How to make Good and Valid Conclusion of Medical Research

Aria Kekalih
Community Medicine Department Epidemiology and Biostatistic Unit

Acknowledgement
L.G.

Portney and M.P.Watkins Bikash Bhatttacharya

Discussion and Conclusion

How

does the author interpret result Did the author clarify if hypothesis were rejected or accepted What alternative explanations does the author consider for the obtained findings How are the findings related to prior report What limitations are described? Are there limitations that are not addressed?

Discussion and Conclusion

If

the result are not significant, does the author consider the possibility of type II error? Regardless of the statistical outcome, are the result clinically important? Does the author discuss how the result apply to practice

Discussion and Conclusion

Does

the author present suggestions for further study Do the stated conclusion flow logically from the obtained result?

Conclusion Validity
Conclusion validity is the degree to which conclusions we reach about relationships in our data are reasonable

Conclusion Validity
Conclusion validity is the degree to which the conclusion we reach is credible or believable
A statistically significant result OR A nonsignificant result Statistically labeled - Often Misunderstood, Least Considered Relevance in Qualitative Research as well as Quantitative Research

Conclusion Validity
Quantitative Example
An inventor has developed a new, energy-efficient lawn mower engine. The inventor claims that the engine will run continuously for 5 hours (300 minutes) on a single gallon of regular gasoline

Qualitative Example
Accountability as practices in our primary health care system creates an undesirable atmosphere of anxiety among nurses

Validity- How they differ?

Internal Validity Conclusion Validity How Credible?

Program Cause-Effect Relation What you do What you See

Accept / Reject

Construct Validity

External Validity

Threats to Conclusion Validity

The primary threat to conclusion validity is the possibility of making an error in the inference process concerning the relationship between the program and the outcome(s) of the program

Error Types
Type I error: Conclude there is a relationship when in fact there is not (concluding/seeing an effect that in reality is not there) Type II error: Conclude there is no relationship when in fact there is a relationship (miss a true effect)

Fishing & the error rate problem

Looking for a specific result by analyzing the data repeatedly under slightly differing conditions or assumptions Conducting multiple analyses and treating each one as though it was independent without error rate adjustment Likely to see a relationship when there isn't one when you keep reanalyzing your data and don't take that fishing into account when drawing your conclusions

Signal-to-noise ratio problem

Is the ratio of Signal Noise
"noise" consists of factors that make it hard to see the relationship low reliability of measures poor reliability of treatment implementation random irrelevancies in the setting random heterogeneity of respondents signal" amount of information collected and the amount of risk taken for decision low statistical power

Problems that can lead to either conclusion error

Assumptions behind analysis when violated - likely to draw erroneous conclusions about relationships
Quantitative Research Example

Assumption that data is normally distributed is violated

Quantitative Research Example

Assumption that the respondent is free to say anything but under pressure from supervisors respond in a particular to

Improving Conclusion Validity

Good Statistical Power Sample size : Collect more information -- use a larger sample size Effect size : Improve the impact of the program relative to the noise Alpha Level : Increase your risk of making a Type I error Power : Ability to see effect thats there Good Reliability Consistency and Repeatability for measures, reducing situational distractions in the measurement context Good Implementation Training program operators and standardizing the protocols for

administering the program

The Four Components to a Statistical Conclusion

Sample size Amount of information
Effect size
Impact of program

Willingness to risk being wrong in Alpha level finding an effect (rejecting the null hypothesis)

Power Ability to see effect thats there

Given Values for Any Three, Possible to Compute the Fourth

= f(effect size, , power) effect size = f(n, , power) = f(n, effect size, power) power = f(n, effect size, )
n

Statistical Conclusions
Statistical conclusions involve constructing two mutually exclusive hypotheses, termed the null (labeled H0) and alternative (labeled H1)

H0:

Program Effect = 0 H1: Program Effect <> 0

The Decision Matrix

In reality

What we conclude

The Decision Matrix

In reality
H0 (null hypothesis) true Alternative H1 false In reality...
There is no real program effect There is no difference, gain Our theory is wrong

What we conclude

The Decision Matrix

In reality
H0 (null hypothesis) true Alternative H1 false In reality...
There is no real program effect There is no difference, gain Our theory is wrong

What we conclude
Accept null Reject alternative We say...
There is no real program effect There is no difference, gain Our theory is wrong

The Decision Matrix

In reality
H0 (null hypothesis) true Alternative H1 false In reality...
There is no real program effect There is no difference, gain Our theory is wrong

What we conclude

Reject null Accept alternative We say...

There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix

In reality
H0 (null hypothesis) true Alternative H1 false In reality...
There is no real program effect There is no difference, gain Our theory is wrong

What we conclude

Reject null Accept alternative We say...

There is a real program effect There is a difference, gain Our theory is correct

TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none
# of times out of 100 when there is no effect, well say there is one

The Decision Matrix

In reality
H0 (null hypothesis) false Alternative H1 true In reality...
There is a real program effect There is a difference, gain Our theory is correct

What we conclude

The Decision Matrix

In reality
H0 (null hypothesis) false Alternative H1 true

What we conclude
Accept null Reject alternative We say...
There is no real program effect There is no difference, gain Our theory is wrong

In reality...
There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix

In reality
H0 (null hypothesis) false Alternative H1 true

What we conclude
Accept null Reject alternative We say...
There is no real program effect There is no difference, gain Our theory is wrong

In reality...
There is a real program effect There is a difference, gain Our theory is correct

TYPE II ERROR The odds of saying there is no effect or gain when in fact there is one
# of times out of 100 when there is an effect, well say there is none

The Decision Matrix

In reality
H0 (null hypothesis) false Alternative H1 true

What we conclude

In reality...
There is a real program effect There is a difference, gain Our theory is correct

Reject null Accept alternative We say...

There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix

In reality
H0 (null hypothesis) false Alternative H1 true

What we conclude

In reality...
There is a real program effect There is a difference, gain Our theory is correct

Reject null Accept alternative We say...

There is a real program effect There is a difference, gain Our theory is correct

1- POWER The odds of saying there is an effect or gain when in fact there is one
# of times out of 100 when there is an effect, well say there is one

The Decision Matrix

In reality
H0 (null hypothesis) true Alternative H1 false In reality...
There is no real program effect There is no difference, gain Our theory is wrong

H0 (null hypothesis) false Alternative H1 true In reality...

There is a real program effect There is a difference, gain Our theory is correct

What we conclude
Accept null Reject alternative We say...
There is no real program effect There is no difference, gain Our theory is wrong

1- THE CONFIDENCE LEVEL The odds of saying there is no effect or gain when in x fact there is none
# of times out of 100 when there is no effect, well say there is none

TYPE II ERROR The odds of saying there is no effect or gain when in fact there is one
# of times out of 100 when there is an effect, well say there is none

Reject null Accept alternative We say...

There is a real program effect There is a difference, gain Our theory is correct

TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none
# of times out of 100 when there is no effect, well say there is one

1- POWER The odds of saying there is an effect or gain when in fact there is one
# of times out of 100 when there is an effect, well say there is one

The Decision Matrix

In reality
H0 (null hypothesis) true Alternative H1 false In reality...
There is no real program effect There is no difference, gain Our theory is wrong

H0 (null hypothesis) false Alternative H1 true In reality...

There is a real program effect There is a difference, gain Our theory is correct

What we conclude
Accept null Reject alternative We say...
There is no real program effect There is no difference, gain Our theory is wrong

1-

THE CONFIDENCE LEVEL

TYPE II ERROR

Reject null Accept alternative We say...

There is a real program effect There is a difference, gain Our theory is correct

1-

TYPE I ERROR

POWER

The Decision Matrix

In reality
H0 (null hypothesis) true Alternative H1 false In reality...
There is no real program effect There is no difference, gain Our theory is wrong

H0 (null hypothesis) false Alternative H1 true In reality...

There is a real program effect There is a difference, gain Our theory is correct

What we conclude
Accept null Reject alternative We say...

1-

THE CONFIDENCE LEVEL

CORRECT

TYPE II ERROR

There is no real program effect There is no difference, gain Our theory is wrong

Reject null Accept alternative We say...

There is a real program effect There is a difference, gain Our theory is correct

1-

TYPE I ERROR

POWER
CORRECT

Thank you

Questions ?