EBM review

B.Barker

Overview of EBM block

week 1 2 3 lecture Intro EBM Basic statistics RCTs Tutorial - no tutorial - Gate article Jackson - practice PICO questions - discuss + interpretation of statistical questions - critical appraisal of RCT - critical appraisal meta-analysis (steroids in meningitis) - critical appraisal cohort study - searching in library - critical appraisal and discuss se/sp & LR Over 2 groups of 18-20

4 5

Meta-analysis Observational studies

6 7 8 9

Searching Diagnostic tests Qualititative study Assignment

Semester 7 Barrier

H&S = 30 marks out of 240 = 8%

EBM = 15 marks other H&S = 15 marks 2 x 2 table calculations will be easy

No calculator necessary

Be able to answer online assessment (as opposed to 2005 barrier & tutorial Qs)

Week 1 Online assessment (S6W2)

27 questions

What is EBM? What is PICO? What type of question is this?

Treatment, prognosis, diagnosis, aeitology, harm, frequency Case-control, case-series, case report, cross-sectional, cohort, randomised control trial, systematic review, metaanalysis

What type of study is this?

What is the ideal & most feasible study design for answering this Q? What is the most reliable study design for interventions? Validity & reliability

What is EBM?

PICO

Population

What are the characteristics of the population?

What disease are you interested in?

Intervention

What do you want to do with this population (e.g. treat, diagnose, observe exposure, screen)? What is the alternative to the intervention (e.g. nothing, placebo, gold standard, different drug, surgery)?

Comparison

Outcome

What is the effect of the intervention? What are the relevant clinical outcomes (e.g. morbidity hospitalisations, death, complications)?

What type of question is this?

Therapy

Diagnosis

Questions of treatment in order to achieve some outcome. May include drugs, surgical intervention, change in diet, counseling, etc. Questions of identification of a disorder in a patient presenting with specific symptoms. Questions of progression of a disease or likelihood of a disease occurring Questions of negative impact from an intervention or other exposure Questions of identification of a disorder in a population not presenting specific symptoms

Prognosis

Harm

Screening

http://library.downstate.edu/EBM2/contents.htm

What type of study is this?

Case-series (report)

Cross-sectional study

Studies that conduct measurements on a group of subjects at one point in time. Cross-sectional studies look at both exposure and outcomes at one point in time and are designed to evaluate associations between risk factors and outcomes in a specific population.

Case-control study

Case control studies are studies in which patients who already have a certain condition are compared with people who do not. For example: a study on which lung cancer patients are asked how much they smoked in the past and the answers are compared with a sample of the general population would be a case control study. Case control studies are less reliable than either randomized controlled trials or cohort studies. Just because there is a statistical relationship between two conditions does not mean that one condition actually caused the other. For instance, lung cancer rates are higher for people without a college education (who tend to smoke more), but that does not mean that someone can reduce his or her cancer risk just by getting a college education. The main advantages of case control studies are:

Cohort study

The strict definition of a cohort study is a study where the participants are enrolled in the study based on their exposure to a putative health modifier (such as smoking or exercise) and followed up over time to compare the effect of the health modifier in the exposed and non-exposed groups.
Sometimes researchers use the term cohort studies to describe a cohort of people who are followed over time. For example, the Framingham study, the British Doctors Cohort Study & the Nurses Health Study are all described as longitudinal cohort studies. A Cohort Study is a study in which patients who presently have a certain condition and/or receive a particular treatment are followed over time and compared with another group who are not affected by the condition under investigation. For instance, since a randomized controlled study to test the effect of smoking on health would be unethical, a reasonable alternative would be a study that identifies two groups, a group of people who smoke and a group of people who do not, and follows them forward through time to see what health problems they develop. Cohort studies are not as reliable as randomized controlled studies, since the two groups may differ in ways other than in the variable under study. For example, if the subjects who smoke tend to have less money than the non-smokers, and thus have less access to health care, that would exaggerate the difference between the two groups. The main problem with cohort studies, however, is that they can end up taking a very long time, since the researchers have to wait for the conditions of interest to develop. Physicians are, of course, anxious to have meaningful results as soon as possible, but another disadvantage with long studies is that things tend to change over the course of the study. People die, move away, or develop other conditions, new and promising treatments arise, and so on. Even so, cohort studies are generally preferred to case control studies , since they involve far fewer statistical problems and generally produce more reliable answers.

Randomised controlled trial

randomized controlled study is one in which: There are two groups, one treatment group and one control group. The treatment group receives the treatment under investigation, and the control group receives either no treatment or some standard default treatment. Patients are randomly assigned to all groups. Assigning patients at random reduces the risk of bias and increases the probability that differences between the groups can be attributed to the treatment. Randomized controlled trials are the standard method of answering questions about the effectiveness of different therapies. With certain research questions, randomized controlled studies cannot be done for ethical reasons.

Double blinding

A double blind study is one in which neither the patient nor the physician knows whether the patient is receiving the treatment of interest or the control treatment

Systematic review with meta-analysis

A systematic review is a comprehensive survey of a topic in which all of the primary studies of the highest level of evidence have been systematically identified, appraised and then summarized according to an explicit and reproducible methodology. A meta-analysis is a survey in which the results of all of the included studies are similar enough statistically that the results are combined and analyzed as if they were one study.

What is the most reliable study design for interventions?

Validity vs. reliability

Valid & reliable Reliable BUT not valid

Valid BUT not reliable

NOT valid or reliable

Week 2 Online assessment (S6W3)

15 questions

Assess the role of chance

P-value CI

Statistical & clinical significance Measures of effect

NNT ARR RRR

If a difference exists between two groups

Chance

Systematic error

Assessing chance: Traditional approach

1. Hypothesis testing Provides a decision about an observed difference P < 0.05 = statistically significant = result is not due to chance = the hypothesis is true = reject the null hypothesis H0 (i.e. no difference between groups)

Statistical significance does not necessarily mean that the effect is real: by chance alone about one in 20 significant findings are spurious

P > 0.05 = not statistically significant = result is due to chance = cannot reject H0

Non-significance does not mean no effect. Small studies will often report non-significance even when there are important, real effects which a large study would have detected

Move away from using P-values

Arbitrary yes/no rule for rejecting H0 Do not provide information on the degree to which the results would vary if measured multiple times (i.e. precision)

Assessing chance New approach

2. 95% Confidence intervals Report a range of plausible results within which the true effect is likely to lie Indicate how precise the estimate is narrow (large study high power) vs. wide 95% of the time CIs contain the true value of the variable of interest

Contains p-value information

Note. for 95 out of 100 studies, the CI contains the truth (and 5 times out of 100 it does not) If CI does not include value of no difference = significant Value of no difference depends on effect measure

Subtraction (ARR): the value of no difference = 0 Ratio (RR, OR): the value for no difference = 1

Confidence intervals go with measures of effect!

Measures of effect

Relative risk

The risk of an event in the exposed/treated group divided by the risk of an event in the control group RR = EER/CER Cohort studies & RCTs

RR = 1 = no difference in risk between the two groups RR of < 1 = event is less likely to occur in the experimental group than in the control group RR of > 1 = event is more likely to occur in the experimental group than in the control group

2 x 2 table

Relative risk = EER/CER EER= a/(a+b)

Probability of developing lung cancer in smokers

Probability of developing lung cancer in non-smokers

CER = c/(c+d)

Odds ratio

Odds of an event in the exposed/treated group divided by the odds of an event in the control group Case-control studies

RR & ORs are similar but not the same

Measures of effect

When presented with a difference need to know starting point!!

Absolute risk reduction Absolute change in risk: the risk of an event in the control group minus the risk of an event in the treated group ARR = CER-EER Absolute risk is always a subtraction Relative risk reduction Proportion of the risk removed by treatment: absolute risk reduction divided by the initial risk in the control group RRR = ARR/CER A relative risk is always a ratio

Measures of effect

Number needed to treat

Number of patients who need to be treated to prevent one event; this is the reciprocal of the risk to treat absolute risk reduction; round up to whole number NNT = 1/ARR
The best NNT would be 1 = Everyone gets better with treatment = antibiotics for susceptible organisms Higher NNTs represent less effective treatment

It is very important to note..

Chance is rarely the most pressing issue

Easy to overemphasize the importance of statistical significance

Biggest threat to a study is not random error (chance) BUT systematic error (bias) Thus, readers must focus on the more difficult questions when determining if the result is correct:

Are these the right patients? Are there measurement biases? Are observed associations confounded by other factors?

Finally readers must consider whether the result is clinically important

Week 3 Online assessment (S6W4)

15 questions

RAMBBO

Double blinding Quality of randomisation Intention to treat (ITT) analysis Were both groups treated equally except for intervention? Loss to follow up

Week 4 online assessment (S6W5)

8 questions

Where to start searching for a systematic review

Obscure topic = pubmed clinical queries Something you suspect there is a lot of work on = cochrane library

Forest plot interpretation What is a systematic review

Week 5 online assessment (S6W6)

22 questions

Define/characteristics/advantages/what type of study is this?

Case-control study Case report Case series Cross-sectional Cohort

Cross-Sectional Survey

A study that examines the relationship between diseases (or other health-related characteristics) and other variables of interest as they exist in a defined population at one particular time (ie exposure and outcomes are both measured at the same time). Best for quantifying the prevalence of a disease or risk factor, and for quantifying the accuracy of a diagnostic test. Advantages:

Disadvantages:

cheap and simple; ethically safe.

establishes association at most, not causality; recall bias susceptibility; confounders may be unequally distributed; Neyman bias; group sizes may be unequal.

http://www.cebm.net/index.aspx?o=1039

Case-Control Studies

Patients with a certain outcome or disease and an appropriate group of controls without the outcome or disease are selected (usually with careful consideration of appropriate choice of controls, matching, etc) and then information is obtained on whether the subjects have been exposed to the factor under investigation. Advantages:

Disadvantages:

quick and cheap; only feasible method for very rare disorders or those with long lag between exposure and outcome; fewer subjects needed than cross-sectional studies.
reliance on recall or records to determine exposure status; confounders; selection of control groups is difficult; potential bias: recall, selection.

Cohort Study

Data are obtained from groups who have been exposed, or not exposed, to the new technology or factor of interest (eg from databases). No allocation of exposure is made by the researcher. Best for study the effect of predictive risk factors on an outcome. Advantages:

Disadvantages:

ethically safe; subjects can be matched; can establish timing and directionality of events; Eligibility criteria and outcome assessments can be standardised; administratively easier and cheaper than RCT. controls may be difficult to identify; exposure may be linked to a hidden confounder; blinding is difficult; randomisation not present; for rare disease, large sample sizes or long follow-up necessary

Randomised Controlled Trial

An experimental comparison study in which participants are allocated to treatment/intervention or control/placebo groups using a random mechanism (see randomisation). Best for study the effect of an intervention. Advantages:

Disadvantages:

unbiased distribution of confounders; blinding more likely; randomisation facilitates statistical analysis.
expensive: time and money; volunteer bias; ethically problematic at times.

Week 6 online assessment (S6W7)

25 questions

PICO how to formulate searchable questions? Type of question?

Aeitology, diagnosis etc

Best study design for following question Search tips e.g. * in pubmed MESH headings what are they Best place to start searching?

Cochrane vs. clinical queries vs. pubmed

Week 7 online assessment (S6W8)

29 questions LO: How to understand diagnostic studies

Positive & negative predictive values Sensitivity, Specificity Likelihood ratio ROC curve & area under the ROC

Receiver Operating Characteristics curve

The diagonal line (representing Sensitivity=0.5 and Specificity=0.5) represents performance no better than chance
Overall shape is predicted by the reciprocal relationship between sensitivity and specificity The closer the curve gets to Sensitivity=1 and Specificity=1, the better the overall performance of the test Hence the area under the curve gives a measure of the tests performance

ROC curve

FALSE POSITIVE RATE (1-Specificity)

AREA UNDER ROC CURVES

100

Sensitivity

AREA=1.0
0 1-Specificity

Sensitivity and specificity both 100% - TEST PERFECT Sensitivity and specificity both 50% - TEST USELESS

100

Sensitivity

AREA=0.5
0 1-Specificity

The area under a ROC curve will be between 0.5 and 1.0

Week 8 online assessment (S6W9)

13 questions LO: Qualitative vs. Quantitative

Purposive sampling Triangulation

likelihood ratios: the answer?

Likelihood ratios LRs
Helps

rule-in or rule-out disease Does not vary with prevalence

Likelihood Ratio

LR is relative frequency of a feature in the disease versus the non-diseased: P (Result for Disease) LR = P (Result for Non Disease)
useful if... ~ >10 <0.1

LR = 1 neutral LR > 1 evidence for disease LR < 1 evidence against disease

another way of seeing it A new diagnostic test for delirium...

The likelihood of a positive test result in someone with the condition is 32/36 = 90%
This is the same as the sensitivity

Likelihood Ratio

The likelihood of a positive test result in someone without the condition is 12/68 or 0.18
This is the same as (1-specificity)

Likelihood Ratio

Likelihood = ratio
=

likelihood of +ve test given the disease likelihood of +ve test in the absence of the disease sensitivity 1- specificity = 5.0