Critical Appraisal of Articles on Therapy

Rani S. Gereige, M.D., MPH

Associate Professor Associate Director, Residency Program University of South Florida

Learning Objectives

Are the results likely to be valid? Are the valid results important? (Magnitude) Applying the valid, important results to a particular patient

The best study design for Therapy studies is

RCT

Are the Results Likely to be Valid?

Was

the assignment of patients to treatment randomized? was the method of randomization?

patients and clinicians kept blind to treatment?

What

Were

Key Concepts

Randomization Blinding Control/Comparison Internal Validity External Validity = Generalizability

Study of Any Intervention

Random Error = Chance deviation from the

underlying truth Bias = Systematic deviation from underlying truth

Limitations for Participation

Exclusion Criteria
By restricting the heterogeneity of the group, we reduce the opportunity for differences in outcome that arent due to the treatment itself Improves INTERNAL VALIDITY Makes generalization of the results more precise but limits EXTERNAL VALIDITY to a smaller portion of the population

Population sampled
Conclusions

Local population

General population

External Validity (Generalizability)

Need a Comparison

Control group

Minimizes the Hawthorne effect

By virtue of being in a study, the patients behavior changes and has a better prognosis Observation group

Still may have a placebo effect unless placebo given to control group
Beyond the Hawthorne effect Giving a pill with an expected/potential result can provide effect even if the pill is inert

Selection of Control Group

Attempts to minimize selection bias

Restriction: limits the range of characteristics of the patients in the study Matching: for each patient in the study group, select one or more patients with the same characteristics for a comparison group Adjustment: mathematical corrections to create an equal weight for dissimilar characteristics Stratification: compare outcomes from subgroups of each group with similar characteristics (i.e. age by decades) Randomization of the study population

Clinical Epidemiology The Essentials. 3rd Ed. Fletcher et al. 1996, p 129.

Are the 2 Groups Comparable?

Selection bias/Confounding bias

Known and unknown confounding variables

Want equal prognostic characteristics Randomize > matching Stratified randomization schemes Can verify effectiveness of allocation

Potential Bias Strategy Against

Sampling Bias Selection Bias Placebo Effect

Cointerventions
Assessment Bias Follow Up

Target population Randomization Placebos and blinding participants Blinding providers, treatment protocols Blinding providers Ensuring completeness

Study of Any Intervention

Bottom Line:

The 2 groups should be the same at the start

Concealed randomization Baseline characteristics

The 2 groups should be treated exactly the same during the study except for the intervention being studied
Tests Non-study treatments Blinding

Variable Regression Analysis

If the baseline prognostic characteristics are not equal in the groups, you can estimate the impact that each variable had on the results through statistical analysis Univariate vs Multivariate regression analysis You may have different prognostic characteristic that had no impact

Guyatt, et al Users Guides to the Medical Literature. AMA Press. 2002: 526-7

Evaluating Studies of Therapy

Are the results valid?

Was the assignment of patients to treatment randomized and was randomization concealed? Was follow-up of patients sufficiently long and complete? Were patients and clinicians kept blind? Were the groups treated equally, except for the experimental therapy? Were the groups similar at the start of the trial? Were all patients analyzed in the groups to which they were randomized?

Intention to Treat preserves the value of randomization

How to Use an Article About

Therapy

Intention to Treat: The principle of attributing all

patients to the group to which they were randomized

Preserves the value of randomization Prognostic factors that we do and dont know about will be, on average, equally distributed between the two groups, and the effect we see will be just that due to the treatment assigned

How to Use an Article About

Therapy
Intention to Treat:
Excluding non-compliant patients from the analysis leaves behind those who may be destined to have a better outcome, and destroys the unbiased comparison provided by randomization In many randomized trials, non-compliant patients from both treatment and placebo groups have fared worse than compliant patients Patients too sick to receive a tx shouldnt only count as control cases

Two Ways to Report the Results

Per-Protocol Results analyzed according to the treatment received Intention-to-Treat Results analyzed according to the treatment assigned whether or not they actually received it

Population of patients with a condition

Per-Protocol
52 Dropped out (stopped the study drug)

1152
52 150 + 50

1152 pts 150

Drug X
Unintentional CROSS-OVER

=1000

50 1136 pts Drug A

36 Dropped out (stopped the study drug)

1136
36 50 + 150

=1200

Population of patients with a condition

Intention-to-Treat
52 Dropped out

1152 pts 150 50 1136 pts

Drug X

1152

Drug A
36 Dropped out

1136

Cross-Over

At the end of a RCT of a medication versus a placebo, the groups switch treatments (still blinded) and the trial is repeated If the initial results also occur in the new treatment group (which previously was the placebo group), it lends credibility to the study

Results

Results Placebo

1152 pts

Drug X

Benefit

No Benefit

1136 pts

Placebo

No Benefit

Drug X

Benefit

Intentional CROSS-OVER

Drop outs change prognosis of original group

Experimental Group Control Group

Ave Age = 58

Ave Age = 57

Drop outs change prognosis of original group

Experimental Group Control Group

Ave Age = 51 Ave Age = 72

Ave Age = 57

Intention-to-Treat

Pros Real life effectiveness What the results will be on a population of patients Needed for economic analysis Intended to account for possible bias from not including subjects that would have affected the results If the results still show benefit of treatment despite including patients who never received the treatment, the results may even be an underestimate

Intention-to-Treat

Cons
Assumes the treatment cant be made more tolerable Results dont apply to the individual If the nonadherence is not equal between the two groups, the results may be biased If the treatment effect is harmful, the inclusion of patients that never received the treatment might make it look less harmful

Intention-to-Treat

Bottom Line
Per protocol analysis is biased if enough subjects arent counted Intention to treat likely to create a false negative result of an effect

Biased effect vs unbiased lack of effect

The Importance of Follow Up

(Not the same as Drop Out)
An Example

Trial A Treatment Control

Trial B Treatment Control

# Patients # Lost to follow up

# Deaths in each group

RRR not accounting for lost to follow up

1000 30 30 (3%) (3%) 200 400 (20%) (40%) (.40 .20)/.40 = .2/.4 = .5 (50%)

1000

1000 30 (3%) 30 (3%)

1000 30 (3%) 60 (6%)

(.06 - .03)/.06 = .03/.06 = .5 (50%)

Worst case scenario (sensitivity analysis): assume the worst outcome in the treatment group for those that were lost to follow up
Guyatt, et al Users Guides to the Medical Literature. AMA Press. 2002: 63-64

Trial A Treatment Control

Trial B Treatment Control

# Patients # Lost to follow up

# Deaths
RRR not accounting for lost to follow up RRR including lost to follow up

1000 30 30 (3%) (3%) 230 400 (23%) (40%) 0.2/0.4 = 0.5 (50%) 0.17/0.4 = 0.43 (43%)

1000

1000 30 (3%) 60 (6%)

1000 30 (3%) 60 (6%)

0.03/0.06 = 0.5 (50%) 0.0/0.06 = 0 No Reduction

Evaluating Studies of Therapy

Are the valid results important?

What is the magnitude of the treatment effect? How precise is this estimate of this treatment effect?

Are the results Clinically Significant?

Outcome Present Drug A Outcome Absent B Totals

Control Event Rate CER= c / c + d

Experimental Event Rate EER = a / a + b Absolute Risk Reduction ARR = CER EER Number Needed to Treat NNT = 1 / ARR

A+B

Placebo

C+D

Totals

A+C

B+D

A+B+C+ D

Magnitude of Effect
RR = Relative Risk or Risk Ratio:

ER in the experimental group divided by the ER in the control group

RR = EER / CER

RRR = Relative Risk Reduction: ARR / AR of the Control An estimate of the proportion of baseline risk (CER) that is removed by the therapy
RRR = ARR / CER = [CER EER] / CER

So How Does it Work???

Therapy Decisions Real-Time

Measures of Treatment Effect:

Shortcut Calculation: Downloadable to PDA EBM calculator EBM Calculator - Website Pre-appraised evidence: ACP Journal Club NNT Nomogram NNT tables

EBM Calculator

Deception
RCT: drug 284 v. placebo for sepsis mortality Drug 284 Placebo 800/2000 1200/2000 = 40% = 60% RR = 40% / 60% = 0.67 = 67% RRR = (60 40) / 60 = 33%

RCT: Panaceanex v. placebo for prostate cancer mortality Panaceanex Placebo 4/2000 6/2000 = 0.2% = 0.3%
RR = 0.2% / 0.3% = 0.67 = 67% RRR = (0.3 0.2) / 0.3 = 33%

Deception
RCT: drug 284 v. placebo for sepsis mortality Drug 284 Placebo 800/2000 1200/2000

RCT: Panaceanex v. placebo for prostate cancer mortality Panaceanex Placebo 4/2000 6/2000

400 lives saved 2 lives saved

Number Needed to Treat (NNT)

Drug 284 Placebo 800/2000 1200/2000 = 40% = 60% RRR = 33% ARR = 60% - 40% = 20% = 0.20

NNT = 1/ARR How many people to treat so that 1 event will be prevented If you reduce the numbers to a common denominator of 100, you can see it more easily

Number Needed to Treat (NNT)

Placebo 1200/2000 120/200 60/100 Drug 284 800/2000 80/200 40/100 So, if you say that you can prevent death for 20 people in every 100 you treat: (= 1 in 5) To save one life you need to treat 5 people

ARR = 60% - 40% = 20%

NNT
Instead of saying if I treat X number of patients, I will prevent Y events, you say in order to prevent 1

event, I have to treat Z number of patients

If the ARR is 25%, the NNT is?

NNT
Instead of saying if I treat X number of patients, I will prevent Y events, you say in order to prevent 1

event, I have to treat Z number of patients

If the ARR is 25%, the NNT is?

What about the Panaceanex?

NNT = 1 / ARR RCT: Panaceanex v. placebo for = 1 / 0.1% prostate cancer mortality = 1 / 0.001 Panaceanex Placebo To prevent one event, 4/2000 6/2000 must treat 1000 = 0.2% = 0.3% patients ARR = 0.3% - 0.2% = 0.1%

How to use an Article About

Therapy

Are the results applicable to my patient?

Is our patient so different from those in the study that its results cannot apply? Is the treatment feasible in our setting? What are our patients potential benefits and harms from the therapy? What are our patients values and expectations for both the outcome we are trying to prevent, and the treatment we are offering?

Analyzing the Results

Efficacy

If taken, does the treatment have an effect? AKA: Explanatory (explains the action of the treatment)

Effectiveness

Will the treatment have good results if offered? Takes into account tolerability and harmful effects Better external validity

The End(s)!!!!!