EVALUATION OF EVIDENCE

Berhe Beyene (Email: berhebeyene2005@gmail.com)

Berhe Beyene
 Purpose of Evaluation of Evidence

 To determine if what is observed is a reflection

of the truth?

Observed:
Prevalence Are they true?
Incidence Are there
Relative Risk alternative
Odds Ratio… explanations?

Berhe Beyene
Common Problems in observed findings

• Inadequacy of the observed sample

• Inappropriate selection of study subjects

• Inappropriate/unfair data collection

methods

• Comparing unequals

Berhe Beyene
 Accuracy
Accuracy = Validity + Precision

Validity - is the degree of closeness to the truth

Precision - is usually related to sampling variation

Berhe Beyene
Validity Vs Reliability

Berhe Beyene
 Precision
• Precision in measurement and estimation
corresponds to the reduction of random error.
– mostly related to sampling variation or sampling
error.

Solution
• Increase sample size
• Improving the efficiency of measurement

Berhe Beyene
Alternative explanations for the observed association other
than cause and effect relationships

A) The association may be the result of chance

B) The association may be the result of bias
C) The association can be the result of a
confounding effect.
D) An apparent cause can be an effect, rather than
a cause (reverse causation)
E) The cause can be both a cause and effect
(reciprocal causation)
e.g Vitamin A deficiency can cause diarrhoea or
diarrhoea can cause Vitamin A deficiency

Berhe Beyene
 Validity of epidemiological studies

Two types of validity - internal and external.

A. Internal validity - is the degree to which the

results of the study are correct for the particular
group of people studied
 
b. External validity (generalizability) - is the extent
to which the results of the study apply to people
not in it.

Berhe Beyene
Validity of epidemiological studies cont…
Judgment about causality should address 2 major areas:
a) whether the observed association between exposure
and disease is valid for any individual study
b) whether the totality of evidence taken from a number
of sources supports the findings of this study
 
First assess whether for any individual study the
observed association is valid ( check the role of chance,
bias and confounding) then assess other supportive
evidences

Berhe Beyene
 The role of chance

• we can draw inferences about the experience of

an entire population based on an evaluation of
only a sample.
• One of the major problems in drawing such
inference is that the play of chance may always
affect the results observed simply because of
random variation from sample to sample.
• One of the major determinants of the degree to
which chance affects the findings in any
particular study is sample size.
Berhe Beyene
 The role of chance cont…

• the smaller the sample , the more variability

there will be in the estimates and the less likely
the findings will reflect the experience of the
total population.
• the larger the sample on which the estimate is
based, the less variability and the more reliable
the inference.
• It is important to quantify the degree to which
chance variability may account for the results
observed in any individual study.
Berhe Beyene
 The role of chance cont…
• This is done by performing an appropriate test
of statistical significance.
• A measure that is often reported from all tests
of statistical significance is the P value.

Berhe Beyene
 The role of chance cont…
• P < 0.05 - statistically significant.
• P> 0.05 – not statistically significant ( chance
can not be excluded as a likely explanation)

Berhe Beyene
 The role of chance cont…
• It is always advisable to report the actual P
value rather than merely that the results did
or did not achieve statistical significance.
• confidence interval (CI) is far more
informative measure than P value to evaluate
the role of chance

Berhe Beyene
 Confidence Interval
1. Provide information that p-value gives.
– If null value is included in a 95% confidence
interval, by definition the corresponding P-
value is >0.05.

2. Indicate the amount of variability (effect

of sample size) by the width of the
confidence interval.
– This information can not be obtained from p-
value.

Berhe Beyene
 Interpretation of CI
 width of CI

Indicate greater variability

suggest inadequacy of the sample size

Particularly important in interpreting non-significant results

Narrow CIsuggest that truly there is no association

Wide CI suggest inadequacy of sample size to have adequate statistical power

Berhe Beyene
 Random Error Vs Bias
Random Error:
• Inaccuracy which is similar
in comparison groups
produced by biological
variations, measurement
variations, error during
recording… (non-
differential/random error)
• Effect is the same for all
groups in the study
Berhe Beyene
Bias:
• Inaccuracy which is
different in size and
direction in one group
than the other
(differential error).
• Effect is not the same
for groups in the study
 The role of bias
• Bias - is any systematic error in the design,
conduct, or analysis of a study that results in a
distorted estimate of what the study is
attempting to measure.
• The key word in the understanding of the
concept of bias is “different”.
• If the way in which participants are selected into
the study is different for cases and controls, for
example, and that difference is related to their
exposure status, then the possibility of a bias
exists in assessment of association between the
exposure and disease.

Berhe Beyene
 The role of bias cont…
Two main types of bias:
A. Selection bias
B. Information (Observation) bias

Berhe Beyene
 Selection Bias (examples)

• Invitational (who gets invited into the study?)

– Berkson's bias
– Incidence-prevalence bias (missing deaths and
recovered cases)
– Healthy worker bias

• Acceptance (who accepted the invitation?)

– Loss to follow-up
– Volunteer/Compliance bias
– Non-response bias

Berhe Beyene
 The role of bias cont…

• Selection bias is a particular problem in case

control and retrospective cohort studies,

• In all circumstances where selection bias does

occur the result is an observed relation between
exposure and disease that is different among
those who are entered into the study than
among those who would have been eligible but
were not chosen to participate.
Berhe Beyene
 Examples of selection bias

Berkson’s bias
Case control studies carried out exclusively
in hospital settings are subject to selection
bias attributable to the fact that risks of
hospitalization can combine in patients
who have more than one condition

Berhe Beyene
 Examples of selection bias cont…
Healthy worker bias
refers to the bias in occupational health
studies which tend to underestimate the
risk associated with an occupation due to
the fact that employed people tend to be
healthier than the general population

Berhe Beyene
 Examples of selection bias cont…

• Healthy worker bias can occur when general

population is used as a comparison group

• preexisting population data on disease

incidence & mortality is used to compare with
data from the exposed group

Berhe Beyene
 Examples of selection bias cont…
• E.g -to evaluate potential risks associated with
occupation in the rubber industry, the
mortality experience of a group of rubber
workers at a tire manufacturing plant in
Akron, Ohio, was compared with mortality
rates for the U.S population of the same age
and sex.
• Result - all-cause mortality rate for the rubber
workers was only 82 percent of that seen in
the general population
Berhe Beyene
 Examples of selection bias cont…
• Comparison with population rates are possible
only for outcomes for which population rates
are available.

• use of the experience of the general

population as a valid indicator of the
experience of a non-exposed group assumes
that only a small proportion of that population
is actually exposed

Berhe Beyene
 Examples of selection bias cont…
• such a comparison will underestimate the true
association between exposure and disease.

• The major disadvantage of using the general

population as a comparison group is that its
members may not be directly comparable to those
of the study cohort.

• people who are employed are, on average,

healthier than those who are not.
Berhe Beyene
 Examples of selection bias cont…
• general population includes people who are
unable to work due to illness as well as those
who are employed

• rates of disease and death among the general

population are almost always higher than they
are for members of the work force

Berhe Beyene
 Examples of selection bias
Diagnostic bias
• Diagnostic bias occurs when a disease is more likely to
be diagnosed in some one with exposure to a
suspected risk factor.
• E.g. Women who take oral contraceptives (OCs) may
be screened more often for breast cancer than women
who do not take OCs because of the suspected link
between oral contraceptive and breast cancer.
• This would result in breast cancer being diagnosed
more readily in those who are exposed to Ocs.
• In turn this would introduce a bias in that exposed
cases may be more likely to come to medical attention
and be included in a study than non-exposed cases.
Berhe Beyene
 Examples of selection bias cont…
Self selection/ Volunteer bias/ Compliance bias
People who accept to participate in a study, or people who
refuse to participate are often quite different from the
general population.

Non-response bias
• This is due to differences in the characteristics between the
responders and non-responders to the study.
• Non-response reduces the effective sample size, resulting
in loss of precision of the survey estimates.
• Rates of response in many studies may be related to
exposure status.

Berhe Beyene
 Examples of selection bias cont…

Loss to follow up
• Major source of bias in cohort studies
• Also a problem in intervention studies
• Relates to the necessity of following individuals for a period
of time after exposure to determine the development of
the outcome
• If the proportion of losses to follow-up is large, >20%, this
would certainly raise serious doubts about the validity of
the study results.
• the more difficult issue for interpretation is that even if the
rate of loss is not that extreme, the probability of loss may
be related to the exposure, to the outcome, or to both.

Berhe Beyene
 Ways of minimizing selection bias

1) Population-based studies are preferable

2) The selection of hospitalized controls in a

case control study will increase comparability
with the cases in terms of willingness to
participate, the presence of selective factors
that influenced the subjects’ choice of a
particular hospital, and awareness of
antecedent exposures and events.

Berhe Beyene
 Ways of minimizing selection bias cont…

3) One should avoid the inclusions as study

subjects of people who have volunteered on
their own to participate in the study
4) In case-control study, it is useful to select
several different control groups, including if
possible a group selected in the community.

Berhe Beyene
 Ways of minimizing selection bias cont…

5) In hospital-based case control study, controls are

usually selected among patients with diseases other
than the disease studied.
-One should ensure that these other diseases are not
related to the exposure & the disease of interest.
6) keep losses to follow-up to an absolute minimum.
• For those who are lost, an assessment of as much
outcome data as can be independently determined by
the investigator should be made.
• This would include, at the very least, an assessment of
mortality status using different sources
 
Berhe Beyene
 Ways of minimizing selection bias cont…
• Previously collected data can also be examined to
determine whether there are systematic
differences in the exposure or other risk factors
between those whose outcome is known and
those who have been lost to follow-up.
•  An indirect approach used to describe the extent
of bias introduced by losses to follow up is to
calculate estimates of the exposure-disease
association assuming the most extreme situations
with respect to both exposure and outcome.

Berhe Beyene
 Ways of minimizing selection bias cont…

• One estimate would be based on the

assumption that all those who were lost to
follow up developed the outcome of interest,
while the other would assume that none
developed it.
• The results of these calculations provide a
range within which the true association will
lie.

Berhe Beyene
B. Information bias /Observation bias

This refers to bias which arises during the data

collection process, because of mistakes in
categorizing study subjects with respect to
their exposure or disease status

Berhe Beyene
 Information bias cont…
Investigator bias/ Interviewer bias/ Observer bias
Occurs when investigators collect information
differently in different comparison groups
Recall bias
Occurs as a result of difficulty to recall prior exposures
Social desirability bias
Occurs because subjects are systematically more likely
to provide a socially acceptable response.
Placebo effect
In experimental studies which are not placebo
controlled, observed changes may be ascribed to the
positive effect of the subjects belief that the
intervention will be beneficial.
Berhe Beyene
 Placebo
• Placebos are inert treatments intended to have no effect other
than the psychological benefit of offering treatment.

• Can only be used if there is no accepted treatment for the

condition under study.

• Use of placebo minimizes the bias in the ascertainment of both

subjective disease outcomes and side effects.

• It facilitates that both groups in the study gain equal attention.

 

Berhe Beyene
 Ways of minimizing information bias
1. Blinding.
A. Single blind
The study subjects doesn't know to which group they
are assigned
B. Double blind
Neither the study subjects nor the data collector know
the group to which the subject has been assigned
C. Triple blind
• The study subjects, the data collector and the
individual who is doing the analysis are ignorant of the
group to which subjects belong
 Blinding is of greatest importance when the outcome is
subjectively determined.

Berhe Beyene
Ways of minimizing information bias cont.
2. same standard procedures, instruments,
questionnaires, interviewing techniques etc
should be used for data collection in both
comparison groups
3. Classification of study subjects according to their
outcome & exposure status should be based on
the most objective & accurate methods available
4. when exposure status is determined by interview,
it should be assessed in several different ways for
both groups, so as to assist all study subjects to
make a thorough attempt at recall

Berhe Beyene
 The role of confounding
• Confounding is distortion of the estimated effect of
an exposure on an outcome, caused by the presence
of an extraneous factor associated both with the
exposure and the outcome,

• Confounding variable, confounder is a variable that

can cause or prevent the outcome of interest, is not
an intermediate variable, and is associated with the
factor under investigation.

Berhe Beyene
 The role of confounding cont…
 To bring a confounding effect, that confounding variable
must fulfill each of the following criteria

1) the variable must be associated with the exposure and,

independent of that exposure, be a risk factor for the
disease.

2) The distribution (frequency) of the confounding variable

should vary between the groups that are compared

3) Confounder must not be an intermediate link in a causal

pathway between exposure and outcome
  
 
Berhe Beyene
 The role of confounding cont…

Example of confounding effect.

• An observed association between
consumption of coffee and increased risk of
MI could be due, at least in part, to the effect
of cigarette smoking, since coffee drinking is
associated with smoking and, independent of
coffee consumption, smoking is a risk factor
for MI.

Berhe Beyene
 The role of confounding cont…
• Confounding is not an all or none condition
described merely by its presence or absence
• Confounding is a quantitative issue
 The role of confounding cont…

It is vital to ask the following questions

1) what is the magnitude or extent of
confounding?
• This can be answered by using the following
formula
• Magnitude of confounding =
(RR crude –RR adjusted)/RR adjusted)
2. Direction of confounding
 The role of confounding cont…

Direction of confounding
• Confounding pulls the observed association
away from the true association
• It can either exaggerate the true association
(positive confounding) or
• Hide the true association (negative
confounding)
 The role of confounding cont…
• Observed finding: statistical significant association between
increased level of physical activity and decreased risk of MI
• Is age a confounder?
• People who exercise heavily are younger than those who do
not exercise
• Independent of exercise, younger individuals have a lower
risk of MI than older people
• Those who exercise could have a lower risk of MI just simply
as a consequence of the greater proportion of younger
individuals in this group
• Age would confound the observed association between
exercise and MI & result in an over estimate of any inverse
relationship
 The role of confounding cont…
RR
Exercis
e

Exercise +
True
more younger RR=0.5
people in the
exerciser
group

RR=0.3
1

Overestimation of the effect of exercise on reducing MI

 The role of confounding cont…
• Observed finding: statistical significant association between
increased level of physical activity and decreased risk of MI
• Is sex a confounder?
• A high level of exercise is likely to be more common in men
• Independent of exercise, men have a greater risk of MI than
women
• An inverse effect of exercise on risk of MI would be
underestimated if differences in gender between exercisers
and non exercisers were not taken in to account
The role of confounding cont…
RR
Exercis
e
True
RR=0.5 Exercise +
more men in
the exerciser
group

RR=0.7
1
Underestimation of the effect of exercise on reducing MI
 The role of confounding cont…
• Failure to control for negative confounding results in an
observed estimate of effect that is diluted towards the null
• i.e any true increased or true decreased risk will be
underestimated and appear as less of an association than is
actually the case

• Failure to control for positive confounding results in a more

extreme estimate of effect observed than is actually the case
• This can be in the direction of either an apparently stronger
increased risk or an apparently more protective, depending
on the direction of true association
 The role of confounding cont…
• Example of positive confounding is when the
true RR is 2 and the confounded RR is 3.0
• Example of the negative confounding is when
the true RR is 2 and the confounded RR is 1
 Control for Confounding Variables
• In the design:
–  Randomization
– Restriction
– Matching

• During analysis:
– Standardization
– Stratification/pooling
– Multivariate analysis

Berhe Beyene
 1. Randomization
• randomization ensure that all potential
confounding factors are evenly distributed
among the treatment groups

• sample size of the study must be

sufficiently large

Berhe Beyene
 2. Restriction

• restrict the admissibility criteria for subjects

and limit entrance into the study to
individuals who fall within a specific category
or categories of the confounder.
• For example, if sex is a potential confounding
factor, the study could include only men or
only women.

Berhe Beyene
 3. Matching

• Randomization & restriction are done during the

design stage
• matching can be done during the design &
analysis stage
•  In matching the particular subjects are selected
in such a way that the potential confounders are
distributed in an identical manner among each of
the study groups
• matching is primarily used in case control studies
Berhe Beyene
 4. Stratified analysis

• used during the analysis stage

• Involves the evaluation of the association within

homogenous categories or strata of the
confounding variable.

Berhe Beyene
 Stratified analysis cont…
Table 1: Data from a case-control study of physical activity and risk of MI, stratified by history of cigarette smoking

Smoking Physical activity Cases Controls Total

history Index

Never smoker 2500+ kcals 41 84 125

<2500 kcals 46 52 98 OR=0.55
Total 87 134 223
Smoker 10+ 2500+ kcals 41 80 121
Years <2500 kcals 30 39 69 OR=0.67
Total 71 119 190
Smoker < 10 2500+ kcals 22 34 56
Years <2500 kcals 21 26 47 OR=0.80
Total 43 60 103
Current smoker2500+ kcals 86 68 154
<2500 kcals 79 40 119 OR=0.64
Total 165 108 273
Total 2500+ kcals 190 266 456
<2500 kcals 176 157 333 OR=0.64
Total 366 423 789

Berhe Beyene
 Stratified analysis cont…
• Formula for the pooled estimate of the OR
for case control study
• ORMH =ad/T
bc/T
• Where the quantities in the numerator
and denominator are summed separately
over each of the individual strata
Berhe Beyene
 Stratified analysis cont…

• Once differences in cigarette smoking have

been taken into account, the OR of MI
associated with high levels of physical
activity is 0.64.

• This estimate is adjusted for, or

unconfounded by, cigarette smoking.

Berhe Beyene
 Stratified analysis cont…
• The magnitude of confounding in any study is
evaluated by observing the degree of
discrepancy between the crude and adjusted
estimates
• Magnitude of confounding =(OR crude –OR adjusted)/OR adjusted)
• The fact that the crude and adjusted OR
estimates in the example above are identical
indicates that there was no confounding

Berhe Beyene
 Illustration of Confounding (1)

Examine results of a study of lung cancer and smoking.

Lung Cancer
Yes No
Smoking Yes 260 820
No 840 1380 OR=0.52

Berhe Beyene
 Illustration of Confounding (2)

Examine results of a study of lung cancer and smoking.

Lung Cancer
Yes No
Smoking Yes 260 (60) 820(800)
No 840(40) 1380 (1200) OR = 0.52

Numbers in brackets indicate exposure to asbestos

Is asbestos a confounding variable?

Berhe Beyene
 Stratified Analysis (Calculating Pooled OR)
Both Groups
Lung Ca+ Lung Ca- ORMH = aidi/Ni
Smoking+ 260 820 bici/Ni
Smoking - 840 1380 i = strata
OR= 0.52 N= total population in strata

Exposed to asbestos Not exposed to asbestos

Lung Ca+ Lung Ca- Lung Ca+ Lung Ca-
Smoking+ 60 800 Smoking+ 200 20
Smoking - 40 1200 Smoking - 800 180
OR= 2.25 OR= 2.25

ORMH = (60X1200/2100) + (200X180/1200)

(40X800/2100) + (800X20/1200)
= 1.17

Berhe Beyene
ORMH= Cell with exposed cases+unexposed controls
Cell with exposed controls + unexposed cases

Berhe Beyene
 5. Multivariate analysis
• A fundamental problem with stratified analysis is its
inability to control simultaneously for even a
moderate number of potential confounders

• Multivariate analysis allows for the sufficient

estimation of measures of association while
controlling for a number of confounding factors
simultaneously

Berhe Beyene
 Multivariate analysis cont…
The most common way that many factors are
controlled for simultaneously is through the use of
a multiple regression model.

Multiple linear regression is most appropriately used

when the dependent outcome variable is
continuously distributed (e.g levels of BP).

 
Berhe Beyene
 Multivariate analysis cont…
• In many epidemiologic studies, the outcome of
interest is a binary variable, such as diseased Vs non
diseased.

• In such circumstances it is possible to use a

specialized type of multiple regression called logistic
regression analysis.

Berhe Beyene
 Judgment of causality
• One cannot conclude to a cause-and-effect
relationship from the results of a single observational
study showing an association between an exposure
and a disease.
• Properly conducted experimental trials do provide
more direct proof of cause and effect, yet are usually
not possible because of ethical considerations.
•  In the absence of an experimental trial, establishing
causation is a difficult process, involving the
considerations of a number of criteria.

Berhe Beyene
 Judgment of causality cont…
 The following formal criteria are widely used
to evaluate the likelihood that an association
is causal
1. Strength of the association
• Refers to the relative risk/odds ratio
• the larger the RR, the greater the likelihood
that the factor is causally related to the
outcome

Berhe Beyene
 Judgment of causality cont…
2. Dose-response relationship
• The likelihood of a causal relation is
strengthen if a dose-response effect (gradient)
can be demonstrated
• the risk of disease often increases with
increasing exposure with the causal agent

Berhe Beyene
 Judgment of causality cont…
3. Consistency of the relationship
• This criterion requires that an association
uncovered in one study persist on testing
under other circumstances, with other study
population, and with different study methods
• The more often the association appears under
diverse circumstances, the more likely it is to
be causal in nature

Berhe Beyene
 Judgment of causality cont…
4. Temporal relationship
• exposure to the suspected factor must antedate the
onset of disease and allow for any necessary period
of induction and latency
• Temporal relationships between environmental
factors and outcome are easy to demonstrate for
events such as a food-borne epidemic.
• They are more difficult to establish in many chronic
conditions, especially those with a long latent
interval

Berhe Beyene
 Judgment of causality cont…
5. Specificity of the association
• The association is more likely causal if a single
exposure is linked to a single disease
• The ideal is a one-to-one relationship, where a cause
is both necessary and sufficient
• Specificity is complete where one manifestation
follows from only one cause

Berhe Beyene
 Judgment of causality cont…
• The requirement of specificity is less satisfactory
than the first 4 criterion for 2 reasons:
• A single factor can cause more than one disease
• Multifactorial nature of disease. None of the factors
alone is sufficient to produce disease
• While one-to-one specificity is stronger evidence for
causal association, lack of specificity is of less
significance

Berhe Beyene
 Judgment of causality cont…
6. Biological plausibility (coherence with existing information)
• Additional support for the causal nature of an
association exists if a causal interpretation is
plausible in terms of the current knowledge
about the factor and the disease
• (e.g its biology, pathology, natural history,
descriptive epidemiology )

Berhe Beyene
 Judgment of causality cont…
7. Prevention
If the exposure is a cause of the disease, then
eliminating the exposure (or modifying host
response to the exposure, for example
through immunization )should be followed by
a decrease in the incidence rate of the disease

Berhe Beyene
 Judgment of causality cont…
• The above criteria are the ones most frequently
employed in trying to establish causation.
• None provides in itself a perfect means of
providing causation, and each has its limitations.
• However, when they are considered together,
the weight of the evidence may allow a tentative
conclusion to be reached.

Berhe Beyene
Questions ?

Berhe Beyene