Validity of a study

Session objectives

Describe internal and external validity of analytic study

Describe the role of chance and methods to control them

Define and describe the common types of bias and methods

to control them
Describe the methods to control bias
Define and describe confounding and methods to control
them
 Internal and external validity
Internal validity: Refers the extent that observed
differences on the dependent variable is attributable to
the independent variable and is not the result of some
other (confounding) variable.

External validity: Refers to the degree to which study

results can be generalized beyond the setting in which it
is conducted (Target population).
 Validity of a study

An epidemiological study is intended to answer a

study question.

Next step of evaluation of study result is validation

of the findings.

An observed association or finding is validated for

bias, chance and confounding
 Epidemiologic Reasoning

Finish

Risk Ratio, Rate

Ratio, Odds Ratio
Start
Analytical
Epidemiology
Determine
Descriptive statistical
Test association
Epidemiology

hypothesis
Form
hypothesis
Soybeans Asthma ED
RR = 23

True association? Chance?

causal Confounding?
non-causal Bias?
 Epidemiologic Reasoning

Finish
R/o Bias, Chance,
Confounding

Risk Ratio, Rate

Start Ratio, Odds Ratio Assess
Analytical validity
Epidemiology
Determine
Descriptive statistical
Test association
Epidemiology

hypothesis
Form
hypothesis
 To show a Valid Statistical Association
S:
Bias: Whether systematic error has been built
into the study design

Role of chance: how likely is that what we

found is a true finding

Confounding: Whether an extraneous factor

is related to both the disease and the exposure
 Chance
Inference: Is the process of making a
generalisation about a larger group of individuals
on the basis of a subset or sample.

There is always the possibility that the inference

will be either inaccurate or imprecise, because of
sampling variability (chance).
 Chance
So, assessing for chance is an important step in making
an inference about general population

How likely is that a result as extreme (as large or as

small) as the one observed, could have occurred simply
by chance ?
 Evaluating the role of chance

1. Hypothesis testing (test of statistical

significance)

2. Estimation of confidence interval

 (1) Hypothesis testing
Requires testing a hypothesis

Ho: Null hypothesis

• No effect or no difference

Ha: Research (alternative) hypothesis

• There is an effect or difference

 Statistical Significance

I believe that Treatment A is better than Treatment B.

Why not test my research hypothesis? Why test the
null hypothesis?
H0 : Treatment A = Treatment B
The research hypothesis requires an infinite number
of statistical tests

If we test the null hypothesis, we only have to

perform one test, that of no difference
 Statistical Significance…
A statistical association tells you the likelihood the
result you obtained happened by chance alone

A strong statistical association does not show cause!

Every time we reject the null hypothesis we risk being

wrong

Every time we fail to reject the null hypothesis we risk

being wrong
 Statistical…
It is conducting of statistical significance, about
sampling variability.

Testing of the role of “null hypothesis ‘Ho’” and

“alternative hypothesis ‘H1’ or ‘HA’”

All tests of statistical significant lead to a probability

of a statement the “P value”

The value of “P” is the probability of obtaining a

result in a study explained by chance alone. Or
 (1) Hypothesis testing
The p-value (probability value) is obtaining a
result at least as extreme as the one observed
in the study, by chance alone, assuming that
there is no association between the exposure and
the outcome in reality (null hypothesis).

The P value set by convention in medical research is at

0.05 (5%)
 (1) Hypothesis testing ...
p-value less than or equal to 0.05 - at most a 5%
chance (1 in 20) chance of observing an
association as large or larger than that found in
the study by chance alone.

chance is an unlikely explanation of the finding.

Reject the null hypothesis

 Hypothesis testing...

p-value greater than 0.05

chance has not been excluded, the null
hypothesis is not rejected and conclude that the
findings are not statistically significant at the 5%
level.
But this does not mean there is no relationship
 Hypothesis testing...
No p-value, however small, excludes chance
completely, even with P< 0.0001.

A large p-value: the observed value is not due

to chance, only that chance cannot be
excluded.
 (2) Confidence intervals

Sometimes we are more concerned with estimating the

true difference than the probability that we are making
the right decision (p-value)

A confidence interval represents the range within

which the true magnitude of an effect lies with a
certain degree of certainty eg 95% certainty.
 (2) Confidence intervals...

The 95% confidence interval provides the interval in

which the true value is likely to be found 95 percent
of the time

If the confidence limit contains 1 (the value of no

difference) we cannot reject the null hypothesis

Larger sample sizes yield smaller confidence intervals

 (2) Confidence intervals...

Confidence interval is more informative of the

magnitude of the effect and variability than simple
“P-value” (especially if it is to look for difference)

Width of a Confidence interval indicates the

variability in the estimate due to sample size

The larger the sample size, the more stable the

estimate, and the narrower the confidence interval
 “A” Statistically significant but wide 95% CI
“B”Statistically significant and with narrow 95% CI

. .
“C” Statistically not significantly associated

.
A
C
B

1 RR/ OR

Null value
 Subjects in the Study
More subjects allows for determining smaller
differences

More subjects yields smaller confidence intervals

More subjects cost more money

More subjects increases the complexity of the project

 Balance the Following
Finding no significant difference in a small study tells us
nothing

Finding a significant difference in a small study may not be

replicable because of sampling variation

Finding no significant difference in a large study tells us

treatments or outcomes are essentially equivalent

Finding a significant difference in a large study reveals a true

difference, but the findings may not be clinically important
 Power of the study
The power of a study is the ability of the study to detect
an effect (increased or decreased RR) if in reality that
effect exists.

– If the study is too small and is under-powered - then

you risk missing a real effect. If the study is too large -
wasteful

• Calculation of sample size Perform at outset of study

1. What effect to detect eg RR of 2.0, 1.5, 3.0 ?

 Power of the study
Comparison group:
– Prevalence of exposure in the control group for case-control
study
– The prevalence of success of standard therapy/placebo in
RCT
– Frequency of disease in the non-exposed group in a cohort

3. Set the level of significance

4. Set the power of the study

– 1 - β is the power i.e the degree of certainty that the RR if
present would be detected.
 Power of the study
β (type II error) is the probability of not detecting a
significant effect when in reality there is one
the risk of a false negative result.
β is often set at 0.2, 0.15 or 0.1 and therefore the power is
80%, 85% or 90%.
The smaller the RR you wish to detect, the higher the level
of stat significance (the higher the α) and the higher the
power (the lower the β), the larger the sample size must
be.
 BIAS

Systematic error (not random) in a study that

leads to an incorrect estimate of the association
between exposure and disease
Can occur in the design or execution of a study

Undesirable
Can’t be ‘adjusted for’

Fundamental in study design

 Bias…
• Systematic error built into the study design

Selection Bias

Information Bias
 Selection Bias
Selection Bias: Error due to a systematic
difference between those selected for a study and
those NOT selected for a study.

distorts the true strength of association

can occur in both case-control and cohort study

designs
difference in way cases vs. controls selected
difference in way exposed vs. non-exposed
selected
 Types of selection bias

Response Bias
Berksonian bias
Diagnostic bias
Self selection/ Volunteer bias/ Compliance
bias
Loss to follow up
 Types of Selection Bias
1. Response Bias:
Those who agree to be in a study may be in some way different from

those who refuse to participate

Volunteers may be different from those who are enlisted.

2. Berksonian Bias:

There may be a spurious association b/n a characteristics and a

disease because of the different probabilities of admission to a hospital

for these with the disease and with out the disease of interest.

Admission criteria of the hospital

 Types of selection bias…
3.Diagnostic bias
Diagnostic bias occurs when a disease is more likely
to be diagnosed in some one with exposure to a
suspected risk factor.

For example women who take oral contraceptives

(OCs) may be screened more often for breast cancer
than women who do not take OCs because of the
suspected link between oral contraceptive and breast
cancer.
 Types of selection bias…

This would result in breast cancer being diagnosed

more readily in those who are exposed to Ocs.

In turn this would introduce a bias in that exposed

cases may be more likely to come to medical attention
and be included in a study than non-exposed cases
 Types of selection bias…

4 .Self selection/ Volunteer bias/ Compliance

bias

People who accept to participate in a study, or

people who refuse to participate are often quite
different from the general population.
 Types of selection bias…
5.Non-response bias

This is due to differences in the characteristics

between the responders and non-responders to the
study.

Non-response reduces the effective sample size,

resulting in loss of precision of the survey estimates.

Rates of response in many studies may be related to

exposure status.
 Types of selection bias…
6. Loss to follow up
major source of bias in cohort studies

also a problem in intervention studies

relates to the necessity of following individuals for a

period of time after exposure to determine the
development of the outcome
 Types of selection bias…

If the proportion of losses to follow-up is large, in the

range of 30 to 40 percent, this would certainly raise
serious doubts about the validity of the study results.

the more difficult issue for interpretation is that even

if the rate of loss is not that extreme, the probability
of loss may be related to the exposure, to the
outcome, or to both
 Ways of minimizing selection bias

Population-based studies are preferable

The selection of hospitalized controls in a case

control study will increase comparability
 Ways of minimizing selection bias cont…

One should avoid the inclusions as study subjects

of people who have volunteered on their own to
participate in the study

In case-control study, it is useful to select several

different control groups, including if possible a
group selected in the community.
Ways of minimizing selection bias cont…
In hospital-based case control study, controls are
usually selected among patients with diseases other
than the disease studied.
One should ensure that these other diseases are not
related to the exposure & the disease of interest.
keep losses to follow-up to an absolute minimum.

For those who are lost, an assessment of as much

outcome data as can be independently determined by the
investigator should be made.

This would include, at the very least, an assessment of

mortality status using different sources
 INFORMATION BIAS
Information Bias: a flaw (error) in measuring exposure or
outcome data that results in a differing quality (accuracy) of
information between comparison groups

Also called Observation Bias

Distorts the true strength of association

Occurs in all study designs but often described as RECALL

BIAS in case-control studies
 Example of information bias
1. Interviewer bias

2. Recall bias
3. Social Desirability bias
4. Healthy worker bias
5. Surveillance bias
6. Misclassification bias
7. Hawthorne effect

8. Lose to follow up
 Types of Information Bias
1. Interviewer Bias: An interviewer’s knowledge may
influence the structure of questions and the manner of
presentation which may influence the response

2. Recall bias: Those with a particular outcome or

exposure may remember events more clearly or
amplify their recollections
 Cont…
3. Observer Bias: Observers may have
preconceived expectations of what they should find
in an examination

4. Lose to follow up: Those that are lost to follow

up or who withdraw from the study may be different
from those who are followed for the entire study
 Cont…
5. Hawthorne effect: An effect first documented at
Hawthorne manufacturing plant; People act
differently if they know they are being watched

6. Surveillance Bias: The group with the known

exposure or outcome may be followed more closely or
longer than the comparison group.

7. Misclassification bias: Errors are made in classifying

either the disease or exposure status
 Controlling For Information BIAS
1.Blinding
– A blinded study (Masked study) is a study in which
observer(s) and/or subjects are kept ignorant of the group to
which the subjects are assigned, as in experiment, or of the
population from which the subjects come, as in a non
experimental study
2.same standard procedures, instruments, questionnaires,
interviewing techniques etc should be used for data collection
in both comparison groups
 Controlling For Information…
3.Classification of study subjects according to their
outcome & exposure status should be based on the
most objective & accurate methods available

4. when exposure status is determined by interview, it

should be assessed in several different ways for both
groups, so as to assist all study subjects to make a
thorough attempt at recall
 Important to Remember

Bias is a result of an error in the design or conduct of

a study.
Efforts should be made to reduce or eliminate bias

OR
the possibility of bias should be recognized and
taken into account when interpreting the findings.
CONFOUNDING
 THREATS TO VALIDITY

A study’s internal validity, or how close its

findings are to the TRUTH, can be compromised
by three things….

Chance
Bias
Confounding
 Confounding…

The influence of third variables in a study which

lead to an incorrect estimate of the association
between the exposure and disease variables

EXAMPLES: age, sex, BMI, smoking status, are

associated with virtually all diseases and are related to the
presence of or level of many exposures. Thus they should
always be considered as potential confounders of an
association.
 Confounding

Positive confounding: Distortion of association occurs

such that there appears to be an association when
actually it is absent.

Negative confounding: There seems no association,

when one exists,
Here’s what we’d like to assess:

Exposure Disease

Here’s where confounding acts:

Exposure Disease

Confounding

Where represents a causal relationship

represents a non-causal relationship

 Factious Example

Consider the following study conducted to

investigate the association between smoking
and a certain disease on 210 smokers and 240
non-smokers
Is smoking related to the disease in males?

Disease Smokers Non-smokers

Present 29 4

Absent 131 36

Total 160 40

29/160 (18%) of male smokers have the disease

4/40 (10%) of male non-smokers have the disease
OR=2.0
Is smoking related to the disease in females?

Disease Smokers Non-smokers?

Present 23 50

Absent 27 140

Total 50 200

23/50 (46%) of female smokers have the disease

60/200 (30%) of female non-smokers have the disease
OR=2.0
Is smoking related to the disease overall?

Disease Smokers Non-smokers?

Present 52 64

Absent 150 176

Total 210 240

52/210 (25%) of smokers have the disease

64/240 (27%) of non-smokers have the disease
OR=0.91
 What is going on?

Disease and Sex

Disease Males Females

Present 33 83
Absent 167 167
Total 200 250

• The prevalence of disease among males is 33/200 (16.5%) much lower

than prevalence of disease among females 83/250 (33%)
 Smoking and sex

Smoking Males Females

Smokers 160 50
Non-smokes 40 200
Total 200 250

The prevalence of smoking among males is 160/200 (80%) much higher

than prevalence of smoking among females 50/250 (20%)
 Why is confounding a problem?

Because…...

The estimate of association between exposure

and disease includes BOTH the contribution of
the exposure AND the confounder

The estimate of association which includes the

confounder gives an incorrect estimate of the
impact of the exposure on the disease outcome
 CRITERIA FOR CONFOUNDING

• Must be an independent predictor of disease with

or without exposure

• must be associated (correlated) with exposure but

not caused by the exposure

• must not be an intermediate link in a causal

pathway between exposure and outcome
 POTENTIAL CONFOUNDER

If…
E D E D

C C

E = Exposure
D = Disease
C = Confounder
NOT A POTENTIAL CONFOUNDER

E D E D

C C
Or
E and C are part of a causal pathway
C E D
part of causal pathway
E C D
 RESULTS OF CONFOUNDING

• can overestimate the true association (POSTIVE

confounding)

• can underestimate the true association

(NEGATIVE confounding)

• can change the direction of the association

between exposure and outcome (protective vs.
harmful)
 Alternative explanations for the observed
association other than cause and effect
relationships
A) The association may be the result of chance
B) The association may be the result of bias
C) The association can be the result of a
confounding effect.
D) An apparent cause can be an effect, rather than a
cause (reverse causation)
E) The cause can be both a cause and effect
(reciprocal causation)
e.g Vitamin A deficiency can cause diarrhea or
diarrhoea can cause Vitamin A deficiency
Causation
 causation

How do we know that one thing causes

another?

How do we know that a risk factor causes a

disease
 What causes an MI?
– Cigarette smoking
– Cholesterol

– Elevated blood pressure

– Stress

– Family history

– Obesity
• Which contributes the most risk?
• What are the relationships among risk factors?
 What causes diarrhea?
– Contaminated water
– Undercooked meat and/or seafood
– Raw fruit
– Prescription medications
– Stress
– Fatty foods/change in diet
 Causation…
Cause (dictionary)– the producer of an effect, result, or
consequence
Cause (Rothman) – an antecedent event, condition, or
characteristic that was necessary for the occurrence of the
disease at the moment it occurred, given that the other
conditions are fixed
Cause (Rothman, lay terms) – an event, condition, or
characteristic that preceded the disease event and without
which the disease event either would not have occurred at
all or would not have occurred until some later time
 Association vs. Causation
• Association is simply an identifiable relationship
between an exposure and a disease

• Implies that exposure might cause disease

• Exposures associated with a difference in

disease risk are often called “risk factors”
 Association vs. Causation
Causation implies that there is a true mechanism
that leads from exposure to disease
– e.g., long-term heavy smoking causes
myocardial infarction

  Finding an association does not make it causal

– e.g., hospital stays are associated with an
increased mortality rate, but this does not
mean they cause death
How do we assess whether risk factor is
indeed causal?

• Does exposure A cause disease B?

• Find out if variables are statistically
associated
• A preponderance of evidence
 Bradford Hill: Considerations for Causal
Inference
• Strength of association
• Consistency of findings
• Temporality
• Biological gradient (dose-response)
• Biological plausibility
• Specificity of the association
• Reversibility
 Strength of Association

Relative risk Interpretation

1.1-1.3 Weak
1.4-1.7 Modest
1.8-3.0 Moderate
3-8 Strong
8-16 Very strong
16-40 Dramatic
40+ Overwhelming
 Coherence and Consistency of
Findings
• Relationships that are demonstrated in multiple
studies are more likely to be causal; i.e.
consistent results are found
– In different populations,
– In different circumstances, and
– With different study designs.
 Specificity of the Association
• An exposure leads to a single or characteristic effect, or
affects people with a specific susceptibility (like the
concept of a “Necessary” cause)
– Easier to support causation when associations are
specific
• But, obviously not always true
•  Many exposures cause multiple diseases
• e.g., smoking causes many diseases including heart
disease, lung and other cancers, emphysema
 Specificity of the Association

Hepatitis A Botulism

Food OR Food OR

Sandwich 2.8
Bread 1.2
Fruit 3.2
Fruit 1.6
Salad 4.0 Tomato 4.0
Pastry 1.8 Tea 1.3
Milk 0.8 Cake 0.8
 Temporal Sequence

• Exposure must precede disease

• In disease with latency periods, exposures must
precede the latent period
• In chronic disease, often long-term exposure for
disease induction
 Biological Gradient

• “Dose- response”

• Changes in exposure are related to a trend in risk

• Risk of outcome increases with increasing
exposure to the suspected risk factor
 Biological Plausibility

• The proposed causal mechanism should be

biologically (etiologically) plausible
– The findings make biologic sense

– Results agree with current biological knowledge

of the disease
 Biological Plausibility
Plausible
– A disease shows a higher incidence in individuals who
are more sexually active
• The disease could be a sexually transmitted disease

Not Plausible
– HIV is caused by smoking
• This would go against the current knowledge on the
biological mechanism for HIV