10/10/2020 Validity, reliability and generalisability | Health Knowledge

Sign in / Register

Search

HOME ABOUT PUBLIC HEALTH TEXTBOOK TEXT COURSES VIDEO COURSES TRAINING

1a - Epidemiology

Validity, reliability and generalisability

PLEASE NOTE:

We are currently in the process of updating this chapter and we appreciate your patience whilst this is being completed.

Validity

Validity is the extent to which an instrument, such as a survey or test, measures what it is intended to measure (also known as internal validity). This is important if
the results of a study are to be meaningful and relevant to the wider population. There are four main types of validity:

Construct validity
Construct validity is the extent to which the instrument specifically measures what it is intended to measure, and avoids measuring other things. For example, a
measure of intelligence should only assess factors relevant to intelligence and not, for instance, whether someone is a hard worker. Construct validity subsumes
the other types of validity.

Content validity
Content validity describes whether an instrument is systematically and comprehensively representative of the trait it is measuring. For example, a questionnaire
aiming to score anxiety should include questions aimed at a broad range of features of anxiety.

Face validity
Face validity is the degree to which a test is subjectively thought to measure what it intends to measure. In other words, does it “look like” it will measure what it
should do. The subjective opinion for face validity can come from experts, from those administering the instrument, or from those using the instrument.

Criterion validity
Criterion validity involves comparing the instrument in question with another criterion which is taken to be representative of the measure. This can take the form
of concurrent validity (where the instrument results are correlated with those of an established, or gold standard, instrument), or predictive validity (where the
instrument results are correlated with future outcomes, whether they be measured by the same instrument or a different one).

Reliability

Reliability is the overall consistency of a measure. A highly reliable measure produces similar results under similar conditions so, all things being equal, repeated
testing should produce similar results. Reliability is also known as reproducibility or repeatability. There are different means for testing the reliability of an
instrument:

Inter-rater (or inter-observer) reliability

The degree of agreement between the results when two or more observers administer the instrument on the same subject under the same conditions.

Intra-rater (or intra-observer) reliability

Also known as test-retest reliability, this describes the agreement between results when the instrument is used by the same observer on two or more occasions
(under the same conditions and in the same test population).

Inter-method reliability
This is the degree to which two or more instruments, that are used to measure the same thing, agree on the result. This is also known as equivalence.

Internal consistency reliability

This is the degree of agreement, or consistency, between different parts of a single instrument.

Internal consistency can be measured using Cronbach’s alpha (a) – a statistic derived from pairwise correlations between items that should produce similar results.
Ranging from minus infinity to one, a Cronbach’s alpha of one indicates perfect internal consistency, and a negative value suggests that there is greater within-

https://www.healthknowledge.org.uk/content/validity-reliability-and-generalisability 1/3
10/10/2020 Validity, reliability and generalisability | Health Knowledge
subject variability than there is between subjects. Cronbach’s alpha values above 0.7 are generally deemed acceptable.

Inter-rater reliability can be measured using the Cohen’s kappa (k) statistic. Kappa indicates how well two sets of (categorical) measurements compare. It is more
robust than simple percentage agreement as it accounts for the possibility that a repeated measure agrees by chance. Kappa values range from -1 to 1, where values
≤0 indicate no agreement other than that which would be expected by chance, and 1 is perfect agreement. Values above 0.6 are generally deemed to represent
moderate agreement. Limitations of Cohen’s kappa are that it can underestimate agreement for rare outcomes, and that it requires the two raters to be independent.

Generalisability

Generalisability is the extent to which the findings of a study can be applicable to other settings. It is also known as external validity. Generalisability requires
internal validity as well as a judgement on whether the findings of a study are applicable to a particular group. In making such a judgement, you can consider factors
such as the characteristics of the participants (including the demographic and clinical characteristics, as affected by the source population, response rate, inclusion
criteria, etc.), the setting of the study, and the interventions or exposures studied. Threats to external validity, that may result in an incorrect generalisation, include
restrictions within the original study (eligibility criteria), and pre-test/post-test effects (where cause-effect relationships within a study are only found when pre-tests
or post-tests are also carried out).

© Saran Shantikumar 2018

‹ Analysis of health and disease in small areas up Intention to treat analysis ›

Navigation
Use of routine vital and health statistics to describe the distribution of disease in time and place and by person
Numerators, denominators and populations at risk
Time at risk
Methods for Summarising Data
Incidence and prevalence including direct and indirect standardisation
Years of Life Lost
Measures of disease burden (event-based and time-based) and population attributable risks including identification of comparison groups appropriate to
Public Health
Sources of variation, its measurement and control
Common errors in epidemiological measurements, their effects on numerator and denominator data and their avoidance
Concepts and measures of risk
Effect measures including odds ratios, rate ratios and risk ratios (relative risk)
Association and Causation
Biases and Confounding
Interactions, methods for assessment of effect modification
Strategies to allow/adjust for confounding in design and analysis
The design, applications, strengths and weaknesses of descriptive studies and ecological studies
Design, applications, strengths and weaknesses of cross-sectional, analytical studies (including cohort, case-control and nested case-control studies), and
intervention studies (including randomised controlled trials)
Analysis of health and disease in small areas
Validity, reliability and generalisability
Intention to treat analysis
Clustered data - effects on sample size and approaches to analysis
Numbers needed to treat (NNTs) - calculation, interpretation, advantages and disadvantages
Time-trend analysis, time series designs
Nested case-control studies
Methods of sampling from a population
Methods of allocation in intervention studies
The design of documentation for recording survey data, construction of valid questionnaires and methods for validating observational techniques
Construction of valid questionnaires
Methods for validating observational techniques
Studies of disease prognosis

https://www.healthknowledge.org.uk/content/validity-reliability-and-generalisability 2/3
10/10/2020 Validity, reliability and generalisability | Health Knowledge

The ethics and etiquette of epidemiological research

Appropriate use of statistical methods in the analysis and interpretation of epidemiological studies, including life-table analysis
Epidemic theory (effective & basic reproduction numbers, epidemic thresholds) & techniques for analysis of infectious disease data (construction & use of
epidemic curves, generation numbers, exceptional reporting & identification of significant clusters)
Systematic reviews, methods for combining data from several studies, and meta-analysis
Electronic bibliographical databases and their limitations
Grey literature
Publication bias
Evidence based medicine and policy
The hierarchy of research evidence - from well conducted meta-analysis down to small case series
The Cochrane collaboration
Understanding of basic issues and terminology in the design, conduct, analysis and interpretation of population-based genetic association studies, including
twin studies, linkage and association studies
Appendix

Our most popular content

Public Health Textbook
Identifying and managing internal and external stakeholder interests
Management models and theories associated with motivation, leadership and change management, and their application to practical situations and problems
Dietary Reference Values (DRVs), current dietary goals, recommendations, guidelines and the evidence for them
Section 1: The theoretical perspectives and methods of enquiry of the sciences concerned with human behaviour
Inequalities in health (e.g. by region, ethnicity, soci-economic position or gender) and in access to health care, including their causes
The impact of political, economic, socio-cultural, environmental and other external influences
Introduction to study designs - intervention studies and randomised controlled trials
2h - Principles and Practice of Health Promotion
Parametric and Non-parametric tests for comparing two or more groups

Recently updated content

3c - Applications
3a - Populations
3b - Sickness and Health
5d - Understanding the Theory and Process of Strategy Development
2a - Epidemiological Paradigms
1a - Epidemiology
2d - Genetics
2c - Diagnosis and Screening
1d - The Principles of Qualitative Methods
1c - Approaches to the assessment of health care needs, utilisation and outcomes, and the evaluation of health and health care

Disclaimer | Copyright © Public Health Action Support Team (PHAST) 2020 | Contact Us | Site Design
Company Information - Public Health Action Support Team CIC
[registered in England and Wales under Company No. 06480440]
Registered Office - 2-4 Packhorse Road, Gerrards Cross, Buckinghamshire, SL9 7QE

https://www.healthknowledge.org.uk/content/validity-reliability-and-generalisability 3/3