EBM Learning: General medicine
BMJ EBM: first published as 10.1136/bmjebm-2018-111115 on 19 December 2018. Downloaded from http://ebm.bmj.com/ on 20 December 2018 by guest. Protected by copyright.
10.1136/bmjebm-2018-111115
►► Additional material is
published online only. To
view please visit the journal
online (http://​dx.​doi.o
​ rg/​
10.1
​ 136/​bmjebm-2 ​ 018-​
111115).
1
Institute of Clinical
Economics (ICE) e.V., Ulm,
Germany
2 ered to have sufficient power.2 The problem is that
Health Care Research at the
Department of General and
Visceral Surgery, University
Hospital Ulm, Ulm, Germany
3
Klinik für Allgemein- und
Viszeralchirurgie, Diakonie-
Klinikum Stuttgart, Stuttgart,
Germany
4
Klinik für Seelische
Gesundheit, Karl Hansen
Klinik, Bad Lippspringe,
Germany
5 criteria.
College of Business,
Kutztown University,
Kutztown, Pennsylvania, USA
Correspondence to:
Dr Franz Porzsolt, Health
Care Research at the Dept
General and Visceral Surgery,
University Hospital Ulm, Ulm
89071, Germany; f​ .​porzsolt@​
gmx.d​ e
© Author(s) (or their
employer(s)) 2018. No
commercial re-use. See
rights and permissions.
Published by BMJ.
To cite: Porzsolt F,
Wiedemann F, Becker SI,
et al. BMJ Evidence-Based
Medicine Epub ahead of
print: [please include Day
Month Year]. doi:10.1136/
bmjebm-2018-111115
Inclusion and exclusion criteria and the problem of
describing homogeneity of study populations in
clinical trials
Franz Porzsolt,1,2 Felicitas Wiedemann,1,3
Susanne Isabel Becker,1,4 C J Rhoads5
Introduction
The description of results in clinical trials is an
unsolved problem.1 The low quality of some clin-
ical trials and the low output of useful data for
healthcare decisions is a problem for clinicians
and patients alike. Statisticians doing system-
atic reviews or meta-analysis try to solve this
problem by establishing lower p-value thresholds
(for example, changing from 0.05 to 0.005). That
results in the acceptance of fewer, larger and more
carefully designed studies in order to be consid-
good useful information might get suppressed
using stricter p-value thresholds. This is an espe-
cially difficult problem in quality of life (QoL)
studies which, by their very nature, are smaller
and less powerful.
QoL researchers believe that the answer to poor
quality is to review more studies that describe
meaningful positive outcomes that contribute to
the improvement of healthcare.3 What is often
missing is a general discussion about the selection
Choosing selection criteria is not easy. On one
hand, wide inclusion criteria should be chosen in
order to demonstrate safety and effectiveness for
a large group of patients. On the other hand, there
are many ethical and scientific reasons to define
exclusion criteria.4 The resulting subjective selec-
tion of inclusion and exclusion criteria should be
a balance of these two considerations.
In this report, we analyse the information
provided by the description of inclusion and
exclusion criteria. We demonstrate that valuable
information on the investigated study popula-
tion can be provided by a complete and precise
description of the inclusion and exclusion criteria
that is not present in all publications. Furthermore,
we propose that researchers can derive important
information on the homogeneity/heterogeneity
of different study populations from completely
reported inclusion and exclusion criteria.
Method
This study builds on on the data of research done
by Wiedemann5 which analysed seven charac-
teristics of 100 publications on QoL assessments.
Wiedemann identified factors which contributed
to the aforementioned problem of poor results
reporting for clinical trials. The seven analysed
factors were:
BMJ Evidence-Based Medicine Month 2018 | volume 0 | number 0 |
►► Explicit and clear description of the study hy-
pothesis.
►► Differentiation of primary and secondary end-
points.
►► Categorisation as randomised or non-ran-
domised trial.
►► Explicit and clear differentiation of the ex-
pected outcome described as efficacy or ef-
fectiveness.
►► Description of the study design as pragmatic
or other than pragmatic trial.
►► Description of the QoL instruments used.
►► Numbers of described inclusion and exclusion
criteria.5
In August 2016, Wiedemann conducted
a PubMed search with the following search
parameters:
►► The terms ‘trial’ and ‘quality of life’ in their
titles.
►► Free full text available.
The number of articles with trial and quality of
life in the title for the past 5 years is depicted in
figure 1. Based on this information, it was reason-
able to choose to review the first 100 free acces-
sible articles that were listed as of August 2016.
Papers were downloaded as PDFs and analysed.
Subsequently, some of the articles were updated
and/or no longer freely available. However, in all
cases, the inclusion and exclusion criteria of the
earlier version and the later version were identical.
The analysis was not impacted by the subsequent
updates. The complete list of 100 articles can be
found in online supplementary appendix A.
In this report, we focus only on the numbers
and types of described inclusion and exclusion
criteria, and leave the other six criteria for future
reports.
Results
Fifteen of the 100 documents were only study
protocols and 85 were reports of completed
studies. Fourteen were published in 2017, 64 in
2016 and 22 in 2015.
Both inclusion and exclusion criteria were
reported in 69 of 100 publications. Only inclusion
but no exclusion criteria were reported in six. There
was no paper that reported only exclusion but no
inclusion criteria, but there were 25 publications
that reported neither inclusion nor exclusion
criteria. Some of these 25 publications described
the selection criteria ‘briefly’ or mentioned only
1
 EBM Learning: General medicine
Figure 1: Number of PubMed articles with Trial and Quality of Life in their
title by year.
‘key inclusion criteria’. All of these 25 documents claimed that
the precise inclusion and exclusion criteria had been described
in previous publications. Unfortunately, the references of some
of these publications were not provided and in other cases it was
necessary to check several referring references until the document
describing the relevant information could finally be identified.
These results are shown in table 1.
Inclusion criteria were reported in detail in 75 of 100 publica-
tions. The median number of explicitly reported inclusion criteria
in these 75 documents was 4 with an IQR of 3 to 5. Twenty-four
of the remaining 25 publications mentioned the exact reference
that describes the inclusion criteria. One document mentioned
the previous description of the inclusion criteria but did not
provide a traceable reference. Exclusion criteria were described
in 69 of 100 publications, the median being 4 with an IQR of 2
to 6. Of the remaining papers, 25 provided a reference for the
exclusion criteria, and 6 did not mention any exclusion criteria
at all. Figure 2 shows the number of studies of the 75 studies that
reported one or more selection criteria. The number of reported
selection criteria varied, with two studies reporting more than 20
selection criteria. Six studies reported only inclusion criteria. The
median number of selection criteria per study was 8 with an IQR
of 5 to 10.
Discussion
Our results confirm that precise inclusion and exclusion criteria
were described in 69% of the investigated publications. But there
are also three negative items: (1) incomplete reporting, (2) lack of
a precise definition and (3) useful information wasted.
Incomplete reporting
If the reader cannot find the information about the inclusion
criteria, then the information provided is incomplete. In 25%
of the investigated publications, neither inclusion nor exclusion
criteria and in 6% no exclusion criteria were directly reported
with references that could be found. Complete reporting of all
Table 1: Number of Inclusion a d Exclusion Criteria Reported
Inclusion criteria
Reported Not reported Total
Exclusion criteria
 Reported 69 0 69
 Not reported 6 25 31
Total 75 25 100
2 BMJ Evidence-Based Medicine Month 2018 | volume 0 | number 0 |
BMJ EBM: first published as 10.1136/bmjebm-2018-111115 on 19 December 2018. Downloaded from http://ebm.bmj.com/ on 20 December 2018 by guest. Protected by copyright.
Figure 2: Number of Studies with Number of Selection Criteria
selection criteria is essential for appropriate interpretation of the
study results.
Lack of precise definitions
Both the inclusion and exclusion criteria can each represent
one of the two endpoints of a dichotomous selection criterion.
Inclusion criteria describe the conditions a patient has to meet
to be included in a study and the exclusion criteria describe the
conditions a patient must not meet to be included in a study. The
problem is that the same criteria can be named inclusion (patients
with a particular disease are included) or exclusion criteria
(patients without this particular disease are excluded).6 7 Criteria
should be specified as inclusion or exclusion.8 9
Waste of information
The quantitative and qualitative information of the inclusion
and exclusion criteria could be used as a description of the target
population. The total number of inclusion and exclusion criteria
can be used as a simple measure for description of the homoge-
neity of a target population. The more criteria used to define the
study population, the more homogeneous the resulting popula-
tion. Our data show that the populations included in our investi-
gated studies were quite heterogeneous.
The total number of selection criteria (ie, the sum of the inclu-
sion and exclusion criteria) as indicator of the homogeneity of the
investigated population is only a very rough and lossy proxy for
the underlying selection. Each of these criteria may have a strong
or weak effect on the specific outcome investigated in a partic-
ular study. Each criteria may be dependent or independent from
each other. So even if the same numbers of selection criteria are
used, the degree of homogeneity will be influenced by the types
of selection criteria. Nonetheless, having this information is useful
and can help a researcher determine whether or not two different
studies are comparable.
Conclusion
We make three concluding recommendations from our research:
1. Report selection criteria completely.
2. Define inclusion and exclusion criteria separately.
3. Clearly differentiate between inclusion and exclusion criteria.
Our recommendations would make it easier to replicate clin-
ical studies in the real world. Exclusion criteria are only appli-
cable to ideal study conditions (ISCs) because some patients would
influence the assessed study outcomes. Inclusion criteria can be

included when the study is replicated under real-world conditions
(RWCs).10 11
In addition, there are ‘research methods’ such as randomisation
or blinding that are needed for completion of high-quality exper-
imental trials but are not used under RWC studies.
A clear distinction of inclusion and exclusion criteria is helpful
for the exact differentiation of efficacy and effectiveness. Efficacy
describes effects under ISC while effectiveness describes effects
under RWC. Efficacy and effectiveness cannot be differentiated
without clear description of inclusion and exclusion criteria.
In summary
In summary, simple modifications to the clarity with which selec-
tion criteria are described in published clinical trials would go a
long way towards the goal of improving the quality and useful-
ness of the study conclusions in QoL studies.
Contributors All authors contributed substantially to the paper.
FW and FP were primarily responsible for the acquisition of the
data, but all authors contributed substantially into the analysis
and review of the data, the development of the report and editing
of the paper for submission.
Competing interests None declared.
Ethics approval This study includes only published information
that cannot be related to individual subjects.
Provenance and peer review Not commissioned; externally peer
reviewed.
Data sharing statement The results on the remaining six criteria
are available (in German language) through FW and FP. We are
planning additional publications on these papers.
References
1. Heneghan C, Mahtani KR, Goldacre B, et al. Evidence based medicine
manifesto for better healthcare: a response to systematic bias, wastage,
BMJ Evidence-Based Medicine Month 2018 | volume 0 | number 0 |
EBM Learning: General medicine
BMJ EBM: first published as 10.1136/bmjebm-2018-111115 on 19 December 2018. Downloaded from http://ebm.bmj.com/ on 20 December 2018 by guest. Protected by copyright.
error and fraud in research underpinning patient care357 doi. Evid Based
Med 2017;22.
2 Ioannidis JPA. The Proposal to Lower P Value Thresholds to.005.
JAMA 2018;319:1429–30.
3. Bullinger M. [The concept of quality of life in medicine: its history and
current relevance]. Z Evid Fortbild Qual Gesundhwes 2014;108:97–103.
4. U.S. Food and Drug Administration. Evaluating Inclusion And Exclusion
Criteria. Clinical Trials: Workshop Report. Washington, DC, 2018. https://
www.​fda.​gov/​downloads/​Regu​lato​ryIn​form​ation/​LawsEnforcedbyFDA/​Sign​
ific​antA​mend​ment​stot​heFDCAct/​FDARA/​UCM613054.​pdf.
5. Wiedemann F. Potential mistakes in clinical trials that use health-related
quality of life as outcome: Thesis at the Medical Faculty University of Ulm,
2018.
6. Williams EM, Lorig K, Glover S, et al. Intervention to Improve Quality
of life for African-AmericaN lupus patients (IQAN): study protocol for a
randomized controlled trial of a unique a la carte intervention approach
to self-management of lupus in African Americans. BMC Health Serv Res
2016;16:339.
7. Omranifard V, Rajabi F, Mohammadian-Sichani M, et al. The effect of
add-on memantine on global function and quality of life in schizophrenia:
a randomized, double-blind, controlled, clinical trial. Adv Biomed Res
2015;4:211–5.
8. Pasquau J, Hidalgo-Tenorio C, Montes ML, et al. High quality of life,
treatment tolerability, safety and efficacy in HIV patients switching from
triple therapy to lopinavir/ritonavir monotherapy: a randomized clinical
trial. PLoS One 2018;13:e0195068.
9. Moretti HD, Colucci VJ, Berry BD. Vitamin D3 repletion versus placebo as
adjunctive treatment of heart failure patient quality of life and hormonal
indices: a randomized, double-blind, placebo-controlled trial. BMC
Cardiovasc Disord 2017;17:274.
10. Porzsolt F, Rocha NG, Toledo-Arruda AC, et al. Efficacy and effectiveness
trials have different goals, use different tools, and generate different
messages. Pragmat Obs Res 2015;6:47–54.
11. Porzsolt F. The assessments of three different dimensions “efficacy”,
“effectiveness”, and “value” require three different tools: the randomized
controlled trial (RCT), the Pragmatic Controlled Trial (PCT), and the
Complete Economic or Cost-Effectiveness Analysis (CEA). Surgery and
Rehabilitation;2.
3