International Journal of

Radiation Oncology
biology physics

www.redjournal.org

Physics Contribution

A Dose of Reality: How 20 Years of Incomplete

Physics and Dosimetry Reporting in Radiobiology
Studies May Have Contributed to the
Reproducibility Crisis
Emily Draeger, PhD,*,y Amit Sawant, PhD,*,z
Christopher Johnstone, PhD,* Brandon Koger, PhD,x
Stewart Becker, PhD,z Zeljko Vujaskovic, MD,*,k
Isabel-Lauren Jackson, PhD,* and Yannick Poirier, PhD*,z
*Division of Translational Radiation Sciences, Department of Radiation Oncology, University of
Maryland School of Medicine, Baltimore Maryland; yDepartment of Therapeutic Oncology, Yale
University School of Medicine, New Haven, Connecticut; zDivision of Medical Physics, Department of
Radiation Oncology, University of Maryland School of Medicine, Baltimore, Maryland; xDepartment
of Radiation Oncology, University of Washington, Seattle, Washington; and kDivision of Clinical
Programs, Department of Radiation Oncology, University of Maryland School of Medicine, Baltimore,
Maryland

Received Mar 12, 2019. Accepted for publication Jun 29, 2019.

Summary
Many preclinical studies
have poor replicability,
partly because of undocu-
mented irradiation methods.
We performed the first large-
scale review of physics and
dosimetry reporting in radi-
ation biology studies, evalu-
ating the methods sections of
1758 publications for 12 pa-
rameters recommended by
the 2011 National Institute of
Standards and Technology
symposium expert panel. We
Purpose: A large proportion of preclinical or translational studies using radiation have
poor replicability. For a study involving radiation exposure to be replicable, interpret-
able, and comparable, its experimental methodology must be well reported, particu-
larly in terms of irradiation protocol, including the amount, rate, quality, and
geometry of radiation delivery. Here we perform the first large-scale literature review
of the current state of reporting of essential experimental physics and dosimetry details
in the scientific literature.
Methods and Materials: For 1758 peer-reviewed articles from 469 journals, we
evaluated the reporting of basic experimental physics and dosimetry details recom-
mended by the authoritative National Institute of Standards and Technology sym-
posium.
Results: We demonstrate that although some physics and dosimetry parameters,
such as dose, source type, and energy, are well reported, the majority are not.
Furthermore, highly cited journals and articles are systematically more likely to
be lacking experimental details related to the irradiation protocol.

Corresponding author: Yannick Poirier; E-mail: yannick.poirier@ Supplementary material for this article can be found at https://doi.org/
umm.edu 10.1016/j.ijrobp.2019.06.2545.
Disclosures: The authors have no conflict of interest to declare.

Int J Radiation Oncol Biol Phys, Vol. -, No. -, pp. 1e10, 2019
0360-3016/$ - see front matter Ó 2019 Elsevier Inc. All rights reserved.
https://doi.org/10.1016/j.ijrobp.2019.06.2545
2 Draeger et al.
demonstrate that although Conclusions: These findings show a crucial deficiency in the reporting of basic
some parameters, such as experimental details and severely affect the reproducibility and translatability of
dose, source type, and en- a large proportion of radiation biology studies. Ó 2019 Elsevier Inc. All rights
ergy, are well reported, the reserved.
majority are not. Further-
more, highly cited journals
are more likely to document
fewer parameters.
Introduction
There has been considerable discussion in recent years
concerning the lack of reproducibility and replicability of
results published in the scientific community at large.1-4 In
a 2011 study, investigators from a major biotechnological
institute could only successfully reproduce 6 of 53 bench-
mark studies on cancer research.5,6 Similarly, a 2012 Na-
tional Cancer Institute (NCI) workshop found 6 randomized
radiation oncology clinical trials in which preclinical
research did not translate to clinical outcome.7 This
inability to reproduce scientific findings has a profound
impact on our ability to translate preclinical research results
to clinical practice.4,7-12 An estimated $28 billion,
approximately half of federally funded preclinical research,
is spent annually on irreproducible studies.13
Radiation oncology benefits roughly half of all patients
with cancer,14-18 a projected 575,000 annual patients within
the United States alone.19 Despite remarkable scientific and
technological advancements in radiation therapy (RT),
outcomes for patients with cancer treated with RT alone or
in combination with other modalities (chemotherapy,
surgery, targeted therapies) is disappointingly lowd67% 5-
year relative survival across all cancers and modalities.16,18
To address these limitations, massive efforts are directed
toward preclinical investigations elucidating the underlying
mechanisms of tumor and normal-tissue radiation response
and developing novel targeted therapies. Indeed, 79.2% of
radiation oncology grants awarded by the National In-
stitutes of Health are directed toward radiation biology
projects, compared with 13.2% to medical physics and
7.6% to clinical investigations.20 In addition to tumor and
normal-tissue radiation biology, radiation is also used in
non-RT biological research to establish chimeric animal
models, probe graft versus host disease mechanisms, or
evaluate adoptive cellular therapies after bone marrow
transplant.
Radiobiological dose-effect relationships depend heavily
on the quantity, quality, and rate of radiation delivery.21-28
If radiation-induced biological effects represent the
observed variable (ie, the vertical axis), the radiation-
absorbed dose represents the controlled variable (ie, the
horizontal axis). Uncertainty, ambiguity, and errors in the
horizontal axis necessarily translate to the vertical axis and
impair the interpretability, reproducibility, and compara-
bility of the study. Of course, this does not invalidate the
International Journal of Radiation Oncology  Biology  Physics
biological conclusions of the study. Nevertheless, without
full description of the irradiation, it is difficult to replicate,
interpret, and compare the results to those of other
studies.21-23,26,28
In contrast to biological sources of error, which are
challenging to isolate and reduce,29 radiation delivery fol-
lows well-understood radiation physics principles, lending
itself to high accuracy, precision, and reproducibility.26,30
Unfortunately, these are often poorly reported: In 2011,
the NCI, the National Institute of Allergies and Infectious
Diseases, and the National Institute of Standards and
Technology held a symposium and established standards
for reporting the experimental physics and dosimetry de-
tails required to reproduce and interpret radiobiology
studies. In addition to communicating these recommenda-
tions to the community, Desrosiers et al21 reviewed a year
of Radiation Research publications and found the wide
majority (>93%) did not report basic physics details.
Similar studies by Pedersen et al22 and Stone et al23 fol-
lowed but were limited in the number of reviewed articles
(28 and 125, respectively). In particular, Stone et al re-
ported that 83% of in vitro studies “had one or more
problems that would make interpretation or replication of
the study difficult”; they cited incomplete reporting of
methods, specifically in dosimetric parameters, as most
important.23
Incomplete physics reporting is often attributed to
insufficient physics knowledge and support among bio-
logical irradiator operators.21,26,28,31 As Desrosiers et al put
it, “Few students or researchers using ionizing radiation in
biological research have training in basic radiation physics.
This leads to the difficult situation that when ‘one does not
know what one does not know.” This lack of physics sup-
port can also manifest in the form of dosimetric
errorsdPedersen et al also conducted a multi-institutional
dosimetric audit and found only 5 out of 11 sites deliv-
ered radiation within a 5% accuracy.22 A similar study by
Seed et al showed only 3 out of 7 sites were able to deliver
radiation to a 5% accuracy, even after remedial action was
taken by the host site.32 Finally, from 1978 to 1999, 8
multi-institution radiation biology dosimetric surveys were
conducted in Europe under the European Late Effects
Project Group and the European Radiation Dosimetry
Group, each of which showed that a large number of sites
require improvement in either the accuracy or homogeneity
of dose delivery.31,33
Volume -  Number -  2019
This study is the first large-scale systematic review of
the current state of physics and dosimetry reporting in
preclinical radiobiology research in the past 2 decades.
Our initial search identified 3542 articles from 968
journals published in the last 20 years (1997-2017). Of
these initial 3542 articles, 1758 from 469 journals were
admissible according to our exclusion criteria and
were scored using the methodology published after the
National Institute of Standards and Technology
symposium.21
Methods and Materials
Search strategy
This review was performed using adapted Preferred
Reporting Systematic Reviews and Meta-Analyses34
guidelines and a search of PubMed35 with the following
criteria: (Animal Studies OR Cell Culture) AND (Ionizing
radiation NOT ultraviolet/UV). The search, which was
limited to papers published 1997 to 2017, identified 3542
articles on August 30, 2017. Ionizing radiation was cho-
sen as the search criteria because UV and nonionizing
radiation represented the vast majority of flagged articles
otherwise.
Exclusion criteria
The review was restricted to peer-reviewed research arti-
cles presenting novel results of radiation-induced biolog-
ical effects in cells or animal subjects resulting from
external radiation sources (Fig. 1), excluding non-English
articles, review articles, articles in nonrefereed journals,
clinical reports (even in animals), internal sources of ra-
diation such as brachytherapy or nuclear medicine (which
demand the reporting of different physics details), imaging
studies, or articles not studying biological effects of
ionizing radiation (eg, radiation as a sterilization agent or
to induce mechanical transformations). A small number of
articles (10) were rejected because of copyright fees,
unreleased manuscripts (eg, under embargo), or article
retraction.
3542 Abstracts
Articles Excluded:
• No ionization radiation
experiment [413] 1035 2793 Articles
• Imaging study only [271]
• Not animals
or cell cultures [240]
• Not external radiation [106]
• Other [5]
Fig. 1. Preferred Reporting Systematic Reviews and Meta-Analyses flow chart.
20 years of incomplete physics in radiobiology 3
Scoring metric
We evaluated the irradiation protocol section following the
recommendations of Desrosiers et al,21 broadly divided in
Table 1 into 4 “mandatory” categories: source specification,
dose specification, calibration/measurement of the radiation
dose, and irradiation geometry. The importance of each of
these parameters on the accuracy of the dose delivered, and
on factors that can affect biological endpoints of the study,
is presented in Appendix EI (available online at https://doi.
org/10.1016/j.ijrobp.2019.06.2545). A fifth nonmandatory
category, “Further details of the irradiation,” was not
included.
Articles were sorted and scored in alphabetical order and
their methods section read carefully. If no irradiation pro-
tocol was present, the article was scanned for the terms
“Gy,” “irradiation,” “radiation,” or “IR” to find other
mentions of the irradiation protocol. If radiation dose was
unspecified, the results section (particularly figures showing
dose-effect relationships) was scanned. Some articles con-
tained supplementary materials or referenced previous
work regarding irradiation protocol; these were read and
scored as well.
The majority of radiation biology experiments involve a
single radiation exposure. Nevertheless, fractionation
schedule is among the mandatory reportable values because
it is impossible to distinguish between an incompletely
described irradiation protocol and an implicit single expo-
sure. For this reason, we counted any mention of “single
exposure,” “single dose,” “single field,” and so forth as
describing the fractionation.
The present study concerns itself purely with the pres-
ence of experimental details and not with their quality. To
be as conservative as possible, all reviewed articles were
given the benefit of the doubt to the furthest extent. It was
sufficient for any given publication to specify a given
quantity to consider that quantity reported, though not all
reported quantities are described in the same amount of
detail, and some were either vague or contradictory.
To compare articles directly, we produced an aggregate
score based on these criteria founded on a 10-point
system counting the quantity of mandatory categories
(see Table 1) reported, where 0 indicates no experimental
749
Screened
Abstracts Excluded:
Reviewed • Review articles [570]
• Not in English [169]
1758 Art.
• Not available
Scored (copyright, embargo,
redacted) [10]
4 Draeger et al. International Journal of Radiation Oncology  Biology  Physics

Table 1 Mandatory reportable categories

Main category Subcategory Score
Source specification [3.0 pts] Radiation source or particle (eg, x-ray, proton, a or b) 1.0
Beam quality (eg, isotope, energy, half-value layer) 1.0
Irradiator model or manufacturer 1.0
Dosimetry/calibration [2.0 pts]* Absolute dose calibration standards or protocol used* 0.5
Radiation detector or equipment 1.0
Medium used for calibration or measurement 0.5
Measurement geometry* 0.5
Dose specification [2.5 pts] Radiation absorbed dose delivered to biological sample 1.0
Dose rate 1.0
Fractionation schedule (eg, number of irradiations and 0.5
intervals)
Irradiation geometry [2.5 pts] Source geometry Field size 0.5
Distance from source 0.5
Number/geometry of fields 0.5
Subject geometry Subject size 0.5
Backscatter/attenuation 0.5

* Measurement geometry only scored if calibration standards protocol not specified; total points in dosimetry calibration cannot exceed 2.0.

details at all and 10 represents a description of all iden-
tified categories. Again, to be as conservative as possible,
even incomplete, flawed, or contradictory reporting was
fully scored, as long as the required category was at least
partially described.
was chosen to reduce the impact of outliers represented by
journals and articles with disproportionate impact factors
and number of citations.
Publication demographics

Publication impact metrics
The following publication impact metrics were recorded for
each scored publication: journal impact factor, number of
citations, and number of citations per year. Number of ci-
tations was obtained from Google Scholar,36 and the
number of citations per year was calculated simply by
dividing the number of citations by the number of years
between publication and 2018 (i.e. when the number of
citations were tabulated). The journal impact factor was
obtained from the SCImago Journal & Country Rank37 as
the number of citations per published article within the last
2 years, available for 2017 at the time of writing.
For 9 journals, distribution had ceased under the publi-
cation’s original name. If the journal had changed names (2
cases), the impact factor of the successor journal was re-
ported instead. If the journal had ceased publishing entirely
(7 cases), the most recent impact factor on record was used
instead. Out of 469 journals, a single journal was unlisted
on SCImago, such that there was no impact factor to report.
A value of 0.000 was used instead.
Spearman correlation coefficients, along with associated
significance (P value) and 95% confidence intervals, were
calculated between the aggregate physics-reporting score
and each publication’s impact metricsdimpact factor,
number of citations, and citations per year, as well as the
number of years since publication. Because the Spearman
correlation coefficient is a nonparametric correlation coef-
ficient that strictly evaluates the monotonicity of the data, it
We also analyzed the aggregate physics-reporting score for
various subcategories of publications. First, we divided by
radiation oncology, biology, or physics specialty journals.
Although this distinction is somewhat arbitrary, specialist
journals were identified as abbreviated journal names
containing “(Radiat) OR (Phys NOT (Biophys OR Phys-
iol))”. This resulted in 389 publications from specialist
journals and 1369 publications from generalist (at least in
terms of radiation oncology, biology, or physics) journals.
Compared with papers from generalist journals, specialist
journal publications tended to be in a lower impact factor
range (average  standard deviation: 2.2  0.8 vs 4.2 
3.1) and to have fewer citations (30  39 vs 46  155) and
citations per year (2.9  2.8 vs 4.6  12.4) but to be
similarly aged (10.4  5.8 vs 9.7  5.7 years since pub-
lication). More importantly, the majority of specialist
publications seem concentrated in a relatively narrow range
of impact factor journals, citations, and citations per year,
whereas generalist publications occupy a much greater
range, as evidenced by their very wide standard deviations.
We also separated publications by biological model,
yielding 961 in vitro studies, 736 in vivo (eg, whole animal)
studies, and 61 studies incorporating both, as well as by
source type (Fig. 2). An unpaired, 2-tailed, heteroscedastic
Student’s t test was performed between the aggregate
physics-reporting score of specialist versus nonspecialist
journals, between in vivo versus in vitro studies, and be-
tween all different radiation sources. Type 1 errors were
avoided because significance values tended to be either
Volume -  Number -  2019 20 years of incomplete physics in radiobiology 5

Biological subject Radiation source

Rabbits, NHP, 12, D or E, 67, None Listed,
Pigs, 10, Other, 34,
8, 0.5% 0.7% Other, 94, 3.8% 132, 7.5%
0.6% 1.9%
5.4% Co-60,
Protons/Heavy
Rats, 138, Ions, 110, 6.3% 421, 23.9%
7.8% "J or x-ray" of
unknown energy,
111, 6.3%
Mice, 540, MV x-rays,
Cells, 1022, 130, 7.4%
30.7%
58.1%
Cs-137,
kV x-rays, 439,
450, 25.6%
25.0%
Fig. 2. Breakdown of articles by biological subject (left) and radiation source (right). Of importance is that the majority of
radiation biology studies are conducted either in vitro or in murine models with more traditional sources of radiation, such as
Co-60, Cs-137, or kV x-rays. Articles reporting more than 1 subject or radiation source are counted multiple times, such that
percentages do not add up to 100%.

above 0.05 or very small (<0.001), such that false positives
were very unlikely.
Results
Our initial search identified 3542 articles from 968 journals
published in the last 20 years (1997-2017), of which 1758
from 469 journals were admissible and evaluated. Figure 2
shows the breakdown in biological subjects and radiation
sources in evaluated articles and shows that the majority of
reported studies are performed in vitro or in murine (ie,
mice or rat) models, with all other models representing
6.8% cumulatively. Additionally, the majority of radiation
biology studies continue to be performed with traditional
Co-60 and Cs-137 irradiators (approximately half of all
reported sources), although they are increasingly becoming
obsolete in both human patients and research. X-rays (kV
and MV) follow in frequency, and a small number (6.3%)
report x-rays or gamma rays without specifying the energy,
manufacturer, or model. Finally, a relatively small propor-
tion of studies used more exotic radiation: a, b, or electrons
(3.8%), protons or heavy ions (6.3%), or other sources,
including neutrons (1.9%). The remaining studies (7.5%)
did not report the radiation source. In total, the source of
radiation was unclear or ambiguous for 243 papers, or
13.8% of the total, which loosely agrees with the findings
of Stone et al, wherein 30.1% of articles did not report the
energy or radiation type.
Figure 3 shows the breakdown of reported physics and
dosimetric experimental details for each category. The
majority of authors reported the source (92.5%), the energy
(81.4%), and the manufacturer or model of the irradiator
(64.8%).
The absolute dosimetry category was by far the least
reported: Only 1.2% of all authors listed the protocol used
for machine calibration, and only 15.9% listed any form of
equipment used to measure the radiation absorbed dose,

Field Size (12.9%)

SSD (12.2%)
Source (92.5%) Protocol (1.2%)
Geometry (2.0%) Dose (99.0%) # Fields Used (18.1%)
Model/Manuf. (64.8%) Size of Target (34.2%)
Equipment (15.9%) Dose Rate (67.5%)
Energy/Isotope (81.4%) Medium (3.9%) Fx Scheme (26.6%) Backscatter (7.5%)

Fig. 3. Breakdown of the number of authors reporting each physics parameter separated into 4 broad categories. From left
to right, these are source specification, dosimetry/calibration, dose specification, and irradiation geometry.
6 Draeger et al.
either for calibration or under irradiation conditions.
Although not explicitly listed, in nearly all cases, no
distinction was made between the dose defined during
calibration under reference conditions and the radiation
dose delivered to biological subjects. Without further
description, the majority of these studies would be difficult
to interpret or replicate.
For irradiation and setup geometry, the majority of au-
thors reported radiation dose delivered (99.0%) and dose
rate (67.5%) but no other parameters. Approximately a
quarter (26.6%) of authors reported fractionation schedule
and irradiation subject dimensions, but an eighth reported
field size (12.9%), distance to the source (12.2%), or
number of fields (18.1%). Without additional description,
many experiments likely involved a single field exposure,
but this is ambiguous. Incidentally, 320 of 468 papers
reporting fractionation specified a single radiation delivery,
showing that at least a portion of authors see value in
deliberately specifying this experimental detail. Almost no
publications (7.5%) described the presence (or dimensions
and composition) of attenuating or backscatter material (see
Fig. 3), which can have a dramatic impact on radiation
delivery, particularly in the case of kV x-ray irradiations
(see Appendix EI for more details; available at https://doi.
org/10.1016/j.ijrobp.2019.06.2545).
Although nearly all publications reported on some pa-
rameters, the pattern of reporting was not uniform. A total
of 196 publications (11.1%) reported at least partially on all
4 main categories described in Table 1. Two publications
(0.1%) had no experimental description at all, save that
ionizing radiation was used to irradiate the biological
samples. Some reported values were predictive of more
complete reportingdfor instance, from 21 publications
reporting the calibration protocol, 18 (85.7%) reported on
all 4 main categories.
The overall level of physics reporting was quantified
through an aggregate score (Fig. 4). In general, these data
show that the majority of articles (75%) received an
aggregate score of 5.5 or less, and a very small minority
(3.0% of articles) received 8 points. As suggested by
Figure 3, publications predominantly report the type of
radiation (1 point) and the absorbed dose (2 points), and a
majority also report 1 to several out of the dose rate, the
irradiator model, and the beam quality (1-3 points), which
explains why most publications score 4 to 6 points. Only
papers reporting irradiation and animal geometry, or spec-
ification of absolute dosimetry, were likely to score higher,
explaining why only a quarter of publications score 6.
We correlated this aggregate physics-reporting score
with a number of common publication impact metri-
csdnamely the impact factor, the total number of citations,
and the citations per year. As shown in Table 2, the physics-
reporting score had a weak but statistically significant
negative correlation with each publication impact metric:
the Spearman correlation coefficient was e0.237 for impact
factor, e0.102 for total number of citations, and e0.067 for
citations per year. The aggregate physics-reporting score
International Journal of Radiation Oncology  Biology  Physics
≤25th percen le (4)
300 ≤50th percen le (5)
≤75th percen le (6)
250
Number of Publica ons
≤100th percen le (10)
200
150
100
50
0
0 1 2 3 4 5 6 7 8 9 10
Aggregate score
Fig. 4. Statistical distribution of the aggregate reporting
score for physics and dosimetry experimental detail
reporting in the reviewed literature. The average score 
standard deviation was 4.80  1.6. The majority of articles
scored in the 4 to 6 range.
correlated with publication age at e0.094, indicating a
higher score for more recent articles. When separating the
publications between specialist and generalist journals, a
different picture emerged: Although the relationships
remained similar in the generalist journals, in specialist
journals the correlation became positive for impact factor
(þ0.152) and citations per year (þ0.140) and stronger
(e0.150) for more recent articles, whereas no correlation
existed with total number of citations (þ0.02, P Z .39).
Figure 5 shows scored publications split evenly among 5
groups for each publication impact metric. The reader
should note that equal groups were not possible because
variables are discrete and not uniformly distributed. In each
case, score frequency is reported for each subcategory as
the cumulative fraction of papers scoring n, where n is the
score on the horizontal axis. The relationship is most
readily apparent for impact factor, where publications in
journals with an impact factor of 2 to 2.5 are much more
likely to have high scores and where the publications are
increasingly weighted toward lower scores with increasing
impact factor. For the other metrics, the relationship is
much more subdued, as expected from the very weak
(approximately e0.07 to 0.10) correlation (see Table 2).
Also shown is the physics-reporting score between several
publication demographicsdbetween so-called specialist
and generalist journals, between in vivo and in vitro ex-
periments, and by radiation source type. Papers published
in specialist journals out-scored generalist journals (t test
P Z 1.4  1015) whereas in vivo studies out-scored
in vitro studies (t test P Z 1.5  1012). Finally, a large
difference in aggregate physics-reporting score is apparent
when separating papers by the radiation source, with un-
specified sources or unknown x-ray/gamma ray sources
scoring dramatically lower than all others and traditional
Volume -  Number -  2019
Table 2 Spearman correlation (r), significance (P), and 95% CI between physics-reporting score and various publication metrics
Metric or group All publications
Impact factor r Z e0.237
P Z 6.9  1024
CI Z –0.280 to –0.192
Total citations r Z e0.102
P Z 1.7  105
CI Z –0.148 to –0.055
Citations per year r Z e0.067
P Z 5.2  10-3
CI Z –0.113 to –0.020
Years since publication r Z e0.094
P Z 8.4  105
CI Z –0.140 to –0.047
Abbreviation: CI Z confidence interval.
* Specialist journals are defined as having “Radiat” or “Phys” but not “Physiol” or “Biophys” in their abbreviated journal title.
(Co-60, Cs-137 sources) scoring lower than MV and kV x-
rays and all other sources. A univariate analysis of variance
showed a highly significant relationship (P Z 9.8 
10216), with subsequent t tests showing significant dif-
ferences between all data sets (P  .0025) except for kV,
MV, and all remaining sources, for which t tests ranged
from P Z .37 to P Z .59. Additionally, the t test between
all remaining sources and Co-60 was only P Z .093.
In the course of the review, several publications were
noted to contain errors in which a quantity was mis-
reported. The most innocuous were simple errors of ter-
minology, such as misattributing Co-60 as x-rays instead of
gamma rays or radiation from x-ray generators as gamma
rays. Although these types of errors do not affect inter-
pretability of a study, they introduce ambiguity. Also
common were vague descriptionsdexperiments conducted
using gamma or x-rays of unknown origin or energy (eg,
“x-rays delivered from a Siemens x-ray machine”) or solely
described as “ionizing radiation.” In a few cases,
unachievable experiments were described, such as a 150
kVp irradiation performed by a Varian TrueBeam medical
linear accelerator, a system operating at MV energiesd2
orders of magnitude higher than reported. Although the
TrueBeam is equipped with a kV x-ray imaging source, its
maximum energy (140 kVp) and dose rate (w0.1 Gy/
min38,39) are both lower than that reported (150 kVp, 1.018
Gy/min). It is unclear whether the irradiator or beam
quality was misreported, but the relative biological effec-
tiveness and, by implication, the study results are conse-
quently difficult to reproduce, interpret, or place in context.
Other papers describe irradiations from nonexistent iso-
topes beyond a simple typographical error (eg, Cs-62 re-
ported vs typical Co-60 or Cs-137).
Discussion
We performed the first systematic large-scale review of the
current state of physics and dosimetry reporting in basic
20 years of incomplete physics in radiobiology 7
Specialist journals* Generalist journals*
r Z þ0.152 r Z e0.169
P Z 1.6  1010 P Z 8.9  1013
CI Z þ0.053 to þ0.247 CI Z –0.220 to –0.117
r Z þ0.020 r Z e0.111
P Z 0.394 P Z 2.9  106
CI Z e0.079 to þ0.120 CI Z –0.163 to –0.059
r Z þ0.140 r Z e0.072
P Z 3.8  109 P Z 2.4  103
CI Z þ0.041 to þ0.236 CI Z –0.125 to –0.019
r Z e0.150 r Z e0.103
P Z 2.7  1010 P Z 1.5  105
CI Z –0.246 to –0.051 CI Z –0.155 to –0.050
and preclinical radiation biology research in the past 2
decades. Although 3 previous reviews were performed,21-23
they were limited in scope. Our results are in broad
agreement with these similar reviews performed by Des-
rosiers et al,21 Pedersen et al,22 and Stone et al.23 Although
differences in scoring categories and precise scoring
methodology make an exact comparison challenging, the
reporting frequencies we found are similar (see Table 3) to
these previous reviews.
Broadly, the type of radiation and delivered radiation
absorbed dose are nearly universally reported. Approxi-
mately 60% to 80% report radiation quality/beam energy,
model or manufacturer of the irradiator, and irradiation
dose rate. Very rarely are details concerning the irradiation
geometry or irradiator calibration reported. Absolute cali-
bration seldom extends beyond specifying the type of
dosimeter used to measure radiation-absorbed dose, itself
only reported in one-seventh of publications.
This endemic failure to report basic experimental details
of the physics and dosimetric irradiation technique of bio-
logical samples can have further-reaching implications. In
all relevant literature, this failure to report or reproduce
radiation absorbed dose is ascribed to the lack of consul-
tation between the radiation biologists performing the study
and the qualified radiation physicists able to design,
document, and validate the irradiation protocol.21,23,32
Furthermore, because plausible but misreported quantities
are undetectable, reporting of experimental details is no
guarantee of their accuracy, and only a comprehensive
description of the irradiation protocol can ultimately be
used to evaluate and reproduce it.
In particular, the very low number of publications
reporting on absolute calibration standards (1.2%) or on
dosimetric measurements (15.9%) implies that few centers
calibrate or verify experimentally the dose delivery of their
irradiators. If true, this constitutes a potential cause of the
repeated difficulty in achieving standardized accurate and
homogeneous dose delivery in multi-institution dosimetric
surveys.22,31-33
8 Draeger et al. International Journal of Radiation Oncology  Biology  Physics

a Impact Factor b Citations per year

1 0–2, n = 329 1 0–1, n = 351
2–2.5, n = 383
Cumulative Fraction

Cumulative Fraction
1–2.5,n = 347
0.8 0.8
2.5–3.3, n = 344 2–3.5, n = 378
3.3–5, n = 362 3.5–6, n = 334
0.6 0.6
5–50, n = 340 6–400, n = 348
0.4 0.4

0.2 0.2

0 0
0 2 4 6 8 10 0 2 4 6 8 10
Reporting Score Reporting Score

c Total Citations d Years since publication

1 0–6, n = 366 1 0–4, n = 393

Cumulative Fraction
7–14, n = 344 5–7, n = 351
Cumulative Fraction

0.8 0.8
15–26, n = 342 8–11, n = 312
27–54, n = 349 12–15, n = 358
0.6 0.6
>54, n = 357 16–21, n = 344
0.4 0.4

0.2 0.2

0 0
0 2 4 6 8 10 0 2 4 6 8 10
Reporting Score Reporting Score

e Journal Specialization and Biological Model f Source type

Co-60, n = 421
1 Specialist j., n = 389 1 Cs-137, n = 450
Generalist j., n = 1369
Cumulative Fraction

kV x-rays, n = 439
Cumulative Fraction

in vivo, n = 736
0.8 0.8 MV x-rays, n = 130
in vitro, n = 961
both, n = 61 All others, n = 211
0.6 0.6 Ukn x/γ-ray, n = 111
No source, n = 132
0.4 0.4

0.2 0.2

0 0
0 2 4 6 8 10 0 2 4 6 8 10
Reporting Score Reporting Score
Fig. 5. Breakdown of the statistical distribution of papers achieving a minimum aggregate physics-reporting score as (a) a
function of impact factor, (b) total number of citations, (c) citations per year, (d) years since publication, (e) journal
specialization and biological model, and (f) source type. In the first 4 cases, the data set was divided into 5 subsets that were
as numerically equal as achievable. In the last 2, categories simply fall as they may. In general, articles with better quan-
titative impact metrics (or less recently published) were statistically more likely to achieve a lower aggregate physics-
reporting score. Journals specializing in radiation oncology, biology, or physics outscored more generalist journals, and
more modern radiation sources (MV or kV x-rays) outsourced traditional sources (Co-60 or Cs-137), but papers that did not
report the source (unknown photons, no source listed) scored drastically lower than all others.

We should stress, however, that although insufficient or
inaccurate reporting of the experimental protocol impairs
reproducibility, it does not necessarily diminish the quality
of results or applicability of conclusions. The review is
limited in scope to the reproducibility of the irradiation
protocol and makes no representations on other scientific
aspects (eg, study design, cell line or animal model selec-
tion, radiation dose delivered, statistics).
Aside from these findings, the radiation biology and
physics community are acutely aware of the urgency of this
Volume -  Number -  2019 20 years of incomplete physics in radiobiology 9

Table 3 Comparison between 4 studies

Category Present study Desrosiers et al21 Pedersen et al22 Stone et al23
Source type, % 92.5 100 N/A* 86/80y
Manufacturer or model, % 64.8 80 N/A* N/A*
Energy, %z 81.4 78 86.2/50.0z 81/63y
Dose, % 99.0 94 100x N/A*
Dose rate, % 67.5 81 - 62/57y
Setup, % 32.1/37.0k 48 0.0/24.1{ 24/55y
Absolute dosimetry/equipment calibration, % 15.9 37 3.4/27.6# 8/12y
Published standards/protocol, % 1.3 7 6.9/10.3** N/A*
Papers reviewed, no. 1758yy w190zz 28 125y

* Not reported.
y
Reported separately for in vitro (left) and in vivo (right) experiments; 104 in vitro and 51 in vivo experiments are involved, though they add to 125
papers total.
z
Reported as “Radio-isotope” and “kV, filtration, HVL” for gamma and kV x-ray sources, respectively.
x
Reported as “Dose details”; unclear if referring to dose only or dose, dose rate, and fractionation.
k
Any detail on source/animal irradiation geometry reported, respectively.
{
Reported as “Field size and shape” and “Geometry of Fields,” respectively.
#
Reported differently under “Absolute dosimetry/calibration” and “Radiation source specification,” respectively.
** Reported differently under “Absolute dosimetry/calibration” and “Determination of dose,” respectively.
yy
Out of an original 3542 evaluated articles.
zz
All issues of Radiation Research between March 2010 and March 2011 (not explicitly enumerated as not all articles may have been analyzed).

problem. In a 2016 report aimed toward improving the
translation of preclinical research to positive clinical trials,
members of the NCI reported that “there is an urgent need
to improve reproducibility and translatability of pre-clinical
data” to improve therapeutic agents involving radiation but
that “one impediment to drug development has been irre-
producibility of preclinical data,” which could be improved
by “fastidious calibration and dosimetry of radiation sour-
ces.”40 Recently, the American Association of Physicists in
Medicine, supported by the Radiation Research Society,
established Task Group 319 on the guidelines for accurate
dosimetry in radiobiology experiments.41 Finally, the Na-
tional Institute of Allergies and Infectious Diseases and the
NCI recently formed a program to standardize and monitor
radiation dosimetry delivery among the entirety of their
preclinical research funded by the radiation countermea-
sures program. The findings in this study only serve to
stress the importance and timeliness of such an initiative.
Finally, a weak but significant negative correlation exists
between the physics-reporting score of an article and its
impact metrics: Well-cited papers published in high-
impact-factor journals were most likely to have incom-
pletely described methods regarding irradiation protocols.
This may be because these journals have more stringent
word limits compared with lower-impact-factor journals.
Furthermore, specialist journals, in vivo studies, and more
modern radiation sources such as MV and kV x-rays were
more likely to be well described than generalist journals,
in vitro studies, or traditional sources. This is likely
because these publications are more likely to enlist the help
of medical physicists who advise on the design and
reporting of irradiation protocols. Indeed, when analyzing
the relationship between publication impact metrics and
physics-reporting scores, generalist journals continued to
exhibit the same trends as before whereas specialist jour-
nals showed positive trends for impact factor and number of
citations per year and an even more pronounced trend to-
ward higher scores in more recent publications, while
showing no relationship with total number of citations.
Conclusions
We performed the first large-scale (>1000 publications)
analysis of the current state and prevalence of physics and
dosimetry reporting in radiation biology preclinical and
translational research. Although most peer-reviewed arti-
cles report basic details on the amount, rate, and type of
radiation dose delivered, the majority fail to report other
details required for a radiation study to be reproducible,
interpretable, and comparable. In general, higher-impact
journals and papers with greater numbers of citations were
correlated with the reporting of fewer experimental details.
These findings expose a deep lack of reporting of basic
radiation physics concepts in a large proportion of the ra-
diation biology literature. In the context where much of
nationally funded preclinical research fails to translate to
clinical applications, and with the increasing role of allied
fields such as immunology in radiobiology studies, journal
editors and reviewers have the responsibility to ensure that
peer-reviewed articles report required experimental details
related to physics and dosimetry to ensure the reproduc-
ibility, interpretability, and comparability of radiation
research.
10 Draeger et al.
References
1. Baker M, Penny D. Is there a reproducibility crisis? Nature 2016;533:
452-454.
2. Peng R. The reproducibility crisis in science: A statistical counterat-
tack. Significance 2015;12:30-32.
3. Challenges in irreproducible research. Nature 2015;1-6; special ed.
4. Karp NA. Reproducible preclinical researchdIs embracing variability
the answer? PLoS Biol 2018;16:e2005413.
5. Mullard A. Reliability of “new drug target” claims called into ques-
tion. Nat Rev Drug Discov 2011;10:643-644.
6. Gegley CG, Ellis LM. Raise standards for preclinical cancer research.
Nature 2012;483:531-533.
7. Liu F-F, Okunieff P, Bernhard EJ, et al. Lessons learned from radiation
oncology clinical trials. Clin Cancer Res 2013;19:6089-6100.
8. Hackam DG, Redelmeier DA. Translation of research evidence from
animals to humans. JAMA 2006;296:1731-1732.
9. Ionnidis JP, Kim B, Trounson A. How to design preclinical studies in
nanomedicine and cell therapy to maximize the prospects of clinical
translation. Nat Biomed Eng 2018;2:797-809.
10. Hatzis C, Bedard PL, Birkbak NJ, et al. Enhancing reproducibility in
cancer drug screening: How do we move forward? Cancer Res 2014;
74:4016-4023.
11. Manolagas SC, Kronenberg HM. Reproducibility of results in pre-
clinical studies: A perspective from the bone field. J Bone Miner Res
2014;29:2131-2140.
12. Samsa G, Samsa L. A guide to reproducibility in preclinical research.
Acad Med 2018;94:47-52.
13. Freedman LP, Cockburn IM, Simcoe TS. The economics of repro-
ducibility in preclinical research. PLoS Biol 2015;13:1-9.
14. Barton MB, Frommer M, Shafiq J. Role of radiotherapy in cancer
control in low-income and middle-income countries. Lancet Oncol
2006;7:584-595.
15. Barton MB, Jacob S, Shafiq J, et al. Estimating the demand for
radiotherapy from the evidence: A review of changes from 2003 to
2012. Radiother Oncol 2014;112:140-144.
16. Miller KD, Siegel RL, Lin CC, et al. Cancer treatment and survivor-
ship statistics, 2016. CA Cancer J Clin 2016;66:271-289.
17. Noone A, Howlader N, Krapcho M, et al., eds. SEER cancer statistics
review, 1975-2015. Available at: https://seer.cancer.gov/archive/csr/
1975_2015/. Accessed October 10, 2018.
18. Bryant AK, Banegas MP, Martinez ME, et al. Trends in radiation
therapy among cancer survivors in the United States, 2000-2030.
Cancer Epidemiol Biomarkers Prev 2017;26:963-970.
19. Smith BD, Haffty BG, Wilson LD, et al. The future of radiation
oncology in the United States from 2010 to 2020: Will supply keep
pace with demand? J Clin Oncol 2010;28:5160-5165.
20. Steinberg M, McBride WH, Vlashi E, et al. National Institutes of
Health funding in Radiation Oncology: A snapshot. Int J Radiat Oncol
Biol Phys 2013;86:234-240.
21. Desrosiers M, Dewerd L, Deye J, et al. The importance of dosimetry
standardization in radiobiology. J Res Natl Inst Stand Technol 2013;
118:403-418.
22. Pedersen KH, Kunugi KA, Hammer CG, et al. Radiation biology
irradiator dose verification survey. Radiat Res 2016;185:163-168.
International Journal of Radiation Oncology  Biology  Physics
23. Stone HB, Bernhard EJ, Coleman CN, et al. Preclinical data on effi-
cacy of 10 drug-radiation combinations: Evaluations, concerns, and
recommendations. Transl Oncol 2016;9:46-56.
24. Dos Santos M, Paget V, Ben Kacem M, et al. Importance of
dosimetry protocol for cell irradiation on a low X-rays facility and
consequences for the biological response. Int J Radiat Biol 2018;94:
597-606.
25. Belley MD, Ashcraft KA, Lee C-T, et al. Microdosimetric and bio-
logical effects of photon irradiation at different energies in bone
marrow. Radiat Res 2015;184:378-391.
26. Yoshizumi T, Brady SL, Robbins ME, et al. Specific issues in small
animal dosimetry and irradiator calibration. Int J Radiat Biol 2011;87:
1001-1010.
27. Kegel P, Riballo E, Kühne M, et al. X-irradiation of cells on glass
slides has a dose doubling impact. DNA Repair (Amst) 2007;6:1692-
1697.
28. Kirsch DG, Diehn M, Kesarwala AH, et al. The future of radiobiology.
J Natl Cancer Inst 2018;110:329-340.
29. Gengenbacher N, Singhal M, Augustin HG. Preclinical mouse solid
tumour models: Status quo, challenges and perspectives. Nat Rev
Cancer 2017;17:751-765.
30. Williams JP, Brown SL, Georges GE, et al. Animal models for
medical countermeasures to radiation exposure. Radiation 2010;173:
557-578.
31. Zoetelief J, Broerse JJ, Davies RW, et al. Protocol for x-ray dosimetry
in radiobiology. Int J Radiat Biol 2001;77:817-835.
32. Seed TM, Xiao S, Manley N, et al. An interlaboratory comparison of
dosimetry for a multi-institutional radiobiological research project:
Observations, problems, solutions and lessons learned. Int J Radiat
Biol 2016;92:59-70.
33. Zoetelief J, Davies RW, Scarpa G, et al. Protocol for x-ray dosimetry
and exposure arrangements employed in studies of late somatic effects
in mammals. Int J Radiat Biol 1985;47:81-102.
34. Moher D, Liberati A, Tezlaff J, et al. Preferred Reporting Items for
Systematic Reviews and Meta-Analyses: The PRISMA statement.
PLoS One 2009;6:e1000097.
35. PubMed. Available at: https://www.ncbi.nlm.nih.gov/pubmed/.
Accessed August 30, 2017.
36. Google Scholar. Available at: https://scholar.google.com/. Accessed
November 30, 2018.
37. SCImago. SJR - SCImago Journal & Country Rank [Journal Rank].
Available at: https://www.scimagojr.com/journalrank.php. Accessed
January 16, 2019.
38. Poirier Y, Tambasco M. Experimental validation of a kV source model
and dose computation method for CBCT imaging in an anthropo-
morphic phantom. J Appl Clin Med Phys 2016;17:155-171.
39. Nobah A, Aldelaijan S, Devic S, et al. Radiochromic film based
dosimetry of image-guidance procedures on different radiotherapy
modalities. J Appl Clin Med Phys 2014;15:229-239.
40. Coleman CN, Higgins GS, Brown JM, et al. Improving the predictive
value of preclinical studies in support of radiotherapy clinical trials.
Clin Cancer Res 2016;22:3138-3147.
41. Kry S, Bazalova-Carter M. Task Group No. 319 - Task Group on the
Guidelines for accurate dosimetry in radiation biology experiments
(TG319). Available at: https://www.aapm.org/org/structure/?commi
ttee_codeZTG319. Accessed August 11, 2019.