Environment International 169 (2022) 107468

Contents lists available at ScienceDirect

Environment International
journal homepage: www.elsevier.com/locate/envint

Full length article

Systematic evidence map (SEM) template: Report format and methods used
for the US EPA Integrated Risk Information System (IRIS) program,
Provisional Peer Reviewed Toxicity Value (PPRTV) program, and other “fit
for purpose” literature-based human health analyses
Kristina A. Thayer a, *, Michelle Angrish a, Xabier Arzuaga a, Laura M. Carlson b, Allen Davis a,
Laura Dishaw a, Ingrid Druwe a, Catherine Gibbons a, Barbara Glenn a, Ryan Jones b,
J. Phillip Kaiser a, Channa Keshava a, Nagalakshmi Keshava a, Andrew Kraft a, Lucina Lizarraga a,
Amanda Persad a, Elizabeth G. Radke a, Glenn Rice a, Brittany Schulz c, Rachel M. Shaffer a,
Teresa Shannon a, Andrew Shapiro b, Shane Thacker b, Suryanarayana V. Vulimiri a,
Antony J. Williams d, George Woodall b, Erin Yost a, Robyn Blain e, Katherine Duke e,
Alexandra E. Goldstone e, Pam Hartman e, Kevin Hobbie e, Brandall Ingle e, Courtney Lemeris e,
Cynthia Lin e, Alex Lindahl e, Kristen McKinley e, Parnian Soleymani e, Nicole Vetter e
a
Center for Public Health and Environmental Assessment, Chemical & Pollutant Assessment Division, US EPA, DC, USA
b
Center for Public Health and Environmental Assessment, Health & Environmental Effects Assessment Division, US EPA, NC, USA
c
Oak Ridge Associated Universities, TN, USA
d
Center for Computational Toxicology and Exposure, US EPA, NC, USA
e
ICF, VA, USA

A R T I C L E I N F O A B S T R A C T

Handling Editor: Adrian Covaci Background: Systematic evidence maps (SEMs) are gaining visibility in environmental health for their utility to
serve as problem formulation tools and assist in decision-making, especially for priority setting. SEMs are now
Keywords: routinely prepared as part of the assessment development process for the US Environmental Protection Agency
Systematic review (EPA) Integrated Risk Information System (IRIS) and Provisional Peer Reviewed Toxicity Value (PPRTV) as­
Risk assessment
sessments. SEMs can also be prepared to explore the available literature for an individual chemical or groups of
Hazard characterization
chemicals of emerging interest.
Toxicity
Scoping review Objectives: This document describes the typical methods used to produce SEMs for the IRIS and PPRTV Programs,
Systematic evidence map as well as “fit for purpose” applications using a variety of examples drawn from existing analyses. It is intended to
serve as an example base template that can be adapted as needed for the specific SEM. The presented methods
include workflows intended to facilitate rapid production. The Populations, Exposures, Comparators and Out­
comes (PECO) criteria are typically kept broad to identify mammalian animal bioassay and epidemiological
studies that could be informative for human hazard identification. In addition, a variety of supplemental content
is tracked, e.g., studies presenting information on in vitro model systems, non-mammalian model systems,

* Corresponding author.
E-mail addresses: thayer.kris@epa.gov (K.A. Thayer), angrish.michelle@epa.gov (M. Angrish), Arzuaga.Xabier@epa.gov (X. Arzuaga), Carlson.Laura@epa.gov
(L.M. Carlson), Davis.Allen@epa.gov (A. Davis), Dishaw.Laura@epa.gov (L. Dishaw), Druwe.Ingrid@epa.gov (I. Druwe), Gibbons.Catherine@epa.gov
(C. Gibbons), Glenn.Barbara@epa.gov (B. Glenn), Jones.Ryan@epa.gov (R. Jones), Kaiser.Jonathan-Phillip@epa.gov (J. Phillip Kaiser), Keshava.Channa@epa.gov
(C. Keshava), keshava.nagu@epa.gov (N. Keshava), kraft.andrew@epa.gov (A. Kraft), lizarraga.lucina@epa.gov (L. Lizarraga), persad.Amanda@epa.gov
(A. Persad), radke-farabaugh.elizabeth@epa.gov (E.G. Radke), rice.glenn@epa.gov (G. Rice), schulz.brittany@epa.gov (B. Schulz), Shaffer.rachel@epa.gov
(R.M. Shaffer), Shannon.Teresa@epa.gov (T. Shannon), Shapiro.Andy@epa.gov (A. Shapiro), Thacker.Samuel@epa.gov (S. Thacker), Vulimiri.Sury@epa.gov
(S.V. Vulimiri), Williams.Antony@epa.gov (A.J. Williams), Woodall.George@epa.gov (G. Woodall), Yost.Erin@epa.gov (E. Yost), robyn.blain@icf.com (R. Blain),
katie.duke@icf.com (K. Duke), Ali.Goldstone@icf.com (A.E. Goldstone), pamela.hartman@icf.com (P. Hartman), kevin.hobbie@icf.com (K. Hobbie), Ingle.
Brandall@epa.gov (B. Ingle), courtney.lemeris@icf.com (C. Lemeris), cynthia.lin@icf.com (C. Lin), alexander.lindahl@icf.com (A. Lindahl), kristen.mckinley@icf.
com (K. McKinley), parnian.soleymani@icf.com (P. Soleymani), nicole.vetter@icf.com (N. Vetter).

https://doi.org/10.1016/j.envint.2022.107468
Received 27 April 2022; Received in revised form 1 August 2022; Accepted 9 August 2022
Available online 13 August 2022
0160-4120/Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
K.A. Thayer et al. Environment International 169 (2022) 107468

exposure-level-only studies in humans, pharmacokinetic models, and absorption, distribution, metabolism, and
excretion (ADME). The availability of New Approach Methods (NAMs) evidence is also tracked (e.g., high
throughput, transcriptomic, in silico, etc.). Genotoxicity studies may be considered as PECO relevant or sup­
plemental material, depending on the topic and context of the review. Standard systematic review practices (e.g.,
two independent reviewers per record) and specialized software applications are used to search and screen the
literature and may include the use of machine learning software. Mammalian bioassay and epidemiological
studies that meet the PECO criteria after full-text review are briefly summarized using structured web-based
extraction forms with respect to study design and health system(s) assessed. Extracted data is available in
interactive visual formats and can be downloaded in open access formats. Methods for conducting study eval­
uation are also presented which is conducted on a case-by-case basis, depending on the usage of the SEM.

Background and how to use this template
Systematic Evidence Maps (SEMs) are proving to be a particularly
valuable analysis tool to inform the scope of complex human health
assessments, like those conducted by the US Environmental Protection
Agency (EPA)’s Integrated Risk Information System (IRIS) Program. The
same workflows can be used to produce SEMs for other EPA Office of
Research and Development (ORD) human health assessments, such as
those developed by the Provisional Peer-Reviewed Toxicity Value
(PPRTV) Program. SEMs can also be prepared to explore the available
literature for an individual chemical or groups of chemicals of emerging
interest. A companion publication describes the genesis and more spe­
cific applications of SEMs within the IRIS Program (Thayer, 2022,
10259559}. The purpose of this document is to disseminate a template
that presents typical methods used to develop an IRIS SEM and a range
of options to communicate the findings (results) of the SEM.

Fig. 1. Overview of the SEM steps, guiding questions and considerations, and associated location in this template.

2
K.A. Thayer et al. Environment International 169 (2022) 107468

Table 1
Example populations, exposures, comparators, and outcomes (PECO) criteria.
PECO element Description

Population
Exposures
Comparators [Note: PECO
element name can be adjusted
toward SEM-specific aims, e.
g., Comparisons or
Comparison group may be
more precise for SEMs
targeted toward identifying
dose response suitable
studies.]
Outcomes
Human: Any population and lifestage (occupational or general population, including children and other sensitive populations).
Animal: Nonhuman mammalian animal species (whole organism) of any lifestage (including fetal, early postnatal, adolescents and adults). [Note:
Full-text retrieval is recommended for studies of transgenic model systems. These studies typically provide mechanistic evidence tracked as “potentially relevant
supplemental material” but may present phenotypic information in wildtype animals that meets PECO criteria but is not mentioned in the abstract].
[Note: For both human and animal studies, any study that includes populations that address susceptibility should also be tagged with that supplemental content
tag.]
Relevant forms:
[chemical X] (CAS number)
Other forms of [chemical X] that readily dissociate (e.g., list any salts, etc.)
Metabolites of interest, including metabolites used to estimate exposures to [chemical X]
Occupations that may be considered surrogates of exposure
Human: Any exposure to [chemical X] [via [oral or inhalation] route[s] if applicable]. Studies will also be included if biomarkers of exposure are
evaluated (e.g., measured chemical or metabolite levels in tissues or bodily fluids) but the exposure route is unclear or likely from multiple routes.
Other exposure routes, such as those that are clearly dermal, will be tracked during title and abstract screening and tagged as “potentially relevant
supplemental material.”
[Note: Specify if certain exposure assessment matrices or methods will NOT be included. Also, although many epidemiology studies measure multiple chemicals,
they are not considered mixture studies (a type of supplemental content) as long as the study presents health outcome analyses specific to the chemical(s) of
interest. For most assessments that include hazard assessment, exposure does not have to be quantitated, e.g., low versus high is considered to meet PECO.]
Animal: Any exposure to [chemical X] via [oral or inhalation] route[s] of >1 day duration, or any duration assessing exposure during reproduction
or development. Studies involving exposures to mixtures will be included only if they include an experimental arm with exposure to [chemical X]
alone. Other exposure routes, including [dermal or injection], will be tracked during title and abstract as “potentially relevant supplemental
material.” [Note: Typically, IRIS assessments to develop chronic toxicity values consider acute non-developmental exposure studies (<1 day duration) as
supplemental content, but this can be adjusted depending on assessment needs. Also, the assessment-specific protocol should specify if certain exposures/study
designs will NOT be included, or if a minimum number of dose or concentration levels tested in experimental animal studies is indicated.]
Human, Example A (general SEM): A comparison or referent population exposed to lower levels (or no exposure/exposure below detection limits),
or exposure for shorter periods of time, or cases versus controls, or a repeated measures design. However, worker surveillance studies are considered
to meet PECO criteria even if no statistical analyses using a referent group is presented. Case reports or case series of > 3 people will be considered to
meet PECO criteria, while case reports describing findings in 1–3 people will be tracked as “potentially relevant supplemental material.”
Human, Example B (targeted SEM to identify studies suitable for dose response): Studies reporting effect measures (e.g., relative risk,
standardized mortality ratio, beta coefficients) based on a comparison or referent population exposed to lower levels (or no exposure/exposure below
detection limits), or cases versus controls, or a repeated measures design.
[Notes: Studies based exclusively on duration of exposure analyses (i.e., longer versus shorter exposure duration) are not likely to be informative for SEMs
focused on identifying studies plausibly suitable for dose response.]
Animal: A concurrent control group exposed to vehicle-only treatment and/or untreated control (control could be a baseline measurement, e.g.,
acute toxicity studies of mortality, or a repeated measure design).
All health outcomes (cancer and noncancer). In general, endpoints related to clinical diagnostic criteria, disease outcomes, biochemical,
histopathological examination, or other apical/phenotypic outcomes are considered to meet PECO criteria
[Note: Studies meeting PECO criteria may also contain supplemental mechanistic content that describes biological or chemical events associated with phenotypic
effects. When this occurs, these studies are also tagged as having supplemental mechanistic information. This typically happens during full-text review or doing
the literature inventory. Full-text retrieval is recommended for studies of transgenic model systems that meet E and C criteria because they may present
phenotypic information in wildtype animals that meet P and O criteria but is not reported in the abstract.]

This template is organized by sections to match how SEMs are often
presented in journal articles (i.e., a study structure of introduction,
methods, results, and conclusions). Within each section there are de­
scriptions of the type of content that can be included. When possible,
template language is provided when the process being described is
similar across different chemical SEMs (e.g., methods to screen studies
for inclusion, study evaluation methods, data extraction and manage­
ment). The template also presents example language to describe SEM
components that are optional or very in level of detail across SEMs,
depending on their decision-making context. For example, there is an
optional section on study evaluation methods and a range of options for
the level of detail used to describe study design and results. When
template text is not possible to develop because the nature of the content
is very chemical-specific, the user is referred to publicly accessible ex­
amples. Fig. 1 provides an overview of the SEM steps, guiding questions
and considerations, and the associated location in the template.
SEM template
1. Introduction
The introduction should include a rationale on why the SEM is being
developed and background on the topic. The level of detail varies
depending on the context for preparing the SEM. Examples of intro­
ductory text developed to support IRIS problem formulation can be
found in published IRIS Assessment Plans (IAPs) and Systematic Review
Protocols e.g.,
• IRIS Assessment Plan for Vanadium and Compounds, Oral (U.S. EPA,
2020b),
• IRIS Assessment Plan for Inorganic Mercury Salts (U.S. EPA, 2019b),
and.
• Systematic Review Protocol for perfluorobutanoic acid (PFBA),
perfluorohexanoic acid (PFHxA), perfluorohexanesulfonic acid
(PFHxS), perfluorononanoic acid (PFNA), and perfluorodecanoic
acid (PFDA) IRIS Assessments (U.S. EPA, 2019c).
Examples of introductory text developed for PPRTV assessments can
be found on the PPRTV website, e.g.,
• Provisional Peer-Reviewed Toxicity Values for Trans-
Crotonaldehyde (U.S. EPA, 2021),
• Provisional Peer-Reviewed Toxicity Value assessment for Glyci­
daldehyde (CASRN 765-34-4) (U.S. EPA, 2020h).
Examples are also available of introductory text for SEMs published
in journal articles (Yost et al., 2021; Keshava et al., 2020; Yost et al.,
2019; Walker et al., 2018; Carlson et al., 2022; Radke et al., In Press).
The introduction may contain information on chemical and physical
properties. Within the IRIS and PPRTV Programs, this information is
typically presented as a table of values and chemical structures (when
available) from the EPA CompTox Chemicals Dashboard (https://comp
tox.epa.gov/dashboard), also referred to as the “Dashboard” (Williams
et al., 2021). [Note:
The physicochemical properties in the summary tables are based on

3
K.A. Thayer et al.
Table 2
Categories of potentially relevant supplemental material.
Category (Tag) Description
Pharmacokinetics Data Potentially Informative to Assessment Analyses
Classical Classical
pharmacokinetic (PK) Pharmacokinetic or
or physiologically Dosimetry Model Studies:
based pharmacokinetic Classical PK or dosimetry
(PBPK) model studies modeling usually divides
the body into just one or
two compartments, which
are not specified by
physiology, where
movement of a chemical
into, between, and out of
the compartments is
quantified empirically by
fitting model parameters to
ADME (absorption,
distribution, metabolism,
and excretion) data. This
category is for papers that
provide detailed
descriptions of PK models
but are not PBPK models.
• The data are typically the
concentration time-
course in blood or plasma
after oral and or intrave­ evaluation of ADME
nous exposure, but other
exposure routes can be
described.
Physiologically Based
Pharmacokinetic or
Mechanistic Dosimetry
Model Studies: PBPK
models represent the body
as various compartments
(e.g., liver, lung, slowly
perfused tissue, richly
perfused tissue) to quantify
the movement of chemicals
or particles into and out of
the body (compartments)
by defined routes of
exposure, metabolism, and
elimination, and thereby
estimate concentrations in
blood or target tissues.
• A defining characteristic
is that key parameters
are determined from a
substance’s
physicochemical
parameters (e.g., particle
size and distribution,
octanol-water partition
coefficient) and physio­
logical parameters (e.g.,
ventilation rate, tissue
volumes).
Pharmacokinetic (ADME) Pharmacokinetic (ADME)
studies are primarily
controlled experiments,
where defined exposures
usually occur by
intravenous, oral,
inhalation, or dermal
routes, and the
concentration of particles, a
chemical, or its metabolites
in blood or serum, other
body tissues, or excreta are
then measured.
Environment International 169 (2022) 107468
Table 2 (continued )
Category (Tag) Description Typical Assessment Use
Typical Assessment Use
• These data are used to inventory and
estimate the amount prioritization.
PBPK and PK model absorbed (A), distributed Standard operating
studies are included in (D), metabolized (M), procedures for PBPK/
the assessment and and/or excreted (E). PK model evaluation
evaluated for possible • ADME data can also be and the identification,
use in conducting collected from human organization, and
quantitative subjects who have had evaluation of ADME
extrapolations. PBPK/ environmental or studies are outlined in
PK models are workplace exposures that An Umbrella Quality
categorized as are not quantified or Assurance Project Plan
supplemental material fully defined. (QAPP) for PBPK models
with the expectation • ADME data, especially (U.S. EPA, 2018).
that each one will be metabolism and tissue
evaluated for partition coefficient
applicability to address information, can be
assessment generated using in vitro
extrapolation needs and model systems. Although
technical conduct. in vitro data may not be
Specialized expertise is as definitive as in vivo
required for their data, these studies should
evaluation. also be tracked as ADME.
Standard operating For large evidence bases
procedures for PBPK/ it may be appropriate to
PK model evaluation separately track the in
and the identification, vitro ADME studies.
organization, and [Note: Studies describing
environmental fate and
studies are outlined in transport or metabolism in
An Umbrella Quality bacteria or model systems
Assurance Project Plan that are not applicable to
(QAPP) for PBPK models humans or animals should
(U.S. EPA, 2018). not be tagged.]
Supplemental Evidence Potentially Informative to Assessment Analyses
Mechanistic endpoints Studies that do not meet Prioritized studies of
PECO criteria but report mechanistic endpoints
measurements that inform are described in the
the biological or chemical mechanistic synthesis
events associated with sections; subsets of the
phenotypic effects related most informative
to a health outcome. studies may become
Experimental design may part of the units of
include in vitro, in vivo (by analysis. Mechanistic
various routes of exposure; evidence can provide
includes all transgenic support for the
models), ex vivo, and relevance of animal
in silico studies in effects to humans and
mammalian and biological plausibility
nonmammalian model for evidence integration
systems. Studies using New judgments [including
Approach Methodologies MOA analyses, e.g.,
(NAMs; e.g., in vitro high using the MOA
throughput testing framework in the US
strategies, read across EPA Cancer Guidelines
applications) are also (2005)].
categorized here. Studies
where the chemical is used
as a laboratory reagent (e.
g., as a chemical probe used
to measure antibody
response) generally should
ADME studies are not be tagged.
inventoried and Mechanistic evidence can
prioritized for possible also help identify factors
inclusion in an ADME contributing to
synthesis section on the susceptibility; these studies
chemical’s PK should also be tagged
properties and for “susceptible populations.”
conducting quantitative [Note: During screening,
adjustments or especially at the title and
extrapolations (e.g., abstract (TIAB) level, it may
animal-to-human). not be readily apparent for
Specialized expertise in studies that meet P, E, and C
PK is necessary for criteria if the endpoint(s) in
(continued on next page) (continued on next page)
4
K.A. Thayer et al.
Table 2 (continued )
Category (Tag) Description
a study are best classified as
phenotypic or mechanistic
with respect to the O
criteria. In these cases, the
study should be screened as
“unclear” during TIAB
screening, and a
determination made based
on full-text review (in
consultation with a content
expert as needed). Full-text
retrieval is performed for
studies of transgenic model
systems that meet E and C
criteria to determine if they
include phenotypic
information in wildtype
animals that meet P and O
criteria that is not reported
in the abstract.]
Non-PECO animal model Studies reporting outcomes
in animal models that meet
the outcome criteria but do
not meet the population
criteria in the PECO.
Depending on the
endpoints measured in
these studies, they can also
provide mechanistic
information (in these cases
studies should also be
tagged “mechanistic
endpoints”).
{Note: This categorization
generally does not apply to
studies that use species
with limited human health
relevance (e.g., ecotoxicity-
focused studies are
typically excluded].
Non-PECO route of Epidemiological or animal
exposure studies that use a non-PECO
route of exposure, e.g.,
injection studies or dermal
studies if the dermal route
is not part of the exposure
criteria.
{Note: This categorization
generally does not apply to
epidemiological studies where
the exposure route is unclear;
such studies are considered to
meet PECO criteria if the
relevant route(s) of exposure
are plausible, with exposure
being more thoroughly
evaluated at later steps.]
Non-PECO exposure For assessments that focus
duration (optional) on chronic exposure, acute
exposure durations
(defined as animal studies
of less than 1 d) are
generally considered
supplemental. In rare cases
and for very large evidence
bases, short-term (i.e., less
than subchronic) exposure
durations could also be
categorized as
supplemental.
{Note: Some assessment
teams may prefer to keep
these studies as PECO
relevant and summarize in
Table 2 (continued )
Typical Assessment Use
Studies of non-PECO
animals, exposures, or
durations can be
summarized to inform
evaluations of
consistency (e.g., across
species or routes or
durations), coherence,
or adversity; subsets of
the most informative
studies may be included
in the unit of analysis.
These studies may also
be used to inform
evidence integration
judgments of biological
plausibility and/or
MOA analyses and thus
may be summarized as
part of the mechanistic
evidence synthesis.
5
Environment International 169 (2022) 107468
Category (Tag) Description Typical Assessment Use
the literature inventory
rather than track as
supplemental.]
Susceptible populations Studies that help to identify Provides information on
potentially susceptible factors that might
subgroups, including predispose sensitive
studies on the influence of populations or
intrinsic factors such as sex, lifestages to a higher
lifestage, or genotype to risk of adverse health
toxicity, as well as some effects following
other factors (e.g., health exposure to the
status). These are often co- chemical. This
tagged with other information is
supplemental material summarized during
categories, such as evidence integration for
mechanistic or ADME. each health effect and is
Studies meeting PECO considered during dose-
criteria that also address response, where it can
susceptibility should be co- directly impact
tagged as supplemental. modeling decisions.
{Note: Susceptibility based
on most extrinsic factors,
such as increased risk for
exposure due to residential
proximity to exposure
sources, is not considered
an indicator of susceptible
populations for the
purposes of IRIS
assessments.]
Background Information Potentially Useful to Problem Formulation and Protocol
Development (These studies fall outside the scope of IRIS assessment analyses)
Human exposure and Information regarding This information may
biomonitoring (no exposure monitoring be useful for developing
health outcome) methods and reporting that exposure criteria for
are unrelated to health study evaluation or
outcomes, but which refining problem
provide information on the formulation decisions.
following: methods for Notably, providing an
measuring human assessment of typical
exposure, biomonitoring (e. human exposures (e.g.,
g., detection of chemical in sources, levels) falls
blood, urine, hair), defining outside the scope of an
exposure sources, or IRIS assessment.
modeled estimates of
exposure (e.g., in
occupational settings).
Studies that compare
exposure levels to a
reference value, risk
threshold or assessment
points of departure are also
included in this category.
Studies related to
environmental fate and
transport are typically
tagged as background
materials unless otherwise
described in the
assessment-specific
protocol.
[Note: Assessment teams
may want to subtag studies
that describe or predict
exposure levels versus those
that present exposure
assessment methods.]
Mixture study Mixture studies use Mixture studies are
methods that do not allow tracked to help inform
investigation of the health cumulative risk
effects of exposure to the analyses, which may
chemical of interest by itself provide useful context
(e.g., animal studies that for risk assessment but
(continued on next page)
K.A. Thayer et al.
Table 2 (continued )
Category (Tag)
Case reports or case series
Other background
information
Reference Materials
Records with no original
data
Posters or conference
abstracts
information from a variety of sources, primarily from the EPA CompTox
Chemicals Dashboard and PubChem. The data obtained from the EPA
CompTox Chemicals Dashboard are of varying quality and include both
experimental and predicted data [predicted data are annotated as such]. The
data associated with the chemical substances in the CompTox Chemicals
Dashboard database have been compiled from public sources, databases and
peer-reviewed literature and have varying levels of reliability and accuracy.
Predicted data in particular have significant limitations in terms of the pre­
dictions of properties for salts, inorganic and organometallic substances Links
to many external resources are provided in the Dashboard. Expansion,
Description
lack exposure to chemical
of interest alone or
epidemiology studies that
do not evaluate associations
of the chemical of interest
with relevant health
outcome(s)).
[Note: Methods used to assess
investigation of the exposure
by itself may not be clear
from the abstract, in
particular for epidemiology
studies. When unclear, the
study is advanced to full-text
review to determine
eligibility.]
Human studies that present
an investigation of a single
exposed individual or group
of ≤ 3 subjects that describe
health outcomes after
exposure but lack a
comparison group (i.e., do
not meet the “C” in the
PECO) and typically do not
include reliable exposure
estimates.
Chemical-specific studies
that may provide
introductory information
on chemical and physical
properties (note: assessors
typically will separately
consult the EPA CompTox
Dashboard); sources,
production, and use; and
environmental occurrence
and fate. Additional
groupings of information
may be determined on an
assessment-specific basis
and some assessments may
decide to separately tag
different subsets of
information (e.g., tag
chemical properties studies
separately from those on
environmental occurrence
and fate).
Records that do not contain
original data, such as other
agency assessments,
informative scientific
literature reviews,
editorials, or
commentaries.
Records that do not contain
sufficient documentation to
support study evaluation
and data extraction.
Typical Assessment Use
fall outside the scope of
an IRIS assessment.
Tracking case studies
can facilitate awareness
of potential human
health issues missed by
other types of studies
during problem
formulation.
Although formal
analyses on these
general background
topics are not part of an
IRIS assessment, this
information may be
useful to protocol
development (e.g.,
chemical properties
information for
evaluating PK or
exposure in animal
studies). In addition,
brief summary
overviews are typically
provided in the
introductory materials.
Studies that are tracked
for potential use in
identifying missing
studies, background
information, or current
scientific opinions (e.g.,
hypothesized MOAs).
6
Environment International 169 (2022) 107468
curation, and validation of the content are ongoing. Any tables presented in
the SEM should be reviewed by chemists for obvious errors and to select the
most appropriate when multiple values were available.].
The introduction can also include a summary of existing health effect
values from federal, state, and international health agencies or other risk
assessment groups. Methods to conduct this summarization are pre­
sented in “Survey of Previous Assessments, Regulatory Reference
Values, Risk Thresholds or Assessment Based Points of Departure.”” The
available values can be summarized graphically as a reference value
array or in a tabular format that includes derivation details; see example
figures and tables from the a naphthalene SEM (Yost et al., 2021), IRIS
Assessment Plan for Vanadium and Compounds, Oral (U.S. EPA, 2020b),
IRIS Assessment Plan for Inorganic Mercury Salts (U.S. EPA, 2019b),
2. Methods
The systematic review methods used to conduct the evidence map
are outlined in the Office of Research and Development (ORD) Staff
Handbook for Developing Integrated Risk Information System (IRIS)
Assessments, referred to as the “IRIS Handbook” (U.S. EPA, 2020c).
[Note: The methods presented in this template reflect adjustments made based
on a November 2021 National Academy of Sciences, Engineering, and
Medicine (NASEM) committee review of the November 2020 IRIS Handbook
(NASEM, 2021). SEM method changes made in response to NASEM focused
mostly on study evaluation, i.e., removing a reporting quality domain,
updating the sensitivity domain, and other minor changes to improve the
clarity of other domain considerations. Other changes were mostly editorial
or to improve the clarity of describing methods (e.g., updating phrasing to
match any changes made in revised handbook). The companion publication
to this template (Thayer, 2022, 10259559} addresses a NASEM recom­
mendation to clarify how SEMs are used in the IRIS assessment development
process.].
2.1. Specific aims
[Note: The specific aims for the SEM should be presented. Below are
a set of specific aims that can be adapted depending on the context for
conducting the SEM.].
• Survey core literature: Develop a SEM to identify epidemiological
(i.e., human) and toxicological (i.e., experimental animal) literature
reporting health effects of exposure to [chemical X] as outlined by
the Populations, Exposures, Comparators and Outcomes (PECO)
criteria (Table 1).
• Identify supplemental content: Identify supplemental material as
outlined in Supplemental content (Table 2.) can include in vivo or ex
vivo mechanistic, in vitro, or in silico studies; non-mammalian model
systems; toxicokinetic and absorption, distribution, metabolism, and
excretion (ADME) studies; pharmacokinetic (PK) or physiologically-
based pharmacokinetic (PBPK) model studies; exposure character­
istics studies (no health outcome); data pertinent to identify sus­
ceptible populations (not otherwise identified by the PECO criteria);
mixture studies; routes of exposure not pertinent to the PECO; case
studies; references with no original data; and conference abstracts.
The availability of New Approach Methods (NAMs) evidence is also
tracked (e.g., high throughput, transcriptomic, in silico, etc.) pri­
marily through the mechanistic and non-mammalian model system
categories. [Note: The example text provided in this template reflects a
SEM with no PECO restrictions placed on exposure duration, but for
certain assessments acute duration studies may be considered supple­
mental and should be included in the list above].
• Provide visual overview: Create literature inventories to provide a
visual overview of studies that meet PECO criteria and are tagged as
supplemental material.
• Evaluate studies (optional): Example A. Conduct study evaluation
of individual epidemiological and toxicological studies that meet
K.A. Thayer et al.
Fig. 2. Overview of Integrated Risk Information System (IRIS) study evaluation process.(a) individual evaluation domains organized by evidence type, and (b)
individual evaluation domains judgements and definitions for overall ratings (i.e., domain and overall judgments are performed on an outcome–specific basis).
PECO criteria. [Note: Typically for SEMs, no study evaluation is con­
ducted for in vitro evidence, PBPK models, or other types of supplemental
material content, but this may be included if desired.] Example B:
Conduct study evaluation for epidemiological and toxicological
studies that are most suitable for dose response analysis based on
study design considerations.
• Summarize evidence base: Provide a narrative overview of the
evidence base that summarizes the available literature (including
study evaluation observations, if conducted), identifying critical data
gaps and highlights key science issues.
• Include this specific aim when the SEM is being used to develop
a more focused PECO for subsequent assessment/full systematic
review (optional). Develop focus for subsequent assessment:
Use the results of the SEM to develop refined PECO criteria for the
assessment, define the unit(s) of analysis at the level of endpoint or
7
Environment International 169 (2022) 107468
health outcome for hazard characterization, and identify priority
analyses of supplemental material to address key science issues,
uncertainties in hazard characterization, and dose–response
analysis.
2.2. Populations, exposures, comparators, and outcomes (PECO) criteria
and supplemental material tagging
PECO criteria are used to focus the research question(s), search
terms, and inclusion/exclusion criteria used in a SEM or systematic re­
view (Table 1). [Note: A set of SEM PECO criteria is presented in and
presents alternative wording that can be used or adapted depending on the
context for conducting the SEM.] In addition to studies that meet the PECO
criteria and studies excluded as not relevant to the SEM, studies con­
taining supplemental material are inventoried during the literature
K.A. Thayer et al.
Fig. 3. Study flow diagram (Example A: no machine learning software used and no/minimal references from gray literature found).
screening process. Table 2 presents major categories of supplemental
material.
2.3. Literature search and screening processes
2.3.1. Survey of previous assessments, regulatory reference values, risk
thresholds or assessment based points of departure
Toxicity value is a broad term that encompasses reference values and
cancer risk estimates (i.e., slope factors and unit risk estimates). The
term reference value applies to values designed to provide a “bench­
mark” or exposure limit below which adverse effects on human health
8
Environment International 169 (2022) 107468
are not expected to occur (U.S. EPA, 2002). Reference values are the
most common final output from the dose response assessment compo­
nent of the risk assessment paradigm set forth by the National Research
Council (NRC, 2009, 1983) and are based on an observed or estimated
threshold for an effect, usually noncancer. Appendix A presents a list of
organizations that disseminate toxicity values that can be queried to
conduct a survey. In addition to these sources, the ToxVal database on
the EPA CompTox Dashboard (https://comptox.epa.gov/dashboard/ch
emical_lists/TOXVAL_V5) can be searched for reference values, risk es­
timate values, and point of departure (PODs) as described in Appendix C
(Williams et al., 2021). [Note: The data obtained from the EPA CompTox
K.A. Thayer et al.
Fig. 4. Study flow diagram (Example B: no machine learning software used, numerous references from gray literature found, and SEM focused on identifying studies
potentially suitable for POD derivation). Studies can be tagged to multiple supplemental tags, therefore, total number of supplemental subtags is greater than the total
number of supplemental references.
Chemicals Dashboard, including the ToxVal database, are of varying quality.
In brief, ToxValDB collates publicly available toxicity dose–effect related
summary values typically used in chemical assessments. Many of the PODs
presented in ToxVal are based on gray literature studies or assessments not
available in databases such as PubMed and Web of Science (WoS). Although
many of the resources included in Appendix A and below in “Searching Other
Resources” are represented in ToxVal, they are also manually searched
because most of the ToxVal entries have not undergone quality control to
ensure accuracy or completeness and might not include recent studies.
Searching ToxVal is considered optional and may perhaps be most useful
when searching chemicals that are relatively data poor to identify unpub­
lished grey literature.].
The available toxicity values can be summarized graphically as a
reference value array and/or in tabular format that includes derivation
details; see examples presented in Fig. 1, Fig. 2, Supplemental Materials
9
Environment International 169 (2022) 107468
Table A1 and A2 from a naphthalene SEM (Yost et al., 2021). Values
without derivation details or that are derived from another agency’s
value are summarized in a separate table from those values that have
derivation details available and are not shown in the toxicity value
array. Tables and figures indicate the month and year the searches were
conducted. A more detailed explanation of the development and evo­
lution of the graphical toxicity value arrays used by the IRIS Program,
along with chemical-specific examples, can be found in reports (U.S.
EPA, 2013, 2009) and a peer-reviewed publication (Woodall et al., In
Press).
2.3.2. Literature search strategies
2.3.2.1. Database search term development. Example A (EPA CompTox
Chemicals Dashboard as primary resource) [Note: Typically Example B is
K.A. Thayer et al.
used but use of the Chemicals Dashboard only can be a pragmatic option
when there is a need to simultaneously develop search strategies for large
numbers of chemicals, e.g., searches on hundreds of per-and polyfluoroalkyl
substances (PFAS) compounds (Carlson et al., 2022).].
The literature search focused on the chemical name (and synonyms,
trade names, and metabolites/degradants of interest) with no additional
limits. No language restrictions were applied. Chemical synonyms were
identified by searching the EPA CompTox Chemicals Dashboard (Wil­
liams et al., 2021; U.S. EPA, 2020a) and selecting those indicated as
“valid” or “good.” The preferred chemical name (as presented in the
CompTox Chemicals Dashboard), CASRN, and synonyms are used to
identify existing toxicity values (See Appendix A) and shared with in­
formation specialists who used these inputs to develop search strategies
tailored for each of the databases below because each database has its
own search architecture. Full details of the search strategy for each
database are presented (see Appendix B for example presentation).
Example B (SWIFT Review + EPA CompTox Chemicals Dashboard).
The literature search focused on the chemical name (and synonyms,
trade names, and metabolites/degradants of interest) with no additional
limits. No language restrictions applied. Chemical synonyms are iden­
tified by first using the “Find Chemical Synonyms” feature in SWIFT
(Sciome Workbench for Interactive computer-Facilitated Text-mining)
Review (Howard et al., 2016). In brief, this feature automatically creates
a PubMed-formatted chemical search using (1) the common name for
the chemical as presented in the Tox21 chemical inventory list (U.S.
EPA, 2020i); (2) the Chemical Abstract Services Registry Number
(CASRN); (3) synonyms from the ChemIDPlus database, which currently
contains chemical names and synonyms for over 400,000 chemicals; and
(4) removal of ambiguous or short alphanumeric terms that could lead
to false positives. This search is manually reviewed to ensure that any
synonyms listed in EPA’s CompTox Chemicals Dashboard (Williams
et al., 2021; U.S. EPA, 2020a) as “valid” or “good” are included. The
preferred chemical name (as presented in the CompTox Chemicals
Dashboard), CASRN, and synonyms are used to identify existing toxicity
values (See Appendix A) and shared with information specialists who
used these inputs The PubMed search created from SWIFT Review is
shared with information specialists to develop search strategies tailored
for each of the databases below because each database has its own
search architecture. Full details of the search strategy for each database
are presented (see Appendix B for example presentation).
When occupations are considered for surrogates for exposure
(optional): When applicable, the SEM should include a methodological
description of how these searches were conducted. Typically, separate
searches are conducted to identify pertinent occupational health studies
because these studies may not mention the chemical name in the title or
abstract. Resources such as prior assessments or reviews, information
from the Occupational Safety and Health Administration (OSHA), or
information from the National Institute for Occupational Safety and
Health (NIOSH) can be used to identify the occupations.
2.3.2.2. Database searches. Example A (Database Search Only) [Note:
Example A is typically only used when the number of records retrieved is
small and can be screened quickly. Example B allows filtering to the most
records most likely pertinent to a human health assessment.]
The databases listed below are searched by an information specialist
in [insert month and year]. Retrieved references are imported into the
EPA’s Health and Environmental Research Online (HERO) database and
undergo a round of deduplication in HERO. Deduplication in HERO
involves first determining whether a matching unique ID exists (e.g.,
PMID, WoSID, or DOI). If one matches one that already exists in HERO,
HERO will tag the existing reference instead of adding the reference
again. Second, HERO checks if the same journal, volume, issue and page
number are already in HERO. Third, HERO matches on the title, year,
and first author. Title comparisons ignore punctuation and case. The
literature search is updated throughout SEM development. The last full
10
Environment International 169 (2022) 107468
literature database search update is conducted several months prior to
the planned dissemination of the SEM. Literature search update dates
are documented in the full details of the search strategy for each data­
base (see Appendix B for example). Searching the other resources
mentioned below is typically done once in the early phases of con­
ducting the SEM.
The literature search should be updated to identify new literature
published during assessment development and review. Studies identi­
fied in literature search updates are generally screened according to
both the SEM and refined PECO criteria, which allows the SEM to be
continually updated throughout assessment development; however,
depending on the size and scope of the assessment, the team may elect to
screen the literature search updates only according to refined PECO
criteria. The last full literature search update is conducted several
months prior to the planned release of the draft document for public
comment.
Returned references (i.e., the number of “hits” from each electronic
database or other literature source), including the results of any litera­
ture search updates, are documented (see Appendix B for example) and
the literature flow diagrams presented in Results. In addition to the
databases listed below, a variety of other resources are subsequently
searched using customized processes (see “Other Resources”). [Note:
Include mention of any special approaches to literature identification (e.g.,
use of other assessments as starting points to identify evidence that meets the
PECO criteria.].
Accurate documentation of the search strategy is essential to the
systematic review process. Documentation of literature searches should
include the database(s) and date range covered by the search, search
terms used and the filters (e.g., matching specific article types or
PubMed Medical Subject Headings [MeSH] terms; matching topic areas
in Web of Science) that were applied, and date(s) that the searches were
performed (see an example template for documentation in Table 4–2).
• PubMed (National Library of Medicine).
• Web of Science (Thomson Reuters).
• ProQuest.
Example B (Database Searches and Subsequent Filtering of Retrieved
References Using SWIFT Review).
The unique studies are imported into SWIFT Review software
(Howard et al., 2016) to identify those references most likely to be
applicable to a human health risk assessment. In brief, SWIFT Review
has pre-set literature search strategies (“filters”) developed by infor­
mation specialists that can be applied to identify studies that are more
likely to be useful for identifying human health content from those that
likely do not (e.g., environmental fate). The filters function like a typical
search strategy where studies are tagged as belonging to a certain filter if
the terms in the filter literature search strategy appear in title, abstract,
keyword or medical subject headings (MeSH) fields content. The applied
SWIFT Review filters focus on all lines of evidence used in human health
assessments: human, animal models for human health, and in vitro
studies. [Note: The lines of evidence filters can be tailored when needed, e.g.,
human evidence only.] The details of the search strategies that underlie
the filters are available online. Studies not retrieved using the search
strategies are not considered further. Studies that include one or more of
the search terms in the title, abstract, keyword, or MeSH fields are
exported as a RIS file for screening as described below. Application of
the SWIFT Review evidence stream filters reduced the number of studies
for title and abstract screening from [insert pre-SWIFT Review number] to
[insert post-SWIFT Review number].
[Note: SWIFT Review also has other filters that allow for more granular
identification of human health literature, including health systems (cancer,
cardiovascular, developmental, endocrine, gastrointestinal, hematological
and immune, hepatic, mortality, musculoskeletal, neurological, nutrition and
metabolic, ocular and sensory, renal, reproductive, respiratory, skin and
connective tissue, ADME, PBPK, and epidemiologic quantitative analyses.
K.A. Thayer et al.
Other evidence stream filters pertinent to environmental assessments are also
available, e.g., environmental fate, physicochemical properties, ecotoxicol­
ogy, plant.].
2.3.2.3. Searching other resources. The literature search strategies
described above are designed to be broad, but like any search strategy,
studies may be missed (e.g., cases where the specific chemical is not
mentioned in title, abstract, or keyword content; ability to capture
“gray” literature that is not indexed in the databases listed above). Thus,
in addition to the database searches, the sources below are used to
identify studies that may have been missed based on the database
search. These sources are searched using customized processes described
in Appendix C. References that appear to meet the PECO criteria are
uploaded into the screening software, annotated with respect to source
of the record, and screened according to PECO as described below.
Searching of these sources is summarized to include the source type or
name, the search string (when applicable), the URL (when available and
applicable), number of results, and number of unique references not
otherwise identified from database searching. To identify unique refer­
ences, a citation for each identified study is generated in HERO and
verified that it is not already identified from the database searches (e.g.,
PubMed, WoS, etc) prior to moving forward to screening. A training
guide for conducting the gray literature searches below is available in
the Health Assessment Workspace Collaborative (HAWC) project “SEM
Template Figures and Resources” (see “Tips for Searching Grey Litera­
ture” attachment). Some of the resources listed below are listed as
optional, e.g., searches of gene expression databases may be more
appropriate for data poor chemicals. [Note: Assessment creation on EPA
deployment of HAWC is currently limited to EPA staff and contractors.
However, a separate free and open source deployment, https://hawcproject.
org/ is available for hosting assessments not affiliated with EPA. This external
website and content on it is not endorsed or supported by the US EPA;
however; it is nearly identical to EPA deployment and shares the same source
code.].
• Manual review (at the title level) of reference/citation list in studies
screened as PECO-relevant after full-text review.
• Review of the reference list from final or publicly available draft or
finalized assessments (e.g., EPA IRIS [Integrated Risk Information
System], ATSDR [Agency for Toxic Substances and Disease Registry]
Toxicological Profile) or from published journal review specifically
focused on human health. Reviews and assessments can be identified
from the database search, the resources listed in Appendix A, or from
the EPA CompTox Chemicals Dashboard ToxVal database (Williams
et al., 2021; U.S. EPA, 2020a).
• EPA CompTox Chemicals Dashboard ToxVal database
(optional.) Retrieval of references from EPA CompTox Chemicals
Dashboard ToxVal database (Williams et al., 2021; U.S. EPA, 2020a)
to identify studies or assessments that present Point of Departure
(POD) information. ToxValDB collates publicly available toxicity
dose–effect related summary values typically used in risk assess­
ments. These include POD data collected from data sources within
ACToR (Aggregated Computational Toxicology Resource) and Tox­
RefDB, and no-observed and lowest-observed (adverse) effect levels
(NOEL, NOAEL, LOEL, LOAEL) data extracted from repeated dose
toxicity studies submitted under REACH (Registration, Evaluation,
Authorisation and Restriction of Chemicals). Also included are
reference dose and concentration values (RfDs and RfCs) from EPA’s
Integrated Risk Information System (IRIS) and dose descriptors from
EPA’s Provisional Peer-Reviewed Toxicity Values (PPRTV) docu­
ments. Acute toxicity information was extracted from a number of
different sources, including: OECD eChemPortal, ECHA (European
Chemicals Agency), NLM (National Library of Medicine) HSDB
(Hazardous Substances Data Bank), ChemIDplus via EPA TEST
(Toxicity Estimation Software Tool), and the EU JRC (Joint Research
11
Environment International 169 (2022) 107468
Centre) AcutoxBase. Data from the EU COSMOS database project
have also been included in ToxValDB. Many of the PODs presented in
ToxValDB are based on gray literature studies or assessments not
available in databases such as PubMed, WoS, etc. Although many of
the resources included in the “Other Sources Consulted” list are
represented in ToxValDB, they are also manually searched because
most of the ToxValDB entries have not undergone quality control to
ensure accuracy or completeness and might not include recent
studies. Searching ToxValDB can be helpful to provide an indication
of how much gray literature may be available for a chemical.
• European Chemicals Agency (ECHA) registration dossiers to identify
data submitted by registrants. Registration dossiers contain data on
substances such as hazardous properties, safe uses, classifications,
environmental fate, and ecotoxicological and toxicological infor­
mation. The amount of information provided for each substance
varies and is obtained directly from companies’ REACH registrations.
ECHA does not give any guarantees or warranties regarding the
quality and correctness of the published information. The informa­
tion in the portal is published ‘as provided’ by industry, and its ac­
curacy has not been verified by ECHA (https://echa.europa.
eu/information-on-chemicals/registered-substances).
• EPA ChemView database (U.S. EPA, 2019a) to identify unpublished
studies, information submitted to EPA under Toxic Substances Con­
trol Act (TSCA) Section 4 (chemical testing results), Section 8(d)
(health and safety studies), Section 8(e) (substantial risk of injury to
health or the environment notices), and FYI (For Your Information,
voluntary documents). Other databases accessible via ChemView
include EPA’s High Production Volume (HPV) Challenge database
(https://iaspub.epa.gov/oppthpv/public_search.html_page) and the
Toxic Release Inventory database.
• National Toxicology Program (NTP) Chemical Effects in Biological
Systems (CEBS) database of study results and research projects (htt
ps://cebs.niehs.nih.gov/cebs/).
• The Organisation for Economic Cooperation and Development
(OECD) eChemPortal to retrieve results for OECD Screening Infor­
mation DataSet (SIDS) and High Production Volume (HPV) Chem­
icals (https://www.echemportal.org/echemportal/).
• ECOTOX database (optional). Review of the list of references in the
ECOTOX database for the chemical(s) of interest (https://cfpub.epa.
gov/ecotox/).
• References identified by technical consultants, during peer-review,
and during public comment periods (when applicable).
• ToxCast or Tox21 high throughput screening information
(optional). The EPA CompTox Chemical Dashboard (Williams et al.,
2021; U.S. EPA, 2020a) to retrieve a summary of any ToxCast or
Tox21 high throughput screening information. This data can be used
to generate mechanistic insight, predict outcome using appropriate
models, and potentially inform dose response modeling. Their
importance for outcome prediction and dose–response modeling
depends on the context, size and quality of retrieved results and the
lack of availability of other data typically used for these purposes.
[Note: Searching for high throughput screening data is done on a case­
–by–case basis, largely dependent on the problem formulation (i.e., de­
cision context and acceptable level of confidence), size of the evidence
base and need to track NAM evidence.].
• Comparative Toxicogenomics Database (CTD) (optional).
Comparative Toxicogenomics Database (CTD), available at
https://ctdbase.org/. The CTD is a freely available public resource
that curates and interrelates chemical, gene/protein, phenotype,
disease, organism, and exposure data (Davis et al., 2021). A list of
genes retrieved from the CTD can be informative, depending on its
size and context, and help generate mechanistic inferences or sup­
port mechanistic evidence generated by other studies. References
that reported the chemical-gene interactions retrieved from the CTD
can enrich the inventory of mechanistic references. [Note: Searching
for gene expression data is done on a case-by-case basis, largely
K.A. Thayer et al.
dependent on the problem formulation (i.e., decision context and
acceptable level of confidence), size of the evidence base and need to
track NAM evidence.].
• Other gene expression databases (optional). Public data from
gene expression studies (Gene Expression Omnibus https://www.nc
bi.nlm.nih.gov/geo/ and ArrayExpress https://www.ebi.ac.uk/arra
yexpress/). These data can be used to generate mechanistic evi­
dence, predict adverse outcomes and derive toxicity values
depending on the design of gene expression studies that produced
these data. [Note: Searching for gene expression data is done on a case-
by-case basis, largely dependent on the on the problem formulation (i.e.,
decision context and acceptable level of confidence, size of the evidence
base and need to track NAM evidence.].
• Backward and Forward Searches (optional). “Backward” searches
(to identify articles cited by included studies, reviews, or prior as­
sessments by other agencies) and “forward” searches (to identify
articles that cite those studies). [Note: This type of searching is done on
a case-by-case basis depending on factors such as whether the PECO has a
targeted evidence stream or health outcome focus, extent of the evidence,
and use of other assessments to serve as a starting point. In general, the
feasibility of conducting backward and forward searches is reduced when
the PECO is broad, and the number of included studies is large. These
searches may be more appropriate to conduct when other assessments are
used as the starting point for a review and those other assessments were
either not conducted using systematic review methods.].
2.3.2.4. Non peer-reviewed data. It is possible that unpublished data
directly relevant to the PECO may be identified during assessment
development. In these instances, EPA will try to get permission to make
the data publicly available (e.g., in HERO); non peer-reviewed/
unpublished data that cannot be made publicly available are not used
in IRIS or PPRTV assessments. In addition, on rare occasions where
unpublished data would be used to support key assessment decisions (e.
g., deriving a toxicity value), EPA may obtain external peer review if the
owners of the data are willing to have the study details and results made
publicly accessible, or if an unpublished report is publicly accessible (or
submitted to EPA in a non-confidential manner) (U.S. EPA, 2015). The
SEM may help identify studies to consider for external peer review in
order to support an assessment analysis, but conducting that review
would be beyond the function of an SEM.
Unpublished data from personal author communication can sup­
plement a peer-reviewed study as long as the information is made
publicly available. If such ancillary information is acquired, it will be
documented in HAWC or HERO project page (depending on the nature
of the information received).
2.3.3. Literature screening processes
2.3.3.1. Title and abstract (TIAB) screening. Example A (Full Manual
Review).
Studies identified from the database searches [and Swift Review, if
applicable] are imported into DistillerSR software (https://www.
evidencepartners.com/products/distillersr-systematic-review-softwa
re/) for screening [Note: DistillerSR is a commonly used screening appli­
cation, but other screening software can be used, see
https://systematicreviewtools.com/ for options]. TIAB screening is con­
ducted by two independent reviewers and any screening conflicts are
resolved by discussion between the primary screeners with consultation
by a third reviewer, if needed. A pilot screening effort is undertaken to
confirm clarity of the screening criteria and adjust instructions. For ci­
tations with no abstract, articles are initially screened based on the
following: title relevance (title should indicate clear relevance), and
page length (articles two pages in length or less are assumed to be
conference reports, editorials, or letters). Eligibility status of non-
English studies is assessed using the same approach with online
12
Environment International 169 (2022) 107468
translation tools or engagement with a native speaker.
Example B (use of machine learning to prioritize studies for TIAB
screening)
The studies identified from the searches described above are im­
ported into SWIFT-Active Screener (https://www.sciome.com/swi
ft-activescreener/) for TIAB screening. SWIFT-Active Screener is a
web-based collaborative software application that utilizes active ma­
chine learning approaches to reduce the screening effort (Howard et al.,
2020). [Note: SWIFT–Active Screener is a machine learning based screening
application, however other screening software can be used, see
https://systematicreviewtools.com/ for options]. TIAB screening is con­
ducted by two independent reviewers and any screening conflicts are
resolved by discussion between the primary screeners with consultation
by a third reviewer, if needed. A pilot screening effort is undertaken to
confirm clarity of the screening criteria and adjust instructions. For ci­
tations with no abstract, articles are initially screened based on the
following: title relevance (title should indicate clear relevance), and
page length (articles two pages in length or less are assumed to be
conference reports, editorials, or letters). Eligibility status of non-
English studies is assessed using the same approach with online trans­
lation tools or engagement with a native speaker.
The machine learning screening process is designed to prioritize
references that appear to meet PECO-criteria or supplemental material
content for manual review (i.e., both types of references are screened as
“include” for machine learning purposes). Screening continues until
SWIFT-Active Screener indicates that it is likely at least 95% of the
relevant studies are identified, a percent identification often used to
evaluate the performance of machine learning applications and
considered comparable to human error rates (Bannach-Brown, 2018;
Howard et al., 2016; Cohen et al., 2006). Any studies with “partially
screened” status at the time of reaching the 95% threshold are then fully
screened. Studies identified as meeting PECO criteria, unclear, or sup­
plemental material during TIAB screening are then imported into Dis­
tillerSR software (https://www.evidencepartners.com/products/dist
illersr-systematic-review-software/) either for conflict resolution or for
an additional round of more specific TIAB tagging (i.e., to separate
studies meeting PECO criteria versus supplemental content and to tag
the evidence stream or specific type of supplemental content). [Note:
DistillerSR is a commonly used screening application, but other screening
software can be used, see https://systematicreviewtools.com/ for options]. In
DistillerSR, TIAB screening is conducted by two independent reviewers
and any screening conflicts resolved by discussion between the primary
screeners with consultation by a third reviewer, if needed. Conflicts
between screeners in applying the supplemental tags, which primarily
occur at the TIAB level, are resolved similarly, erring on the side of over-
tagging based on TIAB content.
2.3.3.2. Full-Text screening. Full-text references are sought through
EPA’s HERO database for studies screened as meeting PECO criteria or
unclear based on the TIAB screening. Full-text screening occurs in Dis­
tillerSR. Full-text copies of these records are independently assessed by
two screeners using a structured form in DistillerSR to confirm eligi­
bility. Screening conflicts are resolved by discussion among the primary
screeners with consultation by a third reviewer or technical advisor (as
needed to resolve any remaining disagreements). References that are not
able to be procured within 45 days of attempt are determined to be
unavailable. Rationales for excluding studies are documented, e.g.,
study did not meet SEM PECO, full-text not available. Approaches for
language translation include online translation tools or engagement of a
native speaker. Fee-based translation services for non-English studies
are typically reserved for studies that are anticipated as being useful for
toxicity value derivation. When there are multiple publications using the
same or overlapping data, all publications are included. An example
screening form is available in Excel format (See Supplemental Excel file)
and access to the example screening form in DistillerSR is available upon
K.A. Thayer et al.
request (https://www.epa.gov/iris/forms/contact-us-about-iris) for re­
questors who have DistillerSR access.
Other exclusions based on full-text content
In addition to failure to meet PECO criteria (described above),
epidemiological and toxicological studies may be excluded at the full-
text level due to critical reporting limitations. Reporting limitations
can be identified during full-text screening but are more commonly
identified during subsequent phases of the assessment (e.g., literature
inventory, study evaluation). Regardless of when the limitation is
identified, exclusions based on full-text content are documented at the
level of full-text exclusions in literature flow diagrams with a rationale
of “critical reporting limitation.” Critical reporting information for
different study types are summarized below. For each piece of infor­
mation, if the information can be inferred (when not directly stated) for
an exposure/endpoint combination, the study should be included.
Epidemiology studies
• Sample size.
• Exposure characterization and/or measurement method.
• Outcome ascertainment method.
• Study design.
Animal studies
• Species.
• Test article name.
• Levels and duration of exposure.
• Route of exposure.
Quantitative or qualitative (e.g., photomicrographs; author-
reported lack of an effect on the outcome) results for at least one
endpoint of interest.
2.3.3.3. Multiple publications of the same data. When there are multiple
publications using the same or overlapping data, all publications are
included, with one selected for use as the primary study; the others are
considered as secondary publications with annotation in HAWC and
HERO indicating their relationship to the primary record during data
extraction. For epidemiology studies, the primary publication is gener­
ally the one with the longest follow-up, the largest number of cases, and
/ or the most recent publication date. For animal studies, the primary
publication is typically the one with the longest duration of exposure,
the largest sample size, and / or with the outcome(s) most informative to
the PECO. For both epidemiology and animal studies, the assessments
include relevant data from all publications of the study, although if the
same data are reported in more than one study, the data is o­
nly extracted once. For corrections, retractions, and other companion
documents to the included publications, a similar approach to annota­
tion is taken and the most recently published data are incorporated into
the assessments.
2.3.3.4. Supplemental material tagging. Supplemental material records
(Table 2) can be identified at both the TIAB or full-text levels. When
tagged during title and abstract screening, it is not always clear whether
the chemical of interest is reported in the study (i.e., abstracts might not
describe all chemicals investigated). In such cases, studies are still tag­
ged with the expectation that additional screening would clarify if the
study is pertinent. Conflicts between screeners in applying the supple­
mental material tags, which typically occurs at the TIAB level, are
resolved by discussion and consultation with a third reviewer (as
needed), erring on the side of over-tagging at the TIAB level when the
article content is relatively unclear.
It is important to emphasize that studies tagged as supplemental
material are not necessarily excluded from consideration in any subse­
quent assessment analysis. The tagging structure in Table 2 is designed
13
Environment International 169 (2022) 107468
to ensure the supplemental material studies are categorized for easy
retrieval while conducting the assessment. Studies that meet the SEM
PECO are those most likely to be used to derive toxicity values and thus
undergo subsequent individual study evaluation and data extraction. In
contrast, it is often difficult to assess the impact of individual studies
tagged as supplemental material on the assessment conclusions during
literature screening. These studies could emerge as being critically
important to the assessment and need to be evaluated and summarized
at the individual study level (e.g., cancer MOA or ADME studies), help
characterize susceptible populations, or might be helpful to provide
context (e.g., summarize current levels of exposure, provide hazard
evidence from routes or durations of exposure not pertinent to the SEM
PECO), or might not be cited at all in the assessment (e.g., individual
studies that contribute to a well-established scientific conclusion).
2.3.3.5. Literature flow diagrams. Results are summarized in a literature
flow diagram and interactive HAWC literature trees that show studies
“tagged” with appropriate category descriptors. i.e.., included, poten­
tially relevant supplemental material, excluded.
2.4. Literature inventory/full data extraction
[Note: At a minimum, describing study methods and results (“data
extraction”) of studies that meet PECO criteria to create the literature in­
ventory should summarize basic features of study design and health system(s)
assessed. Depending on use of the SEM and available resources, data
extraction may include a qualitative description of results (i.e., indication on
whether effects or associations were reported but not at the level of capturing
effect size information) or a quantitative description of results that includes
dose response and effect size information (“full data extraction”). Full data
extraction is not typically pursued for SEMs because it is time and resource
intensive. Full data extraction can be warranted depending on the purpose for
conducting the SEM, e.g., journal publication or to determine whether new
evidence is likely to change an existing toxicity value. Template text covering
these scenarios is presented below.].
2.4.1. Literature inventory
Human and animal studies that met PECO criteria after full-text re­
view are briefly summarized in DistillerSR using a structured data
extraction form [Note: DistillerSR is currently used to prepare literature
inventories, but similar capabilities are increasingly available in HAWC]. The
extraction fields used in these forms are available in Excel format (See
Supplemental Excel file) and access to the forms in DistillerSR is avail­
able upon request (https://www.epa.gov/iris/forms/contact-us-about
-iris) for requestors who have DistillerSR access. These study sum­
maries are referred to as literature inventories and are exported from
DistillerSR in Excel format and imported into Tableau software (https
://www.tableau.com/) or HAWC to create interactive visualizations
(see Results for example formats). For non-English studies online
translation tools (e.g., Google translator), or engagement with a native
speaker can be used to summarize studies at the level of literature in­
ventory. Fee-based translation services for non-English studies are
typically reserved for studies considered potentially informative for dose
response based on an initial analysis of study design from the literature
inventory. Summaries are extracted into DistillerSR by one team mem­
ber and the extracted data are quality checked by at least one other team
member. Discrepancies are resolved by discussion between the primary
extractors and additional consultation when needed.
Example A: Literature Inventory with Summarization of Study Design
Only [Note: Compared to Examples B and C, this level of detail is the least
resource intensive to compile and is often used in preliminary assessment
material (e.g., IAPs) as part of problem formulation to inform assessment
priorities and refine scope, identify data gaps, inform study evaluation
considerations.]
For animal studies, the following information is captured: chemical
K.A. Thayer et al.
assessed, study type (acute, short-term, subchronic, chronic, multigen­
erational, peripubertal, developmental), duration of treatment, route,
species, strain, sex, dose or exposure concentration levels tested, dose
units, health system, and specific endpoints assessed. [Note: Acute to
short-term describes exposures of days to weeks and subchronic to chronic
describes exposure of months to years in animal toxicology studies. Specific
definitions of these study duration types may vary across different SEMs and
should be clearly stated in the SEM.].
For epidemiological studies, the following information is summa­
rized: chemical assessed, population type (e.g., general population-
adult, occupational, pregnant women, infants and children), study
type (e.g., cross-sectional, cohort, case-control), sex, major route of
exposure (if known), description of how exposure was assessed, health
system and specific outcomes assessed.
For ToxVal and ECHA (optional). When the full-text report is not
available for studies identified via ToxVal (Williams et al., 2021) or
ECHA, the study citation name includes “(ToxVal)” or “(ECHA Sum­
mary)” to indicate the literature inventory is based on limited report
content.
Example B: Literature Inventory with Qualitative Summary of Study
Findings. [Note: Compared to Examples A and full data extraction, this level
of detail is intermediate with respect to resource intensity. This level of detail
is the minimal amount needed for journal publication. If resources permit,
SEMs intended for journal publication should consider full-extraction (see
below). This level of summarization may also be used in preliminary
assessment material (e.g., IAPs) as part of problem formulation to inform
assessment priorities and refine scope, identify data gaps, inform study
evaluation considerations.].
For animal studies, the following information is captured: chemical
assessed, study type (acute, short-term, subchronic, chronic, multigen­
erational, peripubertal, developmental), duration of treatment, route,
species, strain, sex, dose or concentration levels tested, dose units,
health system and specific endpoints assessed, and a summary of find­
ings at the health system level (null or no-observed-effect level/low-
observed-effect level (NOAEL/LOAEL) based on author-reported statis­
tical significance (when presented) with an indication of which specific
endpoints were affected). [Note: Acute to short-term describes exposures of
days to weeks and subchronic to chronic describes exposure of months to
years in animal toxicology studies. Specific definitions of these study duration
types may vary across different SEMs and should be clearly stated in the
SEM.] For ToxVal and ECHA (optional). When the full-text report is
not available for studies identified via the Toxicity Values (ToxVal)
(Williams et al., 2021) or the European Chemical Agency (ECHA) da­
tabases, the study citation name includes “(ToxVal)” or “(ECHA Sum­
mary)” to indicate the literature inventory is based on limited report
content.
For epidemiological studies, the following information is summa­
rized: chemical assessed, population type (e.g., general population-
adult, occupational, pregnant women, infants and children), study
type (e.g., cross-sectional, cohort, case-control), sex, major route of
exposure (if known), description of how exposure was assessed, specific
endpoints assessed and a qualitative summary of findings at the health
system level (null or an indication of any associations found).
Example C: Literature Inventory Used to Prioritize Studies for Subsequent
Study Evaluation and Full Data Extraction (Effect Size Level).
For animal studies, the following information is captured: chemical
assessed, study type (acute, short-term, subchronic, chronic, multigen­
erational, peripubertal, developmental), duration of treatment, route of
exposure, species, sex, and health system(s) assessed. [Note: Acute to
short-term describes exposures of days to weeks and subchronic to chronic
describes exposure of months to years in animal toxicology studies. Specific
definitions of these study duration types may vary across different SEMs and
should be clearly stated in the SEM.] For studies in humans, the following
information is summarized: chemical assessed, sex, population, study
design, exposure measurement, and health system(s) assessed. For
ToxVal and ECHA (optional). When the full-text report is not available
14
Environment International 169 (2022) 107468
for studies identified via the ToxVal (Williams et al., 2021) or the Eu­
ropean Chemical Agency (ECHA) databases, the study citation name
includes “(ToxVal)” or “(ECHA Summary)” to indicate the literature
inventory is based on limited report content.
Because study evaluation and full quantitative extraction is time
intensive (often ranging from 30 min to 3 h or more per study for each
activity), the literature inventory is used to prioritize studies for study
evaluation and full quantitative extraction. To prioritize, the literature
inventory inputs are analyzed with respect to the considerations below
to identify studies that plausibly could be considered suitable for iden­
tifying a subchronic or chronic POD. Studies prioritized from this pro­
cess undergo study evaluation (described below) and full data
extraction. Studies not prioritized from this process are summarized at
the literature inventory level only and primarily used to identify data
gaps, identify emerging health concerns, or to provide context when
interpreting findings from prioritized studies. [Note: Studies for which
only ToxVal or ECHA summaries are available can advance to study eval­
uation or full extraction, depending on the amount of detail in the record and
goals of the SEM. However, these studies would not typically be used in an
assessment because of lack of public access to the primary report.] Below are
factors to consider when evaluating literature inventory results to pri­
oritize studies for study evaluation and full quantitative extraction:
• Studies with chronic exposure durations or including exposure dur­
ing reproduction or development, are prioritized over studies with
shorter-term exposure durations.
• Animal studies using a species that is considered a relevant human
surrogate.
• Studies with a broad exposure range and multiple exposure levels are
preferred to the extent that they can provide information about the
shape of the exposure–response relationship [see the EPA Benchmark
Dose Technical Guidance, §2.1.1 (U.S. EPA, 2012)] and facilitate
extrapolation to more relevant (generally lower) exposures. How­
ever, single dose studies can be considered for toxicity value deri­
vation if they test phenotypic health outcomes unexamined in
multidose studies testing similar levels. For human studies, studies
for which quantitative exposure measurements were available and
exposure–response results are presented in sufficient detail (e.g.,
standardized mortality rate or relative risks, numbers of cases/con­
trols, etc) are prioritized. Studies based exclusively on duration of
exposure analyses (i.e., longer versus shorter exposure duration) are
typically not considered suitable for dose response unless additional
information on exposure can be incorporated. Human studies include
experimental (e.g., controlled exposure) and observational studies.
• For epidemiological studies, studies that used biomarker measure­
ments in tissues or bodily fluids as the metric for exposure were only
considered suitable for dose–response analysis if data or PBPK
models are available to extrapolate between the reported biomarker
measurement and the level of exposure. For both animal and human
studies, whether the nature of the outcomes/endpoints assessed were
interpretable with respect to potential adversity was considered.
Typically, apical or clinical measures (“phenotypic”) are preferred
over other endpoints for dose response. However, “mechanistic”
endpoints can be useful in dose–response analyses when they can be
reasonably established as predictive of, or strongly associated with,
phenotypic outcomes interpreted as adverse.
• High or medium confidence studies are highly preferred over low
confidence studies (see “Study Evaluation” below).
2.4.2. Full data extraction of study methods and results (optional)
[Note: Full data extraction includes the same data extraction elements as
the literature inventory but is a more detailed summarization of study design
and results compared to what is conducted in the literature inventory (e.g.,
includes description of animal husbandry, effect size level presentation of
findings). For human studies the additional detail can include information on
which covariates were included in the analysis. A variety of software
K.A. Thayer et al.
platforms can be used to conduct full data extraction and the specific
application should be specified, see https://systematicreviewtools.com/ for
options. The paragraph below has example sentences that describe sole use of
the Health Assessment Workplace Collaborative (HAWC) or a combination
of HAWC and DistillerSR.].
Example A: Full extraction conducted completely in HAWC. Full data
extraction for prioritized studies is conducted using HAWC. General
instructions for data extraction in HAWC are available at https://haw
cprd.epa.gov/about/. An additional resource used to facilitate use of a
consistent vocabulary to summarize endpoints assessed in animal
studies is available in the HAWC project “SEM Template Figures and
Resources” [see the “Environmental Health Vocabulary (EHV) - a rec­
ommended terminology for outcomes/endpoints”].
Data extraction is performed by one member of the evaluation team
and quality checked by at least one other team member. The detailed
study data extracted in HAWC can be downloaded from EPA HAWC as a
Microsoft Excel file at [insert chemical specific HAWC project URL, see
“Download datasets”].
Example B: Full Extraction conducted in HAWC for Toxicology Studies
and DistillerSR for Epidemiology Studies. Full data extraction for priori­
tized animal studies is conducted using HAWC. General instructions for
data extraction in HAWC are available at https://hawcprd.epa.
gov/about/. An additional resource used to facilitate use of a consis­
tent vocabulary to summarize endpoints assessed in animal studies is
available in the HAWC project “SEM Template Figures and Resources”
[see the “Environmental Health Vocabulary (EHV) - a recommended
terminology for outcomes/endpoints”]. For epidemiological studies, full
data extraction is conducted using DistillerSR. Access to the forms in
DistillerSR is available upon request (https://www.epa.gov/iris/fo
rms/contact-us-about-iris) for users who have DistillerSR access. EPA
is in the process of updating HAWC’s epidemiology data extraction
module and anticipates using HAWC instead of DistillerSR for full
extraction by early 2023.
Data extraction is performed by one member of the evaluation team
and quality checked by at least one other team member. The detailed
toxicology study data extracted in HAWC can be download from EPA
HAWC to Excel at [insert chemical specific HAWC project URL, see
“Download datasets”]. Detailed extraction of the epidemiology studies
conducted in DistillerSR is available at [insert chemical specific HAWC
project URL, see “Attachments” at the bottom of the landing page].
2.5. Study evaluation (optional)
2.5.1. Human, animal, and in vitro evidence
The general approach for evaluating epidemiological, animal toxi­
cological, and in vitro data is briefly described below. The approach for
evaluating studies is conceptually the same, but the application specifics
differ based on evidence stream. Key concerns are potential bias (factors
that affect the magnitude or direction of an effect in either direction) and
insensitivity (factors that limit the ability of a study to detect a true
effect; low sensitivity is a bias toward the null when an effect exists). The
study evaluations are aimed at discerning the expected magnitude of any
identified limitations (focusing on limitations that could substantively
change a result), considering the expected direction of the bias.
Examination of specific methodological features for each exposure-
outcome/endpoint combination is accomplished by applying pre­
specified considerations to a set of domains. These domains differ for
epidemiology, animal, and in vitro studies (Fig. 2). Core and prompting
questions for each domain are used to guide the reviewer to seek out and
think about relevant information pertaining to specific aspects of the
study. [Note: Any SEM-specific considerations should be described e.g.,
related to exposure or outcome assessment, confounders for epidemiology
studies. Example sentence: In addition to these base guides, specific consid­
erations were developed for [describe SEM specific considerations].
At least two reviewers independently evaluate health effect studies
for study quality evaluation to identify characteristics that bear on the
15
Environment International 169 (2022) 107468
informativeness (i.e., validity and sensitivity) of the results. The inde­
pendent reviewers use structured web-forms for study evaluation
housed within HAWC to record separate judgments for each domain and
the overall study for each outcome, to reach consensus between re­
viewers, and when necessary, resolve differences by discussion between
the reviewers or engagement of a tertiary reviewer. The web-forms are
available in HAWC and can be customized to reflect the specific tool
being used for the study evaluation.
For studies that examine more than one outcome of interest, the
evaluation process should be health outcome-specific, as the utility of a
study may vary for the different outcomes. The same is true for epide­
miology studies that assess exposure by different approaches. To cali­
brate the assessment –specific considerations for study evaluations, a
pilot phase is conducted to assess and refine the evaluation process.
Final study evaluations housed in HAWC are accessible from the
download page. Additional resources used to facilitate consistency of
study evaluations are available in the HAWC project “SEM Template
Figures and Resources” (see attachments for “Epidemiology Study
Evaluation Core, Prompting, and Follow-up Questions,” “Example An­
swers to the Animal Study Evaluation Domains,” and focused consid­
erations for some types of animal toxicology studies described in “Study
Evaluation Tips - Clinical Chemistry” and “Study Evaluation Tips -
Histopathology“).
During review, for each evaluation, domain reviewers reach a
consensus judgment of good, adequate, deficient, not reported, or critically
deficient (Fig. 2). These four categories are applied to each evaluation
domain for each study, as follows:
• Good represents a judgment that the study was conducted appro­
priately in relation to the evaluation domain, and any minor de­
ficiencies noted would not be expected to influence the study results.
• Adequate indicates a judgment that methodological limitations
related to the evaluation domain might occur, but those limitations
are unlikely to be severe or to notably impact the results.
• Deficient denotes identified biases or deficiencies interpreted as likely
to have had a notable impact on the results or that prevent inter­
pretation of the study findings.
• Not reported indicates the information necessary to evaluate the
domain question was not available in the study. Generally, this term
carries the same functional interpretation as deficient for the pur­
poses of the study confidence classification (described below).
Depending on the number of unreported items and severity of other
limitations identified in the study, reaching out to the study authors
for this information might or might not be worthwhile.
• Critically deficient reflects a judgment that the study conduct relating
to the evaluation domain question introduced a serious flaw inter­
preted to be the primary driver of any observed effect(s) or makes the
study uninterpretable. Studies with critically deficient judgments in
any evaluation domain are almost always classified as overall
uninformative.
Once the evaluation domains have been rated, the identified
strengths and limitations are considered collectively to reach an overall
confidence classification of high, medium, or low confidence, or uninfor­
mative (Fig. 2) for each health outcome of interest. The classifications,
which reflect a consensus judgment between reviewers, are defined as
follows:
• High confidence: No notable deficiencies or concerns were identified;
the potential for bias is unlikely or minimal, and the study used
sensitive methodology. High confidence studies generally reflect
judgments of good across all or most evaluation domains.
• Medium confidence: Possible deficiencies or concerns were noted,
but the limitations are unlikely to be notable. Generally, medium
confidence studies include adequate or good judgments across most
K.A. Thayer et al.
domains, with the impact of any identified limitation not being
judged as severe.
• Low confidence: Deficiencies or concerns are noted, and the potential
for bias or inadequate sensitivity could have a significant impact on
the study results or their interpretation. Typically, low confidence
studies have a deficient evaluation for one or more domains, although
some medium confidence studies might have a deficient rating in
domain(s) considered to have less influence on the magnitude or
direction of the outcome-specific results). Low confidence results are
given less weight compared to high or medium confidence results
during evidence synthesis and integration in an assessment. Studies
rated as medium or low confidence only because of sensitivity con­
cerns about bias towards the null will be asterisked or otherwise
noted because they might require additional consideration during
evidence synthesis. Effects observed in studies biased toward the null
might actually increase confidence in the results, assuming the study
is otherwise well conducted.
• Uninformative: Serious flaw(s) make the study results unusable for
dose–response. Studies with critically deficient judgments in any
evaluation domain are almost always classified as uninformative.
Studies with multiple deficient judgments across domains also might
be considered uninformative. Uninformative results are given the least
weight during evidence synthesis and integration in an assessment.
Outcomes or endpoints judged as uninformative are not used as the
basis for quantitative dose–response analysis.
2.5.2. Pharmacokinetic models
[Note: Pharmacokinetic models are not typically evaluated for SEMs, but
if considered pertinent to the goals of the SEM then the following text can be
used] Any PBPK models identified in the SEM are evaluated using
methods described in the Quality Assurance Project Plan for PBPK
models (U.S. EPA, 2018), which are summarized below and presented in
more detail in the IRIS Handbook. In practice, it has been found that
many published models have errors that affect their predictions to
varying degrees; hence, an evaluation of a model is required before it
can be used in an assessment (McLanahan et al., 2012). Thus, the models
identified as applicable to the assessment are evaluated as described
below.
Considerations for judging the suitability of a model are separated
into two categories: scientific and technical. In summary, the scientific
criteria focus on whether the biology, chemistry, and other information
available for chemical mode(s) of action (MOA[s]) are appropriately
represented by the model structure and equations. Significant to the
overall efficiency of this process, the scientific criteria can be judged by
reading the publication or report that describes the model, without
requiring an evaluation of the computer code. Preliminary technical
criteria include the availability of the computer code and apparent
completeness of parameter listing and documentation. The in-depth
technical and scientific criteria focus on the accurate implementation
of the conceptual model in the computational code, use of correct or
biologically consistent parameters in the model, and reproducibility of
model results reported in journal publications and other documents.
3. Results
3.1. Literature screening results
[Note: Below are examples of presenting the literature screening results,
including example text that can be used when machine learning software is
used, see https://systematicreviewtools.com/ for options. Links to editable
study flow diagrams in PowerPoint are also provided in the publicly accessible
HAWC project “SEM Template Figures and Resources” (see the “Example
Study Flow Figures”). Results may also be presented in interactive HAWC
Literature Trees (see Example C, Fig. 7 below).].
Example A: No Machine Learning Software Used and No/Minimal Ref­
erences from Gray Literature Found
16
Environment International 169 (2022) 107468
The identification of studies during the screening process is sum­
marized in Fig. 3. The database searches yielded 4,388 references after
duplicate removal in HERO. As described earlier, these studies were
imported into SWIFT Review and literature search filters for human,
animal, and in vitro evidence applied. Application of these filters reduced
the number of studies for TIAB screening to 1,918. During TIAB
screening, 6 were included for full-text review, 13 were tagged as sup­
plemental material, and 1,899 were excluded as not relevant to PECO.
During full-text review, 3 studies were excluded as not relevant to PECO
and 1 study was identified as supplemental. No references were identi­
fied from other sources or from a review of the reference lists for the 2
included studies. Thus, only 2 studies (1 human and 1 animal) were
considered to meet PECO criteria and included in the literature
inventory.
Example B: No Machine Learning Software Used, Numerous References
Found from Gray Literature, and SEM Focused on Identifying Studies
Potentially Suitable for POD Derivation
The flow of studies during the screening process is summarized in
Fig. 4. The database searches yielded 3,669 unique references after
duplicate removal. Application of the SWIFT Review filters (human,
animal/human health models, and in vitro) reduced the number of
studies for TIAB screening to 1,452. An additional 135 references found
during searches of other sources (were added to the TIAB screen for a
total of 1,587 unique references (Appendix C, Table C1). After TIAB
screening, 1,028 studies were excluded as not PECO relevant and
another 483 were tagged as supplemental material, leaving 76 studies
that advanced to full-text screening. During full text screening, 11 were
excluded as not meeting PECO criteria and 5 were excluded due to an
inability to obtain the full-text, and another 14 were tagged as supple­
mental material.
Of the 46 studies considered PECO relevant, 28 were human studies
and 18 were animal studies. The human and animal studies were further
inventoried as described earlier to identify those with study designs
most suitable for chronic POD derivation. Fifteen animal studies were
not considered suitable for chronic POD derivation because they
assessed acute (1 day) or short-term (<30 days) exposure duration,
leaving 31 studies (28 human, 3 animal) considered further for study
evaluation, suitability for dose response, and full extraction in HAWC.
The human studies were considered potentially useful for hazard iden­
tification but not for identifying an oral or inhalation POD after analysis
of the exposure domain. Therefore, they were not fully extracted into
HAWC.
Example text to highlight gray literature findings for key sources of animal
bioassay data:
Searches of the ToxVal database yielded 21 references, 6 of which
were mammalian toxicity studies not identified from the database
search or other resources. These 6 studies were screened and inventoried
in DistillerSR. Only 2 of these studies were developmental, reproductive,
or repeated dose studies prioritized for study evaluation and full
extraction in HAWC.
The ECHA dossier had a total of 53 references, of which 39 were
study summaries and 14 were full study reports. Citation information
was not available for many of the ECHA references. Eight of the 53
references with citation information were studies not identified from
other resources and were assigned HERO ID numbers and screened in
DistillerSR. Twenty of the study summaries with citation information
matched references included in the database search or other resources.
The remaining 25 study summaries were cited as unnamed reports.
Three of these studies were determined to be duplicates of references
already included in the literature inventory based on study year, animal
species, strain, sex, administered dose levels, and effect level de­
scriptions. Of the remaining unnamed references, 5 described in vivo
effects of inhalation or oral administration and were included in the
literature using “unnamed report” as the citation information. The other
unnamed reports describing mechanistic, basic toxicokinetics, and
dermal administration studies were tagged to the appropriate
K.A. Thayer et al. Environment International 169 (2022) 107468

Fig. 5. Study flow diagram (Example B PFAS 150 machine learning software used, numerous references from gray literature found). Studies can be tagged to
multiple supplemental tags, therefore, total number of supplemental subtags is greater than the total number of supplemental references.

17
K.A. Thayer et al. Environment International 169 (2022) 107468

Fig. 6. PFAS 150 literature tree. Click here https://hawcprd.epa.gov/lit/assessment/100500085/references/visualization/ for interactive version.

18
K.A. Thayer et al. Environment International 169 (2022) 107468

Fig. 7. Example A: Survey of PFAS 150 SEM human studies that met PECO criteria summarized by study design, population, and health systems assessed. This is a
screenshot image of the Tableau interactive dashboard for a literature inventory for a group of chemicals. The numbers in the heat map inset indicate the number of
studies that investigated a health system within a study design and population. If a study evaluated multiple health outcomes or populations, it is shown here multiple
times. The study details panel includes information on type of study, population, and exposure measurement. The dashboard is searchable by study design and health
system, but all also by chemical name or identifier (CASRN, DTXSID).

supplemental tags, with an additional “unnamed report” tag.
One reference that was not identified in the journal database search
was found by searching EPA’s ChemView Database. This reference was
assigned a HERO ID and screened in DistillerSR according to PECO and
supplemental material criteria. The reference reported an acute exper­
iment (<24 h) and a short-term experiment that were both included in
the literature inventory.
Example C: Machine Learning Software Used, Numerous References from
Gray Literature Found, HAWC Literature Tree
The database searches for per- and polyfluoroalkyl substances
(PFAS) compounds yielded 40,623 references in HERO after duplicate
removal (Fig. 5). Application of the SWIFT Review literature search
filters for human, animal, and in vitro evidence reduced the number of
studies for consideration to 10,397. The studies were screened in
Swift–Active Screener using predictive relevance, resulting in 3,690
studies being manually screened to identify 1,075 studies that were
considered potentially PECO relevant or supplemental (“included” for
the purposes of machine learning) and 2,615 references that were
excluded. After manually reviewing the 3,690 references, screening was
stopped because Swift–Active Screener indicated that it was likely that
96% of the relevant studies were identified. This represented a signifi­
cant reduction in screening effort (i.e., screening was stopped after ~
36% of the references were reviewed and 6,707 references were not
manually screened).
An additional 711 unique studies were identified from the other
sources searched, including 31 that came from reviewing the reference
lists of studies considered PECO relevant after full-text review. These
711 studies were imported into DistillerSR for a total of 1,786 studies
screened at TIAB level. During TIAB screening in DistillerSR, 749 were
included for full-text review, 819 were tagged as supplemental material,
and 218 were excluded as not relevant to PECO.
During full-text review, 225 studies were considered PECO relevant
(132 animal studies and 93 human studies), 94 studies were excluded,
and 430 studies were tagged as supplemental material. Of the 132 ani­
mal studies, 35 were prioritized for study evaluation and full extraction
because they were of 21 day or longer exposure duration, or exposures
that occurred during reproduction or development. Literature search
results are summarized graphically in Fig. 5 and Fig. 6. Chemical-
specific literature trees are also available in HAWC [https://hawcprd.
epa.gov/summary/assessment/https://doi.org/100500085/visuals/;
(U.S. EPA, 2020e)] filter by visualization type “literature tagtree”).
3.2. Characterizing human and animal evidence
[Note: This section should describe the nature of the evidence base
focusing on study design features, population or animal model system
assessed, health outcomes assessed, results (optional), and study evaluation
(optional). The text should be concise. Web-based interactive visualizations
that provide downloadable access to the summarized content are encouraged.
The example text below includes presentations of optional content on study
evaluation and full data extraction. Six Tableau example visualizations are
presented in Figs. 8, 9, 10, 12, 13 and 14 to illustrate a variety of sum­
marized content for human and animal evidence:
• Human Evidence:
o Study design only: Example A presented in Fig. 7 showing from a SEM
on 150 + PFAS chemicals.
o Study design plus quantitative description of findings: Example C pre­
sented in Fig. 8 from an analysis of epidemiological studies identified in
the SEM of 150 + PFAS chemicals.
o Study design plus qualitative description of findings: Example B pre­
sented in Fig. 9 from the vanadium, oral IAP (2020). Note: Main
findings are visible via hover over features from within the Tableau
dashboard.

19
K.A. Thayer et al. Environment International 169 (2022) 107468

Fig. 8. Example C. Survey of PFAS 150 human studies that met PECO criteria summarized by study design, population, and health systems assessed. This is a
screenshot image of the interactive dashboard. The numbers in the heat map inset indicate the number of studies that investigated a health system within a study
design and population. If a study evaluated multiple health outcomes or populations, it is shown here multiple times. The study details panel includes information on
type of study, population, exposure measurement, and findings with additional detail presented via website hover access.

• Animal Evidence:
o Study design only: Example D presented in Fig. 11 from a SEM on 150
+ PFAS chemicals.
o Study design plus qualitative description of findings (Example E):
Example E presented in Fig. 12 from the vanadium, oral IAP (2020).
Note: Main findings are visible via hover over features from within the
Tableau dashboard.].
3.2.1. Human studies
3.2.1.1. Literature inventory. Example from SEM on 150+ PFAS chem­
icals: A total of 118 publications of epidemiology studies were identified
that included information on 10 different PFAS chemicals (Fig. 7).
[Note: a presentation of studies with quantitative findings is presented in
Fig. 8]. The most frequently studied chemicals were per­
fluoroundecanoic acid (86 studies), perfluoroheptanoic acid (26
studies), perfluorooctanesulfonamide (23 studies), per­
fluoroheptanesulfonate (18 studies), perfluorotridecanoic acid (17
studies), and perfluorotetradecanoic acid (12 studies). The majority of
the searched PFAS had zero epidemiology studies available. An
interactive dashboard with the, filterable by PFAS, health effect cate­
gory, outcome, population (adults, adults and children, children < 18
years, pregnant women, occupational, or other), and study design is
available at the hyperlink referenced in Fig. 7. The most commonly
studied health effect categories were endocrine (22 studies), female
reproductive (21), developmental (21), metabolic (19), cardiovascular
(17), and immune (13) effects. The other health effect categories
had<10 studies each. The plurality of studies were cross-sectional (50
studies), followed by cohorts (49), and case-control studies (19).
Example modified from vanadium, oral IAP (U.S. EPA, 2020b): A sur­
vey of study designs and health systems assessed in the human studies
that met PECO criteria and tabular summary of study design and find­
ings is provided in Fig. 9. Human studies identified in the literature
search included nine controlled trials that administered vanadyl sulfate
or sodium metavanadate directly to study participants. Of the controlled
human trials, seven were conducted in diabetic patients for the purpose
of evaluating the therapeutic effects of vanadium supplementation, with
treatment durations of 2–6 weeks (Afkhami-Arekani et al., 2008; Cusi
et al., 2001; Goldfine et al., 2000; Boden et al., 1996; Halberstam et al.,
1996; Cohen et al., 1995; Goldfine et al., 1995); one evaluated effects of
vanadium supplementation on insulin sensitivity in healthy adults, with

20
K.A. Thayer et al. Environment International 169 (2022) 107468

Fig. 9. Example B. survey of vanadium human studies that met PECO criteria summarized by study design, population, and health systems assessed. This is a
screenshot image of the interactive dashboard. The numbers in the heat map inset indicate the number of studies that investigated a health system within a study
design and population. If a study evaluated multiple health outcomes or populations, it is shown here multiple times. The study details panel includes information on
type of study, population, exposure measurement and main findings via website hover access.

a treatment duration of 7 days; and one evaluated effects of vanadium
supplementation in weight training athletes, with a treatment duration
of 12 weeks (Fawcett et al., 1997). The literature search also identified
39 observational epidemiology studies, which evaluated the association
of health outcomes with total vanadium but the specific form of vana­
dium was not determined. This included 37 studies (n = 13 case-control,
14 cross-sectional, and 10 cohort) in which vanadium exposure was
evaluated using biomonitoring of blood (whole blood, plasma, or
serum), urine, hair, seminal plasma, cerebrospinal fluid, saliva, or nails,
but in which the route of exposure was unclear; and two ecological
studies that evaluated the association of human health outcomes with
vanadium levels in soil, drinking water, or food.
[Note: If SEM includes study evaluation, then a paragraph should be
added to describe to overall findings.].
Example study evaluation summary modified from PFAS 150: Study
evaluation results for the 26 available epidemiology studies of per­
fluoroheptanoic acid (PFHpA) are summarized in Fig. 10 and rationales
are available in HAWC (for interactive graphic: https://hawcprd.epa.
gov/summary/visual/assessment/https://doi.org/100500085/PFAS-
150-Epidemiology-Study-Evaluations-for-PFHpA/). Twelve studies
were medium confidence for at least one outcome, nine studies were low
confidence for all outcomes, and four studies were uninformative for all
outcomes. While the majority of studies were generally well conducted,
with adequate and good ratings in most domains, most available studies
were deficient for study sensitivity. This was primarily due to low
exposure levels with limited contrast in the study populations. The
median exposure in most studies (16/26) was<0.1 ng/mL, with the
highest median 0.4 ng/mL. Based on this poor sensitivity, null results in
this set of studies should not be interpreted as evidence of a lack of ef­
fect. In addition, none of the available studies were evaluated as good for
confounding. Across studies, this was influenced by the uncertainty
caused by potential for confounding across PFAS (due to moderate-high
correlations), but in most studies, other potential for residual
confounding was identified as well.
3.2.2. Animal studies
3.2.2.1. Literature inventory. Example from SEM on 150+ PFAS chem­
icals: Of the 133 animal studies, a subset of 41 studies focused on 21 day
or longer exposure durations (or were reproductive or developmental
studies) for 16 PFAS chemicals (Fig. 11). These study designs were
considered most suitable for identifying a subchronic or chronic POD
and were prioritized for study evaluation and full data extraction in
HAWC. The most commonly assessed health systems included the
reproductive (e.g., male and female reproductive organ weights,
fertility); developmental (e.g., pup weight and viability, lactation index,
skull malformation); urinary (e.g., kidney weight, organ function mea­
surements such as blood urea nitrogen); immunological (e.g., spleen
and/or thymus weight; blood components such as monocytes, eosino­
phils, and lymphocytes); and hepatic (e.g., liver weight, enzyme activity,
cholesterol, lipid metabolism, total bilirubin, nonneoplastic lesions such
as hepatocellular hypertrophy). Fig. 12 present examples visualizations
of the data extraction for some of the PFAS animal data. Access to the full
set of extracted results (~80 visualizations of extracted data from 16
PFAS) is available at https://hawcprd.epa.gov/summary/assessment/
https://doi.org/100500085/visuals/.
3.2.2.2. Literature inventory with summary of findings (optional).
Example modified from vanadium, oral IAP (U.S. EPA, 2020b): A pre­
liminary survey of study designs, species, vanadium compounds evalu­
ated, and health effects evaluated in the animal studies that met PECO
criteria is provided in Fig. 13. The animal studies evaluated exposure to
ammonium metavanadate, sodium metavanadate, sodium orthovana­
date, vanadyl sulfate, vanadium pentoxide, calcium orthovanadate, or
calcium pyrovanadate. Of these, vanadyl sulfate and sodium meta­
vanadate were the most frequently studied compounds. Two studies

21
K.A. Thayer et al. Environment International 169 (2022) 107468

Fig. 10. Example A: summary of study evaluation for PFAS 150 SEM human studies of perfluoroheptanoic acid. Click here https://hawcprd.epa.gov/summary/visua
l/assessment/100500085/PFAS-150-Epidemiology-Study-Evaluations-for-PFHpA/ for interactive version.

reported that animals were exposed to “ammonium vanadate” (Susić
and Kentera, 1986) and “sodium vanadate” (Sun et al., 2014), which
were inferred to be ammonium metavanadate and sodium meta­
vanadate (respectively) based on the synonyms described above and are
referred to accordingly here. Four studies reported that animals were
exposed to “vanadium” or “vanadate” but the specific chemical form
was unclear. The majority of studies were conducted in rats and mice,
but data were also available in rabbits, cattle, goats, and sheep. Among
the 94 available animal studies, 23 included experiments in animal
models of diabetes that evaluated the therapeutic effects of vanadium
compounds on diabetic symptoms. Tabular summaries of the study de­
signs and health effects evaluated in chronic, subchronic, and
reproductive or developmental studies that tested multiple dose levels
are provided in [Figs. 7, 8, and 9], respectively [Note: The referenced
figures are presented in the vanadium, oral IAP and not the SEM template]. In
general, these study designs are preferred for toxicity value derivation
over acute/short-term studies or studies that test a single dose level (U.S.
EPA, 2002), although there may be circumstances where other study
designs are more suitable.
3.2.2.3. Study evaluation (optional). Example from SEM on 150+ PFAS
chemicals: Study evaluations of the animal studies are summarized in
Fig. 14 and rationales for each domain and overall confidence ratings
are available in HAWC (click here https://hawcprd.epa.gov/summar

22
K.A. Thayer et al. Environment International 169 (2022) 107468

Fig. 11. Example D: Survey of PFAS 150 SEM animal studies that met PECO criteria by study design, species, and health systems assessed. This is a screenshot image
of the interactive dashboard that is filterable by health system, study design, PFAS name, CASRN, and DTXSID. The numbers in the heat map inset indicate the
number of studies that investigated a health system within a study design. If a study evaluated multiple health outcomes or presented several experiments, it is shown
here multiple times. The study details panel includes information on animal model, exposure duration, route of administration, dose level(s) tested, and presentation
of NOEL and LOEL information with additional detail presented via website hover access.

Fig. 12. Survey exposure-response array of liver weight findings among the 3 most studied PFAS included in the SEM with animal toxicology evidence (tri­
fluoroacetic acid (CAS 76–05-1), 6:2 fluorotelomer alcohol (CAS 647–42-7), 6:2 fuorotelomer methacrylate (CAS 17527–29-6)). Click here https://hawcprd.epa.gov/
summary/data-pivot/assessment/100500085/Main-Report_Hepatic_Weight_Oral/ (U.S. EPA, 2020f) for interactive version. All symbols reflect author-reported,
statistically significant changes only.

23
K.A. Thayer et al. Environment International 169 (2022) 107468

Fig. 13. Example E. survey of vanadium animal studies that met PECO criteria by study design, species, and health systems assessed. This is a screenshot image of the
interactive dashboard that is filterable by health system, study design, and specific vanadium compound. The numbers in the heat map inset indicate the number of
studies that investigated a health system within a study design. If a study evaluated multiple health outcomes or presented several experiments, it is shown here
multiple times. The study details panel includes information on animal model, exposure duration, route of administration, dose level(s) tested, and presentation of
NOEL and LOEL information with additional detail presented via website hover access.

Fig. 14. Study evaluation results for animal studies in the PFAS 150 SEM. Figure is presented as a screenshot for reference. Click here https://hawcprd.epa.
gov/summary/visual/100500560/ for interactive version (U.S. EPA, 2020d).

y/visual/100500560/ (U.S. EPA, 2020d) to see interactive graphic). The
majority of the identified literature for PFAS were from ECHA sum­
maries; however, these summaries typically lack details on the methods
and results, resulting in ratings of low confidence or uninformative for
specific endpoints or overall. For the most part, studies that were not
ECHA summaries were considered well conducted (medium or high
overall confidence for the outcomes assessed (click here https://hawcpr
d.epa.gov/summary/visual/100500561/ to see interactive graphic) (U.
S. EPA, 2020g). The exception was Feng et al. (2015), in which the
Nafion was administered as Nafion membrane (which is comprised of
and releases fluoride) and the study’s internal fluoride measurements
indicated that the Nafion was not absorbed. Outcome-specific judge­
ments sometimes varied within a study usually based on the presenta­
tion of the results.
This figure shows the four risk-of-bias rating options. For each rating,
the associated colored square and symbol and a definition of the rating
are provided. Good (metric) or High confidence (overall) is indicated by
a dark green square with two plus signs. Adequate (metric) or Medium
confidence (overall) is indicated by a light green square with one plus
sign. Deficient (metric) or Low confidence (overall) is indicated by a
yellow square with one minus sign. A yellow square with “NR” for not
reported indicates there is insufficient information provided about the
metric. Critically deficient (metric) or Uninformative (overall) is indi­
cated by a dark red square with two minus signs. Color hatching with an
asterisk occurs where multiple judgements exist. A black solid triangle
inside a square is used to indicate a result that biases away from null.

24
K.A. Thayer et al. Environment International 169 (2022) 107468

3.2.3. Mechanistic evidence

[Note: Mechanistic evidence identified from database searches is tagged
as supplemental content for SEMs. The information may be inventoried at a
high level describing the biological focus of the reference. In some cases, other
resources are searched (see “Other Resources” in Literature Search and
Screening Strategies) to identify high throughput screening information or
gene expression information that might be identified from database searches.
Searching for high throughput screening or gene expression data is done
on a case-by-case basis and is largely dependent on the problem
formulation (i.e., decision context and acceptable level of confidence),
size of the evidence base, and the need to track NAM evidence.].
Example modified from 1-Naphthol SEM.

3.2.3.1. Results from database search. The mechanistic studies tagged as

supplemental material were categorized by the biological focus of the
mechanistic information. Importantly, this categorization was typically
done based on TIAB content only. Full-text retrieval for supplemental
material was not done for the SEM. Access to the underlying references
is provided through the interactive version of Fig. 15.

3.2.3.2. ToxCast and Tox21 high throughput screening data (optional).

ToxCast and Tox21 high throughput screening data are available for
1–naphthol in the EPA CompTox Chemicals Dashboard (Fig. 16; click
here https://comptox.epa.gov/dashboard/dsstoxdb/results?search
=DTXSID6021793#bioactivity to see in the Dashboard) (Williams

Fig. 15. 1-Naphthol studies identified as mechanistic during literature

screening. Click here https://hawcprd.epa.gov/lit/assessment/100500049/refe
rences/visualization/ to view interactive version.

Fig. 17. Top interacting genes for 1–naphthol from the CTD.

Fig. 16. ToxCast activity summary for 1–naphthol.

25
K.A. Thayer et al. Environment International 169 (2022) 107468

et al., 2021).
3.2.3.3. Comparative toxicogenomics database (optional). 1–Naphthol is
included in the Comparative Toxicogenomics Database (CTD). Below is
a summary of the top interacting genes based on analysis of 57 studies
presented in the CTD (Fig. 17; click here http://ctdbase.org/detail.go?
type=chem&acc=C029350 to see in the CTD). Note, these 57 studies
were reviewed to identify any that were not otherwise retrieved from
other sources (Appendix C).
3.2.3.4. Gene expression omnibus, and array express (optional). No in
vivo transcriptomic references were identified for 1-naphthol from
Array Express (https://www.ebi.ac.uk/arrayexpress/search.html?que
ry = 90–15-3). Two references were identified from Gene Expression
Omnibus (ID200092754 or 200060343), although neither was an in
vivo study. One of these studies was not identified from other sources
and was imported into DistillerSR for screening (Appendix D).
4. Conclusions
The conclusion section of an SEM should include observations on the
extent and nature of the evidence and identification of key data gaps.
When study evaluation is conducted, the conclusion section can also
indicate strengths and weaknesses in the evidence base. Examples of
SEM conclusion text can be found in Yost et al. (2021); Keshava et al.
(2020); Yost et al. (2019); Carlson et al. (2022); Radke et al. (In Press)
and Walker et al. (2018).
curation, Writing – review & editing. Glenn Rice: . Brittany Schulz:
Data curation, Writing – review & editing. Rachel M. Shaffer: Writing –
original draft. Teresa Shannon: . Andrew Shapiro: Software. Shane
Thacker: Software. Suryanarayana V. Vulimiri: Methodology,
Writing – review & editing. Antony J. Williams: Methodology, Writing
– review & editing. George Woodall: Methodology, Writing – review &
editing. Erin Yost: Conceptualization, Methodology, Writing – original
draft, Visualization. Robyn Blain: Methodology, Writing – review &
editing. Katherine Duke: Methodology, Writing – review & editing.
Alexandra E. Goldstone: Methodology, Writing – review & editing.
Pam Hartman: Methodology, Writing – review & editing. Kevin Hob­
bie: Methodology, Writing – review & editing. Brandall Ingle: Meth­
odology, Writing – review & editing. Courtney Lemeris: Methodology,
Writing – review & editing. Cynthia Lin: Methodology, Writing – review
& editing. Alex Lindahl: Methodology, Writing – review & editing.
Kristen McKinley: . Parnian Soleymani: Methodology, Writing – re­
view & editing. Nicole Vetter: Methodology, Writing – review &
editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Acknowledgements

CRediT authorship contribution statement
Kristina A. Thayer: Conceptualization, Methodology, Writing –
original draft. Michelle Angrish: Conceptualization, Methodology,
Writing – review & editing. Xabier Arzuaga: Conceptualization,
Methodology, Writing – review & editing. Laura M. Carlson: Concep­
tualization, Methodology, Writing – original draft, Visualization. Allen
Davis: Methodology. Laura Dishaw: Methodology. Ingrid Druwe:
Methodology. Catherine Gibbons: Methodology. Barbara Glenn:
Methodology. Ryan Jones: Software. J. Phillip Kaiser: Methodology.
Channa Keshava: Methodology. Nagalakshmi Keshava: Methodol­
ogy. Andrew Kraft: Conceptualization, Methodology, Writing – review
& editing. Lucina Lizarraga: Methodology. Amanda Persad: Method­
ology. Elizabeth G. Radke: Conceptualization, Methodology, Data
The authors would like to thank the following individuals: Vicki Soto
and Dahnish Shams for document production support; and Barbara
Soares, Casey Lindberg, Brittany Jacobs and Stan Barone for internal
review. The authors would also like to thank staff at ICF for assistance in
helping to evaluate and refine the workflows presented in the SEM.
Funding and Disclaimer: Authors declare no financial conflict of
interest. Funding for U.S. Environmental Protection Agency (EPA) au­
thors was provided through employment with no additional external
funding. Funding for ICF International authors was provided under EPA
contract 68HERC19D0003. The views expressed are those of the authors
and do not necessarily represent the views or policies of the U.S. EPA.
Any mention of trade names, products, or services does not imply an
endorsement by the U.S. Government or the U.S. EPA. The EPA does not
endorse any commercial products, services, or enterprises.

Appendix A. Survey of existing toxicity values

Table A1 lists websites which are searched for relevant human health reference values, along with indications of the results of the search. In
addition to these sources, the Toxicity Values (ToxVal) database on the EPA CompTox Chemicals Dashboard (https://comptox.epa.gov/dashboard/ch
emical_lists/TOXVAL_V5) is searched for both reference values and points of departure (POD)s as described in Appendix C. The ToxVal database is a
compilation of publicly available in vivo toxicity data, including toxicity values presented in a variety of assessments (Williams et al., 2021). [Note: Fill
in Search results column to indicate where results can be found (e.g., See Table 2); if no values were found, write “No values found”. Also note values without
derivation details should be summarized in a separate table from those values that have derivation details available.]
Background
Health-based reference values for noncancer effects are presented either in units of concentration (e.g., mg/L or mg/m3) or in terms of dose (e.g.,
mg/kg-day). Reference values generally are derived by applying uncertainty and adjustment factors to the POD. The POD is either an observed (e.g.,
NOAEL) or estimated (e.g., benchmark lower bound exposure/dose level that elicits an adverse effect observed in studies with human subjects or in
controlled animal experiments (U.S. EPA, 2002). The derivation methods and factors used in moving from a POD to a final reference value varies
according to the organization developing the values, often with consideration of how the resulting values will be applied. Most organizations develop a
human equivalent dose (HED) or concentration (HEC) in this process to account for dosimetric differences between humans and test species. Oral
reference values often are used as the basis for deriving standards for drinking water or acceptable levels in food.
Risk estimates are most often developed for cancer effects where the default assumption is that there is no level of exposure without some effect (i.
e., non-threshold effects) and estimates of risk related to exposure is part of the formulation (U.S. EPA, 2005). As noted previously, noncancer risks
have traditionally been managed by defining a level of exposure (i.e., reference value) below which there is a presumption of minimal health risk but
with no ability to estimate risk relative to any other exposure level. Approaches based on probability distributions to estimate distribution for
noncancer effect levels have also been developed more recently (IOMC ED, 2017; Chiu and Slob, 2015).

26
K.A. Thayer et al. Environment International 169 (2022) 107468

Assessment-based PODs or HECs/HEDs, along with uncertainty/adjustment factors, can also be used in a margin of exposure (MOE) approach to
determine exposure or consumption levels associated with acceptable level of risk. For example, risk evaluations performed under the Toxic Sub­
stances Control Act (TSCA, (Frank R. Lautenberg Chemical Safety for the 21st Century Act, 2016) apply an MOE approach as part of the risk decision
process on specific conditions of use for a subject chemical.

Table A1
Sources searched for existing human health reference values.
Source Search Results Query and/or link

ACGIH e.g., See Table 2. ACGIH. 2007. 2007 TLVs and BEIs: Based on documentation of the threshold limit values for chemical substances and physical agents
and biological exposure indices. Cincinnati, OH: American Conference of Governmental Industrial Hygienists.
AIHA e.g. No results found. AIHA. 2019. 2019 ERPG/WEEL Handbook. Fairfax, VA: American Industrial Hygiene Association. [Latest list of values.]
AIHA. 2002 (and updates). 2002 Emergency Response Planning Guidelines. Fairfax, VA: American Industrial Hygiene Association.
[Details used in deriving values.]
ATSDR e.g., Intermediate https://www.atsdr.cdc.gov/toxprofiles/index.asp
Inhalation MRL of 3 https://wwwn.cdc.gov/TSP/MRLS/mrlsListing.aspx
mg/m3
EPA CompTox https://comptox.epa.gov/dashboard
Chemicals
Dashboard
CT DEEP https://eregulations.ct.gov/eRegsPortal/Browse/getDocument?guid=%7b00D6A654-0300-CC47-9B95-397D2AD21304%7d
DFG https://onlinelibrary.wiley.com/doi/https://doi.org/10.1002/9783527826889.oth
OW (US EPA) https://www.epa.gov/sdwa/drinking-water-contaminant-human-health-effects-information
EPA/NRC AEGL https://www.epa.gov/aegl/access-acute-exposure-guideline-levels-aegls-values#chemicals
European https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32017L0164&from = EN
Commission
Health Canada https://www.canada.ca/en/services/health/publications/healthy-living.html
https://publications.gc.ca/site/archivee-archived.html?url=https://publications.gc.
ca/collections/collection_2012/sc-hc/H128-1–11-638-eng.pdf
https://publications.gc.ca/site/archivee-archived.html?url=https://publications.gc.ca/collections/Collection/H46-2–96-194E.pdf
HEAST https://epa-heast.ornl.gov/heast.php
https://cfpub.epa.gov/ncea/risk/hhra/recordisplay.cfm?deid= 2877
HSA https://www.hsa.ie/eng/publications_and_forms/publications/codes_of_practice/chemical_agents_cop_2020.pdf
IDEM https://www.in.gov/idem/toxic/2343.htm
ID DEQ https://adminrules.idaho.gov/rules/current/58/580101.pdf
IFA https://limitvalue.ifa.dguv.de/WebForm_gw2.aspx
IRIS https://www.epa.gov/iris/
ITER https://iter.tera.org/database.htm
JSOH https://www.sanei.or.jp/?mode= view&cid = 328
MassDEP https://www.mass.gov/service-details/massdep-ambient-air-toxics-guidelines
MDH https://www.health.state.mn.us/communities/environment/risk/guidance/air/table.html
https://www.health.state.mn.us/communities/environment/risk/guidance/gw/table.html
MI EGLE https://www.michigan.gov/documents/deq/deq-rrd-chem-CleanupCriteriaTSD_527410_7.pdf
NATICH https://nepis.epa.gov/Exe/ZyPDF.cgi/2000NS7S.PDF?Dockey= 2000NS7S.PDF
NC DEQ https://files.nc.gov/ncdeq/Air%20Quality/rules/rules/D1104.pdf
NDEP https://ndep.nv.gov/resources/risk-assessment-and-toxicology-basic-comparison-levels
NIOSH https://www.cdc.gov/niosh/npg/npgdcas.html
https://www.cdc.gov/niosh/pubs/criteria_date_desc_nopubnumbers.html
https://www.cdc.gov/niosh/idlh/intridl4.html
NJ DEP https://www.state.nj.us/dep/aqpp/downloads/risk/ToxAll2020.pdf
NY DEC https://www.dec.ny.gov/docs/remediation_hudson_pdf/techsuppdoc.pdf
OAQPS (US EPA) https://www.epa.gov/fera/dose response-assessment-assessing-health-risks-associated-exposure-hazardous-air-pollutants
OEHHA https://www.oehha.ca.gov/tcdb/index.asp
Ontario MOL https://www.labour.gov.on.ca/english/hs/pubs/oel_table.php
OPP (US EPA) https://iaspub.epa.gov/apex/pesticides/f?p= chemicalsearch:1
OR DEQ https://www.oregon.gov/deq/FilterDocs/airtox-abc.pdf
OSHA https://www.osha.gov/chemicaldata/
PAC Database https://edms.energy.gov/pac/Search
PPRTV https://www.epa.gov/pprtv/provisional-peer-reviewed-toxicity-values-pprtvs-assessments
Publications https://legisquebec.gouv.qc.ca/en/showdoc/cr/S-2.1,%20r.%2013?csi_scan_9222d36c6a354dc6=BO9xyrMZ +
Quebec 270UP3j0MGuOD0kZjgFAAAAXrM3HA==&bcsi_scan_filename = S-2.1,%20r.%2013&bcsi_scan_9222d36c6a354dc6 =
KXzmpPueuN0L1AjnJOB1Zerr85YMAAAAyhrPTg==&bcsi_scan_filename = S-2.1,%20r.%2013
RI DEM https://www.dem.ri.gov/programs/benviron/air/pdf/airtoxgl.pdf
RIVM https://www.rivm.nl/bibliotheek/rapporten/711701092.pdf
https://www.rivm.nl/bibliotheek/rapporten/609021044.pdf
https://www.rivm.nl/bibliotheek/rapporten/711701025.pdf
Safe Work https://www.safeworkaustralia.gov.au/exposure-standards#exposure-standards-in-australia
Australia
SWCAA https://www.swcleanair.org
TCEQ https://www.tceq.texas.gov/toxicology/dsd/final
https://www.tceq.texas.gov/remediation/trrp/trrppcls.html
USAPHC https://phc.amedd.army.mil/topics/envirohealth/hrasm/Pages/TG230.aspx
VT DEC https://dec.vermont.gov/sites/dec/files/aqc/laws-regs/documents/AQCD%20Regulations%20ADOPTED_Dec132018.
pdf#page=127
WAC https://apps.leg.wa.gov/WAC/default.aspx?cite=173–460-150
Worksafe https://worksafe.govt.nz/topic-and-industry/work-related-health/monitoring/exposure-standards-and-biological-exposure-indices/
WHO https://www.who.int/ipcs/publications/ehc/en/
https://www.who.int/water_sanitation_health/publications/drinking-water-quality-guidelines-4-including-1st-addendum/en/

ACGIH = American Conference of Governmental Industrial Hygienists; AEGL = Acute Exposure Guideline Levels; ATSDR = Agency for Toxic Substances and Disease
Registry; BEI = biological exposure index; CT DEEP = Connecticut Department of Energy & Environmental Protection; DFG = Deutsche Forschungsgemeinschaft, German
27
K.A. Thayer et al. Environment International 169 (2022) 107468

Research Foundation; EPA = Environmental Protection Agency; HEAST = Health Effects Assessment Summary Tables; HSA = Health and Safety Authority; IDEM = Indiana
Department of Environmental Management; ID DEQ = Idaho Department of Environmental Quality; IFA = Institut für Arbeitsschutz, The Institute for Occupational Safety
and Health; IRIS = Integrated Risk Information System; ITER = International Toxicity Estimates for Risk Assessment; JSOH = Japan Society for Occupational Health;
MassDEP = Massachusetts Department of Environmental Protection; MDH = Minnesota Department of Health; MI EGLE = Michigan Environment, Great Lakes & Energy;
MOL = Ministry of Labour; NATICH = National Air Toxics Information Clearinghouse; NC DEQ = North Carolina Department of Environmental Quality; NDEP = Nevada
Division of Environmental Protection; NIOSH = National Institute for Occupational Safety and Health; NJ DEP = New Jersey Department of Environmental Protection; NRC
= National Research Council; NY DEC = New York State Department of Environmental Conservation; OAQPS = Office of Air Quality Planning and Standards; OEHHA =
California Office of Environmental Health Hazard Assessment; OPP = Office of Pesticide Programs; OR DEQ = Oregon Department of Environmental Quality; OSHA =
Occupational Safety and Health Administration; OW = Office of Water; PAC = Protective Action Criteria; PPRTV = Provisional Peer-Reviewed Toxicity Value; RI DEM =
Rhode Island Department of Environmental Management; RIVM = Rijksinstituut voor Volksgezondheid en Milieu, The Netherlands Institute for Public Health and the Envi­
ronment; SWCAA = Southwest Clean Air Association; TCEQ = Texas Commission on Environmental Quality; TLV = threshold limit value; USAPHC = United States Army
Public Health Center; VT DEC = Vermont Department of Environmental Conservation; WAC = Washington Administrative Code; WHO = World Health Organization.

Appendix B. Literature search strategies

See Table B1.

Table B1
Example presentation for results of literature search strategy.
Database Termsa Search Date and
Results

Web of Science (TS=“chloroform” OR TS=“1,1,1-trichloromethane” OR TS=“chloroforme” OR TS=“trichloromethane”) AND PY=(2000–2018) 12/3/2018

NOT (SU=“physics” OR SU=“plant sciences” OR SU=“energy fuels” OR SU=“instruments instrumentation” OR SU=“computer Number of hits: 2,344
science” OR SU=“legal medicine” OR SU=“metallurgy metallurgical engineering” OR SU=“mechanics” OR SU=“education
educational research” OR SU=“acoustics” OR SU=“geochemistry geophysics” OR SU=“mathematics” OR SU=“forestry” OR 5/31/2019
SU=“automation control systems” OR SU=“mining mineral processing” OR SU=“construction building technology” OR
SU=“astronomy astrophysics” OR SU=“archaeology” OR SU=“operations research management science” OR SU=“anthropology” Number of hits: 30
OR SU=“sport sciences” OR SU=“art” OR SU=“paleontology” OR SU=“telecommunications” OR SU=“chemistry” OR SU=“polymer
science” OR SU=“engineering” OR SU=“environmental sciences ecology” OR SU=“food science technology” OR SU=“science 7/10/2019
technology other topics” OR SU=“biotechnology applied microbiology” OR SU=“agriculture” OR SU=“spectroscopy” OR Number of hits: 7
SU=“crystallography” OR SU=“integrative complementary medicine” OR SU=“water resources” OR SU=“nutrition dietetics” OR
SU=“life sciences biomedicine other topics” OR SU=“parasitology” OR SU=“thermodynamics” OR SU=“optics” OR 4/30/2020
SU=“biophysics” OR SU=“tropical medicine” OR SU=“veterinary sciences” OR SU=“research experimental medicine” OR
SU=“marine freshwater biology” OR SU=“meteorology atmospheric sciences” OR SU=“geology” OR SU=“electrochemistry” OR Number of hits: 65
SU=“general internal medicine” OR SU=“dentistry oral surgery medicine” OR SU=“entomology” OR SU=“nuclear science
technology” OR SU=“infectious diseases” OR SU=“fisheries” OR SU=“oceanography” OR SU=“anesthesiology” OR SU=“zoology” 2/25/2021
OR SU=“virology” OR SU=“radiology nuclear medicine medical imaging” OR SU=“medical laboratory technology” OR Number of hits: 99
SU=“mycology” OR SU=“surgery” OR SU=“biodiversity conservation” OR SU=“obstetrics gynecology” OR SU=“evolutionary
biology” OR SU=“psychiatry” OR SU=“remote sensing” OR SU=“pediatrics” OR SU=“mineralogy” OR SU=“transplantation” OR Total hits: 2,545
SU=“microscopy” OR SU=“rheumatology” OR SU=“geriatrics gerontology” OR SU=“orthopedics” OR SU=“materials science”)

PubMed ((((“chloroform”[MeSH Terms] OR “1,1,1-trichloromethane”[All Fields]) OR “chloroforme”[All Fields]) OR “trichloromethane”[All 12/3/2018

Fields]) OR “67-66-3′′ [EC/RN Number]) AND (”2000/01/01′′ [PDAT]: “3000′′ [PDAT])
Number of hits: 2,527

5/31/2019
Number of hits: 68

7/10/2019
Number of hits: 4

4/30/2020
Number of hits: 56

2/25/2021
Number of hits: 57

Total hits: 2,712

ToxNetb @AND+@OR+(chloroform + chloroforme + trichloromethane+@TERM+@rn + 67 + 66 + 3)+@RANGE + yr + 2000 + 12/3/2018
2018+@NOT+@org + pubmed + pubdart+“nih + reporter”
Number of hits: 365

5/31/2019
Number of hits: 0

7/10/2019
Number of hits: 0

Total hits: 365

Merged Total after duplicate removal in HERO 5,003
Reference Set

Initial search and update history: 12/3/2018 (4,620 unique records identified); 5/31/2019 (98 unique records identified); 7/10/2019 (11 unique records identified);
4/30/2020 (119 unique records identified); 2/25/2021 (155 unique records identified); total of 5,003.
a
Publication year terms were updated for each search date. Other elements of the search strings did not change.
b
As of December 2019, the ToxNet database has shut down and no longer exists as a searchable information venue and its content was moved to other National
Library of Medicine products.
28
K.A. Thayer et al. Environment International 169 (2022) 107468

Appendix C. Process for searching and collecting evidence from selected other resources

An additional resource used to facilitate consistency in searching selected other sources and retrieving results is available in the HAWC project
“SEM Template Figures and Resources”, Gray Literature Training slides.
Review of reference lists from existing assessments (final or publicly available draft), journal reviews articles and studies considered
relevant to PECO based on full-text screening.
Review of the citation reference lists is typically done manually because they are not available in a file format (e.g., RIS) that permits uploading into
screening software applications. Manual review entails scanning the title, study summary, or study details as presented in the resource for those that

Table C1
Summary table of search results for other sources (1-naphthol example).
Sourcea Source address Search terms Search Total References not
date unique otherwise
number of identified that
results were screened
retrieved in DistillerSR

Review of reference lists NA NA NA 67 65

studies considered
relevant to PECO based
on full-text screening.
Review of reference lists NA NA NA 3 0
from existing
assessments (final or
publicly available draft)
or journal review
articles that focused on
human health
EPA CompTox (Chemicals https://comptox.epa.gov/dashboard/dsstoxdb/results? 90–15-3 (Results from Human 12/10/ 21 5
Dashboard (ToxVal) abbreviation= TOXVAL_V5&search = Health: POD, Toxicity Value, 2019
DTXSID6021793#toxicity-values Lethality Effect Level)
EPA CompTox Chemicals https://comptox.epa.gov/dashboard/dsstoxdb/results? 90–15-3 9/19/ 1 1
Dashboard (ToxCast or search= DTXSID6021793 2019
Tox21 high throughput
screening information)
ECHA https://echa.europa. 90–15-3 10/8/ 53 30
eu/information-on-chemicals/registered-substances 2019
EPA ChemView https://chemview.epa.gov/chemview?tf= 0&ch = 90–15- 90–15-3 9/19/ 3 1
3&su = 2–5-6–7-37574985&as = 3–10-9–8&ac = 1–15- 2019
16–6378999&ma = 4–11-1981377&tds = 0&tdl = 10&tas1 =
1&tas2 = asc&tas3 = undefined&tss=
High Production Volume https://ofmpub.epa.gov/oppthpv/quicksearch.display? 90–15-3 9/19/ 4 4
Challenge Database pChem= 101850 2019
(HPVIS)
NTP CEBS https://manticore.niehs.nih.gov/cebssearch/search?q= 90–15-3 9/19/ 0 0
90–15-3 2019
OECD eChemPortal https://www.echemportal.org/echemportal/substance-search 90–15-3 9/19/ 0 0
2019
ECOTOX Database https://cfpub.epa.gov/ecotox/search.cfm 90–15-3 9/19/ 4 3
2019
Comparative https://ctdbase.org/ 90–15-3 12/9/ 57 30
Toxicogenomics 2019
Database (CTD)
ArrayExpress https://www.ebi.ac.uk/arrayexpress/ 90–15-3 & “naphthol” 12/9/ 1 1
2019
Gene Expression Omnibus https://www.ncbi.nlm.nih.gov/geo/ (90–15-3[rn] OR “1- 12/9/ 2 1
Naphthol”[tw] OR “Naphthalen- 2019
1-ol”[tw] OR “1-
Naphthalenol”[tw] OR “1-
naphthalenol”[tw]) AND
(“Expression profiling by RT-
PCR”[Filter] OR “Expression
profiling by MPSS”[Filter] OR
“Expression profiling by
SAGE”[Filter] OR “Expression
profiling by SNP array”[Filter]
OR “Expression profiling by
array”[Filter] OR “Expression
profiling by genome tiling
array”[Filter] OR “Expression
profiling by high throughput
sequencing”[Filter] OR “Protein
profiling by Mass Spec”[Filter]
OR “Protein profiling by protein
array”[Filter]).
a
PECO = Populations, Exposures, Comparators, and Outcomes; NA = not applicable; POD = point of departure; ECHA = European Chemicals Agency; NTP CEBS =
National Toxicology Program Chemical Effects in Biological Systems; OECD = Organisation for Economic Co-operation and Development.

29
K.A. Thayer et al. Environment International 169 (2022) 107468

appear to meet the PECO criteria. Any references that are not identified from the other sources are formatted in a RIS file format, imported into
DistillerSR, annotated with respect to source, and screened as outlined in the methods section. For tracking assessments or reviews, the name of the
source citation and the number of references imported into DistillerSR are noted. The reference list of any study included in the literature inventory are
reviewed manually to identify titles that appeared relevant to the PECO criteria. These citations are tracked in a spreadsheet, compared against the
literature base to determine if they are unique to the project, and then added to DistillerSR to be screened at the title and abstract stage for PECO
relevance.
EPA CompTox Chemicals Dashboard (ToxVal)
ToxVal is searched in the EPA CompTox Chemicals Dashboard (https://comptox.epa.gov/dashboard), and data available from the Hazard tab is
exported (Williams et al., 2021). Using the human health POD summary file, citations are identified that apply to human health PODs. A citation for
each referenced study is generated in HERO and verified that it is not already identified from the database search (or searches of “other sources
consulted”) prior to moving forward to screening in DistillerSR. Full texts are retrieved where possible; if full texts are not available, data from the
ToxVal dashboard are entered and the citation annotated accordingly for Tableau and HAWC visualizations by adding “(ToxVal)” to the citation.
European Chemicals Agency (ECHA)
A search of the ECHA registered substances database (https://echa.europa.eu/information-on-chemicals/registered-substances) is conducted
using the CASRN. The registration dossier associated with the CASRN is retrieved by navigating to and clicking the eye-shaped view icon displayed in
the chemical summary panel. The general information page and all subpages included under the Toxicological Information tab are downloaded in PDF
(Portable Document Format), including all nested reports having unique URLs. Any pages indicated as a toxicological or endpoint summary are
omitted. In addition, data is extracted from each dossier page and used to populate an Excel tracking sheet. An example of a tracking sheet is available
in the Gray Literature Training slides in the HAWC project “SEM Template Figures and Resources”. Extracted fields include data from the general
information page regarding the registration type and publication dates, and on a typical study summary page the primary fields reported in the
administrative data, data source, methods, and results sections. Each study summary results in more than one row in the tracking sheet if more than
one data source or effect level is reported.
At this stage, each study summary is reviewed for inclusion based on PECO criteria. Study summaries labeled “read across” (if any) are screened in
DistillerSR and considered supplemental material. When a study summary considers relevant reported data from a study or lab report, a citation for the
full study is generated in HERO and verified that it is not already identified from the database search (or searches of “other sources consulted”) prior to
moving forward to screening. When citation information is not available and a full text cannot be retrieved, the generated PDF is used as the full text
for screening and extraction and the citation annotated accordingly for Tableau and HAWC visualizations by adding “(ECHA Summary)” to the
citation.
EPA ChemView
The EPA ChemView database (https://chemview.epa.gov/chemview) (U.S. EPA, 2019a) is searched using the chemical CASRN. The prepopulated
CASRN match and the “Information Submitted to EPA” output option filter are selected before generating results. If results are available, the icon
under the “Data Submitted to EPA” column is selected, and the following entries are included:

• High Production Volume Challenge Database (HPVIS).

• Human Health studies (Substantial Risk Reports).
• Monitoring (Includes environmental, occupational, and general entries).
• TSCA Section 4 (Chemical testing results).
• TSCA Section 8(d) (Health and safety studies).
• TSCA Section 8(e) (Substantial risk).
• FYI (Voluntary documents).

All references for ecotoxicology and physical and chemical property entries are excluded. When results are available, extractors navigate into each
record until a substantial risk report link is identified and saved as a PDF file. If the report cannot be saved, due to file corruption or broken links, the
record is not retrieved and is identified as “unable to obtain record.” Most substantial risk reports contain multiple document IDs, so citations are
derived by concatenating the unique report numbers (OTS; 8EHD Num; DCN; TSCATS RefID; and CIS) associated with each document along with the
typical author organization, year, and title. Once a citation is generated, the study moves forward to DistillerSR where it is screened according to PECO
and supplemental material criteria.
National toxicology Program (NTP) chemical effects in biological systems
This database is searched using the chemical CASRN (https://manticore.niehs.nih.gov/cebssearch). All non-NTP data are excluded using the “NTP
Data Only” filter. Data tables for reports undergoing peer review are also searched for studies that have not been finalized (https://ntp.niehs.nih.
gov/data/tables/index.html) based on a manual review of chemical names.
OECD eChemPortal
The OECD eChemPortal (https://www.echemportal.org/echemportal/substance-search) is searched using the chemical CASRN. Only database
entries from the following sources are included and entries from all other databases are excluded in the search. Final assessment reports and other
relevant SIDS reports embedded in the links are captured and saved as PDF files.

• OECD HPV.
• OECD SIDS International Uniform Chemical Information Database (IUCLID).

ECOTOX database
EPA’s ECOTOX Knowledgebase (https://cfpub.epa.gov/ecotox/search.cfm) is searched using the CASRN. Results are refined to terrestrial
mammalian studies by selecting the terrestrial tab at the top of the search page and sorting the results by species group.
EPA CompTox chemical dashboard search to retrieve ToxCast or Tox21 high throughput screening summary information
Version 3.0.9 of the CompTox Chemicals Dashboard (https://comptox.epa.gov/dashboard) is accessed for high throughput screening (HTS) data
by searching the Dashboard by CASRN (Williams et al., 2021). Next, the “Bioactivity” section is selected and the availability of ToxCast/Tox21 HTS
data for active and inactive assays is examined in the “TOXCAST: Summary” tab. If active assays are reported, the figure is copied for presentation in

30
K.A. Thayer et al. Environment International 169 (2022) 107468

the systematic evidence map. This figure presents (i) scatterplot of scaled assay responses vs. AC50 values for each active assay endpoint, and (ii)
cytotoxicity limit as a vertical line. More detailed information on the results of ToxCast and Tox21 assays are available in the CompTox Chemicals
Dashboard section “ToxCast/Tox21,” which includes chemical analysis data, dose response data and model fits, and “flags” assigned by an automated
analysis, which might suggest false positivity/negativity or indicate other anomalies in the data. This information is not summarized further for the
purposes of the systematic evidence map, which is focused on identifying the extent of available evidence.
Comparative toxicogenomics database
This CTD database (https://ctdbase.org/) is searched using the chemical CASRN in the “keyword search” with pulldown menu set to “Chemicals.”
The query results in the “Gene Interactions” tab provide the list genes or proteins reported in the published references as interacting with the query
chemical. Human and rodent genes/proteins interacting with a query chemical are identified and their numbers are provided in the systematic ev­
idence map. If information is available, a figure presenting the interacting genes available in the “Basics” tab is copied to the systematic evidence map.
When many interacting genes are identified it may be appropriate to limit to the top 10 or 20 to represent genes whose interactions with query
chemical are supported by most available references. Details on interaction types and degrees are provided at https://ctdbase.org/help/ixnQueryHe
lp.jsp. The top scoring pathway relevant to the identified interacting genes is provided in the systematic evidence map. The reference list of studies
reporting gene/protein interactions with the query chemical are compared to existing references in DistillerSR. Unique references are added to
DistillerSR and screened according to PECO and supplemental material criteria.
Gene expression omnibus, and ArrayExpress
Public repositories of omics data Gene Expression Omnibus (GEO; https://www.ncbi.nlm.nih.gov/geo/) and ArrayExpress (https://www.ebi.ac.
uk/arrayexpress/) are queried to identify available gene expression datasets relevant to the exposure by the chemical of interest. The GEO is queried
using the custom search string shown below, in which the letters “XX” are replaced by the chemical name as represented in the PubMed search strategy
from Appendix A. Filters are applied to select human, mouse and rat datasets. Note, the filter options might not be available in the GEO website if no
human, mouse, or rat datasets exist. Retrieved studies are reviewed and the studies that report gene expression data for chemical exposures of interest
are reported in the systematic evidence map as follows: Series accession number (GSExxxxx); title of study, species, route of exposure, platform and
tissue type.
GEO Search string
(“XX”[MeSH Terms] OR XX[All Fields]) AND (“Expression profiling by RT-PCR”[Filter] OR “Expression profiling by MPSS”[Filter] OR “Expression
profiling by SAGE”[Filter] OR “Expression profiling by SNP array”[Filter] OR “Expression profiling by array”[Filter] OR “Expression profiling by
genome tiling array”[Filter] OR “Expression profiling by high throughput sequencing”[Filter] OR “Protein profiling by Mass Spec”[Filter] OR
“Protein profiling by protein array”[Filter])
The ArrayExpress (https://www.ebi.ac.uk/arrayexpress/) repository is queried using chemical name as a keyword and filtered to limit datasets to
“Homo sapiens,” “Rattus norvegicus” and “Mus musculus” species. All studies reporting RNA-seq, transcription profiling, proteomic profiling or
translation profiling data are reported in the systematic evidence map.

Appendix D. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.envint.2022.107468.

References Feng, M., Qu, R., Habteselassie, M., Wu, J., Yang, S., Sun, P., Huang, Q., Wang, Z., 2015.
Hepatic transcriptome responses in mice (Mus musculus) exposed to the nafion
membrane and its combustion products. PLoS ONE 10, e0128591. https://doi.org/
Afkhami-Arekani, M., Karimi, M., Mohammadi, S.M., Nourani, F., 2008. Effect of sodium
10.1371/journal.pone.0128591.
metavanadate supplementation on lipid and glucose metabolism biomarkers in type
Frank R. Lautenberg Chemical Safety for the 21st Century Act, ,. Public Law 114-182 -
E diabetic patients. Malaysian J. Nutr. 14, 113–119.
Frank R. Lautenberg Chemical Safety for the 21st Century Act, (2016). https://www.
Bannach-Brown, A, ., P.,rzybyła, P.,., T.,homas, J.,., R.,ice, A.,.S.C., Ananiadou. S., L.,iao,
epa.gov/assessing-and-managing-chemicals-under-tsca/frank-r-lautenberg-
J.,., M.,acleod, M.,.R., 2018. The use of text-mining and machine learning algorithms
chemical-safety-21st-century-act.
in systematic reviews: reducing workload in preclinical biomedical sciences and
Goldfine, A.B., Patti, M.E., Zuberi, L., Goldstein, B.J., Leblanc, R., Landaker, E.J.,
reducing human screening error. Cold Spring Harbor Protocols 0: 1-26. http://dx.
Jiang, Z.Y., Willsky, G.R., Kahn, C.R., 2000. Metabolic effects of vanadyl sulfate in
doi.org/10.1101/255760.
humans with non-insulin-dependent diabetes mellitus: in vivo and in vitro studies.
Boden, G., Chen, X., Ruiz, J., 1996. Effects of vanadyl sulphate on carbohydrate and lipid
Metabolism 49, 400–410. https://doi.org/10.1016/S0026-0495(00)90418-9.
metabolism in patients with non-insulin dependent diabetes mellitus. Metabolism
Goldfine, A.B., Simonson, D.C., Folli, F., Patti, M.E., Kahn, C.R., 1995. Metabolic effects
45, 1130–1135.
of sodium metavanadate in humans with insulin-dependent and noninsulin-
Chiu, W.A., Slob, W., 2015. A unified probabilistic framework for dose-response
dependent diabetes mellitus in vivo and in vitro studies. J. Clin. Endocrinol. Metab.
assessment of human health effects [Review]. Environ. Health Perspect. 123,
80, 3311–3320. https://doi.org/10.1210/jcem.80.11.7593444.
1241–1254. https://doi.org/10.1289/ehp.1409385.
Halberstam, M., Cohen, N., Shlimovich, P., Rossetti, L., Shamoon, H., 1996. Oral vanadyl
Cohen, A.M., Hersh, W.R., Peterson, K., Yen, P.Y., 2006. Reducing workload in
sulfate improves insulin sensitivity in NIDDM but not in obese nondiabetic subjects.
systematic review preparation using automated citation classification. J. Am. Med.
Diabetes 45, 659–666. https://doi.org/10.2337/diab.45.5.659.
Inform. Assoc. 13, 206–219. https://doi.org/10.1197/jamia.M1929.
Howard, B.E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M.R., Holmgren, S.,
Cohen, N., Halberstam, M., Shlimovich, P., Chang, C.J., Shamoon, H., Rossetti, L., 1995.
Pelch, K.E., Walker, V., Rooney, A.A., Macleod, M., Shah, R.R., Thayer, K., 2016.
Oral vanadyl sulfate improves hepatic and peripheral insulin sensitivity in patients
SWIFT-Review: a text-mining workbench for systematic review. Syst. Rev. 5, 87.
with non-insulin-dependent diabetes mellitus. J. Clin. Invest. 95, 2501–2509.
https://doi.org/10.1186/s13643-016-0263-z.
https://doi.org/10.1172/JCI117951.
Howard, B.E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A.,
Cusi, K., Cukier, S., Defronzo, R.A., Torres, M., Puchulu, F.M., Redondo, J.C., 2001.
Thayer, K., Merrick, B.A., Walker, V., Rooney, A., Shah, R.R., 2020. SWIFT-Active
Vanadyl sulfate improves hepatic and muscle insulin sensitivity in type 2 diabetes.
Screener: Accelerated document screening through active learning and integrated
J. Clin. Endocrinol. Metab. 86, 1410–1417. https://doi.org/10.1210/
recall estimation. Environ. Int. 138, 105623 https://doi.org/10.1016/j.
jcem.86.3.7337.
envint.2020.105623.
Davis, A.P., Wiegers, T.C., Wiegers, J., Grondin, C.J., Johnson, R.J., Sciaky, D.,
IOMC ED (Inter-Organization Programme for the Sound Management of Chemicals
Mattingly, C.J., 2021. CTD anatomy: Analyzing chemical-induced phenotypes and
Environment Directorate). (2017). Guidance document on evaluating and expressing
exposures from an anatomical perspective, with implications for environmental
uncertainty in hazard characterization. Harmonization Project Document 11 – 2nd
health studies. Curr. Res. Toxicol. 2, 128–139. https://doi.org/10.1016/j.
edition (2nd ed.). Geneva, Switzerland: World Health Organization. http://www.
crtox.2021.03.001.
who.int/ipcs/methods/harmonization/areas/hazard_assessment/en/.
Fawcett, J.P., Farquhar, S.J., Thou, T., Shand, B.I., 1997. Oral vanadyl sulphate does not
Keshava, C., Davis, J.A., Stanek, J., Thayer, K.A., Galizia, A., Keshava, N., Gift, J.,
affect blood cells, viscosity or biochemistry in humans. Pharmacol. Toxicol. 80,
Vulimiri, S.V., Woodall, G., Gigot, C., Garcia, K., Greenhalgh, A., Schulz, B.,
202–206.
Volkoff, S., Camargo, K., Persad, A.S., 2020. Application of systematic evidence

31
K.A. Thayer et al.
mapping to assess the impact of new research when updating health reference
values: A case example using acrolein. Environ. Int. 143, 105956 https://doi.org/
10.1016/j.envint.2020.105956.
McLanahan, E.D., El-Masri, H.A., Sweeney, L.M., Kopylev, L.Y., Clewell, H.J.,
Wambaugh, J.F., Schlosser, P.M., 2012. Physiologically based pharmacokinetic
model use in risk assessment–Why being published is not enough. Toxicol. Sci. 126,
5–15. https://doi.org/10.1093/toxsci/kfr295.
NASEM (National Academies of Sciences, Engineering, and Medicine), 2021. Review of
U.S. EPA’s ORD staff handbook for developing IRIS assessments: 2020 version.
Washington, DC: National Academies Press. http://dx.doi.org/10.17226/26289.
NRC (National Research Council), 1983. Risk assessment in the federal government:
Managing the process. National Academy Press, Washington, DC http://dx.doi.org/
10.17226/366.
NRC (National Research Council), 2009. Science and decisions: Advancing risk
assessment. The National Academies Press, Washington, DC http://dx.doi.org/
10.17226/12209.
Radke, E.G., Braun, J.M., Meeker, J.D., Cooper, G.S., 2018. Phthalate exposure and male
reproductive outcomes: A systematic review of the human epidemiological evidence
[Review]. Environ. Int. 121, 764–793. https://doi.org/10.1016/j.
envint.2018.07.029.
Sun, L., Shi, D.J., Gao, X.C., Mi, S.Y., Yu, Y., Han, Q., 2014. The protective effect of
vanadium against diabetic cataracts in diabetic rat model. Biol. Trace Elem. Res.
158, 219–223. https://doi.org/10.1007/s12011-014-9925-7.
Susić, D., Kentera, D., 1986. Effect of chronic vanadate administration on pulmonary
circulation in the rat. Respiration 49, 68–72. https://doi.org/10.1159/000194861.
U.S. EPA (U.S. Environmental Protection Agency), 2002. A review of the reference dose
and reference concentration processes. (EPA/630/P-02/002F). Washington, DC: U.S.
Environmental Protection Agency, Risk Assessment Forum. https://www.epa.gov/
sites/production/files/2014-12/documents/rfd-final.pdf.
U.S. EPA (U.S. Environmental Protection Agency), 2005. Guidelines for carcinogen risk
assessment [EPA Report]. (EPA/630/P-03/001B). Washington, DC: U.S.
Environmental Protection Agency, Risk Assessment Forum. https://www.epa.gov/
sites/production/files/2013-09/documents/cancer_guidelines_final_3-25-05.pdf.
U.S. EPA (U.S. Environmental Protection Agency), 2009. Graphical arrays of chemical-
specific health effect reference values for inhalation exposures [EPA Report]. (EPA/
600/R-09/061). Research Triangle Park, NC. http://cfpub.epa.gov/ncea/cfm/
recordisplay.cfm?deid=211003.
U.S. EPA (U.S. Environmental Protection Agency), 2012. Benchmark dose technical
guidance. (EPA/100/R-12/001). Washington, DC: U.S. Environmental Protection
Agency, Risk Assessment Forum. https://www.epa.gov/risk/benchmark-dose-
technical-guidance.
U.S. EPA (U.S. Environmental Protection Agency), 2013. Mn and BTEX reference value
arrays (final reports). (EPA/600/R-12/047F). Washington, DC. https://cfpub.epa.
gov/ncea/risk/recordisplay.cfm?deid=250571.
U.S. EPA (U.S. Environmental Protection Agency), 2015. Peer review handbook [EPA
Report] (4th ed.). (EPA/100/B-15/001). Washington, DC: U.S. Environmental
Protection Agency, Science Policy Council. https://www.epa.gov/osa/peer-review-
handbook-4th-edition-2015.
U.S. EPA (U.S. Environmental Protection Agency), 2018. An umbrella Quality Assurance
Project Plan (QAPP) for PBPK models [EPA Report]. (ORD QAPP ID No: B-0030740-
QP-1-1). Research Triangle Park, NC.
U.S. EPA (U.S. Environmental Protection Agency), 2019a. ChemView [Database].
Retrieved from https://chemview.epa.gov/chemview.
U.S. EPA (U.S. Environmental Protection Agency), 2019b. IRIS assessment plan for
inorganic mercury salts (scoping and problem formulation materials). (EPA/635/R-
19/149). Washington, DC. https://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?
deid=346843.
U.S. EPA (U.S. Environmental Protection Agency), 2019c. Systematic review protocol for
the PFBA, PFHxA, PFHxS, PFNA, and PFDA IRIS assessments [EPA Report].
32
Environment International 169 (2022) 107468
(EPA635R19049). Integrated Risk Information System. Center for Public Health and
Environmental Assessment. Office of Research and Development. U.S.
Environmental Protection Agency. https://cfpub.epa.gov/ncea/iris_drafts/
recordisplay.cfm?deid=345065.
U.S. EPA (U.S. Environmental Protection Agency), 2020a. CompTox Chemicals
Dashboard. Washington, DC. Retrieved from https://comptox.epa.gov/dashboard.
U.S. EPA (U.S. Environmental Protection Agency), 2020b. IRIS assessment plan for oral
exposure to vanadium and compounds (scoping and problem formulation materials).
(EPA/635/R-20/112). Washington, DC. https://cfpub.epa.gov/ncea/iris_drafts/
recordisplay.cfm?deid=348792.
U.S. EPA (U.S. Environmental Protection Agency), 2020c. ORD staff handbook for
developing IRIS assessments (public comment draft) [EPA Report]. (EPA/600/R-20/
137). Washington, DC: U.S. Environmental Protection Agency, Office of Research
and Development, Center for Public Health and Environmental Assessment. https://
cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=350086.
U.S. EPA, 2020d. PFAS 150 (2020) | Visualizations | Animal Study Evaluation Heatmap.
https://hawcprd.epa.gov/summary/visual/100500560/.
U.S. EPA (U.S. Environmental Protection Agency), 2020e. PFAS 150 (2020) |
Visualizations | Available visualizations. https://hawcprd.epa.gov/summary/
assessment/100500085/visuals/.
U.S. EPA (U.S. Environmental Protection Agency), 2020f. PFAS 150 (2020) |
Visualizations | Main Report Hepatic Weight Oral. https://hawcprd.epa.gov/
summary/data-pivot/assessment/100500085/Main-Report_Hepatic_Weight_Oral/.
U.S. EPA (U.S. Environmental Protection Agency), 2020g. PFAS 150 (2020) |
Visualizations | nonECHA animal study evaluation. https://hawcprd.epa.gov/
summary/visual/100500561/.
U.S. EPA (U.S. Environmental Protection Agency), 2020h. Provisional peer-reviewed
toxicity values for glycidaldehyde [EPA Report]. (EPA/690/R-20/003F).
Washington, DC: U.S. Environmental Protection Agency, Center for Public Health
and Environmental Assessment. https://cfpub.epa.gov/ncea/pprtv/recordisplay.
cfm?deid=349744.
U.S. EPA (U.S. Environmental Protection Agency), 2020i. Toxicology testing in the 21st
century (Tox21). Available online at https://ntp.niehs.nih.gov/go/tox21 (accessed
May 4, 2021).
U.S. EPA (U.S. Environmental Protection Agency), 2021. Provisional peer reviewed
toxicity values for trans-Crotonaldehyde [EPA Report]. (EPA/690/R-21/001F).
Washington, DC: U.S. Environmental Protection Agency, Center for Public Health
and Environmental Assessment. https://cfpub.epa.gov/ncea/pprtv/recordisplay.
cfm?deid=351031.
Walker, V.R., Boyles, A.L., Pelch, K.E., Holmgren, S.D., Shapiro, A.J., Blystone, C.R.,
Devito, M.J., Newbold, R.R., Blain, R., Hartman, P., Thayer, K.A., Rooney, A.A.,
2018. Human and animal evidence of potential transgenerational inheritance of
health effects: An evidence map and state-of-the-science evaluation. Environ. Int.
115, 48–69. https://doi.org/10.1016/j.envint.2017.12.032.
Williams, A.J., Lambert, J.C., Thayer, K., Dorne, J., 2021. Sourcing data on chemical
properties and hazard data from the US-EPA CompTox Chemicals Dashboard: A
practical guide for human risk assessment [Review]. Environ. Int. 154, 106566
https://doi.org/10.1016/j.envint.2021.106566.
Woodall, GM; Lipscomb, JC; Schulz, B. (In Press) Visual comparisons of health-based
toxicity values: I. Overview.
Yost, E.E., Euling, S.Y., Weaver, J.A., Beverly, B.E.J., Keshava, N., Mudipalli, A.,
Arzuaga, X., Blessinger, T., Dishaw, L., Hotchkiss, A., Makris, S.L., 2019. Hazards of
diisobutyl phthalate (DIBP) exposure: A systematic review of animal toxicology
studies [Review]. Environ. Int. 125, 579–594. https://doi.org/10.1016/j.
envint.2018.09.038.
Yost, E.E., Galizia, A., Kapraun, D.F., Persad, A.S., Vulimiri, S.V., Angrish, M., Lee, J.S.,
Druwe, I.L., 2021. Health effects of naphthalene exposure: A systematic evidence
map and analysis of potential considerations for dose–response evaluation.