Immunity

Perspective

Expanding the Immunology Toolbox:

Embracing Public-Data Reuse and Crowdsourcing
Rachel Sparks,1 William W. Lau,2 and John S. Tsang1,*
1Systems Genomics and Bioinformatics Unit, Laboratory of Systems Biology, National Institutes of Allergy and Infectious Diseases,

National Institutes of Health

2Office of Intramural Research, Center for Information Technology, National Institutes of Health

*Correspondence: john.tsang@nih.gov
http://dx.doi.org/10.1016/j.immuni.2016.12.008

New technologies have been propelling dramatic increases in the volume and diversity of large-scale public
data, which can potentially be reused to answer questions beyond those originally envisioned. However, this
often requires computational and statistical skills beyond the reach of most bench scientists. The develop-
ment of educational and accessible computational tools is thus critical, as are crowdsourcing efforts that uti-
lize the community’s expertise to curate public data for hypothesis generation and testing. Here we review
the history of public-data reuse and argue for greater incorporation of computational and statistical sciences
into the biomedical education curriculum and the development of biologist-friendly crowdsourcing tools.
Finally, we provide a resource list for the reuse of public data and highlight an illustrative crowdsourcing
exercise to explore public gene-expression data of human autoimmune diseases and corresponding mouse
models. Through education, tool development, and community engagement, immunologists will be poised to
transform public data into biological insights.

Introduction inflammation (Sweeney et al., 2015). These examples highlight

Genetics, biochemistry, and molecular and cellular biology have
long been staples of the immunologists’ toolbox. They have led
to fundamental insights about the immune system, whose
enormous complexity, however, continues to demand new
investigative tools. Recently, the advent of high-throughput, mul-
tiplexed molecular technologies, such as next-generation DNA
sequencing and high-dimensional flow and mass cytometry,
have provided a more global, higher-resolution view of immune
cells and tissues. Gene expression can now be routinely
measured genome-wide (a.k.a. the ‘‘transcriptome’’) in different
immune-cell populations down to the single-cell level (Satija and
Shalek, 2014; Shapiro et al., 2013). Similar measurements can
also be made in other modalities, including chromatin and
methylation states (Laird, 2010; Schwartzman and Tanay,
2015; Zhou et al., 2011), protein expression (Darmanis et al.,
2016; Larance and Lamond, 2015), and post-translational mod-
ifications (Olsen and Mann, 2013). Coupled with the increasingly
broad data-sharing requirements from funding agencies (Paltoo
et al., 2014), the volume and complexity of biological information
have been increasing at an unprecedented rate. Because of the
highly multiplexed nature of these measurements (e.g., covering
most genes), the data can often be reused to answer questions
beyond those posed in the original publication or envisioned
when the dataset was generated. Indeed, there have been
increasing in silico hypothesis testing and generation via reuse
of public data, initially more in cancer biology (Segal et al.,
2005; Segal et al., 2004) but recently also in immunology; for
example, researchers have used public gene-expression data
from multiple studies to infer blood transcriptomic modules
(BTMs) (Chaussabel and Baldwin, 2014; Li et al., 2014), derive
universal blood signatures of viral infection (Andres-Terre et al.,
2015; Tsalik et al., 2016), and develop predictive signatures to
differentiate between infectious and non-infectious sources of
how integration of public data from multiple studies can lead to
new and potentially robust biological insights and how it might
serve as a productive starting point for informing the design of
future experiments. This data-reuse approach differs from and
is complementary to the traditional ‘‘single lab,’’ hypothesis-
driven research paradigm (Rung and Brazma, 2013) and can
be empowering for all scientists. However, to date, computa-
tional biologists have been the primary group of researchers
driving and benefitting the most from public-data reuse and inte-
gration approaches because of their computer programming
and statistical expertise.
Here we argue that it is imperative for the immunology com-
munity to embrace the reuse and exploration of public data
alongside their current arsenal of experimental tools to tackle
immunological complexity. There are obvious cultural and scien-
tific challenges, including a lack of general awareness about the
scientific potential of data reuse and the fact that few bench sci-
entists have sufficient computational and statistical expertise for
productive, hands-on analysis of public data. Below we first sur-
vey the history and examples of large-scale public-data genera-
tion and reuse to highlight the benefits and challenges. We then
discuss ideas involving a combination of education and the
development of open, biologist-friendly software platforms to
not only enable hands-on testing and generation of hypotheses
through the use of public data but also take advantage of the
collective scientific expertise of the immunology research com-
munity (a.k.a. ‘‘crowdsourcing’’) to help enhance the usability
and utility of the growing body of public datasets (Saez-Rodri-
guez et al., 2016; Silberzahn and Uhlmann, 2015). Finally, we
report encouraging observations from an experiment testing
elements of our proposal in the microcosm of the National
Institutes of Health (NIH) immunology community, with whom
we carried out a crowdsourcing, ‘‘jamboree’’ exercise to assess

Immunity 45, December 20, 2016 ª 2016 Published by Elsevier Inc. 1191
 Immunity

Perspective

Box 1. Major Concepts and Glossary

Differentially expressed (DE) gene detection: identifies genes expressed at different levels (i.e., with increased or decreased
expression) between two groups of samples. One can subsequently evaluate the identified DE genes via gene-set enrichment
analysis to determine whether genes annotated in particular biological pathways or processes (e.g., those from the Gene
Ontology Consortium or KEGG [Kyoto Encyclopedia of Genes and Genomes]) are overrepresented in the list of DE genes.
Gene-set enrichment analysis can be performed with programs such as ToppGene (https://toppgene.cchmc.org) (Chen
et al., 2009), which takes in a set of DE genes determined via fixed cutoffs (e.g., a p value and/or a fold-change threshold);
or the analysis can be performed by a cutoff-free approach such as GSEA (http://software.broadinstitute.org/gsea/index.
jsp) (Subramanian et al., 2005).
False Discovery Rate (FDR): the fraction of statistically significant tests that are false positives (Benjamini and Hochberg,
1995).
Replication: the ability to reproduce the findings of one study in a subsequent study (Goodman et al., 2016).
Gene-expression signature: a set of genes whose expression level is associated with a phenotype.
Signature matching: the use of pattern-matching algorithms to compare the gene-expression signature reflecting a particular
phenotype to a set of established gene-expression signatures (Lamb, 2007; Lamb et al., 2006).
Meta-analysis: a statistical technique that pools information from multiple studies to identify coherent signals. This pooling is
often done with a collection of data sets where within each data set the same type of comparison or correlation analysis is per-
formed (e.g., transcripts differentially expressed in lupus patients versus healthy controls), but it is possible to apply the same
approach to pool information across different types of comparisons or correlation analyses (e.g., searching for conserved signals
between lupus versus healthy and rheumatoid arthritis versus healthy comparisons). Statistical methods frequently used for meta-
analysis include combining p values, combining effect sizes, combining ranks, and directly merging the raw data (Campain and
Yang, 2010; Chang et al., 2013; Evangelou and Ioannidis, 2013; Ramasamy et al., 2008; Sweeney et al., 2016; Tseng et al., 2012).
Module analysis: modules are sets of genes that are grouped together in a biologically meaningful way. Gene modules are
often formed through identification of mRNA transcripts with correlated expression across a set of conditions. Module analysis
is a type of dimension reduction, an effort to extract essential information from data sets with a large number of variables
(Chaussabel and Baldwin, 2014; Chaussabel et al., 2008; Li et al., 2014; Segal et al., 2005; Segal et al., 2004).
Machine learning: computer algorithms that can infer ‘‘rules’’ from data (Libbrecht and Noble, 2015). There are two primary
categories: supervised and unsupervised machine learning. Supervised learning creates classification rules from labeled
data (e.g., identification of gene-expression patterns that can separate known cancer types). Unsupervised learning detects
structure or classification rules from unlabeled data (e.g., learning previously unknown subtypes of cancer from data alone.)

Combine data from

Study A Comparison Group Pair (CGP)

Disease group Control group
Genes

Genes

VS. Sample group

Study B
Samples Samples

Study C
expressed genes
Perform meta-analysis Up Down Downstream analyses
PEX19 HADH
Meta-analysis RAB17
HAS3
PGAP3
FBLIM1
Pathway enrichment
Study N techniques SOCS1 SNAI2 analysis
STK4 SSR4
(see Box 1) EVPL PYY Module analysis, etc.
XAF1 ZNFX1
ABRACL FABP4
FDX1 RAD52

(Continued on next page)

1192 Immunity 45, December 20, 2016
Immunity
Perspective
Box 1. Continued
Figure B1. Basic Concept of Meta-Analysis Using Gene-Expression Data
A collection of CGPs (comparison group pairs) where each CGP comprises two groups of gene-expression profiles (e.g., disease versus healthy sample
groups) is shown. Meta-analysis is performed by deriving a CGP collection, typically from more than one study, and evaluating for coherent signals (genes with
consistent increased or decreased expression) across the CGPs. One can use multiple statistical techniques to perform meta-analyses (see text in this Box).
The output of meta-analysis typically includes lists of differentially expressed (i.e., with increased or decreased expression) genes, which can be used for
downstream analyses, such as pathway enrichment analysis or module analysis.
transcriptomic signatures of several common human autoim-
mune diseases and their mouse models by using public data.
Ultimately, we hope that better education, continued tool devel-
opment, and broad community engagement will help usher in an
era where scientists at any level, from the undergraduate student
to the principal investigator, can effectively utilize data from the
public domain to advance the field of immunology.
A Survey of Large-Scale Data Generation and Reuse:
From Cancer to Immunology and Beyond
Gene-expression data obtained from microarrays, and more
recently, RNA-seq, are exemplar large-scale, publicly available
biological data types whose reuse remains largely untapped by
the broad immunology community. Pioneered as a technique
to measure genome-wide transcript expression (Chee et al.,
1996; Schena et al., 1995), microarray technology found early
application in the field of cancer biology, where it enabled a
more comprehensive comparison of gene expression profiles
between tumor and healthy tissue samples (Alon et al., 1999).
In the late 1990s, the development of machine-learning tech-
niques to correlate expression patterns with cancer classes
(e.g., acute myeloid leukemia versus acute lymphoblastic leuke-
mia) led to the use of microarray data both in cancer subtype pre-
diction (i.e., assignment of samples to known classes of cancer)
and in class discovery (identification of new cancer types or sub-
types) (Alizadeh et al., 2000; Golub et al., 1999). Within a few
years, expression data generated from several types of cancer
(e.g., breast, lung, skin, and colon) had provided cancer biolo-
gists with a more global perspective on the transcriptional
changes that occur in carcinogenesis (Alon et al., 1999; Bittner
et al., 2000; Perou et al., 2000; Wang et al., 2000). Shortly there-
after, these techniques were adopted by other fields, including
infectious diseases (Jenner and Young, 2005) and immunology,
as demonstrated by an early evaluation of peripheral blood
mononuclear cell (PBMC) gene expression from patients with
the autoimmune disease systemic lupus erythematosus (SLE),
which identified an ‘‘interferon (IFN) signature’’ involving the up-
regulation of IFN-inducible genes in SLE patients compared to
healthy controls (Baechler et al., 2003; Bennett et al., 2003).
Concurrently, in the late 1990s several groups created
compendia of expression profiles from yeast in a variety of
states. Some yeast had genetic mutations, some had been
exposed to chemicals, drugs, and environmental stresses, and
some were in distinct stages of the cell cycle (Gasch et al.,
2000; Hughes et al., 2000; Spellman et al., 1998). Most notable
was a compendium of expression profiles generated by Hughes
et al. (Hughes et al., 2000) from genetic mutations and chemical
exposures. They used this collection of reference expression
patterns to search for potential pathways affected by chemical
compounds with previously unknown targets or to assess the ef-
fects of gene deletions in uncharacterized open-reading frames
by assessing how these ‘‘unknown’’ profiles clustered with pro-
files in the compendium. This concept later inspired the creation
of the ‘‘Connectivity Map,’’ a collection of mRNA signatures
(i.e., a set of genes with altered expression in one condition
versus another) generated through the exposure of human cell
lines to a large number of small molecules (Lamb, 2007; Lamb
et al., 2006). In similarity to the Hughes et al. approach, re-
searchers can use these reference profiles with pattern-match-
ing computational tools to identify both correlated and anti-
correlated relationships with other gene expression data through
a process termed ‘‘signature matching’’ (Box 1) (Iorio et al., 2013;
Lamb et al., 2006). These datasets have also been amply reused
by bioinformaticians for the development of computational
methods and the construction of gene network models (Pe’er
et al., 2001; Segal et al., 2003a; Segal et al., 2003b). However,
performing such analyses has been primarily limited to re-
searchers with advanced computational and statistical skills
and has thus been largely out of reach for the traditional bench
scientist.
With the rapid accumulation of large datasets such as gene-
expression compendia has come the establishment of numerous
data repositories, including Gene Expression Omnibus (GEO)
and ArrayExpress, for housing such data (see Box 2 for refer-
ences, internet links, and more details on public data repositories
discussed here). Some more recent repositories are organized
around biological areas and contain multi-modal data generated
from the same samples. For example, The Cancer Genome Atlas
(TCGA) and the International Cancer Genome Consortium
contain DNA sequence, gene expression, and other types of mo-
lecular and clinical data from multiple tumor types. In the immu-
nological realm, ImmPort serves as a repository of immunology
research data; ImmuneSpace, a data portal of the Human Immu-
nology Project Consortium (HIPC), contains human immune
profiling data, such as blood transcriptomic and phenotyping
of peripheral immune cells, taken before and after perturbations
such as influenza vaccination. On the mouse side, the Immuno-
logical Genome Project (ImmGen) is a large-scale collaborative
effort to create a gene-expression compendium of mouse im-
mune cells.
The increase in publicly available expression datasets spurred
interest in reusing and integrating data across studies through
meta-analysis—an approach that pools information from multi-
ple studies to identify coherent signals (Figure B1 in Box 1).
For example, the IFN signature originally identified in SLE pa-
tients was subsequently found in several other autoimmune con-
ditions, including rheumatoid arthritis (RA) and Sjogren’s
syndrome, through meta-analysis studies that evaluated several
autoimmune conditions simultaneously (Higgs et al., 2012;
Toro-Domı́nguez et al., 2014). In comparison to single-study
analysis, meta-analysis has several potential advantages,
including increased statistical power, and the results are
Immunity 45, December 20, 2016 1193

typically more robust. Such studies typically have better replica-
bility because platform- and study-specific issues are minimized
through computational techniques that identify congruent sig-
nals across studies [reviewed in (Campain and Yang, 2010;
Chang et al., 2013; Ramasamy et al., 2008; Sweeney et al.,
2016; Tseng et al., 2012); examples in (Andres-Terre et al.,
2015; Khatri et al., 2013; Sweeney et al., 2015)]. Meta-analyses
can still be susceptible to pre-analytic biases, such as those
associated with dataset selection, but having a clear precon-
ceived strategy for the identification of independent discovery
and validation datasets and a well-thought-out analysis plan
can help to mitigate these issues (Ramasamy et al., 2008).
Early examples of gene-expression meta-analyses are found
primarily in cancer biology (Rhodes et al., 2002; Rhodes et al.,
2004; Wirapati et al., 2008), where independent expression
profiling of tumor samples produced numerous datasets that
could be analyzed collectively. More recently, meta-analysis of
TCGA and other public expression data from tumor samples
has been used for molecular and genomic characterization of
multiple cancer subtypes (Weinstein et al., 2013) as well as to un-
cover novel cancer biology, including the effect of copy number
variation on gene expression (Fehrmann et al., 2015), and the
identification of common cancer subtypes among different tu-
mors (Hoadley et al., 2014) and tumor-associated microRNAs
and methylation status (Gross et al., 2015). These examples
demonstrate how data available in the public domain can be em-
powering for scientists who wish to generate and test hypo-
theses, particularly prior to designing an experiment or when
deciding how to prioritize laboratory resources. As the volume
of public gene expression data has grown, there is increased po-
wer to extract biological signals within the data (Lukk et al., 2010;
Torrente et al., 2016) and greater ability to construct independent
discovery and validation study sets that are critical for the devel-
opment and evaluation of robust gene-expression signatures
(Andres-Terre et al., 2015; Khatri et al., 2013; Sweeney et al.,
2015).
As the importance of immunotherapy in cancer becomes
increasingly appreciated, and given the pioneering role that can-
cer biology played in the reuse of gene expression data, it is not
surprising that the field of cancer immunology has been utilizing
public data to assess immune infiltrates in tumors. One example
involves using public expression data from purified subsets of
immune cells to build a compendium of mRNA signatures for
‘‘deconvolving’’ the frequency of these cell subsets in tumors
(Bindea et al., 2013). Researchers used this approach, in combi-
nation with other experimental techniques, to evaluate human
colorectal cancer samples at both the center of the tumor and
at the invasive margin. This approach generated an ‘‘immune
spatiotemporal landscape’’ within a tumor, which highlighted
the important role of the adaptive immune system in the tumor
immunological milieu and led to the discovery that certain im-
mune-cell infiltrates were associated with disease prognosis.
Similarly, a recent study that characterized the infiltrating T cell
receptor repertoire by using more than 9,000 RNA-seq samples
from 29 cancer types in TCGA found that the degree of TCR
CDR3 diversity was positively correlated with the burden of tu-
mor nonsynonymous somatic mutations and the expression
level of two genes encoding cancer/testis antigens (Li et al.,
2016). Gentles et al. compiled public data on gene expression
1194 Immunity 45, December 20, 2016
Immunity
Perspective
and clinical outcomes from approximately 18,000 subjects with
39 cancer subtypes (Gentles et al., 2015). They performed a
‘‘pan-cancer meta-analysis’’ to identify genes associated with
prognosis and used the immune-cell deconvolution tool CIBER-
SORT (Newman et al., 2015) to infer the relative frequency of 22
leukocyte subsets within each tumor type. Together, these data
again suggested a relationship between the tumor immune envi-
ronment and disease outcome; similar to the findings of Bindea
et al. (2013), a larger presence of T cells within the tumor was
associated with better patient outcomes. In a more focused eval-
uation of the specific role of cytotoxic immune cells (CD8+ T cells
and NK cells) in the tumor microenvironment, Rooney et al.
created a quantitative measure of immune cytolytic activity on
the basis of mRNA levels of granzyme A and perforin by using
data from multiple cancers in TCGA (Rooney et al., 2015). They
found that higher expression of this ‘‘cytolytic signature’’ was
associated with a modest increase in survival, greater expres-
sion of tumor neoepitopes, expression of endogenous retrovi-
ruses within several tumor types, and increased presence of mu-
tations in genes involved in antigen presentation and extrinsic
apoptosis. Taken together, these studies of the tumor-immune
relationship are notable for their insights into a complex biology
and for the utility of public gene-expression data in contributing
to this understanding.
There has also been increasing in silico hypothesis testing via
reuse of public gene-expression data in immunology. In an effort
to better understand the human immune response to vaccina-
tion, Li et al. (2014) combined data generated in a human study
of responses to the polysaccharide and conjugate meningo-
coccal vaccines with public gene-expression data from several
other vaccination studies. To assess multi-gene, ‘‘module’’-level
changes in peripheral blood immune cells after vaccination, they
first used public data from more than 30,000 human blood
gene-expression profiles to construct a set of BTMs, each of
which comprises genes whose expression is correlated with
each other (see Box 1). A similar modular approach was used
earlier in a study of SLE (Chaussabel and Baldwin, 2014; Chaus-
sabel et al., 2008). The Li et al. (2014) study found that three
modular transcriptomic patterns were present in PBMCs in the
first few days after vaccination and that, interestingly, these pat-
terns were clustered by the type of vaccine used (live viral versus
polysaccharide versus protein). The authors thus concluded that
there was unlikely to be a common, early-response signature
predictive of the antibody response to multiple types of vac-
cines. However, in separate efforts, universal blood signatures
of viral infection have been derived from infection data (An-
dres-Terre et al., 2015; Tsalik et al., 2016). Notably, one study
used meta-analysis of public expression data from multiple
studies to generate conserved gene signatures for several types
of viral infection or influenza-specific infection; the influenza-
specific signature was able to distinguish influenza infection
from bacterial and non-influenza viral infections (Andres-Terre
et al., 2015). Similarly, a meta-analysis of public microarray data-
sets generated an 11-gene signature that allowed researchers to
differentiate between infectious and non-infectious sources of
inflammation after, for example, trauma (Sweeney et al., 2015).
Other interesting immunological hypotheses recently evalu-
ated solely through the reuse of public data include the question
of a universal signature of organ rejection and the balance of
Immunity
Perspective
leukocyte subsets in schizophrenia. In one case, researchers
combined eight transplant gene-expression datasets from four
solid organs (kidney, lung, heart, and liver) to identify a common
gene expression signature of acute rejection (Khatri et al., 2013).
This signature was reassuringly also positively correlated with
the degree of histologic graft injury in a separate group of
renal-transplant public datasets. In another example, as part of
an investigation of the link between immunology and psychiatry,
researchers applied CIBERSORT (Newman et al., 2015) to public
gene-expression data from subjects with schizophrenia and bi-
polar disorder to infer the immune cell proportions in blood for
each group of subjects. This approach identified a lower number
of circulating natural killer cells in medicated schizophrenia pa-
tients than in healthy controls (Karpin
ski et al., 2016). As more
data become available in the public domain, these hypotheses
can be further refined, and new hypotheses can be generated.
Researchers have also applied public gene-expression data to
identify possible drug targets and potentially novel drug indica-
tions [reviewed in (Chen and Butte, 2016)]. This is facilitated
by the ability to generate disease-specific expression signa-
tures and by the development of drug-specific gene-signature
databases, such as the ‘‘Connectivity Map’’ discussed earlier
(Lamb, 2007; Lamb et al., 2006). Through identification of inverse
drug-disease correlations, numerous novel drug indications,
including fenofibrate for the prevention of graft rejection and
cimetidine in the treatment of lung adenocarcinoma, have been
proposed (Kidd et al., 2016; Roedder et al., 2013; Sirota et al.,
2011). Although some independent experimental data support
these novel drug-disease associations, there is not yet strong
clinical data demonstrating the success of this approach.
In addition to gene-expression data, meta-analyses of publicly
available data from genome-wide association studies (GWASs)
are now routinely used as a way to increase statistical power
and decrease the risk of false positives associated with smaller
studies (Begum et al., 2012; Evangelou and Ioannidis, 2013).
A particularly successful case is autoimmunity, where GWAS
meta-analyses have identified numerous unique and shared
loci among several diseases, including SLE, RA, type I diabetes,
multiple sclerosis, Crohn’s disease, and ulcerative colitis (Ander-
son et al., 2011; Bradfield et al., 2011; Franke et al., 2010; Már-
quez et al., 2016; Morris et al., 2016; Patsopoulos et al., 2011;
Stahl et al., 2010). Although the push for larger meta-analyses
has successfully uncovered new risk loci (Okada et al., 2014),
increasing coverage or sample sizes does not necessarily result
in additional associations (Fuchsberger et al., 2016). Candidate
genes emerging from GWAS meta-analyses could be causal
for the disease, and thus there has been strong interest in using
meta-analysis results to help reposition drugs or identify novel
drug targets (Grover et al., 2015; Hurle et al., 2013; Nelson
et al., 2015; Sanseau et al., 2012). One example is the associa-
tion between CD40 and RA (Okada et al., 2014; Raychaudhuri
et al., 2008); there is currently a phase Ib clinical trial of a
CD40L antagonist in patients with adult-onset RA (clinical trial
number NCT02780388).
Pairing GWAS data with information about biological path-
ways and protein-protein interactions (PPIs) can aid the interpre-
tation of results (Wang et al., 2010); both types of information are
available in public biological repositories (Box 2). Large-scale
PPI data (sometimes referred to as the ‘‘interactome’’) have
been generated via two main strategies: the yeast two-hybrid
strategy for obtaining pairwise interactions and mass spectrom-
etry for characterizing protein complexes [(Rolland et al., 2014);
reviewed in (Lage, 2014)]. Several databases have been estab-
lished to facilitate the collection and dissemination of these
data, including CORUM, a curated database of protein com-
plexes (Ruepp et al., 2010). Another effort is the International Mo-
lecular Exchange (IMEx) consortium, which is an attempt to pro-
vide quality-controlled PPI data available to the public [(Orchard
et al., 2012); www.imexconsortium.org]. Researchers have inte-
grated such pathway and PPI data with GWAS data to identify
potential biological networks involved in specific human pheno-
types (Califano et al., 2012; Cotsapas et al., 2011; Lage, 2014;
Rossin et al., 2011). Similarly, expression quantitative trait locus
(eQTL) data, which link genetic variations to differences in gene
expression, are increasingly available for various tissues and cell
types through public databases such as that generated by the
Genotype-Tissue Expression (GTEx) project (see Box 2) (Gibson
et al., 2015). Meta-analysis also allows the combination of multi-
ple datasets to generate potentially more robust eQTLs (Westra
et al., 2013). Similar to PPI data, eQTLs can help researchers
interpret associations from GWASs by, for example, generating
hypotheses on the identity of genes whose expression level
may be impacted by GWAS variants (Zhu et al., 2016). eQTL
data can also allow researchers to assess the extent of genetic
contributions and generate hypothesis on potential mechanisms
that underlie gene expression signatures associated with partic-
ular phenotypes (Huan et al., 2015).
Immunology research often generates cellular phenotyping
data, such as those obtained from flow and mass cytometry
(Bendall et al., 2012). Until recently, these data have not routinely
been deposited in public repositories (Box 2), but increased mul-
tiplexing of markers (now up to the 40s) and studies with larger
sample sizes have propelled the generation and deposition of
bigger datasets with increasing potential for reuse (Bhattacharya
et al., 2014; Roederer et al., 2015). However, reusing such data
can be labor intensive, and the development of automated anal-
ysis tools, such as software for automatic gating of immune cell
subsets (Saeys et al., 2016), is still in its infancy. Despite these
challenges, community-based efforts to advance the develop-
ment of such tools are underway (Aghaeepour et al., 2013; Finak
et al., 2016), and promising examples of reusing and integrating
cellular phenotyping datasets are emerging (Jujjavarapu et al.,
2016; Lu et al., 2016).
Toward Adding Public-Data Reuse to the
Immunologists’ Toolbox
As the examples above demonstrate, reuse of public data pre-
sents an opportunity for bench scientists to test and generate hy-
potheses; this is particularly useful for labs, including a majority
of immunology labs, that do not routinely generate such large-
scale data. In this sense, the growth of public data also offers
an opportunity to level the playing field for smaller or less-well-
funded groups. This will complement the traditional research
model that involves hypothesis generation followed by experi-
ments testing the hypothesis with a paradigm in which one can
first generate or refine hypotheses by analyzing publicly avail-
able data, then design and execute experiments (Figure 1A).
As illustrated above, starting with an examination of public
Immunity 45, December 20, 2016 1195
 Immunity

Perspective

Box 2. Resources for Reusing Public Data

Commonly used data repositories:

d Gene Expression Omnibus (GEO: http://www.ncbi.nlm.nih.gov/geo/) (Barrett et al., 2013) and ArrayExpress (https://www.ebi.
ac.uk/arrayexpress/) (Kolesnikov et al., 2015): contain multiple forms of data, including expression, chromatin accessibility,
and methylation, obtained from microarray and next-generation sequencing. Some GEO data sets have undergone expert cu-
ration and annotation (GEO DataSets; http://www.ncbi.nlm.nih.gov/gds).
d Flowrepository (https://flowrepository.org) (Spidlen et al., 2012): a database for flow-cytometry data.
d Databases of biological pathways and protein-protein interactions (reviewed in (Klingström and Plewczynski, 2011)). Examples
include the Reactome Pathway Database (http://www.reactome.org) (Fabregat et al., 2016) and the STRING protein-protein
interaction database (http://string-db.org) (Szklarczyk et al., 2015).
Large data compendia (including multi-modal data sets associated with individual studies):
d ImmGen (http://www.immgen.org) (Heng et al., 2008): microarray gene-expression data on the major immune cell populations
of the mouse.
d The Immunology Database and Analysis Portal (ImmPort; http://immport.org/immport-open/public/home/home) (Bhatta-
charya et al., 2014): experimental and clinical trial data generated from studies primarily funded by the National Institute of
Allergy and Infectious Diseases (NIAID).
d ImmuneSpace (https://www.immunespace.org) (Brusic et al., 2014): the data portal of the NIAID HIPC program; contains mul-
tiple data types generated to characterize the human immune system.
d The Cancer Genome Atlas (TCGA, https://cancergenome.nih.gov) (Tomczak et al., 2015): genomic, transcriptomic, proteomic,
epigenomic, and clinical data on more than 30 types of cancer and matched normal tissues.
d International Cancer Genome Consortium (ICGC; http://icgc.org) (Hudson et al., 2010): multi-modal data on more than 20 types
of cancer and matched normal tissues from around the world.
d Genotype-Tissue Expression (GTEx) project (http://www.gtexportal.org/home/) (GTEx Consortium, 2013): expression quanti-
tative trait loci (eQTLs) database created with multiple tissues from deceased human donors.
d Connectivity Map (CMAP) (https://portals.broadinstitute.org/cmap/) (Lamb, 2007; Lamb et al., 2006): a compendium of gene-
expression profiles created from small-molecule perturbations of human cell lines. Using expression-pattern matching, the
user can query a signature against this compendium to find matches.
d Library of Integrated Network-based Cellular Signatures (LINCS) (http://www.lincsproject.org): Similar to the CMAP but with
more extensive perturbations and cell types.
Analysis tools and platforms that enable data reuse:
d Search-Based Exploration of Expression Compendium (SEEK; http://seek.princeton.edu) (Zhu et al., 2015): a cross-platform
gene co-expression search system. Using gene sets provided by the user, it identifies genes and public datasets of interest
(e.g., microarray, next-generation sequencing) that are co-expressed with the query genes.
d OMiCC (https://omicc.niaid.nih.gov) (Shah et al., 2016): open, crowdsourcing-based platform aimed at biologists without
computational training for annotating and performing (meta-) analysis of public microarray datasets.
d Open ImmPort (http://immport.org/immport-open/public/home/home) (Bhattacharya et al., 2014): contains an example of
how to re-analyze data from a published influenza vaccine study with sample code in R and Python. There are also additional
analysis tools, such as those for automatic analysis of flow-cytometry data (access requires registration and approval).
d ImmuNet (http://immunet.princeton.edu) (Gorenshteyn et al., 2015): 15 immunological functional-relationship networks
created with publicly available functional genomics data. Using genes provided by the user, it retrieves immunological subnet-
works functionally associated with those genes.
d GEO2R (http://www.ncbi.nlm.nih.gov/geo/geo2r/) (Barrett et al., 2013): online tool that can be used for comparing groups of
samples within a GEO series. The output is a list of differentially expressed genes. It also generates the R code used to perform
the analysis.
d Interactive Gene Expression Data Browser (https://gxb.benaroyaresearch.org/dm3/landing.gsp) (Speake et al., 2015): a web-
based viewer for interactive exploration of microarray data sets; preloaded with more than 150 public data sets relevant to hu-
man immunology.
d InSilicoDB (www.insilicodb.com) (Coletta et al., 2012): aimed at biologists without computational training, it allows for explo-
ration of public genomics data, with the option to upload and analyze personal, unpublished data.
d NextBio (www.nextbio.com) (Kupershmidt et al., 2010): a database of searchable signatures, including those derived from
gene-expression, genetic, and other types of high-throughput biological data.
d Molecular Signatures Database (MSigDB) (http://www.broadinstitute.org/msigdb) (Liberzon et al., 2011): a large collection of
annotated gene sets, including immunology-specific sets.
d Selected R packages for data reuse and meta-analysis: MetaOmics, a suite of three R packages (MetaQC, MetaDE, and
MetaPath) for quality control, differentially expressed gene identification and enriched pathway detection for microarray
meta-analysis (http://www.pitt.edu/tsengweb/MetaOmicsHome.htm) (Wang et al., 2012). MetaIntegrator, an R package
facilitating multi-cohort gene-expression meta-analysis (Haynes et al., 2016).

1196 Immunity 45, December 20, 2016

Immunity
Perspective
A Toolbox
B and data compendia.
data might result in additional biological insights because (1) the
signal-to-noise ratio can be improved through meta-analysis of
data from multiple, independent studies and (2) as pointed out
by Khatri and colleagues (Andres-Terre et al., 2015; Khatri
et al., 2013; Sweeney et al., 2015), it can be advantageous to uti-
lize heterogeneity across studies (e.g., different microarray plat-
forms, distinct patient cohorts) because the coherent signals
across studies are potentially more likely to be biologically
meaningful and reliable as the basis of generating and testing
new hypotheses.
Despite advances in the development of data standards and
software tools to promote public data reuse and exploration,
reuse of public data remains sparsely practiced outside of the
bioinformatics community as a result of several barriers. First
and foremost, computational and statistical analysis of public
data, such as meta-analysis across multiple studies, remains
technically challenging for most biologists. For example, the abil-
ity to derive or match a gene-expression signature remains
largely out of reach for biologists who lack bioinformatics training
Figure 1. Expanding the Immunologists’
(A) The traditional research paradigm (left) focuses
on hypothesis generation followed by generation
of experimental data for hypothesis testing. An
augmented, complementary paradigm (right)
uses public data to generate and/or refine hy-
potheses prior to designing experiments.
(B) A proposal for adding public-data exploration
and reuse into the immunologists’ toolbox that
involves (1) education in bioinformatics, computer
programming, and relevant areas of statistics
and applied mathematics; (2) development of
software and ‘‘data commons’’ platforms to
enable hands-on exploration of public data for
hypothesis generation and testing; and (3) com-
munity engagement to create reusable content
such as sample and sample group annotations
(Box 1). Commercial offerings such as
NextBio and InSilicoDB offer program-
ming-free analysis of pre-processed
public data (Box 2), but these options
can involve proprietary algorithms or
analysis parameters that are not easily
modified by the user, can be impractical
for individuals with limited resources
(e.g., students or labs from developing
countries), and are limited in their ability
to serve as an open resource to help pro-
pagate new ideas and tools within the
greater immunology community.
Computational and statistical skills are
not the only hurdles; truly embracing the
data-reuse paradigm requires a cultural
change (Rung and Brazma, 2013). Most
bench scientists understandably take
pride in generating their own data and
thus may find it difficult to relegate hy-
pothesis generation and testing to pub-
licly available data generated by others.
However, as a community we are already employing an increas-
ingly large amount of resources generated by ‘‘big science’’ ef-
forts; examples include mouse immune-cell profiling data from
the ImmGen Project (Heng et al., 2008) (Box 2). Reusing existing
data is also philosophically similar to consulting the literature
when designing experiments and generating hypotheses. And
as discussed above, hypotheses developed from meta-analysis
of multiple studies can potentially be more robust against various
biases than data generated within a single lab. However, it is also
worth emphasizing that careful interpretation of meta-analysis re-
sults is needed given study-to-study heterogeneity in both tech-
nical and biological factors. For example, differences in subject
inclusion criteria (e.g., age restrictions) can be prevalent among
different human studies such that coherent signals across
studies might emerge from only a subset of the subjects with
shared biological features; thus, one should cautiously evaluate
the broad applicability of such results to the general human pop-
ulation [see also Tseng et al. (2012) and Ramasamy et al. (2008)
for discussion of potential biases and related statistical issues
Immunity 45, December 20, 2016 1197

on microarray meta-analysis]. Furthermore, examination of pub-
lic data does not replace conducting experiments or human clin-
ical trials, which are required to confirm hypotheses and generate
de novo information most fitting for the scientific questions posed
(Dolinski and Troyanskaya, 2015). With the mandate from funding
agencies to deposit data in the public domain (Paltoo et al., 2014),
a gradual movement toward more accessible supplemental data
associated with publications (Pop and Salzberg, 2015), and a
push for increasing sharing of raw clinical trial data (Doshi et al.,
2013; Ross and Krumholz, 2013), being thoughtful about what
public data can be utilized is a prudent starting point for many hy-
pothesis-generating or -testing endeavors. Furthermore, data
reuse can increase research efficiency and reduce redundancy,
which should be particularly emphasized in research on human
subjects given the inherent risk to participants (Doshi et al., 2013).
Even with the right tools, skills, and mindsets, the varying avail-
ability and quality of the data and of the associated documentation
and annotations pose another challenge. The genetics community
has been a leader in data sharing, as exemplified both by early
large-scale data-generation efforts such as the Human Genome
Project (Collins et al., 2003) [and more recently, the 1000 Genomes
Project (The 1000 Genomes Project Consortium, 2015)] and by
far-reaching data-sharing policies such as the Fort Lauderdale
and Toronto agreements (Birney et al., 2009; Kaye et al., 2009).
However, reuse of more complex functional genomics data typi-
cally requires greater documentation and annotation. Take micro-
array data as an example: the quality of the data depends on the
biological and experimental conditions under which they are
generated and the computational procedures by which the data
are processed. Crucial information about samples can be missing
or provided without the use of standardized vocabularies, and one
often needs to consult the original publication and its authors to
gather the necessary information to reuse the data. Even the
most well-curated datasets might need additional annotation
before they can be reused effectively. For example, it is crucial
to determine what sample groupings and comparisons should
be made to generate a desired gene-expression signature (e.g.,
naive versus activated CD4+ T cells); and one often needs biolog-
ical expertise in the area of inquiry to perform such annotations.
To improve data and annotation quality, some groups have
made significant efforts to promote standards for data submis-
sion to public repositories and encourage the use of standard-
ized vocabularies and ontologies for annotation (Musen et al.,
2012) to facilitate data reuse and increase study reproducibility
(Brazma, 2009; Brazma et al., 2001; Rung and Brazma, 2013).
Despite advances in standardized vocabulary and ontology
development, putting them into practice remains challenging
because large amounts of existing data would need to be re-an-
notated, and many major data repositories have yet to directly
support their use. Data submission and sharing standards would
ideally allow other investigators to more easily reproduce the an-
alyses reported by the primary researchers and enable more
informed comparisons across studies. Although such standards
have not been consistently enforced (Brazma, 2009; Ioannidis
et al., 2009), greater cooperation between journals and public
databases to require data annotation and deposition prior to
publication of papers reporting large-scale datasets could
have a significant positive impact on the quantity and availability
of reusable public data.
1198 Immunity 45, December 20, 2016
Immunity
Perspective
Overcoming the above major obstacles to make public-data
reuse a common practice in immunology and, more broadly, in
all of biology will take time and concerted efforts (Figure 1B). First,
as a community we need to embrace adding rigorous training in
computational and statistical sciences and bioinformatics to
the standard biological sciences curriculum, starting at the un-
dergraduate level. We must also commit to covering more immu-
nology-specific issues, such as the analysis of flow cytometry
and single-cell phenotyping data, in immunology graduate pro-
grams. Doing so would provide future immunologists a solid
theoretical grounding and hands-on experience in computational
immunology. Second, analysis software and platforms designed
specifically for public-data reuse and data integration are needed
for different data modalities, including those, such as flow cytom-
etry, that immunologists use frequently. Although some software
resources have already been created (Box 2), particularly for re-
using and analyzing public gene expression data, they are often
limited to one or a subset of analytical steps, and therefore putting
together an entire workflow often requires some programming
expertise. An important goal of such software should be to enable
direct, hands-on exploration of public data even by those without
formal bioinformatics training. Doing so provides several benefits
(Shah et al., 2016). First, one can directly exercise his/her biolog-
ical intuition. Second, it helps to promote deeper, more inte-
grated collaboration between bench and bioinformatics scien-
tists, for example, by allowing bioinformaticians to focus more
on the computationally challenging aspects while immunologists
explore the biological questions of interest by using well-estab-
lished analytical techniques. The most successful software tools
will also emphasize user education, so that biologists can learn
the statistical language and techniques necessary to properly uti-
lize the software and interpret the results. Software can thus also
play an important role in achieving the educational goals dis-
cussed above by allowing students to learn both biology and bio-
informatics simultaneously. In doing so, students would come to
understand that the two are often inextricably linked.
In addition to enabling programming-free (meta-) analysis of
public data, such software platforms can serve as ‘‘data com-
mons’’ for the entire research community (e.g., see https://
www.synapse.org), for example to allow sharing of raw data, as
well as annotations, data compendia, analysis results, and other
reusable content created by members of the community. Not
only can such practices of crowdsourcing tap into the collective
expertise of the community to create reusable information, they
also promote a culture of sharing and openness (Figure 1B).
The immunological community in particular should embrace
and can benefit from this collaborative approach given both the
complex jargon and biology of the field and the accelerating
use of technologies that generate increasingly complex data
types across multiple biological scales (Germain et al., 2011).
The NIH OMiCC Jamboree: A Microcosmic
Crowdsourcing Experiment
We recently developed OMiCC (Shah et al., 2016) (see Box 2), a
free online platform that implements some of the key ideas dis-
cussed above for programming-free (meta-) analysis of public
gene-expression data. One unique feature of OMiCC is the
capacity to ‘‘crowdshare’’ the work of annotating public data
for automated downstream (meta-) analyses, a critical but
Immunity

Perspective

Box 3. NIH OMiCC Jamboree

Here we provide a summary of the jamboree. The analysis approaches, observations, and caveats can be found at in Lau and
Sparks et al. (Lau et al., 2016) and on the supplemental website accompanying this paper (https://omicc.niaid.nih.gov/
2016-nih-jamboree-analysis/report.html).
Advertised to the NIH immunology community, the jamboree involved a half-day group orientation to the OMiCC platform and a
subsequent day-long jamboree. The 29 volunteer participants, consisting of faculty, fellows, and students, were separated into ten
groups; each group had at least one participant who felt proficient in using OMiCC after the half-day training session. The groups
were each responsible for a human disease (one of five autoimmune diseases: diabetes mellitus type 1, multiple sclerosis, rheu-
matoid arthritis, Sjögren’s syndrome, systemic lupus erythematosus, or the inflammatory condition sarcoidosis) or the corre-
sponding mouse models. The groups were directed to search OMiCC for gene-expression data on their assigned topic and to
focus on studies with both case and control samples generated from specific sources (e.g., PBMC and whole blood (WB) for hu-
man samples). Using the datasets they found, the participants used their biological knowledge to annotate sample groups and
create ‘‘comparison group pairs’’ (CGPs; Figure B1 in Box 1). We then analyzed CGPs from all groups after the jamboree both
within and outside of OMiCC to assess the benefits and caveats of using crowdsourcing for gathering and annotating public
data to drive hypothesis generation (Lau et al., 2016).
We used meta-analysis within OMiCC to derive (1) cross-CGP gene expression signatures for each disease (disease versus control
comparisons) and (2) conserved signatures across all diseases within each species (pan-disease signatures); we also examined
signature conservation between human and mouse (Lau et al., 2016).
Overall, a large number of transcripts with increased or decreased expression in disease versus control comparisons were iden-
tified for each disease (Lau et al., 2016). Comparison of these gene signatures among different diseases within and across species
suggests greater overlap among diseases within a species than across the two species (most likely also reflecting differences be-
tween non-blood murine tissues and human blood). Gene-set and pathway enrichment analysis of individual disease signatures
revealed shared signatures across diseases. Even though the data can be noisy and there are a number of caveats, our jamboree
experiment suggests that there are potentially interesting and robust signals to be mined; given a programming-free, didactic,
community-based platform such as OMiCC, together with some upfront training on the platform, biologists with any level of bio-
informatics experience can benefit from and contribute to utilizing public data for hypothesis exploration. Our post-jamboree user
survey revealed a similarly positive sentiment from a majority of the participants (see supplemental website).
Key Lessons Learned
d CGP quality can vary depending on the biological complexity of the data set and the biological expertise of the participant. One
or more of the following could help mitigate these challenges: (1) having more detailed inclusion and exclusion criteria, (2) early
feedback on CGPs, and (3) emphasis on CGP quality rather than quantity.
d The learning curve to use OMiCC can be steep for some participants, suggesting that more extensive training of the software
beforehand would be helpful.
d For diseases with a sufficient number of data sets, it would be helpful to collect independent discovery and validation CGP sets
for downstream validation.

time-intensive step in data reuse. OMiCC stores and allows
sharing of user-created sample annotations, groupings, two-
group comparisons, and compendia for generating expression
signatures. Thus, OMiCC provides the research community
(‘‘the crowd’’) with the capacity to move science forward through
‘‘virtual,’’ community-wide collaborations; a similar concept was
perhaps first demonstrated through the group effort to annotate
the genomic sequence of Drosophila melanogaster during an
11-day ‘‘Annotation Jamboree’’ in 1999 (Adams et al., 2000;
Pennisi, 2000). Various forms of crowdsourcing have been
used in the biomedical and computing communities in the form
of programming ‘‘hackathons,’’ data-analysis marathons, and
open challenges aimed at taking advantage of collective com-
munity expertise to tackle complex problems (Celi et al., 2014;
Saez-Rodriguez et al., 2016). These approaches have the poten-
tial to become even more broadly applicable and fruitful given
the increasingly connected global research community.
To test the hypothesis that OMiCC could be used effectively
by biologists without formal computational training and that it
could facilitate an organized, concerted group effort to collect,
annotate, and design two-group comparisons (see Figure B1)
by using public gene-expression data for achieving a shared
research goal, we organized a social and scientific experiment
that we called the ‘‘NIH OMiCC Jamboree’’ (see Box 3). Inspired
by the original ‘‘Annotation Jamboree’’ (Adams et al., 2000), we
aimed to compile and annotate gene-expression data across
multiple human autoimmune diseases and the corresponding
mouse models during the jamboree. After the jamboree, we per-
formed meta-analysis of multiple studies within and across dis-
eases by using OMiCC and compared human and mouse to
evaluate pan-disease and pan-species gene-expression signa-
tures. Our jamboree exercise provides evidence that when
enabled by appropriate tools, a ‘‘crowd’’ of biologists without
specialized training in bioinformatics can work together to
potentially accelerate the pace by which the increasingly large
amounts of public data can be meta-analyzed for generating
and testing hypotheses. The data, annotations, and analysis re-
sults generated by this exercise provide a resource for the com-
munity to further explore autoimmune disease gene-expression
signatures (see Box 3 on accessing the jamboree data and anno-
tations). The format and materials used in our jamboree should
also be easily replicable in other institutions (e.g., universities)

Immunity 45, December 20, 2016 1199


to help educate students and to utilize the ‘‘crowd’’ to assess
other biological questions of interest (materials used in the
jamboree are freely available upon request). These ‘‘grassroots’’
efforts can in turn contribute valuable curated and annotated da-
tasets to the larger community.
Conclusion and Outlook
As highlighted here, there is increasing application of data reuse
within the immunological community, but major hurdles remain
for broader adoption. As the volume of public data grows, so
will the extent of missed opportunity if immunologists fail to
take advantage of the wealth of data in the public domain.
Some of the most successful researchers in the future will likely
be those who recognize the value of reusing public data and are
able to utilize it to complement their work at the bench or in
the clinic. It is also increasingly likely that some immunological
questions can be addressed most effectively by utilizing large-
scale public data, owing to the growing size and utility of data-
sets present in the public domain and the difficulty that a single
lab would face in trying to generate the required data de novo.
Our jamboree experience illustrates how free, user-friendly,
community-based platforms (such as OMiCC) can potentially
facilitate data reuse, user education, and crowdsourcing. But
OMiCC is just an illustrative beginning, and much more needs
to be done. Specifically, we need functionalities that (1) go
beyond the meta-analysis of gene expression data by extending
into other modalities, such as flow cytometry data, (2) incorpo-
rate more sophisticated statistical modeling and analysis
approaches, and (3) provide additional annotation and crowd-
sourcing features, for example to enable group-based collabora-
tions. The development of natural text-processing methods that
can automatically annotate samples and extract biologically
meaningful comparisons could also dramatically increase the
amount of reusable content. The expansion and evolution of
data types and data volume will only increase in the future: by
taking the necessary steps as a community to move toward uti-
lizing available data to the fullest extent possible, the field of
immunology will have the opportunity to be a leader in transform-
ing public data—a continually growing, massive resource—into
biological insights.
AUTHOR CONTRIBUTIONS
R.S. designed and organized the jamboree, performed post-jamboree data
curation, and wrote the manuscript; W.W.L. helped design the jamboree, per-
formed post-jamboree data curation, designed and performed post-jamboree
data analysis, and developed and authored the supplemental website; J.S.T.
conceived and guided the project, designed and helped organize the
jamboree, helped design post-jamboree data analysis plan, helped post-
jamboree data curation, and wrote the manuscript.
ACKNOWLEDGMENTS
We thank the jamboree participants for their contributions, including data
gathering, annotation, comparison-group formation via OMiCC, as well as
feedback on the manuscript. Participants include (listed alphabetically by
last name): James Austin, Julián Candia, William Coley, Ehren Dancy, Karen
L. Elkins, Sara Faghihi-Kashani, Julio Gomez-Rodriguez, Liliana Guedez, Ma-
ria J. Gutierrez, Trung Ho, Reiko Horai, Sunmee Huh, Chie Iwamura, Jaimy Joy,
Ju-Gyeong Kang, Sunil Kaul, Laura B. Lewandowski, Nathan P. Manes, Mary
J. Mattapallil, Sarfraz Memon, M. Jubayer Rahman, Kameron B. Rodrigues,
Bruno Silva, Amit Singh, Anthony J. St. Leger, Jessica Tang, Hang Xie, Yongge
1200 Immunity 45, December 20, 2016
Immunity
Perspective
Zhao, and Ofer Zimmerman. We also thank Neha Bansal, Candace Liu, and
Abigail Thorpe for assisting with the jamboree; Yongjian Guo and Yong Lu
for OMiCC support; BCBB/OCICB of NIAID for providing computing support
and web hosting; NIH Facilities for providing the OMiCC Jamboree hosting
venue; and members of the J.S.T. lab for discussions. This research was
funded by the Intramural Programs of the National Institute of Allergy and In-
fectious Diseases (NIAID) and the Center for Information Technology (CIT) at
the NIH.
REFERENCES
Adams, M.D., Celniker, S.E., Holt, R.A., Evans, C.A., Gocayne, J.D., Amanati-
des, P.G., Scherer, S.E., Li, P.W., Hoskins, R.A., Galle, R.F., et al. (2000). The
genome sequence of Drosophila melanogaster. Science 287, 2185–2195.
Aghaeepour, N., Finak, G., Hoos, H., Mosmann, T.R., Brinkman, R., Gottardo,
R., and Scheuermann, R.H.; FlowCAP Consortium; DREAM Consortium
(2013). Critical assessment of automated flow cytometry data analysis tech-
niques. Nat. Methods 10, 228–238.
Alizadeh, A.A., Eisen, M.B., Davis, R.E., Ma, C., Lossos, I.S., Rosenwald, A.,
Boldrick, J.C., Sabet, H., Tran, T., Yu, X., et al. (2000). Distinct types of diffuse
large B-cell lymphoma identified by gene expression profiling. Nature 403,
503–511.
Alon, U., Barkai, N., Notterman, D.A., Gish, K., Ybarra, S., Mack, D., and Lev-
ine, A.J. (1999). Broad patterns of gene expression revealed by clustering anal-
ysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc.
Natl. Acad. Sci. USA 96, 6745–6750.
Anderson, C.A., Boucher, G., Lees, C.W., Franke, A., D’Amato, M., Taylor,
K.D., Lee, J.C., Goyette, P., Imielinski, M., Latiano, A., et al. (2011). Meta-anal-
ysis identifies 29 additional ulcerative colitis risk loci, increasing the number of
confirmed associations to 47. Nat. Genet. 43, 246–252.
Andres-Terre, M., McGuire, H.M., Pouliot, Y., Bongen, E., Sweeney, T.E., Tato,
C.M., and Khatri, P. (2015). Integrated, multi-cohort analysis identifies
conserved transcriptional signatures across multiple respiratory viruses. Im-
munity 43, 1199–1211.
Baechler, E.C., Batliwalla, F.M., Karypis, G., Gaffney, P.M., Ortmann, W.A.,
Espe, K.J., Shark, K.B., Grande, W.J., Hughes, K.M., Kapur, V., et al. (2003).
Interferon-inducible gene expression signature in peripheral blood cells of pa-
tients with severe lupus. Proc. Natl. Acad. Sci. USA 100, 2610–2615.
Barrett, T., Wilhite, S.E., Ledoux, P., Evangelista, C., Kim, I.F., Tomashevsky,
M., Marshall, K.A., Phillippy, K.H., Sherman, P.M., Holko, M., et al. (2013).
NCBI GEO: archive for functional genomics data sets–update. Nucleic Acids
Res. 41, D991–D995.
Begum, F., Ghosh, D., Tseng, G.C., and Feingold, E. (2012). Comprehensive
literature review and statistical considerations for GWAS meta-analysis. Nu-
cleic Acids Res. 40, 3777–3784.
Bendall, S.C., Nolan, G.P., Roederer, M., and Chattopadhyay, P.K. (2012).
A deep profiler’s guide to cytometry. Trends Immunol. 33, 323–332.
Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: A
practical and powerful approach to multiple testing. J. R. Stat. Soc. B 57,
289–300.
Bennett, L., Palucka, A.K., Arce, E., Cantrell, V., Borvak, J., Banchereau, J.,
and Pascual, V. (2003). Interferon and granulopoiesis signatures in systemic
lupus erythematosus blood. J. Exp. Med. 197, 711–723.
Bhattacharya, S., Andorf, S., Gomes, L., Dunn, P., Schaefer, H., Pontius, J.,
Berger, P., Desborough, V., Smith, T., Campbell, J., et al. (2014). ImmPort:
disseminating data to the public for the future of immunology. Immunol. Res.
58, 234–239.
Bindea, G., Mlecnik, B., Tosolini, M., Kirilovsky, A., Waldner, M., Obenauf,
A.C., Angell, H., Fredriksen, T., Lafontaine, L., Berger, A., et al. (2013). Spatio-
temporal dynamics of intratumoral immune cells reveal the immune landscape
in human cancer. Immunity 39, 782–795.
Birney, E., Hudson, T.J., Green, E.D., Gunter, C., Eddy, S., Rogers, J., Harris,
J.R., Ehrlich, S.D., Apweiler, R., Austin, C.P., et al.; Toronto International Data
Release Workshop Authors (2009). Prepublication data sharing. Nature 461,
168–170.
Immunity
Perspective
Bittner, M., Meltzer, P., Chen, Y., Jiang, Y., Seftor, E., Hendrix, M., Radmacher,
M., Simon, R., Yakhini, Z., Ben-Dor, A., et al. (2000). Molecular classification of
cutaneous malignant melanoma by gene expression profiling. Nature 406,
536–540.
Bradfield, J.P., Qu, H.Q., Wang, K., Zhang, H., Sleiman, P.M., Kim, C.E.,
Mentch, F.D., Qiu, H., Glessner, J.T., Thomas, K.A., et al. (2011). A genome-
wide meta-analysis of six type 1 diabetes cohorts identifies multiple associ-
ated loci. PLoS Genet. 7, e1002293.
Brazma, A. (2009). Minimum Information About a Microarray Experiment (MI-
AME)–successes, failures, challenges. ScientificWorldJournal 9, 420–423.
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P.,
Stoeckert, C., Aach, J., Ansorge, W., Ball, C.A., Causton, H.C., et al. (2001).
Minimum information about a microarray experiment (MIAME)—Toward stan-
dards for microarray data. Nat. Genet. 29, 365–371.
Brusic, V., Gottardo, R., Kleinstein, S.H., and Davis, M.M.; HIPC steering
committee (2014). Computational resources for high-dimensional immune
analysis from the Human Immunology Project Consortium. Nat. Biotechnol.
32, 146–148.
Califano, A., Butte, A.J., Friend, S., Ideker, T., and Schadt, E. (2012).
Leveraging models of cell regulation and GWAS data in integrative network-
based association studies. Nat. Genet. 44, 841–847.
Campain, A., and Yang, Y.H. (2010). Comparison study of microarray meta-
analysis methods. BMC Bioinformatics 11, 408.
Celi, L.A., Ippolito, A., Montgomery, R.A., Moses, C., and Stone, D.J. (2014).
Crowdsourcing knowledge discovery and innovations in medicine. J. Med.
Internet Res. 16, e216.
Chang, L.C., Lin, H.M., Sibille, E., and Tseng, G.C. (2013). Meta-analysis
methods for combining multiple expression profiles: comparisons, statistical
characterization and an application guideline. BMC Bioinformatics 14, 368.
Chaussabel, D., and Baldwin, N. (2014). Democratizing systems immunology
with modular transcriptional repertoire analyses. Nat. Rev. Immunol. 14,
271–280.
Chaussabel, D., Quinn, C., Shen, J., Patel, P., Glaser, C., Baldwin, N., Stich-
weh, D., Blankenship, D., Li, L., Munagala, I., et al. (2008). A modular analysis
framework for blood genomics studies: application to systemic lupus erythe-
matosus. Immunity 29, 150–164.
Chee, M., Yang, R., Hubbell, E., Berno, A., Huang, X.C., Stern, D., Winkler, J.,
Lockhart, D.J., Morris, M.S., and Fodor, S.P. (1996). Accessing genetic infor-
mation with high-density DNA arrays. Science 274, 610–614.
Chen, B., and Butte, A.J. (2016). Leveraging big data to transform target selec-
tion and drug discovery. Clin. Pharmacol. Ther. 99, 285–297.
Chen, J., Bardes, E.E., Aronow, B.J., and Jegga, A.G. (2009). ToppGene Suite
for gene list enrichment analysis and candidate gene prioritization. Nucleic
Acids Res. 37, W305–W311.
Coletta, A., Molter, C., Duqué, R., Steenhoff, D., Taminau, J., de Schaetzen, V.,
Meganck, S., Lazar, C., Venet, D., Detours, V., et al. (2012). InSilico DB
genomic datasets hub: an efficient starting point for analyzing genome-wide
studies in GenePattern, Integrative Genomics Viewer, and R/Bioconductor.
Genome Biol. 13, R104.
Collins, F.S., Morgan, M., and Patrinos, A. (2003). The Human Genome Project:
Lessons from large-scale biology. Science 300, 286–290.
Cotsapas, C., Voight, B.F., Rossin, E., Lage, K., Neale, B.M., Wallace, C., Abe-
casis, G.R., Barrett, J.C., Behrens, T., Cho, J., et al.; FOCiS Network of Con-
sortia (2011). Pervasive sharing of genetic effects in autoimmune disease.
PLoS Genet. 7, e1002254.
Darmanis, S., Gallant, C.J., Marinescu, V.D., Niklasson, M., Segerman, A., Fla-
mourakis, G., Fredriksson, S., Assarsson, E., Lundberg, M., Nelander, S., et al.
(2016). Simultaneous multiplexed measurement of RNA and proteins in single
cells. Cell Rep. 14, 380–389.
Dolinski, K., and Troyanskaya, O.G. (2015). Implications of Big Data for cell
biology. Mol. Biol. Cell 26, 2575–2578.
Doshi, P., Goodman, S.N., and Ioannidis, J.P. (2013). Raw data from clinical
trials: Within reach? Trends Pharmacol. Sci. 34, 645–647.
Evangelou, E., and Ioannidis, J.P. (2013). Meta-analysis methods for genome-
wide association studies and beyond. Nat. Rev. Genet. 14, 379–389.
Fabregat, A., Sidiropoulos, K., Garapati, P., Gillespie, M., Hausmann, K., Haw,
R., Jassal, B., Jupe, S., Korninger, F., McKay, S., et al. (2016). The Reactome
pathway Knowledgebase. Nucleic Acids Res. 44 (D1), D481–D487.
Fehrmann, R.S., Karjalainen, J.M., Krajewska, M., Westra, H.J., Maloney, D.,
Simeonov, A., Pers, T.H., Hirschhorn, J.N., Jansen, R.C., Schultes, E.A.,
et al. (2015). Gene expression analysis identifies global gene dosage sensitivity
in cancer. Nat. Genet. 47, 115–125.
Finak, G., Langweiler, M., Jaimes, M., Malek, M., Taghiyar, J., Korin, Y.,
Raddassi, K., Devine, L., Obermoser, G., Pekalski, M.L., et al. (2016). Stan-
dardizing flow cytometryImmunophenotyping analysis from the Human Immu-
noPhenotyping Consortium. Sci. Rep. 6, 20686.
Franke, A., McGovern, D.P., Barrett, J.C., Wang, K., Radford-Smith, G.L.,
Ahmad, T., Lees, C.W., Balschun, T., Lee, J., Roberts, R., et al. (2010).
Genome-wide meta-analysis increases to 71 the number of confirmed Crohn’s
disease susceptibility loci. Nat. Genet. 42, 1118–1125.
Fuchsberger, C., Flannick, J., Teslovich, T.M., Mahajan, A., Agarwala, V.,
Gaulton, K.J., Ma, C., Fontanillas, P., Moutsianas, L., McCarthy, D.J., et al.
(2016). The genetic architecture of type 2 diabetes. Nature 536, 41–47.
Gasch, A.P., Spellman, P.T., Kao, C.M., Carmel-Harel, O., Eisen, M.B., Storz,
G., Botstein, D., and Brown, P.O. (2000). Genomic expression programs in the
response of yeast cells to environmental changes. Mol. Biol. Cell 11, 4241–
4257.
Gentles, A.J., Newman, A.M., Liu, C.L., Bratman, S.V., Feng, W., Kim, D., Nair,
V.S., Xu, Y., Khuong, A., Hoang, C.D., et al. (2015). The prognostic landscape
of genes and infiltrating immune cells across human cancers. Nat. Med. 21,
938–945.
Germain, R.N., Meier-Schellersheim, M., Nita-Lazar, A., and Fraser, I.D.
(2011). Systems biology in immunology: A computational modeling perspec-
tive. Annu. Rev. Immunol. 29, 527–585.
Gibson, G., Powell, J.E., and Marigorta, U.M. (2015). Expression quantitative
trait locus analysis for translational medicine. Genome Med. 7, 60.
Golub, T.R., Slonim, D.K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov,
J.P., Coller, H., Loh, M.L., Downing, J.R., Caligiuri, M.A., et al. (1999). Molec-
ular classification of cancer: Class discovery and class prediction by gene
expression monitoring. Science 286, 531–537.
Goodman, S.N., Fanelli, D., and Ioannidis, J.P. (2016). What does research
reproducibility mean? Sci. Transl. Med. 8, 341ps12.
Gorenshteyn, D., Zaslavsky, E., Fribourg, M., Park, C.Y., Wong, A.K., Tadych,
A., Hartmann, B.M., Albrecht, R.A., Garcı́a-Sastre, A., Kleinstein, S.H., et al.
(2015). Interactive Big Data resource to elucidate human immune pathways
and diseases. Immunity 43, 605–614.
Gross, A.M., Kreisberg, J.F., and Ideker, T. (2015). Analysis of matched tumor
and normal profiles reveals common transcriptional and epigenetic signals
shared across cancer types. PLoS ONE 10, e0142618.
Grover, M.P., Ballouz, S., Mohanasundaram, K.A., George, R.A., Goscinski,
A., Crowley, T.M., Sherman, C.D., and Wouters, M.A. (2015). Novel therapeu-
tics for coronary artery disease from genome-wide association study data.
BMC Med. Genomics 8 (Suppl 2 ), S1.
GTEx Consortium (2013). The Genotype-Tissue Expression (GTEx) project.
Nat. Genet. 45, 580–585.
Haynes, W.A., Vallania, F., Liu, C., Bongen, E., Tomczak, A., Andres-Terrè, M.,
Lofgren, S., Tam, A., Deisseroth, C.A., Li, M.D., et al. (2016). Empowering
multi-cohort gene expression analysis to increase reproducibility. Pac.
Symp. Biocomput. 22, 144–153.
Heng, T.S., and Painter, M.W.; Immunological Genome Project Consortium
(2008). The Immunological Genome Project: Networks of gene expression in
immune cells. Nat. Immunol. 9, 1091–1094.
Higgs, B.W., Zhu, W., Richman, L., Fiorentino, D.F., Greenberg, S.A., Jallal, B.,
and Yao, Y. (2012). Identification of activated cytokine pathways in the blood of
systemic lupus erythematosus, myositis, rheumatoid arthritis, and sclero-
derma patients. Int. J. Rheum. Dis. 15, 25–35.
Immunity 45, December 20, 2016 1201

Hoadley, K.A., Yau, C., Wolf, D.M., Cherniack, A.D., Tamborero, D., Ng, S.,
Leiserson, M.D., Niu, B., McLellan, M.D., Uzunangelov, V., et al.; Cancer
Genome Atlas Research Network (2014). Multiplatform analysis of 12 cancer
types reveals molecular classification within and across tissues of origin.
Cell 158, 929–944.
Huan, T., Esko, T., Peters, M.J., Pilling, L.C., Schramm, K., Schurmann, C.,
Chen, B.H., Liu, C., Joehanes, R., Johnson, A.D., et al.; International
Consortium for Blood Pressure GWAS (ICBP) (2015). A meta-analysis of
gene expression signatures of blood pressure and hypertension. PLoS Genet.
11, e1005035.
Hudson, T.J., Anderson, W., Artez, A., Barker, A.D., Bell, C., Bernabé, R.R.,
Bhan, M.K., Calvo, F., Eerola, I., Gerhard, D.S., et al.; International Cancer
Genome Consortium (2010). International network of cancer genome projects.
Nature 464, 993–998.
Hughes, T.R., Marton, M.J., Jones, A.R., Roberts, C.J., Stoughton, R., Armour,
C.D., Bennett, H.A., Coffey, E., Dai, H., He, Y.D., et al. (2000). Functional dis-
covery via a compendium of expression profiles. Cell 102, 109–126.
Hurle, M.R., Yang, L., Xie, Q., Rajpal, D.K., Sanseau, P., and Agarwal, P.
(2013). Computational drug repositioning: From data to therapeutics. Clin.
Pharmacol. Ther. 93, 335–341.
Ioannidis, J.P., Allison, D.B., Ball, C.A., Coulibaly, I., Cui, X., Culhane, A.C., Fal-
chi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009). Repeatability of pub-
lished microarray gene expression analyses. Nat. Genet. 41, 149–155.
Iorio, F., Rittman, T., Ge, H., Menden, M., and Saez-Rodriguez, J. (2013). Tran-
scriptional data: A new gateway to drug repositioning? Drug Discov. Today 18,
350–357.
Jenner, R.G., and Young, R.A. (2005). Insights into host responses against
pathogens from transcriptional profiling. Nat. Rev. Microbiol. 3, 281–294.
Jujjavarapu, C., Hughey, J., Gheradini, F., Bruggner, R., Nolan, G., Bhatta-
charya, S., and Butte, A.J. (2016). A Framework for Meta-Analysis of Cytome-
try Data. Journal of Immunology 196, 69.16.

ski, P., Frydecka, D., Sa˛siadek, M.M., and Misiak, B. (2016). Reduced
Karpin
number of peripheral natural killer cells in schizophrenia but not in bipolar dis-
order. Brain Behav. Immun. 54, 194–200.
Kaye, J., Heeney, C., Hawkins, N., de Vries, J., and Boddington, P. (2009). Data
sharing in genomics—Re-shaping scientific practice. Nat. Rev. Genet. 10,
331–335.
Khatri, P., Roedder, S., Kimura, N., De Vusser, K., Morgan, A.A., Gong, Y.,
Fischbein, M.P., Robbins, R.C., Naesens, M., Butte, A.J., and Sarwal, M.M.
(2013). A common rejection module (CRM) for acute rejection across multiple
organs identifies novel therapeutics for organ transplantation. J. Exp. Med.
210, 2205–2221.
Kidd, B.A., Wroblewska, A., Boland, M.R., Agudo, J., Merad, M., Tatonetti,
N.P., Brown, B.D., and Dudley, J.T. (2016). Mapping the effects of drugs on
the immune system. Nat. Biotechnol. 34, 47–54.
Klingström, T., and Plewczynski, D. (2011). Protein-protein interaction and
pathway databases, a graphical review. Brief. Bioinform. 12, 702–713.
Kolesnikov, N., Hastings, E., Keays, M., Melnichuk, O., Tang, Y.A., Williams,
E., Dylag, M., Kurbatova, N., Brandizi, M., Burdett, T., et al. (2015).
ArrayExpress update—Simplifying data submissions. Nucleic Acids Res. 43,
D1113–D1116.
Kupershmidt, I., Su, Q.J., Grewal, A., Sundaresh, S., Halperin, I., Flynn, J.,
Shekar, M., Wang, H., Park, J., Cui, W., et al. (2010). Ontology-based meta-
analysis of global collections of high-throughput public data. PLoS ONE 5,
e13066.
Lage, K. (2014). Protein-protein interactions and genetic diseases: The interac-
tome. Biochim. Biophys. Acta 1842, 1971–1980.
Laird, P.W. (2010). Principles and challenges of genomewide DNA methylation
analysis. Nat. Rev. Genet. 11, 191–203.
Lamb, J. (2007). The Connectivity Map: A new tool for biomedical research.
Nat. Rev. Cancer 7, 54–60.
Lamb, J., Crawford, E.D., Peck, D., Modell, J.W., Blat, I.C., Wrobel, M.J.,
Lerner, J., Brunet, J.P., Subramanian, A., Ross, K.N., et al. (2006). The Con-
1202 Immunity 45, December 20, 2016
Immunity
Perspective
nectivity Map: using gene-expression signatures to connect small molecules,
genes, and disease. Science 313, 1929–1935.
Larance, M., and Lamond, A.I. (2015). Multidimensional proteomics for cell
biology. Nat Rev Mol Cell Biol 16, 269–280.
Lau, W.W., Sparks, R., OMiCC Jamboree Working Group, and Tsang, J.S.
(2016). Meta-analysis of crowdsourced data compendia suggests pan-dis-
ease transcriptional signatures of autoimmunity. F1000Research http://dx.
doi.org/10.12688/f1000research.10465.1.
Li, S., Rouphael, N., Duraisingham, S., Romero-Steiner, S., Presnell, S., Davis,
C., Schmidt, D.S., Johnson, S.E., Milton, A., Rajam, G., et al. (2014). Molecular
signatures of antibody responses derived from a systems biology study of five
human vaccines. Nat. Immunol. 15, 195–204.
Li, B., Li, T., Pignon, J.C., Wang, B., Wang, J., Shukla, S.A., Dou, R., Chen, Q.,
Hodi, F.S., Choueiri, T.K., et al. (2016). Landscape of tumor-infiltrating T cell
repertoire of human cancers. Nat. Genet. 48, 725–732.
Libbrecht, M.W., and Noble, W.S. (2015). Machine learning applications in ge-
netics and genomics. Nat. Rev. Genet. 16, 321–332.
Liberzon, A., Subramanian, A., Pinchback, R., Thorvaldsdóttir, H., Tamayo, P.,
and Mesirov, J.P. (2011). Molecular signatures database (MSigDB) 3.0. Bioin-
formatics 27, 1739–1740.
Lu, Y., Biancotto, A., Cheung, F., Remmers, E., Shah, N., McCoy, J.P., and
Tsang, J.S. (2016). Systematic analysis of cell-to-cell expression variation of
T lymphocytes in a human cohort identifies aging and genetic associations.
Immunity 45, 1162–1175.
€, J., Parkinson, H., Goncalves, A., Huber, W.,
Lukk, M., Kapushesky, M., Nikkila
Ukkonen, E., and Brazma, A. (2010). A global map of human gene expression.
Nat. Biotechnol. 28, 322–324.
Márquez, A., Vidal-Bralo, L., Rodrı́guez-Rodrı́guez, L., González-Gay, M.A.,
Balsa, A., González-Álvaro, I., Carreira, P., Ortego-Centeno, N., Ayala-Gutiér-
rez, M.M., Garcı́a-Hernández, F.J., et al. (2016). A combined large-scale meta-
analysis identifies COG6 as a novel shared risk locus for rheumatoid arthritis
and systemic lupus erythematosus. Ann. Rheum. Dis. Published online May
18, 2016.
Morris, D.L., Sheng, Y., Zhang, Y., Wang, Y.F., Zhu, Z., Tombleson, P., Chen,
L., Cunninghame Graham, D.S., Bentham, J., Roberts, A.L., et al. (2016).
Genome-wide association meta-analysis in Chinese and European individuals
identifies ten new loci associated with systemic lupus erythematosus. Nat.
Genet. 48, 940–946.
Musen, M.A., Noy, N.F., Shah, N.H., Whetzel, P.L., Chute, C.G., Story, M.A.,
and Smith, B.; NCBO team (2012). The National Center for Biomedical
Ontology. J. Am. Med. Inform. Assoc. 19, 190–195.
Nelson, M.R., Tipney, H., Painter, J.L., Shen, J., Nicoletti, P., Shen, Y., Flora-
tos, A., Sham, P.C., Li, M.J., Wang, J., et al. (2015). The support of human ge-
netic evidence for approved drug indications. Nat. Genet. 47, 856–860.
Newman, A.M., Liu, C.L., Green, M.R., Gentles, A.J., Feng, W., Xu, Y., Hoang,
C.D., Diehn, M., and Alizadeh, A.A. (2015). Robust enumeration of cell subsets
from tissue expression profiles. Nat. Methods 12, 453–457.
Okada, Y., Wu, D., Trynka, G., Raj, T., Terao, C., Ikari, K., Kochi, Y., Ohmura,
K., Suzuki, A., Yoshida, S., et al.; RACI consortium; GARNET consortium
(2014). Genetics of rheumatoid arthritis contributes to biology and drug dis-
covery. Nature 506, 376–381.
Olsen, J.V., and Mann, M. (2013). Status of large-scale analysis of post-trans-
lational modifications by mass spectrometry. Mol. Cell. Proteomics 12, 3444–
3452.
Orchard, S., Kerrien, S., Abbani, S., Aranda, B., Bhate, J., Bidwell, S., Bridge,
A., Briganti, L., Brinkman, F.S., Cesareni, G., et al. (2012). Protein interaction
data curation: the International Molecular Exchange (IMEx) consortium. Nat.
Methods 9, 345–350.
Paltoo, D.N., Rodriguez, L.L., Feolo, M., Gillanders, E., Ramos, E.M., Rutter,
J.L., Sherry, S., Wang, V.O., Bailey, A., Baker, R., et al.; National Institutes of
Health Genomic Data Sharing Governance Committees (2014). Data use under
the NIH GWAS data sharing policy and future directions. Nat. Genet. 46,
934–938.
Patsopoulos, N.A., Esposito, F., Reischl, J., Lehr, S., Bauer, D., Heubach, J.,
Sandbrink, R., Pohl, C., Edan, G., Kappos, L., et al.; Bayer Pharma MS
Immunity
Perspective
Genetics Working Group; Steering Committees of Studies Evaluating IFNb-1b
and a CCR1-Antagonist; ANZgene Consortium; GeneMSA; International
Multiple Sclerosis Genetics Consortium (2011). Genome-wide meta-analysis
identifies novel multiple sclerosis susceptibility loci. Ann. Neurol. 70, 897–912.
Pe’er, D., Regev, A., Elidan, G., and Friedman, N. (2001). Inferring subnetworks
from perturbed expression profiles. Bioinformatics 17 (Suppl 1 ), S215–S224.
Pennisi, E. (2000). Ideas fly at gene-finding jamboree. Science 287, 2182–
2184.
Perou, C.M., Sørlie, T., Eisen, M.B., van de Rijn, M., Jeffrey, S.S., Rees, C.A.,
Pollack, J.R., Ross, D.T., Johnsen, H., Akslen, L.A., et al. (2000). Molecular
portraits of human breast tumours. Nature 406, 747–752.
Pop, M., and Salzberg, S.L. (2015). Use and mis-use of supplementary mate-
rial in science publications. BMC Bioinformatics 16, 237.
Ramasamy, A., Mondry, A., Holmes, C.C., and Altman, D.G. (2008). Key issues
in conducting a meta-analysis of gene expression microarray datasets. PLoS
Med. 5, e184.
Raychaudhuri, S., Remmers, E.F., Lee, A.T., Hackett, R., Guiducci, C., Burtt,
N.P., Gianniny, L., Korman, B.D., Padyukov, L., Kurreeman, F.A., et al.
(2008). Common variants at CD40 and other loci confer risk of rheumatoid
arthritis. Nat. Genet. 40, 1216–1223.
Rhodes, D.R., Barrette, T.R., Rubin, M.A., Ghosh, D., and Chinnaiyan, A.M.
(2002). Meta-analysis of microarrays: interstudy validation of gene expression
profiles reveals pathway dysregulation in prostate cancer. Cancer Res. 62,
4427–4433.
Rhodes, D.R., Yu, J., Shanker, K., Deshpande, N., Varambally, R., Ghosh, D.,
Barrette, T., Pandey, A., and Chinnaiyan, A.M. (2004). Large-scale meta-anal-
ysis of cancer microarray data identifies common transcriptional profiles of
neoplastic transformation and progression. Proc. Natl. Acad. Sci. USA 101,
9309–9314.
Roedder, S., Kimura, N., Okamura, H., Hsieh, S.C., Gong, Y., and Sarwal, M.M.
(2013). Significance and suppression of redundant IL17 responses in acute
allograft rejection by bioinformatics based drug repositioning of fenofibrate.
PLoS ONE 8, e56657.
Roederer, M., Quaye, L., Mangino, M., Beddall, M.H., Mahnke, Y., Chattopad-
hyay, P., Tosi, I., Napolitano, L., Terranova Barberio, M., Menni, C., et al.
(2015). The genetic architecture of the human immune system: A bioresource
for autoimmunity and disease pathogenesis. Cell 161, 387–403.
Rolland, T., Taşan, M., Charloteaux, B., Pevzner, S.J., Zhong, Q., Sahni, N., Yi,
S., Lemmens, I., Fontanillo, C., Mosca, R., et al. (2014). A proteome-scale map
of the human interactome network. Cell 159, 1212–1226.
Rooney, M.S., Shukla, S.A., Wu, C.J., Getz, G., and Hacohen, N. (2015). Mo-
lecular and genetic properties of tumors associated with local immune cyto-
lytic activity. Cell 160, 48–61.
Ross, J.S., and Krumholz, H.M. (2013). Ushering in a new era of open science
through data sharing: the wall must come down. JAMA 309, 1355–1356.
Rossin, E.J., Lage, K., Raychaudhuri, S., Xavier, R.J., Tatar, D., Benita, Y., Cot-
sapas, C., and Daly, M.J.; International Inflammatory Bowel Disease Genetics
Constortium (2011). Proteins encoded in genomic regions associated with im-
mune-mediated disease physically interact and suggest underlying biology.
PLoS Genet. 7, e1001273.
Ruepp, A., Waegele, B., Lechner, M., Brauner, B., Dunger-Kaltenbach, I.,
Fobo, G., Frishman, G., Montrone, C., and Mewes, H.W. (2010). CORUM:
the comprehensive resource of mammalian protein complexes–2009. Nucleic
Acids Res 38, D497–D501.
Rung, J., and Brazma, A. (2013). Reuse of public genome-wide gene expres-
sion data. Nat. Rev. Genet. 14, 89–99.
Saeys, Y., Gassen, S.V., and Lambrecht, B.N. (2016). Computational flow cy-
tometry: Helping to make sense of high-dimensional immunology data. Nat.
Rev. Immunol. 16, 449–462.
Saez-Rodriguez, J., Costello, J.C., Friend, S.H., Kellen, M.R., Mangravite, L.,
Meyer, P., Norman, T., and Stolovitzky, G. (2016). Crowdsourcing biomedical
research: Leveraging communities as innovation engines. Nat. Rev. Genet. 17,
470–486.
Sanseau, P., Agarwal, P., Barnes, M.R., Pastinen, T., Richards, J.B., Cardon,
L.R., and Mooser, V. (2012). Use of genome-wide association studies for drug
repositioning. Nat. Biotechnol. 30, 317–320.
Satija, R., and Shalek, A.K. (2014). Heterogeneity in immune responses: From
populations to single cells. Trends Immunol. 35, 219–229.
Schena, M., Shalon, D., Davis, R.W., and Brown, P.O. (1995). Quantitative
monitoring of gene expression patterns with a complementary DNA microar-
ray. Science 270, 467–470.
Schwartzman, O., and Tanay, A. (2015). Single-cell epigenomics: Techniques
and emerging applications. Nat. Rev. Genet. 16, 716–726.
Segal, E., Shapira, M., Regev, A., Pe’er, D., Botstein, D., Koller, D., and Fried-
man, N. (2003a). Module networks: Identifying regulatory modules and their
condition-specific regulators from gene expression data. Nat. Genet. 34,
166–176.
Segal, E., Yelensky, R., and Koller, D. (2003b). Genome-wide discovery of
transcriptional modules from DNA sequence and gene expression. Bioinfor-
matics 19 (Suppl 1 ), i273–i282.
Segal, E., Friedman, N., Koller, D., and Regev, A. (2004). A module map
showing conditional activity of expression modules in cancer. Nat. Genet.
36, 1090–1098.
Segal, E., Friedman, N., Kaminski, N., Regev, A., and Koller, D. (2005). From
signatures to models: Understanding cancer using microarrays. Nat. Genet.
37 (Suppl ), S38–S45.
Shah, N., Guo, Y., Wendelsdorf, K.V., Lu, Y., Sparks, R., and Tsang, J.S.
(2016). A crowdsourcing approach for reusing and meta-analyzing gene
expression data. Nat. Biotechnol. 34, 803–806.
Shapiro, E., Biezuner, T., and Linnarsson, S. (2013). Single-cell sequencing-
based technologies will revolutionize whole-organism science. Nat. Rev.
Genet. 14, 618–630.
Silberzahn, R., and Uhlmann, E.L. (2015). Crowdsourced research: Many
hands make tight work. Nature 526, 189–191.
Sirota, M., Dudley, J.T., Kim, J., Chiang, A.P., Morgan, A.A., Sweet-Cordero,
A., Sage, J., and Butte, A.J. (2011). Discovery and preclinical validation of
drug indications using compendia of public gene expression data. Sci. Transl.
Med. 3, 96ra77.
Speake, C., Presnell, S., Domico, K., Zeitner, B., Bjork, A., Anderson, D., Ma-
son, M.J., Whalen, E., Vargas, O., Popov, D., et al. (2015). An interactive web
application for the dissemination of human systems immunology data. Journal
of translational medicine 13, 196.
Spellman, P.T., Sherlock, G., Zhang, M.Q., Iyer, V.R., Anders, K., Eisen, M.B.,
Brown, P.O., Botstein, D., and Futcher, B. (1998). Comprehensive identifica-
tion of cell cycle-regulated genes of the yeast Saccharomyces cerevisiae by
microarray hybridization. Mol. Biol. Cell 9, 3273–3297.
Spidlen, J., Breuer, K., Rosenberg, C., Kotecha, N., and Brinkman, R.R. (2012).
FlowRepository: A resource of annotated flow cytometry datasets associated
with peer-reviewed publications. Cytometry A 81, 727–731.
Stahl, E.A., Raychaudhuri, S., Remmers, E.F., Xie, G., Eyre, S., Thomson, B.P.,
Li, Y., Kurreeman, F.A., Zhernakova, A., Hinks, A., et al.; BIRAC Consortium;
YEAR Consortium (2010). Genome-wide association study meta-analysis
identifies seven new rheumatoid arthritis risk loci. Nat. Genet. 42, 508–514.
Subramanian, A., Tamayo, P., Mootha, V.K., Mukherjee, S., Ebert, B.L., Gil-
lette, M.A., Paulovich, A., Pomeroy, S.L., Golub, T.R., Lander, E.S., and Me-
sirov, J.P. (2005). Gene set enrichment analysis: A knowledge-based
approach for interpreting genome-wide expression profiles. Proc. Natl.
Acad. Sci. USA 102, 15545–15550.
Sweeney, T.E., Shidham, A., Wong, H.R., and Khatri, P. (2015). A comprehen-
sive time-course-based multicohort analysis of sepsis and sterile inflammation
reveals a robust diagnostic gene set. Sci. Transl. Med. 7, 287ra71.
Sweeney, T.E., Haynes, W.A., Vallania, F., Ioannidis, J.P., and Khatri, P. (2016).
Methods to increase reproducibility in differential gene expression via meta-
analysis. Nucleic Acids Res. gkw797, Published online September 14, 2016.
http://dx.doi.org/10.1093/nar/gkw797.
Szklarczyk, D., Franceschini, A., Wyder, S., Forslund, K., Heller, D., Huerta-
Cepas, J., Simonovic, M., Roth, A., Santos, A., Tsafou, K.P., et al. (2015).
Immunity 45, December 20, 2016 1203

STRING v10: protein-protein interaction networks, integrated over the tree of
life. Nucleic Acids Res. 43, D447–D452.
The 1000 Genomes Project Consortium (2015). A global reference for human
genetic variation. Nature 526, 68–74.
Tomczak, K., Czerwin
ska, P., and Wiznerowicz, M. (2015). The Cancer
Genome Atlas (TCGA): An immeasurable source of knowledge. Contemp. On-
col. (Pozn.) 19 (1A), A68–A77.
Toro-Domı́nguez, D., Carmona-Sáez, P., and Alarcón-Riquelme, M.E. (2014).
Shared signatures between rheumatoid arthritis, systemic lupus erythemato-
sus and Sjögren’s syndrome uncovered through gene expression meta-anal-
ysis. Arthritis Res. Ther. 16, 489.
Torrente, A., Lukk, M., Xue, V., Parkinson, H., Rung, J., and Brazma, A. (2016).
Identification of cancer related genes using a comprehensive map of human
gene expression. PLoS ONE 11, e0157484.
Tsalik, E.L., Henao, R., Nichols, M., Burke, T., Ko, E.R., McClain, M.T., Hud-
son, L.L., Mazur, A., Freeman, D.H., Veldman, T., et al. (2016). Host gene
expression classifiers diagnose acute respiratory illness etiology. Sci. Transl.
Med. 8, 322ra11.
Tseng, G.C., Ghosh, D., and Feingold, E. (2012). Comprehensive literature re-
view and statistical considerations for microarray meta-analysis. Nucleic Acids
Res. 40, 3785–3799.
Wang, T., Hopkins, D., Schmidt, C., Silva, S., Houghton, R., Takita, H.,
Repasky, E., and Reed, S.G. (2000). Identification of genes differentially
over-expressed in lung squamous cell carcinoma using combination of
cDNA subtraction and microarray analysis. Oncogene 19, 1519–1528.
Wang, K., Li, M., and Hakonarson, H. (2010). Analysing biological pathways in
genome-wide association studies. Nat. Rev. Genet. 11, 843–854.
1204 Immunity 45, December 20, 2016
Immunity
Perspective
Wang, X., Kang, D.D., Shen, K., Song, C., Lu, S., Chang, L.C., Liao, S.G., Huo,
Z., Tang, S., Ding, Y., et al. (2012). An R package suite for microarray meta-
analysis in quality control, differentially expressed gene analysis and pathway
enrichment detection. Bioinformatics 28, 2534–2536.
Weinstein, J.N., Collisson, E.A., Mills, G.B., Shaw, K.R., Ozenberger, B.A.,
Ellrott, K., Shmulevich, I., Sander, C., and Stuart, J.M.; Cancer Genome Atlas
Research Network (2013). The Cancer Genome Atlas Pan-Cancer Analysis
Project. Nat. Genet. 45, 1113–1120.
Westra, H.J., Peters, M.J., Esko, T., Yaghootkar, H., Schurmann, C., Kettunen,
J., Christiansen, M.W., Fairfax, B.P., Schramm, K., Powell, J.E., et al. (2013).
Systematic identification of trans eQTLs as putative drivers of known disease
associations. Nat. Genet. 45, 1238–1243.
Wirapati, P., Sotiriou, C., Kunkel, S., Farmer, P., Pradervand, S., Haibe-Kains,
€tz, F., et al. (2008). Meta-
B., Desmedt, C., Ignatiadis, M., Sengstag, T., Schu
analysis of gene expression profiles in breast cancer: toward a unified under-
standing of breast cancer subtyping and prognosis signatures. Breast Cancer
Res. 10, R65.
Zhou, V.W., Goren, A., and Bernstein, B.E. (2011). Charting histone modifica-
tions and the functional organization of mammalian genomes. Nat. Rev. Genet.
12, 7–18.
Zhu, Q., Wong, A.K., Krishnan, A., Aure, M.R., Tadych, A., Zhang, R., Corney,
D.C., Greene, C.S., Bongo, L.A., Kristensen, V.N., et al. (2015). Targeted explo-
ration and analysis of large cross-platform human transcriptomic compendia.
Nat. Methods 12, 211–214.
Zhu, Z., Zhang, F., Hu, H., Bakshi, A., Robinson, M.R., Powell, J.E., Montgom-
ery, G.W., Goddard, M.E., Wray, N.R., Visscher, P.M., and Yang, J. (2016).
Integration of summary data from GWAS and eQTL studies predicts complex
trait gene targets. Nat. Genet. 48, 481–487.