Analytical and Bioanalytical Chemistry (2019) 411:6791–8000

https://doi.org/10.1007/s00216-019-02074-9

FEATURE ARTICLE

Challenges of big data integration in the life sciences

Sven Fillinger 1 & Luis de la Garza 1 & Alexander Peltzer 1 & Oliver Kohlbacher 2,3,4,5 & Sven Nahnsen 1

Received: 21 May 2019 / Revised: 8 July 2019 / Accepted: 6 August 2019 / Published online: 28 August 2019
# Springer-Verlag GmbH Germany, part of Springer Nature 2019

Abstract
Big data has been reported to be revolutionizing many areas of life, including science. It summarizes data that is unprecedentedly
large, rapidly generated, heterogeneous, and hard to accurately interpret. This availability has also brought new challenges: How
to properly annotate data to make it searchable? What are the legal and ethical hurdles when sharing data? How to store data
securely, preventing loss and corruption? The life sciences are not the only disciplines that must align themselves with big data
requirements to keep up with the latest developments. The large hadron collider, for instance, generates research data at a pace
beyond any current biomedical research center. There are three recent major coinciding events that explain the emergence of big
data in the context of research: the technological revolution for data generation, the development of tools for data analysis, and a
conceptual change towards open science and data. The true potential of big data lies in pattern discovery in large datasets, as well
as the formulation of new models and hypotheses. Confirmation of the existence of the Higgs boson, for instance, is one of the
most recent triumphs of big data analysis in physics. Digital representations of biological systems have become more compre-
hensive. This, in combination with advances in machine learning, creates exciting new research possibilities. In this paper, we
review the state of big data in bioanalytical research and provide an overview of the guidelines for its proper usage.

Keywords Big data . Bioanalytics . Data integration . Bioinformatics . Scalability

Introduction construction of efficient query interfaces enable entirely new

In 2004, Google launched the ambitious goal to digitize as
many books as publicly available and legally feasible [1, 2].
The transformation of printed pages to digital images
(digitization) was followed by the identification and indexing
of the text blocks (datafication). The comprehensive annota-
tion of these information blocks with metadata and the
Sven Fillinger, Luis de la Garza, and Alexander Peltzer contributed
equally to this work.
* Sven Nahnsen
sven.nahnsen@uni-tuebingen.de
1
Quantitative Biology Center (QBiC), University of Tübingen, Auf
der Morgenstelle 10, 72076 Tübingen, Germany
2
Center for Bioinformatics, University of Tübingen,
72076 Tübingen, Germany
3
Applied Bioinformatics, Department of Computer Science,
72076 Tübingen, Germany
4
Institute for Translational Bioinformatics, University Hospital of
Tübingen, 71016 Tübingen, Germany
5
Biomolecular Interactions, Max Planck Institute for Developmental
Biology, 72076 Tübingen, Germany
research opportunities. Correlative analyses combine hetero-
geneous information and can be performed on big datasets
that grow rapidly.
In the life sciences, similar concepts for digitizing biolog-
ical material, e.g., through sequencing and datafying the
information with bioinformatic tools, have emerged over
the last two decades, paving the way for big data in life
science research. While big data growth rates have been
suggested as early as the 1940s [3], the life sciences have
become important players in the field for about the last
10 years. The widely accepted definition is based on the
four Vs, requiring data to be very large (volume), but also
heterogeneous (variety), rapidly generated (velocity) and as-
sociated with uncertainty (veracity). A part of this veracity
issue in the bioanalytical disciplines arises from the uncer-
tainty that the measured effect is truly an effect and not
simply a measurement error. Thus, it should be noted that
high-throughput data coming from a single source are not
categorized as big data, regardless of the actual size, nor are
data that allow for straightforward and accurate conclusions
[4]. Similarly, datasets of low complexity and/or low gener-
ation rates, but of high volumes are not big data and will
simply be referred to as large data.
6792
While the most recent research data in the biological and
medical sciences show the alignment with big data properties,
related science disciplines have been associated with big data
much earlier, such as experimental particle physics. The large
hadron collider, as installed at the CERN in Geneva,
Switzerland, is generating primary research data at a pace
beyond any current biomedical research center [5].
Nonetheless, other disciplines such as environmental re-
search [6], medicine (Cancer Genome Atlas Research
Network ...), and life sciences (The GTEx Consortium 2015)
are largely benefiting from the model, which was implement-
ed by other disciplines for coping with big data.
Synoptically, the emergence of big data in the biomedical
field can be attributed to three major coinciding events: (1) the
technological revolution for data generation, (2) the develop-
ment of tools for data analysis, and (3) a conceptual change in
science practice towards open data.
Data generation
The technological innovation is visible in almost all fields
where bioanalytical data is generated. While genomics and
other next-generation sequencing–based technologies have
been leading the field with respect to data amounts and the
speed of their generation, other bioanalytical platforms such as
mass spectrometry and imaging are catching up. In recent
years, technological developments in mass spectrometers
and liquid chromatography have led to a quantum leap in
the quality and quantity of protein and metabolite measure-
ments. This trend can clearly be identified using publicly
available proteomics data and studies, as visualized in
Fig. 1. Since 2005, we observe an exponential growth of pub-
licly available proteomics data.
Fig. 1 Development of registered projects and repository size of the PRIDE database from 2005 until October 2018. Blue line: repository size in
terabytes. Cyan line: total number of registered projects (source: https://doi.org/10.5281/zenodo.1464136)
Fillinger S. et al.
Data analysis
The important developments in scientific disciplines such as
bioinformatics and computational biology are akin to the tech-
nological development for data generation.
The emergence of high-throughput technologies fostered
bioinformatics tool development and the genesis of bioinfor-
matics workflow solutions. These developments in applied
computer science together with the appreciation of data man-
agement as an important research field continuously provide
the technical means to share research data beyond the borders
of research groups, institutions, and even countries to fully
leverage the potential of big data. Querying scientific literature
databases, it is obvious to see that the number of studies that
base on at least one omics type is growing rapidly.
Furthermore, the number of published so-called multi-
omics studies, that is, studies using datasets from more than
one area in life sciences, e.g., proteomics, genomics, and
metabolomics, has increased starkly in recent years, as
depicted in Fig. 2. Using a structured, scalable approach, as
workflow engines tend to offer, enables researchers to analyze
and integrate data from several heterogeneous sources. Both
trends are shown in Fig. 2.
Data sharing
The emergence of open data, the growing willingness to share
primary datasets, and increasing resources of public data pro-
vide a tremendously important third cornerstone for the entry
of big data into the life sciences.
Finally, building on the three innovations, the true potential
of big data lies in the discovery of patterns from unprecedent-
edly large datasets and the formulation of new models and
Challenges of big data integration in the life sciences
Fig. 2 Number of published multi-omics studies per year. Based on reported PubMed keywords and publication date (source: https://doi.org/10.5281/
zenodo.1464136)
hypotheses thereof. The growing comprehensiveness of the
digital description of biological systems, in combination with
advances in machine learning and artificial intelligence, opens
fascinating new possibilities for biomedical research. Here we
review the state of big data in bioanalytical research and iden-
tify the general guidelines for its implementation.
Data ecosystems for big data application
The ability to integrate data is a core element of big data
analyses. Research data requires integration at various levels.
Figure 3 illustrates data integration around bioanalytical data
(shown in the blue box), for which a scalable setup requires
unambiguous metadata annotation of all entities (studies, or-
ganisms, samples) involved in a given experimental design.
Denoted as step (1), the research data needs to be embedded
into a data management infrastructure providing domain-
specific-rich metadata annotation, persistent storage, and
human/machine-accessible query interfaces. Following the
annotation, data can be processed by bioinformatics tools
and pipelines. This processing step (2) is classically known
to as bioinformatics processing and involves many, ideally
open, software applications and reference databases.
Harmonized data processing of similar and related data allows
integrating datasets from multiple experimental conditions
and/or a multitude of omics levels. Figure 3 denotes as its
fourth step the integration beyond the current setting with
large-scale global resources (4). Enabling these four steps in
an ideally automated setup enables the construction of a big
data ecosystem that is scalable to easily integrate new datasets
6793
and methods on the local and the global levels. This system
can be built in many different flavors that all share the inten-
tion of integrating large volumes of heterogeneous and rapidly
growing datasets.
Managing bioanalytical data
At the core of the big data ecosystem are methods and systems
for the efficient management of these data. Obviously,
attempting to work with big data is a community effort, im-
plying the necessity to make data accessible in both standard-
ized human- and machine-readable interfaces. In recent years,
scientists, funding agencies, and journals made tremendous
progress in formalizing these requirements through the FAIR
guidelines [7]. Established to help researchers assess and es-
tablish good data management practices and stewardship, the
concept of FAIR data is a prerequisite to perform sustainable
big data-driven research. FAIR data must be Findable,
Accessible, Interoperable, and Reusable. The application of
the FAIR principles is a key enabler for reproducibility in
science. These applications along with stringent deployment
of data standards are thoroughly discussed in [8].
Findability requires the data not only to be registered in a
searchable resource, but also to be described with rich and
standardized metadata. Implementations are diverse, from
web interfaces to command lines, but they must enable re-
searchers to submit queries and provide upload mechanisms
enforcing a strict requirement for detailed metadata provision-
ing. Resources must also provide an application programming
interface (API), enabling software developers to access the
resource data from within their applications and enable, e.g.,
6794
Fig. 3 Bioanalytical data ecosystem for big data applications.
Bioanalytical research aiming at leveraging big data requires several
layers of data integration. The middle level shows integration at the
local (facility, research group) level, including thorough metadata
machine learning or data mining algorithms to operate on the
data. A very common architectural concept of such an inter-
face is RESTful APIs (Representational State Transfer, often
used with HTTP as the network protocol). Typically, data is
then passed around in the common information exchange for-
mats, such as JSON (JavaScript Object Notation) or XML
(Extended Markup Language), which are easily machine-
and human-readable. A good example of a detailed API de-
scription can be found on the ICGC data portal [9]
(International Cancer Genome Consortium; http://docs.icgc.
org/portal/api-endpoints), where cancer-specific controlled
vocabularies can be used to find samples and raw datasets.
Accessibility refers to retrievable research data. A best prac-
tice in accessibility is realized through linking metadata de-
scribing datasets found in central data repositories. Using
these linked metadata, researchers are provided with related
datasets in order to improve search results. Linked metadata
corresponds to a biology-specific, standardized set of descrip-
tions. Most public repositories provide the technical infra-
structure and implementation of access interfaces and proper
usage documentation.
However, the landscape of public data repositories has
grown significantly, rendering the task of data access non-
trivial. The available data repositories profoundly depend on
the scientific domain and the type of data. A curated list of
FAIR data repositories can be found on the Nature journal
Fillinger S. et al.
annotation (1), as well as lending context (3). The bottom layer depicts
community resources required for data processing (2). Lastly, the top
layer illustrates the global integration and access via public data
repositories (4) (source: https://doi.org/10.5281/zenodo.1464136)
website (https://www.nature.com/sdata/policies/repositories).
DataCite, a global non-profit organization, provides a registry
(re3data) that is connected to a number of subscribed reposi-
tories and thus serves as a central access point to submit search
queries. DataCite provides an extensive schema and docu-
mentation on how data can be annotated properly [10].
Careful annotation of data with metadata is critical for the
comparison and interpretation of research data at a global
scale.
Interoperability in the context of big life science data man-
agement requires data to be used by non-cooperating sites to
make use of the data and integrate it with local data for repro-
ducibility or further analysis. A common use case is the use of
existing data builds new research hypotheses. Global stan-
dards to achieve interoperability and provide good practices,
such as the aforementioned DataCite schema definition, are
central to produce interoperable data and thereby facilitate the
sharing of data.
Data reusability repetitively builds upon rich metadata, if
available. Life science researchers are not always aware of
good practices to attribute data licenses to any data or their
subsets. Such identification options help to clarify the terms of
use and to avoid the cost- and time-intensive investment to
untangle the legal basis and the application of country-specific
copyright laws. The terms of use can be set in most cases by
the copyright holder, which is usually the research institution,
Challenges of big data integration in the life sciences
a company, or a single person that created the data. The Digital
Curation Centre (DCC) makes suggestions for data license
selection (http://www.dcc.ac.uk/resources/how-guides/
license-research-data). Two very permissive licenses, and
thus ensuring reusability, are the Open Data Commons
(https://okfn.org/opendata) and the Creative Commons
(https://creativecommons.org) licenses that have become the
de facto standard in publishing life scientific data.
However, we want to point out that there are substantial chal-
lenges in ensuring accessibility and reusability of data,
especially—but not only—when it concerns human (meta) data.
Access is usually controlled and requires written research interest
proposals that also need to be signed by official legal representa-
tives. Oftentimes institutions do not offer good support and staff
for the transaction of these time-consuming processes. For exam-
ple in order to access certain projects hosted by the International
Cancer Genome Consortium (ICGC), one needs to apply for
(meta) data access over data the Data Access Compliance
Office (DACO) plus additional applications for the different
raw data repositories, like PDC or GDC. As these applications
expire within 24 months, it is necessary to keep track and apply
for renewal. In addition, most researchers do not have a legal
background and therefore are unsure about copyright laws, own-
ership regulations, and licensing models. In the worst case, data
and software code is not shared at all due to this uncertainty,
which contradicts the FAIR principles. While founding agencies
mostly require mandatory data publishing, best practices for
FAIR data (and software) are not yet fully implemented.
Preservation of data integrity
If data were obtained from several sources, a special emphasis
should be drawn to the assurance of data correctness. Data
corruption or loss is the most severe damage that can occur
to (scientific) data. Data loss can be observed after a simple
disk failure, after (un)intended physical damage or accidental
deletion. While regular backups to remote systems address
this type of damage quite efficiently, an underestimated dam-
age can occur through data corruption. Data corruptions are
defined as events when the bit-encoded content changes un-
intentionally. The reasons for this can be manifold, such as
radiation on the memory (DRAM [11]), background noise in
the network traffic, or device failure.
Technical solutions that minimize the risk of undetected
data corruption are widely available. For example, at the hard-
ware level, it is considered best practice to employ, e.g.,
Hamming-based error-correcting code memory (ECC [12])
that auto-detects and corrects single- and double-bit errors.
However, as these are regularly more costly and depend on
more expensive hardware that supports this type of memory,
regular laboratory desktop computer environments rarely em-
ploy such advanced hardware solutions. Paradoxically, in the
research setting, investments for data generation are
6795
comparably easy to justify. A mass spectrometer, costing
approx. 1 M € with associated equipment, is a common in-
vestment in research institutes, yet allocating budget for FAIR
data management is frequently not appreciated.
As a best practice recommendation, generated data should
be stored together with a checksum file that contains the
checksum derived from the data. Most common checksum
algorithms for this purpose are based on cryptographic hash
functions such as MD5 or SHA1, or CRC32 (polynomial
division–based). While the abovementioned ones are not con-
sidered to be inherently secure against intended manipulation,
they are sufficient for detecting random errors introduced in
the data by the aforementioned causes. Notably, these check-
sum files should be created as early as possible in a digital
object lifetime and accompany the data during every data
transfer step for validation. It is also considered to be best
practice to deliver the data to local or remote data centers, as
they usually provide the technical measures for sustainable
data storage and delivery.
While there are solutions available that address these prob-
lems, we want to point out that this usually comes with a
higher price for the hardware and needs domain expertise for
the installation and maintenance of such storage solutions.
Neglecting the proper provisioning of such a technical setup
and best practice integrity preservation techniques inevitably
leads to data that could have been altered without notice or even
rendered unusable. For example, such alterations can potential-
ly lead to different numerical measurement values and change
the file content. Obviously, this can have severe consequences
for further scientific evaluation of the data. For the sake of
reproducibility and research data integrity, this should be min-
imized by implementing an appropriate big data infrastructure.
Bioinformatics data processing
With the availability of biological big data, the need to process
considerable amounts of data in computations in a timely
m a n n er ha s b e co m e a p r e s s i ng m a t t er [1 3 , 1 4] .
Bioinformatic calculations are therefore turning more com-
plex and require more computing resources to complete in a
timely manner. Simple personal computers and monolithic
approaches to resolve scientific questions in bioinformatics
can no longer keep pace with the growth of available biolog-
ical data, the complexity of computations, and the need for
faster results. Structuring solutions to scientific questions
using workflows not only continues to be seen as a good
practice, but also helps users to obtain reproducible results in
a scalable and timely manner.
There are several comprehensive bioinformatics software
libraries [15–17] designed to be integrated into workflow en-
gines, thus facilitating scientists to design scalable workflows.
The primary annotation of experimental raw data with bioin-
formatics workflows requires the integration of many
6796
community resources, such as reference genomes and
proteomes as well as spectral libraries.
Integration beyond institutions
Institutional research is building considerable amounts of data
in their application domains and on specific research ques-
tions. Studying correlations in heterogeneous datasets, how-
ever, requires the integration of large volumes of data that
exceed the capacities of single institutions. These needs have
been largely appreciated by the respective communities, and
the required mindset towards data sharing and open data is
currently being pushed forward by peers, funding agencies,
and journals. The infrastructure on a global scale is partially
provided by public repositories that operate as a central data
hub for scientific high-throughput data and allow reusability
beyond the original research questions. Secondly, to make use
of these global resources, the well-established process of
copying the relevant data to a local infrastructure is no longer
applicable, but the tool sets, e.g., bioinformatics workflows,
will need to be brought to the data and not vice versa.
Public repositories
Over the last decades, multiple data-hosting platforms have
been established for a variety of data. Most of these platforms
aim at providing a centralized location for specific data types,
e.g., mass spectrometry (MS)–based proteomics/metabolomics
or next-generation sequencing (NGS)–based genomics or tran-
scriptomics experiments.
PRIDE [18] (https://www.ebi.ac.uk/pride/archive/) aims at
providing access to protein and peptide identifications as well
as post-translational modifications. All entries into PRIDE are
curated by a team of experts in the field, ensuring that data
entering the repository is of high quality. Standardized datasets
are furthermore kept in a preprocessed form, enabling direct
comparisons between different experiments in the repository.
MetaboLights (Haug et al. 2013) is a cross-species data
repository for metabolomics raw data and the corresponding
metadata. Similar to PRIDE, it offers the deposition of mass
spectrometric data in open standard formats which is the de
facto standard repository for many journals publishing meta-
bolomics studies.
SRA [19] (https://www.ncbi.nlm.nih.gov/sra) and ENA
[20] (https://www.ebi.ac.uk/ena) provide means to store
sequencing data from both Sanger and next-generation se-
quencing methodologies in a standardized manner. Results
from various publications are archived publicly in these repos-
itories, enabling researchers to utilize datasets for comparative
studies in their own research projects.
The gene expression omnibus (GEO) [21] provides public
access to archived sets of microarray or other next-generation
Fillinger S. et al.
sequencing functional genomic data. This approach aims at
providing a central archive for researchers to download and
combine expression data within their own research projects in
order to for example unravel novel transcriptome changes in
different types of tissue.
One of the first projects in the cancer context was the
International Cancer Genome Consortium (ICGC)’s database
TCGA [9] that stores somatic and germline datasets for more
than 25,000 patients and aims at providing a central data resource
for the personalized treatment and research in cancer genomics.
Other applications of big data in the life sciences focused
on the generation of large gene expression datasets such as the
GTex [22] that covers more than 50 different sample tissues.
Such an atlas of expression can be used intensively to inves-
tigate reference gene expression in new measures or compare
these against putatively disease gene expression.
Bringing computation to data
The onset of large-scale research projects in genomics can be seen
articulately in the size of datasets being created in the last years.
While the first 1000 Genomes Project (including stage 3 of the
project) [23] now provides access to more than 2500 sequenced
whole genome sequencing (WGS) datasets, newer projects such
as the Icelandic whole genome diversity project generated WGS
samples of 15,220 Icelanders [24]—a 6-fold increase.
Current projects such as the UK 100,000 Genomes Project
[25] aim to sequence a total of 100,000 WGS samples and the
European Union initialized a project among 13 states of the
European Union to sequence a total of 1,000,000 individuals
[26] in order to enable rare disease research as well as provide
a basis for personalized medicine in healthcare.
Nevertheless, the last years have seen the generation of
large life science databases that incorporate data from various
sources and are utilized to derive new knowledge using data
mining technologies on large-scale data available.
All these approaches have in common that they intend to
provide a solid data basis for research scientists. An obvious
consequence is that they facilitate the means for future treat-
ment methods that rely on large-scale databases of
sequence—as well as other omics data to answer specific
questions that rise especially in the treatment of cancer and
rare disease. The sheer mass of accumulated data in these
repositories provides, unfortunately, a challenging environ-
ment for the classic analysis path that researchers were relying
on before: Downloading and integrating datasets in the
petabyte (PB) range is oftentimes impractical on local com-
puting resources for different reasons, primarily because it
requires a good understanding of these resources and financial
possibilities that are not accessible to many researchers.
Additionally, acquiring huge datasets is very time consuming
and computationally expensive as the data integrity needs to
be verified after the download. Last but not least, it produces
Challenges of big data integration in the life sciences
high amounts of network traffic which burdens especially
smaller institutions with limited bandwidth capabilities.
One example that provides a technical solution is the
International Cancer Genome Consortium (ICGC) data portal
(https://dcc.icgc.org/), which provides controlled access to
human genetic data in the context of cancer research. The
sequencing raw data is then stored in different central data
repositories that are embedded in cloud computing
environments such as AWS. The big advantage of this
approach is that once a researcher has granted access to the
data repositories, computational workflows can be deployed
on virtual machines with the data accessible via, i.e., the S3
protocol. No raw data transfer is required (Downs et al. 2014).
A different model would be extending cloud server APIs from
offering mere data access (data portals) to performing data oper-
ations. For instance, the International Human Epigenome
Consortium allows massive epigenome datasets to be aggregated
and analyzed efficiently on the server-side (Albrecht et al. 2016).
Furthermore, in some research contexts, especially in the
personalized medicine field, stricter data security efforts than
local institutions can typically provide are required, thus hin-
dering data sharing with researchers outside of data collecting
institutions that initially have access to, e.g., biopsy material
from patients. Particularly the data sharing policies but also
the hindrances that occur when large-scale data repositories
need to be shared among researchers, therefore, led to several
developments that aim to bring computing and therefore data
analysis to the actual data hosts and repositories.
A limitation of the aforementioned cloud approach is that it
requires data from various sources to be aggregated in a secure
manner. This is particularly problematic when it comes to
sensitive patient data. Another possibility is thus federated
machine learning, where algorithms operate in a distributed
fashion across individual data hubs such as hospitals or re-
search institutes and allow for shared machine learning on
the entire dataset. This approach allows for privacy by design
and is explored by initiatives such as the European
FeatureCloud project (http://featurecloud.eu/).
Initiatives like the Global Alliance for Genomics in Health
(GA4GH, www.ga4gh.org) aim at providing generalized
policies and interaction interfaces for researchers across the
globe. With such interfaces, researchers should be enabled to
analyze data at various locations, by directly initiating data
analysis at these distinct locations and integrating obtained
results with their own research data. The biggest change to
current big data analysis methods is the context in which such
analysis is performed by, e.g., computing summary metrics
from cancer patients at a hospital and searching the initiative
for patients with similar mutational burden in their tumors
respectively. Such information could be incredibly beneficial
for cancer treatment, as the outcome of treatment of other
patients with very similar mutational burden would allow for
a direct assessment of feasibility for certain cancer treatments.
6797
Shivom (https://shivom.io/) and Nebula Genomics (https://
nebula.org/) are initiatives that aim to resolve data sharing and
protection issues in the context of personalized medicine. The
principle behind both concepts is a blockchain-based infra-
structure that ingests DNA samples from users, e.g., from
personalized genetic testing companies such as 23andme or
AncestryDNA, but also healthcare and sequencing providers.
Both raw and processed data belong to the data owner at any
time, but the owner can share access to their data with re-
searchers of their choice, enabled through the blockchain tech-
nology running the service. Users can get tokens to gain ac-
cess to healthcare products (e.g., more or differently targeted
health reports) in return for providing researchers with more
material. The distributed ledger of blockchain is likely to pro-
vide a solution to data privacy and IP issues, when it comes to
integrating many datasets in a big data type of experiment.
Big data quo Vadis?
Although bioinformatics as a general research field is existing
since about 25 years, there have been significant changes in
terms of especially data volume that is handled nowadays in
bioinformatics workloads.
Intriguingly, big data, if comprehensive enough, carries the
potential to challenge the scientific method for new discover-
ies in science, through studying data correlations at a large
scale. Theoretically, comprehensive data circumvents model-
ing the unknown, but allows direct readout. However, trans-
lating this ambitious goal to the labs requires restructuring
many of today’s research routines.
The prevalent tendency to build and maintain project-
specific and institutional data silos will prevent fostering the
full potential of big data. Instead, the field’s future is the cre-
ation of centralized (meta) data repositories or “data lakes” of
structured and unstructured data, as dominated by genomics
and ongoing for other omics technologies. An early success of
the paradigm shift towards sharing gave rise to initiatives such
as GA4GH. These big data ecosystems finally allow the inte-
grative analysis for the entire globe that was not possible years
back. As data protection and ethical concerns limit the possi-
bilities of local computing facilities to data that has been col-
lected at the respective institution, decentralized and standard-
ized data-sharing measures have to be developed in the next
years to make big data in science a true success story.
Following FAIR principles, big data ecosystems will need to
provide a scalable, metadata-aware framework for future anal-
yses in various research areas. Only if the research community
conceptionally prepares for sharing and integrating data across
the global, artificial intelligence (AI) methods will be able to
learn from data, fulfill the high expectations, and favorably
contribute to the progress of biomedical research.
6798
Funding information This work was carried out with the support of the
German Research Foundation (DFG) within project INF, SFB/TR 209
“Liver Cancer.”
Compliance with ethical standards
Conflict of interest The authors declare that they have no conflict of
interest.
Research involving human participants and/or animals Not applicable.
Informed consent Not applicable.
References
1. Mayer-Schönberger V, Cukier K. Big data: a revolution that will
transform how we live, work and think. In: Houghton Mifflin
Harcourt Publishing Company, vol. 215. New York: Park Avenue
South; 2013. p. 10003.
2. NGRAM Viewer. https://books.google.com/ngrams. Accessed
Oct 2018
3. Price MO, Rider F. The scholar and the future of the research library.
A problem and its solution. Columbia Law Rev. 1944;44:938.
4. Yao Q, Tian Y, Li P-F, Tian L-L, Qian Y-M, Li J-S. Design and
development of a medical big data processing system based on
Hadoop. J Med Syst. 2015;39:23.
5. CERN Data Centre passes the 200-petabyte milestone | CERN.
https://home.cern/about/updates/2017/07/cern-data-centre-passes-
200-petabyte-milestone. Accessed 16 Oct 2018.
6. Savage N. Big data goes green. Nature. 2018;558:S19.
7. Wilkinson MD, Dumontier M, Aalbersberg IJJ, Appleton G, Axton
M, Baak A, et al. The FAIR guiding principles for scientific data
management and stewardship. Sci Data. 2016;3:160018.
8. Sansone S-A, McQuilton P, Rocca-Serra P, Gonzalez-Beltran A,
Izzo M, Lister AL, et al. FAIRsharing as a community approach
to standards, repositories and policies. Nat Biotechnol. 2019;37:
358–67.
9. Zhang J, Baran J, Cros A, Guberman JM, Haider S, Hsu J, et al.
International Cancer Genome Consortium Data Portal—a one-stop
shop for cancer genomics data. Database. 2011. https://doi.org/10.
1093/database/bar026.
10. DataCite Schema. In: DataCite Schema. https://schema.datacite.
org/meta/kernel-4.1/index.html. Accessed 9 Oct 2018.
11. Schroeder B, Pinheiro E, Weber W-D. DRAM errors in the wild: a
large-scale field study. In: Proceedings of the Eleventh International
Joint Conference on Measurement and Modeling of Computer
Systems. New York: ACM; 2009. p. 193–204.
12. Hamming RW. Error detecting and error correcting codes. Bell Syst
Tech J. 1950;29:147–60.
13. Savage N. Bioinformatics: big data versus the big C. Nature.
2014;509:S66–7.
14. Dai L, Gao X, Guo Y, Xiao J, Zhang Z. Bioinformatics clouds for
big data manipulation. Biol Direct. 2012;7:43 discussion 43.
15. Röst HL, Sachsenberg T, Aiche S, Bielow C, Weisser H, Aicheler F,
et al. OpenMS: a flexible open-source software platform for mass
spectrometry data analysis. Nat Methods. 2016;13:741–8.
16. Hildebrandt A, Dehof AK, Rurainski A, Bertsch A, Schumann M,
Toussaint NC, et al. BALL–biochemical algorithms library 1.3.
BMC Bioinformatics. 2010;11:531.
17. Döring A, Weese D, Rausch T, Reinert K. SeqAn an efficient,
generic C++ library for sequence analysis. BMC Bioinformatics.
2008;9:11.
Fillinger S. et al.
18. Vizcaíno JA, Csordas A, del-Toro N, Dianes JA, Griss J, Lavidas I,
et al. 2016 update of the PRIDE database and its related tools.
Nucleic Acids Res. 2016;44:D447–56.
19. Leinonen R, Sugawara H, Shumway M, International Nucleotide
Sequence Database Collaboration. The sequence read archive.
Nucleic Acids Res. 2011;39:D19–21.
20. Cochrane G, Alako B, Amid C, Bower L, Cerdeño-Tárraga A,
Cleland I, et al. Facing growth in the European Nucleotide
Archive. Nucleic Acids Res. 2013;41:D30–5.
21. Barrett T, Wilhite SE, Ledoux P, Evangelista C, Kim IF,
Tomashevsky M, et al. NCBI GEO: archive for functional geno-
mics data sets—update. Nucleic Acids Res. 2013;41:D991–5.
22. GTEx Consortium. The Genotype-Tissue Expression (GTEx) pro-
ject. Nat Genet. 2013;45:580–5.
23. The 1000 Genomes Project Consortium, Auton A, Abecasis GR,
Altshuler DM, Durbin RM, Abecasis GR, Bentley DR, Chakravarti
A, Clark AG, Donnelly P, Eichler EE, Flicek P, Gabriel SB, Gibbs
RA, Green ED, Hurles ME, Knoppers BM, Korbel JO, Lander ES,
Lee C, Lehrach H, Mardis ER, Marth GT, McVean GA, Nickerson
DA, Schmidt JP, Sherry ST, Wang J, Wilson RK, Gibbs (Principal
Investigator) RA, Boerwinkle E, Doddapaneni H, Han Y, Korchina
V, Kovar C, Lee S, Muzny D, Reid JG, Zhu Y, Wang (Principal
Investigator) J, Chang Y, Feng Q, Fang X, Guo X, Jian M, Jiang H,
Jin X, Lan T, Li G, Li J, Li Y, Liu S, Liu X, Lu Y, Ma X, Tang M,
Wang B, Wang G, Wu H, Wu R, Xu X, Yin Y, Zhang D, Zhang W,
Zhao J, Zhao M, Zheng X, Lander (Principal Investigator) ES,
Altshuler DM, Gabriel (Co-Chair) SB, Gupta N, Gharani N, Toji
LH, Gerry NP, Resch AM, Flicek (Principal Investigator) P, Barker
J, Clarke L, Gil L, Hunt SE, Kelman G, Kulesha E, Leinonen R,
McLaren WM, Radhakrishnan R, Roa A, Smirnov D, Smith RE,
Streeter I, Thormann A, Toneva I, Vaughan B, Zheng-Bradley X,
Bentley (Principal Investigator) DR, Grocock R, Humphray S,
James T, Kingsbury Z, Lehrach (Principal Investigator) H,
Sudbrak (Project Leader), Ralf, Albrecht MW, Amstislavskiy VS,
Borodina TA, Lienhard M, Mertes F, Sultan M, Timmermann B,
Yaspo M-L, Mardis (Co-Principal Investigator) (Co-Chair) ER,
Wilson (Co-Principal Investigator) RK, Fulton L, Fulton R,
Sherry (Principal Investigator) ST, Ananiev V, Belaia Z,
Beloslyudtsev D, Bouk N, Chen C, Church D, Cohen R, Cook C,
Garner J, Hefferon T, Kimelman M, Liu C, Lopez J, Meric P,
O’Sullivan C, Ostapchuk Y, Phan L, Ponomarov S, Schneider V,
Shekhtman E, Sirotkin K, Slotta D, Zhang H, McVean (Principal
Investigator) GA, Durbin (Principal Investigator) RM,
Balasubramaniam S, Burton J, Danecek P, Keane TM, Kolb-
Kokocinski A, McCarthy S, Stalker J, Quail M, Schmidt
(Principal Investigator) JP, Davies CJ, Gollub J, Webster T, Wong
B, Zhan Y, Auton (Principal Investigator) A, Campbell CL, Kong
Y, Marcketta A, Gibbs (Principal Investigator) RA, Yu (Project
Leader), Fuli, Antunes L, Bainbridge M, Muzny D, Sabo A,
Huang Z, Wang (Principal Investigator) J, Coin LJM, Fang L,
Guo X, Jin X, Li G, Li Q, Li Y, Li Z, Lin H, Liu B, Luo R, Shao
H, Xie Y, Ye C, Yu C, Zhang F, Zheng H, Zhu H, Alkan C, Dal E,
Kahveci F, Marth (Principal Investigator) GT, Garrison (Project
Lead), Erik P, Kural D, Lee W-P, Fung Leong W, Stromberg M,
Ward AN, Wu J, Zhang M, Daly (Principal Investigator) MJ,
DePristo (Project Leader), Mark A, Handsaker (Project Leader),
Robert E, Altshuler DM, Banks E, Bhatia G, del Angel G,
Gabriel SB, Genovese G, Gupta N, Li H, Kashin S, Lander ES,
McCarroll SA, Nemesh JC, Poplin RE, Yoon (Principal
Investigator) SC, Lihm J, Makarov V, Clark (Principal
Investigator) AG, Gottipati S, Keinan A, Rodriguez-Flores JL,
Korbel (Principal Investigator) JO, Rausch (Project Leader),
Tobias, Fritz MH, Stütz AM, Flicek (Principal Investigator) P,
Beal K, Clarke L, Datta A, Herrero J, McLaren WM, Ritchie
GRS, Smith RE, Zerbino D, Zheng-Bradley X, Sabeti (Principal
Investigator) PC, Shlyakhter I, Schaffner SF, Vitti J, Cooper
Challenges of big data integration in the life sciences
(Principal Investigator) DN, Ball EV, Stenson PD, Bentley
(Principal Investigator) DR, Barnes B, Bauer M, Keira Cheetham
R, Cox A, Eberle M, Humphray S, Kahn S, Murray L, Peden J,
Shaw R, Kenny (Principal Investigator) EE, Batzer (Principal
Investigator) MA, Konkel MK, Walker JA, MacArthur (Principal
Investigator) DG, Lek M, Sudbrak (Project Leader), Ralf,
Amstislavskiy VS, Herwig R, Mardis (Co-Principal Investigator)
ER, Ding L, Koboldt DC, Larson D, Ye K, Gravel S, Swaroop A,
Chew E, Lappalainen (Principal Investigator) T, Erlich (Principal
Investigator) Y, Gymrek M, Frederick Willems T, Simpson JT,
Shriver (Principal Investigator) MD, Rosenfeld (Principal
Investigator) JA, Bustamante (Principal Investigator) CD,
Montgomery (Principal Investigator) SB, De La Vega (Principal
Investigator) FM, Byrnes JK, Carroll AW, DeGorter MK,
Lacroute P, Maples BK, Martin AR, Moreno-Estrada A,
Shringarpure SS, Zakharia F, Halperin (Principal Investigator) E,
Baran Y, Lee (Principal Investigator) C, Cerveira E, Hwang J,
Malhotra (Co-Project Lead), Ankit, Plewczynski D, Radew K,
Romanovitch M, Zhang (Co-Project Lead), Chengsheng, Hyland
FCL, Craig (Principal Investigator) DW, Christoforides A, Homer
N, Izatt T, Kurdoglu AA, Sinari SA, Squire K, Sherry (Principal
Investigator) ST, Xiao C, Sebat (Principal Investigator) J, Antaki D,
Gujral M, Noor A, Ye K, Burchard (Principal Investigator) EG,
Hernandez (Principal Investigator) RD, Gignoux CR, Haussler
(Principal Investigator) D, Katzman SJ, James Kent W, Howie B,
Ruiz-Linares (Principal Investigator) A, Dermitzakis (Principal
Investigator) ET, Devine (Principal Investigator) SE, Abecasis
(Principal Investigator) (Co-Chair) GR, Min Kang (Project
Leader), Hyun, Kidd (Principal Investigator) JM, Blackwell T,
Caron S, Chen W, Emery S, Fritsche L, Fuchsberger C, Jun G, Li
B, Lyons R, Scheller C, Sidore C, Song S, Sliwerska E, Taliun D,
Tan A, Welch R, Kate Wing M, Zhan X, Awadalla (Principal
Investigator) P, Hodgkinson A, Li Y, Shi (Principal Investigator)
X, Quitadamo A, Lunter (Principal Investigator) G, McVean
(Principal Investigator) (Co-Chair) GA, Marchini (Principal
Investigator) JL, Myers (Principal Investigator) S, Churchhouse
C, Delaneau O, Gupta-Hinch A, Kretzschmar W, Iqbal Z,
Mathieson I, Menelaou A, Rimmer A, Xifara DK, Oleksyk
(Principal Investigator) TK, Fu (Principal Investigator) Y, Liu X,
Xiong M, Jorde (Principal Investigator) L, Witherspoon D, Xing J,
Eichler (Principal Investigator) EE, Browning (Principal
Investigator) BL, Browning (Principal Investigator) SR,
Hormozdiari F, Sudmant PH, Khurana (Principal Investigator) E,
Durbin (Principal Investigator) RM, Hurles (Principal Investigator)
ME, Tyler-Smith (Principal Investigator) C, Albers CA, Ayub Q,
Balasubramaniam S, Chen Y, Colonna V, Danecek P, Jostins L,
Keane TM, McCarthy S, Walter K, Xue Y, Gerstein (Principal
Investigator) MB, Abyzov A, Balasubramanian S, Chen J, Clarke
D, Fu Y, Harmanci AO, Jin M, Lee D, Liu J, Jasmine Mu X, Zhang
J, Zhang Y, Li Y, Luo R, Zhu H, Alkan C, Dal E, Kahveci F, Marth
(Principal Investigator) GT, Garrison EP, Kural D, Lee W-P, Ward
AN, Wu J, Zhang M, McCarroll (Principal Investigator) SA,
Handsaker (Project Leader), Robert E, Altshuler DM, Banks E,
del Angel G, Genovese G, Hartl C, Li H, Kashin S, Nemesh JC,
Shakir K, Yoon (Principal Investigator) SC, Lihm J, Makarov V,
Degenhardt J, Korbel (Principal Investigator) (Co-Chair) JO, Fritz
MH, Meiers S, Raeder B, Rausch T, Stütz AM, Flicek (Principal
Investigator) P, Paolo Casale F, Clarke L, Smith RE, Stegle O,
Zheng-Bradley X, Bentley (Principal Investigator) DR, Barnes B,
Keira Cheetham R, Eberle M, Humphray S, Kahn S, Murray L,
Shaw R, Lameijer E-W, Batzer (Principal Investigator) MA,
Konkel MK, Walker JA, Ding (Principal Investigator) L, Hall I,
Ye K, Lacroute P, Lee (Principal Investigator) (Co-Chair) C,
Cerveira E, Malhotra A, Hwang J, Plewczynski D, Radew K,
Romanovitch M, Zhang C, Craig (Principal Investigator) DW,
Homer N, Church D, Xiao C, Sebat (Principal Investigator) J,
6799
Antaki D, Bafna V, Michaelson J, Ye K, Devine (Principal
Investigator) SE, Gardner (Project Leader), Eugene J, Abecasis
(Principal Investigator) GR, Kidd (Principal Investigator) JM,
Mills (Principal Investigator) RE, Dayama G, Emery S, Jun G,
Shi (Principal Investigator) X, Quitadamo A, Lunter (Principal
Investigator) G, McVean (Principal Investigator) GA, Chen
(Principle Investigator) K, Fan X, Chong Z, Chen T, Witherspoon
D, Xing J, Eichler (Principal Investigator) (Co-Chair) EE, Chaisson
MJ, Hormozdiari F, Huddleston J, Malig M, Nelson BJ, Sudmant
PH, Parrish NF, Khurana (Principal Investigator) E, Hurles
(Principal Investigator) ME, Blackburne B, Lindsay SJ, Ning Z,
Walter K, Zhang Y, Gerstein (Principal Investigator) MB, Abyzov
A, Chen J, Clarke D, Lam H, Jasmine Mu X, Sisu C, Zhang J,
Zhang Y, Gibbs (Principal Investigator) (Co-Chair) RA, Yu
(Project Leader), Fuli, Bainbridge M, Challis D, Evani US, Kovar
C, Lu J, Muzny D, Nagaswamy U, Reid JG, Sabo A, Yu J, Guo X,
Li W, Li Y, Wu R, Marth (Principal Investigator) (Co-Chair) GT,
Garrison EP, Fung Leong W, Ward AN, del Angel G, DePristo MA,
Gabriel SB, Gupta N, Hartl C, Poplin RE, Clark (Principal
Investigator) AG, Rodriguez-Flores JL, Flicek (Principal
Investigator) P, Clarke L, Smith RE, Zheng-Bradley X,
MacArthur (Principal Investigator) DG, Mardis (Principal
Investigator) ER, Fulton R, Koboldt DC, Gravel S, Bustamante
(Principal Investigator) CD, Craig (Principal Investigator) DW,
Christoforides A, Homer N, Izatt T, Sherry (Principal
Investigator) ST, Xiao C, Dermitzakis (Principal Investigator) ET,
Abecasis (Principal Investigator) GR, Min Kang H, McVean
(Principal Investigator) GA, Gerstein (Principal Investigator) MB,
Balasubramanian S, Habegger L, Yu (Principal Investigator) H,
Flicek (Principal Investigator) P, Clarke L, Cunningham F,
Dunham I, Zerbino D, Zheng-Bradley X, Lage (Principal
Investigator) K, Berg Jespersen J, Horn H, Montgomery
(Principal Investigator) SB, DeGorter MK, Khurana (Principal
Investigator) E, Tyler-Smith (Principal Investigator) (Co-Chair) C,
Chen Y, Colonna V, Xue Y, Gerstein (Principal Investigator) (Co-
Chair) MB, Balasubramanian S, Fu Y, Kim D, Auton (Principal
Investigator) A, Marcketta A, Desalle R, Narechania A, Wilson
Sayres MA, Garrison EP, Handsaker RE, Kashin S, McCarroll
SA, Rodriguez-Flores JL, Flicek (Principal Investigator) P, Clarke
L, Zheng-Bradley X, Erlich Y, Gymrek M, Frederick Willems T,
Bustamante (Principal Investigator) (Co-Chair) CD, Mendez FL,
David Poznik G, Underhill PA, Lee C, Cerveira E, Malhotra A,
Romanovitch M, Zhang C, Abecasis (Principal Investigator) GR,
Coin (Principal Investigator) L, Shao H, Mittelman D, Tyler-Smith
(Principal Investigator) (Co-Chair) C, Ayub Q, Banerjee R, Cerezo
M, Chen Y, Fitzgerald TW, Louzada S, Massaia A, McCarthy S,
Ritchie GR, Xue Y, Yang F, Gibbs (Principal Investigator) RA,
Kovar C, Kalra D, Hale W, Muzny D, Reid JG, Wang (Principal
Investigator) J, Dan X, Guo X, Li G, Li Y, Ye C, Zheng X, Altshuler
DM, Flicek (Principal Investigator) (Co-Chair) P, Clarke (Project
Lead), Laura, Zheng-Bradley X, Bentley (Principal Investigator)
DR, Cox A, Humphray S, Kahn S, Sudbrak (Project Lead), Ralf,
Albrecht MW, Lienhard M, Larson D, Craig (Principal Investigator)
DW, Izatt T, Kurdoglu AA, Sherry (Principal Investigator) (Co-
Chair) ST, Xiao C, Haussler (Principal Investigator) D, Abecasis
(Principal Investigator) GR, McVean (Principal Investigator) GA,
Durbin (Principal Investigator) RM, Balasubramaniam S, Keane
TM, McCarthy S, Stalker J, Chakravarti (Co-Chair) A, Knoppers
(Co-Chair) BM, Abecasis GR, Barnes KC, Beiswanger C,
Burchard EG, Bustamante CD, Cai H, Cao H, Durbin RM, Gerry
NP, Gharani N, Gibbs RA, Gignoux CR, Gravel S, Henn B, Jones
D, Jorde L, Kaye JS, Keinan A, Kent A, Kerasidou A, Li Y, Mathias
R, McVean GA, Moreno-Estrada A, Ossorio PN, Parker M, Resch
AM, Rotimi CN, Royal, Charmaine D, Sandoval K, Su Y, Sudbrak
R, Tian Z, Tishkoff S, Toji LH, Tyler-Smith C, Via M, Wang Y,
Yang H, Yang L, Zhu J, Bodmer W, Bedoya G, Ruiz-Linares A, Cai
6800 Fillinger S. et al.

Z, Gao Y, Chu J, Peltonen L, Garcia-Montero A, Orfao A, Dutil J, Alexander Peltzer is coordinat-

Martinez-Cruzado JC, Oleksyk TK, Barnes KC, Mathias RA, ing the Research & Development
Hennis A, Watson H, McKenzie C, Qadri F, LaRocque R, Sabeti in Data Science team at the
PC, Zhu J, Deng X, Sabeti PC, Asogun D, Folarin O, Happi C, Quantitative Biology Center
Omoniwa O, Stremlau M, Tariyal R, Jallow M, Sisay Joof F, Corrah (QBiC), Tübingen. His main fo-
T, Rockett K, Kwiatkowski D, Kooner J, Tịnh Hiê’n T, Dunstan SJ, cus lies on the integration of effi-
Thuy Hang N, Fonnie R, Garry R, Kanneh L, Moses L, Sabeti PC, cient workflow management and
Schieffelin J, Grant DS, Gallo C, Poletti G, Saleheen D, Rasheed A, execution systems in data science,
Brooks LD, Felsenfeld AL, McEwen JE, Vaydylevich Y, Green with a detailed focus on cloud
ED, Duncanson A, Dunn M, Schloss JA, Wang J, Yang H, Auton computing and the application of
A, Brooks LD, Durbin RM, Garrison EP, Min Kang H, Korbel JO, machine learning.
Marchini JL, McCarthy S, McVean GA, Abecasis GR (2015) A
global reference for human genetic variation. Nature 526:68.
24. Jónsson H, Sulem P, Kehr B, Kristmundsdottir S, Zink F, Hjartarson
E, et al. Whole genome characterization of sequence diversity of 15,
220 Icelanders. Sci Data. 2017;4:170115.
Oliver Kohlbacher is a Chair for
25. Turnbull C, Scott RH, Thomas E, Jones L, Murugaesu N, Pretty
Applied Bioinformatics at the
FB, et al. The 100 000 Genomes Project: bringing whole genome
University of Tübingen, Director
sequencing to the NHS. BMJ. 2018;361:k1687.
of the Institute for Translational
26. Anonymous (2018) EU countries will cooperate in linking genomic Bioinformatics at University
databases across borders - digital single market - European Hospital Tübingen, and a Fellow
Commission. In: Digital single market - European Commission. at the Max Planck Institute for
https://ec.europa.eu/digital-single-market/en/news/eu-countries- Developmental Biology. The
will-cooperate-linking-genomic-databases-across-borders. lab’s current research focus is on
Accessed 1 Jul 2019. developing methods and tools for
the analysis of biomedical high-
Publisher’s note Springer Nature remains neutral with regard to throughput data and their applica-
jurisdictional claims in published maps and institutional affiliations. tion in translational research.

Sven Fillinger is coordinating the Sven Nahnsen is the Scientific

IT and software development at
the Quantitative Biology Center
(QBiC) Tübingen. He is actively
working on shaping the software
and IT infrastructure there to con-
tribute to efficient data manage-
ment and analysis platforms.
Director of the Quantitative
Biology Center (QBiC),
Tübingen. His current research in-
terests range from bioinformatics
method development and data
management to data science ap-
plications in the life and environ-
mental sciences.

Luis de La Garza is currently

finishing his doctoral degree pro-
gram in bioinformatics at the
University of Tübingen.
Throughout the years, he has spe-
cialized in workflows, workflow
engines, and high-performance
and distributed computing.