A
American Library Association
David Brown
Southern New Hampsire University, University of
Central Florida College of Medicine, Independent
Consultant, Huntington Beach, CA, USA
The American Library Association (ALA) is a
voluntary organization that represents libraries
and librarians around the world. Worldwide, the
ALA is the largest and oldest professional
organization for libraries, librarians, information
science centers, and information scientists. The
association was founded in 1876 in Philadelphia,
Pennsylvania. Since its inception, the ALA has
provided leadership for the development, promo-
tion, and improvement of libraries, information
access, and information science. The ALA is pri-
marily concerned with learning enhancement and
information access for all people. The organiza-
tion strives to advance the profession through its
initiatives and divisions within the organization.
The primary action areas for the ALA are advo-
cacy, education, lifelong learning, intellectual
freedom, organizational excellence, diversity,
equitable access to information and services,
expansion of all forms of literacy, and library
transformation to maintain relevance in a dynamic
and increasing global digitalized environment.
While ALA is composed of several different divi-
sions, there is no single division devoted exclu-
sively to big data. Rather, a number of different
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_6-1
divisions are working to develop and implement
policies and procedures that will enhance the
quality of, the security of, the access to, and the
utility of big data.
ALA Divisions Working with Big Data
At this time, the Association of College & Research
Libraries (ACRL) is a primary division of the
ALA that is concerned with big data issues. The
ACRL has published a number of papers, guides,
and articles related to the use of, promise of, and
the risks associated with big data. Several other
ALA divisions are also involved with big data.
The Association for Library Collections &
Technical Service (ALCTS) division discusses
issues related to the management, organization,
and cataloging of big data and its sources. The
Library Information Technology Association
(LITA) is an ALA division that is involved with
the technological and user services activities that
advance the collection, access, and use of big data
and big data sources.
Big Data Activities of the Association of
College & Research Libraries (ACRL)
The Association of College & Research Libraries
(ACRL) is actively involved with the opportuni-
ties and challenges presented by big data. As
science and technology advance, our world
2 American Library Association
becomes more and more connected and linked.
These links in and of themselves may be consid-
ered big data, and much of the information that
they transmit is big data. Within the ACRL, big
data is conceptualized in terms of the three Vs: its
volume, its velocity, and its variety. Volume refers
to the tremendously large size of the big data.
However, ACRL stresses that the size of the data
set is a function of the particular problem one is
investigating and size is only one attribute of big
data. Velocity refers to the speed at which data is
generated, needed, and used. As new information
is generated exponentially, the need to catalogue,
organize, and develop user-friendly means of
accessing these big data increases multiple expo-
nentially. The utility of big data is a function of the
speed at which it can be accessed and used. For
maximum utility, big data needs to be accurately
catalogued, interrelated, and integrated with other
big data sets. Variety refers to the many different
types of data that are typically components of and
are integrated into big data. Traditionally, data sets
consist of a relatively small number of different
types of data, like word-processed documents,
graphs, and pictures. Big data on the other hand
is typically concerned with many additional types
of information such as emails, audio and video-
tapes, sketches, artifacts, data sets, and many
other kinds of quantitative and qualitative data.
In addition, big data information is usually pre-
sented in many different languages, dialects, and
tones. A key point that ACRL stresses is that as
disciplines advance, the need for and the value of
big data will increase. However, this advancement
can be facilitated or inhibited by the degree to
which the big data can be accessed and used.
Within this context, librarians who are also infor-
mation scientists are and will continue to be
invaluable resources that can assist with the col-
lection, storage, retrieval, and utilization of big
data. Specifically, ACRL anticipates needs for
specialists in the areas of big data management,
big data security, big data cataloguing, big data
storage, big data updating, and big data accessing.
Conclusion
The American Library Association and its
member libraries, librarians, and information sci-
entists are involved in shaping the future of big
data. As disciplines and professions continue to
advance with big data, librarians and information
scientists’ skills need to advance to enable them
to provide valuable resources for strategists,
decision-makers, policy-makers, researchers, mar-
keters, and many other big data users. The ability
to effectively use big data will be a key to success
as the world economy and its data sources expand.
In this rapidly evolving environment, the work of
the ALA will be highly valuable and an important
human resource for business, industry, govern-
ment, academic and research planners, decision-
makers, and program evaluators who want and
need to use big data.
Cross-References
▶ Automated Modeling/Decision Making
▶ Big Data Curation
▶ Big Data Quality
▶ Data Preservation
▶ Data Processing
▶ Data Storage
▶ Digital Libraries
Further Readings
American Library Association. About ALA. http://www.
ala.org/aboutala/. Accessed 10 Aug 2014.
American Library Association. Association for Library
Collections and Technical Services. http://www.ala.
org/alcts/. Accessed 10 Aug 2014.
American Library Association. Library Information Tech-
nology Association (LITA). http://www.ala.org/lita/.
Accessed 10 Aug 2014.
Bieraugel, Mark. Keeping up with... big data. American
Library Association. http://www.ala.org/acrl/publica
tions/keeping_up_with/big_data. Accessed 10 Aug
2014.
American Library Association
Carr, P. L. (2014). Reimagining the library as a technology:
An analysis of Ranganathan’s five laws of library sci-
ence within the social construction of technology
framework. The Library Quarterly, 84(2), 152–164.
Federer, L. (2013). The librarian as research information-
ist: A case study. Journal of the Medical Library
Association, 101(4), 298–302.
3
Finnemann, N. O. (2014). Research libraries and the
Internet: On the transformative dynamic between institu-
tions and digital media. Journal of Documentation, 70(2),
202–220.
Gordon-Murnane, L. (2012). Big data: A big opportunity
for Librarians. Online, 36(5), 30–34.
A
Anonymization Techniques
Mick Smith1 and Rajeev Agrawal2
1
North Carolina A&T State University,
Greensboro, NC, USA
2
Information Technology Laboratory, US Army
Engineer Research and Development Center,
Vicksburg, MS, USA
Synonyms
Anonymous data; Data anonymization; Data pri-
vacy; De-Identification; Personally identifiable
information
Introduction
Personal information is constantly being collected
on individuals as they browse the internet or share
data electronically. This collection of information
has been further exacerbated with the emergence
of the Internet of things and the connectivity of
many electronic devices. As more data is dissem-
inated into the world, interconnected patterns are
created connecting one data record to the next.
The massive data sets that are collected are of
great value to businesses and data scientists
alike. To properly protect the privacy of these
individuals, it is necessary to de-identify or
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_9-1
anonymize the data. In other words, personally
identifiable information (PII) needs to be
encrypted or altered so that a person’s sensitive
data remains indiscernible to outside sources and
readable to the pre-approved parties. Some popu-
lar anonymization techniques include noise addi-
tion, differential privacy, k-anonymity, l-diversity,
and t-closeness.
The need for anonymizing data has come from
the availability of data through big data. Cheaper
storage, improved processing capabilities, and a
greater diversity of analysis techniques have cre-
ated an environment in which big data can thrive.
This has allowed organizations to collect massive
amounts of data on the customer/client base. This
information in turn can then be subjected to a
variety of business intelligence applications so as
to improve the efficiency of the collecting organi-
zation. For instance, a hospital can collect various
patient health statistics over a series of visits. This
information could include vital statistics measure-
ments, family history, frequency of visits, test
results, or any other health-related metric. All of
this data could be analyzed to provide the patient
with an improved plan of care and treatment,
ultimately improving the patient’s overall health
and the facilities ability to provide a diagnosis.
However, the benefits that can be realized from
the analysis of massive amounts of data come with
the responsibility of protecting the privacy of the
entities whose data is collected. Before the data is
2 Anonymization Techniques
released, or in some instances analyzed, the sen-
sitive personal information needs to be altered.
The challenge comes in deciding upon a method
that can achieve anonymity and preserve the data
integrity.
Noise Addition
The belief with noise addition is that by adding
noise to data sets that the data becomes ambiguous
and the individual subjects will not be identified.
The noise refers to the skewing of an attribute so
that it is displayed as a value within a range. For
instance, instead of giving one static value for a
person’s age, it could be adjusted
2 years. If the
subject’s age is displayed as 36, the observer
would not know the exact value, only that the
age may be between 34 and 38. The challenge
with this technique comes in identifying the
appropriate amount of noise. There needs to be
enough to mask the true attribute value, while at
the same time preserving the data mining relation-
ships that exist within the dataset.
Differential Privacy
Differential privacy is similar to the noise addition
technique in that the original data is altered
slightly to prevent any de-identification. How-
ever, it is done in a manner that if a query is
done on two databases that differ in only one
row that the information contained in the missing
row is not discernable. Cynthia Dwork provides
the following definition:
A randomized function K gives e-differential
privacy if for all data sets D1and D2differing on
at most one element, and all S  Range(K),
Pr ½KðD1 Þ  S  expðeÞ  Pr ½KðD1 Þ  S
As an example think of a database containing
the incomes of 75 people in a neighborhood and
the average income is $75,000. If one person were
to leave the neighborhood and the average income
dropped to $74,000, it would be easy to identify
the income of the departing individual. To
overcome this, it would be necessary to apply
minimum noise so that the average income before
and after would not be representative of the
change. At the same time, the computational
integrity of the data is maintained. The amount
of noise and whether an exponential or Laplacian
mechanism is used is still subject to ongoing
research/discussion.
K-Anonymity
In the k-anonymity algorithm, two common
methods for anonymizing data are suppression
and generalization. By using suppression, the
values of categorical variable, such as name, are
removed entirely from the data set. With general-
ization quantitative variables, such as age or
height, are replaced with a range. This in turn
makes each record in a data set indistinguishable
from at least k–1 other records. One of the major
drawbacks to k-anonymity is that it may be pos-
sible to infer identity if certain characteristics are
already known. As a simple example consider a
data set that contains credit decisions from a bank
(Table 1). The names have been omitted, the age
categorized, and the last two digits of the zip code
have been removed.
This obvious example is for the purposes of
demonstrating the weakness of a potential homo-
geneity attack in k-anonymity. In this case, if it
was known that a 23-year-old man living in
14,999 was in this data set, the credit decision
information for that particular individual could
be inferred.
L-Diversity
L-diversity can be viewed as an extension to k-
anonymity in which the goal is to anonymize
specific sensitive values of a data record. For
instance, in the previous example, the sensitive
information would be the credit decision. As with
k-anonymity generalization and suppression tech-
niques are used to mask the true values of the
target variable. The authors of the l-diversity prin-
ciple, Ashwin Machanavajjhala, Daniel Kifer,
Anonymization Techniques 3

Anonymization Techniques, Table 1 K-anonymity credit example

Age Gender Zip Credit decision
18–25 M 149** Yes
18–25 M 148** No
32–39 F 149** Yes
40–47 M 149** Yes
25–32 F 148** No
32–39 M 149** Yes

Johannes Gehrke, and Muthuramakrishnan
Venkitasubramniam, define it as follows:
A q*-block is l-diverse if it contains at least l well-
represented values for the sensitive attribute S.
A table is l-diverse if every q*-block is l-diverse.
However, it should be noted that the distance
metric may differ depending on the data types.
This includes the following distance measures:
numerical, equal, and hierarchical.

The concept of well-represented has been

defined in three possible methods: distinct l-
diversity, entropy l-diversity, and recursive
(c, l)-diversity. A criticism of the l-diversity
model is that it does not hold up well when the
sensitive value has a minimal number of states.
As an example, consider the credit decision table
from above. If that table were extended to
include 1000 records and 999 of them had a
decision of “yes,” then l-diversity would not be
able to provide sufficient equivalence classes.
T-Closeness
Continuing with the refinement of de-identifica-
tion techniques, t-closeness is an extension of l-
diversity. The goal of t-closeness is to create
equivalence classes that approximate the original
distribution of the attributes in the initial database.
Privacy can be considered a measure of informa-
tion gain. T-Closeness takes this characteristic
into consideration by assessing an observer’s
prior and posterior belief about the content of a
data set as well as the influence of the sensitivity
attribute. As with l-diversity, this approach hides
the sensitive values within a data set while
maintaining association through “closeness.”
The algorithm uses a distance metric known as
the Earth Mover Distance to measure the level of
closeness. This takes into consideration the
semantic interrelatedness of the attribute values.
Conclusion
To be effective each anonymization technique
should prevent against the following risks: sin-
gling out, linkability, and inference. Singling out
is the process of isolating data that could identify
an individual. Linkability occurs when two or
more records in a data set can be linked to either
an individual or grouping of individuals. Finally
inference is the ability to determine the value of
the anonymized data through the values of other
elements within the set. An anonymization
approach that can mitigate these risks should be
considered robust and will reduce the possibility
of re-identification. Each of the techniques pre-
sented address each of these risks differently. The
following table outlines their respective perfor-
mance (Table 2):
For instance, unlike k-anonymity, l-diversity,
and t-closeness are not subject to inference attacks
that utilize the homogeneity or background
knowledge of the data set. Similarly, the three
generalization techniques (k-anonymity, l-diver-
sity, and t-closeness), all present differing levels
of association that can be made due to the cluster-
ing nature of each approach.
As with any aspect of data collection, sharing,
publishing, and marketing, there is the potential
for malicious activity. However, the benefits that
can be achieved from the potential analysis of
such data cannot be overlooked. Therefore, it is
4 Anonymization Techniques

Anonymization Techniques, Table 2 Anonymization algorithm comparison

Technique Singling out Linkability Inference
Noise addition At risk Possibly Possibly
K-anonymity Not at risk At risk At risk
L-diversity Not at risk At risk Possibly
T-closeness Not at risk At risk Possibly
Differential privacy Possibly Possibly Possibly

extremely important to mitigate such risks
through the use of effective de-identification tech-
niques so as to protect sensitive personal informa-
tion. As the amount of data becomes more
abundant and accessible, there is an increased
importance to continuously modify and refine
existing anonymization techniques.
Further Reading
Machanavajjhala, A., et al. (2007). l-diversity: Privacy
beyond k-anonymity. ACM Transactions on Knowl-
edge Discovery from Data, 1(1), Article 3, 1–12.
Sweeney, L. (2002). k-anonymity: A Model for Protecting
Privacy. International Journal of Uncertainty, Fuzzi-
ness and Knowledge-Based Systems, 10(5).
The European Parliament and of the Council Working
Party. (2014). Opinion 05/2014 on anonymisation tech-
niques. http://ec.europa.eu/justice/data-protection/arti
cle-29/documentation/opinion-recommendation/files/
2014/wp216_en.pdf. Retrieved on 29 Dec 2014.

Dwork, C. (2006). Differential privacy. In Automata, lan-

guages and programming. Berlin: Springer.
Li, Ninghui, et al. (2007). t-Closeness: Privacy beyond k-
anonymity and l-diversity. IEEE 23rd International
Conference on Data Engineering, 7.
A
Archaeology
Stuart Dunn
Department of Digital Humanities, King’s
College London, London, UK
Introduction
In one sense, archaeology deals with the biggest
dataset of all: the entire material record of human
history, from the earliest human origins c. 2.2
million years Before Present (BP) to the present
day. However this dataset is, by its nature, incom-
plete, fragmentary, and dispersed. Archaeology
therefore brings a very particular kind of chal-
lenge to the concept of big data. Rather than
real-time analyses of the shifting digital landscape
of data produced by the day to day transactions of
millions of people and billions of devices,
approaches to big data in archaeology refer to
the sifting and reverse-engineering of masses of
data derived from both primary and secondary
investigation into the history of material culture.
Big Data and the Archaeological
Research Cycle
Whether derived from excavation, post-
excavation analysis, experimentation, or simula-
tion, archaeologists have only tiny fragments of
the “global” dataset that represents the material
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_12-1
record, or even the record of any specific time
period or region. If one takes any definition of
“Big Data” as it is generally understood, a corpus
of information which is too massive for desktop-
based or manual analysis or manipulation, no
single archaeological dataset is likely to have
these attributes of size and scale. The significance
of Big Data for archaeology lies not so much in
the analysis and manipulation of single or multi-
ple collections of vast datasets but rather in the
bringing together of multiple data, created at dif-
ferent times, for different purposes and according
to different standards; the interpretive and critical
frameworks needed to create knowledge from
them. Archaeology is “Big Data” in the sense
that it is “data that is bigger than the sum of its
parts.”
Those parts are massively varied. Data in
archaeology can be normal photographic images,
images and data from remote sensing, tabular data
of information such as artifact findspots, numeri-
cal databases, or text. It should also be noted that
the act of generating archaeological data is rarely,
if ever, the end of the investigation or project. Any
dataset produced in the field or the lab typically
forms part of a larger interpretation and interpola-
tion process and – crucially – archaeological data
is often not published in a consistent or interoper-
able manner; although approaches to so-called
Grey Literature, which constitutes reports from
archaeological surveys and excavations that typi-
cally do not achieve a wide readership, are
discussed below. This fits with a general
2 Archaeology
characteristic of Big Data, as opposed to the “e-
Science/Grid Computing” paradigm of the 2000s.
Whereas the latter was primarily concerned with
“big infrastructure,” anticipating the need for sci-
entists to deal with a “deluge” of monolithic data
emerging from massive projects such as the Large
Hardron Collider, as described by Tony Hey and
Anne Trefethen, Big Data is concerned with the
mass of information which grows organically as
the result of the ubiquity of computing in every-
day life and in everyday science. In the case of
archaeology, it may be considered more as a
“complexity deluge,” where small data, produced
on a daily basis, forms part of a bigger picture.
There are exceptions: Some individual projects
in archaeology are concerned with terabyte-scale
data. The most obvious example in the UK is the
North Sea Paleolandscapes, led by the University
of Birmingham, a project which has reconstructed
the Early Holocene landscape of the bed of the
North Sea, which was an inhabitable landscape
until its inundation between 20,000 and 8,000
BP – so-called Doggerland. Vince Gaffney and
others describe drawing on 3D seismic data gath-
ered during the process of oil prospection, this
project has used large-scale data analytics and
visualization to reconstruct the topography of the
preinundation land surface spanning an area larger
than the Netherlands, and to thus allow inferences
as to what environmental factors might have
shaped human habitation of it; although it must
be stressed that there is no direct evidence at all of
that human occupation. While such projects dem-
onstrate the potential of Big Data technologies for
conducting large-scale archaeological research,
they remain the exception. Most applications in
archaeology remain relatively small scale, at least
in terms of the volume of data that is produced,
stored, and preserved.
However, this is not to say that approaches
which are characteristic of Big Data are not chang-
ing the picture significantly in archaeology, espe-
cially in the field of landscape studies. Data from
geophysics, the science of scanning subterranean
features using techniques such as magentometry
and resistivity typically produce relatively large
datasets, which require holistic analysis in order to
be understood and interpreted. This trend is
accentuated by the rise of more sophisticated
data capture techniques in the field, which is
increasing the capacity of data that can be gath-
ered and analyzed. Although still not “big” in the
literal sense of “Big Data,” this class of material
undoubtedly requires the kinds of approaches in
thinking and interpretation familiar from else-
where in the Big data agenda. Recent applications
in landscape archaeology have highlighted the
need both for large capacity and interoperation.
For example, integration of data from the in the
Stonehenge Hidden Landscape project, also
directed by Gaffney, provides for “seamless” cap-
ture of reams of geophysical data from remote
sensing, visualizing the Neolithic landscape
beneath modern Wiltshire to a degree of clarity
and comprehensiveness that would only have
been possible hitherto with expensive and labori-
ous manual survey. Due to improved capture tech-
niques, this project succeeded in gathering a
quantity of data in its first two weeks equivalent
to that of the landmark Wroxeter survey project in
the 1990s.
These early achievements of big data in an
archaeological context fall against a background
of falling hardware costs, lower barriers to usage,
and the availability of generic web-based plat-
forms where large-scale distributed research can
be conducted. This combination of affordability
and usability is bringing about a revolution in
applications such as those described above,
where remote sensing is reaching new concepts
and applications. For example, coverage of freely
available satellite imagery is now near-total;
graphical resolution is finer for most areas than
ever before (1 m or less); and pre-georeferenced
satellite and aerial images are delivered to the
user’s desktop, removing the costly and highly
specialized process of locating imagery of the
Earth’s surface. Such platforms also allow access
to imagery of archaeological sites in regions
which are practically very difficult or impossible
to survey, such as Afghanistan, where declassified
CORONA spy satellite data are now being
employed to construct inventories of the region’s
(highly vulnerable) archaeology. If these develop-
ments cannot be said to have removed the bound-
aries within which archaeologists can produce,
Archaeology
access, and analyze data, then it has certainly
made them more porous.
As in other domains, strategies for the storage
and preservation of data in archaeology have a
fundamental relationship with relevant aspects of
the Big Data paradigm. Much archaeological
information lives on the local servers of institu-
tions, individuals, and projects; this has always
constituted an obvious barrier to their integration
into a larger whole. However, weighing against
this is the ethical and professional obligation to
share, especially in a discipline where the process
of gathering the data (excavation) destroys its
material context. National strategies and bodies
encourage the discharge of this obligation. In the
UK, as well as data standards and collections held
by English Heritage, the main repository for
archaeological data is the Archaeology Data Ser-
vice, based at the University of York. The ADS
considers for accession any archaeological data
produced in the UK in a variety of formats. This
includes most of the data formats used in day-to-
day archaeological workflows: Geographic Infor-
mation System (GIS) databases and shapefiles,
images, numerical data, and text. In the latter
case, particular note should be given to the
“Grey Literature” library of archaeological reports
from surveys and excavations, which typically
present archaeological information and data in a
format suitable for rapid publication, rather than
the linking and interoperation of that data. Cur-
rently, the Library contains over 27,000 such
reports. Currently, the total volume of the ADS’s
collections stands at 4.5 Tb (I thank Michael
Charno for this information). While this could be
considered “big” in terms of any collection of data
in the humanities, it is not of a scale which would
overwhelm most analysis platforms; however
what is key here is that it is most unlikely to be
useful to perform any “global” scale analysis
across the entire collection. The individual
datasets therein relate to each other only inasmuch
as they are “archaeological.” In the majority of
cases, there is only fragmentary overlap in terms
of content, topic, and potential use. A 2007
ADS/English Heritage report on the challenges
of Big Data in archaeology identified four types
of data format potentially relevant to Big Data in
3
the field: LIDAR (Light Detection and Ranging or
Laser Imaging Detection and Ranging) data,
which models terrain elevation modelled from
airborne sensors, 3D laser scanning, maritime sur-
vey, and digital video. At first glance this appears
to underpin an assumption that the primary focus
is data formats which convey larger individual
data objects, such as images and geophysics
data, with the report noting that “many formats
have the potential to be Big Data, for example, a
digital image library could easily be gigabytes in
size. Whilst many of the conclusions reached here
would apply equally to such resources this study
is particularly concerned with Big Data formats in
use with technologies such as lidar surveys, laser
scanning and maritime surveys.”
However, the report also acknowledges that “If
long term preservation and reuse are implicit goals
data creators need to establish that the software to
be used or toolsets exist to support format migra-
tion where necessary.” It is true that any “Big
Data” which is created from an aggregation of
“small data” must interoperate. In the case of
“social data” from mobile devices, for example,
location is a common and standardizable attribute
that can be used to aggregate Tb-scale datasets:
heat maps of mobile device usage can be created
which show concentrations of particular kinds of
activity in particular places at particular times. In
more specific contexts hashtags can be used to
model trends and exchanges between large
groups. Similarly intuitive attributes that can be
used for interoperation, however, elude archaeo-
logical data, although there is much emerging
interest in Linked Data technologies, which
allow the creation of linkages between web-
exposed databases, provided they conform
(or can be configured to conform) to predefined
specifications in descriptive languages such as
RDF. Such applications have proved immensely
successful in areas of archaeology concerned with
particular data types, such as geodata, where there
is a consistent base reference (such as latitude and
longitude). However, this raises a question which
is fundamental to archaeological data in any
sense. Big Data approaches here, even if the data
is not “Big” in terms of relative terms to the social
and natural sciences, potentially allows an
4 Archaeology
“n=all” picture of the data record. As noted
above, however, this record represents only a
tiny fragment of the entire picture. A key question,
therefore, is does “Big data” thinking risk techno-
logical determination, constraining what ques-
tions can be asked? This is a point which has
concerned archaeologists since the very earliest
days of computing in the discipline. In 1975, a
skeptical Sir Moses Finley noted that “It would be
a bold archaeologist who believed he could antic-
ipate the questions another archaeologist or a his-
torian might ask a decade or a generation later, as
the result of new interests or new results from
older researchers. Computing experience has pro-
duced examples enough of the unfortunate conse-
quences . . . of insufficient anticipation of the
possibilities at the coding stage.”
Conclusion
Such questions probably cannot be predicted, but
big data is (also) not about predicting questions.
The kind of critical framework that Big Data is
advancing, in response to the ever-more linkable
mass of pockets of information, each themselves
becoming larger in size as hardware and software
barriers lower, allows us to go beyond what is
available “just” from excavation and survey. We
can look at the whole landscape in greater detail
and at new levels of complexity. We can harvest
public discourse about cultural heritage in social
media and elsewhere and ask what that tells us
about that heritage’s place in the contemporary
world. We can examine what are the fundamental
building blocks of our knowledge about the past
and ask what do we gain, as well as lose, by
putting them into a form that the World Wide
Web can read.
References
Archaeology data service. http://archaeologydataservice.
ac.uk. Accessed 25 May 2017.
Austin, T., & Mitcham, J. (2007). Preservation and man-
agement strategies for exceptionally large data for-
mats: ‘Big Data’. Archaeology Data Service &
English Heritage: York, 28 Sept 2007.
Gaffney, V., Thompson, K., & Finch, S. (2007). Mapping
Doggerland: The Mesolithic landscapes of the South-
ern North Sea. Oxford: Archaeopress.
Gaffney, C., Gaffney, V., Neubauer, W., Baldwin, E.,
Chapman, H., Garwood, P., Moulden, H., Sparrow, T.,
Bates, R., Löcker, K., Hinterleitner, A., Trinks, I., Nau,
W., Zitz, T., Floery, S., Verhoeven, G., & Doneus,
M. (2012). The Stonehenge Hidden Landscapes Pro-
ject. Archaeological Prospection, 19(2), 147–155.
Tudhope, D., Binding, C., Jeffrey, S., May, K., &
Vlachidis, A. (2011). A STELLAR role for knowledge
organization systems in digital archaeology. Bulletin of
the American Society for Information Science and
Technology, 37(4), 15–18.
A
Asian Americans Advancing Justice
Francis Dalisay
Communication & Fine Arts, College of Liberal
Arts & Social Sciences, University of Guam,
Mangilao, GU, USA
Asian Americans Advancing Justice (AAAJ) is a
national nonprofit organization founded in 1991.
It was established to empower Asian Americans,
Pacific Islanders, and other underserved groups,
ensuring a fair and equitable society for all. The
organization’s mission is to promote justice, unify
local and national constituents, and empower
communities. To this end, AAAJ dedicates itself
to develop public policy, educate the public, liti-
gate, and facilitate in the development of grass-
roots organizations. Some of their recent
accomplishments have included increasing Asian
Americans and Pacific Islanders’ voter turnout
and access to polls, enhancing immigrants’ access
to education and employment opportunities, and
advocating for greater protections of rights as they
relate to the use of “big data.”
The Civil Rights Principles for the Era of
Big Data
In 2014, AAAJ joined a diverse coalition com-
prising of civil, human, and media rights groups,
such as the ACLU, the NAACP, and the Center for
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_14-1
Media Justice, to propose, sign, and release the
“Civil Rights Principles for the Era of Big Data.”
The coalition acknowledged that progress and
advances in technology would foster improve-
ments in the quality of life of citizens and help
mitigate discrimination and inequality. However,
because various types of “big data” tools and
technologies – namely, digital surveillance, pre-
dictive analytics, and automated decision-
making – could potentially ease the level in
which businesses and governments are able to
encroach upon the private lives of citizens, the
coalition found it critical that such tools and tech-
nologies are developed and employed with the
intention of respecting equal opportunity and
equal justice.
According to civilrights.org (2014), the Civil
Rights Principles for the Era of Big Data proposes
five key principles: (1) stop high-tech profiling,
(2) guarantee fairness in automated decisions,
(3) maintain constitutional protections,
(4) enhance citizens’ control of their personal
information, and (5) protect citizens from inaccu-
rate data. These principles were intended to
inform law enforcement, companies, and policy-
makers about the impact of big data practices on
racial justice and the civil and human rights of
citizens.
1. Stop high-tech profiling. New and emerging
surveillance technologies and techniques have
made it possible to piece together comprehen-
sive details on any citizen or group, resulting in
2 Asian Americans Advancing Justice
an increased risk of profiling and discrimina-
tion. For instance, it was alleged that police in
New York had used license plate readers to
document vehicles that were visiting certain
mosques; this allowed the police to track
where the vehicles were traveling. The acces-
sibility and convenience of this technology
meant that this type of surveillance could hap-
pen without policy constraints. The principle
of stopping high-tech profiling was thus
intended to limit such acts through setting
clear limits and establishing auditing proce-
dures for surveillance technologies and
techniques.
2. Ensure fairness in automated decisions. Today,
computers are responsible for making critical
decisions that have the potential to affect the
lives of citizens’ in the areas of health, employ-
ment, education, insurance, and lending. For
example, major auto insurers are able to use
monitoring devices to track drivers’ habits, and
as a result, insurers could potentially deny the
best coverage rates to those who often drive
when and where accidents are more likely to
occur. The principle of ensuring fairness in
automated decisions advocates that computer
systems should be operating fairly in situations
and circumstances such as the one described.
The coalition had recommended, for instance,
that independent reviews be employed to
assure that systems are working fairly.
3. Preserve constitutional protections. This prin-
ciple advocates that government databases
must be prohibited from undermining core
legal protections, including those concerning
citizens’ privacy and their freedom of associa-
tion. Indeed, it has been argued that data from
warrantless surveillance conducted by the
National Security Agency have been used by
federal agencies, including the DEA and the
IRS, even though such data were gathered out-
side the policies that rule those agencies. Indi-
viduals with access to government databases
could also potentially use them for improper
purposes. The principle of preserving constitu-
tional protections is thus intended to limit such
instances from occurring.
4. Enhance citizens’ control of their personal
information. According to this principle, citi-
zens should have direct control over how cor-
porations gather data from them, and how
corporations use and share such data. Indeed,
personal and private information known and
accessible to a corporation can be shared with
companies and the government. For example,
unscrupulous companies can find vulnerable
customers through accessing and using highly
targeted marketing lists, such as one that might
contain the names and contact information of
citizens who have cancer. In this case, the
principle of enhancing citizens’ control of per-
sonal information ensures that the government
and companies should not be able to disclose
private information without a legal process to
do so.
5. Protect citizens from inaccurate data. This
principle advocates that when it comes to mak-
ing important decisions about citizens – partic-
ularly, the disadvantaged (the poor, persons
with disabilities, the LGBT community,
seniors, and those who lack access to the
Internet) – corporations and the government
should work to ensure that their databases con-
tain accurate of personal information about
citizens. To ensure the accuracy of data, this
could require disclosing the underlying data
and granting citizens the right to correct infor-
mation that is inaccurate. For instance, govern-
ment employment verification systems have
had higher error rates for legal immigrants
and individuals with multiple surnames
(including many Hispanics) than for other
legal workers; this has created a barrier to
employment. In addition, some individuals
have lost job opportunities because of inaccu-
racies in their criminal history information, or
because their information had been expunged.
The five principles above continue to help
inspire subsequent movements highlighting the
growing need to strengthen and protect civil rights
in the face of technological change. Asian Amer-
icans Advancing Justice and the other members of
the coalition also continue to advocate for these
rights and protections.
Asian Americans Advancing Justice 3

Cross-References Further Readings

▶ American Civil Liberties Union Civil rights and big data: Background material. http://
www.civilrights.org/press/2014/civil-rights-and-big-
▶ Center for Democracy and Technology
data.html. Accessed 20 June 2016.
▶ Center for Digital Democracy
▶ National Hispanic Media Coalition
A
Automated Modeling/Decision
Making
Murad A. Mithani
School of Business, Stevens Institute of
Technology, Hoboken, NJ, USA
Big data promises a significant change in the
nature of information processing, and hence, deci-
sion making. The general reaction to this trend is
that the access and availability of large amounts of
data will improve the quality of individual and
organizational decisions. However, there are also
concerns that our expectations may not be entirely
correct. Rather than simplifying decisions, big
data may actually increase the difficulty of mak-
ing effective choices. I synthesize the current state
of research and explain how the fundamental
implications of big data offer both a promise for
improvement but also a challenge to our capacity
for decision making.
Decision making pertains to the identification
of the problem, understanding of the potential
alternatives, and the evaluation of those alterna-
tives to select the ones that optimally resolve the
problem. While the promise of big data relates to
all aspects of decision making, it more often
affects the understanding, the evaluation, and the
selection of alternatives. The resulting implica-
tions comprise of the dual decision model, higher
granularity, objectivity, and transparency of deci-
sions, and the bottom-up decision making in
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_17-1
organizational contexts. I explain each of these
implications in detail to illustrate the associated
opportunities and challenges.
With data and information exceeding our
capacity for storage, there is a need for decisions
to be made on the fly. While this does not imply
that all decisions have to be immediate, our inabil-
ity to store large amounts of data that is often
generated continuously suggests that decisions
pertaining to the use and storage of data, and
therefore the boundaries of the eventual decision
making context, need to be defined earlier in the
process. With the parameters of the eventual deci-
sion becoming an apriori consideration, big data is
likely to overcome the human tendency of pro-
crastination. It imposes the discipline to recognize
the desired information content early in the pro-
cess. Whether this entails decision processes that
prefer immediate conclusions or if the early
choices are limited to the identification of critical
information that will be used for later evaluation,
the dual decision model with a preliminary deci-
sion far removed from the actual decision offers
an opportunity to examine the available alterna-
tives more comprehensively. It allows decision
makers to have a greater understanding of the
alignment between goals and alternatives. Com-
pare this situation to the recruitment model for a
human resource department that screens as well as
finalizes prospective candidates in a single round
of interviews, or separates the process into two
stages where the potential candidates are first
identified from the larger pool and they are then
2 Automated Modeling/Decision Making
selected from the short-listed candidates in the
second stage. The dual decision model not only
facilitates greater insights, it also eliminates the
fatigue that can seriously dampen the capacity for
effective decisions. Yet this discipline comes at a
cost. Goals, values, and biases that are part of the
early phase of a project can leave a lasting imprint.
Any realization later in the project that was not
deliberately or accidently situated in the earlier
context becomes more difficult to incorporate
into the decision. In the context of recruitment, if
the skills desired of the selected candidate change
after the first stage, it is unlikely that the short-
listed pool will rank highly in that skill. The more
unique is the requirement that emerges in the later
stage, the greater is the likelihood that it will not
be sufficiently fulfilled. This tradeoff suggests that
an improvement in our understanding of the
choices comes at the cost of limited maneuver-
ability of an established decision context.
In addition to the benefits and costs of early
decisions in the data generation cycle, big data
allows access to information at a much more gran-
ular level than possible in the past. Behaviors,
attitudes, and preferences can now be tracked in
extensive detail, fairly continuously, and over lon-
ger periods of time. They can in turn be combined
with other sources of data to develop a broader
understanding of consumers, suppliers,
employees, and competitors. Not only can we
understand in much more depth the activities and
processes that pertain to various social and eco-
nomic landscapes, higher level of granularity
makes decisions more informed and, as a result,
more effective. Unfortunately, granularity also
brings with it the potential of distraction. All
data that pertains to a choice may not be necessary
for the decision, and excessive understanding can
overload our capacity to make inferences. Ima-
gine the human skin which is continuously sens-
ing and discarding thermal information generated
from our interaction with the environment. What
if we had to consciously respond to every signal
detected by the skin? It is this loss of granularity
that comes through the human mind responsive
only to significant changes in temperature that
saves us from being overwhelmed by data. Even
though information granularity makes it possible
to know what was previously impossible, infor-
mation overload can lead us astray towards inap-
propriate choices, and at worse, it can incapacitate
our ability to make effective decisions.
The third implication of big data is the poten-
tial for objectivity. When a planned and compre-
hensive examination of alternatives is combined
with a deeper understanding of the data, the result
is more accurate information. This makes it less
likely for individuals to come up to an incorrect
conclusion. This eliminates the personal biases
that can prevail in the absence of sufficient infor-
mation. Since traditional response to overcome
the effect of personal bias is to rely on individuals
with greater experience, big data predicts an elim-
ination of the critical role of experience. In this
vein, Andrew McAfee and Erik
Brynjolfson (2012) find that regardless of the
level of experience, firms that extensively rely
on data for decision making are, on average, 6%
more profitable than their peers. This suggests that
as decisions become increasingly imbibed with an
objective orientation, prior knowledge becomes a
redundant element. This however does not elimi-
nate the value of domain-level experts. Their role
is expected to evolve into individuals who know
what to look for (by asking the right questions)
and where to look (by identifying the appropriate
sources of data). Domain expertise and not just
experience is the mantra to identify people who
are likely to be the most valuable in this new
information age. However, it needs to be
acknowledged that this belief in objectivity is
based on a critical assumption: individuals endo-
wed with identical information that is sufficient
and relevant to the context, reach identical con-
clusions. Yet anyone watching the same news
story reported by different media outlets knows
the fallacy of this assumption. The variations that
arise when identical facts lead individuals to
contrasting conclusions are a manifestation of
the differences in the way humans work with
information. Human cognitive machinery associ-
ates meanings to concepts based on personal his-
tory. As a result, even while being cognizant of
our biases, the translation of information into
Automated Modeling/Decision Making
conclusion can be unique to individuals. More-
over, this effect compounds with the increase in
the amount of information that is being translated.
While domain experts may help ensure consis-
tency with the prevalent norms of translation,
there is little reason to believe that all domain
experts are generally in agreement. The consensus
is possible in the domains of physical sciences
where objective solutions, quantitative measure-
ments, and conceptual boundaries leave little
ambiguity. However, the larger domain of
human experience is generally devoid of stan-
dardized interpretations. This may be one reason
that a study by the Economist Intelligence
Unit (2012) found a significantly higher propor-
tion of data-driven organizations in the industrial
sectors such as the natural resources, biotechnol-
ogy, healthcare, and financial services. Lack
of extensive reliance on data in the other indus-
tries is symptomatic of our limited ability for
consensual interpretation in areas that challenge
the positivistic approach.
The objective nature of big data produces two
critical advantages for organizations. The first is
transparency. A clear link between data, informa-
tion, and decision implies the absence of personal
and organizational biases. Interested stakeholders
can take a closer look at the data and the associ-
ated inferences to understand the basis of conclu-
sions. Not only does this promise a greater buy-in
from participants that are affected by those deci-
sions, it develops a higher level of trust between
decision makers and the relevant stakeholders,
and it diminishes the need for external monitoring
and governance. Thus, transparency favors the
context in which human interaction becomes eas-
ier. It paves the way for richer exchange of infor-
mation and ideas. This in turn facilitates the
quality of future decisions. But due to its very
nature, big data makes replications rather difficult.
The time, energy, and other resources required to
fully understand or reexamine the basis of choices
makes transparency not an antecedent but a con-
sequence of trust. Participants are more likely to
believe in transparency if they already trust the
decision makers, and those that are less receptive
to the choices remain free to accuse the process as
3
opaque. Regardless of the comprehensiveness of
the disclosed details, transparency largely remains
a symbolic expression of the participants’ faith in
the people managing the process.
A second advantage that arises from the objec-
tive nature of data is decentralization. Given that
decisions made in the presence of big data are
more objective and require lower monitoring,
they are easier to delegate to people who are closer
to the action. By relying on proximity and expo-
sure as the basis of assignments, organizations can
save time and costs by avoiding the repeated
concentration and evaluation of information that
often occurs at the various hierarchical levels as
the information travels upwards. So unlike the
flatter organizations of the current era which rely
on the free flow of information, lean organizations
of the future may decrease the flow of information
altogether, replacing it with data-driven, contex-
tually rich, and objective findings. In fact, this is
imminent since the dual decision model defines
the boundaries of subsequent choices. Any
attempt to disengage the later decision from the
earlier one is likely to eliminate the advantages of
granularity and objectivity. Flatter organizations
of the future will delegate not because managers
have greater faith in the lower cadres of the orga-
nization but because individuals at the lower
levels are the ones that are likely to be best posi-
tioned to make timely decisions. As a result, big
data is moving us towards a bottom-up model of
organizational decisions where people at the inter-
face between data and findings determine the stra-
tegic priorities within which higher-level
executives can make their call. Compare this
with the traditional top-down model of organiza-
tional decisions where strategic choices of the
higher executives define the boundaries of actions
for the lower-level staff. However, the bottom-up
approach is also fraught with challenges. It mini-
mizes the value of executive vision. The subjec-
tive process of environmental scanning allows
senior executives to imbibe their valued prefer-
ences into organizational choices through selec-
tive attention to information. It enables
organizations to do what would be uninformed
and at times, highly irrational. Yet it sustains the
4 Automated Modeling/Decision Making

Automated Modeling/Decision Making, Table 1 Opportunities and challenges for the decision implications of
big data
Big data implication Opportunity Challenge
1. Dual decision model Comprehensive examination of Early choices can constrain later considerations
alternatives
2. Granularity In-depth understanding Critical information can be lost due to
information overload
3. Objectivity Lack of dependence on experience Inflates the effect of variations in translation
4. Transparency Free-flow of ideas Difficult to validate
5. Bottom-up decision Prompt decisions Impairment of vision
making

spirit of beliefs that take the form of entrepreneur- Cross-References

ial action. By setting up a mechanism where facts
and findings run supreme, organization of the ▶ Big Data Quality
future may constrain themselves to do only what ▶ Data Governance
is measureable. Extensive reliance on data can ▶ Decision Theory
impair our capacity to imagine what lies beyond ▶ Decision Tree
the horizon (Table 1).
In sum, the big data revolution promises a
change in the way individuals and organiza-
Further Readings
tions make decisions. But it also brings with
it a host of challenges. The opportunities and Boyd, D., & Crawford, K. (2012). Critical questions for big
threats discussed in this article reflect different data. Information, Communication & Society, 15(5),
facets of the implications that are fundamental 662–679.
to this revolution. They include the dual deci- Economist Intelligence Unit. (2012). The deciding factor:
Big data & decision making. New York, NY, USA:
sion model, granularity, objectivity, transpar- Capgemini/The Economist.
ency, and the bottom-up approach to McAfee, A., & Brynjolfsson, E. (2012). Big data: The
organizational decisions. The table above sum- management revolution. Harvard Business Review,
marizes how the promise of big data is an 90(10), 61–67.
opportunity as well as a challenge for the
future of decision making.
B
Behavioral Analytics
Lourdes S. Martinez
School of Communication, San Diego State
University, San Diego, CA, USA
Behavioral analytics can be conceptualized as a
process involving the analysis of large datasets
comprised of behavioral data in order to extract
behavioral insights. This definition encompasses
three goals of behavioral analytics intended to
generate behavioral insights for the purposes of
improving organizational performance and deci-
sion-making as well as increasing understanding
of users. Coinciding with the rise of big data and
the development of data mining techniques, a
variety of fields stand to benefit from the emer-
gence of behavioral analytics and its implications.
Although there exists some controversy regarding
the use of behavioral analytics, it has much to
offer organizations and businesses that are willing
to explore its integration into their models.
Definition
The concept of behavioral analytics has been
defined by Montibeller and Durbach as an analyt-
ical process of extracting behavioral insights from
datasets containing behavioral data. This defini-
tion is derived from previous conceptualizations
of the broader overarching idea of business
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_18-1
analytics put forth by Davenport and Harris as
well as Kohavi and colleagues. Business analytics
in turn is a subarea within business intelligence
and described by Negash and Gray as systems that
integrate data processes with analytics tools to
demonstrate insights relevant to business planners
and decision-makers. According to Montibeller
and Durbach, behavioral analytics differs from
traditional descriptive analysis of behavioral data
by focusing analyses on driving action and
improving decision-making among individuals
and organizations. The purpose of this process is
threefold. First, behavioral analytics facilitates the
detection of users’ behavior, judgments, and
choices. For example, a health website that tracks
the click-through behavior, views, and downloads
of its visitors may offer an opportunity to person-
alize user experience based on profiles of different
types of visitors.
Second, behavioral analytics leverages find-
ings from these behavioral patterns to inform
decision-making at the organizational level and
improve performance. If personalizing the visitor
experience to a health website reveals a mismatch
between certain users and the content provided on
the website’s navigation menu, the website may
alter the items on its navigation menu to direct this
group of users to relevant content in a more effi-
cient manner. Lastly, behavioral analytics informs
decision-making at the individual level by
improving judgments and choices of users. A
health website that is personalized to unique
health characteristics and demographics of
2 Behavioral Analytics
visitors may help users fulfill their informational
needs so that they can apply the information to
improve decisions they make about their health.
Applications
According to Kokel and colleagues, the largest
behavioral databases can be found at Internet
technology companies such as Google as well as
online gaming communities. The sheer size of
these datasets is giving rise to new methods,
such as data visualization, for behavioral analyt-
ics. Fox and Hendler note the opportunity in
implementing data visualization as a tool for
exploratory research and argue for a need to create
a greater role for it in the process of scientific
discovery. For example, Carneiro and Mylonakis
explain how Google Flu relies on data visualiza-
tion tools to predict outbreaks of influenza by
tracking online search behavior and comparing it
to geographical data. Similarly, Mitchell notes
how Google Maps analyzes traffic patterns
through data provided via real-time cell phone
location to provide recommendations for travel
directions. In the realm of social media, Bollen
and colleagues have also demonstrated how anal-
ysis of Twitter feeds can be used to predict public
sentiments.
According to Jou, the value of behavioral ana-
lytics has perhaps been most notably observed in
the area of commercial marketing. The consumer
marketing space has borne witness to the progress
made through extracting actionable and profitable
insights from user behavioral data. For example,
between recommendation search engines for
Amazon and teams of data scientists for LinkedIn,
behavioral analytics has allowed these companies
to transform their plethora of user data into
increased profits. Similarly, advertising efforts
have turned toward the use of behavioral analytics
to glean further insights into consumer behavior.
Yamaguchi discusses several tools on which dig-
ital marketers rely that go beyond examining data
from site traffic.
Nagaitis notes observations that are consistent
with Jou’s view of behavioral analytics’ impact on
marketing. According to Nagaitis, in the absence
of face-to-face communication, behavioral analyt-
ics allows commercial marketers to examine e-
consumers through additional lenses apart from
the traditional demographic and traffic tracking.
In approaching the selling process from a relation-
ship standpoint, behavioral analytics uses data
collected via web-based behavior to increase
understanding of consumer motivations and
goals, and fulfill their needs. Examples of these
sources of data include keyword searchers, navi-
gation paths, and click-through patterns. By input-
ting data from these sources into machine learning
algorithms, computational social scientists are
able to map human factors of consumer behavior
as it unfolds during purchases. In addition, behav-
ioral analytics can use web-based behaviors of
consumers as proxies for cues typically conveyed
through in-person face-to-face communication.
Previous research suggests that web-based dia-
logs can capture rich data pointing toward behav-
ioral cues, the analysis of which can yield highly
accurate predictions comparable to data collected
during face-to-face interactions. The significance
of this ability to capture communication cues is
reflected in marketers increased ability to speak to
their consumers with greater personalization that
enhances the consumer experience.
Behavioral analytics has also enjoyed increas-
ingly widespread application in game develop-
ment. El-Nasr and colleagues discuss the
growing significance of assessing and uncovering
insights related to player behavior, both of which
have emerged as essential goals for the game
industry and catapulted behavioral analytics into
a central role with commercial and academic
implications for game development. A combina-
tion of evolving mobile device technology and
shifting business models that focus on game dis-
tribution via online platforms has created a
situation for behavioral analytics to make impor-
tant contributions toward building profitable
businesses.
Increasingly available data on user behavior
has given rise to the use of behavioral analytic
approaches to guide game development. Fields
and Cotton note the premium placed in this indus-
try on data mining techniques that decrease
behavioral datasets in complexity while extracting
Behavioral Analytics
knowledge that can drive game development.
However, determining cutting-edge methods in
behavioral analytics within the game industry is
a challenge due to reluctance on the part of various
organizations to share analytic methods. Drachen
and colleagues observe a difficulty in assessing
both data and analytical methods applied to data
analysis in this area due to a perception that these
approaches represent a form of intellectual prop-
erty. Sifa further notes that to the extent that data
mining, behavioral analytics, and the insights
derived from these approaches provide a compet-
itive advantage over rival organizations in an
industry that already exhibits fierce competition
in the entertainment landscape, organizations will
not be motivated to share knowledge about these
methods.
Another area receiving attention for its appli-
cation of behavioral analytics is business manage-
ment. Noting that while much interest in applying
behavioral analytics has focused on modeling and
predicting consumer experiences, Géczy and col-
leagues observe a potential for applying these
techniques to improve employee usability of inter-
nal systems. More specifically, Géczy and col-
leagues describe the use of behavioral analytics
as a critical first step to user-oriented management
of organizational information systems through
identification of relevant user characteristics.
Through behavioral analytics, organizations can
observe characteristics of usability and interaction
with information systems and identify patterns of
resource underutilization. These patterns are
important in providing implications for designing
streamlined and efficient user-oriented processes
and services. Behavioral analytics can also offer
prospects for increasing personalization during
the user experience by drawing from user infor-
mation provided in user profiles. These profiles
contain information about how the user interacts
with the system, and the system can accordingly
adjust based on clustering of users.
Despite advances made in behavioral analytics
within the commercial marketing and game indus-
tries, several areas are ripe with opportunities for
integrating behavioral analytics to improve per-
formance and decision-making practices. One
area that has not yet reached its full potential for
3
capitalizing on the use of behavioral analytics is
security. Although Brown reports on exploration
in the use of behavioral analytics to track cross-
border smuggling activity in the United Kingdom
through vehicle movement, the application of
these techniques under the broader umbrella of
security remains understudied. Along these lines
and in the context of an enormous amount of
available data, Jou discusses the possibilities for
implementing behavioral analytics techniques to
identify insider threats posed by individuals
within an organization. Inputting data from a vari-
ety of sources into behavioral analytics platforms
can offer organizations an opportunity to contin-
uously monitor users and machines for early indi-
cators and detection of anomalies. These sources
may include email data, network activity via
browser activity and related behaviors, intellec-
tual property repository behaviors related to how
content is accessed or saved, end-point data show-
ing how files are shared or accessed, and other less
conventional sources such as social media or
credit reports. Connecting data from various
sources and aggregating them under a comprehen-
sive data plane can provide enhanced behavioral
threat detection. Through this, robust behavioral
analytics can be used to extract insights into pat-
terns of behavior consistent with an imminent
threat. At the same time, the use of behavioral
analytics can also measure, accumulate, verify,
and correctly identify real insider threats while
preventing inaccurate classification of nonthreats.
Jou concludes that the result of implementing
behavioral analytics in an ethical manner can pro-
vide practical and operative intelligence while
raising the question as to why implementation in
this field has not occurred more quickly.
In conclusion, behavioral analytics has been
previously defined as a process in which large
datasets consisting of behavioral data are analyzed
for the purpose of deriving insights that can serve
as actionable knowledge. This definition includes
three goals underlying the use of behavioral ana-
lytics, namely, to enhance organizational perfor-
mance, improve decision-making, and generate
insights into user behavior. Given the burgeoning
presence of big data and spread of data mining
techniques to analyze this data, several fields have
4 Behavioral Analytics
begun to integrate behavioral analytics into their
approaches for problem-solving and perfor-
mance-enhancing actions. While concerns related
to accuracy and ethical use of these insights
remain to be addressed, behavioral analytics can
present organizations and business with unprece-
dented opportunities to enhance business, man-
agement, and operations.
Cross-References
▶ Big Data
▶ Business Analytics
▶ Data Mining
▶ Data Science
▶ Data Scientist
▶ Data-Driven Decision-Making
Further Readings
Bollen, J., Mao, H., & Pepe, A. (2011). Modeling public
mood and emotion: Twitter sentiment and socio-eco-
nomic phenomena. Proceedings of the Fifth Interna-
tional Association for Advancement of Artificial
Intelligence Conference on Weblogs and Social Media.
Brown, G. M. (2007). Use of kohonen self-organizing
maps and behavioral analytics to identify cross-border
smuggling activity. Proceedings of the World Congress
on Engineering and Computer Science.
Carneiro, H. A., & Mylonakis, E. (2009). Google trends: A
web-based tool for real-time surveillance of disease
outbreaks. Clinical Infectious Diseases, 49(10).
Davenport, T., & Harris, J. (2007). Competing on analyt-
ics: The new science of winning. Boston: Harvard
Business School Press.
Drachen, A., Sifa, R., Bauckhage, C., & Thurau, C. (2012).
Guns, swords and data: Clustering of player behavior in
computer games in the wild. Proceedings of the IEEE
Computational Intelligence and Games.
El-Nasr, M. S., Drachen, A., & Canossa, A. (2013). Game
analytics: Maximizing the value of player data. New
York: Springer Publishers.
Fields, T. (2011). Social game design: Monetization
methods and mechanics. Boca Raton: Taylor &
Francis.
Fox, P., & Hendler, J. (2011). Changing the equation on
scientific data visualization. Science, 331(6018).
Géczy, P., Izumi, N., Shotaro, A., & Hasida, K. (2008).
Toward user-centric management of organizational
information systems. Proceedings of the Knowledge
Management International Conference, Langkawi,
Malaysia (pp. 282-286).
Kohavi, R., Rothleder, N., & Simoudis, E. (2002). Emerg-
ing trends in business analytics. Communications of the
ACM, 45(8).
Mitchell, T. M. (2009). Computer science: Mining our
reality. Science, 326(5960).
Montibeller, G., & Durbach, I. (2013). Behavioral analyt-
ics: A framework for exploring judgments and choices
in large data sets. Working Paper LSE OR13.137. ISSN
2041-4668.
Negash, S., & Gray, P. (2008). Business intelligence. Ber-
lin/Heidelberg: Springer.
Sifa, R., Drachen, A., Bauckhage, C., Thurau, C., &
Canossa, A. (2013). Behavior evolution in tomb raider
underworld. Proceedings of the IEEE Computational
Intelligence and Games.
B
Big Humanities Project
Ramon Reichert
Department for Theatre, Film and Media Studies,
Vienna University, Vienna, Germany
“Big Humanities” are a heterogenic field of
research between IT, cultural studies, and human-
ities in general. Recently, because of higher avail-
ability of digital data, they gained even more
importance. The term “Big Humanities Data”
has prevailed due to the wider usage of the Inter-
net, and it replaced the terms like “computational
science” and “humanities computing,” which
have been used since the beginning of the com-
puter era in the 1960s. These terms were related
mostly to the methodological and practical devel-
opment of digital tools, infrastructures, and
archives.
In addition to the theoretical explorations on
science according to Davidson (2008), Svensson
(2010), Anne et al. (2010) and Gold (2012), “Big
Humanities Data” are divided into three trendset-
ting theoretical approaches, simultaneously cov-
ering the historical development and changes in
the field of research according to the epistemolog-
ical policy:
1. The usage of computers and digitalization of
“primary data” within humanities and cultural
studies are in the center of Digital humanities.
On the one hand the digitization projects relate
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_22-1
to the digitalized portfolios. On the other hand
they relate to the computerized philology tools
for the application of secondary data or results.
Even today these elementary methods of digi-
tal humanities are based on philological tradi-
tion, which sees the evidence-driven collection
and management of data as the foundation of
hermeneutics and interpretation. Beyond the
narrow discussions about the methods,
computer-based measuring within humanities
and cultural studies claims the media-like pos-
tulates of objectivity within modern sciences.
Contrary to the curriculum of text studies in the
50s and 60s within the “Humanities Comput-
ing” (McCarthy 2005) the research area of
related disciplines has been differentiated and
broadened to history of art, culture and sociol-
ogy, media studies, technology, archaeology,
history and musicology (Gold 2012).
2. According to the second phase, in addition to
the quantitative digitalization of texts, the
research practices are being developed in
accordance with the methods and processes of
production, analysis and modeling of digital
research environments for work within human-
ities with digital data. This approach stands
behind the enhanced humanities and tries to
find new methodological approaches of quali-
tative application of generated, processed and
archived data for reconceptualization of tradi-
tional research subjects. (Ramsey and
Rockwell 2012, pp. 75–84).
2 Big Humanities Project
3. The development from humanities 1.0 to
humanities 2.0 (Davidson 2008, pp. 707–717)
marks the transition from digital development
of methods within “Enhanced Humanities” to
the “Social Humanities” which use the possi-
bility of web 2.0 to construct the research
infrastructure. Social humanities use interdis-
ciplinarity of scientific knowledge by making
use of software for open access, social reading
and open knowledge and by enabling online
cooperative and collaborational work on
research and development. On the basis of the
new digital infrastructure of social web
(hypertext systems, Wiki tools, Crowd funding
software etc.) these products transfer the
computer-based processes from the early
phase of digital humanities into the network
culture of the social sciences. Today it is Blog-
ging Humanities (work on digital publications
and mediation in peer-to-peer networks) and
Multimodal humanities (presentation and rep-
resentation of knowledge within multimedia
software environments) that stand for the tech-
nical modernization of academic knowledge
(McPherson 2008). Because of them Big
Social Humanities claims the right to represent
paradigmatically alternative form of knowl-
edge production. In this context one should
reflect on the technical fundamentals of the
computer-based process of gaining insights
within the research of humanities and cultural
studies while critically considering data,
knowledge genealogy and media history in
order to evaluate properly the understanding
of a role in the context of digital knowledge
production and distribution (Thaller 2012,
pp. 7–23).
History of Big Humanities
Big Humanities have been considered only occa-
sionally from the perspective of science and
media history in the course of the last few years
(Hockey 2004). Historical approach to the
interdependent relation between humanities and
cultural studies and the usage of computer-based
processes relativize the aspiration of digital
methods on the evidence and truth and support
the argumentation that digital humanities were
developed from a network of historical cultures
of knowledge and media technologies with their
roots in the end of the nineteenth century.
The relevant research literature of the historical
context and genesis of Big Humanities is regarded
as one of the first projects of genuine humanistic
usages of computer a Concordance of Thomas of
Aquino based on punch cards by Roberto Busa
(Vanhoutte 2013, p. 126). Roberto Busa
(1913–2011), an Italian Jesuit priest, is considered
as a pioneer of Digital Humanities. This project
enabled the achievement of uniformity in histori-
ography of computational science in its early
stage (Schischkoff 1952). Busa, who in 1949
developed the linguistic corpus of “Index
Thomisticus” together with Thomas J. Watson,
the founder of IBM, (Busa 1951, 1980,
pp. 81–90), is regarded a founder of the point of
intersection between humanities and IT. The first
digital edition on punch cards initiated a series of
the following philological projects: “In the 60s the
first electronic version of ‘Modern Language
Association International Bibliography’
(MLAIB) came up, a specific periodical bibliog-
raphy of all modern philologies, which could be
searched through with a telephone coupler. The
retrospective digitalization of cultural heritage
started after that, having had ever more works
and lexicons such as German vocabulary by
Grimm brothers, historical vocabularies as the
Krünitz or regional vocabularies” (Lauer 2013,
p. 104).
At first, a large number of other disciplines and
non-philological areas were formed such as liter-
ature, library, and archive studies. They had lon-
ger epistemological history in the field of
philological case studies and practical information
studies. Since the introduction of punch card
methods, they have been dealing with quantitative
and IT procedures for facilities of knowledge
management. As one can see, neither the research
question nor Busa’s methodological procedure
have been without its predecessors, so they can
be seen as a part of a larger and longer history of
knowledge and media archeology. Sketch models
of mechanical knowledge apparatus capable of
Big Humanities Project
combining information were found in the manu-
scripts of Suisse Archivar Karl Wilhelm Bührer
(1861–1917, Bührer 1890, pp. 190–192). This
figure of thought of flexible and modularized
information unit was made to a conceptional
core of mechanical data processing. The archive
and library studies took part directly in the histor-
ical change of paradigm of information pro-
cessing. It was John Shaw Billings, the doctor
and later director of the National Medical Library,
who worked further on the development of appa-
ratus for machine-driven processing of statistical
data, a machine developed by Hermann Hollerith
in 1886 (Krajewski 2007, p. 43). Technology of
punch cards traces its roots in technical pragmat-
ics of library knowledge organization; even if
only later – within the rationalization movement
in the 1920s – the librarian working procedure
was automatized in specific areas. Other projects
of data processing show that the automatized pro-
duction of an index or a concordance marks the
beginning of computer-based humanities and cul-
tural studies for the lexicography and catalogue
apparatus of libraries. Until the late 1950s, it was
the automatized method of processing large text
data with the punch card system after Hollerith
procedure that stood in the center of the first
applications/usages. The technical procedure of
punch cards changed the lecture practice of text
analysis by transforming a book into a database
and by turning the linear-syntagmatic structure of
text into a factual and term-based system. As early
as 1951, the academic debate among the contem-
poraries started in academic journals. This debate
saw the possible applications of the punch card
system as largely positive and placed them into
the context of economically motivated rationality.
Between December 13 and 16, 1951, the German
Society for Documentation and the Advisory
Board of German Economical Chamber orga-
nized a working conference on the study of mech-
anization and automation of documentation
process, which was enthusiastically discussed by
philosopher Georgi Schischkoff. He talked about
a “significant simplification and acceleration [. . .]
by mechanical remembrance” (Schischkoff 1952,
p. 290). The representatives of computer-based
humanities saw in the “literary computing,”
3
starting in the early 1950s, the first autonomous
research area, which could provide an “objective
analysis of exact knowledge” (Pietsch 1951). In
the 1960s, the first studies in the field of computer
linguistics concerning the automatized indexing
of large text corpora appeared, publishing the
computer-based analysis about word indexing,
word frequency, and word groups.
The automatized evaluation procedure of texts
for the editorial work within literary studies was
described already in the early stages of “humani-
ties computing” (mostly within its areas of “com-
puter philology” and “computer linguistics”) on
the ground of two discourse figures relevant even
today. The first figure of discourse describes the
achievements of the new tool usage with instru-
mental availability of data (“helping tools”); the
other figure of discourse focuses on the econom-
ical disclosure of data and emphasizes the effi-
ciency and effectivity of machine methods of
documenting. The media figure of automation
was finally combined with the expectance that
interpretative and subjective influences from the
processing and analysis of information can be
systematically removed. In the 1970s and 1980s,
the computer linguistics was established as an
institutionally positioned area of research with
its university facilities, its specialist journals
(Journal of Literary and Linguistic Computing,
Computing in the Humanities), discussion panels
(HUMANIST), and conference activities. The
computer-based work in the historical-
sociological research has its first large rise, but it
remains regarded in the work reports less than an
autonomous method, and it is seen mostly as a
tool for critical text examination and as a simpli-
fication measure by quantifying the prospective
subjects (Jarausch 1976, p. 13).
A sustainable media turn both in the field of
production and in the field of reception aesthetics
appeared with the application of standardized
markup texts such as the Standard Generalized
Markup Language established in 1986 and
software-driven programs for text processing.
They made available the additional series of dig-
ital modules, analytical tools, and text functions
and transformed the text into a model of a data-
base. The texts could be loaded as structured
4 Big Humanities Project
information and were available as (relational)
databases. In the 1980s and 1990s, the technical
development and the text reception were domi-
nated by the paradigm of a database.
With the domination of the World Wide Web,
the research and teaching practices changed dras-
tically: the specialized communication experi-
enced a lively dynamics through the digital
network culture of publicly accessible online
resources, e-mail distribution, chats, and forums,
and it became largely responsive through the
media-driven feedback mentality of rankings and
voting. With its aspiration to go beyond the hier-
archical structures of academic system through
the reengineering of scientific knowledge, the
Digital Humanities 2.0 made the ideals of equal-
ity, freedom, and omniscience attainable again.
As opposed to its beginnings in the 1950s, the
Digital Humanities today have also an aspiration
to reorganize the knowledge of the society. There-
fore, they regard themselves “both as a scientific
as well as a socioutopistic project” (Hagner and
Hirschi 2013, p. 7). With the usage of social media
in the humanities and cultural studies, the techno-
logical possibilities and the scientific practices of
Digital Humanities not only developed but they
also brought to life new phantasmagoria of scien-
tific distribution, quality evaluation, and transpar-
ency in the World Wide Web (Haber 2013,
pp. 175–190). In this context, Bernhard Rieder
and Theo Röhle identified five central problematic
perspectives for the current “Digital Humanities”
in their text from 2012 “five challenges.” These
are the following: the temptation of objectivity,
the power of visual evidence, black-boxing
(fuzziness, problems of random sampling, etc.),
institutional turbulences (rivaling service facilities
and teaching subjects), and the claim of univer-
sality. Computer-based research is usually domi-
nated by the evaluation of data so that some
researchers see the advanced analysis within the
research process even as a substitution for a sub-
stantial theory construction. That means that the
research interests are almost completely data
driven. This evidence-based concentration on the
data possibilities can deceive the researcher to
neglect the heuristic aspects of his own subject.
Since the social net is not only a neutral reading
channel of research, writing, and publication
resources without any power but also a govern-
mental structure of power of scientific knowledge,
the epistemological probing of social, political,
and economic contexts of Digital Humanities
includes also a data critical and historical
questioning of its computer-based reformation
agenda (Schreibmann 2012, pp. 46–58).
What did the usage of computer technology
change for cultural studies and humanities on the
basis of theoretical essentials? Computers did
reorganize and accelerated the quantification and
calculation process of scientific knowledge; they
did entrench the metrical paradigm in the cultural
studies and humanities and promoted the
hermeneutical-interpretative approaches with a
mathematical formalization of the respective sub-
ject field. In addition to these epistemological
shifts, the research practices within the Big
Humanities have been shifted, since the research
and development are seen as project related, col-
laborative, and network formed, and on the net-
work horizon, they become the subject of research
of network analysis. The network analysis itself
has its goal to reveal the correlations and relation-
patterns of digital communication of scientific
networks and to declare the Big Humanities itself
to the subject of reflections within a social con-
structivist actor-network-theory.
Further Readings
Anne, B, Drucker, J., Lunenfeld, P., Presner, T., &
Schnapp, J. (2010). Digital_humanities. Cambridge,
MA: MIT Press, 201(2). Online: http://mitpress.mit.
edu/sites/default/files/titles/content/9780262018470_
Open_Access_Edition.pdf
Bührer, K. W. (1890). Ueber Zettelnotizbücher und
Zettelkatalog. Fernschau, 4, 190–192.
Busa, R. (1951). S. Thomae Aquinatis Hymnorum
Ritualium Varia Specimina Concordantiarum. Primo
saggio di indici di parole automaticamentecomposti e
stampati da macchine IBM a schede perforate. Milano:
Bocca.
Busa, R. (1980). The annals of humanities computing: The
index Thomisticus. Computers and the Humanities,
14(2), 83–90.
Big Humanities Project
Davidson, C. N. (2008). Humanities 2.0: Promise, perils,
predictions. Publications of the Modern Language
Association (PMLA), 123(3), 707–717.
Gold, M. K. (Ed.). (2012). Debates in the digital humani-
ties. Minneapolis: University of Minnesota Press.
Haber, P. (2013). ‘Google Syndrom‘. Phantasmagorien des
historischen Allwissens im World Wide Web. Zürcher
Jahrbuch für Wissensgeschichte, 9, 175–190.
Hagner, M., & Hirschi, C. (2013). Editorial Digital
Humanities. Zürcher Jahrbuch für Wissensgeschichte,
9, 7–11.
Hockey, S. (2004). History of humanities computing. In
S. Schreibman, R. Siemens, & J. Unsworth (Eds.), A
companion to digital humanities. Oxford: Blackwell.
Jarausch, K. H. (1976). Möglichkeiten und Probleme der
Quantifizierung in der Geschichtswissenschaft. In:
ders., Quantifizierung in der Geschichtswissenschaft.
Probleme und Möglichkeiten (pp. 11–30). Düsseldorf:
Droste.
Krajewski, M. (2007). In Formation. Aufstieg und Fall der
Tabelle als Paradigma der Datenverarbeitung. In
D. Gugerli, M. Hagner, M. Hampe, B. Orland,
P. Sarasin, & J. Tanner (Eds.), Nach Feierabend.
Zürcher Jahrbuch für Wissenschaftsgeschichte (Vol.
3, pp. 37–55). Zürich/Berlin: Diaphanes.
Lauer, G. (2013). Die digitale Vermessung der Kultur.
Geisteswissenschaften als Digital Humanities. In
H. Geiselberger & T. Moorstedt (Eds.), Big Data. Das
neue Versprechen der Allwissenheit (pp. 99–116).
Frankfurt/M: Suhrkamp.
McCarty, W. (2005). Humanities computing. London:
Palgrave.
McPherson, T. (2008). Dynamic vernaculars: Emergent dig-
ital forms in contemporary scholarship. Lecture presented
5
to HUMLab Seminar, Umeå University, 4 Mar. http://
stream.humlab.umu.se/index.php?streamName=dynami
cVernaculars
Pietsch, E. (1951). Neue Methoden zur Erfassung des
exakten Wissens in Naturwissenschaft und Technik.
Nachrichten für Dokumentation, 2(2), 38–44.
Ramsey, S., & Rockwell, G. (2012). Developing things:
Notes toward an epistemology of building in the digital
humanities. In M. K. Gold (Ed.), Debates in the digital
humanities (pp. 75–84). Minneapolis: University of
Minnesota Press.
Rieder, B., & Röhle, T. (2012). Digital methods: Five
challenges. In D. M. Berry (Ed.), Understanding digital
humanities (pp. 67–84). London: Palgrave.
Schischkoff, G. (1952). Über die Möglichkeit der
Dokumentation auf dem Gebiete der Philosophie.
Zeitschrift für Philosophische Forschung, 6(2),
282–292.
Schreibman, S. (2012). Digital humanities: Centres and
peripheries. In: M. Thaller (Ed.), Controversies around
the digital humanities (Historical social research, Vol.
37(3), pp. 46–58). Köln: Zentrum für Historische
Sozialforschung.
Svensson, P. (2010). The landscape of digital humanities.
Digital Humanities Quarterly (DHQ), 4(1). Online:
http://www.digitalhumanities.org/dhq/vol/4/1/000080/
000080.html
Thaller, M. (Ed.). (2012). Controversies around the digital
humanities: An agenda. Computing Historical Social
Research, 37(3), 7–23.
Vanhoutte, E. (2013). The gates of hell: History and defi-
nition of digital | humanities. In M. Terras, J. Tyham, &
E. Vanhoutte (Eds.), Defining digital humanities
(pp. 120–156). Farnham: Ashgate.
B
Biomedical Data
Qinghua Yang1 and Fan Yang2
1
Department of Communication Studies, Texas
Christian University, Fort Worth, TX, USA
2
Department of Communication Studies,
University of Alabama at Birmingham,
Birmingham, AL, USA
Thanks to the development of modern data col-
lection and analytic techniques, biomedical
research generates increasingly large amounts of
data in various formats and at all levels, which is
referred to as big data. Big data is a collection of
data sets, which are large in volume and complex
in structure. To illustrate, the data managed by
America’s leading healthcare provider Kaiser is
4,000 times more than the amount of information
stored in the Library of Congress. As to data
structure, the range of nutritional data types and
sources make it really difficult to normalize. Such
volume and complexity of big data make it diffi-
cult to be processed by traditional data analytic
techniques. Therefore, to further knowledge and
uncover hidden value, there is an increasing need
to better understand and mine biomedical big data
by innovative techniques and new approaches,
which requires interdisciplinary collaborations
involving data providers and users (e.g., biomed-
ical researchers, clinicians, and patients), data sci-
entists, funders, publishers, and librarians.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_25-1
The collection and analysis of big data in bio-
medical area have demonstrated its ability to
enable efficiencies and accountability in health
care, which provides strong evidence for the ben-
efits of big data usage. Electronic health records
(EHRs), an example of biomedical big data, can
provide timely data for assisting monitoring of
infectious diseases, disease outbreaks, and
chronic illnesses, which could be particularly
valuable during public health emergencies. By
collecting and extracting data from EHRs, public
health organizations and authorities could receive
extraordinary amount of information. By analyz-
ing the massive data from EHRs, public health
researchers could conduct comprehensive obser-
vational studies with uncountable patients who
are treated in real clinical settings over years.
Disease progress, clinical outcomes, treatment
effectiveness, and public health intervention effi-
cacies can also be studied by analyzing EHRs
data, which may influence public health deci-
sion-making (Hoffman and Podgurski 2013).
As a crucial juncture of addressing the oppor-
tunities and challenges presented by biomedical
big data, the National Institutes of Health (NIH)
has initiated a Big Data to Knowledge (BD2K)
initiative to maximize the use of biomedical big
data. BD2K, a response to the Data and Informat-
ics Working Groups (DIWG), focuses on
enhancing:
(a) the ability to locate, access, share, and apply
biomedical big data,
2 Biomedical Data
(b) the dissemination of data analysis methods
and software,
(c) the training in biomedical big data and data
science,
(d) the establishment of centers of excellence in
data science (Margolis et al. 2014)
First, BD2K initiative fosters the emergence of
data science as a discipline relevant to biomedi-
cine by developing the solutions to specific high-
need challenges confronting the research commu-
nity. For instance, the Centers of Excellence in
Data Science initiated the first BD2K Funding
Opportunity to test and validate new ideas in
data science. Second, BD2K aims to enhance the
training of methodologists and practitioners in
data science by improving their skills in demand
under the data science “umbrella,” such as com-
puter science, mathematics, statistics, biomedical
informatics, biology, and medicine. Third, given
the complex questions posed by the generation of
large amounts of data requiring interdisciplinary
teams, BD2K initiative facilitates the develop-
ment of investigators in all parts of the research
enterprise for interdisciplinary collaboration to
design studies and perform subsequent data ana-
lyses (Margolis et al. 2014).
Besides these promotive initiatives proposed
by national research institutes, such as NIH,
great endeavors in improving biomedical big
data processing and analysis have also been
made by biomedical researchers and for-profit
organizations. National cyberinfrastructure has
been suggested by biomedical researchers as one
of the systems that could efficiently handle many
of big data challenges facing the medical infor-
matics community. In the United States, the
national cyberinfrastructure (CI) refers to an
existing system of research supercomputer centers
and high-speed networks that connect them
(LeDuc et al. 2014). CI has been widely used by
physical and earth scientists, and more recently
biologists, yet little used by biomedical
researchers. It has been argued that more compre-
hensive adoption of CI could facilitate many chal-
lenges in biomedical area. One example of
innovative biomedical big data technique pro-
vided by for-profit organizations is GENALICE
MAP, a next-generation sequencing (NGS) DNA
processing software launched by a Dutch Soft-
ware Company GENALICE. Processing biomed-
ical big data one hundred times faster than
conventional data analytic tools, MAP demon-
strated robustness and spectacular performance
and raised the NGS data processing and analysis
to a new level.
Challenges
Despite the opportunities brought by biomedical
big data, certain noteworthy challenges also exist.
First, to use big biomedical data effectively, it is
imperative to identify the potential sources of
healthcare information and to determine the
value of linking them together (Weber et al.
2014). The “bigness” of biomedical data sets is
multidimensional: some big data, such as EHRs,
provide depth by including multiple types of data
(e.g., images, notes, etc.) about individual patient
encounters; others, such as claims data, provide
longitudinality, which refers to patients’ medical
information over a period of time. Moreover,
social media, credit cards, census records, and a
various number of other types of data can help
assemble a holistic view of a patient and shed light
on social and environmental factors that may be
influencing health.
The second technical obstacle in linking big
biomedical data results from the lack of a national
unique patient identifier (UPI) in the United
States (Weber et al. 2014). To address the absence
of a UPI to enable precise linkage, hospitals and
clinics have developed sophisticated probabilistic
linkage algorithms based on other information,
such as demographics. By requiring enough vari-
ables to match, hospitals and clinics are able to
reduce the risk of linkage errors to an acceptable
level even though two different patients share the
same characteristics (e.g., name, age, gender, zip
code). In addition, the same techniques used to
match patients across different EHRs can be
extended to data sources outside of health care,
which is an advantage of probabilistic linkage.
Third, besides the technical challenges, pri-
vacy and security concerns turn to be a social
Biomedical Data
challenge in linking biomedical big data (Weber et
al. 2014). As more data are linked, they become
increasingly more difficult to be de-identified. For
instance, although clinical data from EHRs offer
considerable opportunities for advancing clinical
and biomedical research, unlike most other forms
of biomedical research data, clinical data are typ-
ically obtained outside of traditional research set-
tings and must be converted for research use. This
process raises important issues of consent and
protection of patient privacy (Institute of Medi-
cine 2009). Possible constructive responses could
be to regulate legality and ethics, to ensure that
benefits outweigh risks, to include patients in the
decision-making process, and to give patients
control over their data. Additionally, changes in
policies and practices are needed to govern
research access to clinical data sources and facil-
itate their use for evidence-based learning in
healthcare. Improved approaches to patient con-
sent and risk-based assessments of clinical data
usage, enhanced quality and quantity of clinical
data available for research, and new methodolo-
gies for analyzing clinical data are all needed for
ethical and informed use of biomedical big data.
3
Cross-References
▶ Biometrics Databases
▶ Data sharing
▶ Health Informatics
▶ National Institutes of Health
Further Readings
Hoffman, S., & Podgurski, A. (2013). Big bad data: Law,
public health, and biomedical databases. The Journal of
Law, Medicine & Ethics, 41(8), 56–60.
Institute of Medicine. (2009). Beyond the HIPAA privacy
rule: Enhancing privacy, improving health through
research. Washington, DC: The National Academies
Press.
LeDuc, R., Vaughn, M., Fonner, J. M., Sullivan, M., Wil-
liams, J. G., Blood, P. D., et al. (2014). Leveraging the
national cyberinfrastructure for biomedical research.
Journal of the American Medical Informatics Associa-
tion, 21(2), 195–199.
Margolis, R., Derr, L., Dunn, M., Huerta, M., Larkin,
J., Sheehan, J., et al. (2014). The National Institutes of
Health’s Big Data to Knowledge (BD2K) initiative:
Capitalizing on biomedical big data. Journal of the
American Medical Informatics Association, 21(6),
957–958.
Weber, G., Mandl, K. D., & Kohane, I. S. (2014). Finding
the missing link for big biomedical data. Journal of
American Medical Association, 331(4), 2479-2480.
B
Biosurveillance
Ramón Reichert
Universität Wien, Wien, Austria
Internet biosurveillance, or Digital Disease
Detection, represents a new paradigm of Public
Health Governance. While traditional approaches
to health prognosis operated with data collected in
the clinical diagnosis, Internet biosurveillance
studies use the methods and infrastructures of
Health Informatics. That means, more precisely,
that they use unstructured data from different
web-based sources and targets using the collected
and processed data and information about changes
in health-related behavior. The two main tasks of
the Internet biosurveillance are (1) the early detec-
tion of epidemic diseases, biochemical, radiolog-
ical, and nuclear threats (Brownstein et.al. 2009)
and (2) the implementation of strategies and mea-
sures of sustainable governance in the target areas
of health promotion and health education (Walters
et al. 2010). Biosurveillance has established itself
as an independent discipline in the mid-1990s, as
military and civilian agencies began to get inter-
ested in automatic monitoring systems. In this
context, the biosurveillance program of the
Applied Physics Laboratory of Johns Hopkins
University has played a decisive and pioneering
role (Burkom et al. 2008).
The Internet biosurveillance uses the accessi-
bility to data and analytic tools provided by digital
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_27-1
infrastructures of social media, participatory
sources, and non-text-based sources. The struc-
tural change generated by digital technologies, as
main driver for Big Data, offers a multitude of
applications for sensor technology and biometrics
as key technologies. Biometric analysis technolo-
gies and methods are finding their way into all
areas of life, changing people’s daily lives. In
particular the areas of sensor technology, biomet-
ric recognition process, and the general tendency
toward convergence of information and commu-
nication technologies are stimulating the Big Data
research. The conquest of mass markets through
sensor and biometric recognition processes can
sometimes be explained by the fact that mobile,
web-based terminals are equipped with a large
variety of different sensors. More and more users
come this way into contact with the sensor tech-
nology or with the measurement of individual
body characteristics. Due to the more stable and
faster mobile networks, many people are perma-
nently connected to the Internet using their mobile
devices, providing connectivity an extra boost.
With the development of apps, application
software for mobile devices such as smartphones
(iPhone, Android, BlackBerry, Windows Phone)
and Tablet computer, the application culture of
biosurveillance changed significantly, since these
apps are strongly influenced by the dynamics of
the bottom-up participation. Andreas
Albrechtslund speaks in this context of the “Par-
ticipatory Surveillance” (2008) on the social net-
working sites, in which biosurveillance
2 Biosurveillance
increasingly assumes itself as a place for open
production of meaning and permanent negotia-
tion, by providing comment functions, hypertext
systems, and ranking and voting procedures
through collective framing processes. This is the
case of the sports app Runtastic, monitoring dif-
ferent sports activities, using GPS, mobile
devices, and sensor technology, and making infor-
mation, such as distance, time, speed, and burned
calories, accessible and visible for friends and
acquaintances in real time. The Eatery app is
used for weight control and requires its users the
ability to do self-optimization through self-track-
ing. Considering that health apps also aim to
influence the attitudes of their users, they can
additionally be understood as persuasive media
of Health Governance. With their feedback tech-
nologies, the apps facilitate not only issues related
to healthy lifestyles but also multiply the social
control over compliance with the health regula-
tions in peer-to-peer networks. Taking into con-
sideration the network connection of information
technology equipment, as well as the commercial
availability of biometric tools (e.g., “Nike Fuel,”
“Fitbit,” “iWatch”) and infrastructure (apps), the
biosurveillance is frequently associated, in the
public debates, to dystopian ideas of a society of
control biometrically organized.
Organizations and networks for health promo-
tion, health information, and health education and
formation observed with great interest that, every
day, millions of users worldwide search for infor-
mation about health using the Google search
engine. During the influenza season, the searches
for flu increase considerably, and the frequency of
certain search terms can provide good indicators
of flu activity. Back in 2006, Eysenbach evaluated
in a study on “Infodemiology” or “Infoveillance”
the Google AdSense click quotas, with which he
analyzed the indicators of the spread of influenza
and observed a positive correlation between
increasing search engine entries and increased
influenza activity. Further studies on the volume
of search patterns have found that there is a sig-
nificant correlation between the number of flu-
related search queries and the number of people
showing actual flu symptoms (Freyer-Dugas et al.
2012). This epidemiological correlation structure
was subsequently extended to provide early warn-
ing of epidemics in cities, regions, and countries,
in cooperation with the 2008 established Google
Flu Trends in collaboration with the US authority
for the surveillance of epidemics (CDC). On the
Google Flu Trends website, users can visualize
the development of influenza activity both geo-
graphically and chronologically. Some studies
criticize that the predictions of the Google project
are far above the actual flu cases.
Ginsberg et al. (2009) point out that in the case
of an epidemic, it is not clear whether the search
engines behavior of the public remains constant
and thus whether the significance of Google Flu
Trends is secured or not. They refer to the
medialized presence of the epidemic as distorting
cause of an “Epidemic of Fear” (Eysenbach 2006,
p. 244), which can lead to miscalculations
concerning the impending influenza activity. Sub-
sequently, the prognostic reliability of the corre-
lation between increasing search engine entries
and increased influenza activity has been
questioned. In recent publications on digital
biosurveillance, communication processes in
online networks are more intensely analyzed.
Especially in the field of Twitter Research (Paul
and Dredze 2011), researchers developed specific
techniques and knowledge models for the study of
future disease development and work backed up
by context-oriented sentiment analysis and social
network analysis to hold out the prospect of a
socially and culturally differentiated
biosurveillance.
Further Readings
Albrechtslund, A. (2008). Online social networking as
participatory surveillance. First Monday, 13(3).
Online: http://firstmonday.org/ojs/index.php/fm/arti
cle/viewArticle/2142/1949
Brownstein, J. S., et al. (2009). Digital disease detection –
Harnessing the web for public health surveillance. The
New England Journal of Medicine, 360(21),
2153–2157.
Burkom, H. S., et al. (2008). Decisions in biosurveillance
tradeoffs driving policy and research. Johns Hopkins
technical digest, 27(4), 299–311.
Biosurveillance
Eysenbach, G. (2006). Infodemiology: Tracking flu-related
searches on the Web for syndromic surveillance. In
AMIA Annual Symposium, Proceedings 8/2, 244–248.
Freyer-Dugas, A., et al. (2012). Google Flu Trends: Cor-
relation with emergency department influenza rates and
crowding metrics. Clinical Infectious Diseases, 54(15),
463–469.
Ginsberg, J., et al. (2009). Detecting influenza epidemics
using search engine query data. In Nature. Interna-
tional weekly journal of science (Vol. 457, pp.
1012–1014).
3
Paul, M. J., & Dredze, P. (2011). You are what you Tweet:
Analyzing Twitter for public health. In Proceedings of
the Fifth International AAAI Conference on Weblogs
and Social Media. Online: www.aaai.org/ocs/index.
php/ICWSM/ICWSM11/paper/.../3264
Walters, R. A., et al. (2010). Data sources for
biosurveillance. In G. Voeller John (Ed.), Wiley hand-
book of science and technology for homeland security
(Vol. 4, pp. 2431–2447). Hoboken: Wiley.
C
Cancer
Christine Skubisz
Department of Communication Studies, Emerson
College, Boston, MA, USA
Department of Behavioral Health and Nutrition,
University of Delaware, Newark, DE, USA
Cancer is an umbrella term that encompasses
more than 100 unique diseases related to the
uncontrolled growth of cells in the human body.
Cancer is not completely understood by scientists,
but it is generally accepted to be caused by both
internal genetic factors and external environmen-
tal factors. The US National Cancer Institute
describes cancer on a continuum, with points of
significance that include prevention, early detec-
tion, diagnosis, treatment, survivorship, and end-
of-life care. This continuum provides a frame-
work for research priorities. Cancer prevention
includes lifestyle interventions such as tobacco
control, diet, physical activity, and immunization.
Detection includes screening tests that identify
atypical cells. Diagnosis and treatment involves
informed decision making, the development of
new treatments and diagnostic tests, and outcomes
research. Finally, end-of-life care includes pallia-
tive treatment decisions and social support. Large
data sets can be used to uncover patterns, view
trends, and examine associations between vari-
ables. Searching, aggregating, and cross-
referencing large data sets is beneficial at all
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_32-1
stages within the cancer continuum. Sources of
data include laboratory investigations, feasibility
studies, clinical trials, cancer registries, and
patient medical records. The paragraphs that fol-
low describe current practices and future direc-
tions for cancer-related research in the era of big
data.
Cancer Prevention and Early Detection
Epidemiology is the study of the causes and pat-
terns of human diseases. Aggregated data allows
epidemiologists to study why and how cancer
forms. Researchers study the causes of cancer
and ultimately make recommendations about
how to prevent cancer. Data provides medical
practitioners with information about populations
at risk. This can facilitate proactive and preventive
action. Data is used by expert groups including the
American Cancer Society and the United States
Preventive Services Task Force to write recom-
mendations about screening for detection. Screen-
ing tests, including mammography and
colonoscopy, have advantages and disadvantages.
Evidence-based results, from large representative
samples, can be used to recommend screening for
those who will gain the largest benefit and sustain
the fewest harms. Data can be used to identify
where public health education and resources
should be disseminated.
At the individual level, aggregated information
can guide lifestyle choices. With the help of
2 Cancer
technology, people have the ability to quickly and
easily measure many aspects of their daily lives.
Gary Wolf and Kevin Kelly coined this rapid
accumulation of personal data the quantified self
movement. Individual-level data can be collected
through wearable devices, activity trackers, and
smartphone applications. The data that is accumu-
lated is valuable for cancer prevention and early
detection. Individuals can track their physical
activity and diet over time. These wearable
devices and applications also allow individuals
to become involved in cancer research. Individ-
uals can play a direct role in research by contrib-
uting genetic data and information about their
health. Health care providers and researchers can
view genetic and activity data to understand the
connections between health behaviors and
outcomes.
Diagnosis and Treatment
Aggregated data that has been collected over long
periods of time has made a significant contribu-
tion to research on the diagnosis and treatment of
cancer. The Human Genome Project, completed
in 2003, was one of the first research endeavors to
harness large data sets. Researchers have used
information from the Human Genome Project to
develop new medicines that can target genetic
changes or drivers of cancer growth. The ability
to sequence the DNA of large numbers of tumors
has allowed researchers to model the genetic
changes underlying certain cancers.
Genetic data is stored in biobanks, repositories
in which samples of human DNA are stored for
testing and analysis. Researchers draw from these
samples and analyze genetic variation to observe
differences in the genetic material of someone
with a specific disease compared to a healthy
individual. Biobanks are run by hospitals,
research organizations, universities, or other med-
ical centers. Many biobanks do not meet the needs
of researchers due to an insufficient number of
samples. The burgeoning ability to aggregate
data across biobanks, within the United States
and internationally, is invaluable and has the
potential to lead to new discoveries in the future.
Data is also being used to predict which med-
ications may be good candidates to move forward
into clinical research trials. Clinical trials are sci-
entific studies that are designed to determine if
new treatments and diagnostic procedures are safe
and effective. Margaret Mooney and Musa Mayer
estimate that only 3% of adult cancer patients
participate in clinical trials. Much of what is
known about cancer treatment is based on data
from this small segment of the larger population.
Data from patients who do not participate in clin-
ical trials exists, but this data is unconnected and
stored in paper and in electronic medical records.
New techniques in big data aggregation have the
potential to facilitate patient recruitment for clin-
ical trials. Thousands of studies are in progress
worldwide at any given point in time. The tradi-
tional, manual, process of matching patients with
appropriate trials is both time consuming and
inefficient. Big data approaches can allow for the
integration of medical records and clinical trial
data from across multiple organizations. This
aggregation can facilitate the identification of
patients for inclusion in an appropriate clinical
trial. Nicholas LaRusso writes that IBM’s super-
computer Watson will soon be used to match
cancer patients with clinical trials. Patient data
can be mined for lifestyle factors and genetic
factors. This can allow for faster identification of
participants that meet inclusion criteria. Watson,
and other supercomputers, can shorten the patient
identification process considerably, matching
patients in seconds. This has the potential to
increase enrollment in clinical trials and ulti-
mately advance cancer research.
Health care providers’ access to large data sets
can improve patient care. When making a diagno-
sis, providers can access information from
patients exhibiting similar symptoms, lifestyle
choices, and demographics to form more accurate
conclusions. Aggregated data can also improve a
patient’s treatment plan and reduce the costs of
conducting unnecessary tests. Knowing a
patient’s prognosis helps a provider decide how
aggressively to treat cancer and what steps to take
after treatment. If aggregate data from large and
diverse groups of patients were available in a
single database, providers would be better
Cancer
equipped to predict long-term outcomes for
patients. Aggregate data can help providers select
the best treatment plan for each patient, based on
the experiences of similar patients. This can also
allow providers to uncover patterns to improve
care. Providers can also compare their patient out-
comes to outcomes of their peers. Harlan
Krumholz, a professor at the Yale School of Med-
icine, argued that the best way to study cancer is to
learn from everyone who has cancer.
Survivorship and End-of-Life Care
Cancer survivors face physical, psychological,
social, and financial difficulties after treatment
and for the remaining years of their lives. As sci-
ence advances, people are surviving cancer and
living in remission. A comprehensive database on
cancer survivorship could be used to develop, test,
and maintain patient navigation systems to facili-
tate optimal care for cancer survivors.
Treating or curing cancer is not always possi-
ble. Health care providers typically base patient
assessments on past experiences and the best data
available for a given condition. Aggregate data
can be used to create algorithms to model the
severity of illness and predict outcomes. This
can assist doctors and families who are making
decisions about end-of-life care. Detailed infor-
mation, based on a large number of cases, can
allow for more informed decision making. For
example, if a provider is able to tell a patient’s
family with confidence that it is extremely
unlikely that the patient will survive, even with
radical treatment, this eases the discussion about
palliative care.
Challenges and Limitations
The ability to search, aggregate, and cross-refer-
ence large data sets has a number of advantages in
the prevention and treatment of cancer. Yet, there
are multiple challenges and limitations to the use
of big data in this domain. First, we are limited to
3
the data that is available. The data set will always
be incomplete and will fail to cover the entire
population. Data from diverse sources will vary
in quality. Self-reported survey data will appear
alongside data from randomized, clinical trials.
Second, the major barrier to using big data for
diagnosis and treatment is the task of integrating
information from diverse sources. Allen Lichter
explained that 1.6 million Americans are diag-
nosed with cancer every year, but in more than
95% of cases, details of their treatments are in
paper medical records, file drawers, or electronic
systems that are not connected to each other.
Often, the systems in which useful information
is currently stored cannot be easily integrated.
The American Association of Clinical Oncology
is working to overcome this barrier and has devel-
oped software that can accept information from
multiple formats of electronic health records. A
prototype system has collected 100,000 breast
cancer records from 27 oncology groups. Third,
traditional laboratory research is necessary to
understand the context and meaning of the infor-
mation that comes from the analysis of big data.
Large data sets allow researchers to explore cor-
relations or relationships between variables of
interest. Danah Boyd and Kate Crawford point
out that data are often reduced to what can fit
into a mathematical model. Taken out of context,
results lose meaning and value. The experimental
designs of clinical trials will ultimately allow
researchers to show causation and identify vari-
ables that cause cancer. Bigger data, in this case
more data, is not always better. Fourth, patient
privacy and security of information must be pri-
oritized at all levels. Patients are, and will
continue to be, concerned with how genetic and
medical profiles are secured and who will have
access to their personal information.
Cross-References
▶ DNA
▶ Evidence Based Medicine
▶ Genetics
4 Cancer

▶ Genome Data Further Readings

▶ Health Care Delivery
▶ Human Genome Project Murdoch, T. B., & Detsky, A. S. (2013). The inevitable
application of big data to health care. Journal of the
▶ Nutrition
American Medical Association, 309(13), 1351–1352.
▶ Prevention
▶ Treatment
C
Cloud Services
Paula K. Baldwin
Department of Communication Studies, Western
Oregon University, Monmouth, OR, USA
As consumers and institutions congregate larger
and larger portions of data, hardware storage has
become inadequate. These additional storage
needs led to the development of virtual data cen-
ters, also known as the cloud, cloud computing,
or, in the case of the cloud providers, cloud ser-
vices. The origin of the term, cloud computing, is
somewhat unclear, but a cloud-shaped symbol is
often used as a representation on the Internet of
the cloud. The cloud symbol also represents the
remote, complex system infrastructure used to
store and manage the consumer’s data.
The first reference to cloud computing in the
contemporary age appeared in the mid-1990s, and
it became popular in the mid-2000s. As cloud
services become much more versatile and eco-
nomical, consumers’ use is increasing. The
cloud offers users immediate access to a shared
pool of computer resources. As processors con-
tinue to develop both in power and economic
feasibility, the expansion of these data centers
(the cloud) has expanded on an enormous scale.
Cloud services incentivize migration to the cloud
as users recognize the elastic potential for data
storage as a reasonable cost. Cloud services are
the new generation of computing infrastructures,
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_37-1
and there are multiple cloud vendors providing a
range of cloud services. The fiscal benefit of cloud
computing is the consumer only pays for the use
on the resources they need without any concern
over compromising their physical storage areas.
The cloud service manages the data on the back
end. In an era where physical storage limitations
has become problematic with increased down-
loads of movies, books, graphics, and other high
data memory products, cloud computing has been
a welcome development.
Choosing a Cloud Service
As the cloud service industry grows, choosing a
cloud service can be confusing for the consumer.
One of the first areas to consider is the unique
cloud service configurations. Cloud services are
configured in four ways. One, public clouds may
be free or bundled with other services or offered as
pay per usage. Generally speaking, public cloud
service providers like Amazon AWS, Microsoft,
and Google own and operate their own infrastruc-
ture data centers, and access to these providers’
services is through the Internet. Private cloud
services are data management infrastructures cre-
ated solely for one particular organization. Man-
agement of the private cloud may be internal or
external. Community cloud services exist when
multiple organizations from a specific community
with common needs choose to share an infrastruc-
ture. Again, management of the community cloud
2 Cloud Services
service may be internal or external, and fiscal
responsibility is shared between the organiza-
tions. Hybrid clouds are a grouping of two or
more clouds, public or private community,
where the cloud service is comprised of variant
combination that extends the capacity of the ser-
vice through aggregation, integration, or
customizations with another cloud service. Some-
times a hybrid cloud is used on a temporary basis
to meet short-term data needs that cannot be ful-
filled by the private cloud. Having the ability to
use the hybrid cloud enables the organization to
only pay for the extra resources when they are
needed, so this exists as a fiscal incentive for
organizations to use a hybrid cloud service.
The other aspect to consider when evaluating
cloud services is the specific service models
offered for the consumer or organization. Cloud
computing offers three different levels of service:
Software as a Service (SaaS), Platform as a Ser-
vice (PaaS), and Infrastructure as a Service (IaaS).
The SaaS has a specific application or service
subscription for the customer (e.g., Dropbox,
Salesforce.com, and QuickBooks). With the
SaaS, the service provider handles the installation,
setup, and running of the application with little to
no customization. The PaaS allows businesses an
integrated platform on which they can create and
deploy custom apps, databases, and line-of-busi-
ness service (e.g., Microsoft Windows Azure,
IBM Bluemix, Amazon Web Services (AWS),
Elastic Beanstalk, Heroku, Force.com, Apache
Stratos, Engine Yard, and Google App Engine).
The PaaS service model includes the operating
system, programming language execution envi-
ronment, database, and web servicer designed
for a specific framework with a high level of
customization. With Infrastructure as a Service
(IaaS), businesses can purchase infrastructure
from providers as virtual resources. Components
include servers, memory, firewalls, and more, but
the organization provides the operating system.
IaaS providers include Amazon Elastic Cloud
Computer (Amazon EC2), GoGrid, Joyent,
AppNexus, Rackspace, and Google Compute
Engine.
Once the correct cloud service configuration is
determined, the next step is to match user needs
with correct service level. When looking at cloud
services, it is important to examine four different
aspects: application requirements, business
expectations, capacity provisioning, and cloud
information collection and process. These four
areas complicate the process of selecting a cloud
service. First, the application requirements refer to
the different features such as data volume, data
production rate, data transfer and updating, com-
munication, and computing intensities. These fac-
tors are important because the differences in these
factors will affect the CPU (central processing
unit), memory, storage, and network bandwidth
for the user. Business expectations fluctuate
depending on the applications and potential
users, which, in turn, affect the cost. The pricing
model depends on the level of the service required
(e.g., voicemail, a dedicated service, amount of
storage required, additional software packages,
and other custom services). Capacity provisioning
is based on the concept that, according to need,
different IT technologies are employed and, there-
fore, each technology has its own unique strengths
and weaknesses. The downside for the consumer
is the steep learning curve required. The final
challenge requires that the consumers invest a
substantial amount of time to investigate individ-
ual websites, collect information about each cloud
service offering, collate their findings, and employ
their own assessments to determine their best
match. If an organization has an internal IT
department or employs an IT consultant, the deci-
sion is easier to make; for the individual con-
sumer, without an IT background, the choice
may be considerably more difficult.
Cloud Safety and Security
For the consumer, two primary issues are relevant
to cloud usage: a check and balance system on the
usage versus service level purchased and data
safety. This on-demand computation model of
cloud computing is processed through large vir-
tual data centers (clouds), offering storage and
computation needs for all types of cloud users.
These needs are based on service level agree-
ments. Although cloud services are relatively
Cloud Services
low cost, there is no way to know if the services
they are purchasing are equivalent to the service
level purchased. Although being able to deter-
mine that a consumer’s usage in relationship to
the service level purchased is appropriate, the
more serious concern for consumers is data safety.
Furthermore, because users do not have physical
possession of their data, public cloud services are
underutilized due to trust issues. Larger organiza-
tions use privately held clouds, but if a company
does not have the resources to develop their own
cloud service, most organizations are unlikely to
use public cloud services due to safety concerns.
Currently, there is no global standardization of
data encryption between cloud services, and
there have been some concerns raised by experts
who say there is no way to be completely sure that
data, once moved to the cloud, remains secure.
With most cloud services, control of the encryp-
tion keys is retained by the cloud service, making
your data vulnerable to a rogue employee or a
governmental request to see your data.
The Electronic Frontier Foundation (EFF) is a
privacy advocacy group that maintains a section
on their website (Who Has Your Back) that rates
the largest Internet companies on their data pro-
tections. The EFF site uses six criteria to rate the
companies: requires a warrant for content, tells
users about government data requests, publishes
transparency reports, publishes law enforcement
guidelines, fights for user privacy rights in courts,
and fights for user privacy rights in Congress.
Another consumer and corporate data protection
group is the Tahoe Least Authority File System
(Tahoe-LAFS) project. Tahoe-LAFS protects a
free, open-source storage system created and
developed by Zooko Wilcox-O’Hearn with the
goal of data security and protection from hard-
ware failure. The strength of this storage system is
their encryption and integrity – checks first go
through gateway servers, and after the process is
complete, the data is stored on a secondary set of
servers that cannot read or modify the data.
Security for data storage via cloud services is a
global concern whether for individuals or organi-
zations. From a legal perspective, there is a great
3
deal of variance in how different countries and
regions deal with security issues. At this point in
time, until there are universal rules or legacy
specifically addressing data privacy legislation,
the consumers must take responsibility for their
own data. There are five strategies for keeping
your data secure in the cloud, outside of what the
cloud services offer. First, consider storing crucial
information somewhere other than the cloud. For
this type of information, perhaps utilizing the
available hardware storage might be the best solu-
tion rather than using a cloud service. Second,
when choosing a cloud service, take the time to
read the user agreement. The user agreement
should clearly delineate the parameters of their
service level and that will help with the decision-
making. Third, take creating passwords seriously.
Oftentimes, the easy route for passwords is famil-
iar information such as dates of birth, hometowns,
and pet’s or children’s names. With the advances
in hardware and software designed specially to
crack passwords, it is particularly important to
use robust, unique passwords for each of your
accounts. Fourth, the best way to protect data is
through encryption. The way encryption works in
this instance is to use an encryption software on a
file before you move the file to the cloud. Without
the password to the encryption, no one will be
able to read the file content. When considering a
cloud service, investigate their encryption ser-
vices. Some cloud services encrypt and decrypt
user files local as well as provide storage and
backup. Using this type of service ensures that
data is encrypted before it is stored in the cloud
and after it is downloaded from the cloud provid-
ing the optimal safety net for consumer data.
Cross-References
▶ Cloud
▶ Cloud Computing
▶ Cloud Safety
▶ Cloud Storage
▶ Computer Network Storage
4 Cloud Services
Further Readings
Ding, S., et al. (2014). Decision support for personalized
cloud service selection through multi-attribute trust-
worthiness evaluation. PLoS One, 9(6), e97762.
Gui, Z., et al. (2014). A service brokering and recommen-
dation mechanism for better selecting cloud services.
PLoS One, 8(8). e105297. https://doi.org/10.1371/jour-
nal.pone.0105297
Hussain, M., et al. (2014). Software quality in the clouds: A
cloud-based solution. Cluster Computing, 17(2),
389–402.
Kun, H., et al. (2014). Securing the cloud storage audit
service: Defending against frame and collude attacks of
third party auditor. IET Communications, 8(12), 2106–
2113.
Mell, P., et al. (2011). National Institute of Standards and
Technology, U.S. Department of Commerce. The
NIST definition of cloud computing. Special Publica-
tion 800-145, 9–17.
Qi, Q., et al. (2014). Cloud service-aware location update
in mobile could computing. IET Communications, 8(8),
1417–1424.
Rehman, Z., et al. (2014). Parallel could service selection
and ranking based on QoS history. International Jour-
nal of Parallel Programming, 42(5), 820–852.
C
Communications
Alison N. Novak
Department of Public Relations and Advertising,
Rowan University, Glassboro, NJ, USA
There is much debate about the origins and history
of the field of Communications. While many
researchers point to a rhetorical origin in ancient
Greece, others suggest the field is much newer,
developing from psychology and propaganda
studies of the 1940s. The discipline includes
scholars exploring subtopics such as political
communication, media effects, and organizational
relationships. The field generally uses both qual-
itative and quantitative approaches, as well as
developing a variety of mixed-methods tech-
niques to understand social phenomena.
Russell W. Burns argues that the field of Com-
munications developed from a need to explore the
ways in which media influenced people to behave,
support, or believe in a certain idea. Much of
Communication studies investigates the idea of
media and texts, such as newspaper discourses,
social media messages, or radio transcripts. As the
field has developed, it has investigated new tech-
nologies and media, including those still in their
infancies.
Malcom R. Parks states that the field of Com-
munications has not adopted one set definition of
big data, but rather sees the term as a means to
identify datasets and archival techniques.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_39-1
Singularly thinking of big data as a unit of measure-
ment or a size fails to underscore the many uses and
methods used by Communications to explore big
datasets.
One frequent source of big data analysis in
Communications is that of network analysis or
social network analysis. This method is used to
explore the ways in which individuals are
connected in physical and digital spaces. Commu-
nications research on social networks particularly
investigates how close individuals are to each
other, whom they are connected through, and
what resources can be shared amongst networks.
These networks can be archived from social net-
working sites such as Twitter or Facebook, or
alternatively can be constructed through surveys
of people within a group, organization, or com-
munity. The automated data aggregation of digital
social networks makes the method appealing to
Communications researchers because it produces
large networks quickly and with limited possibil-
ity of human error in recording nodes. Addition-
ally, the subfield of Health Communications has
adopted the integration of big datasets in an effort
to study how healthcare messages are spread
across a network.
Natural language processing is another area of
big data inquiry in the field of Communications. In
this vein of research, scholars explore the way that
computers can develop an understanding of lan-
guage and generate responses. Often studied along
with Information Science researchers and Artificial
intelligence developers, natural language processing
2 Communications
draws from Communications association with lin-
guistics and modern languages. Natural language
processing is an attempt to build communication
into computers so they can understand and provide
more sender-tailored messages to users.
The field of communication has also been out-
spoken about the promises levied with big data
analytics as well as the ethics of big data use.
Recognizing that the field is still early in its devel-
opment, scholars point to the lifespan of other
technologies and innovations as examples of
how optimism early in the lifecycle often turns
into critique. Pierre Levy is one Communications
scholar who explains that although new datasets
and technologies are viewed as positive changes
with big promises early in their trajectory, as more
information is learned about their effects, scholars
often begin to challenge their use and ability to
provide insight.
Communications scholars often refer to big data
as the “datafication” of society, meaning turning
everyday interactions and experiences into quanti-
fiable data that can be segmented and analyzed
using brad techniques. This in particular refers to
analyzing data that has not been previously viewed
as data before. Although this is partially where the
value of big data develops from, for Communica-
tions researchers, this complicates the ability to
think holistically or qualitatively.
Specifically, big datasets in Communications
research include information taken from social
media sites, health records, media texts, political
polls, and brokered language transcriptions. The
wide variety of types of datasets reflects the truly
broad nature of the discipline and its subfields.
Malcom Parks offers suggestions on the future
of big data research within the field of Communi-
cations. First, the field must situate big data
research with larger theoretical contexts. One cri-
tique of the data-revolution is the false identifica-
tion of this form of analysis as being new. Rather
than consider big data as an entirely new phenom-
ena, by situating it within a larger history of Com-
munications theory, more direct comparisons
between past and present datasets can be drawn.
Second, the field requires more attention to the
topic of validity in big data analysis. While quan-
titative and statistical measurements can support
the reliability of a study, validity asks researchers
to provide examples or other forms of support for
their conclusions. This greatly challenges the eth-
ical notions of anonymity in big data, as well as
the consent process for individual protections.
This is one avenue in which the quality of big
data research needs more work within the field of
communications.
Communications asserts that big data is an
important technological and methodological
advancement within research, however, due to its
newness, researchers need to exercise caution when
considering its future. Specifically, researchers must
focus on the ethics of inclusion in big datasets,
along with the quality of analysis and long term
effects of this type of dataset on society.
Further Readings
Burns, R. W. (2003). Communications: An international
history of the formative years. New York: IEE History
of Technology Series.
Levy, P. (1997). Collective intelligence: Mankind’s emerg-
ing world in cyberspace. New York: Perseus Books.
Parks, M. R. (2014). Big data in communication research:
Its contents and discontents. Journal of Communica-
tion, 64, 355–360.
C
Computational Social Sciences
Ines Amaral
University of Minho, Braga, Minho, Portugal
Autonomous University of Lisbon, Lisbon,
Portugal
Computational social sciences is a research disci-
pline at the interface between computer science
and the traditional social sciences. This interdis-
ciplinary and emerging scientific field uses com-
putationally methods to analyze and model social
phenomena, social structures, and collective
behavior. The main computational approaches to
the social sciences are social network analysis,
automated information extraction systems, social
geographic information systems, complexity
modeling, and social simulation models.
New areas of social science research have
arisen due the existence of computational and
statistical tools, which allow social scientists to
extract and analyze large datasets of social infor-
mation. Computational social sciences diverges
from conventional social science because of the
use of mathematical methods to model social phe-
nomena. As an intersection of computer science,
statistics, and the social sciences, computational
social science is an interdisciplinary subject,
which uses large-scale demographic, behavioral,
and network data to analyze individual activity,
collective behaviors, and relationships. Modern
distributed computing frameworks, algorithms,
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_41-1
statistics, and machine learning methods can
improve several social science fields like anthro-
pology, sociology, economics, psychology, polit-
ical science, media studies, and marketing.
Therefore, computational social sciences is an
interdisciplinary scientific area, which explores
social dynamics of society through advanced
computational systems.
Computational social science is a relatively
new field, and its development is closely related
to the computational sociology that is often asso-
ciated to the study of social complexity, which is a
useful conceptual framework for the analysis of
society. Social complexity is theory neutral that
frames both local and global approaches to social
research. The theoretical background of this con-
ceptual framework dates back to the work of
Talcott Parsons on action theory, the integration
of the study of social order with the structural
features of macro and micro factors. Several
decades later, in the early 1990s, social theorist
Niklas Luhmann began to work on the themes of
complex behavior. By then, new statistical and
computational methodologies were being devel-
oped for social science problems.
Nigel Gilbert, Klaus G. Troitzsch, and Joshua M.
Epstein are the founders of modern computational
sociology, merging social science research with sim-
ulation techniques in order to model complex policy
issues and essential features of human societies.
Nigel Gilbert is a pioneer in the use of agent-based
models in the social sciences. Klaus G. Troitzsch
introduces the method of computer-based simulation
2 Computational Social Sciences
in the social sciences. Joshua M. Epstein developed,
with Robert Axtell, the first large-scale agent-based
computational model, which aims to explore the role
of social experiences such as seasonal migrations,
pollution, and transmission of disease.
As an instrument-based discipline, computa-
tional social sciences enables the observation and
empirical study of phenomena through computa-
tional methods and quantitative datasets. Quantita-
tive methods such as dynamical systems, artificial
intelligence, network theory, social network analy-
sis, data mining, agent-based modeling, computa-
tional content analysis, social simulations
(macrosimulation and microsimulation), and statis-
tical mechanics are often combined to study com-
plex social systems.
Technological developments are constantly
changing society, ways of communication, behav-
ioral patterns, the principles of social influence,
and the formation and organization of groups and
communities, enabling the emergence of self-
organized movements. As technology-mediated
behaviors and collectives are primary elements
in the dynamics and in the design of social struc-
tures, computational approaches are critical to
understand the complex mechanisms that form
part of many social phenomena in contemporary
society. Big data can be used to understand many
complex phenomena as it offers new opportuni-
ties to work toward a quantitative understanding
of our complex social systems. Technological-
mediated social phenomena emerging over multi-
ple scales are available in complex datasets. Twit-
ter, Facebook, Google, and Wikipedia showed
that it is possible to relate, compare, and predict
opinions, attitudes, social influences, and collec-
tive behaviors. Online and offline big data can
provide insights that allow the understanding of
social phenomena like diffusion of information,
polarization in politics, formation of groups, and
evolution of networks.
Big data is dynamic, heterogeneous, and inter-
related. But it is also often noisy and unreliable.
However, even so, big data may be more valuable
to social sciences than small samples because the
overall statistics obtained from frequent patterns
and correlation analysis disclose often hidden pat-
terns and more reliable knowledge. Furthermore,
when big data is connected, it forms large net-
works of heterogeneous information with data
redundancy that can be exploited to compensate
for the lack of data, to validate trust relationships,
to disclose inherent groups, and to discover hid-
den patterns and models. Several methodologies
and applications in the context of modern social
science datasets allow scientists to understand and
study different social phenomena, from political
decisions to the reactions of economic markets to
the interactions of individuals and the emergence
of self-organized global movements.
Trillions of bytes of data can be captured by
instruments or generated by simulation. Through
better analysis of these large volumes of data that
are becoming available, there is the potential to
make further advances in many scientific disci-
plines and improve the social knowledge and the
success of many companies. More than ever, sci-
ence is now a collaborative activity. Computational
systems and techniques created new ways of
collecting, crossing and interconnecting data.
Analysis of big data are now at the disposal of
social sciences, allowing the study of cases in
macro- and in microscales in connection to other
scientific fields.
Cross-References
▶ Computer Science
▶ Data Visualization
▶ Network Analytics
▶ Network Data
▶ Physics
▶ Social Network Analysis (SNA)
▶ Sociology
▶ Visualization
Further Readings
Bankes, S., Lempert, R., & Popper, S. (2002). Making
computational social science effective epistemology,
methodology, and technology. Social Science Com-
puter Review, 20(4), 377–388.
Bainbridge, W. S. (2007). Computational sociology. In
The Blackwell Encyclopedia of Sociology. Malden,
MA: Blackwell Publishing.
Computational Social Sciences
Cioffi-Revilla, C. (2010). Computational social science.
Wiley Interdisciplinary Reviews: Computational Statis-
tics, 2(3), 259–271.
Conte, R., et al. (2012). Manifesto of computational social
science. The European Physical Journal Special
Topics, 214(1), 325–346.
Lazer, D., et al. (2009). Computational social science.
Science, 323(5915), 721–723.
3
Miller, J. H., & Page, S. E. (2009). Complex adaptive
systems: An introduction to computational models of
social life. Princeton: Princeton University Press.
Oboler, A., et al. (2012). The danger of big data: Social
media as computational social science. First Monday 17
(7). Retrieved from http://firstmonday.org/article/view/
3993/3269/
C
Content Moderation
Sarah T. Roberts
Department of Information Studies, University of
California, Los Angeles, Los Angeles, CA, USA
Synonyms
Community management; Community modera-
tion; Content screening
Definition
Content moderation is the organized practice of
screening user-generated content (UGC) posted to
Internet sites, social media, and other online out-
lets, in order to determine the appropriateness of
the content for a given site, locality, or jurisdic-
tion. The process can result in UGC being
removed by a moderator, acting as an agent of
the platform or site in question. Increasingly,
social media platforms rely on massive quantities
of UGC data to populate them and to drive user
engagement; with that increase has come the con-
comitant need for platforms and sites to enforce
their rules and relevant or applicable laws, as the
posting of inappropriate content is considered a
major source of liability.
The style of moderation can vary from site to
site, and from platform to platform, as rules
around what UGC is allowed are often set at a
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_44-1
site or platform level and reflect that platform’s
brand and reputation, its tolerance for risk, and the
type of user engagement it wishes to attract. In
some cases, content moderation may take place in
haphazard, disorganized, or inconsistent ways; in
others, content moderation is a highly organized,
routinized, and specific process. Content modera-
tion may be undertaken by volunteers or, increas-
ingly, in a commercial context by individuals or
firms who receive remuneration for their services.
The latter practice is known as commercial con-
tent moderation, or CCM. The firms who own
social media sites and platforms that solicit UGC
employ content moderation as a means to protect
the firm from liability and negative publicity and
to curate and control user experience.
History
The Internet and its many underlying technologies
are highly codified and protocol-reliant spaces
with regard to how data are transmitted within it
(Galloway 2006), yet the subject matter and
nature of content itself has historically enjoyed a
much greater freedom. Indeed, a central claim to
the early promise of the Internet as espoused by
many of its proponents was that it was highly
resistant, as a foundational part of both its architec-
ture and ethos, to censorship of any kind.
Nevertheless, various forms of content moder-
ation occurred in early online communities. Such
content moderation was frequently undertaken by
2 Content Moderation
volunteers and was typically based on the enforce-
ment of local rules of engagement around com-
munity norms and user behavior. Moderation
practices and style therefore developed locally
among communities and their participants and
could inform the flavor of a given community,
from the highly rule-bound to the anarchic: the
Bay Area-based online community the WELL
famously banned only three users in its first
6 years of existence, and then only temporarily
(Turner 2005, p. 499).
In social communities, on the early text-based
Internet, mechanisms to enact moderation were
often direct and visible to the user and could
include demanding that a user alters a contribution
to eliminate offensive or insulting material, the
deletion or removal of posts, the banning of
users (by username or IP address), the use of text
filters to disallow posting of specific types of
words or content, and other overt moderation
actions. Examples of sites of this sort of content
moderation include many Usenet groups, BBSes,
MUDs, listservs, and various early commercial
services.
Motives for people participating in voluntary
moderation activities varied. In some cases, users
carried out content moderation duties for prestige,
status, or altruistic purposes (i.e., for the better-
ment of the community); in others, moderators
received non-monetary compensation, such as
free or reduced-fee access to online services,
e.g., AOL (Postigo 2003). The voluntary model
of content moderation persists today in many
online communities and platforms; one such
high-profile site where volunteer content modera-
tion is used exclusively to control site content is
Wikipedia.
As the Internet has grown into large-scale
adoption and a massive economic engine, the
desire for major mainstream platforms to control
the UGC that they host and disseminate has also
grown exponentially. Early on in the proliferation
of so-called Web 2.0 sites, newspapers and other
news media outlets, in particular, began noticing a
significant problem with their online comments
areas, which often devolved into unreadable
spaces filled with invective, racist and sexist dia-
tribes, name-calling, and irrelevant postings.
These media firms began to employ a variety of
techniques to combat what they viewed as the
misappropriation of the comments spaces, using
in-house moderators, turning to firms that special-
ized in the large-scale management of such inter-
active areas and deploying technological
interventions such as word filter lists or
disallowing anonymous posting, to bring the com-
ments sections under control. Some media outlets
went the opposite way, preferring instead to close
their comments sections altogether.
Commercial Content Moderation and
the Contemporary Social Media
Landscape
The battle with text-based comments was just the
beginning of a much larger issue. The rise of
Friendster, MySpace, and other social media
applications in the early part of the twenty-first
century has given way to more persistent social
media platforms of enormous scale and reach. As
of the second quarter of 2016, Facebook alone
approached two billion users worldwide, all of
whom generate content by virtue of their partici-
pation on the platform. YouTube reported receiv-
ing upwards of 100 hours of UGC video per
minute as of 2014.
The contemporary social media landscape is
therefore characterized by vast amounts of UGC
uploads made by billions of users to massively
popular commercial Internet sites and social
media platforms with a global reach. Mainstream
platforms, often owned by publicly traded firms
responsible to shareholders, simply cannot afford
the risk – legal, financial, and to reputation – that
unchecked UGC could cause. Yet, contending
with the staggering amounts of transmitted data
from users to platforms is not a task that can
currently be addressed reliably and at large scale
by computers. Indeed, making nuanced decisions
about what UGC is acceptable and what is not
currently exceeds the abilities of machine-driven
processes, save for the application of some algo-
rithmically informed filters or bit-for-bit or hash
value matching, which occur at relatively low
levels of computational complexity.
Content Moderation
The need for adjudication of UGC – video- and
image-based content, in particular – therefore
calls on human actors who rely upon their own
linguistic and cultural knowledge and competen-
cies to make decisions about UGC’s appropriate-
ness for a given site or platform. Specifically,
“they must be experts in matters of taste of the
site’s presumed audience, have cultural knowl-
edge about location of origin of the platform and
of the audience (both of which may be very far
removed, geographically and culturally, from
where the screening is taking place), have linguis-
tic competency in the language of the UGC (that
may be a learned or second language for the
content moderator), be steeped in the relevant
laws governing the site’s location of origin and
be experts in the user guidelines and other
platform-level specifics concerning what is and
is not allowed” (Roberts 2016). These human
workers are the people who make up the legions
of commercial content moderators: moderators
who work in an organized way, for pay, on behalf
of the world’s largest social media firms, apps, and
websites who solicit UGC.
CCM processes may take place prior to mate-
rial being submitted for inclusion or distribution
on a site, or they may take place after material has
already been uploaded, particularly on high-
volume sites. Specifically, content moderation
may be triggered as the result of complaints
about material from site moderators or other site
administrators, from external parties (e.g., compa-
nies alleging misappropriation of material they
own; from law enforcement; from government
actors) or from other users themselves who are
disturbed or concerned by what they have seen
and then invoke protocols or mechanisms on a
site, such as the “flagging” of content, to prompt
a review by moderators (Crawford and Gillespie
2016). In this regard, moderation practices are
often uneven, and the removal of UGC may rea-
sonably be likened to censorship, particularly
when it is undertaken in order to suppress speech,
political opinions, or other expressions that
threaten the status quo.
CCM workers are called upon to match and
adjudicate volumes of content, typically at rapid
speed, against the specific rules or community
3
guidelines of the platform for which they labor.
They must also be aware of the laws and statutes
that may govern the geographic or national loca-
tion from where the content emanates, for which
the content is destined, and for where the platform
or site is located – all of which may be distinct
places in the world. They must be aware of the
platform’s tolerance for risk, as well as the expec-
tations of the platform for whether or how CCM
workers should make their presence known.
In many cases, CCM workers may work at
organizational arm’s length from the platforms
they moderate. Some labor arrangements in
CCM have workers located at great distances
from the headquarters of the platforms for which
they are responsible, in places such as the Philip-
pines and India. The workers may be structurally
removed from those firms, as well, via
outsourcing companies who take on CCM con-
tracts and then hire the workers under their aus-
pices, in call center (often called BPO, or business
process outsourcing) environments. Such
outsourcing firms may also recruit CCM workers
using digital piecework sites such as Amazon
Mechanical Turk or Upwork, in which the rela-
tionships between the social media firms, the
outsourcing company, and the CCM worker can
be as ephemeral as one review.
Even when CCM workers are located on-site at
a headquarters of a social media firm, they often
are brought on as contract laborers and are not
afforded the full status, or pay, of a regular full-
time employee. In this regard, CCM work, wher-
ever it takes place in the world and by whatever
name, often shares the characteristic of being rel-
atively low wage and low status as compared to
other jobs in tech. These arrangements of institu-
tional and geographic removal can pose a risk for
workers, who can be exposed to disturbing and
shocking material as a condition of their CCM
work but can be a benefit to the social media
firms who require their labor, as they can distance
themselves from the impact of the CCM work on
the workers. Further, the working conditions,
practices, and existence of CCM workers in social
media are little known to the general public, a fact
that is often by design. CCM workers are fre-
quently compelled to sign NDAs, or
4 Content Moderation
nondisclosure agreements, that preclude them
from discussing the work that they do or the
conditions in which they do it. While social
media firms often gesture at the need to maintain
secrecy surrounding the exact nature of their mod-
eration practices and the mechanisms they used to
undertake them, claiming the possibility of users’
being able to game the system and beat the rules if
armed with such knowledge, the net result is that
CCM workers labor in secret. The conditions of
their work – its pace, the nature of the content they
screen, the volume of material to be reviewed, and
the secrecy – can lead to feelings of isolation,
burnout, and depression among some CCM
workers. Such feelings can be enhanced by the
fact that few people know such work exists,
assuming, if they think of it at all, that algorithmi-
cally driven computer programs take care of social
media’s moderation needs. It is a misconception
that the industry has been slow to correct.
Conclusion
Despite claims and conventional wisdom to the
contrary, content moderation has likely always
existed in some form on the social Internet. As
the Internet’s many social media platforms grow
and their financial, political, and social stakes
increase, the undertaking of organized control of
user expression through such practices as CCM
will likewise only increase. Nevertheless, CCM
remains a little discussed and little acknowledged
aspect of the social media production chain,
despite its mission-critical status in almost every
case in which it is employed. The existence of a
globalized CCM workforce abuts many difficult,
existential questions about the nature of the Inter-
net itself and the principles that have long been
thought to undergird it, particularly, the free
expression and circulation of material, thought,
and ideas. These questions are further compli-
cated by the pressures related to contested notions
of jurisdiction, borders, application and
enforcement of laws, social norms, and mores
that frequently vary and often are in conflict with
each other. The acknowledgement and under-
standing of the history of content moderation
and the contemporary reality of large-scale CCM
is central to many of these core questions of what
the Internet has been, is now, and will be in the
future, and yet the continued invisibility and lack
of acknowledgment of CCM workers by the firms
for which their labor is essential means that such
questions cannot fully be addressed. Neverthe-
less, discussions of moderation practices and the
people who undertake them are essential to the
end of more robust, nuanced understandings of
the state of the contemporary Internet and to better
policy and governance based on those
understandings.
Cross-References
▶ Algorithm
▶ Facebook
▶ Internet
▶ Social Media
▶ Wikipedia
▶ YouTube
Further Readings
Crawford, K., & Gillespie, T. (2016). What is a flag for?
Social media reporting tools and the vocabulary of
complaint. New Media & Society, 18(3), 410–428.
Galloway, A. R. (2006). Protocol: How control exists after
decentralization. Cambridge, MA: MIT Press.
Postigo, H. (2003). Emerging sources of labor on the
internet: The case of America online volunteers. Inter-
national Review of Social History, 48(S11), 205–223.
Roberts, S. T. (2016). Commercial content moderation:
Digital laborers’ dirty work. In S. U. Noble &
B. Tynes (Eds.), The intersectional internet: Race,
sex, class and culture online (pp. 147–160). New
York: Peter Lang.
Turner, F. (2005). Where the counterculture met the new
economy: The WELL and the origins of virtual com-
munity. Technology and Culture, 46(3), 485–512.
C
Crowdsourcing
Heather McIntosh
Mass Media, Minnesota State University,
Mankato, MN, USA
Crowdsourcing is an online participatory culture
activity that brings together large, diverse sets of
people and directs their energies and talents
toward varied tasks designed to achieve specific
goals. The concept draws on the principle that the
diversity of knowledge and skills offered by a
crowd exceeds the knowledge and skills offered
by an elite, select few. For big data, it offers access
to abilities for tasks too complex for computa-
tional analysis. Corporations, government groups,
and nonprofit organizations all use crowdsourcing
for multiple projects, and the crowds consist of
volunteers who choose to engage tasks toward
goals determined by the organizations. Though
these goals may benefit the organizations more
so than the crowds helping them, ideally the ben-
efit is shared between the two. Crowdsourcing
breaks down into basic procedures, the tasks and
their applications, the crowds and their makeup,
and the challenges and ethical questions.
Crowdsourcing follows a general procedure.
First, an organization determines the goal or the
problem that requires a crowd’s assistance in order
to achieve or solve. Next, the organization defines
the tasks needed from the crowd in order to fulfill
its ambitions. After, the organization seeks the
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_47-1
crowd’s help, and the crowd engages the tasks.
In selective crowdsourcing, the best solution from
the crowd is chosen, while in integrative
crowdsourcing, the crowd’s solutions become
worked into the overall project in a useful manner.
Working online is integral to the
crowdsourcing process. It allows the gathering
of diverse individuals who are geographically dis-
persed to “come together” for working on the
projects. The tools the crowds need to engage
the tasks also appear online. Since using an orga-
nization’s own tools can prove too expensive for
big data projects, organizations sometimes use
social networks for recruitment and task fulfill-
ment. The documentary project Life in a Day, for
example, brought together video footage from
people’s everyday lives from around the world.
When possible, people uploaded their footage to
YouTube, a video-sharing platform. To address
the disparities of countries without access to
digital production technologies and the Internet,
the project team sent cameras and memory storage
cards through the mail. Other services assist with
recruitment and tasks. LiveWork and Amazon
Mechanical Turk are established online service
marketplaces, while companies such as
InnoCentive and Kaggle offer both the crowds
and the tools to support an organization’s project
goals.
Tasks vary depending on the project’s goals,
and they vary in structure, interdependence, and
commitment. Some tasks follow definite bound-
aries or procedures, while others are open-ended.
2 Crowdsourcing
Some tasks depend on other tasks for completion,
while others stand alone. Some tasks require but a
few seconds, while others demand more time and
mental energy. More specifically, tasks might
include finding and managing information, ana-
lyzing information, solving problems, and pro-
ducing content. With big data, crowds may enter,
clean, and validate data. The crowds may even
collect data, particularly geospatial data, which
prove useful for search and rescue, land manage-
ment, disaster response, and traffic management.
Other tasks might include transcription of audio or
visual data and tagging.
When bringing crowdsourcing to big data, the
crowd offers skills that benefit through matters of
judgment, contexts, and visuals – skills that
exceed computational models. In terms of judg-
ment, people can determine the relevance of items
that appear within a data set, identify similarities
among items, or fill in holes within the set. In
terms of contexts, people can identify the situa-
tions surrounding the data and how those situa-
tions influence them. For example, a person can
determine the difference between the Statue of
Liberty on Ellis Island in New York and the rep-
lica on The Strip in Las Vegas. The contexts then
allow determination of accuracy or ranking, such
as in this case differentiating the real from the
replica. People also can determine more in-depth
relationships among data within a set. For exam-
ple, people can better decide the accuracy of
search engine terms and results matches, deter-
mine better the top search result, or even predict
other people’s preferences.
Properly managed crowdsourcing begins
within an organization that has clear goals for its
big data. These organizations can include govern-
ment, corporations, and nonprofit organizations.
Their goals can include improving business prac-
tices, increasing innovations, decreasing project
completion times, developing issue awareness,
and solving social problems. These goals fre-
quently involve partnerships that occur across
multiple entities, such as government or corpora-
tions partnering with not-for-profit initiatives.
At the federal level and managed through Mas-
sachusetts Institute for Technology’s Center for
Collective Intelligence, Climate CoLab brings
together crowds to analyze issues related to global
climate change, registering more than 14,000
members who participate in a range of contests.
Within the contests, members create and refine
proposals that offer climate change solutions.
The proposals then are evaluated by the commu-
nity and, through voting, recommended for imple-
mentation. Contest winners presented their
proposals to those who might implement them at
a conference. Some contests build their initiatives
on big data, such as Smart Mobility, which relies
on mobile data for tracking transportation and
traveling patterns in order to suggest ways for
people to reduce their environmental impacts
while still getting where they want to go.
Another government example comes from the
city of Boston, wherein a mobile app called Street
Bump tracks and maps potential potholes
throughout the city in order to guide crews toward
fixing them. The crowdsourcing for this initiative
comes from two levels. One, the information gath-
ered from the app helps city crews do their work
more efficiently. Two, the app’s first iteration
reported too many false positives, leading crews
to places where no potholes existed. The city
worked with a crowd drawn together through
InnoCentive to improve the app and its efficiency,
with the top suggestions coming from a hackers
group, a mathematician, and a software engineer.
Corporations also use crowdsourcing to work
with their big data. AOL needed help with
cataloging the content on its hundreds of thou-
sands web pages, specifically the videos and their
sources, and turned to crowdsourcing as a means
to expedite and streamline the project’s costs.
Between 2006 and 2010, Netflix, an online
streaming and mail DVD distributor, sought help
with perfecting its algorithm for predicting user
ratings of films. The company developed a contest
with a $1 million dollar prize, and for the contest,
it offered data sets consisting of multiple million
units for analysis. The goal was to beat Netflix’s
current algorithm by 10%, which one group
achieved and took home the prize.
Not-for-profit groups also incorporate
crowdsourcing as part of their initiatives. AARP
Foundation, which works on behalf of older
Americans, used crowdsourcing to tackle such
Crowdsourcing
issues as eliminating food insecurity and food
deserts (areas where people do not have conve-
nient or close access to grocery stores). Humani-
tarian Tracker crowdsources data from people “on
the ground” about issues such as disease, human
rights violations, and rape. Focusing particularly
on Syria, Humanitarian Tracker aggregates these
data into maps that show the impacts of systematic
killings, civilian targeting, and other human tolls.
Not all crowdsourcing and big data projects
originate within these organizations. For example,
Galaxy Zoo demonstrates the expanses of both
big data and crowds. The project asked people to
classify a data set of one million galaxies into
three categories: elliptical, merger, and spiral. By
the project’s completion, 150,000 people had con-
tributed 50 million classifications. The data fea-
ture multiple independent classifications as well,
adding reliability. The largest crowdsourcing pro-
ject involved searching satellite images for wreck-
age from Malaysia Airlines flight MH370, which
went missing in March 2014. Millions of people
searched for signs among the images made avail-
able by Colorado-based Digital Globe. The
amount of crowdsourcing traffic even crashed
websites.
Not all big data crowdsourced projects suc-
ceed, however. One example is the Google Flu
tracker. The tracker included a map to show the
disease’s spread throughout the season. It was
later revealed that the tracker overestimated the
expanse of the flu spreading, predicting twice as
much as actually occurred.
In addition to their potentially not succeeding,
another drawback to these projects is their overall
management, which tends to be time-consuming
and difficult. Several companies attempt to fulfill
this role. InnoCentive and Kaggle use crowds to
tackle challenges brought to them by industries,
government, and nonprofit organizations. Kaggle
in particular offers almost 150,000 data scientists
– statisticians – to help companies develop more
efficient predictive models, such as deciding the
best order in which to show hotel rooms for a
travel app or guessing which customers would
leave an insurance company within a year. Both
InnoCentive and Kaggle run their crowdsourcing
activities as contests or competitions as these are
3
often tasks that require a higher time and mental
commitment than others.
Crowds bring wisdom to crowdsourced tasks
on big data through their diversity of skills and
knowledge. Determining the makeup of that
crowd proves more challenging, but one study of
Mechanical Turk offers some interesting findings.
It found that US females outnumber males by 2 to
1 and that many of the workers hold bachelor’s
and even master’s degrees. Most live in small
households of two or fewer people, and most use
the crowdsourcing work to supplement their
household incomes as opposed to being the pri-
mary source of income.
Crowd members choose the projects on which
they want to work, and multiple factors contribute
to their motivations for joining a project and
staying with it. For some working on projects
that offer no further incentive to participate, the
project needs to align with their interests and
experience so that they feel they can make a
contribution. Others enjoy connecting with other
people, engaging in problem-solving activities,
seeking something new, learning more about the
data at hand, or even developing a new skill. Some
projects offer incentives such as prize money or
top-contributor status. For some entertainment
motivates them to participate in that the tasks
offer a diversion. For others, though, working on
crowdsourced projects might be addiction as well.
While crowdsourcing offers multiple benefits
for the processing of big data, it also draws some
criticism. A primary critique centers on the notion
of labor, wherein the crowd contributes knowl-
edge and skills for little-to-no pay, while the orga-
nization behind the data stands to gain much more
financially. Some crowdsourcing sites offer low
cash incentives for the crowd participants, and in
doing so, they sidestep labor laws requiring min-
imum wage and other worker benefits. Opponents
of this view cite that the labor involved frequently
requires menial tasks and that the labor faces no
obligation in completing the assigned tasks. They
also cite that crowd participants engage the tasks
because they enjoy doing so.
Ethical concerns come back to the types of
crowdsourced big data projects and the intentions
behind them, such as information gathering,
4 Crowdsourcing
surveillance, and information manipulation. With
information manipulation, for example, crowd
participants might create fake product reviews
and ratings for various web sites, or they might
crack anti-spam devices such as CAPTCHAs
(Completely Automated Public Turing test to tell
Computers and Humans Apart). Other activities
involve risks and possible violations of other indi-
viduals, such as gathering large amounts of per-
sonal data for sale. Overall, the crowd participants
remain unaware that they are engaging in
unethical activities.
Cross-References
▶ Amazon
▶ Cell Phone Data
▶ Netflix
▶ Predictive Analytics
Further Readings
Brabham, D. C. (2013). Crowdsourcing. Cambridge, MA:
MIT Press.
Howe, J. (2009). Crowdsourcing: why the power of the
crowd is driving the future of business. New York:
Crown.
Nakatsu, R. T., Grossman, E. B., & Charalambos, L. I.
(2014). A taxonomy of crowdsourcing based on task
complexity. Journal of Information Science, 40(6),
823–834.
Shirky, C. (2009). Here comes everybody: the power of
organizing without organizations. New York: Penguin.
Surowiecki, J. (2005). The wisdom of crowds. New York:
Anchor.
C
Curriculum, Higher Education, and
Social Sciences
Stephen T. Schroth
Department of Early Childhood Education,
Towson University, Baltimore, MD, USA
Big data, which has revolutionized many practices
in business, government, healthcare, and other
fields, promises to radically change the curricu-
lum offered in many of the social sciences. Big
data involves the capture, collection, storage,
collation, search, sharing, analysis, and visualiza-
tion of enormous data sets so that this information
may be used to spot trends, to prevent problems,
and to proactively engage in activities that make
success more likely. The social sciences, which
includes fields as disparate as anthropology, eco-
nomics, education, political science, psychology,
and sociology, is a disparate area, and the tools of
big data are being embraced differently within
each. The economic demands of setting up sys-
tems that permit the use of big data in higher
education have also hindered some efforts to use
these processes, as these institutions often lack the
infrastructure necessary to proceed with such
efforts. Opponents of the trend toward using big
data tools for social science analyses often stress
that while these tools may provide helpful for
certain analyses, it is also crucial for students to
receive training in more traditional methods. As
equipment and training concerns are overcome,
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_50-1
however, the use of big data social sciences
departments at colleges and universities seems
likely to increase.
Background
A variety of organizations, including government
agencies, businesses, colleges, universities,
schools, hospitals, research centers, and others,
collect data regarding their operations, clients,
students, patients, and findings. Disciplines
within the social sciences, which are focused
upon society and the relationships among individ-
uals within a society, often use such data to inform
studies related to these. Such a volume of data has
been generated, however, that many social scien-
tists have found it impossible to use this in their
work in a meaningful manner. The emergence of
computers and other electronic forms of data stor-
age has resulted in more data than ever before
being collected, especially during the last two
decades of the twentieth century. This data was
generally stored in separate databases. This
worked to make data from different sources inac-
cessible to most social science users. As a result,
much of the information that could potentially be
obtained from such sources was not used.
Over the past decade and a half, many busi-
nesses became increasingly interested in making
use of data they had but did not use regarding
customers, processes, sales, and other matters.
Big data became seen as a way of organizing
2 Curriculum, Higher Education, and Social Sciences
and using the numerous sources of information in
ways that could benefit organizations and individ-
uals. Infonomics, the study of how information
could be used for economic gain, grew in impor-
tance as companies and organizations worked to
make better use of the information they possessed,
with the end goal being to use it in ways that
increased profitability. A variety of consulting
firms and other organizations began working
with large corporations and organizations in an
effort to accomplish this. They defined big data
as consisting of three “v”s, volume, variety, and
velocity.
Volume, as used in this context, refers to the
increase in data volume caused by technological
innovation. This includes transaction-based data
that has been gathered by corporations and orga-
nizations over time but also includes unstructured
data that derives from social media and other
sources as well as increasing amounts of sensor
and machine-to-machine data. For years, exces-
sive data volume was a storage issue, as the cost of
keeping much of this information was prohibitive.
As storage costs have decreased, however, cost
has diminished as a concern. Today, how best to
determine relevance within large volumes of data
and how best to analyze data to create value have
emerged as the primary issues facing those wish-
ing to use it.
Velocity refers to the amount of data streaming
in at great speed raises the issue of how best to
deal with this in an appropriate way. Technologi-
cal developments, such as sensors and smart
meters, and client and patient needs emphasize
the necessity of overseeing and handling inunda-
tions of data in near real time. Responding to data
velocity in a timely manner represents an ongoing
struggle for most corporations and other organi-
zations. Variety in the types of formats in which
data today comes to organizations presents a prob-
lem for many. Data today includes that in struc-
tured numeric forms which is stored in traditional
databases but has grown to include information
created from business applications, e-mails, text
documents, audio, video, financial transactions,
and a host of others. Many corporations and orga-
nizations struggle with governing, managing, and
merging different forms of data.
Some have added two additional criteria to
these: variability and complexity. Variability con-
cerns the potential inconsistency that data can
demonstrate at times, which can be problematic
for those who analyze the data. Variability can
hamper the process of managing and handling
the data. Complexity refers the intricate process
that data management involves, in particular when
large volumes of data come from multiple and
disparate sources. For analysts and other users to
fully understand the information that is contained
in these data, they must be must first be connected,
correlated, and linked in a way that helps users
make sense of them.
Big Data Comes to the Social Sciences
Colleges, universities, and other research centers
have tracked the efforts of the business world to
use big data in a way that helped to shape organi-
zational decisions and increase profitability. Many
working in the social sciences were intrigued by
this process, as they saw it as a useful tool that
could be used in their own research. The typical
program in these areas, however, did not provide
students, be they at the undergraduate or graduate
level, the training necessary to engage in big data
research projects. As a result, many programs in
the social sciences have altered their curriculum in
an effort to assure that researchers will be able to
carry out such work. For many programs across
the social sciences that have pursued curricular
changes that will enable students to engage in
big data research, these changes have resulted in
more coursework in statistics, networking, pro-
gramming, analytics, database management, and
other related areas. As many programs already
required a substantial number of courses in other
areas, the drive toward big data competency has
required many departments to reexamine the work
required of their students.
This move toward more coursework that sup-
ports big data has not been without its critics.
Some have suggested that changes in curricular
offerings have come at a high cost, with students
now being able to perform certain operations
involved with handling data but unable to
Curriculum, Higher Education, and Social Sciences
competently perform other tasks, such as
establishing a representative sample or composing
a valid survey. These critics also suggest that
while big data analysis has been praised for offer-
ing tremendous promise, in truth the analysis
performed is shallow, especially when compared
to that done with smaller data sets. Indeed, repre-
sentative sampling would negate the need for, and
expense of, many big data projects. Such critics
suggest that increased emphasis in the curriculum
should focus on finding quality, rather than big,
data sources and that efforts to train students to
load, transform, and extract data is sublimating
other more important skills.
Despite these criticisms, changes to the social
sciences curriculum are occurring at many insti-
tutions. Many programs now require students to
engage in work that examines practices and para-
digms of data science, which would provide stu-
dents with a grounding in the core concepts of
data science, analytics, and data management.
Work in algorithms and modeling, which provide
proficiency in basic statistics, classification, clus-
ter analysis, data mining, decision trees, experi-
mental design, forecasting, linear algebra, linear
and logistic regression, market basket analysis,
predictive modeling, sampling, text analytics,
summarization, time series analysis, and
unsupervised learning constrained optimization,
is also an area of emphasis in many programs.
Students require exposure to tools and platforms,
which provides proficiency in modeling, develop-
ment and visualization tools to be used on big data
projects, as well as knowledge about the platforms
used for execution, governance, integration, and
storage of big data. Finally, work with applica-
tions and outcomes, which emphasize the primary
applications of data science to one’s field, and
3
how it interacts with disciplinary issues and con-
cerns have been emphasized by many programs.
Some programs have embraced big data tools
but suggested that not every student needs mas-
tery of them. Instead, these programs have
suggested that big data has emerged as a field of
its own and that certain students should be trained
in these skills so that they can work with others
within the discipline to provide support for those
projects that require big data analysis. This
approach offers more incremental changes to the
social science curricular offerings, as it would
require fewer changes for most students yet still
enable departments to produce scholars who are
equipped to engage in research projects involving
big data.
Cross-References
▶ Big Data Quality
▶ Correlation vs. Causation
▶ Curriculum, Higher Education, Business
▶ Curriculum, Higher Education, Humanities
▶ Education
▶ Public Administration/Government
Further Readings
Foreman, J. W. (2013). Data smart: Using data science to
transform information into insight. Hoboken: Wiley.
Lane, J. E., & Zimpher, N. L. (2014). Building a smarter
university: Big data, innovation, and analytics.
Albany: The State University of New York Press.
Mayer-Schönberger, V., & Cukier, K. (2013). Big data.
New York: Mariner Books.
Siegel, E. (2013). Predictive analytics: The power to pre-
dict who will click, buy, lie, or die. Hoboken: Wiley.
D
Data Science
Lourdes S. Martinez
School of Communication, San Diego State
University, San Diego, CA, USA
Data science has been defined as the structured
study of data for the purpose of producing
knowledge. Going beyond simply using data,
data science revolves around extracting actionable
knowledge from said data. Despite this definition,
confusion exists surrounding the conceptual
boundaries of data science in large part due to its
intersection with other concepts, including big
data and data-driven decision making. Given that
increasingly unprecedented amounts of data are
generated and collected every day, the growing
importance of the data science field is undeniable.
As an emerging area of research, data science
holds promise for optimizing performance of
companies and organizations. The implications
of advances in data science are relevant for fields
and industries spanning an array of domains.
Defining Data Science
The basis of data science centers around
established guiding principles and techniques
that help organize the process of drawing out
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_60-1
information and insights from data. Conceptually,
data science closely resembles data mining, or a
process relying on technologies that implement
these techniques in order to extract insights from
data. According to Dhar, Jarke, and Laartz, data
science seeks to move beyond simply explaining
a phenomenon. Rather its main purpose is to
answer questions that explore and uncover action-
able knowledge that informs decision making
or predicts outcomes of interest. As such, most
of the challenges currently facing data science
emanate from properties of big data and the size
of its datasets, which are so massive they require
the use of alternative technologies for data
processing.
Given these characteristics, data science as a
field is charged with navigating the abundance of
data generated on a daily basis, while supporting
machine and human efforts in using big data to
answer the most pressing questions facing indus-
try and society. These aims point toward the
interdisciplinary nature of data science.
According to Loukides, the field itself falls
inside the area where computer programming
and statistical analysis converge within the con-
text of a particular area of expertise. However,
data science differs from statistics in its holistic
approach to gathering, amassing, and examining
user data to generate data products. Although
several areas across industry and society are
beginning to explore the possibilities offered by
2 Data Science
data science, the idea of what constitutes data
science remains nebulous.
Controversy in Defining the Field
According to Provost and Fawcett, one reason
why data science is difficult to define relates to
its conceptual overlap with big data and data-
driven decision making. Data-driven decision
making represents an approach characterized by
the use of insights gleaned through data analysis
for deciding on a course of action. This form of
decision making may also incorporate varying
amounts of intuition, but does not rely solely on
it for moving forward. For example, a marketing
manager faced with a decision about how much
promotional effort should be invested in a partic-
ular product has the option of solely relying on
intuition and past experiences, or using a combi-
nation of intuition and knowledge gained from
data analysis. The latter represents the basis for
data-driven decision making. At times, however,
in addition to enabling data-driven decision mak-
ing, data science may also overlap with data-
driven decision making. The case of automated
online recommendations of products based on
user ratings, preferences, and past consumer
behavior is an example of where the distinction
between data science and data-driven decision
making is less clear.
Similarly, differentiating between the concepts
of big data and data science becomes murky when
considering that approaches used for processing
big data overlay with the techniques and princi-
ples used to extract knowledge and espoused by
data science. This conceptual intersection exists
where big data technologies meet data mining
techniques. For example, technologies such as
Apache™ Hadoop ® which are designed to store
and process large-scale data can also be used to
support a variety of data science efforts related to
solving business problems, such as fraud detec-
tion, and social problems, such as unemployment
reduction. As the technologies associated with big
data are also often used to apply and bolster
approaches to data mining, the boundary between
where big data ends and data science begins
continues to be imprecise.
Another source of confusion in defining data
science stems from the absence of formalized
academic programs in higher education. The
lack of these programs exists in part due to chal-
lenges in launching novel programs that cross
disciplines and the natural pace at which these
programs are implemented within the academic
environment. Although several institutions within
higher education now recognize the importance
of this emerging field and the need to develop
programs that fulfill industry’s need for practi-
tioners of data science, the result up to now has
been to leave the task for defining the field to data
scientists.
Data scientists currently occupy an enviable
position as among the most coveted employees
for twenty-first-century hiring according to Dav-
enport and Patil. They describe data scientists as
professionals, usually of senior-level status, who
are driven by curiosity and guided by creativity
and training to prepare and process big data.
Their efforts are geared toward uncovering find-
ings that solve problems in both private and
public sectors. As businesses and organizations
accumulate greater volumes of data at faster
speeds, Davenport and Patil predict the need for
data scientists will to continue in a very steep and
upward trajectory.
Opportunities in Data Science
Several sectors stand to gain from the explosion in
big data and acquisition of data scientists to ana-
lyze and extract insights from it. Chen, Chiang,
and Storey note the opportunities inherent through
data science for various areas. Beginning with e-
commerce and the collection of market intelli-
gence, Chen and colleagues focus on the develop-
ment of product recommendation systems via e-
commerce vendors such as Amazon that are com-
prised of consumer-generated data. These product
recommendation systems allow for real-time
access to consumer opinion and behavior data in
record quantities. New data analytic techniques to
Data Science
harness consumer opinions and sentiments have
accompanied these systems, which can help busi-
nesses become better able to adjust and adapt
quickly to needs of consumers. Similarly, in the
realm of e-government and politics, a multitude of
data science opportunities exist for increasing the
likelihood for achieving a range of desirable out-
comes, including political campaign effective-
ness, political participation among voters, and
support for government transparency and
accountability. Data science methods used to
achieve these goals include opinion mining, social
network analysis, and social media analytics.
Public safety and security represents another
area that Chen and colleagues observe has pros-
pects for implementing data science. Security
remains an important issue for businesses and
organizations in a post-September 11th 2001 era.
Data science offers unique opportunities to pro-
vide additional protections in the form of security
informatics against terrorist threats to transporta-
tion and key pieces of infrastructure (including
cyberspace). Security informatics uses a three-
pronged approach coordinating organizational,
technological, and policy-related efforts to
develop data techniques designed to promote
international and domestic security. The use of
data science techniques such as crime data min-
ing, criminal network analysis, and advanced
multilingual social media analytics can be instru-
mental in preventing attacks as well as
pinpointing whereabouts of suspected terrorists.
Another sector flourishing with the rise of data
science is science and technology (S&T). Chen
and colleagues note that several areas within S&T,
such as astrophysics, oceanography, and geno-
mics, regularly collect data through sensor sys-
tems and instruments. The result has been an
abundance of data in need of analysis, and the
recognition that information sharing and data ana-
lytics must be supported. In response, the National
Science Foundation (NSF) now requires the sub-
mission of a data management plan with every
funded project. Data-sharing initiatives such as
the 2012 NSF Big Data program are examples of
government endeavors to advance big data
3
analytics for science and technology research.
The iPlant Collaborative represents another
NSF-funded initiative that relies on cyber infra-
structure to instill skills related to computational
techniques that address evolving complexities
within the field of plant biology among emerging
biologists.
The health field is also flush with opportunities
for advances using data science. According to
Chen and colleagues, opportunities for this field
are rising in the form of massive amounts of
health- and healthcare-related data. In addition to
data collected from patients, data are also gener-
ated through advanced medical tools and instru-
mentation, as well as online communities formed
around health-related topics and issues. Big data
within the health field is primarily comprised of
genomics-based data and payer-provider data.
Genomics-based data encompasses genetic-
related information such as DNA sequencing.
Payer-provider data comprises information col-
lected as part of encounters or exchanges between
patients and the healthcare system, and includes
electronic health records and patient feedback.
Despite these opportunities, Miller notes that
application of data science techniques to health
data remains behind that of other sectors, in part
due to a lack of initiatives that leverage scalable
analytical methods and computational platforms.
In addition, research and ethical considerations
surrounding privacy and protection of patients’
rights in the use of big data present some chal-
lenges to full utilization of existing health data.
Challenges to Data Science
Despite the enthusiasm for data science and the
potential application of its techniques for solving
important real-world problems, there are some
challenges to full implementation of tools from
this emerging field. Finding individuals with the
right training and combination of skills to become
data scientists represents one challenge. Daven-
port and Pital discuss the shortage of data scien-
tists as a case in which demand has grossly
4 Data Science
exceeded supply, resulting in intense competition
among organizations to attract highly sought-after
talent.
Concerns related to privacy represent
another challenge to data science analysis of big
data. Errors, mismanagement, or misuse of data
(specifically data that by its nature is traceable
to individuals) can lead to potential problems.
One famous incident involved Target correctly
predicting the pregnancy status of a teenaged girl
before her father was aware of the situation,
resulting in wide media coverage over issues
equating big data with “Big Brother.” This per-
ception of big data may cause individuals to
become reluctant to provide their information, or
choose to alter their behavior when they suspect
they are being tracked, potentially undermining
the integrity of data collected.
Data science has been characterized as a field
concerned with the study of data for the purpose
of gleaning insight and knowledge. The primary
goal of data science is to produce knowledge
through the use of data. Although this definition
provides clarity to the conceptualization of data
science as a field, there persists confusion as to
how data science differs from related concepts
such as big data and data-driven decision making.
The future of data science appears very bright, and
as the amount and speed with which data is col-
lected continues to increase, so too will the need
for data scientists to harness the power of big data.
The opportunities for using data science to maxi-
mize corporate and organizational performance
cut across several sectors and areas.
Cross-References
▶ Big Data
▶ Data Mining
▶ Data Scientist
▶ Data-Driven Decision-Making
Further Readings
Chen, H. (2006). Intelligence and security informatics for
international security: Information sharing and data
mining. New York: Springer Publishers.
Chen, H. (2009). AI, E-government, and politics 2.0. IEEE
Intelligent Systems, 24(5), 64–86.
Chen, H. (2011). Smart health and wellbeing. IEEE Intel-
ligent Systems, 26(5), 78–79.
Chen, H., Chiang, R. H. L., & Storey, V. C. (2012). Busi-
ness intelligence and analytics: From big data to big
impact. MIS Quarterly, 36(4), 1165–1188.
Davenport, T. H., & Patil, D. J. (2012). Data scientist: The
sexiest job of the 21st century. Harvard Business
Review, 90, 70–76.
Dhar, V., Jarke, M., & Laartz, J. (2014). Big data. Business
& Information Systems Engineering, 6(5), 257–259.
Hill, K. (2012). How target figured out a teen girl was
pregnant before her father did. Forbes magazine.
Forbes Magazine.
Loukides, M. (2011). What is data science? The future
belongs to the companies and people that turn data
into products. Sebastopol: O’ Reilly Media.
Miller, K. (2012). Big data analytics in biomedical
research. Biomedical Computation Review, 2, 14–21.
Provost, F., & Fawcett, T. (2013). Data science and its
relationship to big data and data-driven decision mak-
ing. Big Data, 1(1), 51–59.
Wactlar, H., Pavel, M., & Barkis, W. (2011). Can computer
science save healthcare? IEEE Intelligent Systems, 26
(5), 79–83.
I
Industrial and Commercial Bank of
China
Jing Wang1 and Aram Sinnreich2
1
School of Communication and Information,
Rutgers University, New Brunswick, NJ, USA
2
School of Communication, American University,
Washington, DC, USA
The Industrial and Commercial Bank of
China (ICBC)
The Industrial and Commercial Bank of China
(ICBC) was the first state-owned commercial
bank of the People’s Republic of China (PRC). It
was founded on January 1st, 1984, and is
headquartered in Beijing. In line with Deng
Xiaoping’s economic reform policies launched
in the late 1970s, the State Council (chief admin-
istrative authority of China) decided to relay all
the financial businesses related to industrial and
commercial sectors from the central bank
(People’s Bank of China) to ICBC (China Indus-
trial Map Committee 2016). This decision made
in September 1983 is considered a landmark event
in the evolution of China’s increasingly special-
ized banking system (Fu and Hefferman 2009).
While the government retains control over ICBC,
the bank began to take on public shareholders in
October, 2006. As of May 2016, ICBC was
ranked as the world’s largest public company by
Forbes “Global 2000.” (Forbs Ranking 2016)
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_113-1
With its combination of state and private owner-
ship, state governance, and commercial dealings,
ICBC serves as a perfect case study to examine the
transformation of China’s financial industry.
Big data collection and database construction
are fundamental to ICBC’s management strate-
gies. Beginning in the late 1990s, ICBC paid
unprecedented attention on the implication of
information technology (IT) in their daily opera-
tions. Several branches adopted computerized
input and internet communication of transactions,
which had previously relied upon manual prac-
tices by bank tellers. Technological upgrades
increased work efficiency and also helped to
save labor costs. More importantly, compared to
the labor-driven mechanism, the computerized
system was more effective for retrieving data
from historical records and analyzing these data
for business development. At the same time, it
became easier for the headquarters to control the
local branches by checking digitalized informa-
tion records. Realizing the benefits of these
informatization and centralization tactics, the
head company assigned its Department of Infor-
mation Management to develop a centralized
database collecting data from every single branch.
This database is controlled and processed by
ICBC headquarters but is also available for use
by local branches with the permission of top
executives.
In this context, “big data” refers to all the
information collected from ICBC’s daily opera-
tions and can be divided into two general
2 Industrial and Commercial Bank of China
categories: “structured data” (which is organized
according to preexisting database categories) and
“unstructured data” (which does not) (Davenport
and Kim 2013). For example, a customer’s
account information is typically structured data.
The branch has to input the customer’s gender,
age, occupation, etc., into the centralized network.
This information then flows into the central data-
base which is designed specifically to accommo-
date it. Any data other than the structured data will
be stored as raw data and preserved without pro-
cessing. For example, the video recorded at a local
branch’s business hall will be saved with only a
date and a location label. Though “big data” in
ICBC’s informational projects refers to both struc-
tured and unstructured data, the former is the core
of ICBC’s big data strategy and is primarily used
for data mining.
Since the late 1990s, ICBC has invested in big
data development with increasingly large eco-
nomic and human resources. On September 1st,
1999, ICBC inaugurated its “9991” project, which
aimed at centralizing the data collected from
ICBC branches nationwide. This project took
more than 3 years to accomplish its goal. Begin-
ning in 2002, all local branches were connected to
ICBC’s Data Processing Center in Shanghai – a
data warehouse with a 400 terabyte (TB) capacity.
The center’s prestructured database enables ICBC
headquarters to process and analyze data as soon
as they are generated, regardless of the location.
With its enhanced capability in storing and man-
aging data, ICBC also networked and digitized its
local branch operations. Tellers are able to input
customer information (including their profiles and
transaction records) into the national Data Center
through their computers at local branches. These
two-step strategies of centralization and digitiza-
tion allow ICBC to converge local operations on
one digital platform, which intensifies the head-
quarters’ control over national businesses. In
2001, ICBC launched another data center in
Shenzhen, China, which is in charge of the big
data collected from its oversea branches. ICBC’s
database thus enables the headquarters’ control
over business and daily operations globally and
domestically.
By 2014, ICBC’s Data Center in Shanghai had
collected more than 430 million individual cus-
tomers’ profiles and more than 600,000 commer-
cial business records. National transactions –
exceeding 215 million on daily basis – have all
been documented at the Data Center. Data storage
and processing on such a massive scale cannot be
fulfilled without a powerful and reliable computer
system. The technology infrastructure supporting
ICBC’s big data strategy consists of three major
elements: hardware, software, and cloud comput-
ing. Suppliers are both international and domestic,
including IBM, Teradata, and Huawei.
Further, ICBC has also invested in data backup
to secure its database infrastructure and data
records. The Shanghai Data Center has a backup
system in Beijing which can record data when the
main server fails to work properly. The Beijing
data center serves as a redundant system in case
the Shanghai Data Center fails. It only takes less
than 30 s to switch between two centers. To speed
data backup and minimize data loss in significant
disruptive events, ICBC undertakes multiple
disaster recovery (DR) tests on a regular basis.
The accumulation and construction of big data
is significant for ICBC’s daily operation in three
respects. First of all, big data allows ICBC to
develop its customers’ business potential through
a so-called “single-view” approach. A customer’s
business data collected from one of ICBC’s
35 departments are available for all the other
departments. By mining the shared database,
ICBC headquarters is able to evaluate both a
customer’s comprehensive value and the overall
quality of all existing customers. Cross depart-
mental business has also been propelled (e.g.,
the Credit Card Department may share business
opportunities with the Savings Department). Sec-
ond, the ICBC marketing department has been
using big data for email-based marketing
(EBM). Based on the data collected from
branches, the Marketing and Business Develop-
ment Department is able to locate their target
customers and follow up with customized market-
ing/advertising information via customized email
communications. This data-driven marketing
approach is increasingly popular among financial
institutions in China. Third, customer
Industrial and Commercial Bank of China
management systems rely directly on big data. All
customers have been segmented into six levels,
ranging from “one star” to “seven stars,” (one star
and two stars fall into a single segment which
indicates the customers’ savings or investment
levels at ICBC). “Seven Stars” clients have the
highest level of credit and enjoy the best benefits
provided by ICBC.
Big data has influenced ICBC’s decision-
making on multiple levels. For local branches,
market insights are available at a lower cost. Con-
sumer data generated and collected at local
branches have been stored on a single platform
provided and managed by the national data center.
For example, a branch in an economically devel-
oping area may predict demand for financial prod-
ucts by checking the purchase data from branches
in more developed areas. The branch could also
develop greater insights regarding the local con-
sumer market by examining data from multiple
branches in the geographic area. For ICBC head-
quarters, big data fuels a dashboard through which
it monitors ICBC’s overall business and is alerted
to potential risks. Previously, individual depart-
ments used to manage their financial risk through
their own balance sheets. This approach was
potentially misleading and even dangerous for
ICBC’s overall risk profile. A given branch pro-
viding many loans and mortgages may be consid-
ered to be performing well, but if a large number
of branches overextended themselves, the emer-
gent financial consequences might create a crisis
for ICBC or even for the financial industry at
large. Consequently, today, a decade after its
data warehouse construction, ICBC considers
big data indispensable in providing a holistic per-
spective, mitigating risk for its business and
development strategies.
To date, ICBC has been a pioneer in big data
construction among all the financial enterprises in
China. It was the first bank to have all local data
centralized in a single database. As the Director of
ICBC’s Informational Management Department
claimed in 2014, ICBC has the largest Enterprise
Database (EDB) in China.
Parallel to its aggressive strategies in big data
construction, the issue of privacy protection has
always been a challenge in ICBC’s customer data
3
collection and data mining. The governing poli-
cies primarily regulate the release of data from
ICBC to other institutions, yet the protection of
customer privacy within ICBC itself has rarely
been addressed. According to the central bank’s
Regulation on the Administration of the Credit
Investigation Industry issued by the State Council
in 2013, interbank sharing of customer informa-
tion is forbidden. Further, a bank is not eligible to
release customer information to its nonbanking
subsidiaries. For example, the fund management
company (ICBCCS) owned by ICBC is not allo-
wed access customer data collected from ICBC
banks. The only situation in which ICBC could
release customer data to a third party is when such
information has been linked to the official inquiry
by law enforcement. These policies prevent con-
sumer information from leaking to other compa-
nies for business purposes. Yet, the policies have
also affirmed the fact that ICBC has full owner-
ship of the customer information, thus giving
ICBC greater power to use the data in its own
interests.
Cross-References
▶ Data Driven Marketing
▶ Data Mining
▶ Data Warehouse
▶ Hardware
▶ Structured Data
Further Reading
China Industrial Map Editorial Committee, China Eco-
nomic Monitoring & Analysis Center & Xinhua Hold-
ings. 2016. Industrial map of China’s financial sectors,
Chapter 6. World Scientific Publishing.
Davenport, T., & Kim, J. (2013). Keeping up with the
quants: Your guide to understanding and using analyt-
ics. Boston: Harvard Business School Publishing.
Fu, M., & Hefferman, S. (2009). The effects of reform on
China’s bank structure and performance. Journal of
Banking & Finance, 33(1), 39–52.
Forbs Ranking (2016). The World’s Biggest Public Com-
pany. Retrieved from https://www.forbes.com/compa
nies/icbc/
I
Information Commissioner, United
Kingdom
Ece Inan
Provost&Academic Dean, Girne American
University Canterbury, Canterbury, UK
The Information Commissioner’s Office (ICO) is
the UK’s independent public authority which is
responsible for data protection mainly in England,
Scotland, Wales, and Northern Ireland; and also
ICO has right to conduct some international
duties. ICO was firstly set up to uphold informa-
tion rights by implementing the Data Protection
Act 1984. The ICO declared their mission state-
ment as to promote respect for the private lives of
individuals and in particularly, for the privacy of
their information by implementing the Data Pro-
tection Act 1984 and also influencing national and
international thinking on privacy and personal
information.
ICO enforces and oversees all the data protec-
tion issues by following the Freedom of Informa-
tion Act 2000, Environmental Information
Regulations 2004, and Privacy and Electronic
Communications Regulations 2003, and also
ICO has some limited responsibilities under the
INSPIRE Regulations 2009, in England, Wales,
Northern Ireland, and UK-wide public authorities
based in Scotland. On the other hand, Scotland
has complementary INSPIRE Regulations and
its own Scottish Environmental Information
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_114-1
Regulations regulated by the Scottish Information
Commissioner and the Freedom of Information
(Scotland) Act 2002.
The Information Commissioner is appointed
by the Queen and reports directly to Parliament.
The Commissioner is supported by the manage-
ment board. The ICO’s headquarter is in
Wilmslow, Cheshire; in addition to this, three
regional offices in Northern Ireland, Scotland,
and Wales are aimed to provide relevant services
where legislation or administrative structure is
different.
Under the Freedom of Information Act, Envi-
ronmental Information Regulations, INSPIRE
Regulations, and associated codes of practice,
the functions of the ICOs contain noncriminal
enforcement and assessments of good practice,
providing information to individuals and organi-
zations, taking appropriate action when the law an
freedom of information is broken, considering
complaints, disseminating publicity and encour-
aging sectoral codes of practice, and taking action
to change the behavior of organizations and indi-
viduals that collect, use, and keep personal infor-
mation. The main aim is to promote data privacy
for individuals, for providing this service, the ICO
has different tools such as criminal prosecution,
noncriminal enforcement, and audit. The Informa-
tion Commissioner also has the power to serve a
monetary penalty notice on a data controller and
promotes openness to public.
The Data Protection Act 1984 introduced basic
rules of registration for users of data and rights of
2 Information Commissioner, United Kingdom
access to that data for the individuals to which it
related. In order to comply with the Act, a data
controller must comply with the following eight
principles as “data should be processed fairly and
lawfully; should be obtained only for specified
and lawful purposes; should be adequate, rele-
vant, and not excessive; should be accurate and,
where necessary, kept up to date; should not be
kept longer than is necessary for the purposes for
which it is processed; should be processed in
accordance with the rights of the data subject
under the Act; should be appropriate technical
and organisational measures should be taken
against unauthorised or unlawful processing of
personal data and against accidental loss or
destruction of, or damage to, personal data; and
should not be transferred to a country or territory
outside the European Economic Area unless that
country or territory ensures an adequate level of
protection for the rights and freedoms of data
subjects in relation to the processing of personal
data.”
In 1995, The EU formally adopted the General
Directive on Data Protection. In 1997, DUIS, the
Data User Information System, was implemented,
and the Register of Data Users was published on
the internet. In 2000, the majority of the Data
Protection Act comes into force. The name of
the office was changed from the Data Protection
Registrar to the Data Protection Commissioner.
Notification replaced the registration scheme
established by the 1984 Act. Revised regulations
implementing the provisions of the Data Protec-
tion Telecommunications Directive 97/66/EC
came into effect. In January 2001, the office was
given the added responsibility of the Freedom of
Information Act and changed its name to the
Information Commissioner’s Office. On 1 Janu-
ary, 2005, the Freedom of Information Act 2000
was fully implemented. The Act was intended to
improve the public’s understanding of how public
authorities carry out their duties, why they make
the decisions they do, and how they spend their
money. Placing more information in the public
domain would ensure greater transparency and
trust and widen participation in policy debate. In
October 2009, the ICO adopted a new mission
statement: “The ICO’s mission is to uphold infor-
mation rights in the public interest, promoting
openness by public bodies and data privacy for
individuals.” In 2011, ICO launched the “data
sharing code of practice” at the House of Com-
mons and enable to impose monetary penalties of
up to £500,000 for serious breaches of the Privacy
and Electronic Communications Regulations.
Cross-References
▶ Data Protection
▶ Open Data
Further Readings
Data Protection Act 1984. http://www.out-law.com/page-
413. Accessed Aug 2014.
DataProtectionAct 1984. http://www.legislation.gov.uk/
ukpga/1984/35/pdfs/ukpga_19840035_en.pdf?view=
extent. Accessed Aug 2014.
Smartt, U. (2014). Media & entertainment law (2nd ed.).
London: Routledge.
I
Interactive Data Visualization
Andreas Veglis
School of Journalism and Mass Communication,
Aristotle University of Thessaloniki,
Thessaloniki, Greece
Definition
Data visualization is a modern branch of descrip-
tive statistics that involves the creation and study
of the visual representation of data. It is the graph-
ical display of abstract information for data anal-
ysis and communication purposes. Static data
visualization offers only precomposed “views”
of data. Interactive data visualization supports
multiple static views in order to present a variety
of perspectives on the same information. Impor-
tant stories include “hidden” data, and interactive
data visualization is the appropriate mean to dis-
cover, understand, and present these stories. In
interactive data visualization there is a user input
(a control of some aspect of the visual representa-
tion of information), and the changes made by the
user must be incorporated into the visualization in
a timely manner. They are based on existing sets
of data, and obviously this subject is strongly
related with the issue of big data. Data visualiza-
tions is the best method in order to transform
chunks of data to meaningful information (Ward
et al. 2015).
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_116-1
History
Although people have been using tables in order
to arrange data since the second century BC, the
idea of representing quantitative information
graphically first appeared in the seventeenth cen-
tury. Rene Descartes, who was a French philoso-
pher and mathematician, proposed a two-
dimensional coordinate system for displaying
values, consisting of a horizontal axis for one
variable and a vertical axis for another, primarily
as a graphical means of performing mathematical
operations. In the eighteenth century William
Playfair began to exploit the potential of graphics
for the communication of quantitative, by devel-
oping many of the graphs that are commonly used
today. He was the first to employ a line moving up
and down as it progressed from left to right to
show how values changed through time. He
invented the bar graph, as well as the pie chart.
In the 1960s Jacques Bertin proposed that visual
perception operates according to rules that can be
followed to express information visually in ways
that represented it intuitively, clearly, accurately,
and efficiently. Also John Tukey, a statistics pro-
fessor set the basis of the exploratory data analy-
sis, by demonstrating the power of data
visualization as a means for exploring and making
sense of quantitative data (Few 2013).
In 1983, Edward Tufte published his ground-
breaking book “The Visual Display of Quantita-
tive Information,” in which he distinguished
between the effective ways of displaying data
2 Interactive Data Visualization
visually and the ways that most people are doing it
without much success. Also around this time,
William Cleveland extended and refined data
visualization techniques for statisticians. At the
end of the century, the term information visuali-
zation was proposed. In 1999, Stuart Card, Jock
Mackinlay, and Ben Shneiderman published their
book entitled “Readings in Information Visuali-
zation: Using Vision to Think.” Moving to the
twenty-first century, Colin Ware published two
books entitled “Information Visualization: Per-
ception for Design (2004) and Visual Thinking
for Design (2008)” in which he compiled, orga-
nized, and explained what we have learned from
several scientific disciplines about visual thinking
and cognition and applied that knowledge to data
visualization (Few 2013).
Since the turn of the twenty-first century, data
visualization has been popularized, and it has
reached the masses through commercial software
products that are distributed through the web.
Many of these data visualization products pro-
mote more superficially appealing esthetics and
neglect the useful and effective data exploration,
sense-making, and communication. Nevertheless
there are a few serious contenders that offer prod-
ucts which help users fulfill data visualization
potential in practical and powerful ways.
From Static to Interactive
Visualization can be categorized into static and
interactive. In the case of the static visualization,
there is only one view of data, and in many occa-
sions, multiple cases are needed in order to fully
understand the available information. Also the
number of dimensions of data is limited. Thus
representing multidimensional datasets fairly in
static images is almost impossible. Static visuali-
zation is ideal when alternate views are neither
needed nor desired and is special suited for static
medium (e.g., print) (Knaffic 2015). It is worth
mentioning that infographics are also part of the
static visualization. Infographics (or information
graphics) are graphic visual representations of
data or knowledge, which are able to present
complex information quickly and clearly.
Infographics are being used for many years, and
recently the availability of many easy-to-use free
tools have made the creation of infographics
available to every Internet user (Murray 2013).
Of course static visualizations can also be
published on the World Wide Web in order to
disseminate more easily and rapidly. Publishing
on the web is considered to be the quickest way to
reach a global audience. An online visualization is
accessible by any Internet user that employs a
recent web browser, regardless of the operating
system (Windows, Mac, Linux, etc.) and device
type (laptop, desktop, smartphone, tablet). But the
true capabilities of the web are being exploited in
the case of interactive data visualization.
Dynamic, interactive visualizations can
empower people to explore data on their own.
The basic functions of most interactive visualiza-
tion tools have been set back in 1996, when Ben
Shneiderman proposed a “Visual Information-
Seeking Mantra” (overview first, zoom and filter,
and then details on demand). The above functions
allow data to be accessible from every user, from
the one who is just browsing or exploring the
dataset to the one who approaches the visualiza-
tion with a specific question in mind. This design
pattern is the basic guide for every interactive
visualization today.
An interactive visualization should initially
offer an overview of the data, but it must also
include tools for discovering details. Thus it will
be able to facilitate different audiences, from those
who are new to the subject to those who are
already deeply familiar with the data. Interactive
visualization may also include animated transi-
tions and well-crafted interfaces in order to
engage the audience to the subject it covers.
User Control
In the case of interactive data visualization, users
interact with the visualization by introducing a
number of input types. Users can zoom in a par-
ticular part of an existing visualization, pinpoint
an area that interest them, select an option from an
offered list, choose a path, and input numbers or
text that customize the visualization. All the
Interactive Data Visualization
previous mentioned input types can be accom-
plished by using keyboard, mice, touch screen,
and other more specialized input devices. With
the help of these input actions, users can control
both the information being represented on the
graph or the way that the information is being
presented. In the second case, the visualization is
usually part of a feedback loop. In most cases the
actual information remains the same, but the rep-
resentation of the information does change. One
other important parameter in the interactive data
visualizations is the time it takes for the visuali-
zation to be updated after the user has introduced
an input. A delay of more than 20 ms is noticeable
by most people. The problem is that when large
amounts of data are involved, this immediate ren-
dering is impossible.
Interactive framerate is a term that is often
being used to measure the frequency with which
a visualization system generates an image. In
case that the rapid response time, which is
required for interactive visualization, is not fea-
sible, there are several approaches that have
been explored in order to provide people with
rapid visual feedback based on their input.
These approaches include:
Parallel rendering: in this case the image is being
rendered simultaneously by two or more com-
puters (or video cards). Different frames are
being rendered at the same time by different
computers, and the results are transferred over
the network for display on the user’s computer.
Progressive rendering: in this case a framerate is
guaranteed by rendering some subset of the
information to be presented. It also provides
progressive improvements to the rendering
when the visualization is no longer changing.
Level-of-detail (LOD) rendering: in this case sim-
plified representations of information are ren-
dered in order to achieve the desired frame rate,
while a user is providing input. When the user
has finished manipulating the visualization, then
the full representation is used in order to generate
a still image.
Frameless rendering: in this type of rendering,
the visualization is not presented as a time
series of images. Instead a single image is
3
generated where different regions are
updated over time.
Types of Interactive Data Visualizations
The information and more specifically statistical
information is abstract, since it describes things
that are not physical. It can concern education,
sales, diseases, and various other things. But
everything can be displayed visually, if the way
is found to give them a suitable form. The trans-
formation of the abstract into physical representa-
tion can only succeed if we understand a bit about
visual perception and cognition. In other words, in
order to visualize data effectively, one must
follow design principles that are derived from an
understanding of human perception.
Heer, Bostock and Ogievetsky (2010) defined
the types (and also their subcategories) of data
visualization:
(i) Time series data (index charts, stacked
graphs, small multiples, horizon graphs)
(ii) Statistical distributions (stem-and-leaf plots,
Q-Q plots, scatter plot matrix (SPLOM),
parallel coordinates)
(iii) Maps (flow maps, choropleth maps, gradu-
ated symbol maps, cartograms)
(iv) Hierarchies (node-link diagrams, adjacency
diagrams, enclosure diagrams)
(v) Networks (force-directed layout, arc dia-
grams, matrix views)
Tools
There are a lot of tools that can be used for
creating interactive data visualizations. All of
them are either free or offer a free version (except
a paid version that includes more features).
According to datavisualization.ch, the list of the
tools that most users employ includes: Arbor.js,
CartoDB, Chroma.js, Circos, Cola.js,
ColorBrewer, Cubism.js, Cytoscape, D3.js,
Dance.js, Data.js, DataWrangler, Degrafa, Envi-
sion.js, Flare, GeoCommons, Gephi, Google
Chart Tools, Google Fusion Tables, I Want
4 Interactive Data Visualization
Hue, JavaScript InfoVis Toolkit, Kartograph,
Leaflet, Many Eyes, MapBox, Miso, Modest
Maps, Mr. Data Converter, Mr. Nester, NVD3.
js,. NodeBox, OpenRefine, Paper.js, Peity, Poly-
maps, Prefuse, Processing, Processing.js, Pro-
tovis, Quadrigram, R, Raphael, Raw, Recline.js,
Rickshaw, SVG Crowbar, Sigma.js. Tableau
Public, Tabula, Tangle, Timeline.js, Unfolding,
Vega, Visage, and ZingCharts.
Conclusion
Data visualization is a significant discipline that
is expected to become even more important as
we gradually moving, as a society, in the era of
big data. Especially the case of interactive data
visualization allows data analysts to convey
complex data to meaningful information that
can be searched, explored, and understood by
end users.
Cross-References
▶ Business Intelligence
▶ Tableau Software
▶ Visualization
Further Readings
Few, S. (2013). Data visualization for human perception. In
S. Mads & D. R. Friis (Eds.), The encyclopedia of human-
computer interaction (2nd ed.). Aarhus: The Interaction
Design Foundation. http://www.interaction-design.org/lit
erature/book/the-encyclopedia-of-human-computer-inter
action-2nd-ed/data-visualization-for-human-perception.
Accessed 12 July 2016.
Heer, J., Bostock, M., & Ogievetsky, V. (2010). A tour
through the visualization zoo. Communications of the
ACM, 53(6), 59–67.
Knaffic, C. N. (2015). Storytelling with data: A data visu-
alization guide for business professionals. Hoboken,
New Jersey: John Wiley & Sons Inc.
Murray, S. (2013). Interactive data visualization for the
web. Sebastopol, CA: O’Reilly Media, Inc.
Ward, M., Grinstein, G., & Keim, D. (2015). Interactive
data visualization: Foundations, techniques, and
applications. Boca Raton, FL: CRC Press, Taylor &
Francis Group.
I
International Development
Jon Schmid
Georgia Institute of Technology, Atlanta, GA,
USA
Big data can affect international development in
two primary ways. First, big data can enhance our
understanding of underdevelopment by
expanding the evidence base available to
researchers, donors, and governments. Second,
big data-enabled applications can affect interna-
tional development directly by facilitating eco-
nomic behavior, monitoring local conditions,
and improving governance. The following sec-
tions will look first at the role of big data in
increasing our understanding of international
development and then look at examples where
big data has been used to improve the lives of
the world’s poor.
Big Data in International Development
Research
Data quality and data availability tend to be low in
developing countries. In Kenya, for example,
poverty data was last collected in 2005, and
income surveys in other parts of sub-Saharan
Africa often take up to 3 years to be tabulated.
When national income-accounting methodologies
were updated in Ghana (2010) and Nigeria
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_117-1
(2014), GDP calculations had to be revised
upward by 63% and 89%, respectively. Poor-
quality or stale data prevent national policy
makers and donors from making informed policy
decisions.
Big data analytics has the potential to amelio-
rate this problem by providing alternative
methods for collecting data. For example, big
data applications may provide a novel means by
which national economic statistics are calculated.
The Billion Prices Project – started by researchers
at the Massachusetts Institute of Technology –
uses daily price data from hundreds of online
retailers to calculate changes in price levels. In
countries where inflation data is unavailable – or
in cases such as Argentina where official data is
unreliable – these data offer a way of calculating
national statistics that does not require a high-
quality national statistics agency.
Data from mobile devices is a particularly rich
source of data in the developing world. Roughly
20% of mobile subscriptions are held by individ-
uals that earn less than 5 $ a day. Besides emitting
geospatial, call, and SMS data, mobile devices are
increasingly being used in the developing world
to perform a broad array of economic functions
such as banking and making purchases. In many
African countries (nine in 2014), more people
have online mobile money accounts than have
traditional bank accounts. Mobile money services
such M-Pesa and MTN Money produce trace data
and thus offer intriguing possibilities for increas-
ing understanding of spending and saving
2 International Development
behavior in the developing world. As the func-
tionality provided by mobile money services
extends into loans, money transfers from abroad,
cash withdrawal, and the purchase of goods, the
data yielded by these platforms will become even
richer.
The data produced by mobile devices has
already been used to glean insights into complex
economic or social systems in the developing
world. In many cases, the insights into local eco-
nomic conditions that result from the analysis of
mobile device data can be produced more quickly
than national statistics. For example, in Indonesia
the UN Global Pulse monitored tweets about the
price of rice and found them to be highly corre-
lated with national spikes in food prices. The same
study found that tweets could be used to identify
trends in other types of economic behavior such as
borrowing. Similarly, research by Nathan Eagle
has shown that reductions in additional airtime
purchases are associated with falls in income.
Researchers Han Wang and Liam Kilmartin
examined Call Detail Record (CDR) data gener-
ated from mobile devices in Uganda and identified
differences in the way that wealthy and poor indi-
viduals respond to price discounts. The
researchers also used the data to identify centers
of economic activity within Uganda.
Besides providing insight into how individuals
respond to price changes, big data analytics
allows researchers to explore the complex ways
in which the economic lives of the poor are orga-
nized. Researchers at Harvard’s Engineering
Social Systems lab have used mobile phone data
to explore the behavior of inhabitants of slums in
Kenya. In particular, the authors tested theories of
rural-to-urban migration against spatial data emit-
ted by mobile devices. Some of the same
researchers have used mobile data to examine
the role of social networks on economic develop-
ment and found that diversity in individuals’ net-
work relationships is associated with greater
economic development. Such research supports
the contention that insular networks – i.e., highly
clustered networks with few ties to outside
nodes – may limit the economic opportunities
that are available to members.
Big data analytics are also being used to
enhance understanding of international develop-
ment assistance. In 2009, the College of William
and Mary, Brigham Young University, and Devel-
opment Gateway created AidData (aiddata.org), a
website that aggregates data on development pro-
jects to facilitate project coordination and provide
researchers with a centralized source for develop-
ment data. AidData also maps development pro-
jects geospatially and links donor-funded projects
to feedback from the project’s beneficiaries.
Big Data in Practice
Besides expanding the evidence base available to
international development scholars and practi-
tioners, large data sets and big data analytic tech-
niques have played a direct role in promoting
international development. Here the term “devel-
opment” is considered in its broad sense as refer-
ring not to a mere increase in income, but to
improvements in variables such as health and
governance.
The impact of infectious diseases on develop-
ing countries can be devastating. Besides the
obvious humanitarian toll of outbreaks, infectious
diseases prevent the accumulation of human cap-
ital and strain local resources. Thus there is great
potential for big data-enabled applications to
enhance epidemiological understanding, mitigate
transmission, and allow for geographically
targeted relief. Indeed, it is in the tracking of
health outcomes that the utility of big data analyt-
ics in the developing world has been most obvi-
ous. For example, Amy Wesolowski and
colleagues used mobile phone data from 15 mil-
lion individuals in Kenya to understand the rela-
tionship between human movement and malaria
transmission. Similarly, after noting in 2008 that
search trends could be used to track flu outbreaks,
researchers at Google.org have used data on
searches for symptoms to predict outbreaks of
the dengue virus in Brazil, Indonesia, and India.
In Haiti, researchers from Columbia University
and the Karolinska Institute used SIM card data
to track the dispersal of people following a cholera
outbreak. Finally, the Centers for Disease Control
International Development
and Prevention used mobile phone data to direct
resources to appropriate areas during the 2014
Ebola outbreak.
Big data applications may also prove useful in
improving and monitoring aspects of governance
in developing countries. In Kenya, India, and
Pakistan, witnesses of public corruption can
report the incident online or via text message to
a service called “I Paid A Bribe.” The provincial
government in Punjab, Pakistan, has created a
citizens’ feedback model, whereby citizens are
solicited for feedback regarding the quality of
government services they received via automated
calls and texts. In effort to discourage absenteeism
in India and Pakistan, certain government officials
are provided with cell phones and required to text
geocoded pictures of themselves at jobsites. These
mobile government initiatives have created a rich
source of data that can be used to improve gov-
ernment service delivery, reduce corruption, and
more efficiently allocate resources.
Applications that exploit data from social
media have also proved useful in monitoring elec-
tions in sub-Saharan Africa. For example, Aggie,
a social media tracking software designed to mon-
itor elections, has been used to monitor elections
in Liberia (2011), Ghana (2012), Kenya (2013),
and Nigeria (2011 and 2014). The Aggie system is
first fed with a list of predetermined keywords,
which are established by local subject matter
experts. The software then crawls social media
feeds – Twitter, Facebook, Google+, Ushahidi,
and RSS – and generates real-time trend visuali-
zations based on keyword matches. The reports
are monitored by a local Social Media Tracking
Center, which identifies instances of violence or
election irregularities. Flagged incidents are
passed on to members of the election commission,
police, or other relevant stakeholders.
The history of international economic devel-
opment initiatives is fraught with would-be pana-
ceas that failed to deliver. White elephants – large-
scale capital investment projects for which the
social surplus is negative – are strewn across
3
poor countries as reminders of the preferred devel-
opment strategies of the past. While more recent
approaches to reducing poverty that have focused
on improving institutions and governance within
poor countries may produce positive development
effects, the history of development policy sug-
gests that optimism should be tempered. The
same caution holds in regard to the potential role
of big data in international economic develop-
ment. Martin Hilbert’s 2016 systematic review
article rigorously enumerates both the causes for
optimism and reasons for concern. While big data
may assist in understanding the nature of poverty
or lead to direct improvements in health or gover-
nance outcomes, the availability and ability to
process large data sets are not a panacea.
Cross-References
▶ Economics
▶ Epidemiology
▶ U.S. Agency International Development
▶ United Nations Global Pulse (Development)
▶ World Bank
Further Reading
Hilbert, M. (2016). Big data for development: A review of
promises and challenges. Development Policy Review,
34(1), 135–174.
Wang, H., & Kilmartin, L. (2014). Comparing rural and
urban social and economic behavior in Uganda:
Insights from mobile voice service usage. Journal of
Urban Technology, 21(2), 61–89.
Wesolowski, A., et al. (2012). Quantifying the impact of
human mobility on malaria. Science, 338(6104),
267–270.
World Economic Forum. (2012). Big data, big impact:
New possibilities for international development. In
Big data, big impact: New possibilities for interna-
tional development, Cologny/Geneva, Switzerland:
World Economic Forum. http://www3.weforum.org/
docs/WEF_TC_MFS_BigDataBigImpact_Briefing_
2012.pdf
I
International Labor Organization
Jennifer Ferreira
Centre for Business in Society, Coventry
University, Coventry, UK
Every day people across the world in both devel-
oped and developing economies are creating an
ever-growing ocean of digital data. This “big
data” represents a new resource for international
organizations with the potential to revolutionize
the way policies, programs, and projects are gen-
erated. The International Labour Organization
(ILO) is no exception to this and has begun to
discuss and engage with the potential uses of big
data to contribute to its agenda.
Focus
The ILO, founded in 1919 in the wake of the First
World War, became the first specialized agency of
the United Nations. It focuses on labor issues
including child labor, collective bargaining, cor-
porate social responsibility, disability, domestic
workers, forced labor, gender equality, informal
economy, international labor migration, interna-
tional labor standards, labor inspection, micro-
finance, minimum wages, rural development,
and youth employment. By 2013 the ILO had
185 members (of the 193 member states of the
United Nations). Among its multifarious
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_118-1
activities, it is widely known for its creation of
Conventions and Recommendations (189 and
203, respectively by, 2014) related to labor market
standards.
Where Conventions are ratified, come into
force, and are therefore legally binding, they cre-
ate a legal obligation for ratifying nations. For
many Conventions even in countries where they
are not ratified, they are often adopted and
interpreted as the international labor standard.
There have been many important milestones cre-
ated by the ILO to shape the landscape to encour-
age the promotion of improved working lives
globally, although a significant milestone is often
considered to be the 1998 Declaration on the
Fundamental Principles and Rights to Work
which had four key components: the right of
workers to associate freely and collectively, the
end of forced and compulsory labor, the end of
child labor, and the end of unfair discrimination
among workers. ILO members have an obligation
to work toward these objectives and respect the
principles which are embedded in the
Conventions.
Decent Work Agenda
The ILO believes that work plays a crucial role in
the well-being of workers and families and there-
fore the broader social and economic develop-
ment of individuals, communities, and societies.
While the ILO works on many issues related to
2 International Labor Organization
employment, their key agenda which has domi-
nated activities in recent decades is “decent
work.”
“Decent work” refers to an aspiration for peo-
ple to have a work that is productive, provides a
fair income with security and social protection,
safeguards basic rights, and offers equal opportu-
nities and treatment, opportunities for personal
development, and a voice in society. “Decent
work” is central to efforts to reduce poverty and
is a path to achieving equitable, inclusive, and
sustainable development; ultimately it is seen as
a feature which underpins peace and security in
communities and societies (ILO 2014a).
The “decent work” concept was formulated by
the ILO in order to identify the key priorities to
focus their efforts. “Decent work” is designed to
reflect priorities on the social, economic, and
political agenda of countries as well as the inter-
national system. In a relatively short time, this
concept has formed an international consensus
among government, employers, workers, and
civil equitable globalization, a path to reduce pov-
erty as well as inclusive and sustainable develop-
ment. The overall goal of “decent work” is to
instigate positive change in/for people at all spa-
tial scales.
Putting the decent work agenda into practice is
achieved through the implementation of the ILO’s
four strategic objectives, with gender equality as a
crosscutting objective:
1. Creating jobs to foster an economy that gener-
ates opportunities for investment, entrepre-
neurship, skills development, job creation,
and sustainable livelihoods.
2. Guaranteeing rights at work in order to obtain
recognition for work achieved as well as
respect for the rights of all workers.
3. Extending social protection to promote both
inclusion and productivity of all workers. To
be enacted by ensuring both women and men
experience safe working conditions, allowing
free time, taking into account family and social
values and situations, and providing compen-
sation where necessary in the case of lost or
reduced income.
4. Promoting social dialogue by involving both
workers and employers in the organizations in
order to increase productivity, avoid disputes
and conflicts at work, and more broadly build
cohesive societies.
ILO Data
The ILO produces research on important labor
market trends and issues to inform constituents,
policy makers, and the public about the realities of
employment in today’s modern globalized econ-
omy and the issues facing workers and employers
in countries at all development stages. In order to
do so, it draws on data from a wide variety of
sources.
The ILO is a major provider of statistics as
these are seen as important tools to monitor pro-
gress toward labor standards. In addition to the
maintenance of key databases (ILO 2014b) such
as LABOURSTA, it also publishes compilations
of labor statistics, such as the Key Indicators of
Labour Markets (KILM) which is a comprehen-
sive database of country level data for key indica-
tors in the labor market which is used as a research
tool for labor market information. Other databases
include the ILO STAT, a series of databases with
labor-related data; NATLEX which includes leg-
islation related to labor markets, social security,
and human rights; and NORMLEX which brings
together ILO labor standards and national labor
and security laws (ILO 2014c). The ILO database
provides a range of datasets with annual labor
market statistics including over 100 indicators
worldwide including annual indicators as well as
short-term indicators, estimates and projections of
total population, and labor force participation
rates.
Statistics are vital for the development and
evaluation of labor policies, as well as more
broadly to assess progress toward key ILO objec-
tives. The ILO supports member states in the
collection and dissemination of reliable and recent
data on labor markets. While the data produced by
the ILO are both wide ranging and widely used,
they are not considered by most to be “big data,”
and this has been recognized.
International Labor Organization
ILO, Big Data, and the Gender Data
In October 2014, a joint ILO-Data2X roundtable
event held in Switzerland identified the impor-
tance of developing innovative approaches to the
better use of technology to include big data, in
particular where it can be sourced and where
innovations can be made in survey technology.
This event, which brought together representa-
tives from national statistics offices, key interna-
tional and regional organizations, and
nongovernmental organizations, was organized
to discuss where there were gender data gaps,
particularly focusing on informal and unpaid
work as well as agriculture. These discussions
were sparked by wider UN discussions about the
data revolution and the importance of develop-
ment data in the post-2015 development agenda.
It is recognized that big data (including adminis-
trative data) can be used to strengthen existing
collection of gender statistics, but there need to
be more efforts to find new and innovative ways to
work with new data sources to meet a growing
demand for more up to date (and frequently
updating) data on gender and employment
(United Nations, 2013). The fundamental goal of
the discussion was to improve gender data collec-
tion which can then be used to guide policy and
inform the post-2015 development agenda, and
here big data is acknowledged as a key compo-
nent. At this meeting, four types of gender data
gaps were identified: coverage across countries
and/or regular country production, international
standards to allow comparability, complexity,
and granularity (sizeable and detailed datasets
allowing disaggregation by demographic and
other characteristics). Furthermore a series of big
data types that have the potential to increase col-
lection of gender data were identified:
• Mobile phone records: for example, mobile
phone use and recharge patterns could be
used as indicators of women’s socioeconomic
welfare or mobility patterns.
• Financial patterns: exploring engagement with
financial systems.
• Online activity: for example, Google searches
or Twitter activity which might be used to gain
3
insights into women’s maternal health, cultural
attitudes, or political engagement.
• Sensing technologies: for example, satellite
data which might be used to examine agricul-
tural productivity, access to healthcare, and
education services.
• Crowdsourcing: for example, disseminating
apps to gain views about different elements of
societies.
A primary objective of this meeting was to
highlight that existing gender data gaps are large,
and often reflect traditional societal norms, and
that no data (or poor data) can have significant
development consequences. Big data here has the
potential to transform the understanding of
women’s participation in work and communities.
Crucially it was posited that while better data is
needed to monitor the status of women in informal
employment conditions, it is not necessarily
important to focus on trying to extract more data
but to make an impact with the data that is avail-
able to try and improve wider social, economic,
and environmental conditions.
ILO, the UN, and Big Data
The aforementioned meeting represented one
example of where the ILO has engaged with
other stakeholders to not only acknowledge the
importance of big data but begin to consider
potential options for its use with respect to their
agendas. However, as a UN agency, they partake
in wider discussion with the UN regarding the
importance of big data, as was seen in the 45th
session of the UN Statistical Commission in
March 2014 where the report of the secretary
general on “big data and the modernization of
statistical systems” was discussed (United
Nations, 2014). This report is significant as it
touches upon important issues, opportunities,
and challenges that are relevant for the ILO with
respect to the use of big data.
The report makes reference to the UN “Global
Pulse” which is an initiative on big data
established in 2009 which included a vision of a
4 International Labor Organization
future where big data was utilized safely and
responsibly. Its mission was to accelerate the
adoption of big data innovation. Partnering with
UN agencies such as the ILO, governments, aca-
demics, and the private sector, it sought to achieve
a critical mass of implemented innovation and
strengthen the adoption of big data as a tool to
foster the transformation of societies.
There is a recognition that the national statisti-
cal system is essentially now subject to competi-
tion from other actors producing data outside of
their system, and there is a need for data collection
of national statistics to adjust in order to make use
of the mountain of data now being produced
almost continuously (and often automatically).
To make use of the big data, a shift may be
required from the traditional survey-oriented col-
lection of data to a more secondary data-focused
orientation from data sources that are high in
volume, velocity, and variety. Increasing demand
from policy makers for real-time evidence in com-
bination with declining response rates to national
household and business survey means that orga-
nizations like the ILO will have to acknowledge
the need to make this shift. There are a number of
different sources of big data which may be poten-
tially useful for the ILO: sources from administra-
tion, e.g., bank records; commercial and
transaction data, e.g., credit card transactions;
sensor data, e.g., satellite images or road sensors;
tracking devices, e.g., mobile phone data; behav-
ioral data, e.g., online searches; and opinion data,
e.g., social media. Official statistics like those
presented in ILO databases often rely on admin-
istrative data, and these are traditionally produced
in a highly structured manner which can in turn
limit their use. If administrative data was collected
in real time, or in a more frequent basis, then it has
the potential to become “big data.”
There are, however, a number of challenges
related to the use of big data which face the UN,
its agencies, and national statistical services alike:
• Legislative: in many countries, there will not
be legislation in place to enable the access to,
and use of, big data particularly from the pri-
vate sector.
• Privacy: a dialogue will be required in order to
gain public trust around the use of data.
• Financial: related to costs for access data.
• Management: policies and directives to ensure
management and protection of data.
• Methodological: data quality, representative-
ness, and volatility are all issues which present
potential barriers to the widespread use of
big data.
• Technological: the nature of big data, particu-
larly the volume in which it is often created
meaning that some countries would need
enhanced information technology.
An assessment of the use of big data for official
statistics carried out by the UN indicates that there
are good examples where it has been used, for
example, using transactional, tracking, and sensor
data. However, in many cases, a key implication is
that statistical systems and IT infrastructures need
to be enhanced in order to be able to support the
storage and processing of big data as it accumu-
lates over time.
Modern society has witnessed an explosion of
the quantity and diversity of real-time information
known more commonly as big data, presenting a
potential paradigm shift in the way official statis-
tics are collected and analyzed. In the context of
increased demand for statistics information, orga-
nizations recognize that big data has the potential
to generate new statistical products in a timelier
manner than traditional official statistical sources.
The ILO, alongside a broader UN agenda to
acknowledge the data revolution, recognizes the
potential for future uses of big data at the global
level, although there is a need for further investi-
gation of the data sources, challenges and areas of
use of big data, and its potential contribution to
efforts working toward the “better work” agenda.
Cross-References
▶ Scientific and Cultural Organization
(UNESCO) United Nations Global Pulse
▶ United Nations
▶ United Nations Educational
International Labor Organization
Further Readings
International Labour Organization. (2014a). Key indica-
tors of the labour market. International Labour Orga-
nization. http://www.ilo.org/empelm/what/WCMS_
114240/lang–en/index.htm. Accessed 10 Sep 2014.
International Labour Organization. (2014b). ILO databases.
International Labour Organization. http://www.ilo.org/pub
lic/english/support/lib/resource/ilodatabases.htm. Accessed
1 Oct 2014.
International Labour Organization. (2014c). ILOSTAT data-
base. International Labour Organization. http://www.ilo.
5
org/ilostat/faces/home/statisticaldata?_afrLoop=342428
603909745. Accessed 10 Sep 2014.
United Nations. (2013). Big data and modernization of sta-
tistical systems. Report of the Secretary-General. United
Nations. United Nations Economic and Social Council.
Available at: http://unstats.un.org/unsd/statcom/doc14/
2014-11-BigData-E.pdf. Accessed 1 Dec 2014.
United Nations. (2014). UN global pulse. United Nations.
Available at: http://www.unglobalpulse.org/. Accessed
10 Sep 2014.
I
Internet Association, The
David Cristian Morar
Schar School of Policy and Government, George
Mason University, Fairfax, VA, USA
Synonyms
Internet Lobby; Internet Trade Association;
Internet Trade Organization
Introduction
The Internet Association is a trade organization
that represents a significant number of the world’s
largest Internet companies, all of whom are based,
founded, or ran in the United States of America.
While issues such as net neutrality or copyright
reform are at the forefront of their work, the Inter-
net Association is also active in expressing the
voice of the Internet industry in matters of Big
Data. On this topic, it urges a commitment to
status quo in privacy regulation and increased
government R&D for innovative ways of enhanc-
ing the benefits of Big Data, while also calling for
dispelling the belief that the web is the only sector
that collects large data sets, as well as for a more
thorough review of government surveillance.
These proposals are underlined by the perspective
that the government has a responsibility to protect
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_121-1
the economic interests of the US industries, inter-
nationally, and a responsibility to protect the pri-
vacy of the American citizens, nationally.
Main Text
Launched in 2012 with 14 members and designed
as the unified voice in Washington D.C. for the
industry, the Internet Association now boasts
41 members and is dedicated, according to their
statements, to protecting the future of the free and
innovative Internet. Among these 41 members,
some of the more notable include Amazon,
AOL, Groupon, Google, Facebook, Twitter,
eBay, Yelp, IAC, Uber Technologies Inc,
Expedia, and Netflix. As part of both their purpose
and mission statements, the Internet Association
believes that the decentralized architecture of the
Internet, which it vows to protect, is what led it to
become one of the world’s most important engines
for growth, economically and otherwise. The
Association’s representational role, also referred
to as a lobbying, is portrayed as not simply an
annex of Silicon Valley but as a voice of its com-
munity of users as well. The policy areas it pro-
motes are explained with a heavy emphasis on the
user and the benefits and rights the user gains.
The President and CEO, Michael Beckerman,
a former congressional staffer, is the public face of
the Internet Association, and he is usually the one
that signs statements or comments on important
2 Internet Association, The
issues on behalf of the members. Beyond their
“business crawl” efforts promoting local busi-
nesses and their connection to, and success yield-
ing from the Internet economy, the Association is
active in many other areas. These areas include
Internet freedom (nationally and worldwide) and
patent reform, among others, with their most
important concern being net neutrality. As Big
Data is associated with the Internet, and the indus-
try is interested in being an active stakeholder in
related policy, the Association has taken several
opportunities to make its opinions heard on the
matter. These opinions can also be traced through-
out the policies it seeks to propose in other
connected areas.
Most notably, after the White House Office of
Science and Technology Policy’s (OSTP) 2014
request for information, as part of their 90-day
review on the topic of Big Data, the Internet
Association has released a set of comments that
crystallize their views on the matter. Prior com-
munications have also brought up certain aspects
related to Big Data; however, the comments made
to the OSTP have been the most comprehensive
and detailed public statement to date by the indus-
try on issues of Big Data, privacy, and govern-
ment surveillance.
In matters of privacy regulation, the Associa-
tion believes that the current framework is both
robust and effective in relation to commercial
entities. In their view, reform is mostly necessary
in the area of government surveillance, by
adopting an update to the Electronic Communica-
tions Privacy Act (which would give service pro-
viders a legal basis in denying government
requests for data that are not accompanied by a
warrant), prohibiting bulk governmental collec-
tion of metadata from communications and clearly
bounding surveillance efforts by law.
The Internet Association subscribes to the
notion that the current regime for private sector
privacy regulation is not only sufficient but also
perfectly equipped to deal with potential concerns
brought about by Big Data issues. The status quo
is, in the acceptation of the Internet industry, a
flexible and multilayered framework, designed for
businesses that embrace privacy protective prac-
tices. The existing framework, beyond a
sometimes overlapping federal-state duality of
levels, also includes laws in place through the
Federal Trade Committee that guard against
unfair practices and that target and swiftly punish
the bad actors that perpetrate the worst harms.
This allows companies to harness the potential
of Big Data within a privacy-aware context that
does not allow or tolerate gross misconduct. In
fact, the Association even cites the White House’s
2012 laudatory comments on the existing privacy
regimes, to strengthen its argument for regulatory
status quo, beyond simply an industry’s desire to
be left to its own devices to innovate without
major restrictions.
The proposed solutions by the industry would
center on private governance mechanisms that
include a variety of stakeholders in the decision-
making process and are not, in fact, a product of
the legislative system. Such actions have been
taken before and, according to the views of the
Association, are successful in the general sector of
privacy, and they allow industry and other actors
that are involved in the specific areas to have a seat
at the table beyond the traditional lobbying route.
One part that needs further action, according to
the views of the Association, is educating the
public on the entire spectrum of activities that
lead to the collection and analysis of large data
sets. With websites as the focus of most privacy-
related research, the industry advocates a more
consumer-oriented approach that would permeate
the whole range of practices from understudied
sectors to the Internet, centered around increasing
user knowledge on how their data is being han-
dled. This would allow the user to understand the
entire processes that go on beyond the visible
interfaces, without putting any more pressure on
the industries to change their actions.
While the Internet Association considers that
commercial privacy regulation should be left vir-
tually intact, substantial government funding for
research and development should be funneled into
unlocking future and better societal benefits of
Big Data. These funds, administered through the
National Science Foundation and other instru-
ments, would be directed toward a deeper under-
standing of the complexities of Big Data,
including accountability mechanisms,
Internet Association, The
de-identification, and public release. Prioritizing
such government-funded research over new regu-
lation, the industry believes that current societal
benefits from commercial Big Data usage
(ranging from genome research to better spam
filters) would multiply in number and effect.
The Association deems that the innovation
economy would suffer from any new regulatory
approaches that are designed to restrict the free
flow of data. In their view, not only would the
companies not be able to continue with their com-
mercial activities, which would hurt the sector,
and the country, but the beneficial aspects of Big
Data would suffer as well. Coupled with the rev-
elations about the data collection projects of the
National Security Agency, this would signifi-
cantly impact the standing of the United States
internationally, as important international agree-
ments, such as the Transatlantic Trade and Invest-
ment Partnership with the EU, are in jeopardy,
says the industry.
Conclusion
The Internet Association thus sees privacy as a
significant concern with regard to Big Data. How-
ever, it strongly emphasizes governmental mis-
steps in data surveillance, and offers an
unequivocal condemnation of such actions,
while lauding and extolling the virtues of the
regulatory framework in place to deal with the
commercial aspect. The Association believes
that current nongovernmental policies, such as
agreements between users and service providers,
or industry self-regulation, are also adequate, and
promoting such a user-facing approach to a major-
ity of privacy issues would continue to be useful.
Governmental involvement is still desired by the
industry, primarily through funding for what
might be called basic research into the Big Data
territory, as the benefits of this work would be
3
spread around not just between the companies
involved but also with the government, as best
practices would necessarily involve governmental
institutions as well.
Cross-References
▶ Amazon
▶ De-identification, Re-identification
▶ Genome Data
▶ Google
▶ National Security Agency
▶ Netflix
▶ Office of Science and Technology Policy:
White House Report (2014 Report)
▶ Twitter
Further Readings
The Internet Association. Comments of the Internet Associa-
tion in response to the White House Office of Science and
Technology Policy’s Government ‘Big Data’ Request for
Information. http://internetassociation.org/wp-content/
uploads/2014/03/3_31_-2014_The-Internet-Association-
Comments-Regarding-White-House-OSTP-Request-for-
Information-on-Big-Data.pdf. Accessed July 2016.
The Internet Association. Comments on ‘Big Data’ to the
Department of Commerce. http://internetassociation.org/
080614comments/. Accessed July 2016.
The Internet Association. Policies. https://internetassociation.
org/policy-platform/protecting-internet-freedom/.
Accessed July 2016.
The Internet Association. Privacy. http://internetassociation.
org/policies/privacy/. Accessed July 2016.
The Internet Association. Statement on the White House Big
Data Report. http://internetassociation.org/050114bigdata/.
Accessed July 2016.
The Internet Association. The Internet Association’s Press Kit.
http://internetassociation.org/the-internet-associations-press-
kit/. Accessed July 2016.
The Internet Association. The Internet Association Statement
on White House Big Data Filed Comments. http://
internetassociation.org/bigdatafilingstatement/. Accessed
July 2016.
I
Italy
Chiara Valentini
Department of Management, Aarhus University,
School of Business and Social Sciences, Aarhus,
Denmark
Introduction
Italy is a Parliamentary republic in southern
Europe. It has a population of about 60 million
people of which, 86.7%, are Internet users
(Internet World Stat 2017). Public perception of
handling big data is generally very liberal, and the
phenomenon has been associated with more trans-
parency and digitalized economic and social sys-
tems. The collection and processing of personal
data have been increasingly used to counter tax
evasion which is one of the major problems of
Italian economy. The Italian Revenue Agency is
using data collected through different private and
public data collectors to cross-check tax declara-
tions (DPA 2014a).
According to the results of a study on Italian
companies' perception of big data conducted by
researchers at the Big Data Analytics & Business
Intelligence Observatory of Milan Polytechnic,
more and more companies (þ22% in 2013) are
interested in investing in technologies that allow
to handle and use big data. Furthermore, the num-
ber of companies seeking professional managers
that are capable of interpreting data and assisting
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_123-1
senior management on decision-making is also
increasing. Most of the Italian companies (76%
of 184 interviewed) claim that they use basic
analytics strategically and another 36% use more
sophisticated tools for forecasting activities
(Mosca 2014, January 7).
Data Protection Agency and Privacy
Issues
Despite the positive attitude and increased use of
big data by Italian organizations, an increasing
public expectation for privacy protection has
emerged as a result of raising debates on personal
data, data security, and protection in the whole
European Union. In the past years, the Italian
Data Protection Authority (DPA) reported several
instances of data collection of telephone and Inter-
net communications of Italian users which may
have harmed Italians’ fundamental rights (DPA
2014b). Personal data laws have been developed
as these are considered important instruments for
the overall protection of fundamental human
rights, thereby adding new legal specifications to
the existing privacy framework. The first specific
law on personal data was adopted by the Italian
Parliament in 1996 and this incorporated a num-
ber of guidelines already included in the European
Union 1995 Data Protection Directive. At the
same time, an indepedent authority, the Italian
Data Protection Authority (Garante per la pro-
tezione dei dati personali), was created in 1997 to
2 Italy
protect fundamental rights and freedoms of peo-
ple when personal data are processed. The Italian
Data Protection Authority (DPA) is run by a four-
member committee elected by the Italian Parlia-
ment for a seven-year mandate (DPA 2014a).
The main activities of DPA consist of monitor-
ing and assuring that organizations comply with
the latest regulations on data protection and indi-
vidual privacy. In order to do so, DPA carries out
inspections on organizations’ databases and data
storage systems to guarantee that their require-
ments for preserving individual freedom and pri-
vacy are of high standards. It checks that the
activities of the police and the Italian Intelligence
Service comply with the legislation, reports pri-
vacy infringements to judicial authorities, and
encourages organizations to adopt codes of con-
duct promoting fundamental human rights and
freedom. The authority also handles citizens’
reports and complaints of privacy loss or any
misuse or abuse of personal data. It bans or blocks
activities that can cause serious harm to individual
privacy and freedom. It grants authorizations to
organizations and institutions to have access and
use sensitive and/or judicial data. Sensitive and
judicial data concern, for instance, information on
a person’s criminal records, ethnicity, religion or
other beliefs, political opinions, membership of
parties, trade unions and/or associations, health,
or sex life. Access to sensitive and judicial data is
granted only for specific purposes, for example, in
situations where it is necessary to know more
about a certain individual for national security
reasons (DPA 2014b).
The DPA participates to data protection activ-
ities involving the European Union and other
international supervisory authorities and follows
existing international conventions (Schengen,
Europol, and Customs Information System)
when regulating Italian data protection and secu-
rity matters. It carries out an important role in
increasing public awareness of privacy legislation
and in soliciting the Italian Parliament to develop
legislation on new economic and social issues
(DPA 2014b). The DPA has also formulated spe-
cific guidelines on cloud computing for helping
Italian businesses. Yet, according to this authority,
these cloud computing guidelines require that Ital-
ian laws are updated to be fully effective in regu-
lating this area. Critics indicate that there are
limits in existing Italian laws concerning the allo-
cation of liabilities, data security, jurisdiction, and
notification of infractions to the supervisory
authority (Russo 2012).
Another area of great interest for the DPA is the
collection of personal data via video surveillance
both in the public and in the private sector. The
DPA has acted on specific cases of video surveil-
lance, sometimes banning and other times allo-
wing it (DPA 2014c). For instance, the DPA
reported to have banned the use of webcams in a
nursery school to protect children’s privacy and to
safeguard freedom of teaching. It banned police
headquarters to process images collected via
CCTV cameras installed in streets for public
safety purposes because such cameras also cap-
tured images of people’s homes. The use of cus-
tomers’ pre-recorded, operator-unassisted phone
calls for debt collection purposes is among those
activities that have been prohibited by this author-
ity. Yet, the DPA permits the use of video surveil-
lance in municipalities for counter-vandalism
purposes (DPA 2014b).
Conclusion
Overall, Italy is advancing with the regulation of
big data phenomenon following also the impetus
given by the EU institutions and international
debates on data protection, security, and privacy.
Nonetheless, Italy is still lagging behind many
western and European countries regarding the
adoption and development of frameworks for a
full digital economy. According to the Networked
Readiness Index 2015 published by the World
Economic Forum, Italy is ranked 55th. As indi-
cated by the report, Italy’s major weakness is still
a political and regulatory environment that does
not facilitate the development of a digital econ-
omy and its innovation system (Bilbao-Osorio
et al. 2014).
Italy 3

Cross-References DPA (2014b). Who we are. http://www.garanteprivacy.it/web/

▶ Cell Phone Data
▶ Data Security
▶ European Union
▶ Privacy
▶ Security Best Practices
▶ Surveillance Cameras
References
Bilbao-Osorio, B., Dutta, S. & Lanvin, B. (2014). The
global information technology report 2014. Reword
and risks of big data. World Economic Forum. http://
www3.weforum.org/docs/WEF_GlobalInformationTec
hnology_Report_2014.pdf. Accessed 31 Oct 2014.
DPA (2014a). Summary of key activities by the Italian DPA
in 2013. http://www.garanteprivacy.it/web/guest/home/
docweb/-/docweb-display/docweb/3205017. Accessed
31 Oct 2014.
guest/home_en/who_we_are. Accessed 31 Oct 2014.
DPA. (2014c) “Compiti del Garante” [Tasks of DPA].
http://www.garanteprivacy.it/web/guest/home/autorita/
compiti. Accessed 31 Oct 2014.
Internet World Stat (2017). Italy. http://www.
internetworldstats.com/europa.htm. Accessed 15
May 2017.
Mosca, G. (2014, January 7). Big data, una grossa
opportunità per il business, se solo si sapesse come
usarli. La situazione in Italia. La Stampa. http://www.
ilsole24ore.com/art/tecnologie/2014-01-07/big-data-gr
ossa-opportunita-il-business-se-solo-si-sapesse-come-us
arli-situazione-italia-110103.shtml?uuid=ABuGM6n.
Accessed 31 Oct 2014.
Russo, M. (2012). Italian data protection authority releases
guidelines on cloud computing. In McDermott Will &
Emery (Eds.), International News (Focus on Data Pri-
vacy and Security, 4). http://documents.lexology.com/
475569eb-7e6b-4aec-82df-f128e8c67abf.pdf. Accessed
31 Oct 2014.
J

Journalism methodologies to achieve the objectives of jour-

nalism. Although big data offer many opportuni-
Brian E. Weeks1, Trevor Diehl2, Brigitte Huber2 ties for journalists to report the news in novel and
and Homero Gil de Zúñiga2 interesting ways, critics have noted data journal-
1
Communication Studies Department, University ism also faces potential obstacles that must be
of Michigan, Ann Arbor, USA considered.
2
Media Innovation Lab (MiLab), Department of
Communication, University of Vienna, Wien,
Austria
Origins of Journalism and Big Data

Contemporary data journalism is rooted in the

The Pew Research Center notes that journalism is
work of reporters like Philip Meyer, Elliot Jaspin,
a mode of communication that provides the public
Bill Dedman, and Stephen Doig. In his 1973
verified facts and information in a meaningful
book, Meyer introduced the concept of “precision
context so that citizens can make informed judg-
journalism” and advocated applying social sci-
ments about society. As aggregated, large-scale
ence methodology to investigative reporting prac-
data have become readily available and the prac-
tices. Meyer argued that journalists needed to
tice of journalism has increasingly turned to big
employ the same tools as scientific researchers:
data to help fulfill this mission. Journalists have
databases, spreadsheets, surveys, and computer
begun to apply a variety of computational and
analysis techniques.
statistical techniques to organize, analyze, and
Based on the work of Meyer, computer-
interpret these data, which are then used in con-
assisted reporting developed as a niche form of
junction with traditional news narratives and
investigative reporting by the late 1980s, as com-
reporting techniques. Big data are being applied
puters became smaller and more affordable.
to all facets of news including politics, health, the
A notable example from this period was Bill
economy, weather, and sports.
Dedman’s Pulitzer Prize winning series “The
The growth of “data-driven journalism” has
Color of Money.” Dedman obtained lending sta-
changed many journalists’ news gathering rou-
tistics on computer tape through the federal Free-
tines by altering the way news organizations inter-
dom of Information Act. His research team
act with their audience, providing new forms of
combined that data with demographic information
content for the public and incorporating new
from the US Census. Dedman found widespread

# Springer International Publishing AG 2017

L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_124-1
2 Journalism
racial discrimination in mortgage lending prac-
tices throughout the Atlanta metropolitan area.
Over the last decade, the ubiquity of large,
often free, data sets has created new opportunities
for journalists to make sense of the world of big
data. Where precision journalism was once the
domain of a few investigative reporters, data-
driven reporting techniques are now a common,
if not necessary, component of contemporary
news work. News organizations like The Guard-
ian, The New York Times’ Upshot, and The Texas
Tribune represent the mainstream embrace of big
data. Some websites, like Nate Sliver’s
FiveThirtyEight, are entirely devoted to data
journalism.
How Do Journalists Use Big Data?
Big data provide journalists with new and alterna-
tive ways to approach the news. In traditional
journalism, reporters collect and organize infor-
mation for the public, often relying on interviews
and in-depth research to report their stories. Big
data allow journalists to move beyond these stan-
dard methods and report the news by gathering
and making sense of aggregated data sets. This
shift in methods has required some journalists and
news organizations to change their information-
gathering routines. Rather than identifying poten-
tial sources or key resources, journalists using big
data must first locate relevant data sets, organize
the data in a way that allows them to tell a coher-
ent story, analyze the data for important patterns
and relationships, and, finally, report the news in a
comprehensible manner. Because of the complex-
ity of the data, news organizations and journalists
are increasingly working alongside computer pro-
grammers, statisticians, and graphic designers to
help tell their stories.
One important aspect of big data is visualiza-
tion. Instead of writing a traditional story with
text, quotations, and the inverted-pyramid format,
big data allow journalists to tell their stories using
graphs, charts, maps, and interactive features.
These visuals enable journalists to present
insights from complicated data sets in a format
that is easy for the audience to understand. These
visuals can also accompany and buttress news
articles that rely on traditional reporting methods.
Nate Silver writes that big data analyses pro-
vide several advantages over traditional journal-
ism. They allow journalists to further explain a
story or phenomenon through statistical tests that
explore relationships, to more broadly generalize
information by looking at aggregate patterns over
time and to predict future events based on prior
occurrences. For example, using an algorithm
based on historical polling data, Silver’s website,
FiveThirtyEight (formerly hosted by the New York
Times), correctly predicted the outcome of the
2012 US presidential election in all 50 states.
Whereas methods of traditional journalism often
lend themselves to more microlevel reporting,
more macrolevel and general insights can be
gleaned from big data.
An additional advantage of big data is that, in
some cases, they reduce the necessary resources
needed to report the story. Stories that would
otherwise have taken years to produce can be
assembled relatively quickly. For example,
WikiLeaks provided news organizations nearly
400,000 unreleased US military reports related
to the war in Iraq. Sifting through these docu-
ments using traditional reporting methods would
take a considerable amount of time, but news
outlets like The Guardian in the UK applied com-
putational techniques to quickly identify and
report the important stories and themes stemming
from the leak, including a map noting the location
of every death in the war.
Big data also allow journalists to interact with
their audience to report the news. In a process
called crowdsourcing the news, large groups of
people contribute relevant information about a
topic, which in the aggregate can be used to
make generalizations and identify patterns and
relationships. For example, in 2013 the
New York Times website released an interactive
quiz on American dialects that used responses to
questions about accents and phrases to demon-
strate regional patterns of speech in the US. The
quiz became the most visited content on the
website that year.
Journalism 3

Data Sets and Methodologies be designed to automatically write news stories,

without a human author. These automated “robot
Journalists have a multitude of large data sets and journalists” have been used to produce stories for
methodologies at their disposal to create news news outlets like the Associated Press and The
stories. Much of the data used is public and orig- Los Angeles Times. Algorithms have also changed
inates from government agencies. For example, the way news is delivered, as news aggregators
the US government has created a website, data. like Google News employ these methods to col-
gov, which offers over 100,000 datasets in a vari- lect and provide users personalized news feed.
ety of areas including education, finance, health,
jobs, and public safety. Other data, like the
WikiLeaks reports, were not intended to be public
Limitations of Big Data for Journalism
but became primary sources of big data for jour-
nalists. News organizations can also utilize publi-
Although big data offer numerous opportunities to
cally available data from private Internet
journalists reporting the news, scholars and prac-
companies like Google or social networking
titioners have both highlighted several potential
sites such as Facebook and Twitter to help report
general limitations of these data. As much as big
the news.
data can help journalists in their reporting, they
Once the data are secured, journalists can apply
need to make an active effort to contextualize the
numerous techniques to make sense of the data.
information. Big data storytelling also elicits
For example, at a basic level, journalists could get
moral and ethical concerns with respect the data
a sense of public interest about a topic or issue by
collection of individuals as aggregated informa-
examining the volume of online searches about
tion. These reporting techniques also need to bear
the topic or the number of times it was referenced
in mind potential data privacy transgressions.
in social media. Mapping or charting occurrences
of events across regions or countries also offers
basic descriptive visualizations of the data. Jour-
nalists can also apply content or sentiment ana- Cross-References
lyses to get a sense of the patterns of phrases or
tone within a set of documents. Further, network ▶ Big Data Storytelling (Digital Storytelling)
analyses could be utilized to assess connections ▶ Computational Social Sciences
between points in the data set, which could pro- ▶ Data Visualization
vide insights on the flow or movement of infor- ▶ Information Society
mation, or on power structures. ▶ Interactive Data Visualization
These methods can be combined to produce a ▶ Open Data
more holistic account of events. For example,
journalists at the Associated Press used textual
and network analysis to examine almost 400,000
Further Readings
WikiLeaks documents related to the Iraq war that
identified related clusters of words used in the Pew Research Center. The core principles of journalism.
reports. In doing so, they were able to demonstrate http://www.people-press.org/1999/03/30/section-i-the-
patterns of content within the documents, which core-principles-of-journalism. Accessed April 2016.
Shorenstein Center on Media, Politics and Public Policy.
shed previously unseen light on what was happen-
Understanding data journalism: Overview of resources,
ing on the ground during the war. tools and topics. http://journalistsresource.org/reference/
Computer algorithms, and self-taught machine reporting/understanding-data-journalism-overview-tools-
learning techniques, also play an important role in topics. Accessed April 2016.
Silver, N. What the fox knows. http://fivethirtyeight.com/
the big data journalistic process. Algorithms can
features/what-the-fox-knows. Accessed August 2014.
4 Journalism

Special Issues and Volumes
Digital Journalism–Journalism in an Era of Big Data:
Cases, concepts, and critiques. v. 3/3 (2015).
Social Science Computer Review – Citizenship, Social
Media, and Big Data: Current and Future Research in
the Social Sciences (in press).
The ANNALS of American of the American Academy of
Political and Social Science – Toward Computational
Social Science: Big Data in Digital Environments.
v. 659/1 (2015).
K
Keystroke Capture
Gordon Alley-Young
Department of Communications and Performing
Arts, Kingsborough Community College, City
University of New York, New York, NY, USA
Synonyms
Keycatching; Keylogger; Keystroke logger;
Keystroke recorder
Introduction
Keystroke capture (KC) tracks a computer or
mobile device users’ keyboard activity using
hardware or software. KC is used by businesses
to keep employees from misusing company tech-
nology, in families to monitor the use possible
misuse of family computers, and by computer
hackers who seek gain through secretly
possessing an individual’s personal information
and account passwords. KC software can be pur-
chased for use on a device or may be placed
maliciously without the user’s knowledge through
contact with untrusted websites or e-mail attach-
ments. KC hardware can also be purchased and is
disguised to look like computer cords and acces-
sories. KC detection can be difficult because soft-
ware and hardware are designed to avoid
detection by anti-KC programs. KC can be
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_125-1
avoided by using security software as well as
through careful computing practices. KC affects
individual computer users as well as small,
medium, and large organizations internationally.
How Keystroke Capture (KC) Works
Keystroke capture (KC), also called keystroke
logger, keylogger, keystroke recorder, and
keycatching, tracks a computer or mobile device
users’ activities, including keyboard activity,
using hardware or software. KC is knowingly
employed by businesses to deter its employees
from misusing company devices and also by fam-
ilies seeking to monitor the technology activities
of vulnerable family members (e.g., teens, chil-
dren). Romantic partners and spouses use KC to
catch their significant others engaged in deception
and/or infidelity. Computer hackers install KC
onto unsuspecting users’ devices in order to steal
their personal data, website passwords, financial
information, read their correspondence/online
communication, to stalk/harass/intimidate users,
and/or to sabotage organizations or individuals
that hackers consider unethical. When used
covertly to hurt and/or steal from others, KC is
called malware, malicious software used to inter-
fere with a device, and/or spyware, software used
to steal information or to spy on someone.
KC software (e.g., WebWatcher, SpectorPro,
Cell Phone Spy) is available for free and also for
purchase, and it is usually downloaded onto the
2 Keystroke Capture
device where it either saves captured data onto the
hard drive or sends it through networks/wirelessly
to another device/website. KC hardware (e.g.,
KeyCobra, KeyGrabber, KeyGhost) may be an
adaptor device into which a keyboard/mouse
USB cord is plugged before it is inserted in to
the computer or may look like an extension cable.
Hardware can also be installed inside the com-
puter/keyboard. KC is placed on devices mali-
ciously by hackers when computer and mobile
device users visit websites, open e-mail attach-
ments, or click links to files that are from
untrusted sources. Individual technology users
are frequently lured by untrusted sources and
websites that offer free music files or pornogra-
phy. KC’s infiltrate organizations’ computers
when an employee is completing company busi-
ness (i.e., financial transactions) on a device that
he/she also uses to surf the Internet in their
free time.
When a computer is infected with a malicious
KC, it can be turned into what is called a zombie, a
computer that is hijacked and used to spread KC
malware/spyware to other unsuspecting individ-
uals. A network of zombie computers that is con-
trolled by someone other than the legitimate
network administrator is called a botnet. In 2011,
the FBI shut down the Coreflood botnet, a global
KC operation affecting 2 million computers. This
botnet spread KC software via an infected e-mail
attachment and seemed to infect only computers
using Microsoft Windows operating systems. The
FBI seized the operators’ computers and charged
13 “John Doe” defendants with wire fraud, bank
fraud, and illegally intercepting electronic com-
munication. Then in 2013 security firm
SpiderLabs found 2 million passwords in the
Netherlands stolen by the Pony botnet. While
researching the Pony botnet, SpiderLabs discov-
ered that it contained over a million and a half
Twitter and Facebook passwords and over
300,000 Gmail and Yahoo e-mail passwords. Pay-
roll management company ADP, with over
600,000 clients in 125 countries, was also hacked
by this botnet.
The Scope of the Problem
Internationally
In 2013 the Royal Canadian Mounted Police
(RCMP) served White Falcon Communications
with a warrant that alleged that the company was
controlling an unknown number of computers
known as the Citadel botnet (Vancouver Sun
2013). In addition to distributing KC malware/
spyware, the Citadel botnet also distributed spam
and conducted network attacks that reaped over
$500 million dollars illegal profit affecting more
than 5 million people globally (Vancouver Sun
2013). The Royal Bank of Canada and HSBC in
Great Britain were among the banks attacked by
the Citadel botnet (Vancouver Sun 2013). The
operation is believed to have originated from
Russia or Ukraine as many websites hosted by
White Falcon Communications end in the .ru suf-
fix (i.e., country code for Russia). Microsoft
claims that the 1,400 botnets running Citadel
malware/spyware were interrupted due to the
RCMP action with the highest infection rates in
Germany (Vancouver Sun 2013). Other countries
affected were Thailand, Italy, India, Australia, the
USA, and Canada. White Falcon owner Dmitry
Glazyrin’s voicemail claimed he was out of the
country on business when the warrant was served
(Vancouver Sun 2013).
Trojan horses allow others to access and install
KC and other malware. Trojan horses can alter or
destroy a computer and its files. One of the most
infamous Trojan horses is called Zeus. Don Jack-
son, a senior security researcher with Dell
SecureWorks and who has been widely
interviewed, claims that Zeus is so successful
because those behind it, seemingly in Russia, are
well funded and technologically experienced, and
this allows them to keep Zeus evolving into dif-
ferent variations (Button 2013). In 2012 Micro-
soft’s Digital Crimes Unit with its partners
disrupted a variation of Zeus botnets in Pennsyl-
vania and Illinois responsible for an estimated
13 million infections globally. Another variation
of Zeus called GameOver tracks computer users’
every login and uses the information to lock them
out and drain their bank accounts (Lyons 2014). In
some instances GameOver works in concert with
Keystroke Capture
CryptoLocker. If GameOver finds that an individ-
ual has little in the bank then CryptoLocker will
encrypt users’ valuable personal and business files
agreeing to release them only once a ransom is
paid (Lyons 2014). Often ransoms must be paid in
Bitcoin, Internet based and currently anonymous
and difficult to track. Victims of CryptoLocker
will often receive a request for a one Bitcoin
ransom (estimated to be worth 400€/$500USD)
to unlock the files on their personal computer that
could include records for a small business, aca-
demic research, and/or family photographs
(Lyons 2014).
KC is much more difficult to achieve on a
smartphone as most operating systems operate
only one application at a time, but it is not impos-
sible. As an experiment Dr. Hao Chen, an Asso-
ciate Professor in the Department of Computer
Science at the University of California, Davis,
with an interest in security research created a KC
software that operates using smartphone motion
data. When tested, Chen’s application correctly
guessed more than 70% of the keystrokes on a
virtual numerical keypad though he asserts that it
would probably be less accurate on an alphanu-
merical keypad (Aron 2011). Point-of-sale (POS)
data, gathered when a credit card purchase is made
in a retail store or restaurant, is also vulnerable to
KC software (Beierly 2010). In 2009 seven Lou-
isiana restaurant companies (i.e., Crawfish Town
USA Inc., Don’s Seafood & Steak House Inc.,
Mansy Enterprises LLC, Mel’s Diner Part II Inc.,
Sammy’s LLC, Sammy’s of Zachary LLC, and
B.S. & J. Enterprises Inc.) sued Radiant Systems
Inc., a POS system maker, and Computer World
Inc., a POS equipment distributor, charging that
the vendors did not secure the Radiant POS sys-
tems. The customers were then defrauded by KC
software, and restaurant owners incurred financial
costs related to this data capture. Similarly, Patco
Construction Company, Inc. sued People’s United
Bank for failing to implement sufficient security
measures to detect and address suspicious trans-
actions due to KC. The company finally settled for
$345,000, the cost that was stolen plus interest.
Teenage computer hackers, so-called hactivists
(people who protest ideologically by hacking
computers), and governments under the auspices
3
of cyber espionage engage in KC activities, but
cyber criminals attain the most notoriety. Cyber
criminals are as effective as they are evasive due
to the organization of their criminal gangs. After
taking money from bank accounts via KC, many
cyber criminals send the payments to a series of
money mules. Money mules are sometimes
unwitting participants in fraud who are recruited
via the Internet with promises of money for work-
ing online. The mules are then instructed to wire
the money to accounts in Russia and China (Krebs
2009). Mules have no face-to-face contact with
the heads of KC operations so it can be difficult to
secure prosecutions, though several notable cyber
criminals have been identified, charged, and/or
arrested. In late 2013 the RCMP secured a warrant
for Dmitry Glazyrin, the apparent operator of a
botnet who left Canada before the warrant could
be served. Then in early 2014, Russian SpyEye
creator Aleksandr Panin was arrested for cyber
crime (IMD 2014). Also, Estonian Vladimir
Tsastsin, the cyber criminal who created
DNSChanger and became rich of online advertis-
ing fraud and KC by infecting millions of com-
puters. Finnish Internet security expert Mikko
Hermanni Hyppönen claimed that Tsastsin
owned 159 Estonian properties when he was
arrested in 2011 (IMD 2014). Tsastsin was
released 10 months after his arrest due to insuffi-
cient proof. As of 2014 Tsastsin has been extra-
dited to the US for prosecution (IMD 2014). Also
in 2014 the US Department of Justice Department
(DOJ) filed papers accusing a Russian Evgeniy
Mikhailovich Bogachev of leading the gang
behind GameOver Zeus. The DOJ claims
GameOver Zeus caused $100 million in losses
from individuals and large organizations.
Suspected Eastern European malware/spyware
oligarchs have received ample media attention for
perpetuating KC via botnets and Trojan horses
while other perpetrators have taken the public by
surprise. In 2011 critics accused software com-
pany Carrier IQ of placing KC and geographical
position spyware in millions of users’ Android
devices (International Business Times 2011).
The harshest of critics have alleged illegal
wiretapping on the part of the company while
Carrier IQ has rebutted that what was identified
4 Keystroke Capture
as spyware is actually diagnostic software that
provides network improvement data (Interna-
tional Business Times 2011). Further the com-
pany stated that the data was both encrypted and
secured and not sold to third parties. In January
2014, 11 students were expelled from Corona del
Mar High School in California’s affluent Orange
County for allegedly using KC to cheat for several
years with the help of tutor Timothy Lai. Police
report being unable to find Lai, a former resident
of Irvine, CA, since the allegations surfaced in
December 2013. The students are accused of plac-
ing KC hardware onto teachers’ computers to get
passwords to improve their grades and steal
exams. All 11 students signed expulsion agree-
ments in January 2014 that whereby they aban-
doned their right to appeal their expulsions in
exchange for being able to transfer to other
schools in the district. Subsequently, five of the
students’ families sued the district for denying the
students the right to appeal and/or claiming tutor
Lai committed the KC crimes. By the end of
March, the school district had spent almost
$45,000 in legal fees.
When large organizations are hacked via KC,
the news is reported widely. For instance, Visa
found KC software being able to transmit card
data to a fixed e-mail or IP address where hackers
could retrieve it. Here the hackers attached KC to
a POS system. Similarly KC was used to capture
the keystrokes of pilots flying the US military’s
Predator and Reaper drones that have been used in
Afghanistan (Shachtman 2011). Military officials
were unsure whether the KC software was already
built into the drones was the work of a hacker
(Shachtman 2011). Finally, Kaspersky Labs has
publicized how it is possible to get control of
BMW’s Connected Drive system via KC and
other malware, and this gain control of a luxury
car that uses this Internet-based system.
Research by Internet security firm Symantec
shows that many small and medium-sized busi-
nesses believe that malware/spyware is a problem
for large organizations (e.g., Visa, the US mili-
tary). However, since 2010 the company notes
that 40% of all companies attacked have fewer
than 500 employees while only 28% of attacks
target large organizations. A case in point is a
2012–2013 attack on a California escrow firm,
Efficient Services Escrow Group of Huntington
Beach, CA, that had one location and nine
employees. Using KC malware/spyware, the
hackers drained the company of $1.5 million dol-
lars in three transactions wired to bank accounts in
China and Russia. Subsequently, $432,215 sent to
a Moscow Bank was recovered, while the $1.1
million sent to China was never recouped. The
loss was enough to shutter the business’s one
office and put its nine employees out of work.
Though popular in European computer circles,
the relatively low-profile Chaos Computer Club
learned that German state police were using KC
malware/spyware as well as saving screenshots
and activating the cameras/microphones of club
members (Kulish and Homola 2014). News of the
police’s actions led the German justice minister to
call for stricter privacy rules (Kulish and Homola
2014). This call echoes a 2006 commission report
to the EU Parliament that calls for strengthening
the regulatory framework for electronic commu-
nications. KC is a pressing concern in the US for
as of 2014, 18 states and one territory (i.e., Alaska,
Arizona, Arkansas, California, Georgia, Illinois,
Indiana, Iowa, Louisiana, Nevada, New Hamp-
shire, Pennsylvania, Rhode Island, Texas, Utah,
Virginia, Washington, Wyoming, Puerto Rico) all
have anti-spyware laws on the books (NCSL
2015).
Tackling the Problem
The problem of malicious KC can be addressed
through software interventions and changes in
computer users’ behaviors, especially when
online. Business travelers may be at a greater
risk for losses if they log onto financial accounts
using hotel business centers as these high-traffic
areas provide ample opportunities to hackers
(Credit Union Times 2014). Many Internet secu-
rity experts recommend not using public wireless
networks where of KC spyware thrives. Experts at
Dell also recommend that banks have separate
computers dedicated only to banking transactions
with no emailing or web browsing.
Keystroke Capture
Individuals without the resources to devote one
computer to financial transactions can, experts
argue, protect themselves from KC through
changing several computer behaviors. First, indi-
viduals should change their online banking pass-
words regularly. Second, they should not use the
same password for multiple accounts or use com-
mon words or phrases. Third is checking one’s
bank account on a regular basis for unauthorized
transfers. Finally, it is important to log off of
banking websites when finished with them and
to never click on third-party advertisements that
post to online banking sites and take you to a new
website upon clicking.
Configurations of one’s computer features,
programs, and software are also urged to thwart
KC. This includes removing remote access (i.e.,
accessing one’s work computer from home) con-
figurations when they are not needed in addition
to using a strong firewall (Beierly 2010). Users
need to continually check their devices for unfa-
miliar hardware attached to mice or keyboards as
well as check the listings of installed software
(Adhikary et al. 2012; Beierly 2010). Many finan-
cial organizations are opting for virtual keypads
and virtual mice, especially for online transactions
(Kumar 2009). Under this configuration instead of
typing a password and username on the keyboard
using number and letter keys, the user scrolls
through numbers and letters using the cursors’
virtual keyboard. Always use the online virtual
keyboard for your banking password to avoid
the risk of keystrokes being logged when
available.
Conclusion
Having anti-KC/malware/spyware alone does not
guarantee protection, but experts agree that it is an
important component of an overall security strat-
egy. Anti-KC programs include SpyShelter Stop-
Logger, Zemana AntiLogger, KeyScrambler Pre-
mium, Keylogger Detector, and GuardedID Pre-
mium. Some computer experts claim that PC’s are
more susceptible to KC malware/spyware than are
Mac’s as KC malwares/spywares are often
reported to exploit holes in PC’s operating
5
systems, but new wisdom suggests that all devices
can be vulnerable especially when programs and
plug-ins are added to devices. Don Jackson, a
senior security researcher with Dell SecureWorks,
argues that one of the most effective methods for
preventing online business fraud, the air-gap tech-
nique, is not widely utilized despite being around
since 2005. The air-gap technique creates a unique
verification code that is transmitted as a digital
token, text message, or other device not connected
to the online account device, so the client can read
and then key in the code as a signature for each
transaction over a certain amount. Alternately in
2014 Israeli researchers presented research on a
technique to hack an air-gap network using just a
cellphone.
Cross-References
▶ Banking Industry
▶ Canada
▶ China
▶ Cyber Espionage
▶ Cyber Threat/Attack
▶ Department of Homeland Security
▶ Germany
▶ Microsoft
▶ Point-of-Sales Data
▶ Royal Bank of Canada
▶ Spyware
▶ Visa
Further Readings
Adhikary, N., Shrivastava, R., Kumar, A., Verma, S., Bag,
M., & Singh, V. (2012). Battering keyloggers and
screen recording software by fabricating passwords.
International Journal of Computer Network & Infor-
mation Security, 4(5), 13–21.
Aron, J. (2011). Smartphone jiggles reveal your private
data. New Scientist, 211(2825), 21.
Beierly, I. (2010). They’ll be watching you. Retrieved from
http://www.hospitalityupgrade.com/_files/File_Articles/
HUSum10_Beierly_Keylogging.pdf
Button, K. (2013). Wire and online banking fraud continues
to spike for businesses. Retrieved from http://www.
americanbanker.com/issues/178_194/wire-and-online-
banking-fraud-continues-to-spike-for-businesses-1062
666-1.html
6 Keystroke Capture
Credit Union Times. (2014). Hotel business centers
hacked. Credit Union Times, 25(29), 11.
IMD: International Institute for Management Develop-
ment. (2014). Cybercrime buster speaks at IMD.
Retrieved from http://www.imd.org/news/Cybercrime-
buster-speaks-at-IMD.cfm
International Business Times. (2011). Carrier iq spyware:
Company’s Android app logging the keystrokes of mil-
lions. Retrieved from http://www.ibtimes.com/carrier-
iq-spyware-companys-android-app-logs-keystrokes-
millions-video-377244
Krebs, B. (2009). Data breach highlights role of ‘money
mules’. Retrieved from http://voices.washingtonpost.
com/securityfix/2009/09/money_mules_carry_loot_for_
org.html
Kulish, N., & Homola, V. (2014). Germans condemn police
use of spyware. Retrieved from http://www.nytimes.
com/2011/10/15/world/europe/uproar-in-germany-on-
police-use-of-surveillance-software.html?_r=0
Kumar, S. (2009). Handling malicious hackers & assessing
risk in real time. Siliconindia, 12(4), 32–33.
Lyons, K. (2014). Is your computer already infected with
dangerous Gameover Zeus software? Virus could be
lying dormant in thousands of Australian computers.
Retrieved from http://www.dailymail.co.uk/news/article-
2648038/Gameover-Zeus-lying-dormant-thousands-
Australian-computers-without-knowing.html#ixzz3
AmHLKlZ9
NCSL: National Conference of State Legislatures. (2015).
State spyware laws. Retrieved from http://www.ncsl.
org/research/telecommunications-and-information-tech
nology/state-spyware-laws.aspx
Shachtman, N. (2011). Exclusive: Computer virus hits US
drone fleet. Retrieved from http://www.wired.com/
2011/10/virus-hits-drone-fleet/
Vancouver Sun. (2013). Police seize computers linked to
large cybercrime operation: Malware Responsible for
over $500 million in losses has affected more than five
million people globally. Retrieved from http://www.
vancouversun.com/news/Police+seize+computers+
linked+large+cybercrime+operation/8881243/story.html
#ixzz3Ale1G13s
L
LexisNexis
Jennifer Summary-Smith
Culver-Stockton College, Canton, MO, USA
As stated on its website, LexisNexis is a leading
global provider of content-enabled workflow
solutions. This corporation provides data and
solutions for professionals in areas such as the
academia, accounting, corporate world, govern-
ment, law enforcement, legal, and risk manage-
ment. LexisNexis is a subscription-based service,
with two data centers located in Springfield and
Miamisburg, Ohio. The centers are among the
largest complexes of their kind in the United
States, providing LexisNexis with “one of the
most complete comprehensive collections of
online information in the world.”
Data Centers
The LexisNexis data centers hold network
servers, software, and telecommunication equip-
ment, which is a vital component of the entire
range of LexisNexis products and services. The
data centers service the LexisNexis Group
Inc. providing assistance for application develop-
ment, certification and administrative services,
and testing. The entire complex serves its Reed
Elsevier sister companies while also providing
LexisNexis customers with the following: backup
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_127-1
services, data hosting, and online services.
LexisNexis opened its first remote data center
and development facility in
Springfield, Ohio, in 2004, which hosts new prod-
uct development. Both data centers function as a
backup and recovery facility for each other.
According to the LexisNexis’ website, its cus-
tomers use services that span multiple servers and
operating systems. For example, when a sub-
scriber submits a search request, the systems
explore and sift through massive amounts of
information. The answer set is typically returned
to the customer within 6–10 s, resulting in a
99.99% average for reliability and availability of
the search. This service is accessible to five mil-
lion subscribers, with nearly five billion docu-
ments of source information available online and
stored in the Miamisburg facility. The online ser-
vices also provide access to externally hosted data
from the Delaware Secretary of State, Dun &
Bradstreet Business Reports, Historical Quote,
and Real-Time Quote. Given that a large incentive
for data center services is to provide expansion
capacity for all future hosting opportunities, this
has led to an increase in the percentage of total
revenue for Reed Elsevier. Currently, the
Miamisburg data center supports over two billion
dollars in online revenue for Reed Elsevier.
2 LexisNexis
Mainframe Servers
There are over 100 servers housed in the Spring-
field center, managing over 100 terabytes of data
storage. As for the Miamisburg location, this com-
plex holds 11 huge mainframe servers, running
34 multiple virtual storage (MVS) operating sys-
tem images. The center also has 300 midrange
Unix servers and almost 1,000 multiprocessor
NT servers. They provide a wide range of com-
puter services including patent images to cus-
tomers, preeminent US case law citation
systems, a hosting channel data for Reed Elsevier,
and computing resources for the LexisNexis
enterprise. As the company states, its processors
have access to over 500 terabytes (or one trillion
characters) of data storage capacity.
Telecommunications
LexisNexis has developed a large telecommuni-
cations network, permitting the corporation to
support its data collection requirements while
also serving its customers. As noted on its
website, subscribers to the LexisNexis Group
have a search rate of one billion times annually.
LexisNexis also provides bridges and routers and
maintains firewalls, high-speed lines, modems,
and multiplexors, providing an exceptional degree
of connectivity.
Physical Dimensions of the Miamisburg
Data Center
LexisNexis Group has hardware, software, elec-
trical, and mechanical systems housed in a
73,000 ft2 data center hub. Its sister complex,
located in Springfield, comprises a total of
80,000 ft2. In these facilities, the data center hard-
ware, software, electrical, and mechanical sys-
tems have multiple levels of redundancy, in the
event that a single component fails, ensuring
uninterrupted service. The company’s website
states that its systems are maintained and tested
on a regular basis to ensure they perform correctly
in case of an emergency. The LexisNexis Group
also holds and stores copies of critical data off-
site. Multiple times a year, emergency business
resumption plans are tested. Furthermore, the data
center has system management services 365 days
a year and 24 h a day provided by skilled opera-
tions engineers and staff. If needed, there are
additional specialists on site, or on call, to provide
the best support to customers. According to its
website, LexisNexis invests a great deal in protec-
tion architecture to prevent hacking attempts,
viruses, and worms. In addition, the company
also has third-party contractors which conduct
security studies.
Security Breach
In 2013, Byron Acohido reported that a hacking
group hit three major data brokerage companies.
LexisNexis, Dun & Bradstreet, and Kroll Back-
ground America are companies that stockpile and
sell sensitive data. The group that hacked these
data brokerage companies specialized in
obtaining and selling social security numbers.
The security breach was disclosed by a cyberse-
curity blogger Brian Kebs. He stated that the
website ssndob.ms (SSNDOB), their acronym
stands for social security number and date of
birth, markets itself on underground cybercrime
forums, offering services to its customers who
want to look up social security numbers, birth-
days, and other data on any US resident.
LexisNexis found an unauthorized program called
nbc.exe on its two systems listed in the botnet
interface network located in Atlanta, Georgia.
The program was placed as far back as April
2013, compromising their security for at least
5 months.
LexisNexis Group Expansion
As of July 2014, LexisNexis Risk Solutions
expanded its healthcare solutions to the life sci-
ence marketplace. In an article by Amanda Hall,
she notes that an internal analysis revealed that
40% of the customer files have missing or inaccu-
rate information in a typical life science company.
LexisNexis
LexisNexis Risk Solutions has leveraged its lead-
ing databases, reducing costs, improving effec-
tiveness, and strengthening identity transparency.
LexisNexis is able to deliver data to over 6.5
million healthcare providers in the United States.
This will benefit life science companies allowing
them to tailor their marketing and sales strategies,
to identify the correct providers to pursue. The
LexisNexis databases are more efficient, which
will help health science organizations gain com-
pliance with federal and state laws.
Following the healthcare solutions announce-
ment, Elisa Rodgers writes that Reed Technology
and Information Services, Inc., a LexisNexis com-
pany, acquired PatentCore. PatentCore is an inno-
vator of patent data analytics. PatentAdvisor is a
user-friendly suite, delivering information to
assist with a more effective patent prosecution
and management. Its web-based patent analytic
tools will help IP-driven companies and law
firms by making patent prosecution a more strate-
gic and probable process.
The future of the LexisNexis Group should
include more acquisitions, expansion, and
increased capabilities for the company. According
to its website, the markets for their companies
have grown over the last three decades, servicing
professionals in academic institutes, corporations,
governments, and business people. LexisNexis
Group provides critical information, in easy-to-
use electronic products, to the benefit of sub-
scribed customers. The company has a long his-
tory of fulfilling its mission statement “to enable
its customers to spend less time searching for
critical information and more time using
LexisNexis knowledge and management tools to
guide critical decisions.” For more than a century,
3
legal professionals have trusted the LexisNexis
Group. It appears that the company will continue
to maintain this status and remain one of the
leading providers in the data brokerage
marketplace.
Cross-References
▶ American Bar Association
▶ Big Data Quality
▶ Data Breach
▶ Data Center
▶ Legal Issues
▶ Reed Elsevier
Further Readings
Acohido, B. LexisNexis, Dunn & Bradstreet, Kroll Hacked.
http://www.usatoday.com/story/cybertruth/2013/09/26/
lexisnexis-dunn–bradstreet-altegrity-hacked/2878769/.
Accessed July 2014.
Hall, A. LexisNexis verified data on more than 6.5 million
providers strengthens identity transparency and reduces
costs for life science organizations. http://www.benzi
nga.com/pressreleases/14/07/b4674537/lexisnexis-veri
fied-data-on-more-than-6-5-million-providers-strengt
hens. Accessed July 2014.
Krebs, B. Data broker giants hacked by ID theft service.
http://krebsonsecurity.com/2013/09/data-broker-giants-
hacked-by-id-theft-service/. Accessed July 2014.
LexisNexis. http://www.lexisnexis.com. Accessed July
2014.
Rodgers, E. Adding multimedia reed tech strengthens line
of LexisNexis intellectual property solutions by acquir-
ing PatentCore, an innovator in patent data analytics.
http://in.reuters.com/article/2014/07/08/supp-pa-reed-
technology-idUSnBw015873a+100+BSW20140708.
Accessed July 2014.
L
Link/Graph Mining
Derek Doran
Department of Computer Science and
Engineering, Wright State University, Dayton,
OH, USA
Synonyms
Network analysis; Network science; Relational
data analytics
Definition/Introduction
Link/graph mining is defined as the extraction of
information within a collection of interrelated
objects. Whereas conventional data mining ima-
gines a database as a collection of “flat” tables,
where entities are rows and attributes of these
entities are columns, link/graph mining imagines
entities as nodes or vertices in a network, with
attributes attached to the nodes themselves. Rela-
tionships among datums in a “flat” database may
be seen by primary key relationships or by com-
mon values across a set of attributes. In the link/
graph mining view of a database, these relation-
ships are made explicit by defining links or edges
between vertices. The edges may be homoge-
neous, where a single kind of relationship defines
the edges that are formed, or heterogeneous,
where multiple kinds of data are used to develop
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_129-1
a vertex set, and relationships define edges among
network vertices. For example, a relation from
vertex A to B and a relation from vertex C to
D in a homogeneous graph means that A is related
to B in the same way that C is related to D. An
example of a homogeneous graph may be one
where nodes represent individuals and connec-
tions represent a friendship relationship. An
example of a heterogeneous graph is one where
different types of network devices connect to each
other to form a corporate intranet. Different node
types correspond to different device types, and
different relationships may correspond to the
type of network protocol that two devices use to
communicate with each other. Networks may be
directed (e.g., a link may be presented from A to
B but not vice versa) or undirected (e.g., a link
from A to B exists if and only if a link from B to
A exists). Link/graph mining is intimately related
to network science, which is the scientific study
of the structure of complex systems. Common
link/graph mining tasks include discovering
shortest or expected paths in the network, an
importance ranking of nodes or vertices, under-
standing relationship patterns, identifying com-
mon clusters or regions of a graph, and modeling
propagation phenomena across the graph. Ran-
dom graph models give researchers a way to iden-
tify whether a structural or interaction pattern seen
within a dataset is statistically significant.
2 Link/Graph Mining
Network Representations of Data
While a traditional “tabular” representation of a
dataset contains information necessary to under-
stand a big dataset, a network representation
makes explicit datum relations that may be
implicit in a data table. For example, in a database
of employee personal and their meeting calendars,
a network view may be constructed where
employees are nodes and edges are present if
two employees will participate in the same meet-
ing. The network thus captures a “who works with
who” relationship that is only implicit in the data
table. Analytics over the network representation
itself can answer queries such as “how did some-
body at meeting C hear about information that was
only discussed during meeting A?”, or “which
employee may have been exposed to the most
amount of potential information, rumors, and
views, as measured by participating in many
meetings where few other participants overlap?”
The network representation of data has another
important advantage: the network itself represents
the structure of a complex system of
interconnected participants. These participants
could be people or even components of a physical
system. There is some agreement in the scientific
community that the complexity of most techno-
logical, social, biological, and natural systems is
best captured by its representation as a network.
The field of network science is devoted to the
scientific application of link and graph mining
techniques to quantitatively understand, model,
and make predictions over complex systems. Net-
work science defines two kinds of frameworks
under which link/graph mining is performed:
(i) exploratory analysis and (ii) hypothesis-driven
analysis. In exploratory analysis, an analyst has no
specific notion about why and now nodes in a
complex system connect or are related to each
other or why a complex network takes on a spe-
cific structure. Exploratory analysis leads to a
hypothesis about an underlying mechanism of
the system based on regularly occurring patterns
or based on anomalous graph metrics. In
hypothesis-driven analysis, the analyst has some
at hand evidence supporting an underlying mech-
anism about how a system operates and is
interested in understanding how the structural
qualities of the system speak in favor or in oppo-
sition to the mechanism. Under either setting,
hypotheses may be tested by comparing observa-
tions against random network models to identify
whether or not patterns in support or in opposition
of a hypothesis are significant or merely occurred
by chance. Network science is intimately tied to
link/graph mining: it defines an apparatus for ana-
lysts to use link/graph mining methods that can
answer important questions about a complex sys-
tem. Similarly, network science procedures and
analyses are the primary purpose for the develop-
ment of link/graph mining techniques. The utility
of one would thus not nearly be as high without
the other.
Representation
The mathematical representation of a graph is a
basic preprocessing step for any link/graph mining
task. One form may be as follows: every node in
the graph is labeled with an integer i = 1 . . . n and
a tuple (i, j) is defined for a relationship between
nodes i and j. A network may then be defined by
the value n and a list of all tuples. For example, let
n = 5 and define the set {(1, 2), (3, 4), (2, 4), (4, 1),
(2, 3)}. This specifies a graph with five vertices,
one of which is disconnected (vertex 5) and the
others that have edges between them as defined by
the set. Such a specification of a network is called
an edge list. Another approach is to translate the
edge list representation into an adjacency matrix
A. This is defined as an n  n matrix where the
element Aij, corresponding to the ith row and jth
column of the matrix, is equal to 1 if the tuple (i, j)
or (j, i) exists in the edge list. When edges are
unlabeled or unweighted, A is simply a binary
matrix. Alternatively, if the graph is heteroge-
neous or allows multiple relationships between
the same pair of nodes, then Aij is equal to the
number of edges between i and j. When A is not
symmetric, the graph is directed rather than
undirected.
Types of Link/Graph Mining Techniques
The discovery and analysis of algorithms for
extracting knowledge from networks are ongoing.
Link/Graph Mining
Common types of analyses, emphasizing those
types often used in practice, are explained below.
Path analysis: A path p in a graph is a sequence
of vertices p = (v1, v2, . . . , vm) , vi  V such
that for each consecutive pair vi,vj of vertices in
p is matched by an edge of the form (vj,vi) (if the
network is undirected) or (vi,vj) (if the network is
directed or undirected). If one were to draw a graph
graphically, a path is any sequence of movements
along the edges of the network that brings you from
one vertex to another. Any path is valid, even ones
that have loops or crosses the same vertex many
times. Paths that do not intersect with themselves
(i.e., vi does not equal vj for any vi,vj  p) are self-
avoiding. The length of a path is defined by the
total number of edges along it. Geodesic paths
between vertices i and j is a minimum length path
of size k where p1 = i and pk = j. A breadth-first
search starting from node d, which iterates over all
paths of length 1, and then 2 and 3, and so on up to
the largest path that originates at d, is one way to
compute geodesic paths.
Network interactions: Whereas path analysis
considers the global structure of a graph, the inter-
actions among nodes are a concept related to sub-
graphs or microstructures. Microstructural
measures consider a single node, members of its
nth degree neighborhood (the set of nodes no
more than n hops from it), and the collection of
interactions that run between them. If macro-
measures study an entire system as a whole (the
“forest”), micro-measures such as interactions try
to get at the heart of the individual conditions that
cause nodes to bind together locally (the “trees”).
Three popular features for microstructural analy-
sis are reciprocity, transitivity, and balance.
Reciprocity measures that degree to which
two nodes are mutually connected to each other
in a directed graph. In other words, if one observes
that a node A connects to B, what is the chance
that B will also connect A? The term reciprocity
comes from the field of social network analysis,
which describes a particular set of link/graph min-
ing techniques designed to operate over graphs
where nodes represent people and edges represent
the social relationships among them. For example,
if A does a favor for B, will B also do a favor for
A? If A sends a friend request to B on an online
3
social system, will B reply? On the World Wide
Web, if website A has a hyperlink to B, will B link
to A?
Transitivity refers to the degree to which two
nodes in a network have a mutual connection in
common. In other words, if there is an edge
between nodes A and B and B to C, graphs that
are highly transitive indicate a tendency for an
edge to also exist between A and C. In the context
of social network analysis, transitivity carries an
intuitive interpretation based on the old adage “a
friend of my friend is also my friend.” Transitivity
is an important measure in other contexts, as well.
For example, in a graph where edges correspond
to paths of energy as in a power grid, highly
transitive graphs correspond to more efficient sys-
tems compared to less transitive ones: rather than
having energy take the path A to B to C, a transi-
tive relation would allow a transmission from A to
C directly. The transitivity of a graph is measured
by counting the total number of closed triangles in
the graph (i.e., counting all subgraphs that are
complete graphs of three nodes) multiplied by
three and divided by the total number of
connected triples in the graph (e.g., all sets of
three vertices A, B, and C where at least the
edges (A,B) and (B,C) exist).
Balance is defined for networks where edges
carry a binary variable that, without loss of gen-
erality, is either “positive” (i.e., a “+,” “1,” “Yes,”
“True,” etc.) or “negative” (i.e., a “ ,” “0,” “No,”
“False,” etc.). Vertices incident to positive edges
are harmonious or non-conflicting entities in a
system, whereas vertices incident to negative
edges may be competitive or introduce a tension
in the system. Subgraphs over three nodes that are
complete are balanced or imbalanced depending
on the assignment of + and labels to the edges
of the triangle as follows:
• Three positive: Balanced. All edges are “posi-
tive” and in harmony with each other.
• One positive, two negative: Balanced. In this
triangle, two nodes exhibit a harmony, and
both are in conflict with the same other. The
state of this triangle is “balanced” in the sense
that every node is either in harmony or in
conflict with all others in kind.
4 Link/Graph Mining
• Two positive, one negative: Imbalanced. In this
triangle, node A is harmonious with B, and B is
harmonious with C, yet A and C are in conflict.
This is an imbalanced disagreement since, if
A does not conflict with B, and B does not
conflict with C, one would expect A to also
not conflict with C. For example, in a social
context where positive means friend and neg-
ative means enemy, B can fall into a conflicting
situation when friends A and C disagree.
• Three negative: In this triangle, all vertices are
in conflict with one another. This is a danger-
ous scenario in systems of almost any context.
For example, in a dataset of nations, mutual
disagreements among three states has conse-
quence to the world community. In a dataset of
computer network components, three routers
that are interconnected but in “conflict” (e.g.,
a down connection or a disagreement among
routing tables) may lead to a system outage.
Datasets drawn from social process always
tend toward balanced states because people do
not like tension or conflict. It is thus interesting
to use link/graph mining to study social systems
where balance may actually not hold. If a graph
where most triangles are not balanced comes from
a social system, one may surmise that there exist
latent factors pushing the system toward imbal-
anced states. A labeled complete graph is bal-
anced if every one of its triangles is balanced.
Quantifying node importance: The impor-
tance of a node is related to its ability to reach
out or connect to other nodes. A node may also be
important if it carries a strong degree of “flow,”
that is, if the values of relationships connected to it
are very high (so that it acts as a strong conduit for
the passage of information). Nodes may be impor-
tant if they are vital to maintain network connec-
tivity, so that if an important node was removed,
the graph may suddenly fragment or become dis-
connected. Importance may be measured recur-
sively: a node is important if it is connected to
other nodes that themselves are important. For
example, people who work in the United States
White House or serve as Senior Aids to the Pres-
ident are powerful people, not necessarily because
of their job title but because they have a direct and
strong relationship with the Commander in Chief.
Importance is measured by calculating the cen-
trality of a node in a graph. Different centrality
measures that encode different interpretations of
node importance exist and should thus be selected
according to the analysis at hand. Degree central-
ity defines importance as being proportional to the
number of connections a node has. Closeness
centrality defines importance as having a small
average distance to all other nodes in the graph.
Betweenness centrality defines importance as
being part of as many shortest paths in graph
from other pairs of nodes as possible. Eigenvec-
tor centrality defines importance as being
connected to not only many other nodes but also
to many other nodes that are themselves are
important.
Graph partitioning: In the same way that clus-
ters of datums in a dataset correspond to groups of
points that are similar, interesting, or signify some
other demarcation, vertices in graphs may also be
divided into groups that correspond to a common
affiliation, property, or connectivity structure.
Graph partitioning methods. Graph partitioning
takes as an input the number and size of the groups
and then searches for the “best” partitioning under
these constraints. Community detection algo-
rithms are similar to graph partitioning methods
except that they do not require the number and
size of groups to be specified a priori. But this is
not necessarily a disadvantage to graph
partitioning methods; if a graph miner under-
stands the domain from where the graph came
from well, or if for her application she requires a
partitioning into exactly k groups, graph
partitioning methods should be used.
Conclusion
As systems that our society relies on become ever
more complex, and as technological advances
continue to help us capture the structure of this
complexity at high definition, link/graph mining
methods will continue to rise in prevalence. As the
primary means to understand and extract knowl-
edge from complex systems, link/graph mining
Link/Graph Mining 5

methods need to be included in the toolkit of any Further Readings

big data analyst.
Cook, D. J., & Holder, L. B. (2006). Mining graph data.
Wiley.
Getoor, L., & Diehl, C. P. (2005). Link mining: A survey.
Cross-References ACM SIGKDD Explorations Newsletter, 7(2), 3–12.
Lewis, T. G. (2011). Network science: Theory and appli-
▶ Computer Science cations. Wiley.
Newman, M. (2010). Networks: An introduction. New
▶ Computational Science and Engineering
York: Oxford University Press.
▶ Computational Social Sciences Philip, S. Y., Han, J., & Faloutsos, C. (2010). Link mining:
▶ Graph-Theoretic Computations Models, algorithms, and applications. Berlin: Springer.
▶ Mathematics
▶ Statistics
L
LinkedIn
Jennifer Summary-Smith
Culver-Stockton College, Canton, MO, USA
According to its website, LinkedIn is the largest
professional network in the world servicing over
300 million members in over 200 territories and
countries. Their mission statement is to “connect
the world’s professionals to make them more pro-
ductive and successful. When you join LinkedIn,
you get access to people, jobs, news, updates, and
insights that help you be great at what you do.”
Through its online service, LinkedIn earns around
$473.2 million from premium subscriptions, mar-
keting solutions, and talent solutions. It offers free
and premium memberships allowing people to
network, obtain knowledge, and locate potential
job opportunities. The greatest asset to LinkedIn is
its data, making a significant impact in the job
industry.
Company Information
Cofounder Reid Hoffman conceptualized the
company in his living room in 2002, launching
LinkedIn on May 5, 2003. Hoffman, a Stanford
graduate, became one of PayPal’s earliest execu-
tives. After PayPal was sold to eBay, he
cofounded LinkedIn. The company had one mil-
lion members by 2004. Today, the company is ran
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_130-1
by chief executive, Jeff Weiner, who is also the
former CEO of Yahoo! Inc. LinkedIn’s headquar-
ters are located in Mountain View, California,
with US offices in Chicago, Los Angeles,
New York, Omaha, and San Francisco. LinkedIn
also has international offices in 21 locations and
its online content is available in 23 languages.
LinkedIn currently employs 5,400 full-time
employees with offices in 27 cities globally.
LinkedIn states that professionals are signing up
to join the service at the rate of two new members
per second with 67% of its membership located
outside of the United States. The fastest growing
demographic using LinkedIn are students and
recent college graduates, accounting for around
39 million users. LinkedIn’s corporate talent solu-
tions product lines and its memberships include
all executives from the 2013 Fortune 500 compa-
nies and 89 Fortune 100 companies. In 2012, its
members conducted over 5.7 billion profession-
ally oriented searches, with three million compa-
nies utilizing LinkedIn company pages.
As noted on cofounder Reid Hoffman’s
LinkedIn account, a person’s network is how one
stays competitive as a professional, keeping up-
to-date on one’s industry. LinkedIn provides a
space where professionals learn about key trends,
information, and transformations of their industry.
It provides opportunities for people to find jobs,
clients, and other business connections.
2 LinkedIn
Relevance of Data
MIT Sloan Management Review contributing edi-
tor, Renee Boucher Ferguson, interviewed
LinkedIn’s director of relevance science, Deepak
Agarwal, who states that relevance science at
LinkedIn plays the role of improving the rele-
vancy of its products by extracting information
from LinkedIn data. In other words, LinkedIn
provides recommendations using its data to pre-
dict user responses to different items.
To achieve this difficult task, LinkedIn has
relevance scientists who provide an interdisciplin-
ary approach with backgrounds in computer sci-
ence, economics, information retrieval, machine
learning, optimization, software engineering, and
statistics. Relevance scientists work to improve
the relevancy of LinkedIn’s products. According
to Deepak Agarwal, LinkedIn relevance scientists
significantly enhance products such as advertis-
ing, job recommendations, news, LinkedIn feed,
people recommendations, and much more. He
further points out that most of the company’s
products are based upon its use of data.
Impact on the Recruiting Industry
As it states on LinkedIn’s website, the company’s
free membership allows its members the opportu-
nity to upload resumes and/or curriculum vitae,
join groups, follow companies, establish connec-
tions, view and/or search for jobs, endorse con-
nections, and update profiles. It also suggests to its
members several people that they may know,
based on their connections. LinkedIn’s premium
service provides members with additional bene-
fits, allowing access to hiring managers and
recruiters. Members can send personalized mes-
sages to any person on LinkedIn. Additionally,
members can also find out who has viewed their
profile, detailing how others found them for up to
90 days. There are four premium search filters,
permitting premium members to find decision
makers at target companies. The membership
also provides individuals the opportunity to get
noticed by potential employers. When one applies
as a featured applicant, it raises his or her rank to
the top of the application list. OpenLink is a
network that also lets any member on LinkedIn
to view another member’s full profile to make a
connection.
The premium LinkedIn membership assists
with drawing attention to members’ profile,
adding an optional premium or job seeker badge.
When viewing the job listings, members have the
option to sort by salary range, comparing salary
estimates for all jobs in the United States,
Australia, Canada, and the United Kingdom.
LinkedIn’s premium membership also allows
users to see more profile data in one’s extended
network, including first-, second-, and third-
degree connections. A member’s first-level con-
nections are people that have either received an
invitation from the member or the member sent an
invitation to connect. Second-level connections
are people who are connected to first-level con-
nections but are not connected to the actual mem-
ber. Third-level connections are only connected to
the second-level members. Moreover, members
can receive advice and support from a private
group of LinkedIn experts, assisting with job
searches.
In a recent article by George Anders, he notes
the impact that LinkedIn has made on the
recruiting industry. He spoke with the chief exec-
utive of LinkedIn, Jeff Weiner, who brushes off
comparisons between LinkedIn and Facebook.
While both companies connect a vast amount of
people via the Internet, each social media platform
occupies a different niche within the social net-
working marketplace. Facebook generates 85% of
its revenue from advertisements, whereas
LinkedIn focuses its efforts on monetizing mem-
bers’ information. Furthermore, LinkedIn’s
mobile media experience is growing significantly,
changing the face of job searching, career net-
working, and online profiles. George Anders
also interviewed the National Public Radio head
of talent acquisition, Lars Schmidt, who notes that
recruiters no longer remain chiefly in their offices
but are becoming more externally focused. The
days of exchanging business cards is quickly
being replaced by smartphone applications such
as CardMunch. CardMunch is an iPhone app that
captures business card photos, transferring them
LinkedIn
into digital contacts. In 2011, LinkedIn bought the
company, retooling it to pull up existing LinkedIn
profiles from each card improving the ability of
members to make connections. A significant part
of LinkedIn’s success comes from its dedication
to selling services to people who purchase talent.
The chief executive of LinkedIn, Jeff Weiner,
has created an intense sales-focused culture. The
company celebrates new account wins during its
biweekly meetings. According to George Anders,
LinkedIn has doubled the number of sales
employees in the past year. In addition, the com-
pany has made a $27 billion impact on the
recruiting industry. Jeff Weiner also states that
every time LinkedIn expands its sales team for
hiring solutions, the payoff increases “off the
charts.” He also talks about how sales keep rising
and its customers are spreading enthusiasm for
LinkedIn’s products. Jeff Weiner further states
that once sales are made, LinkedIn customers are
loyal, reoccurring, and low maintenance. This
trend is reflected in current stock market prices in
the job-hunting sector. George Anders writes that
older search firm companies, such as Heidrick &
Struggles that recruits candidates the old fashion
way, have slumped 67%. Monster Worldwide has
experienced a more dramatic drop, tumbling 81%.
As noted on its website, “LinkedIn operates the
world’s largest professional network on the Inter-
net.” This company has made billions of dollars,
hosting a massive amount of data with a member-
ship of 300 million people worldwide. The social
network for professionals is growing at a fast pace
under the tenure of Chief Executive Jeff Weiner.
In a July 2014 article by David Gelles, he reports
that LinkedIn has made its second acquisition in
the last several weeks buying Bizo for $175 mil-
lion dollars. A week prior, it purchased Newsle,
which is a service that combs the web for articles
3
that are relevant to members. It quickly notifies a
person whenever friends, family members,
coworkers, and so forth are mentioned online in
the news, blogs, and/or articles.
LinkedIn continues to make great strides by
leveraging its large data archives, to carve out a
niche in the social media sector specifically
targeting the needs of online professionals. It is
evident that, through the use of big data, LinkedIn
is changing and significantly influencing the job-
hunting process. This company provides a service
that allows its member to connect and network
with professionals. LinkedIn is the world’s largest
professional network, proving to be an innovator
in the employment service industry.
Cross-References
▶ Facebook
▶ Information Society
▶ Online Identity
▶ Social Media
Further Readings
Anders, G. How LinkedIn has turned your resume into a cash
machine. http://www.forbes.com/sites/georgeanders/
2012/06/27/how-linkedin-strategy/. Accessed July 2014.
Boucher Ferguson, R. The relevance of data: Behind the
scenes at LinkedIn. http://sloanreview.mit.edu/arti
cle/the-relevance-of-data-going-behind-the-scenes-
at-linkedin/. Accessed July 2014.
Gelles, D. LinkedIn makes another deal, buying Bizo.
http://dealbook.nytimes.com/2014/07/22/linkedin-does-
another-deal-buying-bizo/?_php=true&_type=blogs&_
php=true&_type=blogs&_php=true&_type=blogs&_
r=2. Accessed July 2014.
LinkedIn. https://www.linkedin.com. Accessed July 2014.
M
Media
Colin Porlezza
IPMZ - Institute of Mass Communication and
Media Research, University of Zurich, Zürich,
Switzerland
Synonyms
Computer-assisted reporting; Data journalism;
Media ethics
Definition/Introduction
Big data can be understood as “the capacity to
search, aggregate and cross-reference large data
sets” (Boyd and Crawford 2012, p. 663). The
proliferation of large amounts of data concerns
the media in at least three different ways. First,
large-scale data collections are becoming an
important resource for journalism. As a result,
practices such as data journalism are increasingly
gaining attention among newsrooms and become
relevant resources as data collected and published
in the Internet expands and legal frameworks to
access public data such as Freedom of Informa-
tion Acts come into effect. Recent success stories
of data journalism such as uncovering the MPs’
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_133-1
expenses scandal in the UK or the giant data leak
in the case of the Panama Papers have contributed
to further improve the capacities to deal with large
amounts of data in newsrooms. Second, big data
are not only important in reference to the practice
of reporting. They also play a decisive role with
regard to what kind of content gets finally
published. Many newsrooms are no longer using
the judgment of human editors alone to decide
what content ends up on their websites; instead
they use real-time data analytics generated by the
clicks of their users to identify trends, to see how
content is performing, and to boost virality and
user engagement. Data is also used in order to
improve product development in entertainment
formats. Social media like Facebook have
perfected this technique by using personal prefer-
ences, tastes, and moods of their users, to offer
personalized content and targeted advertising.
This datafication means that social media trans-
forms intangible elements such as relationships
and transform them into a valuable resource or
an economic asset on which to build entire busi-
ness models. Third, datafication and the use of
large amounts of data give also rise to risks with
regard to ethics, privacy, transparency, and sur-
veillance. Big data can have huge benefits because
it allows organizations to personalize and target
products and services. But at the same time, it
requires clear and transparent information
2 Media
handling governance and data protection. Han-
dling big data increases the risk of paralyzing
privacy, because (social) media or internet-based
services require a lot of personal information in
order to use them. Moreover, analyzing big data
entails higher risks to incur in errors, for instance,
when it comes to statistical calculations or visual-
izations of big data.
Big Data in the Media Context
Within media, big data mainly refers to huge
amounts of structured (e.g., sales, clicks) or
unstructured (e.g., videos, posts, or tweets) data
generated, collected, and aggregated by private
business activities, governments, public adminis-
trations, or online-based organizations such as
social media. In addition, the term big data usually
includes references to the analysis of huge bulks
of data, too. These large-scale data collections are
difficult to analyze using traditional software or
database techniques and request new methods in
order to identify patterns in such a massive and
often incomprehensible amount of data. The
media ecosystem has therefore developed special-
ized practices and tools not only to generate big
data but also to analyze it in turn. One of these
practices to analyze data is called data or data-
driven journalism.
Data Journalism
We live in an age of information abundance. One
of the biggest challenges for the media industry,
and journalism in particular, is to bring order in
this data deluge. It is therefore not surprising that
the relationship between big data and journalism
is becoming stronger, especially because large
amounts of data need new and better tools that
are able to provide specific context, to explain the
data in a clear way, and to verify the information it
contains. Data journalism is thus not entirely dif-
ferent from more classic forms of journalism.
However, what makes it somehow special are
the new opportunities given by the combination
of traditional journalistic skills like research and
innovative forms of investigation thanks to the use
of key information sets, key data and new
processing, analytics, and visualization software
that allows journalists to peer through the massive
amounts of data available in a digital environment
and to show it in a clear and simple way to the
publics. The importance of data journalism is
given by its ability to gather, interrogate, visual-
ize, and mash up data with different sources or
services, and it requires an amalgamation of a
journalist’s “nose for news” and tech savvy
competences.
However, data journalism is not as new as it
seems to be. Ever since organizations and public
administrations collected information or built up
archives, journalism has been dealing with large
amounts of data. As long as journalism has been
practiced, journalists were keen to collect data and
to report them accurately. When the data
displaying techniques got better in the late eigh-
teenth century, newspapers started to use this
know-how to present information in a more
sophisticated way. The first example of data jour-
nalism can be traced back to 1821 and involved
The Guardian, at the time based in Manchester,
UK. The newspaper published a leaked table list-
ing the number of students and the costs for each
school in the British city. For the first time, it was
publicly shown that the number of students
receiving free education was higher than what
was expected in the population. Another example
of early data journalism dates back to 1858, when
Florence Nightingale, the social reformer and
founder of modern nursing, published a report to
the British Parliament about the deaths of soldiers.
In her report she revealed with the help of visual
graphics that the main cause of mortality resulted
from preventable diseases during cure rather than
as a cause from battles.
By the middle of the twentieth century, news-
rooms started to use systematically computers to
collect and analyze data in order to find and enrich
news stories. In the 1950s this procedure was
called computer-assisted reporting (CAR) and is
perhaps the evolutionary ancestor of what we call
data journalism today. Computer-assisted
reporting was, for instance, used by the television
network CBS in 1952 to predict the outcome of
the US presidential election. CBS used a then
famous Universal Automatic Computer
Media
(UNIVAC) and programmed it with statistical
models based on voting behavior from earlier
elections. With just 5% of votes in, the computer
correctly predicted the landslide win of former
World War II general Dwight D. Eisenhower
with a margin of error less than 1%. After this
remarkable success of computer-assisted
reporting at CBS, other networks started to use
computers in their newsrooms as well, particu-
larly for voting prediction. Not one election has
since passed without a computer-assisted predic-
tion. However, computers were slowly introduced
in newsrooms, and only in the late 1960s, they
started to be regularly used in the news production
as well.
In 1967, a journalism professor from the Uni-
versity of North Carolina, Philip Meyer, used for
the first time a quicker and better equipped IBM
360 mainframe computer to do statistical analyses
on survey data collected during the Detroit riots.
Meyer was able to show that not only less edu-
cated Southerners were participating in the riots
but also people who attended college. This story,
published on the Detroit Free Press, won him a
Pulitzer Prize together with other journalists and
marked a paradigm shift in computer-assisted
reporting. On the grounds of this success, Meyer
not only supported the use of computers in jour-
nalistic practices but developed a whole new
approach to investigative reporting by introducing
and using social science research methods in jour-
nalism for data gathering, sampling, analysis, and
presentation. In 1973 he published his thoughts in
the seminal book entitled “Precision Journalism.”
The fact that computer-assisted reporting entered
newsrooms especially in the USA was also
revealed through the increased use of computers
in news organizations. In 1986, the Time maga-
zine wrote that computers are revolutionizing
investigative journalism. By trying to analyze
larger databases, journalists were able to offer a
broader perspective and much more information
about the context of specific events.
The practice of computer-assisted reporting
spread further until, at the beginning of the
1990s, it became a standard routine particularly
in bigger newsrooms. The use of computers,
together with the application of social science
3
methods, has helped – according to Philip
Meyer – to make journalism scientific. Besides,
Meyer’s approach tried also to tackle some of the
common shortcomings of journalism like the
increasing dependence on press releases, shrink-
ing accuracy and trust, or the critique of political
bias. An important factor of precision journalism
was therefore the introduction and the use of sta-
tistical software. These programs enabled journal-
ists for the first time to analyze bigger databases
such as surveys or public records. This new
approach might also be seen as a reaction to alter-
native journalistic trends that came up in the
1990s, for instance, the concept of new journal-
ism. While precision journalism stood for scien-
tific rigor in data analysis and reporting, new
journalism used techniques from fiction to
enhance reading experience.
There are some similarities between data jour-
nalism and computer-assisted reporting: both rely
on specific software programs that enable journal-
ists to transform raw data into news stories. How-
ever, there are also differences between computer-
assisted reporting and data journalism, which are
due to the context in which the two practices were
developed. Computer-assisted reporting tried to
introduce both informatics and scientific methods
into journalism, given that at the time, data was
scarce, and many journalists had to generate their
own data. The rise of the Internet and new media
contributed to the massive expansion of archives,
databases, and to the creation of big data. There is
no longer a poverty of information, data is now
available in abundance. Therefore, data journal-
ism is less about the creation of new databases, but
more about data gathering, analysis, and visuali-
zation, which means that journalists have to look
for specific patterns within the data rather than
merely seeking information – although recent dis-
cussions call for journalists to create their own
databases due to an overreliance on public data-
bases. Either way, the success of data journalism
also led to new practices, routines, and mixed
teams of journalists working together with pro-
grammers, developers, and designers within the
same newsrooms, allowing them to tell stories in a
different and visually engaging way.
4 Media
Media Organizations and Big Data
Big data is not only a valuable resource for data
journalism. Media organizations are data gath-
erers as well. Many media products, whether
news or entertainment, are financed through
advertising. In order to satisfy the advertisers’
interests in the site’s audience, penetration, and
visits, media organizations track user behavior on
their webpages. Very often, media organizations
share this data with external research bodies,
which then try to use the data on their behalf.
Gathering information about their customers is
therefore not only an issue when it comes to the
use of social media. Traditional media organiza-
tions are also collecting data about their clients.
However, media organizations track the user
behavior on news websites not only to provide
data to their advertisers. Through user data, they
also adapt the website’s content to the audience’s
demand, with dysfunctional consequences for
journalism and its democratic function within
society. Due to web analytics and the generation
of large-scale data collections, the audience exerts
an increasing influence over the news selection
process. This means that journalists – particularly
in the online realm – are at the risk of increasingly
adapting their news selections on the audience’s
feedback through data generated via web analyt-
ics. Due to the grim financial situation and their
shrinking advertising revenue, some print media
organizations especially in western societies try to
apply strategies to compensate these deficits
through a dominant market-driven discourse,
manufacturing cheaper content that appeals to
broader masses – publishing more soft news, sen-
sationalism, and articles of human interest without
any connection to public policy issues. This is also
due to the different competitive environment:
while there are fewer competitors in traditional
newspaper or broadcast markets, in the online
world, the next competitor is just one click away.
Legacy media organizations, particularly news-
papers and their online webpages, offer more soft
news to increase traffic, to attract the attention of
more readers, and thus to keep their advertisers at
it. A growing body of literature about the conse-
quences of this behavior shows that journalists, in
general, are becoming much more aware of the
audiences’ preferences. At the same time, however,
there is also a growing concern among journalist
with regard to their professional ethics and the
consequences for the function of journalism in
society if they base their editorial decision-making
processes on real-time data. The results of web
analytics not only influence the placement of
news on the websites; they also have an impact
on the journalists’ beliefs about what the audience
wants. Particularly in online journalism, the news
selection is carried out grounding the decisions on
data generated by web analytics and no longer
on intrinsic notions such as news values or
personal beliefs. Consequently, online journalism
becomes highly responsive to the audiences’
preferences – serving less what would be in the
public interest. As many news outlets are integrated
organizations, which means that they apply a
crossmedia strategy by joining previously sepa-
rated newsrooms such as the online and the print
staff, it might be possible that factors like data-
based audience feedback will also affect print
newsrooms. As Tandoc Jr. and Thomas state, if
journalism continues to view itself as a sort of
“conduit through which transient audience prefer-
ences are satisfied, then it is no journalism worth
bearing the name” (Tandoc and Thomas 2015, p.
253).
While news organizations still struggle with
self-gathered data due to the conflicts that can
arise in journalism, media organizations active in
the entertainment industry rely much more
strongly on data about their audiences. Through
large amounts of data, entertainment media can
collect significant information about the audi-
ence’s preferences for a TV series or a movie –
even before it is broadcast. Particularly for big
production companies or film studios it is essen-
tial to observe structured data like ratings, market
share, and box office stats. But also unstructured
data like comments or videos in social media are
equally important in order to understand con-
sumer habits, given that they provide information
about the potential success or failure of a (new)
product.
An example of such use of big data is the
launch of the TV show “House of Cards” by the
Internet-based on demand streaming provider
Media
Netflix. Before launching the first original content
with the political drama, Netflix was already
collecting huge amounts of data about the stream-
ing habits of their customers. Of more than 25 mil-
lion users, they tracked around 30 million views a
day (recording also when people are pausing,
rewinding, or fast-forwarding the videos), about
four million ratings, and three million searches
(Carr 2013). On top of that, they also try to gather
unstructured data from social media, and they
look how customers are tagging the selected
videos with metadata descriptors and whether
they recommend the content. Based on these
data, Netflix predicted possible preferences and
decided to buy “House of Cards.” It was a major
success for the online-based company.
There are also potential risks associated with
the collection of such huge amounts of data:
Netflix recommends specific movies or TV
shows to their customers based on what they
liked or what they have watched before. These
recommendation algorithms might well guide the
user toward more of their original content, without
taking into account the consumers’ actual prefer-
ences. In addition, consumers might not be able to
discover new TV shows that transcend their usual
taste. Given that services like Netflix know so
much about their users’ habits, another concern
with regard to privacy arises.
Big Data Between Social Media, Ethics, and
Surveillance
Social media are a main source for big data. Since
the first major social media webpages have been
launched in the 2000s, they began to collect and
store massive amounts of data. These sites started
to gather information about the behavior, prefer-
ences, and interests of their users in order to know
how their users would both think and act. In
general, this process of datafication is used to
target and tailor the services better to the users’
interests. At the same time, social media use these
large-scale data collections to help advertiser tar-
get the users. Big data in social media have there-
fore also a strong commercial connotation.
Facebook’s business model, for instance, is
entirely based on hyper-targeted display ads.
While display ads are a relatively old-fashioned
5
way of addressing customers, Facebook can make
it up with its incredible precision about the cus-
tomers’ interests and its ability to target advertis-
ing more effectively.
Big data are an integrative part of social
media’s business model: they possess far more
information on their customers given that they
have access not only to their surf behavior but
above all to their tastes, interests, and networks.
This might not only bear the potential to predict
the users’ behavior but also to influence it, partic-
ularly as social media such as Facebook and Twit-
ter adapt also their noncommercial content to the
individual users: the news streams we see on our
personal pages are balanced by various variables
(differing between social media) such as interac-
tions, posting habits, popularity, the number of
friends, user engagement, and others, being how-
ever constantly recombined. Through such
opaque algorithms, social media might well use
their own data to model voters: in 2010, for exam-
ple, 61 million users in the USA were shown a
banner message on their pages about how many of
their friends already voted for the US Congressio-
nal Elections. The study showed that the banner
convinced more than 340,000 additional people to
cast their vote (Bond et al. 2012). The individually
tailored and modeled messaging does not only
bear the potential to harm the civic discourse; it
also enhances the negative effects deriving from
“asymmetry and secrecy built into this mode of
computational politics” (Tufekci 2014).
The amount of data stored on social media will
continue to rise, and already today, social media
are among the largest data repositories in the
world. Since the data collecting mania of social
media will not decrease, which is also due to the
explorative focus of big data, it raises issues with
regard to the specific purpose of the data collec-
tion. Particularly if the data usage, storage, and
transfer remain opaque and are not made transpar-
ent, the data collection might be disproportionate.
Yet, certain social media allow third parties to
access their data, particularly as the trade of data
increases because of its economic potential. This
policy raises ethical issues with regard to trans-
parency about data protection and privacy.
6 Media

Particularly in the wake of the Snowden reve- was that the users in the sample were not aware
lations, it has been shown that opaque algorithms that their newsfeed was altered. This study shows
and big data practices are increasingly important that the use of big data generated in social media
to surveillance: “[...] Big Data practices are can entail ethical issues, not the least because the
skewing surveillance even more towards a reli- constructed reality within Facebook can be
ance on technological “solutions,” and that these distorted. Ethical questions with regard to media
both privileges organizations, large and small, and big data are thus highly relevant in our soci-
whether public or private, reinforce the shift in ety, given that both the privacy of citizens and the
emphasis toward control rather than discipline protection of their data are at stake.
and rely increasingly on predictive analytics to
anticipate and preempt” (Lyon 2014, p. 10). Over-
all, the Snowden disclosures have demonstrated
Conclusion
that surveillance is no longer limited to traditional
instruments in the Orwellian sense but have
Big data plays a crucial role in the context of the
become ubiquitous and overly reliant on practices
media. The instruments of computer-assisted
of big data – as governmental agencies such as the
reporting and data journalism allow news organi-
NSA and GCHQ are allowed to access not only
zations to engage in new forms of investigations
the data of social media and search giants but also
and storytelling. Big data also allow media orga-
to track and monitor telecommunications of
nizations to better adapt their services to the pref-
almost every individual in the world. However,
erences of their users. While in the news business
the big issue even with the collect-all approach is
this may lead to an increase of soft news, the
that data is subject to limitations and bias, partic-
entertainment industry benefits from such data in
ularly if they rely on automated data analysis:
order to predict the audience’s taste with regard to
“Without those biases and limitations being
potential TV shows or movies. One of the biggest
understood and outlined, misinterpretation is the
issues with regard to media and big data are its
result” (Boyd and Crawford 2012, p. 668). This
ethical implications, particularly with regard to
might well lead to false accusation or failure of
data collection, storage, transfer, and surveillance.
predictive surveillance as could be seen in the case
As long as the urge to collect large amounts of
of the Boston Marathon bombing case: first, a
data and the use of opaque algorithms continue to
picture of the wrong suspect was massively shared
prevail in many already powerful (social) media
on social media, and second, the predictive radar
organizations, the risks of data manipulation and
grounded on data gathering was ineffective.
modeling will increase, particularly as big data are
In addition, the use of big data generated by
becoming even more important in many different
social media entails also ethical issues in reference
aspects of our lives. Furthermore, as the Snowden
to scientific research. Normally, when human
revelations showed, collect-it-all surveillance
beings are involved in research, strict ethical
already relies heavily on big data practices. It is
rules, such as the informed consent of the people
therefore necessary to increase both the research
participating in the study, have to be observed.
into and the awareness about the ethical implica-
Moreover, in social media there are “public” and
tions of big data in the media context. Only thanks
“private,” which can be accessed. An example of
to a critical discourse about the use of big data in
such a controversial use of big data is a study
our society, we will be able to determine “our
carried out by Kramera et al. (2014). The authors
agency with respect to big data that is generated
deliberately changed the newsfeed of Facebook
by us and about us, but is increasingly being used
users: some got more happy news, others more
at us” (Tufekci 2014). Being more transparent,
sad ones, because the goal of the study was to
accountable, and less opaque about the use and,
investigate whether emotional shifts in those sur-
in particular, the purpose of data collection might
rounding us – in this case virtually – can change
be a good starting point.
our own moods as well. The issue with the study
Media
Cross-References
▶ Advertising Targeting
▶ Big Data Storytelling
▶ Crowdsourcing
▶ Transparency
References
Bond, R. M., Fariss, C. J., Jones, J. J., Kramer, A. D. I.,
Marlow, C., Settle, J. E., & Fowler, J. H. (2012).
A 61-million-person experiment in social influence
and political mobilization. Nature, 489, 295–298.
Boyd, D., & Crawford, K. (2012). Critical questions for big
data. Information, Communication & Society, 15(5),
662–679.
Carr, D. (2013, February 24). Giving readers what they
want. New York Times. http://www.nytimes.com/
7
2013/02/25/business/media/for-house-of-cards-using-
big-data-to-guarantee-its-popularity.html. Accessed
11 July 2016.
Kramera, A. D. I., Guilloryb, J. E., & Hancock, J. T.
(2014). Experimental evidence of massive-scale emo-
tional contagion through social networks. Proceedings
of the National Academy of Sciences of the United
States of America, 111(24), 8788–8790.
Lyon, D. (2014, July–December). Surveillance, Snowden,
and Big Data: Capacities, consequences, critique. Big
Data & Society, 1–13.
Tandoc Jr., E. C., & Thomas, R. J. (2015). The ethics of
web analytics. Implications of using audience metrics
in news construction. Digital Journalism, 3(2),
243–258.
Tufekci, Z. (2014). Engineering the public: Big data, sur-
veillance and computational politics. First Monday,
19(7). http://journals.uic.edu/ojs/index.php/fm/article/
view/4901/4097. Accessed 12 July 2016.
M
Metadata
Xiaogang Ma
Department of Computer Science, University of
Idaho, Moscow, ID, USA
Metadata are data about data, or in a more general
sense, they are data about resources. They provide
a snapshot about a resource, such as information
about the creator, date, subject, location, time and
methods used, etc. There are high-level metadata
standards that can provide a general description of
a resource. In recent years, community efforts
have been taken to develop domain-specific meta-
data schemas and encode the schemas with
machine readable formats for the World Wide
Web. Those schemas can be reused and extended
to fit requirements of specific applications. Com-
paring with the long-term archive of data and
metadata in traditional data management and anal-
ysis, the velocity of Big Data leads to short-term
and quick applications addressing scientific and
business issues. Accordingly, there is a metadata
data life cycle in Big Data applications. Commu-
nity metadata standards and machine readable
formats will be a big advantage to facilitate the
metadata data life cycle on the Web.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_135-1
Know Before Use
Few people are able to use a piece of data before
knowing its subject, origin, structure, and mean-
ing. A primary functionality of metadata is to help
people to obtain an overview of some data, and
this functionality can be understood through a few
real-world examples. If data are comparable with
goods in a grocery, then metadata are like the
information on the package of an item.
A consumer may care more about the ingredients
due to allergies to some substances, the nutrition
facts due to dietary needs, and/or the manufacturer
and date of expiration due to personal preferences.
Most people want to know the information about a
grocery item before purchasing and consuming
it. The information on the package provides a
concise and essential introduction about the item
inside. Such nutrition and ingredient information
of grocery items is mandatory for manufacturers
in many countries. Similarly, an ideal situation for
data users is that they can receive clear metadata
from data providers. However, compared to the
food industry, the rules and guidelines for meta-
data are still less developed.
Another comparable subject is the 5W1H
method for storytelling or context description,
especially in journalism. The 5W1H represents
the question words who, what, when, where,
why, and how, which can be used to organize a
number of questions about a certain object or
event, such as: Who is responsible for a research
project? What are the planned output data? Where
2 Metadata
will the data be archived? When will the data be
open access? Why a specific instrument is needed
for data collection? How will the data be
maintained and updated? In journalism, the
5W1H is often used to evaluate whether the infor-
mation covered in a news article is complete or
not. Normally, the first paragraph of a news article
gives a brief overview of the article and provides
concise information to answer the 5W1H ques-
tions. By reading the first paragraph, a reader can
grasp the key information of an article even before
reading through the full text. Metadata is data
about data; such functionality is similar to what
the first paragraph works for a news article, and
metadata items used for describing a dataset are
equal to the 5W1H question words.
Metadata Hierarchy
Metadata are used for describing resources. The
description can be general or detailed according to
the actual needs. Accordingly, there is a hierarchy
of metadata items corresponding to the actual
needs of describing an object. For instance, the
abovementioned 5W1H question words can be
regarded as a list of general metadata items, and
they can also be used to describe datasets. How-
ever, the six question words only offer a start
point, and there may be various derived metadata
items in actual works. In early days there was such
a heterogeneous situation among the metadata
provided by different stakeholders. To promote
standardization of metadata items, a number of
international standards have been developed.
The most well-known standard is the Dublin
Core Metadata Element Set (DCMI Usage Board
2012). The name “Dublin” originates from a 1995
workshop at Dublin, OH, USA. The word “Core”
means that the elements are generic and broad.
The 15 core elements are contributor, coverage,
creator, date, description, format, identifier, lan-
guage, publisher, relation, rights, source, subject,
title, and type. Those elements are more specific
than the 5W1H question words and can be used
for describing a wide range of resources, includ-
ing datasets. The Dublin Core Metadata Element
Set was published as a standard by the
International Organization for Standardization
(ISO) in 2003 and later revised in 2009. It has
also been endorsed by a number of other national
or international organizations such as the Ameri-
can National Standards Institute and the Internet
Engineering Task Force.
The 15 core elements are part of an enriched
specification of metadata terms maintained by the
Dublin Core Metadata Initiative (DCMI). The
specification includes properties in the core ele-
ments, properties in an enriched list of terms,
vocabulary encoding schemes, syntax encoding
schemes, and classes (including the DCMI Type
Vocabulary). The enriched terms include all the
15 core elements and cover a number of more
specific properties, such as abstract, access rights,
has part, has version, medium, modified, spatial,
temporal, valid, etc. In practice, the metadata
terms in the DCMI specification can be further
extended by combining with other compatible
vocabularies to support various application pro-
files. With the 15 core elements, one is able to
provide rich metadata for a certain resource, and
by using the enriched DCMI metadata terms and
external vocabularies, one can create an even
more specific metadata description for the same
object. This can be done in a few ways. For
example, one way is to use terms that are not
included in the core elements, such as spatial and
temporal. Another possible way is to use a refined
metadata term that is more appropriate for
describing an object. For instance, the term
“description” in the core elements is with broad
meaning, and it may include an abstract, a table of
contents, a graphical representation, or a free-text
account of a resource. In the enrich DCMI terms,
there is a more specific term “abstract,” which
means a summary of a resource. Compared to
“description,” the term “abstract” is more specific
and appropriate if one wants to collect a literal
summary of an academic article.
Domain-Specific Metadata Schemas
High-level metadata terms such as those in the
Dublin Core Metadata Element Set have broad
meaning and are applicable to various resources.
Metadata
However, those metadata elements are too general
in meaning and sometimes are implicit. If one
wants a more specific and detailed description of
the resources, a domain-specific metadata schema
is needed. Such a metadata schema is a list of
organized metadata items for describing a certain
type of resource. For example, there could be a
metadata schema for each type defined in the
DCMI Type Vocabulary, such as dataset, event,
image, physical object, service, etc. There have
been various national and international commu-
nity efforts for building domain-specific metadata
schemas. Especially, many schemas developed in
recent years face the data management and
exchange on the Web. A few recent works are
introduced below.
The data catalog vocabulary (DCAT)
(Erickson and Maali 2014) was approved as a
World Wide Web Consortium (W3C) recommen-
dation in January 2014. It was designed to facili-
tate interoperability among data catalogs
published on the Web. DCAT defines a metadata
schema and provides a number of examples on
how to use it. DCAT reuses a number of DCMI
metadata terms in combination with terms from
other schemas such as the W3C Simple Knowl-
edge Organization System (SKOS). It also defines
a few new terms to make the resulted schema
more appropriate for describing datasets in data
catalogs.
The Darwin Core is a group of standards for
biodiversity applications. By extending the Dub-
lin Core metadata elements, the Darwin Core
establishes a vocabulary of terms to facilitate the
description and exchange of data about the geo-
graphic occurrence of organisms and the physical
existence of biotic specimens. The Darwin Core
itself is also extensible, which provides a mecha-
nism for describing and sharing additional
information.
The ecological metadata language (EML) is a
metadata standard developed for the none-
geospatial datasets in the field of ecology. It is a
set of schemas encoded in the format of extensible
markup language (XML) and thus allows struc-
tured expression of metadata. EML can be used to
describe digital resources and also nondigital
resources such as paper maps.
3
The international geo sample number (IGSN),
initiated in 2004, is a sample identification code
for the geoscience community. Each registered
IGSN identifier is accompanied with a group of
metadata providing detailed background informa-
tion about that sample. Top concepts in the current
IGSN metadata schema are sample number, reg-
istrant, related resource identifiers, and log. A top
concept may include a few child concepts. For
example, there are two child concepts for “regis-
trant”: registrant name and name identifier.
The ISO 19115 and ISO 19115-2 geographic
information metadata are regarded as a best prac-
tice of metadata schemas for geospatial data.
Geospatial data are about objects with some posi-
tion on the surface of the Earth. The ISO 19115
standards provide guidelines on how to describe
geographical information and services. Detailed
metadata items cover topics about contents, spa-
tiotemporal extents, data quality, channels for
access and rights to use, etc. Another standard,
ISO 19139, provides an XML schema implemen-
tation for the ISO 19115. The catalog service for
the Web (CSW) is an open geospatial consortium
(OGC) standard for describing online geospatial
data and services. It adopts ISO 19139, the Dublin
Core elements and items from other metadata
efforts. Core elements in CSW include title, for-
mat, type, bounding box, coordinate reference
system, and association.
Annotating a Web of Data
Recent efforts on metadata standards and
schemas, such as the abovementioned Dublin
Core, DCAT, Darwin Core, EML, IGSN meta-
data, ISO 19139, and CSW, show a trend of pub-
lishing metadata on the Web. More importantly,
by using standard encoding formats, such as the
XML and W3C resource description framework
(RDF), they are making metadata machine dis-
coverable and readable. This mechanism moves
the burden of searching, evaluating, and integrat-
ing massive datasets from humans to computers,
and for computers such burden is not real burden
because they can find ways to access various data
sources through standardized metadata on the
4 Metadata
Web. For example, the project OneGeology aims
to enable online access to geological maps across
the world. By the end of 2014, the OneGeology
has 119 participating nations, and most of them
share national or regional geological maps
through OGC geospatial data service standards.
Those map services are maintained by their
corresponding organizations, and they also enable
standardized metadata services, such as CSW. On
the one hand, OneGeology provides technical
supports to organizations who want to set up
geologic map services using common standards.
On the other hand, it also provides a central data
portal for end users to access various distributed
metadata and data services. The OneGeology pro-
ject presents a successful example on how to
rescue the legacy data, update them with well-
organized metadata, and make them discoverable,
accessible, and usable on the Web.
Comparing with domain-specific structured
datasets, such as those in OneGeology, many
other datasets in the Big Data are not structured,
such as webpages and data stream on social
media. In 2011, the search engines Bing, Google,
Yahoo!, and Yandex launched an initiative called
schema.org, which aims at creating and
supporting a common set of schemas for struc-
tured data markup on web pages. The schemas are
presented as lists of tags in hypertext markup
language (HTML). Webmasters can use those
tags to mark up their web pages, and search engine
spiders and other parsers can recognize those tags
and record what a web page is about. This makes
it easier for search engine users to find the right
web pages. Schema.org adopts a hierarchy to
organize the schemas and vocabularies of terms.
The concept on the top is thing, which is very
generic and is divided into schemas of a number
of child concepts, including creative work, event,
intangible, medical entity, organization, person,
place, product, and review. These schemas are
further divided into smaller schemas with specific
properties. A child concept inherits characteristics
from a parent concept. For example, book is a
child concept of creative work. The hierarchy of
concepts and properties does not intend to be a
comprehensive model that covers everything in
the world. The current version of schema.org only
represents those entities that the search engines
can handle in a short term. Schema.org provides a
mechanism for extending the scope of concepts,
properties, and schemas. Webmasters and devel-
opers can define their own specific concepts,
properties, and schemas. Once those extensions
are commonly used on the Web, they can also be
included as a part of the schema.org schemas.
Linking for Tracking
If the recognition of domain-specific topics is a
work to identify resource types, then the definition
of metadata items is a work of annotating those
types. The work in schema.org is an excellent
reflection of those two works. Various structured
and unstructured resources can be categorized and
annotated by using metadata and are ready to be
discovered and accessed. In a scientific or busi-
ness procedure, various resources are retrieved
and used, and outputs are generated and archived
and perhaps be reused elsewhere. In recent years,
people take a further step to make links among
those resources, their types, and properties, as
well as the people and activities involved in the
generation of those outputs. The work of catego-
rization, annotation, and linking as a whole can be
used to describe the origin of a resource, which is
called provenance. There have been community
efforts developing specifications of commonly
usable provenance models.
The Open Provenance Model was initiated in
2006. It includes three top classes: artifact, pro-
cess, and agent and their subclasses, as well as a
group of properties, such as was generated by, was
controlled by, was derived from, and used, for
describing the classes and the interrelationships
among them. Another earlier effort is the proof
markup language, which was used to represent
knowledge about how information on the Web
was asserted or inferred from other information
sources by intelligent agents. Information, infer-
ence step/inference rule, and inference engine are
the three key building blocks in the proof markup
language.
Works on the Open Provenance Model and the
proof markup language have set up the basis for
Metadata
community actions. Most recently, the W3C
approved the PROV Data Model as a recommen-
dation in 2013. The PROV Data Model is a
generic model for provenance, which allows spe-
cific representations of provenance in research
domains or applications to be translated into the
model and be interchangeable among systems
(Moreau and Missier 2013). There are intelligent
knowledge systems that can import the prove-
nance information from multiple sources, process
it, and reason over it to generate clues for potential
new findings. The PROV Data Model includes
three core classes, entity, activity, and agent,
which are comparable to the Open Provenance
Model and the proof markup language. W3C
also approved the PROV Ontology as a recom-
mendation for the expression of the PROV Data
Model with semantic Web languages. It can be
used to represent machine readable provenance
information and can also be specialized to create
new classes and properties to represent prove-
nance information of specific applications and
domain. The extension and specification here are
similar to the idea of a metadata hierarchy.
A typical application of the PROV Ontology is
the Global Change Information System for the US
Global Change Research Program (Ma et al.
2014), which captures and presents provenance
of global change research, and links to the publi-
cations, datasets, instruments, models, algo-
rithms, and workflows that support key research
findings. The provenance information in the sys-
tem increases understanding, credibility, and trust
in the works of the US Global Change Research
Program and aids in fostering reproducibility of
results and conclusions.
A Metadata Life Cycle
Velocity is a unique feature that differentiates Big
Data from traditional data. Both traditional data
and Big Data traditional data can also be big, but
they have a relatively longer life cycle compared
to social media data stream in Big Data. Big Data
life cycles are featured by short-term and quick
deployments to solve specific scientific or
5
business issues. In traditional data management,
especially for a single data center or data reposi-
tory, the metadata life cycle is less addressed.
Now, facing the short-lived and quick Big Data
life cycles, attention should also be paid to the
metadata life cycle.
In general, a data life cycle covers steps of
context recognition, data discovery, data access,
data management, data archive, and data distribu-
tion. Correspondingly, a metadata life cycle
covers similar steps but they focus on the descrip-
tion of data rather than the data themselves. The
context recognition allows people to study a spe-
cific domain or application and reuse any existing
metadata standards and schemas. Then in the
metadata discovery step, it is possible to develop
applications to automatically harvest machine
readable metadata from multiple sources and har-
monize them. Commonly used domain-specific
metadata standards and machine readable formats
will significantly facilitate the metadata life cycle
in applications using Big Data, because most of
such applications will be on the Web and inter-
changeable schemas and formats will be an
advantage.
Cross-References
▶ Data Model, Data Modeling
▶ Data Profiling
▶ Data Provenance
▶ Data Sharing
▶ Open Data
▶ Semantic Web
Further Readings
DCMI Usage Board. (2012). DCMI metadata terms. http://
dublincore.org/documents/dcmi-terms
Erickson, J., Maali, F. (2014). Data catalog vocabulary
(DCAT). http://www.w3.org/TR/vocab-dcat
Ma, X., Fox, P., Tilmes, C., Jacobs, K., & Waple,
A. (2014). Capturing provenance of global change
information. Nature Climate Change, 4/6, 409–413.
Moreau, L., Missier, P.. (2013). PROV-DM: The PROV
data model. http://www.w3.org/TR/prov-dm
M
Mobile Analytics
Ryan Eanes
Department of Business Management,
Washington College, Chestertown, MD, USA
Analytics, broadly defined, refers to a series of
quantitative measures that allow marketers, ven-
dors, business owners, advertisers, and interested
parties the ability to gauge consumer engagement
and interaction with a property. When properly
deployed and astutely analyzed, analytics can
help to inform a range of business decisions
related to user experience, advertising, budgets,
marketing, product development, and more.
Mobile analytics, then, refers to the measurement
of consumer engagement with a brand, property,
or product via a mobile platform, such as a
smartphone or tablet computer.
Despite the fact that the mobile Internet and
app markets have exploded in growth over the
past decade, and despite the fact that more than
half of all American adults now own at least one
smartphone, according to the Pew Research Cen-
ter, marketers have been relatively slow to jump
into mobile marketing. In fact, American adults
spend at least 20% of their time online via mobile
devices; the advertising industry has been playing
“catch-up” over the past few years in an attempt to
chase this market. Even so, analyst Mary Meeker
notes that advertising budgets still devote only
about a tenth of their expenditures to mobile –
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_138-1
though this is a fourfold increase from just a few
years ago.
Any entity that is considering the deployment
of a mobile strategy must understand consumer
behavior as it occurs via mobile devices. Web
usability experts have known for years that online
browsing behavior can be casual, with people
quickly clicking from one site to another and
making judgments about content encountered in
mere seconds. Mobile users, on the other hand, are
far more deliberate in their efforts – generally
speaking, a mobile user has a specific task in
mind when he or she pulls out his phone. Brows-
ing is far less likely to occur in a mobile context.
This is due to a number of factors, including
screen size, connection speed, and the environ-
mental context in which mobile activity takes
place – the middle of the grocery store dairy
case, for example, is not the ideal place for one
to contemplate the purchase of an eight-person
spa for the backyard.
The appropriate route to the consumer must be
considered, as well. This can be a daunting pros-
pect, particularly for small businesses, businesses
with limited IT resources, or businesses with little
previous web or tech experience. If a complete
end-user experience is desired, there are two pri-
mary strategies that a company can employ: an
all-in-one web-based solution, or a stand-
alone app.
All-in-one web-based solutions allow the same
HTML5/CSS3-based site to appear elegant and
functional in a full-fledged computer-based
2 Mobile Analytics
browser while simultaneously “degrading” on a
mobile device in such a way that no functionality
is lost. In other words, the same underlying code
provides the user experience regardless of what
technological platform one uses to visit a site.
There are several advantages to this approach,
including singularity of platform (that is, no
need to duplicate properties, logos, databases,
etc.), ease of update, unified user experience, and
relative ease of deployment. However, there are
downsides: full implementation of HTML5 and
CSS3 are relatively new. As a result, it can be
costly to find a developer who is sufficiently
knowledgeable to make the solution as seamless
as desired, and who can articulate the solution in
such a way that non-developers will understand
the full vision of the end product. Furthermore,
development of a polished finished product can be
time-consuming and will likely involve a great
deal of compromise from a design perspective.
Mobile analytics tools are relatively easy to
deploy when a marketer chooses to take this
route, as most modern smartphone web browsers
are built on the same technologies that drive
computer-based web browsers – in other words,
most mobile browsers support both JavaScript
and web “cookies,” both of which are typically
requisites for analytics tools. Web pages can be
“tagged” in such a way that mobile analytics can
be measured, which will allow for the collection
of a variety of information on visitors. This might
include device type, browser identification, oper-
ating system, GPS location, screen resolution/
size, and screen orientation, all of which can pro-
vide clues as to the contexts in which users are
visiting the website on a mobile device. Some
mainstream web analytics tools, such as Google
Analytics, already include a certain degree of
information pertaining to mobile users (i.e., it is
possible to drill down into reports and determine
how many mobile users have visited and what
types of devices they were using); however, mar-
keting entities that want a greater degree of insight
into the success of their mobile sites will likely
need to seek out a third-party solution to monitor
performance.
There are a number of providers of web-based
analytics solutions that cover mobile web use.
These include, but are not limited to, ClickTale,
which offers mobile website optimization tools;
comScore, which is known for its audience mea-
surement metrics; Flurry, which focuses on use
and engagement metrics; Google, which offers
both free and enterprise-level services; IBM,
which offers the ability to record user sessions
and perform deep analysis on customer actions;
Localytics, which offers real-time user tracking
and messaging options; Medio, which touts “pre-
dictive” solutions that allow for custom content
creation; and Webtrends, which incorporates
other third-party (e.g., social media) data.
The other primary mobile option: development
of a stand-alone smartphone or tablet app. Stand-
alone apps are undeniably popular, given that
50 billion apps were downloaded from the Apple
App Store between July 2008 and June 2014.
A number of retailers have had great success
with their apps, including Amazon, Target,
Zappos, Groupon, and Walgreens, which speaks
to the potential power of the app as a marketing
tool. However, consider that there are more than
one million apps in the Apple App Store alone, as
of this writing – those odds greatly reduce the
chances that an individual will simply “stumble
across” a company’s app in the absence of some
sort of viral advertising, breakout product, or
buzzworthy word-of-mouth. Furthermore, devel-
oping a successful and enduring app can be quite
expensive, particularly considering that a mar-
keter will likely want to make versions of the
app available for both Apple iOS and Google
Android (the two platforms are incompatible
with each other). Estimates for app development
vary widely, from a few thousand dollars at the
low end all the way up to six figures for a complex
app, according to Mark Stetler of AppMuse – and
these figures do not include ongoing updates, bug
fixes, or recurring content updates, all of which
require staff with specialized training and know-
how.
If a full-fledged app or redesigned website
proves too daunting or beyond the scope of what
a marketer needs or desires, there are a number of
other techniques that can be used to reach con-
sumers, including text and multimedia messaging,
email messaging, mobile advertising, and so forth.
Mobile Analytics
Each of these techniques can reveal a wealth of
data about consumers, so long as the appropriate
analytic tools are deployed in advance of the
launch of any particular campaign.
Mobile app analytics are quite different from
web analytics in a number of ways, including the
vocabulary. For example, there are no page views
in the world of app analytics – instead, “screen
views” are referenced. Likewise, an app “session”
is analogous to a web “visit.” App analytics often
have the ability to access and gauge the use of
various features built into a phone or tablet,
including the accelerometer, GPS, and gyroscope,
which can provide interesting kinesthetic aspects
to user experience considerations. App analytics
tools are also typically able to record and retain
data related to offline usage for transmission when
a device has reconnected to the network, which
can provide a breadth of environmentally contex-
tual information to developers and marketers
alike. Finally, multiple versions of a mobile
app can exist “in the wild” simultaneously
because users’ proclivities differ when it comes
to updating apps. Most app analytic packages
have the ability to determine which version of an
app is in use so that a development team can track
interactional differences between versions and
confirm that bugs have been “squashed.”
As mentioned previously, marketers who
choose to forego app development and develop a
mobile version of their web page often choose to
stick with their existing web analytics provider,
and oftentimes these providers do not provide a
level of detail regarding mobile engagement that
would prove particularly useful to marketers who
want to capture a snapshot of mobile users. In
many cases, companies simply have not given
adequate consideration to mobile engagement,
despite the fact that it is a growing segment of
online interaction that is only going to grow, par-
ticularly as smartphone saturation continues.
However, for those entities that wish to delve
further into mobile analytics, there are a growing
number of options available, with a few key dif-
ferences between the major offerings. There are
both free and paid mobile analytics platforms
available; the key differentiator between these
offerings seems to come down to data ownership.
3
A third-party provider that shares the data with
you, like Google, is more likely to come at a
bargain price, whereas a provider that grants you
exclusive ownership of the data is going to come
at a premium. Finally, implementation will make a
difference in costs: SaaS (software-as-a-service)
solutions, which are typically web based, run on
the third-party service’s own servers, and rela-
tively easy to install, tend to be less expensive,
whereas “on-premises” solutions are both rare and
quite expensive.
There are a small but growing number of com-
panies that provide app-specific analytic tools,
typically deployed as SDKs (software develop-
ment kits) that can be “hooked” into apps. These
companies include, but are by no means limited
to, Adobe Analytics, which has been noted for its
scalability and depth of analysis; Artisan Mobile,
an iOS-focused analytics firm that allows cus-
tomers to conduct experiments with live users in
real time; Bango, which focuses on ad-based
monetization of apps; Capptain, which allows
specific user segments to be identified and
targeted with marketing campaigns; Crittercism,
which is positioned as a transaction-monitoring
service; Distimo, which aggregates data from a
variety of platforms and app stores to create a
fuller position of an app in the larger marketplace;
ForeSee, which has the ability to record customer
interactions with apps; and Kontagent, which
touts itself as a tool for maintaining customer
retention and loyalty.
As mobile devices and the mobile web grow
increasingly sophisticated, there is no doubt that
mobile analytics tools will also grow in sophisti-
cation. Nevertheless, it would seem that there are
a wide range of promising toolkits already avail-
able to the marketer who is interested in better
understanding customer behaviors and increasing
customer retention, loyalty, and satisfaction.
Cross-References
▶ Data Aggregation
▶ Location Data
▶ Network Data
▶ Telecommunications
4 Mobile Analytics

Further Readings smartphone-ownership-2013/. Accessed September

Meeker, M. Internet trends 2014. http://www.kpcb.com/
insights/2014-internet-trends. Accessed September
2014.
Smith, A. Smartphone ownership 2013. Pew Research
Center. http://www.pewinternet.org/2013/06/05/
2014.
Stetler, M. How much does it cost to develop a mobile app?
AppMuse. http://appmuse.com/appmusing/how-much-
does-it-cost-to-develop-a-mobile-app/. Accessed Sep-
tember 2014.
N
National Association for the
Advancement of Colored People
Steven Campbell
University of South Carolina, Lancaster,
Lancaster, SC, USA
The National Association for the Advancement of
Colored People (NAACP) is an African-
American civil rights organization headquartered
in Baltimore, MD. Founded in 1909, its member-
ship advocates civil rights by engaging in activi-
ties such as mobilizing voters and tracking equal
opportunity in government, industry, and commu-
nities. Over the past few years, the NAACP has
shifted its attention to digital advocacy and the
utilization of datasets to better mobilize activists
online. In the process, the NAACP has become a
leading organization in how it harnesses big data
for digital advocacy and related campaigns. The
NAACP’s application of specially tailored data to
its digital approach, from rapid response to
targeted messaging to understanding recipients’
interests, has become an example for other groups
to follow. At the same time, the NAACP has
challenged other big data (both in the public and
private sectors), highlighting abuse of such data in
ways that can directly impact disadvantaged
minority groups.
With a membership of over 425,000 members,
the NAACP is the nation’s largest civil rights
organization. Administered by a 64-member
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_139-1
board headed by a chairperson, various depart-
ments within the NAACP govern particular areas
of action. The Legal Department tracks court
cases with potentially extensive implications for
minorities, including recurring discrimination in
areas such as education and employment. The
Washington, D.C., office lobbies Congress and
the Presidency on a wide range of policies and
issues, while the Education Department seeks
improvements in the sphere of public education.
Overall, the NAACP’s mission is to bolster equal
rights for all people in political, educational, and
economic terms as well as stamp out racial biases
and discrimination.
In order to extend this mission into the twenty-
first century, the NAACP launched a digital media
department in 2011. This entailed a mobile sub-
scriber project that led to 423,000 contacts,
233,000 Facebook supporters, and 1.3 million
email subscribers, due in large part to greater
social media outreach. The NAACP’s “This is
my Vote!” campaign, launched prior to the 2012
presidential election, dramatically advanced the
organization’s voter registration and mobilization
programs. As a result, the NAACP registered
twice the number of individuals – over
374,000 – than it did in 2008 and mobilized over
1.2 million voters. In addition, the NAACP
conducted an election eve poll that surveyed
1,600 African-American voters. This was done
in order to assess their potential influence as well
as key issue areas prior to the election results and
in looking forward to 2016. Data from the poll
2 National Association for the Advancement of Colored People
highlighted the predominate role played by
African-Americans in major battleground states
and divulged openings for the Republican Party
in building rapport with the African-American
community. In addition, the data signaled to Dem-
ocrats a message not to assume levels of Black
support in 2016 on par with that realized in the
2008 and 2012 elections.
By tailoring its outreach to individuals, the
NAACP has been successful in achieving rela-
tively high rates of engagement. The organization
segments supporters based on their actions, such
as whether they support a particular issue based on
past involvement. For instance, many NAACP
members view gun violence as a serious problem
in today’s society. If such a member connects with
NAACP’s online community via a particular
webpage or internet advertisement, s/he will be
recognized as one espousing stronger gun control
laws. Future outreach will entail tailored mes-
sages expressing attributes that resonate on a per-
sonal level with the supporter, not unlike that from
a friend or colleague.
The NAACP also takes advantage of major
events that reflect aspects of the organization’s
mission statement. Preparation for such moments
entails much advance work, as evidenced in the
George Zimmerman trial involving the fatal
shooting of 17-year-old Trayvon Martin. As the
trial was concluding in 2013, the NAACP formed
contingency plans in advance of the court’s deci-
sion. Website landing pages and prewritten emails
were set in place, adapted for whatever result may
come. Once the verdict was read, the NAACP sent
out emails within 5 min that detailed specific
actions for supporters to take. This resulted in
over a million petition signatures demanding
action on the part of the US Justice Department,
which it eventually took.
Controversy
While government and commercial surveillance
potentially affect all Americans, minorities face
these risks at disproportionate rates. Thus, the
NAACP has raised concerns about whether big
data needs to provide greater protections for
minorities in addition to the general privacy pro-
tections commonly granted. Such controversy
surrounding civil rights and big data may not be
self-evident; however, big data often involves the
targeting and segmenting of one type of individual
from another. This serves as a threat to basic civil
rights –which are protected by law – in ways that
were inconceivable in recent decades. For
instance, the NAACP has expressed alarm regard-
ing the collection of information by credit
reporting agencies. Such collections can result in
the making of demographic profiles and stereo-
typical categories, leading to the marketing of
predatory financial instruments to minority
groups.
The US government’s collection of massive
phone records for purposes of intelligence has
also drawn harsh criticism from the NAACP as
well as other civil rights organizations. They have
vented warnings regarding such big data by
highlighting how abuses can uniquely affect dis-
advantaged minorities. The NAACP supports
principles aimed at curtailing the pervasive use
of data in areas such as law enforcement and
employment. Increasing collections of data are
viewed by the NAACP as a threat since such big
data could allow for unjust targeting of, and dis-
crimination against, African-Americans. Thus,
the NAACP strongly advocates measures such
as a stop to “high-tech profiling,” greater pressure
on private industry for more open and transparent
data, and greater protections for individuals from
inaccurate data.
Cross-References
▶ Demographic Data
▶ Discrimination
▶ Facebook
▶ Pattern Recognition
▶ Targeting
Further Reading
Fung, Brian (27 Feb 2014). Why civil rights groups are
warning against ‘big data’. Washington Post. http://
National Association for the Advancement of Colored People 3

www.washingtonpost.com/blogs/the-switch/wp/2014/ ahead of the curve, making smarter data decisions.

02/27/why-civil-rights-groups-are-warning-against- Advertising Age. http://adage.com/article/datadriven-
big-data/. Accessed Sept 2014. marketing/brands-learn-data-advocacy-groups/
Murray, Ben (3 Dec 2013). What brands can learn about 245498/. Accessed Sept 2014).
data from the NAACP: Some advocacy groups are NAACP. http://www.NAACP.org. Accessed Sept 2014.
N
National Oceanic and Atmospheric
Administration
Steven J. Campbell
University of South Carolina Lancaster,
Lancaster, SC, USA
The National Oceanic and Atmospheric Adminis-
tration (NOAA) is an agency housed within the
US Commerce Department that monitors the sta-
tus and conditions of the oceans and the atmo-
sphere. NOAA oversees a diverse array of
satellites, buoys, ships, aircraft, tide gauges, and
supercomputers in order to closely track environ-
mental changes and conditions. This network
yields valuable and critical data that is crucial for
alerting the public to potential harm and pro-
tecting the environment nationwide. The vast
sums of data collected daily have served as a
challenge to NOAA in storing as well as making
the information readily accessible and meaningful
to the public and interested organizations. In the
future, as demand grows for ever-greater amounts
and types of climate data, NOAA must be
resourceful in meeting the demands of public
officials and other interested parties.
First proposed by President Richard Nixon,
who wanted a new department in order to better
protect citizens and their property from natural
dangers, NOAA was founded in October 1970.
Its mission is to comprehend and foresee varia-
tions in the environment, from the conditions of
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_141-1
the oceans to the state of the sun, and to better
safeguard and preserve seashores and marine life.
NOAA provides alerts to dangerous weather,
maps the oceans and atmosphere, and directs the
responsible handling and safeguarding of the seas
and coastal assets. One key way NOAA pursues
its mission is by conducting research in order to
further awareness and better management of envi-
ronmental resources. With a workforce of over
12,000, NOAA consists of six major line offices,
including the National Weather Service (NWS), in
addition to over a dozen staff offices.
NOAA’s collection and dissemination of vast
sums of data on the climate and environment
contribute to a multibillion-dollar weather enter-
prise in the private sector. The agency has sought
ways to release extensive new troves of this data,
an effort that could be of great service to industry
and those engaged in research. NOAA announced
a call in early 2014 for ideas from the private
sector to assist the agency’s efforts in freeing up
a large amount of the 20 terabytes of data that it
collects on a daily basis pertaining to the environ-
ment and climate change. In exchange,
researchers stand to gain critical access to impor-
tant information about the planet, and private
companies can receive help and assistance in
advancing new climate tools and assessments.
This request by NOAA shows that it is plan-
ning to place large amounts of its data into the
cloud, benefitting both the private and public sec-
tors in a number of ways. For instance, climate
data collected by NOAA is currently employed
2 National Oceanic and Atmospheric Administration
for forecasting the weather over a week in
advance. In addition, marine navigation and off-
shore oil and gas drilling operations are very
interested in related data. NOAA has pursued
unleashing ever-greater amounts of its ocean and
atmospheric data by partnering with groups out-
side government. This is seen as paramount to
NOAA’s data management, where tens of
petabytes of information are recorded in various
ways, engendering over 15 million results daily –
from weather forecasts for US cities to coastal tide
monitoring – which totals twice the amount of all
the printed collections of the US Library of
Congress.
Maneuvering through NOAA’s mountain of
weather and climate, the data has proved to be a
great challenge over the years. To help address
this issue, NOAA made available, in late 2013, an
instrument that helped further open up the data to
the public. With a few clicks of a mouse, individ-
uals can create interactive maps illustrating natu-
ral and manmade changes in the environment
worldwide. For the most part, the data is free to
the public, but much of the information has not
always been organized in a user-friendly format.
NOAA’s objective was to bypass that issue and
allow public exploration of environmental condi-
tions from hurricane occurrences to coastal tides
to cloud formations. The new instrument, named
NOAA View, allows ready access to many of
NOAA’s databases, including simulations of
future climate models. These datasets grant users
the ability to browse various maps and informa-
tion by subject and time frame. Behind the scenes,
numerous computer programs manipulate
datasets into maps that can demonstrate environ-
mental attributes and climate change over time.
NOAA View’s origins were rooted in data visual-
ization instruments present on the web, and it is
operational on tablets and smartphones that
account for 44% of all hours spent online by the
US public.
Advances to NOAA’s National Weather Ser-
vice supercomputers have allowed for much faster
calculations of complex computer models,
resulting in more accurate weather forecasts. The
ability of these enhanced supercomputers to
analyze mounds of scientific data proves vital in
helping public officials, communities, and indus-
trial groups to better comprehend and prepare for
perils linked with turbulent weather and climatic
occurrences. Located in Virginia, the supercom-
puters operate with 213 teraflops (TF) – up from
the 90 TF with the computers that came before
them. This has helped to produce an advanced
Hurricane Weather Research and Forecasting
(HWRF) model that the National Weather Service
can more effectively employ. By allowing more
effective monitoring of violent storms and more
accurate predictions regarding the time, place, and
intensity of their impact, the HWRF model can
result in saved lives.
NOAA’s efforts to build a Weather-Ready
Nation have evolved from a foundation of super-
computer advancements that have permitted more
accurate storm-tracking algorithms for weather
prediction. First launched in 2011, this initiative
on the part of NOAA has resulted in advanced
services, particularly in ways that data and infor-
mation can be made available to the public, gov-
ernment agencies, and private industry.
Cross-References
▶ Climate Change, Hurricanes/Typhoons
▶ Cloud or Cloud Computing
▶ Data Storage
▶ Environment
▶ Predictive Analytics
Further Readings
Freedman, A. (2014, February 24). U.S. readies big-data
dump on climate and weather. http://mashable.com/
2014/02/24/NOAA-data-cloud/. Accessed September
2014.
Kahn, B. (2013). NOAA’s new cool tool puts climate on
view for all. http://www.climatecentral.org/news/
noaas-new-cool-tool-puts-climate-on-view-for-all-
16703. Accessed September 2014.
National Oceanic and Atmospheric Administration
(NOAA). www.noaa.gov. Accessed September 2014.
N
National Organization for Women
Deborah Elizabeth Cohen
Smithsonian Center for Learning and Digital
Access, Washington, DC, USA
The National Organization for Women (NOW) is
an American feminist organization that is the
grassroots arm of the women’s movement and
the largest organization of feminist activists in
the United States. Since its founding in 1966,
NOW has engaged in activity to bring about
equality for all women. NOW has been participat-
ing in recent dialogues to identify how common
big data working methods lead to discriminatory
practices against protected classes including
women. This entry discusses NOW’s mission
and issues related to big data and the activities
NOW has been involved with to end discrimina-
tory practices resulting from the usage of big data.
As written in its original statement of purpose,
the purpose of NOW is to take action to bring
women into full participation in the mainstream
of American society, exercising privileges and
responsibilities in completely equal partnership
with men. NOW strives to make change through
a number of activities including lobbying, rallies,
marches, and conferences. NOW’s six core issues
are economic justice, promoting diversity and
ending racism, lesbian rights, ending violence
against women, constitutional equality, and
access to abortion and reproductive health.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_142-1
NOW’s current president Terry O’Neill has
stated that big data practices can render obsolete
the USA’s landmark civil rights and anti-
discrimination laws with special challenges for
women, the poor, people of color, trans-people,
and the LGBT community. While the technolo-
gies of automated decision-making are hidden and
largely not understood by average people, they are
being conducted with an increasing level of per-
vasiveness and used in contexts that affect indi-
viduals’ access to health, education, employment,
credit, and products. Problems with big data prac-
tices include the following:
• Big data technology is increasingly being used
to assign people to ideologically or culturally
segregated clusters, profiling them and in
doing so leaving room for discrimination.
• Through the practice of data fusion, big data
tools can reveal intimate personal details, erod-
ing personal privacy.
• As people are often unaware of this “scoring”
activity, it can be hard for individuals to break
out of being mislabeled.
• Employment decisions made through data
mining have the potential to be discriminatory.
• Metadata collection renders legal protection of
civil rights and liberties less enforceable, undo-
ing civil rights law.
Comprehensive US civil rights legislation in
the 1960s and 1970s resulted from social actions
2 National Organization for Women
organized to combat discrimination. A number of
current big data practices are in misalignment with
these laws and can lead to discriminatory
outcomes.
NOW has been involved with several impor-
tant actions in response to these recognized prob-
lems with big data. In January of 2014, the US
White House engaged in a 90-day review of big
data and privacy issues, to which NOW as a par-
ticipating stakeholder provided input. Numerous
policy recommendations resulted from this pro-
cess especially related to data privacy and the
need for the federal government to develop tech-
nical expertise to stop discrimination.
The NOW Foundation also belongs to a coali-
tion of 200 progressive organizations named the
Leadership Conference on Civil and Human
Rights whose mission is to promote the civil and
human right of all persons in the United States.
NOW President Terry O’Neill serves on the Coa-
lition’s Board of Directors. In February 2014, The
Leadership Conference released five “Civil Rights
Principles for the Era of Big Data” and in August
2014 provided testimony based on their work to
the US National Telecommunications and Infor-
mation Administration’s Request for Public Com-
ment related to Big Data and Consumer Privacy.
The five civil rights principles to ensure that big
data is designed and used in ways that respect the
values of equal opportunity and equal justice
include the following:
1. Stop high tech profiling – ensure that clear
limits and audit mechanisms are in place to
make sure that data gathering and surveillance
tools that can assemble detailed information
about a person or group are used in a respon-
sible and fair way.
2. Ensure fairness in automated decisions –
require through independent review and other
measures that computerized decision-making
systems in areas such as employment, health,
education, and lending operate fairly for all
people and protect the interests of those that
are disadvantaged and have historically been
discriminated against. Systems that are blind to
preexisting disparities can easily reach deci-
sions that reinforce existing inequities.
3. Preserve constitutional principles – govern-
ment databases must not be allowed to under-
mine core legal protections, including those of
privacy and freedom of association. Indepen-
dent oversight of law enforcement is particu-
larly important for minorities who often
receive disproportionate scrutiny.
4. Enhance individual control of personal
information – individuals, and in particular
those in vulnerable populations including
women and the LGBT community, should
have meaningful and flexible control over
how a corporation gathers data from them and
how it uses and shares that data. Nonpublic
information should not be shared with the gov-
ernment without judicial process.
5. Protect people from inaccurate data – Govern-
ment and corporate databases must allow
everyone to appropriately ensure the accuracy
of personal information used to make impor-
tant decisions about them. This requires disclo-
sure of the data and the right to correct it when
inaccurate.
Big data has been called the civil rights battle
of our time. Consistent with its mission, NOW is
engaged in this battle, protecting civil rights of
women and others against discriminatory prac-
tices that can result from current big data
practices.
Cross-References
▶ Data Fusion
▶ Data Mining
▶ Discrimination
▶ National Telecommunication and Information
Administration
▶ White House Big Data Initiative
Further Readings
Big data: Seizing opportunities, preserving values. (2014).
Washington, DC: The White House. www.whitehouse-
gov/sites/default/files/docs/big-data-privacy-report-5.1.1.
14-final-print.pdf. Accessed 7 Sep 2014.
National Organization for Women
Eubanks, V. (2014). How big data could undo our civil-
rights laws. The American Prospect. www.prospect.
org/article/how-big-data-could-undo-our-civil-rights-
laws. Accessed 7 Sep 2014.
Gangadharan, S. P. (2014). The dangers of high-tech profil-
ing, using big data. The New York Times. www.nytimes.
com/roomfordebate/204/08/06/Is-big-data-spreading-
inequality/the-dangers-of-high-tech-profiling-using-
big-data. Accessed 5 Sep 2014.
3
NOW website. (2014). Who we are. National Organization
for Women. http://now.org/about/who-we-are/.
Accessed 2 Sep 2014.
The Leadership Conference on Civil and Human Rights.
(2014). Civil rights principles for the era of big data.
www.civilrights.org/press/2014/civil-rights-principles-
big-data.html. Accessed 7 Sep 2014.
N
Netflix
J. Jacob Jenkins
California State University Channel Islands,
Camarillo, CA, USA
Introduction
Netflix is a film and television provider
headquartered in Los Gatos, California. Netflix
was founded in 1997 as an online movie rental
service, using Permit Reply Mail to deliver
DVDs. In 2007, the company introduced stream-
ing content, which allowed customers instant
access to its online video library. Netflix has
since continued its trend toward streaming ser-
vices by developing a variety of original and
award-winning programming. Due to its success-
ful implementation of Big Data, Netflix has expe-
rienced exponential growth since its inception. It
currently offers over 100,000 titles on DVD and is
the world’s largest on-demand streaming service
with more than 80 million subscribers in over
190 countries worldwide.
Netflix and Big Data
Software executives Marc Randolph and Reed
Hastings founded Netflix in 1997. Randolph was
a previous cofounder of MicroWarehouse, a mail-
order computer company; Hastings was a
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_144-1
previous math teacher and founder of Pure Soft,
a software company he sold for $700 million. The
idea for Netflix was prompted by Hastings’ expe-
rience of paying $40 in overdue fees at a local
Blockbuster. Using $2.5 million dollars in start-up
money from his sale of Pure Soft, Hastings
envisioned a video provider whose content could
be returned from the comfort of one’s own home,
void of due dates or late fees. Netflix’s website
was subsequently launched on August 29, 1997.
Netflix’s original business model used a tradi-
tional pay-per-rental approach, charging 0.50
cents per film. Netflix introduced its monthly
flat-fee subscription service in September 1999,
which led to the termination of its pay-per-rental
model by early 2000. Netflix has since built its
global reputation on the flat-fee business model,
as well as its lack of due dates, late fees, or
shipping and handling charges. Netflix delivers
DVDs directly to its subscribers using the United
States Postal Service and a series of regional
warehouses located throughout the United States.
Based upon which subscription plan is chosen,
users can keep between one and eight DVDs at a
time, for as long as they desire. When subscribers
return a disc to Netflix using one of its prepaid
envelopes, the next DVD on their online rental
queue is automatically mailed in its stead. DVD-
by-mail subscribers can access and manage their
online rental queue through Netflix’s website in
order to add and delete titles or rearrange their
priority.
2 Netflix
In 2007 Netflix introduced streaming content
as part of its “Watch Instantly” initiative. When
Netflix first introduced streaming video to its
website, subscribers were allowed 1 h of access
for every $1 spent on their monthly subscription.
This restriction was later removed due to emerg-
ing competition from Hulu, Apple TV, Amazon
Prime, and other on-demand services. There are
substantially less titles available through Netflix’s
streaming service than its disc library. Despite this
limitation, Netflix has become the most widely
supported streaming service in the world by
partnering with Sony, Nintendo, and Microsoft
to allow access through Blu-ray DVD players, as
well as the Wii, Xbox, and PlayStation gaming
consoles. In subsequent years, Netflix has increas-
ingly turned attention toward its streaming ser-
vices. In 2008 the company added 2500 new
“Watch Instantly” titles through a partnership
with Starz Entertainment. In 2010 Netflix inked
deals with Paramount Pictures, Metro-Goldwyn-
Mayer, and Lions Gate Entertainment; in 2012 it
inked a deal with DreamWorks Animation.
Netflix has also bolstered its online library by
developing its own programming. In 2011 Netflix
announced plans to acquire and produce original
content for its streaming service. That same year it
outbid HBO, AMC, and Showtime to acquire the
production rights for House of Cards, a political
drama based on the BBC miniseries of the same
name. House of Cards was released on Netflix in
its entirety in early 2013. Additional program-
ming released during 2013 included Lilyhammer,
Hemlock Grove, Orange is the New Black, and the
fourth season of Arrested Development – a series
that originally aired on Fox between 2003 and
2006. Netflix later received the first Emmy
Award nomination for an exclusively online tele-
vision series. House of Cards, Hemlock Grove,
and Arrested Development received a total of
14 nominations at the 2013 Primetime Emmy
Awards; House of Cards received an additional
four nominations at the 2014 Golden Globe
Awards. In the end, House of Cards won three
Emmy Awards for “Outstanding Casting for a
Drama Series,” “Outstanding Directing for a
Drama Series,” and “Outstanding Cinematogra-
phy for a Single-Camera Series.” It won one
Golden Globe for “Best Actress in a Television
Series Drama.”
Through its combination of DVD rentals,
streaming services, and original programming,
Netflix has grown exponentially since 1997. In
2000, the company had approximately 300,000
subscribers. By 2005 that number grew to nearly
4 million users, and by 2010 it grew to 20 million.
During this time, Netflix’s initial public offering
(IPO) of $15 per share soared to nearly $500, with
a reported annual revenue of more than $6.78
billion in 2015. Today, Netflix is the largest source
of Internet traffic in all of North America. Its sub-
scribers stream more than 1 billion hours of media
content each month, approximating one-third of
total downstream web traffic. Such success has
resulted in several competitors for online stream-
ing and DVD rentals. Wal-Mart began its own
online rental service in 2002 before acquiring the
Internet delivery network, Vudu, in 2010. Ama-
zon Prime, Redbox Instant, Blockbuster @ Home,
and even “adult video” services like WantedList
and SugarDVD have also entered the video
streaming market. Competition from Blockbuster
sparked a price war in 2004, yet Netflix remains
the industry leader in online movie rentals and
streaming.
Netflix owes much of its success to the inno-
vative use of Big Data. Because it is an Internet-
based company, Netflix has access to an unprece-
dented amount of viewer behavior. Broadcast net-
works have traditionally relied on approximated
ratings and focus group feedback to make deci-
sions about their content and airtime. In contrast,
Netflix can aggregate specified data about cus-
tomers’ actual viewing habits in real time, allo-
wing it to understand subscriber trends and
tendencies at a much more sophisticated level.
The type of information Netflix gathers is not
limited to what viewers watch and the ratings
they ascribe. Netflix also tracks the specific dates
and times in which viewers watch particular pro-
gramming, as well as their geographic locations,
search histories, and scrolling patterns; when they
use pause, rewind, or fast-forward; the types of
streaming devices employed; and so on.
The information Netflix collects allows it to
deliver unrivaled personalization to each
Netflix
individual customer. This customization not only
results in better recommendations but also helps
to inform what content the company should invest
in. Once content has been acquired/developed,
Netflix’s algorithms also help to optimize their
marketing and to increase renewal rates on origi-
nal programming. As an example, Netflix created
ten distinct trailers to promote their original series
House of Cards. Each trailer was designed for a
different audience and seen by various customers
based on those customers’ previous viewing
behaviors. Meanwhile, the renewal rate for origi-
nal programming on traditional broadcast televi-
sion is approximately 35%; the current renewal
rate for original programming on Netflix is nearly
70%.
As successful as Netflix’s use of Big Data has
been, the company strives to keep pace with
changes in viewer habits, as well as changes in
its own product. When the majority of subscribers
used Netflix’s DVD-by-mail service, for instance,
those customers consciously added new titles to
their queue. Streaming services demand a more
instantaneous and intuitive process of generating
future recommendations. In response to develop-
ments such as this, Netflix initiated the “Netflix
Prize” in 2006: a $1 million payout to the first
person or group of persons to formulate a superior
algorithm for predicting viewer preferences. Over
the next 3 years, more than 40,000 teams from
183 countries were given access to over 100 mil-
lion user ratings. BellKor’s Pragmatic Chaos was
able to improve upon Netflix existing algorithm
by approximately 10% and was announced as the
award winner in 2009.
Conclusion
In summation, Netflix is presently the world’s
largest “Internet television network.” Key turning
points in the company’s development have
included a flat-rate subscription service, streaming
content, and original programming. Much of the
company’s success has also been due to its
3
innovative implementation of Big Data. An
unprecedented level of information about cus-
tomers’ viewing habits has allowed Netflix to
make informed decisions about programming
development, promotion, and delivery. As a
result, Netflix currently streams more than 1 bil-
lion hours of content per month to over 80 million
subscribers in 190 countries and counting.
Cross-References
▶ Algorithm
▶ Amazon
▶ Apple
▶ Communications
▶ Consumer Action
▶ Entertainment
▶ Facebook
▶ Internet
▶ Internet Tracking
▶ Microsoft
▶ Social Media
▶ Streaming Data
▶ Streaming Data Analytics
▶ Video
Further Readings
Keating, G. (2013). Netflixed: The epic battle for America’s
eyeballs. London: Portfolio Trade.
McCord, P. (2014). How Netflix reinvented HR. Harvard
Business Review. http://static1.squarespace.com/
static/5666931569492e8e1cdb5afa/t/56749ea457eb
8de4eb2f2a8b/1450483364426/How+Netflix+Reinven
ted+HR.pdf. Accessed 5 Jan 2016.
McDonald, K., & Smith-Rowsey, D. (2016). The Netflix
effect: Technology and entertainment in the 21st cen-
tury. London: Bloomsbury Academic.
Simon, P. Big data lessons from Netflix. Wired. Retrieved
from https://www.wired.com/insights/2014/03/big-
data-lessons-netflix/
Wingfield, N., & Stelter, B. (2011, October 24). How Netflix
lost 800,000 members, and good will. The New York
Times. http://faculty.ses.wsu.edu/rayb/econ301/Arti
cles/Netflix%20Lost%20800,000%20Members%20.
pdf. Accessed 5 Jan 2016.
N
Network Analytics
Jürgen Pfeffer
Bavarian School of Public Policy, Technical
University of Munich, Munich, Germany
Synonyms
Network science; Social network analysis
Much of big data comes with relational informa-
tion. People are friends with or follow each other
on social media platforms, send each other emails,
or call each other. Researchers around the world
copublish their work, and large-scale technology
networks like power grids and the Internet are the
basis for worldwide connectivity. Big data net-
works are ubiquitous and are more and more
available for researchers and companies to extract
knowledge about our society or to leverage new
business models based on data analytics. These
networks consist of millions of interconnected
entities and form complex socio-technical sys-
tems that are the fundamental structures
governing our world, yet defy easy understand-
ing. Instead, we must turn to network analytics to
understand the structure and dynamics of these
large-scale networked systems and to identify
important or critical elements or to reveal groups.
However, in the context of big data, network
analytics is also faced with certain challenges.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_147-1
Network Analytical Methods
Networks are defined as a set of nodes and a set of
edges connecting the nodes. The major questions
for network analytics, independent from network
size, are “Who is important?” and “Where are the
groups?” Stanley Wasserman and Katherine Faust
have authored a seminal work on network analyt-
ical methods. Even though this work was
published in the mid-1990s, it can still be seen as
the standard book on methods for network analyt-
ics, and it also provides the foundation for many
contemporary methods and metrics. With respect
to identifying the most important nodes in a given
network, a diverse array of centrality metrics have
been developed in the last decades. Marina
Henning and her coauthors classified centrality
metrics into four groups. “Activity” metrics
purely count the number or summarize the volume
of connections. For “radial” metrics, a node is
important if it is close to other nodes, and
“medial” metrics account for being in the middle
of flows in networks or for bridging different areas
of the network. “Feedback” metrics are based on
the idea that centrality can result from the fact that
a node is connected (directly or even indirectly) to
other central nodes. For the first three groups,
Linton C. Freeman has defined “degree central-
ity,” “closeness centrality,” and “betweenness
centrality” as the most intuitive metrics. These
metrics are used in almost every network analyt-
ical research project nowadays. The fourth metric
category comprises mathematically advanced
2 Network Analytics
methods based on eigenvector computation.
Phillip Bonacich presented eigenvector centrality
which led to important developments of metrics
for web analytics like Google’s PageRank algo-
rithm or the HITS algorithms by John Kleinberg,
which is incorporated into several search engines
to rank search results based on the website’s struc-
tural importance on the Internet.
The second big pile of research questions
related to networks is about identifying groups.
Groups can refer to a broad array of definitions,
e.g., nodes sharing of certain socioeconomic attri-
butes, membership affiliations, or geographic
proximity. When analyzing networks, we are
often interested in structurally identifiable groups,
i.e., sets of nodes of a network that are denser
connected among them and sparser connected to
all other nodes. The most obvious group of nodes
in a network would be a clique – a set of nodes
where each node is connected to all other nodes.
Other definitions of groups are more relaxed.
K-cores are a set of nodes for which every node
is connected to at least k other nodes in the set. It
turns out that k-cores are more realistic for real-
world data than cliques and much faster to calcu-
late. For any form of group identification in net-
works, we are often interested in evaluating the
“goodness” of the identified groups. The most
common approach to assess the quality of group-
ing algorithms is to calculate the modularity index
developed by Michelle Girvan and Mark
Newman.
Algorithmic Challenges
The most widely used algorithms in network ana-
lytics were developed in the context of small
groups of (less than 100) humans. When we
study big networks with millions of nodes, several
major challenges emerge. To begin with, most
network algorithms run in Y(n2) time or slower.
This means that if we double the number of nodes,
the calculation time is quadrupled. For instance,
let us assume we have a network with 1,000 nodes
and a second network with one million nodes
(thousandfold). If a certain centrality calculation
with quadratic algorithmic complexity takes
1 min on the first network, the same calculation
would take 1 million minutes (approximately
2 years) on the second network (millionfold).
This property of many network metrics makes it
nearly impossible to apply them to big data net-
works within reasonable time. Consequently,
optimization and approximation algorithms of tra-
ditional metrics are developed and used to speed
up analysis for big data networks.
A straight forward approach for algorithmic
optimization of network algorithms for big data
is parallelization. The abovementioned algorithms
closeness and betweenness centralities are based
on all-pairs shortest path calculation. In other
words, the algorithm starts at a node, follows its
links, and visits all other nodes in concentric cir-
cles. The calculation for one node is independent
from the calculation for all other nodes; thus,
different processors or different computers can
jointly calculate a metric with very little coordi-
nation overhead.
Approximation algorithms try to estimate a
centrality metric based on a small part of the
actual calculations. The calculations of the all-
pairs shortest path calculation can be restricted in
two ways. First, we can limit the centrality calcu-
lation to the k-step neighborhood of nodes, i.e.,
instead of visiting all other nodes in concentric
circles, we stop at a distance k. Second, instead of
all nodes, we just select a small proportion of
nodes as starting points for the shortest path cal-
culations. Both approaches can speed up calcula-
tion time tremendously as just a small proportion
of the calculations are needed to create these
results. Surprisingly, these approximated results
have very high accuracy. This is because real-
world networks are far from random and have
specific characteristics. For instance, networks
created from social interactions among people
often have core-periphery structure and are highly
clustered. These characteristics facilitate the accu-
racy of centrality approximation calculations. In
the context of optimizing and approximating tra-
ditional network metrics, a major future challenge
will be to estimate time/fidelity trade-offs(e.g.,
develop confidence intervals for network metrics)
and to build systems that incorporate the con-
straints of user and infrastructure into the
Network Analytics
calculations. This is especially crucial as certain
network metrics are very sensitive and small data
change can lead to big change of results.
New algorithms are especially developed for
very large networks. These algorithms have sub-
quadratic complexity so that they are applicable
for very large networks. Vladimir Batagelj and
Andrej Mrvar have developed a broad array of
new metrics and a network analytical tool called
“Pajek” to analyze networks with tens of millions
of nodes.
However, some networks are too big to fit into
the memory of a single computer. Imagine a net-
work with 1 billion nodes and 100 billion edges –
social media networks have already reached this
size. Such a network would require a computer
with about 3,000 gigabyte RAM to hold the pure
network structure with no additional information.
Even though supercomputer installations already
exist that can cope with these requirements, they
are rare and expensive. Instead, researchers make
use of computer clusters and analytical software
optimized for distributed systems, like Hadoop.
Streaming Data
Most modern big data networks come from
streaming data of interactions. Messages are sent
among nodes, people call each other, and data
flows are measured among servers. The observed
data consist of dyadic interaction. As the nodes of
the dyads overlap over time, we can extract net-
works. Even though networks extracted from
streaming data are inherently dynamic, the actual
analysis of these networks is often done with static
metrics, e.g., by comparing the networks created
from daily aggregation of data. The most interest-
ing research questions with respect to streaming
data are related to change detection. Centrality
metrics for every node or network level indices
that describe the structure of the network can be
calculated for every time interval. Looking at
these values as time series can help to identify
structural change in the dynamically changing
networks over time.
3
Visualizing Big Data Networks
Visualizing networks can be a very efficient ana-
lytical approach as human perception is capable of
identifying complex structures and patterns. To
facilitate visual analytics, algorithms are needed
that present network data in an interpretable way.
One of the major challenges for network visuali-
zation algorithms is to calculate the positions of
the nodes of the network in a way that it reveals
the structure of the network, i.e., show communi-
ties and put important nodes in the center of the
figure. The algorithmic challenges for visualizing
big networks are very similar to the ones
discussed above. Most commonly used layout
algorithms scale very poorly. Ulrich Brandes and
Christian Pich developed a layout algorithm based
on eigenvector analysis that can be used to visu-
alize networks with millions of nodes. The
method that they applied is similar to the before-
mentioned approximation approaches. As real-
world networks normally have a certain topology
that is far from random, calculating just a part of
the actual layout algorithm can be a good enough
approximation to reveal interesting aspects of a
network.
Networks are often enriched with additional
information about the nodes or the edges. We
often know the gender or the location of people.
Nodes might represent different types of infra-
structure elements. We can incorporate this infor-
mation by mapping data to visual elements of our
network visualization. Nodes can be visualized
with different shapes (circles, boxes, etc.) and
can be colored with different colors resulting in
multivariate network drawings. Adding contex-
tual information to compelling network visualiza-
tions can make the difference between pretty
pictures and valuable pieces of information
visualization.
Methodological Challenges
Besides algorithmic issues, we also face serious
conceptual challenges when analyzing big data
networks. Many “traditional” network analytical
metrics were developed for groups of tens of
4 Network Analytics
people. Applying the same metrics to very big
networks raises questions whether the algorithmic
assumptions or the interpretations of results are
still valid. For instance, the abovementioned met-
rics closeness and betweenness centralities just
incorporate the shortest paths between every pair
of nodes ignoring possible flow of information on
non-shortest paths. Even more, these metrics do
not take path length into account. In other words,
if a node is on the shortest path of length, two or
eight is treaded identically. Most likely this does
not reflect real-world assumptions of information
flow. All these issues can be addressed by apply-
ing different metrics that incorporate all possible
paths or a random selection of paths with length k.
In general, when accomplishing network analyt-
ics, we need to ask which of the existing network
algorithms are suitable under which assumptions
to be used for very large networks? Moreover,
what research questions are appropriate for very
large networks? Does being a central actor in a
group of high school kids has the same interpre-
tation as being a central user of an online social
network with millions of users?
Conclusions
Networks are everywhere in big data. Analyzing
these networks can be challenging. Due of the
very nature of network data and algorithms,
many traditional approaches of handling and
analyzing these networks are not scalable. None-
theless, it is worthwhile coping with these chal-
lenges. Researchers from different academic areas
have been optimizing existing and developing
new metrics and methodologies as network ana-
lytics can provide unique insights into big data.
Cross-References
▶ Algorithmic Complexity
▶ Complex Networks
▶ Data Visualization
▶ Streaming Data
Further Readings
Batagelj, V., Mrvar, A., & de Nooy, W. (2011). Exploratory
social network analysis with Pajek. (Expanded edi-
tion.). New York: Cambridge University Press.
Brandes, U., & Pich, C. (2007). Eigensolver Methods for
progressive multidimensional scaling of large data.
Proceedings of the 14th International Symposium on
Graph Drawing (GD’06), 42–53.
Freeman, L. C. (1979). Centrality in social networks: Con-
ceptual clarification. Social Networks, 1(3), 215–239.
Hennig, M., Brandes, U., Pfeffer, J., & Mergel, I. (2012).
Studying social networks. A guide to empirical
research. Frankfurt: Campus Verlag.
Wasserman, S., & Faust, K. (1994). Social network analy-
sis: Methods and applications. Cambridge: Cambridge
University Press.
N
Nutrition
Qinghua Yang1 and Yixin Chen2
1
Department of Communication Studies, Texas
Christian University, Fort Worth, TX, USA
2
Department of Communication Studies, Sam
Houston State University, Huntsville, TX, USA
Nutrition is a science that helps people to make
good choices of foods to keep healthy, by identi-
fying the amount of nutrients they need and the
amount of nutrients each food contains. Nutrients
are chemicals obtained from diet and are indis-
pensable to people’s health. Keeping a balanced
diet containing all essential nutrients can prevent
people from diseases caused by nutritional defi-
ciencies such as scurvy and pellagra.
Although the United States has one of the most
advanced nutrition sciences in the world, the
nutrition status of the U.S. population is not opti-
mistic. While nutritional deficiencies as a result of
dietary inadequacies are not very common, many
Americans are suffering from overconsumption-
related diseases. Due to the excessive intake of
sugar and fat, the prevalence of overweight and
obesity in the American adult population
increased from 47% to over 65% over the past
three decades, currently with two-thirds of Amer-
ican adults being overweight and among whom
36% being obese. Overweight and obesity are
concerns not only for the adult population, but
also for the childhood population, with one third
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_151-1
of American children being overweight or obese.
Obesity kills more than 2.8 million Americans
every year, and the obesity-related health prob-
lems cost American taxpayers more than $147
billion every year. Thus, reducing the obesity
prevalence in the United States has become a
national health priority.
Big data research on nutrition holds tremen-
dous promise for preventing obesity and improv-
ing population health. Recently, researchers have
been trying to apply big data to nutritional
research, by taking advantages of the increasing
amount of nutritional data and the accumulation
of nutritional studies. Big data is a collection of
data sets, which are large in volume and complex
in structure. For instance, the data managed by
America’s leading health care provider Kaiser is
more than 4,000 times the amount of information
stored in the Library of Congress. As to data
structure, nutritional data and ingredients are
really difficult to normalize. The volume and com-
plexity of nutritional big data make it difficult to
process them using traditional data analytic
techniques.
Big data analyses can provide more valuable
information than traditional data sets and reveal
hidden patterns among variables. In a big data
study sponsored by the National Bureau of Eco-
nomic Research, economists Matthew Harding
and Michael Lovenheim analyzed data of over
123 million purchasing decisions on food and
beverage made in the U.S. between 2002 and
2007 and simulated the effects of various taxes
2 Nutrition

on Americans’ buying habits. Their model pre- of nutritional epidemiology in 2013 versus just 1
dicted that an increase of 20% tax on sugar would in 1985. However, in the era of “big data”, there is
reduce Americans’ total caloric intake by 18% and an urgent need to translate big-data nutrition
reduce sugar consumption by over 16%. Based on research to practice, so that doctors and
their findings, they proposed a new policy of policymakers can utilize this knowledge to
implementing a broad-based tax on sugar to improve individual and population health.
improve public health. In another big-data study
on human nutrition, two researchers at West Vir
ginia University tried to understand and monitor
Controversy
the nutrition status of a population. They designed
intelligent data collection strategies and examined
Despite the exciting progress of big-data applica-
the effects of food availability on obesity occur-
tion in nutrition research, several challenges are
rence. They concluded that modifying environ-
equally noteworthy. First, to conduct big-data
mental factors (e.g., availability of healthy food)
nutrition research, researchers often need access
could be the key in obesity prevention.
to a complete inventory of foods purchased in all
Big data can be applied to self-tracking, that is,
retail outlets. This type of data, however, is not
monitoring one’s nutrition status. An emerging
readily available and gathering such information
trend in big data studies is quantified self (QS),
site by site is a time-consuming and complicated
which refers to keeping track of one’s nutritional,
process. Second, information provided by nutri-
biological and physical information, such as cal-
tion big data may be incomplete or incorrect. For
ories consumed, glycemic index, and specific
example, when doing self-tracking for nutrition
ingredients of food intake. By pairing the self-
status, many people fail to do consistent daily
tracking device with a web interface, the QS solu-
documentation or suffer from poor recall of food
tions can provide users with nutrient-data aggre-
intake. Also, big data analyses may be subject to
gation, infographic visualization, and personal
systematic biases and generate misleading
recommendations for diet.
research findings. Lastly, since an increasing
Big data can also enable researchers to monitor
amount of personal data is being generated
the global food consumption. One pioneering pro-
through quantified self-tracking devices, it is
ject is the Global Food Monitoring Group
important to consider privacy rights in personal
conducted by the George Institute for global
data. That individuals’ personal nutritional data
health with participations from 26 countries.
should be well-protected and that data shared and
With the support of these countries, the Group is
posted publicly should be used appropriately are
able to monitor the nutrition composition of var-
key ethical issues for nutrition researchers and
ious foods consumed around the world, identify
practitioners. In light of these challenges, techni-
the most effective food reformulation strategies,
cal, methodological, and educational interven-
and explore effective approaches on food produc-
tions are needed to deal with issues related to
tion and distribution by food companies in differ-
big-data accessibility, errors and abuses.
ent countries.
Thanks to the development of modern data
collection and analytic technologies, the amount
of nutritional, dietary, and biochemical data con- Cross-References
tinues to increase at a rapid pace, along with a
growing accumulation of nutritional epidemio- ▶ Biomedical Data
logic studies during this time. The field of nutri- ▶ Data Mining
tional epidemiology has witnessed a substantial ▶ Diagnostics
increase in systematic reviews and meta-analyses ▶ Health Informatics
over the past two decades. There were 523 meta-
analyses and systematic reviews within the field
Nutrition
Further Readings
Harding, M., & Lovenheim, M. (2017). The effect of prices
on nutrition: Comparing the impact of product-and
nutrient-specific taxes. Journal of Health Economics,
53.
Insel, P., et al. (2013). Nutrition. Boston: Jones and Bartlett
Publishers.
3
Satija, A., & Hu, F. (2014). Big data and systematic
reviews in nutritional epidemiology. Nutrition Reviews,
72(12).
Swan, M. (2013). The quantified self: Fundamental disrup-
tion in big data science and biological discovery. Big
Data, 1(2).
WVU Today. WVU researchers work to track nutritional
habits using ‘Big Data’. http://wvutoday.wvu.edu/n/
2013/01/11/wvu-researchers-workto-track-nutritional-
habits-using-big-data. Accessed Dec 2014.
O
Online Advertising
Yulia A. Strekalova
College of Journalism and Communications,
University of Florida, Gainesville, FL, USA
In a broad sense, online advertising means adver-
tising through cross-referencing on a business’s
own web portal or on the websites of other online
businesses. The goal of online advertising is to
attract attention to advertised websites and prod-
ucts and, potentially, lead to an enquiry about a
project, mail list subscription, or product pur-
chase. Online advertising creates new cost-saving
opportunities for businesses by reducing some of
the risks of ineffective advertising resources.
Online advertising types include banners, targeted
ads, and social media community interactions,
and each type requires careful planning and con-
sideration of potential ethical challenges.
Online advertising analytics and measurement
is necessary to assess the effectiveness of adver-
tising efforts and the return on the investment of
funds. However, measurement is challenged by
the fact that advertising across media platforms
is increasingly interactive. For example, a TV
commercial may lead to an online search, which
will result in a relevant online ad, which may lead
to a sale. The vast amounts of data and powerful
analytics are necessary to allow advertisers
performing high-definition cross-channel ana-
lyses of the public and its behaviors, evaluate the
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_152-1
return on investments across media, generate pre-
dictive models, and modify their campaigns in
near-real time. The proliferation of data collection
gave rise to increased concerns among the Internet
users and advocacy groups. As the user data are
collected by shared among multiple parties, they
may amount to become personally identifiable to a
particular person.
Types of Online Advertising
Online advertising, a multibillion-dollar industry
today, started from a single marketing email offer-
ing a new computer system sent in 1978 to
400 users of the Advanced Research Projects
Agency Network (ARPAnet). While the reactions
to this first online advertising campaign were neg-
ative and identified the message as spam, email
and forum-based advertising continued to develop
and grow. In 1993, a company called Global Net-
work Navigator sold the first clickable online
ad. AT&T, one of the early adopters of this adver-
tising innovation, received clicks from almost half
of the Internet users who were exposed to its
“Have you ever clicked your mouse right
HERE? – You will.” banner ad. In 1990s, online
advertising industry was largely fragmented, but
first ad networks started to appear and offer their
customers opportunities to develop advertising
campaigns that will place ads across a diverse set
of websites and reach particular audience seg-
ments. An advertising banner may be placed on
2 Online Advertising
high-traffic sites statically for a predefined period
of time. While this method may be the least costly
and targeted to a niche audience, it does not allow
for rich data collection. Banner advertising is a
less sophisticated form of online advertising. Ban-
ner advertising could also be used as a hybrid of
cost per mille (CPM), or cost per thousand, as
another advertising option which will deliver an
ad to website users. This option is usually priced
in a multiple of 1,000 impressions (or the number
of times an ad was shown) and an additional cost
for clicks. It also allows businesses to assess how
many times an ad was shown. However, this
method is limited in its ability to measure if the
return on an investment in advertising covered the
costs. However, proliferation of banners on sites
and the overall volume of information on sites
lead to “banner blindness” among the Internet
users. In addition, with rapid increase of mobile
phones as Internet connection devises, the average
effectiveness of banners became even lower. The
use of banner and pop-up ads increased in the late
1990s and early 2000s, but the users of the Inter-
net started to block these ads with pop-up
blockers, and the clicks on banner ads dropped
to about 0.1%.
The next innovation in the online advertising is
tied to the growth in sophistication of search
engines. The search engines started to allow
advertisers to place ad relevant to particular key-
words. Tying advertising to relevant search key-
words gave rise to the pay-per-click (PPC)
advertising. PPC provides advertisers with most
robust data to assess if expended costs generated
sufficient return. PPC advertising means that
advertisers are charged per click on an ad. This
advertising method ties exposure to advertising to
an action from a potential consumer thus provid-
ing advertisers with the data on the sites that are
more effective. Google AdWords is an example of
pay-per-click advertising, which is linked to the
keywords and phrases used in search. AdWords
ads are correlated with these keywords and shown
only to the Internet users with relevant searches.
By using PPC in conjunction with a search
engine, like Google, Bing, or Yahoo, advertisers
can also obtain insights on the environment or
search terms that led a consumer to the ad in the
first place.
Online advertising may also include direct
newsletter advertising delivered to potential cus-
tomers who have purchased before. However, the
decision to use this way of advertising should be
coupled with an ethical way of employing
it. Email addresses became a commodity and can
be bought. However, a newsletter sent to users
who never bought from a company may fire
back and lead to unintended negative conse-
quences. Overall, this low-cost advertising
method can be effective in keeping past customers
informed about new products and other cam-
paigns run by the company.
Social media is another advertising channel,
which is rapidly growing in its popularity. Social
media networks created repositories of psycho-
graphic data, which include user-reported demo-
graphic information, hobbies, travel destinations,
lifetime events, and topics of interest. Social
media can be used as more traditional advertising
channels for PPC ad placements. However, they
can also serve as a base for customer engagement.
Social media, although require a commitment and
time investment from advertisers, may generate
brand loyalty. Social media efforts, therefore,
require careful evaluation as they can be both
costly in terms of direct advertising costs and the
cost of time spent by company employees on
developing and executing social media campaign
and keeping the flow of communication active.
Data collected from social media channels can be
analyzed on the individual level, which was
nearly impossible with earlier online advertising
methods. Companies can collect information
about specific user communication and engage-
ment behavior, track communication activities of
individual users, and analyze comments shared by
the social media users. At the same time, aggre-
gate data may allow for general sentiment analysis
to assess if overall comments about a brand are
positive or negative and seek out product-related
signals shared by users. Social media evaluation,
however, is challenged by the absence of deep
understanding of the audience engagement met-
rics and lack of industry-wide benchmarks and
evaluation standards. As a fairly new area of
Online Advertising
advertising, social media evaluation of likes, com-
ments, and shares may be interpreted in a number
of ways. Social media networks provide a frame-
work for a new type of advertising, community
exchange, but they also are channels of online
advertising through real-time advertising
targeting. It is likely that focused targeting will
continue to be the focus of advertisers as it leads to
the increases in the effectiveness of advertising
efforts. At the same time, tracking of user web
behavior throughout the Web creates privacy con-
cerns and policy challenges.
Targeting
Innovations in online advertising introduced
targeting techniques that based advertising on
the past browsing and purchase behaviors of Inter-
net users. Proliferation of data collection enabled
advertisers to target potential clients based on a
multitude of web activities, like site browsing, key
word searchers, past purchasing across different
merchants, etc. These targeting techniques led to
the development of data collection systems that
track user activity in real time and make decisions
to advertise or not advertise right as the user is
browsing a particular page. Online advertising
lacks rigorous standardization and several recent
targeting typologies have been proposed.
Reviewing strategies for online advertising,
Gabriela Taylor identifies nine distinct targeting
methods, which overlap or complement the dis-
cussion of targeting methods proposed by other
authors. In general, targeting refers to situation
when ads that are shown to an Internet user are
relevant to their interests. The latter are deter-
mined by the keywords used on searchers, pages
visited, or online purchases made.
Contextual targeting ads are delivered to web
users based on the content of the sites these users
visit. In other words, contextually targeted adver-
tising matches ads to the content of the webpage
an Internet user is browsing. Systems managing
contextual advertising scan websites for key-
words and place ads that match these keywords
most closely. For example, a user viewing a
website about gardening may see ads for
3
gardening and house-keeping magazines or
home improvement stores.
Geo, or local, targeting is focused on the deter-
mination of the geographical location of a website
visitor. This information, in turn, is used to deliver
ads that are specific to a particular location, coun-
try, region or state, city, or metro area. In some
cases, targeting can go as deep as an organiza-
tional level. Internet protocol (IP) address,
assigned to each device participating a computer
network, is used as the primary data point in this
targeting method. The use of this method may
prevent the delivery of ads to users where product
or service is not available – for example, a content
restriction for Internet television or region-
specific advertising that complies with regional
regulations.
Demographic targeting, as implied by its name,
tailors ads based on website users’ demographic
information, like gender, age, income and educa-
tion level, marital status, ethnicity, language pref-
erences, and other data points. Users may supply
this information is social networking site registra-
tion. The sites, additionally, may also encourage
its users to “complete” their profiles after the
initial registration to get access to the fullest set
of data.
Behavioral targeting looks at users’ declared or
expressed interests to tailor the content of deliv-
ered ads. Web-browsing information, data on the
pages visited, the amount of time spent on partic-
ular pages, meta-data for the links that were
clicked, the searches conducted recently, and
information about recent purchases is collected
and analyzed by advertisement delivery systems
to select and display the most relevant ads. In a
sense, website publishers can create user profiles
based on the collected data and use it to predict
future browsing behavior and potential products
of interest. This approach, using rich past data,
allows advertisers to target their ads more effec-
tively to the page visitors who are more likely to
have interest in these products or services. Com-
bined with other strategies, including contextual,
geographic, and demographic targeting, this
approach may lead to finely tuned and interest-
tailored ads. The approach proves effective as
several studies showed that also Internet users
4 Online Advertising
prefer to have no ads on the web-pages they visit,
they favor relevant ads over random ones.
DayPart and time-based targeting is run during
specific times of the day or the week, for example,
10 am to 10 pm local time Monday through
Friday. Ads targeted based on this method are
displayed only during these days and times and
go off during the off-times. Ads run through
DayPart campaigns may focus on time-limited
offers and create a sense of urgency among audi-
ence members. At the same time, such ads may
create an increased sense of monitoring and lack
of privacy among the users exposed to these ads.
Real-time targeting allows for the ad place-
ment systems to place bids for advertisement
placement in real time. Additionally, this adver-
tising method allows to track every unique site
user and collect real-time data to assess the likeli-
hood of each visitor to make a purchase.
Affinity targeting creates a partnership
between a product producer and an interest-
based organization to promote the use of a third-
party product. This method targets customers who
share interest in a particular topic. These cus-
tomers are assumed to have positive attitude
toward a website they visit and therefore have a
positive attitude toward more relevant advertis-
ing. This method is akin to niche advertising,
and its success is based on the close match
between the advertising content and that of the
passions and interests of website users.
Look-alike targeting aims to identify prospec-
tive customers who are similar to the advertiser’s
customer base. Original customer profiles are
determined based on the website use and previous
behaviors of active customers. These profiles are
then matched against a pool of independent Inter-
net users who share common attributes and behav-
iors and are the likely targets for an advertised
product. The challenge with identifying these
look-alike audiences is challenged by the large
number of possible input data points which may
or may not be defining for a particular behavior or
user group.
Act-alike targeting is an outcome of predictive
analytics. Advertisers using this method define
profiles of customers based on their information
consumption and spending habits. Customers and
their past behaviors are identified; they are seg-
mented into groups to predict their future pur-
chase behavior. The goal of this method is to
identify the most loyal group of customers, who
generate revenue for the company and engage
with this group in a most effective and
supportive way.
Privacy Concerns
Technology is developing at a speed too rapid for
policy-making to catch up. Whichever advertising
targeting method is used, each is based on an
extended collections and analysis of personal
and behavioral data for each user. Ongoing and
potentially pervasive data collection raises impor-
tant privacy questions and concerns. Omer Tene
and Jules Polonetsky identify several privacy
risks associated with big data. First is an incre-
mental adverse effect on privacy from an ongoing
accumulation of information. More and more data
points are collected about individual Internet users
and once information about real identify has been
linked to a virtual identify of a user, the anonymity
is lost. Furthermore, disassociation of a user with
a particular service may be insufficient to break a
previously existing link as other networks and
online resources may have already harvested
missing data points. Second area of privacy risks
is an automated decision-making process. These
automated algorithms may lead to discrimination
and self-determination. Targeting and profiling
used in online advertising gives ground to poten-
tial threats to the free access to information and
open, democratic society. The third area of pri-
vacy concerns is predictive analysis, which may
identify and predict stigmatizing behaviors or
characteristics, like susceptibility to disease or
undisclosed sexual orientation. In addition, pre-
dictive analysis may give ground to social strati-
fication by putting users in like-behaving clusters
and ignoring outliers and minority groups.
Finally, the fourth area of concern is the lack of
access to information and exclusion of smaller
organizations and individuals from the access to
the benefits of big data. Large organizations are
able to collect and use big data to price products
Online Advertising
close to an individual’s reservation price or
cornering an individual with a deal impossible to
resist. At the same time, large organizations are
seldom forthcoming with sharing individuals’
information with these individuals in an assess-
able and understandable format.
Cross-References
▶ Advertising Self-Regulatory Council, Council
of Better Business Bureaus
▶ Content Management
▶ Data-Driven Marketing
▶ Data-Information-Knowledge-Wisdom
(DIKW) Pyramid, Framework, Continuum
5
▶ Predictive Analytics
▶ Social Media
Further Readings
Siegel, E. (2013). Predictive analytics: The power to pre-
dict who will click, buy, lie, or die. Hoboken: Wiley.
Taylor, G. (2013). Advertising in a digital age: Best prac-
tices & tips for paid search and social media advertis-
ing. Global & Digital.
Tene, O., & Polonetsky, J. (2013). Privacy in the age of big
data: A time for big decisions. Stanford Law Review
Online, 11/5.
Turow, J. (2012). The daily you: How the advertising
industry is defining your identity and your worth.
New Haven: Yale University Press.
O
Online Identity
Catalina L. Toma
Communication Science, University of
Wisconsin-Madison, Madison, WI, USA
Identity refers to the stable ways in which indi-
viduals or organizations think of and express
themselves. The availability of big data has
enabled researchers to examine online communi-
cators’ identity using generalizable samples.
Empirical research to date has focused on per-
sonal, rather than organizational, identity, and on
social media platforms, particularly Facebook and
Twitter, given that these platforms require users to
present themselves and their daily reflections to
audiences. Research to date has investigated the
following aspects of online identity: (1) expres-
sion, or how users express who they are, espe-
cially their personality traits and demographics (e.
g., gender, age) through social media activity; (2)
censorship, or how users suppress their urges to
reveal aspects of themselves on social media; (3)
detection, or the extent to which it is possible to
use computational tools to infer users’ identity
from their social media activity; (4) audiences,
or who users believe accesses their social media
postings and whether these beliefs are accurate;
(5) families, or the extent to which users include
family ties as part of their identity portrayals; and
(6) culture, or how users express their identities in
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_153-1
culturally determined ways. Each of these areas of
research is described in detail below.
Identity Expression
In its early days, the Internet appealed to many
users because it allowed them to engage with one
another anonymously. However, in recent years,
users have overwhelmingly migrated toward per-
sonalized interaction environments, where they
reveal their real identities and often connect with
members of their offline networks. Such is the
case with social media platforms. Therefore,
research has taken great interest in how users
communicate various aspects of their identities
to their audiences in these personalized
environments.
One important aspect of people’s identities is
their personality. Big data has been used to exam-
ine how personality traits get reflected in people’s
social media activity. How do people possessing
various personality traits talk, connect, and pre-
sent themselves online? The development of the
myPersonality Facebook application was instru-
mental in addressing these questions.
myPersonality administers personality question-
naires to Facebook users and then informs them
of their personality typology in exchange for
access to all their Facebook data. The application
has attracted millions of volunteers on Facebook
and has enabled researchers to correlate Facebook
activities with personality traits. The application,
2 Online Identity
used in all the studies summarized below, mea-
sures personality using the Big Five Model, which
specifies five basic personality traits: (1) extraver-
sion, or an individual’s tendency to be outgoing,
talkative, and socially active; (2) agreeableness, or
an individual’s tendency to be compassionate,
cooperative, trusting, and focused on maintaining
positive social relations; (3) openness to experi-
ence, or an individual’s tendency to be curious,
imaginative, and interested in new experiences
and ideas; (4) conscientiousness, or an individ-
ual’s tendency to be organized, reliable, consis-
tent, and focused on long-term goals and
achievement; and (5) neuroticism, or an individ-
uals’ tendency to experience negative emotions,
stress, and mood swings.
One study conducted by Yoram Bachrach and
his colleagues investigated the relationship
between Big Five personality traits and Facebook
activity for a sample of 180,000 users. Results
show that individuals high in extraversion had
more friends, posted more status updates, partici-
pated in more groups, and “liked” more pages on
Facebook; individuals high in agreeableness
appeared in more photographs with other
Facebook users but “liked” fewer Facebook
pages; individuals high in openness to experience
posted more status updates, participated in more
groups, and “liked” more Facebook pages; indi-
viduals high in conscientiousness posted more
photographs but participated in fewer groups and
“liked” fewer Facebook pages; and individuals
high in neuroticism had fewer friends but partici-
pated in more groups and “liked” more Facebook
pages. A related study, conducted by Michal
Kosinski and his colleagues, replicated these find-
ings on a sample of 350,000 American Facebook
users, the largest dataset to date on the relationship
between personality and Internet behavior.
Another study examined the relationship
between personality traits and word usage in the
status updates of over 69,000 English-speaking
Facebook users. Results show that personality
traits were indeed reflected in natural word use.
For instance, extroverted users used words
reflecting their sociable nature, such as “party,”
whereas introverted users used words reflecting
their more solitary interests, such as “reading” and
“Internet.” Similarly, highly conscientious users
expressed their achievement orientation through
words such as “success,” “busy,” and “work,”
whereas users high in openness to experience
expressed their artistic and intellectual pursuits
through words like “dreams,” “universe,” and
“music.”
In sum, this body of work shows that people’s
identity, operationalized as personality traits, is
illustrated in the actions they undertake and
words they use on Facebook. Given social media
platforms’ controllable nature, which allows users
time to ponder their claims and the ability to edit
them, researchers argue that these digital traces
likely illustrate users’ intentional efforts to com-
municate their identity to their audience, rather
than being unintentionally produced.
Identity Censorship
While identity expression is frequent in social
media and, as discussed above, illustrated by
behavioral traces, sometimes users suppress iden-
tity claims despite their initial impulse to divulge
them. This process, labeled “last-minute self-cen-
sorship,” was investigated by Sauvik Das and
Adam Kramer using data from 3.9 million
Facebook users over a period of 17 days. Censor-
ship was measured as instances when users
entered text in the status update or comment
boxes on Facebook but did not post it in the next
10 min. The results show that 71% of the partic-
ipants censored at least one post or comment
during the time frame of the study. On average,
participants censored 4.52 posts and 3.20 com-
ments. Notably, 33% of all posts and 13% of all
comments written by the sample were censored,
indicating that self-censorship is a fairly prevalent
phenomenon. Men censored more than women,
presumably because they are less comfortable
with self-disclosure. This study suggests that
Facebook users take advantage of controllable
media affordances, such as editability and unlim-
ited composition time, in order to manage their
identity claims. These self-regulatory efforts are
perhaps a response to the challenging nature of
addressing large and diverse audiences, whose
Online Identity
interpretation of the poster’s identity claims may
be difficult to predict.
Identity Detection
Given that users leave digital traces of their per-
sonal characteristics on social media platforms,
research has been concerned with whether it is
possible to infer these characteristics from social
media activity. For instance, can we deduce users’
gender, sexual orientation, or personality from
their explicit statements and patterns of activity?
Is their identity implicit in their social media
activity, even though they might not disclose it
explicitly?
One well-publicized study by Michal Kosinski
and his colleagues sought to predict Facebook
users’ personal characteristics from their “likes”
– that is, Facebook pages dedicated to products,
sports, music, books, restaurant, and interests –
that users can endorse and with which they can
associate by clicking the “like” button. The study
used a sample of 58,000 volunteers recruited
through the myPersonality application. Results
show that, based on Facebook “likes,” it is possi-
ble to predict a user’s ethnic identity (African-
American vs. Caucasian) with 95% accuracy, gen-
der with 93% accuracy, religion (Christian vs.
Muslim) with 82% accuracy, political orientation
(Democrat vs. Republican) with 85% accuracy,
sexual orientation among men with 88% accuracy
and among women with 75% accuracy, and rela-
tionship status with 65% accuracy. Certain “likes”
stood out as having particularly high predictive
ability for Facebook users’ personal characteris-
tics. For instance, the best predictors of high intel-
ligence were “The Colbert Report,” “Science,”
and, unexpectedly, “curly fries.” Conversely, low
intelligence was indicated by “Sephora,” “I Love
Being a Mom,” “Harley Davidson,” and “Lady
Antebellum.”
In the area of personality, two studies found
that users’ extraversion can be most accurately
inferred from Facebook profile activity (e.g.,
group membership, number of friends, number
of status updates); neuroticism, conscientious-
ness, and openness to experience can be
3
reasonably inferred; and agreeableness cannot be
inferred at all. In other words, Facebook activity
renders extraversion highly visible and agreeable-
ness opaque.
Language can also be used to predict online
communicators’ identity, as shown by Andrew
Schwartz and his colleagues in a study of 15.4
million Facebook status updates, totaling over
700 million words. Language choice, including
words, phrases, and topics of conversation, was
used to predict users’ gender, age, and Big Five
personality traits with high accuracy.
In sum, this body of research suggests that it is
possible to infer many facets of Facebook users’
identity through automated analysis of their
online activity, regardless of whether they explic-
itly choose to divulge this identity. While users
typically choose to reveal their gender and ethnic-
ity, they can be more reticent in disclosing their
relational status or sexual orientation and might
themselves be unaware of their personality traits
or intelligence quotient. This line of research
raises important questions about users’ privacy
and the extent to which this information, once
automatically extracted from Facebook activity,
should be used by corporations for marketing or
product optimization purposes.
Real and Imagined Audience for Identity
Claims
The purpose of many online identity claims is to
communicate a desired image to an audience.
Therefore, the process of identity construction
involves understanding the audience and targeting
messages to them. Social media, such as
Facebook and Twitter, where identity claims are
posted very frequently, pose a conundrum in this
regard, because audiences tend to be unprecedent-
edly large, sometimes reaching hundreds and
thousands of members, and diverse. Indeed,
“friends” and “followers” are accrued over time
and often belong to different social circles (e.g.,
high school, college, employment). How do users
conceptualize their audiences on social media
platforms? Are users’ mental models of their audi-
ences accurate?
4 Online Identity
These questions were addressed by Michael
Bernstein and his colleagues in a study focusing
specifically on Facebook users. The study used a
survey methodology, where Facebook users indi-
cated their beliefs about how many of their
“friends” viewed their Facebook postings,
coupled with large-scale log data for 220,000
Facebook users, where researchers captured the
actual number of “friends” who viewed users’
postings. Results show that, by and large,
Facebook users underestimated their audiences.
First, they believed that any specific status update
they posted was viewed, on average, by 20
“friends,” when in fact it was viewed by 78
“friends.” The median estimate for the audience
size for any specific post was only 27% of the
actual audience size, meaning that participants
underestimated the size of their audience by a
factor of 4. Second, when asked how many total
audience members they had for their profile post-
ings during the past month, Facebook users
believed it was 50, when in fact it was 180. The
median perceived audience for the Facebook pro-
file, in general, was only 32% of the actual audi-
ence, indicating that users underestimated their
cumulative audience by a factor of 3. Slightly
less than half of Facebook users indicated they
wanted a larger audience for their identity claims
than they thought they had, ironically failing to
understand that they did in fact have this larger
audience. About half of Facebook users indicated
that they were satisfied with the audience they
thought they had, even though their audience
was actually much greater than they perceived it
to be. Overall, this study highlights a substantial
mismatch between users’ beliefs about their audi-
ences and their actual audiences, suggesting that
social media environments are translucent, rather
than transparent, when it comes to audiences. That
is, actual audiences are somewhat opaque to users,
who as a result may fail to properly target their
identity claims to their audiences.
Family Identity
One critical aspect of personal identity is family
ties. To what extent do social media users reveal
their family connections to their audience, and
how do family members publically talk to one
another on these platforms? Moira Burke and her
colleagues addressed these questions in the con-
text of parent-child interactions on Facebook.
Results show that 37.1% of English-speaking
US Facebook users specified either a parent or
child relationship on the site. About 40% of teen-
agers specified at least one parent on their profile,
and almost half of users age 50 or above specified
a child on their profile. The most common family
ties were between mothers and daughters (41.4%
of all parent-child ties), followed by mothers and
sons (26.8%), fathers and daughters (18.9%), and
least of all fathers and sons (13.1%). However,
Facebook communication between parents and
children was limited, accounting for only 1–4%
of users’ public Facebook postings. When com-
munication did happen, it illustrated family iden-
tities: Parents gave advice to children, expressed
affection, and referenced extended family mem-
bers, particularly grandchildren.
Cultural Identity
Another critical aspect of personal identity is cul-
tural identity. Is online communicators’ cultural
identity revealed by their communication pat-
terns? Jaram Park and colleagues show that Twit-
ter users create emoticons that reflect an
individualistic or collectivistic cultural orienta-
tion. Specifically, users from individualistic cul-
tures preferred horizontal and mouth-oriented
emoticons, such as :), whereas users from collec-
tivistic cultures preferred vertical and eye-ori-
ented emoticons, such as ^_^. Similarly, a study
of self-expression using a sample of four million
Facebook users from several English-speaking
countries (USA, Canada, UK, Australia) shows
that members of these cultures can be differenti-
ated through their use of formal or informal
speech, the extent to which they discuss positive
personal events, and the extent to which they
discuss school. In sum, this research shows that
cultural identity is evident in linguistic self-
expression on social media platforms.
Online Identity 5

Cross-References and Social Media (ICWSM) (pp. 41–50). Association

▶ Anonymity
▶ Behavioral Analytics
▶ Facebook
▶ Privacy
▶ Profiling
▶ Psychology
▶ Twitter
Further Readings
Bachrach, Y., et al. (2012). Personality and patterns of
Facebook usage. In Proceedings of the 3rd Annual
Web Science Conference (pp. 24–32). Association for
Computing Machinery.
Bernstein, M., et al. (2013). Quantifying the invisible audi-
ence in social networks. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems
(pp. 21–30). Association for Computing Machinery.
Burke, M., et al. (2013). Families on Facebook. In Pro-
ceedings of the International Conference on Weblogs
for the Advancement of Artificial Intelligence.
Das, S., & Kramer, A. (2013). Self-censorship on
Facebook. In Proceedings of the 2013 Conference on
Computer-Supported Cooperative Work (pp.
793–802). Association for Computing Machinery.
Kern, M., et al. (2014). The online social self: An open
vocabulary approach to personality. Assessment, 21,
158–169.
Kosinski, M., et al. (2013). Private traits and attributes are
predictable from digital records of human behavior.
Proceedings of the National Academy of Sciences,
110, 5802–5805.
Kramer, A., & Chung, C. (2011). Dimensions of self-
expression in Facebook status updates. In Proceedings
of the International Conference on Weblogs and Social
Media (ICWSM) (pp. 169–176). Association for the
Advancement of Artificial Intelligence.
Park, J., et al. (2014). Cross-cultural comparison of non-
verbal cues in emoticons on twitter: Evidence from big
data analysis. Journal of Communication, 64, 333–354.
Schwartz, A., et al. (2013). Personality, gender, and age in
the language of social media: The open-vocabulary
approach. PloS One, 8, e73791.
O
Open-Source Software
Marc-David L. Seidel
Sauder School of Business, University of British
Columbia, Vancouver, BC, Canada
Open-source software refers to computer software
where the copyright holder provides anybody the
right to edit, modify, and distribute the software
free of charge. The initial creation of such soft-
ware spawned the open-source movement. Fre-
quently the only limitation on the intellectual
property rights are that any subsequent changes
made by others are required to be made with
similarly open intellectual property rights. Such
software is often developed in an open collabora-
tive manner by a Community Form (C-form)
organization. A large percentage of the internet
infrastructure is operated utilizing such software
which handles the majority of networking, web
serving, e-mail, and network diagnostics. With the
spread of the internet, the volume of user gener-
ated data has expanded exponentially, and open-
source software to manage and analyze big data
has flourished through open-source big data pro-
jects. This entry explains the history of open-
source software, the typical organizational struc-
ture used to create such software, prominent pro-
ject examples of the software focused on
managing and analyzing big data, and the future
evolution suggested by current research on the
topic.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_157-1
History of Open-Source Software
Two early software projects leading to the mod-
ern-day open-source software growth were at the
Massachusetts Institute of Technology (MIT) and
the University of California at Berkeley. The Free
Software Foundation, created by Richard
Stallman of the MIT Artificial Intelligence Lab,
was launched as a nonprofit organization to pro-
mote the development of free software. Stallman
is credited with creating the term “copyleft” and
created the GNU operating system as an operating
system composed entirely of free software. The
free BSD Unix operating system was developed
by Bill Jolitz of the University of California at
Berkeley Computer Science Research Group and
served as the basis for many later Unix operating
system releases. Many open-source software pro-
jects were unknown outside of the highly techni-
cal computer science community. Stallman’s
GNU was later popularized by Linus Torvalds, a
Finish computer science student, who released a
Linux kernel based upon the earlier work. The
release of Linux triggered substantial media atten-
tion for the open-source movement when an inter-
nal Microsoft strategy document, dubbed the
Halloween Documents, was leaked. It outlined
Microsoft’s perception of the threat of Linux to
Microsoft’s dominance of the operating system
market. Linux was portrayed in the mass media
as a free alternative to the Microsoft Windows
operating system. Eric S. Raymond and Bruce
Perens further formalized open source as a
2 Open-Source Software
development method by creating the Open Source
Initiative in 1998. By 1998, open-source software
routed 80% of the e-mail on the internet. It has
continued to flourish to the modern day being
responsible for a large number of software and
information-based products today produced by
the open-source movement.
C-form Organizational Architecture
The C-form organizational architecture is the pri-
mary organizational structure for open-source
development projects. A typical C-form has four
common organizing principles. First, there are
informal peripheral boundaries for developers.
Contributors can participate as much or as little
as they like and join or leave a project on their
own. Second, many contributors receive no finan-
cial compensation at all for their work, yet some
may have employment relationships with more
traditional organizations which encourage their
participation in the C-form as part of their regular
job duties. Third, C-forms focus on information-
based product, of which software is a major sub-
set. Since the product of a typical C-form is infor-
mation based, it can be replicated with minimal
effort and cost. Fourth, typical C-forms operate
with a norm of open transparent communication.
The primary intellectual property of an open-
source C-form is the software code. This, by def-
inition, is made available for any and all to see,
use, and edit.
Prominent Examples of Open-Source Big
Data Projects
Apache Casandra is a distributed database man-
agement system originally developed by Avinash
Lakshman and Prashant Malik at Facebook as a
solution to handle searching an inbox. It is now
developed by the Apache Software Foundation, a
distributed community of developers. It is
designed to handle large amounts of data distrib-
uted across multiple datacenters. It has been rec-
ognized by University of Toronto researchers as
having leading scalability capabilities.
Apache CouchDB is a web-focused database
system originally developed by Damien Katz, a
former IBM developer. Similar to Apache
Casandra, it is now developed by the Apache
Software Foundation. It is designed to deal with
large amounts of data through multi-master repli-
cation across multiple locations.
Apache Hadoop is designed to store and pro-
cess large-scale datasets using multiple clusters of
standardized low-level hardware. This technique
allows for parallel processing similar to a super-
computer but using mass market off the shelf
commodity computing systems. It was originally
developed by Doug Cutting and Mike Cafarella.
Cutting was employed at Yahoo, and Cafarella
was a Masters student at the University of Wash-
ington at the time. It is now developed by the
Apache Software Foundation. It serves a similar
purpose as Storm.
Apache HCatalog is a table and storage man-
agement layer for Apache Hadoop. It is focused
on assisting grid administrators with managing
large volumes of data without knowing exactly
where the data is stored. It provides relational
views of the data, regardless of what the source
storage location is. It is developed by the Apache
Software Foundation.
Apache Lucene is an information retrieval soft-
ware library which tightly integrates with search
engine projects such as ElasticSearch. It provides
full text indexing and searching capabilities. It
treats all document formats similarly by extracting
textual components and as such is independent of
file format. It is developed by the Apache Soft-
ware Foundation and released under the Apache
Software License.
D3.js is a data visualization package originally
created by Mike Bostock, Jeff Heer, and Vadim
Ogievetsky who worked together at Stanford Uni-
versity. It is now licensed under the Berkeley
Software Distribution (BSD) open-source license.
It is designed to graphically represent large
amounts of data and is frequently used to generate
rich graphs and for map making.
Drill is a framework to support distributed
applications for data intensive analysis of large-
scale datasets in a self-serve manner. It is inspired
by Google’s BigQuery infrastructure service. The
Open-Source Software
stated goal for the project is to scale to 10,000 or
more servers to make low-latency queries of
petabytes of data in seconds in a self-service man-
ner. It is being incubated by Apache currently. It is
similar to Impala.
ElasticSearch is a search server that provides
near real-time full-text search engine capabilities
for large volumes of documents using a distrib-
uted infrastructure. It is based upon Apache
Lucene and is released under the Apache Software
License. It spawned a venture-funded company in
2012 created by the people responsible for
ElasticSearch and Apache Lucene to provide sup-
port and professional services around the
software.
Impala is an SQL query engine which enables
massively parallel processing of search queries on
Apache Hadoop. It was announced in 2012 and
moved out of beta testing in 2013 to public avail-
ability. It is targeted at data analysts and scientists
who need to conduct analysis on large-scale data
without reformatting and transferring the data to a
specialized system or proprietary format. It is
released under the Apache Software License and
has professional support available from the ven-
ture-funded Cloudera. It is similar to Drill.
Julia is a technical computing high-perfor-
mance dynamic programming language with a
focus on distributed parallel execution with high
numerical accuracy using an extensive mathemat-
ical function library. It is designed to use a simple
syntax familiar to many developers of older pro-
gramming languages while being updated to be
more effective with big data. The aim is to speed
development time by simplifying coding for par-
allel processing support. It was first released in
2012 under the MIT open-source license after
being originally developed starting in 2009 by
Alan Edelman (MIT), Jeff Bezanson (MIT),
Stefan Karpinski (UCSB), and Viral Shah
(UCSB).
Kafka is a distributed, partitioned, replicated
message broker targeted on commit logs. It can be
used for messaging, website activity tracking,
operational data monitoring, and stream pro-
cessing. It was originally developed by LinkedIn
and released open source in 2011. It was subse-
quently incubated by the Apache Incubator and as
3
of 2012 is developed by the Apache Software
Foundation.
Lumify is a big data analysis and visualization
platform originally targeted to investigative work
in the national security space. It provides real-time
graphical visualizations of large volumes of data
and automatically searches for connections
between entities. It was originally created by Alta-
mira Technologies Corporation and then released
under the Apache License in 2014.
MongoDB is a NoSQL document focused
database focused on handling large volumes of
data. The software was first developed in 2007
by 10gen. In 2009, the company made the soft-
ware open source and focused on providing pro-
fessional services for the integration and use of the
software. It utilizes a distributed file storage, load
balancing, and replication system to allow quick
ad hoc queries of large volumes of data. It is
released under the GNU Affero General Public
License and uses drivers released under the
Apache License.
R is a technical computing high-performance
programming language focused on statistical
analysis and graphical representations of large
datasets. It is an implementation of the S program-
ming language created by Bell Labs’ John Cham-
bers. It was created by Ross Ihaka and Robert
Gentleman at the University of Auckland. It is
designed to allow multiple processors to work
on large datasets. It is released under the GNU
License.
Scribe is a log server designed to aggregate
large volumes of server data streamed in real
time from a high volume of servers. It is com-
monly described as a scaling tool. It was origi-
nally developed by Facebook and then released in
2008 using the open-source Apache License.
Spark is a data analytic cluster computing
framework designed to integrate with Apache
Hadoop. It has the capability to cache large
datasets in memory to interactively analyze the
data and then extract a working analysis set to
further analyze quickly. It was originally devel-
oped at the University of California at Berkeley
AMPLab and released under the BSD License.
Later it was incubated in 2013 at the Apache
Incubator and released under the Apache License.
4 Open-Source Software
Major contributors to the project include Yahoo
and Intel.
Storm is a programming library focused on
real-time storage and retrieval of dynamic object
information. It allows complex querying across
multiple database tables. It handles unbound
streams of data in an instantaneous manner allo-
wing real-time analytics of big data and continu-
ous computation. The software was originally
developed by Canonical Ltd., also known for the
Ubuntu Linux operating system, and is released
under the GNU Lesser General Public License. It
is similar to Apache Hadoop but with a more real-
time and less batch-focused nature.
The Future
The majority of open-source software focused on
big data applications has primarily been targeting
web-based big data sources and corporate data
analytics. Current developments suggest a shift
toward more analysis of real-world data as sensors
spread more widely into everyday use by mass
market consumers. As consumers provide more
and more data passively through pervasive sen-
sors, the open-source software used to manage
and understand big data appears to be shifting
toward analyzing a wider variety of big data
sources. It appears likely that the near future will
provide more open-source software tools to ana-
lyze real-world big data such as physical
movements, biological data, consumer behavior,
health metrics, and voice content.
Cross-References
▶ Apache
▶ Crowdsourcing
▶ Distributed Computing
▶ Global Open Data Initiative
▶ Google Flu
▶ Wikipedia
Further Readings
Bretthauer, D. (2002). Open source software: A history.
Information Technology and Libraries, 21(1), 3–11.
Lakhani, K. R., & von Hippel, E. (2003). How open source
software works: ‘Free’ user-to-user assistance.
Research Policy, 32(6), 923–943.
Marx, V. (2013). Biology: The big challenges of big data.
Nature, 498, 255–260.
McHugh, J. (1998, August). For the love of hacking.
Forbes.
O’Mahony, S., & Ferraro, F. (2007). The emergence of
governance on an open source project. Academy of
Management Journal, 50(5), 1079–1106.
Seidel, M.-D. L., & Stewart, K. (2011). An initial descrip-
tion of the C-form. Research in the Sociology of Orga-
nizations, 33, 37–72.
Shah, S. K. (2006). Motivation, governance, and the via-
bility of hybrid forms in open source software devel-
opment. Management Science, 52(7), 1000–1014.
P
Participatory Health and Big Data
Muhiuddin Haider,
Yessenia Gomez and Salma Sharaf
School of Public Health Institute for Applied
Environmental Health, University of Maryland,
College Park, MD, USA
The personal data landscaped has changed drasti-
cally with the rise of social networking sites and
the Internet. The Internet and social media sites
have allowed for the collection of large amounts
of personal data. Every keystroke typed, website
visited, Facebook post liked, Tweet posted, or
video shared becomes part of a user’s digital his-
tory. A large net is cast collecting all the personal
data into big data sets that may be subsequently
analyzed. This type of data has been analyzed for
years by marketing firms through the use of algo-
rithms that analyze and predict consumer purchas-
ing behavior. The digital history of an individual
paints a clear picture about their influence in the
community and their mental, emotional, and
financial state, and much about an individual can
be learned through the tracking of his or her data.
When big data is fine-tuned, it can benefit the
people and community at large. Big data can be
used to track epidemics, and its analysis can be
used in the support of patient education, treatment
of at-risk individuals, and encouragement of par-
ticipatory community health. However, with the
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_159-1
rise of big data comes concern about the security
of health information and privacy.
There are advantages and disadvantages to
casting large data nets. Collecting data can help
organizations learn about individuals and commu-
nities at large. Following online search trends and
collecting big data can help researchers under-
stand health problems currently facing the studied
communities and can similarly be used to track
epidemics. For example, increases in Google
searches for the term flu have been correlated
with an increase in flu patient visits to emergency
rooms. In addition, a 2008 Pew study revealed
that 80% of Internet users use the Internet to
search for health information. Today, many
patients visit doctors after having already
searched their symptoms online. Furthermore,
more patients are now using the Internet to search
health information, seek medical advice, and
make important medical decisions. The rise of
the Internet has led to more patient engagement
and participation in health.
Technology has also encouraged participatory
health through an increase in interconnectedness.
Internet technology has allowed for constant
access to medical specialists and support groups
for people suffering from diseases or those
searching for health information. The use of tech-
nology has allowed individuals to take control of
their own health, through the use of online
searches and the constant access to online health
records and tailored medical information. In the
United States, hospitals are connecting
2 Participatory Health and Big Data
individuals to their doctors through the use of
online applications that allow patients to email
their doctors, check prescriptions, and look at
visit summaries from anywhere where they have
an Internet connection. The increase in patient
engagement has been seen to play a major role
in promotion of health and improvement in qual-
ity of healthcare.
Technology has also helped those at risk of
disease seek treatment early or be followed care-
fully before contracting a disease. Collection of
big data has helped providers see health trends in
their communities, and technology has allowed
them to reach more people with targeted health
information. A United Nations International Chil-
dren’s Emergency Fund (UNICEF) project in
Uganda asked community members to sign up
for U-report, a text-based system that allows indi-
viduals to participate in health discussions
through weekly polls. This system was
implemented to connect and increase communi-
cation between the community and the govern-
ment and health officials. The success of the
program helped UNICEF prevent disease out-
breaks in the communities and encouraged
healthy behaviors. U-report is now used in other
countries to help mobilize communities to play
active roles in their personal health.
Advances in technology have also created
wearable technology that is revolutionizing par-
ticipatory health. Wearable technology is a cate-
gory of devices that are worn by individuals and
are used to track data about the individuals, such
as health information. Examples of wearable tech-
nology are wrist bands that collect information
about the individual’s global positioning system
(gps) location, amount of daily exercise, sleep
patterns, and heart rate. Wearable technology
enables users to track their health information,
and some wearable technology even allows the
individual to save their health information and
share it with their medical providers. Wearable
technology encourages participatory health, and
the constant tracking of health information and
sharing with medical providers allow for more
accurate health data collection and tailored care.
The increase in health technology and collection
and analysis of big data has led to an increase in
participatory health, better communication
between individuals and healthcare providers,
and more tailored care.
Big data collected from these various sources,
whether Internet searches, social media sites, or
participatory health through applications and
technology, strongly influences our modern health
system. The analysis of big data has helped med-
ical providers and researchers understand health
problems facing their communities and develop
tailored programs to address health concerns, pre-
vent disease, and increase community participa-
tory health. Through the use of big data
technology, providers are now able to study health
trends in their communities and communicate
with their patients without scheduling any medi-
cal visits. However, big data also creates concern
for the security of health information.
There are several disadvantages to the collec-
tion of big data. One being that not all the data
collected is significant and much of the informa-
tion collected may be meaningless. Additionally,
computers lack the ability to interpret information
the way humans do, so something that may have
multiple interpretations may be misinterpreted by
a computer. Therefore, data may be flawed if
simply interpreted based on algorithms, and any
decisions regarding the health of the communities
that were made based on this inaccurate data
would also be flawed. Of greater concern is the
issue of privacy with regards to big data. Much of
the data is collected automatically based on peo-
ple’s online searches and Internet activities, so the
question arises as to whether people have the right
to choose what data is collected about them. Ques-
tions that arise regarding big data and health
include how long is personal health data saved?
Will data collected be used against individuals?
How will the Health Insurance Portability and
Accountability Act (HIPPA) change with the
incorporation of big data in medicine? Will data
collected determine insurance premiums? Privacy
concerns need to be addressed before big health
data, health applications, and wearable technol-
ogy become a security issue.
Today, big data can help health providers better
understand their target populations and can lead to
an increase in participatory health. However,
Participatory Health and Big Data 3

concerns arise about the safety of health informa- Further Readings

tion that is automatically collected in big data sets.
With this in mind, targeted data collected may be a Eysenbach, G. (2008). Medicine 2.0: Social networking,
collaboration, participation, apomediation, and open-
more beneficial method for data collection with
ness. Journal of Medical Internet Research, 10(3), e22.
regard to health. All these concerns need to be doi:10.2196/jmir.1030.
addressed today as the use of big data in health Gallant, L. M., Irizarry, C., Boone, G., & Kreps, G. (2011).
becomes more commonplace. Promoting participatory medicine with social media:
New media applications on hospital websites that
enhance health education and e-patients’ voices. Jour-
nal of Participatory Medicine, 3, e49.
Gallivan, J., Kovacs Burns, K. A., Bellows, M., & Eigen-
Cross-References seher, C. (2012). The many faces of patient engage-
ment. Journal of Participatory Medicine, 4, e32.
▶ Epidemiology Lohr, S. (2012). The age of big data. The New York Times.
▶ Marketing/Advertising Revolutionizing social mobilization, monitoring and
response efforts. (2012) UNICEF [video file].
▶ Medical/Health Care Retrieved from https://www.youtube.com/watch?v=
▶ Patient-Centered (Personalized) Health gRczMq1Dn10
▶ PatientsLikeMe The promise of personalized medicine. (2007, Winter).
▶ Prevention NIH Medline Plus, pp. 2–3.
P
Patient Records
Barbara Cook Overton
Communication Studies, Southeastern Louisiana
University, Hammond, LA, USA
Patient records have existed since the first hospi-
tals were opened. Early handwritten accounts of
patients’ hospitalizations were recorded for edu-
cational purposes but most records were simply
tallies of admissions and discharges used to justify
expenditures. Standardized forms would eventu-
ally change how patient care was documented.
Content shifted from narrative to numerical
descriptions, largely in the form of test results.
Records became unwieldy as professional guide-
lines and malpractice concerns required more and
more data be recorded. Patient records are owned
and maintained by individual providers, meaning
multiple records exist for most patients. Nonethe-
less, the patient record is a document meant to
ensure continuity of care and is a communication
tool for all providers engaged in a patient’s current
and future care. Electronic health records may
facilitate information sharing, but that goal is
largely unrealized.
Modern patient records evolved with two pri-
mary goals: facilitating fiscal justification and
improving medical education. Early hospitals
established basic rules to track patient admissions,
diagnoses, and outcomes. The purpose was
largely bureaucratic: administrators used patient
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_160-1
tallies to justify expenditures. As far back as 1737,
Berlin surgeons were required to note patients’
conditions each morning and prescribe lunches
accordingly (e.g., soup was prescribed for patients
too weak to chew). The purpose, according to
Volker Hess and Sophie Ledebur, was helping
administrators track the hospital’s food costs and
had little bearing on actual patient care. In 1791,
according to Eugenia Siegler in her analysis of
early medical recordkeeping, the New York Board
of Governors required complete patient logs along
with lists of prescribed medications, but no
descriptions of the patients’ conditions. Formally
documenting the care that individual patients
received was fairly uncommon in American hos-
pitals at that time. It was not until the end of the
nineteenth century that American physicians
began recording the specifics of daily patient
care for all patients. Documentation in European
hospitals, by contrast, was much more complete.
From the mid-eighteenth century on, standardized
medical forms were widely used to record
patients’ demographic data, their symptoms, treat-
ments, daily events, and outcomes. By 1820, these
forms were collected in preprinted folders with
multiple graphs and tables (by contrast, American
hospitals would not begin using such forms until
the mid-1860s). Each day, physicians in training
were tasked with transcribing medical data into
meaningful narratives, describing patterns of dis-
ease progression. The resulting texts became valu-
able learning tools. Similar narratives were
complied by American physicians and used for
2 Patient Records
medical training as well. In 1805, Dr. David
Hosack had suggested recording the specifics of
particularly interesting cases, especially those
holding the greatest educational value for medical
students. The New York Board of Governors
agreed and mandated compiling summary reports
in casebooks. As Siegler noted, there were very
few reports written at first: the first casebook
spanned 1810–1834. Later, as physicians in train-
ing were required to write case reports in order to
be admitted to their respective specialties, the
number of documented cases grew. Eventually,
reports were required for all patients. The reports,
however, were usually written retrospectively and
in widely varying narrative styles.
Widespread use of templates in American hos-
pitals helped standardize patient records, but the
resulting quantitative data superseded narrative
content. By the start of the twentieth century,
forms guaranteed documentation of specific
tasks like physical exams, histories, orders, and
test results. Graphs and tables dominated patient
records and physicians’ narrative summaries
began disappearing. The freestyle narrative form
that had previously comprised the bulk of the
patient record allowed physicians to write as
much or as little as they wished. Templates left
little room for lengthy narratives, no more than a
few inches, so summary reports gave way to brief
descriptions of pertinent findings. As medical
technology advanced, according to Siegler, the
medical record became more complicated and
cumbersome with the addition of yet more forms
for reporting each new type of test (e.g., chemis-
try, hematology, and pathology tests). While most
physicians kept working notes on active patients,
these scraps of paper notating observations, daily
tasks, and physicians’ thoughts seldom made their
way into the official patient record. The official
record emphasized tests and numbers, as Siegler
noted, and this changed medical discourse: inter-
actions and care became more data driven. Care
became less about the totality of the patient’s
experience and the physician’s perception of
it. Nonetheless, patient records had become a
mainstay and they did help ensure continuity of
care. Despite early efforts at a unifying style,
however, the content of patient records still varied
considerably.
Although standardized forms ensured certain
events would be documented, there were no
methods to ensure consistency across documenta-
tions or between providers. Dr. Larry Weed pro-
posed a framework in 1964 to help standardize
recording medical care: SOAP notes. SOAP notes
are organized around four key areas: subjective
(what patients say), objective (what providers
observe, including vital signs and lab results),
assessment (diagnosis), and plan (prescribed treat-
ments). Other standardized approaches have been
developed since then. The most common charting
formats today, in addition to SOAP notes, include
narrative charting, APIE charting, focus charting,
and charting by exception. Narrative charting,
much as in the early days of patient
recordkeeping, involves written accounts of
patients’ conditions, treatments, and responses
and is documented in chronological order. Charts
include progress notes and flow sheets which are
multi-column forms for recording dates, times,
and observations that are updated every few
hours for inpatients and upon each subsequent
outpatient visit. They provide an easy-to-read
record of change over time; however their limited
space cannot take the place of more complete
assessments, which should appear elsewhere in
the patient record. APIE charting, similar to
SOAP notes, involves clustering patient notes
around assessment (both subjective and objective
findings), planning, implementation, and evalua-
tion. Focus charting is a more concise method of
inpatient recording and is organized by keywords
listed in columns. Providers note their actions and
patients’ responses under each keyword heading.
Charting by exception involves documenting only
significant changes or events using specially for-
matted flow sheets. Computerized charting, or
electronic health records (EHR), combines several
of the above approaches but proprietary systems
vary widely. Most hospitals and private practices
are migrating to EHRs, but the transition has been
expensive, difficult, and slower than expected.
The biggest challenges include interoperability
issues impeding data sharing, difficult-to-use
Patient Records
EHRs, and perceptions that EHRs interfere with
provider-patient relationships.
Today, irrespective of the charting format used,
patient records are maintained according to strict
guidelines. Several agencies publish
recommended guidelines including the American
Association of Nurses, the American Medical
Association (AMA), the Joint Commission of
Accreditation of Healthcare Organizations
(JCAHO), and the Centers for Medicare and Med-
icaid Services (CMS). Each regards the medical
record as a communication tool for everyone
involved in the patient’s current and future care.
The primary purpose of the medical record is to
identify the patient, justify treatment, document
the course of treatment and results, and facilitate
continuity of care among providers. Data stored in
patient records have other functions; aside from
ensuring continuity of care, data can be extracted
for evaluating the quality of care administered,
released to third-party payers for reimbursement,
and analyzed for clinical research and/or epidemi-
ological studies. Each agency’s charting guide-
lines require certain fixed elements in the patient
record: the patient’s name, address, birthdate,
attending physician, diagnosis, next of kin, and
insurance provider. The patient record also con-
tains physicians’ orders and progress notes, as
well as medication lists, X-ray records, laboratory
tests, and surgical records. Several agencies
require the patient’s full name, birthdate, and a
unique patient identification number appear on
each page of the record, along with the name of
the attending physician, date of visit or admission,
and the treating facility’s contact information.
Every entry must be legibly signed or initialed
and date/time stamped by the provider.
The medical record is a protected legal docu-
ment and because it could be used in a malpractice
case, charting takes on added significance. Incom-
plete, confusing, or sloppy patient records could
signal poor medical care to a jury, even in the
absence of medical incompetence. For that rea-
son, many malpractice insurers require additional
documentation above and beyond what profes-
sional agencies recommend. For example, pro-
viders are urged to: write legibly in permanent
ink, avoid using abbreviations, write only
3
objective/quantifiable observations and use quo-
tation marks to set apart patients’ statements, note
communication between all members of the care
team while documenting the corresponding dates
and times, document informed consent and
patient education, record every step of every pro-
cedure and medication administration, and chart
instances of patients’ noncompliance or lack of
cooperation. Providers should avoid writing over,
whiting out, or attempting to erase entries, even if
made in error – mistakes should be crossed
through with a single line, dated, and signed.
Altering a patient chart after the fact is illegal in
many states, so corrections should be made in a
timely fashion and dated/signed. Leaving blank
spaces on medical forms should be avoided as
well; if space is not needed for documenting
patient care, providers are instructed to draw a
line through the space or write “N/A.” The fol-
lowing should also be documented to ensure both
good patient care and malpractice defense: the
reason for each visit, chief complaint, symptoms,
onset and duration of symptoms, medical and
social history, family history, both positive and
negative test results, justifications for diagnostic
tests, current medications and doses, over-the-
counter and/or recreational drug use, drug aller-
gies, any discontinued medications and reactions,
medication renewals or dosage changes, treatment
recommendations and suggested follow-up or
specialty care, a list of other treating physicians,
a “rule-out” list of considered but rejected diag-
noses, final definitive diagnoses, and canceled or
missed appointments.
Patient records contain more data than ever
before because of professional guidelines,
malpractice-avoidance strategies, and the ease of
data entry many EHRs make possible. The result
is that providers are experiencing data overload.
Many have difficulty wading through mounds of
data, in either paper or electronic form, to discern
important information from insignificant attesta-
tions and results. While EHRs are supposed to
make searching for data easier, many providers
lack the needed skills and time to search for and
review patients’ medical records. Researchers
have found some physicians rely on their own
memories or ask patients about previous visits
4 Patient Records
instead of searching for the information them-
selves. Other researchers have found providers
have trouble quickly processing the amount of
quantitative data and graphs in most medical
records. Donia Scott and colleagues, for example,
found that providers given narrative summaries of
patient records culled from both quantitative and
qualitative data performed better on questions
about patients’ conditions than those providers
given complete medical records, and did so in
half the time. Their findings highlight the impor-
tance of narrative summaries that should be
included in patients’ records. There is a clear
need for balancing numbers with words in ensur-
ing optimal patient care.
Another important issue is ownership of and
access to patient records. For each healthcare pro-
vider and/or medical facility involved in a
patient’s care, there is a unique patient record
owned by that provider. With patients’ permis-
sion, those records are frequently shared among
providers. The Health Insurance Portability and
Accountability Act (HIPAA) protects the confi-
dentiality of patient data, but patients, guardians
or conservators of minor or incompetent patients,
and legal representatives of deceased patients may
request access to records. Providers in some states
can withhold records if, in the providers’ judg-
ment, releasing information could be detrimental
to patients’ well-being or cause emotional or men-
tal distress. In addition to HIPAA mandates, many
states have strict confidentiality laws restricting
the release of HIV test results, drug and alcohol
abuse treatment, and inpatient mental health
records. While HIPAA guarantees patient access
to their medical records, providers can charge
copying fees. Withholding records because a
patient cannot afford to pay for them is prohibited
in many states because it could disrupt the conti-
nuity of care. HIPAA also allows patients the right
to amend their medical records if they believe
mistakes have been made. While providers are
encouraged to maintain records in perpetuity,
there are not requirements that they do so. Given
the costs associated with data storage, both on
paper and electronically, many providers will
only maintain charts on active patients. Many
inactive patients, those who have not seen a
given provider in 8 years, will likely have their
records destroyed. Additionally, many retiring
physicians typically only maintain records for
10 years. Better data management capabilities
will inevitably change these practices in years
to come.
While patient records have evolved to ensure
continuity of patient care, many claim the current
form that records have taken facilitates billing
over communication concerns. Many EHRs, for
instance, are modeled after accounting systems:
providers’ checkbox choices of diagnoses and
tests are typically categorized and notated in bill-
ing codes. Standardized forms are also designed
with billing codes in mind. Diagnosis codes are
reported in the International Statistical Classifica-
tion of Diseases and Related Health Problems
terminology, commonly referred to as ICD. The
World Health Organization maintains this coding
system for epidemiological, health management,
and research purposes. Billable procedures and
treatments administered in the United States are
reported in Current Procedural Terminology
(CPT) codes. The AMA owns this coding schema
and users must pay a yearly licensing fee for the
CPT codes and codebooks, which are updated
annually. Critics claim this amounts to a monop-
oly, especially given HIPAA, CMS, and most
insurance companies require CPT-coded data to
satisfy reporting requirements and for reimburse-
ment. CPT-coded data may impact patients’ abil-
ity to decipher and comprehend their medical
records, but the AMA does have a limited search
function on its website for non-commercial use
allowing patients to look up certain codes.
Patient records are an important tool ensuring
continuity of care, but data-heavy records are
cumbersome and often lacking narrative summa-
ries which have been shown to enhance providers’
understanding of patients’ histories and inform
better medical decision-making. Strict guidelines
and malpractice concerns produce thorough
records that while ensuring complete documenta-
tion, sometimes impede providers’ ability to dis-
cern important from less significant past findings.
Better search and analytical tools are needed for
managing patient records and data.
Patient Records
Cross-References
▶ Electronic Health Records (EHR)
▶ Health Care Delivery
▶ Health Informatics
▶ Medical/Health Care
▶ Patient-Centered (Personalized) Health
Further Reading
American Medical Association. CPT – current proce-
dural terminology. http://www.ama-assn.org/ama/
pub/physician-resources/solutions-managing-your-prac
tice/coding-billing-insurance/cpt.page. Accessed October
2014.
Christensen, T., & Grimsmo, A. (2008). Instant availability
of patient records, but diminished availability of patient
5
information: A multi-method study of GP’s use of
electronic health records. BMC Medical Informatics
and Decision Making, 8(12).
Hess, V., & Ledebur, S. (2011). Taking and keeping: A note
on the emergence and function of hospital patient
records. Journal of the Society of Archivists, 32, 1.
Lee, J. Interview with Lawrence Weed, MD – The father of
the problem-oriented medical record looks ahead.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2911807/.
Accessed October 2014.
Medical Insurance Exchange of California. Medical
record documentation for patient safety and physi-
cian defensibility. http://www.miec.com/Portals/0/
pubs/MedicalRec.pdf. Accessed October 2014.
Scott, D., et al. (2013). Data-to-text summarisation of
patient records: Using computer-generated summaries
to access patient histories. Patient Education and
Counseling, 92.
Siegler, E. (2010). The evolving medical record. Annals of
Internal Medicine, 153.
P
Patient-Centered (Personalized)
Health
Barbara Cook Overton
Southeastern Louisiana University, Baton Rouge,
LA, USA
Patient-centered health privileges patient partici-
pation and results in tailored interventions incor-
porating patients’ needs, values, and preferences.
Although this model of care is preferred by
patients and encouraged by policy makers, many
healthcare providers persist in using a biomedical
approach which prioritizes providers’ expertise
and downplays patients’ involvement. Patient-
centered care demands collaborative partnerships
and quality communication, both requiring more
time than is generally available during medical
exams. While big data may not necessarily
improve patient-provider communication, it can
facilitate individualized care in several
important ways.
The concept of patient-centered health,
although defined in innumerable ways, has gained
momentum in recent years. In 2001, the Institute
of Medicine (IOM) issued a report recommending
healthcare institutions and providers adopt six
basic tenets: safety, effectiveness, timeliness, effi-
ciency, equity, and patient-centeredness. Patient-
centeredness, according to the IOM, entails deliv-
ering quality health care driven by patients’ needs,
values, and preferences. The Institute for Patient-
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_161-1
and Family-Centered Care expands the IOM def-
inition by including provisions for shared
decision-making, planning, delivery, and evalua-
tion of health care that is situated in partnerships
comprising patients, their families, and providers.
The concept is further elucidated in terms of four
main principles: respect, information sharing, par-
ticipation, and collaboration. According to the
Picker Institute, patient-centered care encom-
passes seven basic components: respect, coordi-
nation, information and education, physical
comfort, emotional support, family involvement,
and continuity of care. All of the definitions basi-
cally center on two essential elements: patient
participation in the care process and
individualized care.
The goal of patient-centered care, put forth by
the IOM, is arguably a return to old-fashioned
medicine. Dr. Abraham Flexner, instrumental in
revamping physician training during the 1910s
and 1920s, promoted medical interactions that
were guided by both clinical reasoning and com-
passion. He encouraged a biopsychosocial
approach to patient communication, which incor-
porates patients’ feelings, thoughts, and expecta-
tions. Scientific and technological advances
throughout the twentieth century, however, grad-
ually shifted medical inquiry away from the whole
person and towards an ever-narrowing focus on
symptoms and diseases. Once the medical inter-
view became constricted, scientific, and objective,
collaborative care gave way to a provider-driven
approach. The growth of medical specialties (like
2 Patient-Centered (Personalized) Health
cardiology and gastroenterology) further
compounded the problem by reducing patients to
collections of interrelated systems (such as circu-
latory and digestive). This shift to specialty care
coincided with fewer providers pursuing careers
in primary care, the specialty most inclined to
adopt a patient-centered perspective. The
resulting biomedical model downplays patient
participation while privileging provider control
and expertise. Although a return to patient-
centered care is being encouraged, many pro-
viders persist in using a biomedical approach.
Some researchers fault patients for not actively
co-constructing the medical encounter, while
others blame medical training that
de-emphasizes relationship development and
communication skills.
Several studies posit quality communication as
the single most important component necessary
for delivering patient-centered care. Researchers
find patient dissatisfaction is associated with pro-
viders who are insensitive to or misinterpret
patients’ socio-emotional needs, fail to express
empathy, do not give adequate feedback or infor-
mation regarding diagnoses and treatment proto-
cols, and disregard patients’ input in decision-
making. Patients who are dissatisfied with pro-
viders’ communication are less likely to comply
with treatment plans and typically suffer poorer
outcomes. Conversely, patients satisfied with the
quality of their providers’ communication are
more likely to take medications as prescribed
and adhere to recommended treatments. Satisfied
patients also have lower blood pressure and better
overall health. Providers, however, routinely sac-
rifice satisfaction for efficiency, especially in man-
aged care contexts.
Many medical interactions proceed according
to a succinct pattern that does not prioritize
patients’ needs, values, and preferences. The
asymmetrical nature of the provider-patient rela-
tionship preferences providers’ goals and discour-
ages patient participation. Although patients
expect to have all or most of their concerns
addressed, providers usually pressure them to
focus on one complaint per visit. Providers also
encourage patients to get to the point quickly,
which means patients rarely speak without
interruption or redirection. While some studies
note patients are becoming more involved in
their health care by offering opinions and asking
questions, others find ever-decreasing rates of
participation during medical encounters. Studies
show physicians invite patients to ask questions in
fewer than half of exams. Even when patients do
have concerns, they rarely speak up because they
report feeling inhibited by asymmetrical relation-
ships: many patients simply do not feel
empowered to express opinions, ask questions,
or assert goals. Understandably, communication
problems stem from these hierarchical differences
and competing goals, thereby making patient-
centered care difficult.
There are several other obstacles deterring
patient-centered communication and care. While
medical training prioritizes the development of
clinical skills over communication skills, lack of
time and insufficient financial reimbursement are
the biggest impediments to patient-centered care.
The “one complaint per visit” approach to health
care means most conversations are symptom spe-
cific, with little time left for discussing patients’
overall health goals. Visits should encompass
much broader health issues, moving away from
the problem presentation/treatment model while
taking each patient’s unique goals into account.
The goal of patient-centered care is further
compromised by payment structures incentivizing
quick patient turnaround over quality communi-
cation, which takes more time than is currently
available in a typical medical encounter. Some
studies, however, suggest that patient-centered
communication strategies, like encouraging ques-
tions, co-constructing diagnoses, and mutually
deciding treatment regimens, do not necessarily
lengthen the overall medical encounter. Further-
more, collaboratively decided treatment plans are
associated with decreased rates of hospitalization
and emergency room use. Despite the challenges
that exist, providers are implored to attempt
patient-centered communication.
Big data has helped facilitate asynchronous
communication between medical providers,
namely through electronic health records which
ensure continuity of care, but big data’s real prom-
ise lies elsewhere. Using the power of predictive
Patient-Centered (Personalized) Health
analytics, big data can play an important role in
advancing patient-centered health by helping
shape tailored wellness programs. The provider-
driven, disease-focused approach to health care
has, heretofore, impacted the kind of health data
that exist: data that are largely focused on patients’
symptoms and diseases. However, diseases do not
develop in isolation. Most conditions develop
through a complicated interplay of hereditary,
environmental, and lifestyle factors. Expanding
health data to include social and behavioral data,
elicited via a biopsychosocial/patient-centered
approach, can help medical providers build better
predictive models. By examining comprehensive
rather than disease-focused data, providers can,
for example, leverage health data to predict
which patients will participate in wellness pro-
grams, their level of commitment, and their poten-
tial for success. This can be done using data
mining techniques, like collaborative filtering. In
much the same way Amazon makes purchase
recommendations for its users, providers may
similarly recommend wellness programs by tak-
ing into account patients’ past behavior and health
outcomes. Comprehensive data could also be use-
ful for tailoring different types of programs based
on patients’ preferences, thereby facilitating
increased participation and retention. For exam-
ple, programs could be customized for patients
that go beyond traditional racial, ethnic, or socio-
demographic markers and include characteristics
such as social media use and shopping habits. By
designing analytics aimed at understanding indi-
vidual patients and not just their diseases, pro-
viders may better grasp how to motivate and
support the necessary behavioral changes
required for improved health.
The International Olympic Committee (IOC),
in a consensus meeting on noncommunicable dis-
ease prevention, has called for an expansion of
health data collected and a subsequent conversion
of that data into information providers and
patients may use to achieve better health out-
comes. Noncommunicable/chronic diseases,
such as diabetes and high blood pressure, are
largely preventable. These conditions are related
to lifestyle choices: too little exercise, an
unhealthy diet, smoking, and alcohol abuse. The
3
IOC recommends capturing data from pedometers
and sensors in smart phones, which provide
details about patients’ physical activity, and com-
bining that with data from interactive smart phone
applications (such as calorie counters and food
logs) to customize behavior counseling. This
approach individualizes not only patient care but
also education, prevention, and treatment inter-
ventions and advances patient-centered care with
respect to information sharing, participation, and
collaboration. The IOC also identifies several
other potential sources of health data: social
medial profiles, electronic medical records, and
purchase histories. Collectively, this data can
yield a “mass customization” of prevention pro-
grams. Given chronic diseases are responsible for
60 percent of deaths and 80 percent of healthcare
spending is dedicated to chronic disease manage-
ment, customizable programs have the potential to
save lives and money.
Despite the potential, big data’s impact are
largely unrealized in patient-centered care efforts.
Although merging social, behavioral, and medical
data to improve health outcomes has not hap-
pened on a widespread basis, there is still a lot
that can be done analyzing medical data alone.
There is, however, a clear need for computational/
analytical tools that can aid providers in recogniz-
ing disease patterns, predicting individual
patients’ susceptibility, and developing personal-
ized interventions. Nitesh Chawla and Darcy
Davis propose aggregating and integrating big
data derived from millions of electronic health
records to uncover patients’ similarities and con-
nections with respect to numerous diseases. This
makes a proactive medical model possible, as
opposed to the current treatment-based approach.
Chawla and Davis suggest that leveraging clini-
cally reported symptoms from a multitude of
patients, along with their health histories, pre-
scribed treatments, and wellness strategies, can
provide a summary report of possible risk factors,
underlying causes, and anticipated concomitant
conditions for individual patients. They devel-
oped an analytical framework called the Collabo-
rative Assessment and Recommendation Engine
(CARE), which applies collaborative filtering
using inverse frequency and vector similarity to
4 Patient-Centered (Personalized) Health
generate predictions based on data from similar
patients. The model was validated using a Medi-
care database of 13 million patients with two
million hospital visits over a 4-year period by
comparing diagnosis codes, patient histories, and
health outcomes. CARE generates a short list that
includes high-risk diseases and early warning
signs that a patient may develop in the future,
enabling a collaborative prevention strategy and
better health outcomes. Using this framework,
providers can improve the quality of care through
prevention and early detection and also advance
patient-centered health care.
Data security is a factor that merits discussion.
Presently, healthcare systems and individual pro-
viders exclusively manage patients’ health data.
Healthcare systems must comply with security
mandates set forth by the Health Insurance Porta-
bility and Accountability Act of 1996 (HIPAA).
HIPAA demands data servers are firewall and
password protected, and use encrypted data trans-
mission. Information sharing is an important com-
ponent of patient-centered care. Some proponents
of the patient-centered care model advocate trans-
ferring control of health data to patients, who may
then use and share it as they see fit. Regardless as
to who maintains control of health data, storing
and electronically transferring that data pose
potential security and privacy risks.
Patient-centered care requires collaborative
partnerships and wellness strategies that incorpo-
rate patients’ thoughts, feelings, and preferences.
It also requires individualized care, tailored to
meet patients’ unique needs. Big data facilitates
patient-centered/individualized care in several
ways. First, it ensures continuity of care and
enhanced information sharing through integrated
electronic health records. Second, analyzing pat-
terns embedded in big data can help predict dis-
ease. APACHE III, for example, is a prognostic
program that predicts hospital inpatient mortality.
Similar programs help predict the likelihood of
heart disease, Alzheimer’s, cancer, and digestive
disorders. Lastly, big data accrued from not only
patients’ health records but from their social
media profiles, purchase histories, and
smartphone applications have the potential to pre-
dict enrollment in wellness programs and improve
behavioral modification strategies thereby
improving health outcomes.
Cross-References
▶ Biomedical Data
▶ Electronic Health Records (EHR)
▶ Epidemiology
▶ Health Care Delivery
▶ Health Infomatics
▶ HIPAA
▶ Medical/Health Care
▶ Predictive Analytics
Further Readings
Chawla, N. V., & Davis, D. A. (2013). Bringing big data to
personalized healthcare: A patient-centered frame-
work. Journal of General Internal Medicine, 28(3),
660–665.
Duffy, T. P. (2011). The Flexner report: 100 years later. Yale
Journal of Biology and Medicine, 84(3), 269–276.
Institute of Medicine. (2001). Crossing the quality chasm.
Washington, DC: National Academies Press.
Institute for Patient- and Family-Centered Care. FAQs.
http://www.ipfcc.org/faq.html. Accessed Oct 2014.
Matheson, G., et al. (2013). Prevention and management of
non-communicable disease: The IOC consensus state-
ment, Lausanne 2013. Sports Medicine, 43,
1075–1088.
Picker Institute. Principles of patient-centered care. http://
pickerinstitute.org/about/picker principles/. Accessed
Oct 2014.
P
PatientsLikeMe
Niccolò Tempini
Department of Sociology, Philosophy and
Anthropology and Egenis, Centre for the Study
of the Life Sciences, University of Exeter,
Exeter, UK
Introduction
PatientsLikeMe is a for-profit organization based
in Cambridge, Massachusetts, managing a social
media-based health network that supports patients
in activities of health data self-reporting and
socialization. As of January 2015, the network
counts more than 300,000 members and 2,300+
associated conditions and it is one of the most
established networks in the health social media
space. The web-based system is designed and
managed to encourage and enable patients to
share data about their health situation and
experience.
Business Model
Differently from most prominent social media
sites, the network is not ad-supported. Instead, the
business model centers on the sale of anonymized
data access and medical research services to com-
mercial organizations (mostly pharmaceutical
companies). The organization has been partnering
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_162-1
with clients, in order to develop patient communi-
ties targeted on a specific disease, or kind of patient
experience. In the context of a sponsored project,
PatientsLikeMe staff develop disease-specific tools
required for patient health self-reporting (Patient-
reported outcome measures – PROMs) on a web-
based platform, then collect and analyze the patient
data, and produce research outputs, either commer-
cial research reports or peer-reviewed studies.
Research has regarded a wide range of issues,
from drug efficacy discovery for neurodegenera-
tive diseases, or symptom distribution across
patient populations, to sociopsychological issues
like compulsive gambling.
While the network has produced much of its
research in occasion of sponsored research pro-
jects, this has mostly been discounted from criti-
cism. This because, for its widespread involvement
of patients in medical research, PatientsLikeMe is
often seen as a champion of the so-called partici-
patory turn in medicine, the issue of patient
empowerment and more generally of the forces of
democratization that several writers argued to be
promise of the social web. While sustaining its
operations through partnerships with commercial
corporations, PatientsLikeMe also gathers on the
platform a number of patient-activism NGOs. The
system provides them customized profiles and
communication tools, with which these organiza-
tions can try to improve the reach with the patient
population of reference, while the network in
return gains a prominent position as the center, or
enabler, of health community life.
2 PatientsLikeMe
Patient Members
PatientsLikeMe attracts patient members because
the system is designed to allow patients to find
others and socialize. This can be particularly use-
ful for patients of rare, chronic, or life-changing
diseases: patient experiences for which an indi-
vidual might feel helpful to learn from the expe-
rience of others, whom however might be not easy
to find through traditional, “offline” socialization
opportunities. The system is also designed to
enable self-tracking of a number of health dimen-
sions. The patients record both structured data,
about diagnoses, treatments, symptoms, disease-
specific patient-reported questionnaires (PROs),
or results of specific lab test, and semi-structured
or unstructured data, in the form of comments,
messages, and forum posts. All of these data are
at the disposal of the researchers that have access
to the data. A paradigmatic characteristic of
PatientsLikeMe as social media research network
is that the researchers do not learn about the
patients in any other way than through the data
that the patients share.
Big Data and PatientsLikeMe
As such, it is the approach to data and to research
that defines PatientsLikeMe as a representative
“Big Data” research network – one that, however,
does not manage staggeringly huge quantities of
data nor employs extremely complex technologi-
cal solutions for data storage and analysis.
PatientsLikeMe is a big data enterprise because,
first, it approaches medical research through an
open (to data sharing by anyone and about user-
defined medical entities), distributed (relative to
availability of a broadband connection, from any-
where and at anytime), and data-based (data are all
that is transacted between the participating
parties) research approach. Second, the data used
by PatientsLikeMe researchers are highly varied
(including social data, social media user-
generated content, browsing session data, and
most importantly structured and unstructured
health data) and relatively fast, as they are
updated, parsed, and visualized dynamically in
real time through the website or other data-
management technologies. The research process
involves practices of pattern detection, analysis of
correlations, and investigation of hypotheses
through regression and other statistical
techniques.
The vision of scientific discovery that is under-
lying the PatientsLikeMe project is one based on
the assumption that given a broad enough base of
users and a granular, frequent and longitudinal
exercise of data collection, new, small patterns
ought to emerge from the data and invite further
investigation and explanation. This assumption
implies that for medical matters to be discovered
further, the development of an open, distributed
and data-based socio-technical system that is
more sensitive to their forms and differences is a
necessary step. But also, the hope is that important
lessons can be learned by opening the medical
framework to measure and represent a broader
collection of entities and events than traditional,
profession-bound medical practice accepted. The
PatientsLikeMe database includes symptoms and
medical entities as described in the terms used by
the patients themselves. This involves sensitive
and innovative processes of translation from the
patient language to expert terminology. Questions
about the epistemological consequence of the
translation of the patient voice (until now a
neglected form of medical information) over
data fields and categories, and the associated con-
cerns about reliability of patient-generated data,
cannot have a simple answer. In any case, from a
practice-based point of view these data are none-
theless being mobilized for research through inno-
vative technological solutions for coordinating
the patient user-base. The data can then be ana-
lyzed in multiple ways, all of which include the
use of computational resources and databases –
given the digital nature of the data.
As ethnographic research of the organization
has pointed out (see further readings section,
below), social media companies that try to
develop knowledge from the aggregation and
analysis of the data contributed by their patients
are involved in complex efforts to “cultivate” the
information lying in the database – as they have to
come to grips with the dynamics and trade-offs
PatientsLikeMe
that are specific to understanding health through
social media. Social media organizations try to
develop meaningful and actionable information
from their database by trying to make data struc-
tures more precise in differentiating between phe-
nomena and reporting about them in data records,
and make the system easier and flexible in use in
order to generate more data. Often these demands
work at cross-purposes. The development of
social media for producing new knowledge
through distributed publics involves the engineer-
ing of social environment where sociality and
information production are inextricably
intertwined. Users need to be steered towards
information-productive behaviors as they engage
in social interaction of sorts, for information is the
worth upon which social media businesses
depend. In this respect, it has been argued that
PatientsLikeMe is representative of the construc-
tion of sociality that takes place in all social media
sites, where social interaction unfolds along the
paths that the technology continuously and
dynamically draws based on the data that the
users are sharing.
As such, many see PatientsLikeMe as incarnat-
ing an important dimension of the much-expected
revolution of personalized medicine. Improve-
ments in healthcare will not be limited to a capil-
lary application of genetic sequencing and other
micro and molecular biology tests that try to open
up the workings of individual human physiology
at unprecedented scale, instead the information
produced by these tests will often the related
with the information about the subjective patient
experience and expectations that new information
technology capabilities are increasingly making
possible.
Other Issues
Much of the public debate about the
PatientsLikeMe network involves issues of pri-
vacy and confidentiality of the patient users. The
network is a “walled garden,” with patient profiles
remaining inaccessible to unregistered users by
default. However, once logged in, every user can
browse all patient profiles and forum
3
conversations. In more than one occasion,
unauthorized intruders (including journalists and
academics) were detected and found screen-
scraping data from the website. Despite the orga-
nization employing state-of-the-art techniques to
protect patient data from unauthorized exporting,
any sensitive data shared on a website remains at a
risk, given the widespread belief – and public
record on other websites and systems – that skilled
intruders could always execute similar exploits
unnoticed. Patients can have a lot to be concerned
about, especially if they have conditions with a
social stigma or if they shared explicit political or
personal views in the virtual comfort of a forum
room. In this respect, even if the commercial pro-
jects that the organization has undertaken with
industry partners implied the exchange of user
data that had been pseudonymised before being
handed over, the limits of user profile
anonymization are well known. In the case of
profiles of patients living with rare diseases,
which are a consistent portion of the users in
PatientsLikeMe, it can arguably be not too diffi-
cult to reidentify individuals, upon determined
effort. These issues of privacy and confidentiality
remain a highly sensitive topic as society does not
dispose of standard and reliable solutions against
the various forms that data misuse can take. As
both news and scholars have often reported, the
malleability of digital data makes it impossible to
stop the diffusion of sensitive data once that func-
tion creep happens.
Moreover, as it is often discussed in the social
media and big data public debate, data networks
increasingly put pressure on the notion of
informed consent as an ethically sufficient device
for conducting research with user and patient data.
The need for moral frameworks of operation that
overperform over strict compliance with law has
often been called for, and recently by the report on
data in biomedical research by the Nuffield Coun-
cil for Bioethics. In the report, PatientsLikeMe
was held as a paramount example of new kinds
of research networks that rely on extensive patient
involvement and social (medical) data – these
networks are often dubbed as citizen science or
participatory research.
4 PatientsLikeMe
On another note, some have argued that
PatientsLikeMe, as many other prominent social
media organizations, has been exploiting the rhe-
toric of sharing (one’s life with a network and its
members) to encourage data-productive behav-
iors. The business model of the network is built
around a traditional, proprietary model of data
ownership. The network facilitates the data flow
inbound and makes it less easy for the data to flow
outbound, controlling their commercial applica-
tion. In this respect, we must notice that the cur-
rent practice in social media management in
general is often characterized by data sharing
evangelism by the managing organization, which
at the same time requires monopoly of the most
important data resources that the network gener-
ates. In the general public debate, this kind of
social media business model has been linked as a
factor contributing to the erosion of user privacy.
On a different level, one can notice how the
kind of patient-reported data collection and med-
ical research that the network makes possible to
perform is a much cheaper and under many
respects more efficient model than what the
professional-laden institutions such as the clinical
research hospital, with their specific work loci and
customs, could put in place. This way of
organising the collection of valuable data operates
by including large amounts of end users who are
not remunerated. Despite this, running and orga-
nizing such an enterprise is expensive and labor-
intensive and as such, critical analysis of this kind
of “crowdsourcing” enterprise needs to look
beyond the more superficial issue of the absence
of a contract to sanction the exchange of a mone-
tary reward for distributed, small task perfor-
mances. One connected problem in this respect
is that since data express their value only when
they are re-situated through use, no data have a
distinct, intrinsic value upon generation; not all
data generated will ever be equal.
Finally, the affluence of medical data that this
network makes available can have important con-
sequences on therapy or lifestyle decisions that a
patient might take. Sure, patients can make up
their mind and take critical decisions without
appropriate consultation at any time, as they
have always done. Nonetheless, the sheer amount
of information that networks such as
PatientsLikeMe or search engines such as Google
make available at a click’s distance is without
antecedents and what this implies for healthcare
must still be fully understood. Autonomous deci-
sions by the patients do not necessarily happen for
the worst. As healthcare often falls short of pro-
viding appropriate information and counseling,
especially about everything that is not strictly
therapeutic, patients can eventually devise
improved courses of action, through a consulta-
tion of appropriate information-rich web
resources. At the same time, risks and harms are
not fully appreciated and there is a pressing need
to understand more on the consequences of these
networks for individual health and the future of
healthcare and health research.
There are other issues besides these more evi-
dent and established topics of discussion. As it has
been pointed out, questions of knowledge transla-
tion (from the patient vocabulary to the clinical-
professional) remain open, and unclear is also the
capacity of these distributed and participative net-
works to consistently represent and organize the
patient populations that they are deemed to serve,
as the involvement of patients is however limited
and relative to specific tasks, most often of data-
productive character. The afore-mentioned issues
are not exhaustive nor exhausted in this essay.
They require in-depth treatment; with this intro-
duction the aim has been to give a few coordinates
on how to think about the subject.
Further Readings
Angwin, J. (2014). Dragnet nation: A quest for privacy,
security, and freedom in a world of relentless surveil-
lance. New york: Henry Holt and Company.
Arnott-Smith, C., & Wicks, P. (2008). PatientsLikeMe:
Consumer health vocabulary as a folksonomy. Ameri-
can Medical Informatics Association Annual Sympo-
sium Proceedings, 2008, 682–686.
Kallinikos, J., & Tempini, N. (2014). Patient data as med-
ical facts: Social media practices as a foundation for
medical knowledge creation. Information Systems
Research, 25, 817–833. doi:10.1287/isre.2014.0544.
Lunshof, J. E., Church, G. M., & Prainsack, B. (2014).
Raw personal data: Providing access. Science, 343,
373–374. doi:10.1126/science.1249382.
PatientsLikeMe
Prainsack, B. (2013). Let’s get real about virtual: Online
health is here to stay. Genetical Research, 95, 111–113.
doi:10.1017/S001667231300013X.
Richards, M., Anderson, R., Hinde, S., Kaye, J., Lucassen,
A., Matthews, P., Parker, M., Shotter, M., Watts, G.,
Wallace, S., & Wise, J. (2015). The collection, linking
and use of data in biomedical research and health care:
Ethical issues. London: Nuffield Council on Bioethics.
Tempini, N. (2014). Governing social media: Organising
information production and sociality through open,
distributed and data-based systems (Doctoral disserta-
tion). School of Economics and Political Science,
London.
Tempini, N. (2015). Governing PatientsLikeMe: Informa-
tion production and research through an open,
5
distributed and data-based social media network. The
Information Society, 31, 193–211.
Wicks, P., Vaughan, T. E., Massagli, M. P., & Heywood,
J. (2011). Accelerated clinical discovery using self-
reported patient data collected online and a patient-
matching algorithm. Nature Biotechnology, 29,
411–414. doi:10.1038/nbt.1837.
Wyatt, S., Harris, A., Adams, S., & Kelly, S. E. (2013).
Illness online: Self-reported data and questions of trust
in medical and social research. Theory Culture & Soci-
ety., 30, 131–150. doi:10.1177/0263276413485900.
Zuboff, S. (2015). Big other: surveillance capitalism and
the prospects of an information civilization. Journal of
Information Technology, 30, 75–89.
P
Pharmaceutical Industry
Janelle Applequist
The Zimmerman School of Advertising and Mass
Communications, University of South Florida,
Tampa, FL, USA
Globally, the pharmaceutical industry is worth
more than $1 trillion, encompassing one of the
world’s most profitable industries, focusing on the
development, production, and marketing of pre-
scription drugs for use by patients. Over one-third
of the pharmaceutical industry is controlled by
just ten companies, with six of these companies
in the United States alone. The World Health
Organization has reported an inherent conflict of
interest between the pharmaceutical industry’s
business goals and the medical needs of the pub-
lic, attributable to the fact that twice the amount is
spent on promotional spending (including adver-
tisements, marketing, and sales representation)
than is on the research and development for future
prescription drugs needed for public health
efforts. The average pharmaceutical company in
the United States sees a profit of greater than $10
billion annually, while pharmaceutical companies
contribute 50 times more spending on promoting
and advertising for their own products than spend-
ing on public health information initiatives.
Big data can be described as the collection,
manipulation, and analysis of massive amounts
of data – and the decisions made from that
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_163-1
analysis. Having the ability to be described as
both a problem and an opportunity, big data and
its techniques are continuing to be utilized in
business by thousands of major institutions. The
sector of health care is not immune to massive
data collection efforts, and pharmaceuticals in
particular comprise an industry that relies on
aggregating information.
Literature on data mining in the pharmaceuti-
cal industry generally points to a disagreement
regarding the intended use of health-care informa-
tion. On the one hand, historically, data mining
techniques have proved useful for the research
and development (R&D) of current and future
prescription drugs. Alternatively, continuing con-
sumerist discourses in health care that have posi-
tion the pharmaceutical industry as a massive and
successful corporate entity have acknowledged
how this data is used to increase business sales,
potentially at the cost of patient confidentiality
and trust.
History of Data Mining Used for
Pharmaceutical R&D
Proponents of data mining in the pharmaceutical
industry have cited its ability to aide in: organiz-
ing information pertaining to genes, proteins, dis-
eases, organisms, and chemical substances,
allowing predictive models to be built for analyz-
ing the stages of drug development; keeping track
of adverse effects of drugs in a neural network
2 Pharmaceutical Industry
during clinical trial stages; listing warnings and
known reactions reported during the post-drug
production stage; forecasting new drugs needed
in the marketplace; providing inventory control
and supply chain management information; and
managing inventories. Data mining was first used
in the pharmaceutical industry as early as the
1960s alongside the increase in prescription drug
patenting. With over 1,000 drug patents a year
being introduced at that time, data collection
assisted pharmaceutical scientists in keeping up
with patents being proposed. At this time, infor-
mation was collected and published in an edito-
rial-style bulletin categorized according to areas
of interest in an effort to make relevant issues for
scientists easier to navigate. Early in the 1980s,
technologies allowed biological sequences to be
identified and stored, such as the Human Genome
Project, which led to the increased use and pub-
lishing of databanks. Occurring alongside the
popularity of personal computer usage, bioinfor-
matics was born, which allowed biological
sequence data to be used for discovering and
studying new prescription drug targets. Ten
years later, in the 1990s, microarray technology
developed, posing a problem for data collection,
as this technology permitted the simultaneous
measurement of large numbers of genes and col-
lection of experimental data on a large scale. As
the ability to sequence a genome occurred in the
2000s, the ability to manage large levels of raw
data was still maturing, creating a continued prob-
lem for data mining in the pharmaceutical indus-
try. As the challenges presented for data mining in
relation to R&D have continued to increase since
the 1990s, the opportunities for data mining in
order to increase prescription drug sales have
steadily grown.
Data Mining in the Pharmaceutical
Industry as a Form of Controversy
Since the early 1990s, health-care information
companies have been purchasing the electronic
records of prescriptions from pharmacies and
other data collection resources in order to
strategically link this information with specific
physicians.
Prescription tracking refers to the collection of
data from prescriptions as they are filled at phar-
macies. When a prescription gets filled, data
miners are able to collect: the name of the drug,
the date of the prescription, and the name or
licensing number of the prescribing physician.
Yet, it is simple for the prescription drug industry
to identify specific physicians through protocol in
place by the American Medical Association
(AMA). The AMA has a “Physician Masterfile”
that includes all US physicians, whether or not
they belong to the AMA, and this file allows the
physician licensing numbers collected by data
miners to be connected to a name. Information
distribution companies (such as IMS Health, Den-
drite, Verispan, Wolters Kluwer, etc.) purchase
records from pharmacies. What many consumers
do not realize is that most pharmacies have these
records for sale and are able to do so legally by not
including patient names and only providing a
physician’s state licensing number and/or name.
While pharmacies cannot release a patient’s name,
they can provide data miners with a patient’s age,
sex, geographic location, medical conditions, hos-
pitalizations, laboratory tests, insurance copays,
and medication use. This has caused a significant
area of concern on behalf of patients, as it not only
may increase instances of prescription detailing,
but it may compromise the interests of patients.
Data miners do not have access to patient names
when collected prescription data; however, data
miners assign unique numbers to individuals so
that future prescriptions for the patient can be
tracked and analyzed together. This means that
data miners can determine: how long a patient
remains on a drug, whether the drug treatment is
continued, and which new drugs become pre-
scribed for the patient.
As information concerning a patient’s health is
highly sensitive, data mining techniques used by
the pharmaceutical industry have perpetuated the
notion that personal information carries a substan-
tial economic value. By data mining companies
paying pharmacies to extract prescription drug
information, the relationships between patients
and their physicians and/or pharmacists is being
Pharmaceutical Industry
exploited. The American Medical Association
(AMA) established the Physician Data Restriction
Program in 2006, giving any physician the oppor-
tunity to opt out from data mining initiatives. To
date, no such program for patients exists that
would give them the opportunity to have their
records removed from data collection procedures
and subsequent analyses. Three states have
enacted statutes that do not permit data mining
of prescription records. The Prescription Confi-
dentiality Act of 2006 in New Hampshire was the
first state to decide that prescription information
could not be sold or used for any advertising,
marketing, or promotional purposes. However, if
the information is de-identified, meaning that the
physician and patient names cannot be accessed,
then the data can be aggregated by geographical
region or zip code, meaning that data mining
companies could still provide an overall, more
generalized report for small geographic areas but
could not target specific physicians. Maine and
Vermont have statutes that limit the presence of
data mining. Physicians in Maine can register with
the state to prevent data mining companies from
obtaining their prescribing records. Data miners in
Vermont must obtain consent from the physician
for which they are analyzing prior to using “pre-
scriber-identifiable” information for marketing or
promotional purposes.
The number one customer for information dis-
tribution companies is the pharmaceutical indus-
try, which purchases the prescribing data to
identify the highest prescribers and also to track
the effects of their promotional efforts. Physicians
are given a value, a ranking from one to ten, which
identifies how often they prescribe drugs. A sales
training guide for Merck even states that this value
issued to identify which products are currently in
favor with the physician in order to develop a
strategy to change those prescriptions into Merck
prescriptions. The empirical evidence provided by
information distribution companies offers a
glimpse into the personality, behaviors, and
beliefs of a physician, which is why these num-
bers are so valued by the drug industry.
By collecting and analyzing this data, pharma-
ceutical sales representatives are able to better
target their marketing activities toward
3
physicians. For example, as a result of data mining
in the pharmaceutical industry, pharmaceutical
sales representatives could: determine which phy-
sicians are already prescribing specific drugs in
order to reinforce already-existent preferences, or,
could learn when a physician switches from a drug
to a competing drug, so that the representative can
attempt to encourage the physician to switch back
to the original prescription.
The Future of Data Mining in the
Pharmaceutical Industry
As of 2013, only 18% of pharmaceutical compa-
nies work directly with social media to promote
their prescription drugs, but this number is
expected to increase substantially in the next
year. As more individuals tweet about their med-
ical concerns, symptoms, the drugs they take, and
respective side effects, pharmaceutical companies
have noticed that social media has become an
integrated part of personalized medicine for indi-
viduals. Pharmaceutical companies are already in
the process of hiring data miners to collect and
analyze various forms of public social media in an
effort to: discover unmet needs, recognize new
adverse events, and determine what types of
drugs consumers would like to enter the market.
Based on the history of data mining used by
pharmaceutical corporations, it is evident that the
lucrative nature of prescription drugs serves as a
catalyst for data collection and analysis. By hav-
ing the ability to generalize what should be very
private information about patients for the pre-
scription drug industry, the use of data allows
prescription drugs to make more profit than ever,
as individual information can be commoditized to
benefit the bottom line of a corporation. Although
there are evident problems associated with pre-
scription drug data mining, the US Supreme Court
has continued to recognize that the pharmaceuti-
cal industry has a first amendment right to adver-
tise and solicit clients for goods and future
services. The Court has argued that legal safe-
guards, such as the Health Information Portability
and Accountability Act (HIPAA), are put in place
to combat the very concerns posed by practices
4 Pharmaceutical Industry
such as pharmaceutical industry data mining.
Additionally, the Court has found that by stripping
pharmaceutical records of patient information that
could lead to personal identification (e.g., name,
address, etc.), patients have their confidentiality
adequately protected. The law, therefore, leaves it
to the discretion of the physician to decide
whether they will associate with pharmaceutical
sales representatives and various data collection
procedures.
An ongoing element to address in analyzing
the pharmaceutical industry’s use of data mining
techniques will be the level of transparence used
with patients while utilizing the information col-
lected. Research shows that the majority of
patients in the United States are not only unfamil-
iar with data mining use by the pharmaceutical
industry, but that they are against any personal
information (e.g., prescription usage information
and personal diagnoses) being sold and shared
with outside entities, namely, corporations. As
health care continues to change in the United
States, it will be important for patients to under-
stand the ways in which their personal informa-
tion is being shared and used, in an effort to
increase national understandings of how privacy
laws are connected to the pharmaceutical industry.
Cross-References
▶ Electronic Health Records (EHR)
▶ Food and Drug Administration (FDA)
▶ Health Care Industry
▶ Patient Records
▶ Privacy
Further Readings
Altan, S., et al. (2010). Statistical considerations in design
space development. Pharmaceutical Technology, 34
(7), 66–70.
Fugh-Berman, A. (2008). Prescription tracking and public
health. Journal of General Internal Medicine, 23(8),
1277–1280.
Greene, J. A. (2007). Pharmaceutical marketing research
and the prescribing physician. Annals of Internal Med-
icine, 146(10), 742–747.
Klocke, J. L. (2008). Comment: Prescription records for
sale: Privacy and free speech issues arising from the
sale of de-identified medical data. Idaho Law Review,
44(2), 511536.
Orentlicher, D. (2010). Prescription data mining and the
protection of patients’ interests. The Journal of Law,
Medicine & Ethics, 38(1), 74–84.
Steinbrook, R. (2006). For sale: Physicians’ prescribing
data. The New England Journal of Medicine, 354(26),
2745–2747.
Wang, J., et al. (2011). Applications of data mining in
pharmaceutical industry. The Journal of Management
and Engineering Integration, 4(1), 120–128.
White paper: Big Data and the needs of the Pharmaceuti-
cal Industry. (2013). Philadelphia: Thomson Reuters.
World Health Organization. (2013). Pharmaceutical
Industry. Retrieved online from http://www.who.int/
trade/glossary/story073/en/.
P
Pollution, Air
Zerrin Savasan
Department of International Relations, Sub-
Department of International Law, Faculty of
Economics and Administrative Sciences, Selcuk
University, Konya, Turkey
The air contains many different substances, gases,
aerosols, particulate matter, trace metals, and a
variety of other compounds. If those are not at
the same concentration and change in space, and
over time to an extent that the air quality deterio-
rates, some contaminants or pollutant substances
exist in the air. The release of these air pollutants
causes harmful effects to both environment and
humans, to all organisms. This is regarded as air
pollution.
The air is a common/shared resource of all
human beings. After released, air pollutants can
be carried by natural events like winds, rains, and
so on. So, some pollutants, e.g., lead or chloro-
form, often contaminate more than one environ-
mental occasions, so, many air pollutants can also
be water or land pollutants. They can combine
with other pollutants and thus can undergo chem-
ical transformations, and then they can be eventu-
ally deposited on different locations. Their effects
can emerge in different locations far from their
main resources. Thus, they can detrimentally
affect upon all organisms on local or regional
scales and also upon the climate on global scale.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_167-1
Hence, concern for air pollution and its influ-
ences on the earth and efforts to prevent/and to
mitigate it have increased greatly in global scale.
However, today, it still stands as one of the pri-
mary challenges that should be addressed globally
on the basis of international cooperation. So, it
becomes necessary to promote the widespread
understanding on air pollution, its pollutants,
sources, and impacts.
Sources of Air Pollution
The air pollutants can be produced from natural-
based reasons (e.g., fires from burning vegetation,
forest fires, volcanic eruptions, etc.) or anthropo-
genic (human-caused) reasons. When outdoor
pollution – referring to the pollutants found in
outdoors – is thought, smokestacks of industrial
plants can be given as an example of human-made
ones. However, natural processes also produce
outdoor air pollution, e.g., volcanic eruptions.
The main causes of indoor air pollution, on the
other hand, again raise basically from human-
driven reasons, e.g., technologies used for
cooking, heating, and lighting. Nonetheless,
again there are also natural indoor air pollutants,
like radon, and chemical pollutants from building
materials and cleaning products.
Among those, human-based reasons, specifi-
cally after industrialization, have produced a vari-
ety of sources of air pollution, and thus more
contributed to the global air pollution. They can
2 Pollution, Air
emanate from point and nonpoint sources or from
mobile and stationary sources. A point source
describes a specific location from which large
quantities of pollutants are discharged, e.g., coal-
fired power plants. A nonpoint source, one the
other hand, is more diffuse often involving many
small pieces spread across a wide range of area,
e.g., automobiles. Automobiles are also known as
mobile sources, and the combustion of gasoline is
responsible for released emissions from mobile
sources. Industrial activities are also known as
stationary sources, and the combustion of fossil
fuels (coal) is accountable for their emissions.
These pollutants producing from distinct
sources may cause harm directly or indirectly. If
they are emitted from the source directly into the
atmosphere, and so cause harm directly, they are
called as primary pollutants, e.g., carbon oxides,
carbon monoxide, hydrocarbons, nitrogen oxides,
sulfur dioxide, particulate matter, and so on. If
they are produced from chemical reactions includ-
ing also primary pollutants in the atmosphere,
they are known as secondary pollutants, e.g.,
ozone and sulfuric acid.
The Impacts of Air Pollution
The air pollutants result in a wide range of impacts
both upon humans and environment. Their detri-
mental effects upon humans can be briefly sum-
marized as follows: health problems resulting
particularly from toxicological stress, like respi-
ratory diseases such as emphysema and chronic
bronchitis, chronic lung diseases, pneumonia, car-
diovascular troubles, and cancer, and immune
system disorders increasing susceptibility to
infection and so on. Their adverse effects upon
environment, on the other hand, are the following:
acid deposition, climate change resulting from the
emission of greenhouse gases, degradation of air
resources, deterioration of air quality, noise,
photooxidant formation (smog), reduction in the
overall productivity of crop plants, stratospheric
ozone (O3) depletion, threats to the survival of
biological species, etc.
While determining the extent and degree of
harm given by these pollutants, it becomes
necessary to know sufficiently about the features
of that pollutant. This is because some pollutants
can be the reason of environmental or health
problems in the air, they can be essential in the
soil or water, e.g., nitrogen is harmful as it can
form ozone in the air, and it is necessary for the
soil as it can also act beneficially as fertilizer in the
soil. Additionally, if toxic substances exist below
a certain threshold, they are not necessarily
harmful.
New Technologies for Air Pollution:
Big Data
Before the industrialization period, the compo-
nents of pollution are thought to be primarily
smoke and soot; but with industrialization, they
have been expanded to include a broad range of
emissions, including toxic chemicals and biolog-
ical or radioactive materials. Therefore, even
today it is still admitted that there are six conven-
tional pollutants (or criteria air pollutants) identi-
fied by the US Environmental Protection Agency
(EPA): carbon monoxide, lead, nitrous oxides,
ozone, particulate matter, and sulfur oxides.
Hence, it is expectable that there can be new
sources for air pollution and so new threats for
the earth soon. Indeed, very recently, through
Kigali (Rwanda) Amendment (14 October,
2016) to the Montreal Protocol adopted at the
Meeting of the Parties (MOP 28), it is accepted
to address hydrofluorocarbons (HFCs) – green-
house gases having a very high global warming
potential even if not harmful as much as CFCs and
HCFCs for the ozone layer under the Protocol – in
addition to chlorofluorocarbons (CFCs) and
hydrochlorofluorocarbons (HCFCs).
Air pollution first becomes an international
issue with the Trail Smelter Arbitration
(1941) between Canada and the United States.
Indeed, prior to the decision made by the Tribunal,
disputes over air pollution between two countries
had never been settled through arbitration. Since
this arbitration case – specifically with increasing
efforts since the early 1990s – attempts to mea-
sure, to reduce, and to address rapidly growing
impacts of air pollution have been continuing.
Pollution, Air
Developing new technologies, like Big Data,
arises as one of those attempts.
Big Data has no uniform definition (ELI 2014;
Keeso 2014; Simon 2013; Sowe and Zettsu 2014).
In fact, it is defined and understood in diverse
ways by different researchers (Boyd 2010; Boyd
and Crawford 2012; De Mauro et al. 2016; Gogia
2012; Mayer-Schönberger and Cukier 2013;
Manyika et.al 2011) and interested companies
like Experian, Forrester, Forte Wares, Gartner,
and IBM. It is initially identified by 3Vs – volume
(data amount), velocity (data speed), and variety
(data types and sources) (Laney 2001). By the
time, it has included fourth Vs like veracity (data
accuracy) (IBM) and variability (data quality of
being subject to structural variation) (Gogia 2012)
and a fifth V, value (data capability to turn into
value) together with veracity (Marr), and a sixth
one, vulnerability (data security-privacy)
(Experian 2016). It can be also defined by veracity
and value together with visualization (visual rep-
resentation of data) as additional 3Vs (Sowe and
Zettsu 2014) and also by volume, velocity, and
variety requiring specific technology and analyti-
cal methods for its transformation into value
(De Mauro et al. 2016). However, it is generally
referred as large and complex data processing
sets/applications that conventional systems are
not able to cope with them.
Because air pollution has various aspects that
should be measured as mentioned above, it
requires massive data that should be collected at
different spatial and temporal levels. Therefore, it
is observed in practice that Big Data sets and
analytics are increasingly used in the field of air
pollution, for monitoring, predicting its possible
consequences, responding timely to them, con-
trolling and reducing its impacts, and mitigating
the pollution itself.
They can be used by different kind of organi-
zations, such as governmental agencies, private
firms, and nongovernmental organizations
(NGOs). To illustrate, under US Environmental
Protection Agency (EPA), samples of Big Data
use include:
• Air Quality Monitoring (collaborating with
NASA on the DISCOVER-AQ initiative, it
3
involves research on Apps and Sensors for
Air Pollution (ASAP), National Ambient Air
Quality Standards (NAAQS) compliance, and
data fusion methods)
• Village Green Project (on improving Air Qual-
ity Monitoring and awareness in communities)
• Environmental Quality Index (EQI) (a dataset
consisting of an index of environmental quality
based on air, water, land, build environment,
and sociodemographic space)
There are also examples generated by local
governments like “E-Enterprise for the Environ-
ment,” by environmental organizations like “Per-
sonal Air Quality Monitoring,” or by citizen
science like “Danger Maps,” or by private firms
like “Aircraft Emissions Reductions” (ELI 2014)
or Green Horizons Project (IBM 2015).
The Environmental Performance Index (EPI) is
also another platform – using Big Data compiled
from a great number of sensors and models – pro-
viding a country and an issue ranking on how each
country manages environmental issues and also a
Data Explorer allowing users to investigate the
global data comparing environmental performance
with GDP, population, land area, or other variables.
Despite all, as the potential benefits and costs
of the use of Big Data are still under discussion
(Boyd 2010; Boyd and Crawford 2012; De Mauro
et al. 2016; Forte Wares, – ; Keeso 2014; Mayer-
Schönberger and Cukier 2013; Simon 2013; Sowe
and Zettsu 2014), various concerns can be raised
about the use of Big Data to monitor, measure, and
forecast air pollution as well. Therefore, it is
required to make further research to identify
gaps, challenges, and solutions for “making the
right data (not just higher volume) available to the
right people (not just higher variety) at the right
time (not just higher velocity)” (Forte Wares, ).
Cross-References
▶ Climate Change
▶ Environment
▶ Pollution, Land
▶ Pollution, Water
4 Pollution, Air
References
Boyd, Danah. Privacy and publicity in the context of big
data. WWW Conference. Raleigh, (2010). Retrieved
from http://www.danah.org/papers/talks/2010/
WWW2010.html. Accession 3 Feb 2017.
Boyd, Danah & Crawford, Kate. Critical questions for big
data, information, communication & society, 15(5),
662–679, (2012). Retrieved from: http://www.tandf
online.com/doi/abs/10.1080/1369118X.2012.678878.
Accession3 Feb 2017.
De Mauro, Andrea, Greco, Marco, Grimaldi, Michele.
A formal definition of big data based on its Essential
features. (2016). Retrieved from: https://www.
researchgate.net/publication/299379163_A_formal_
definition_of_Big_Data_based_on_its_essential_fea
tures. Accession 3 Feb 2017.
Environmental Law Institute (ELI). (2014). Big data and
environmental protection: An initial survey of public and
private initiatives. Washington, DC: Environmental Law
Institute. Retrieved from: https://www.eli.org/sites/
default/files/eli-pubs/big-data-and-environmental-pro
tection.pdf. Accession 3 Feb 2017.
Environmental Performance Index (EPI) (n.d.). Available
at: http://epi.yale.edu/. Accession 3 Feb 2017.
Experian. A data powered future. White Paper (2016).
Retrieved from: http://www.experian.co.uk/assets/
resources/white-papers/data-powered-future-2016.pdf.
Accession 3 Feb 2017.
Gartner. Gartner says solving ‘big data’ challenge involves
more than just managing volumes of data. June
27, 2011. (2011). Retrieved from: http://www.gartner.
com/newsroom/id/1731916. Accession 3 Feb 2017.
Gogia, Sanchit. The big deal about big data for customer
engagement, June 1, 2012, (2012). Retrieved from:
http://www.iab.fi/media/tutkimus-matskut/130822_
forrester_the_big_deal_about_big_data.pdf. Acces-
sion 3 Feb 2017.
IBM. IBM expands green horizons initiative globally to
address pressing environmental and pollution chal-
lenges. (2015). Retrieved from: http://www-03.ibm.
com/press/us/en/pressrelease/48255.wss. Accession
3 Feb 2017.
IBM (n.d.). What is big data? Retrieved from: https://
www-01.ibm.com/software/data/bigdata/what-is-big-
data.html. Accession 3 Feb 2017.
Keeso, Alan. Big data and environmental sustainability:
A conversation starter. Smith School Working Paper
Series, December 2014, Working paper 14-04, (2014).
Retrieved from: http://www.smithschool.ox.ac.uk/
library/working-papers/workingpaper%2014-04.pdf.
Accession 3 Feb 2017.
Laney, D. 3D data management: Controlling data volume,
velocity, and variety. Meta Group (2001). Retrieved
from: Available at: https://blogs.gartner.com/doug-
laney/files/2012/01/ad949-3D-Data-Management-Cont
rolling-Data-Volume-Velocity-and-Variety.pdf. Acces-
sion 3 Feb 2017.
Manyika, J. et al. Big data: The next frontier for innova-
tion, competition, and productivity. McKinsey
Global Institute (2011). Retrieved from: https://
file:///C:/Users/cassperr/Downloads/MGI_big_da-
ta_full_report.pdf. Accession 3 Feb 2017.
Marr, Bernard (n.d.). Big data: The 5 vs everyone must know.
Retrieved from: Available at: https://www.linkedin.
com/pulse/20140306073407-64875646-big-data-the-
5-vs-everyone-must-know. Accession 3 Feb 2017.
Mayer-Schönberger, V., & Cukier, K. (2013). Big data:
A revolution that will transform how We live, work and
think. London: John Murray.
Simon, P. (2013). Too big to ignore: The business case for
big data. Hoboken: Wiley.
Sowe, S. K. & Zettsu, K. “Curating big data made simple:
Perspectives from scientific communities.” Big Data, 2,
1. 23–33. Mary Ann Liebert, Inc. (2014).
Wares, F. (n.d.). Failure to launch: From big data to big
decisions why velocity, variety and volume is not
improving decision making and how to fix it. White
Paper. A Forte Consultancy Group Company.
Retrieved from http://www.fortewares.com/Administra
tor/userfiles/Banner/forte-wares–pro-active-reporting_
EN.pdf. Accession:3 Feb 2017.
Further Reading
Gillespie, A. (2006). Climate change, ozone depletion and
air pollution. Leiden: Martinus Nijhoff Publishers.
Gurjar, B. R., et al. (Eds.). (2010). Air pollution, health and
environmental impacts. Boca Raton: CRC Press.
Jacobson, M. Z. (2012). Air pollution and global warming.
New York: Cambridge University Press.
Louka, E. (2006). International environmental law,
fairness, effectiveness, and world order. New York:
Cambridge University Press.
Raven, P. H., & Berg, L. R. (2006). Environment. Danvers:
Wiley.
The Open University. (2007). T210-environmental control
and public health. Milton Keynes: The Open
University.
Vallero, D. A. (2008). Fundamentals of air pollution.
Amsterdam: Elsevier.
Vaughn, J. (2007). Environmental politics. Belmont:
Thomson Wadsworth.
Withgott, J., & Brennan, S. (2011). Environment. San
Francisco: Pearson.
P
Pollution, Land
Zerrin Savaşan
Department of International Relations, Sub-
Department of International Law, Faculty of
Economics and Administrative Sciences, Selçuk
University, Konya, Turkey
Pollution, in its all types (air, water, land), means
the entrance of some substances beyond the
threshold concentration level into the natural envi-
ronment which do not naturally belong there and
not present there, resulting in its destruction and
causing harmful effects on both humans/all living
organisms and the environment. So, in land pol-
lution as well, solid or liquid waste materials get
deposited on land and further degrade and deteri-
orate the quality and the productive capacity of
land surface. It is sometimes used as a substitute
of/or together with soil pollution where the upper
layer of the soil is destroyed. However, in fact, soil
pollution is just one of the causes of the land
pollution.
Like the other types, land pollution also arises
as a global environmental problem, specifically
associated with urbanization and industrialization,
that should be dealt with globally concerted envi-
ronmental policies. However, as a first and fore-
most step, it requires to be understood very well
with its all dimensions by all humankind, but
particularly the researchers studying on it.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_168-1
What Causes Land Pollution?
The degradation of land surfaces are caused
directly or indirectly by human (anthropogenic)
activities. It is possible to mention several reasons
temporally or permanently changing the land
structure and so causing land pollution. However,
three main reasons are generally identified as
industrialization, overpopulation, and urbaniza-
tion, and the others are counted as the reasons
stemming from these main reasons. Some of
them are as follows: improper waste disposal
(agricultural/domestic/industrial/solid/radioactive
waste) littering; mining polluting the land through
removing the topsoil which forms the fertile layer
of soil, or leaving behind waste products and the
chemicals used for the process; misuse of land
(deforestation, land conversion, desertification);
soil pollution (pollution on the topmost layer of
the land); soil erosion (loss of the upper (the most
fertile) layer of the soil); and the chemicals
(pesticides, insecticides, and fertilizers) applied
for crop enhancement on the lands.
Regarding these chemicals used for crop
enhancement, it should be underlined that, while
they are enhancing the crop yield, they can also
kill the insects, mosquitoes, and some other small
animals. So, they can harm the bigger animals that
feed on these tiny animals. In addition, most of
these chemicals can remain in the soil or accumu-
late there for many years. To illustrate, DDT
(dichlorodiphenyltrichloroethane) is one of these
pesticides. It is now widely banned with the great
2 Pollution, Land

effect of Rachel Carson’s very famous book,
Silent Spring (1962), which documents detrimen-
tal effects of pesticides on the environment, par-
ticularly on birds. Nonetheless, as it is not
ordinarily biodegradable, so known as persistent
organic pollutant, it has remained in the environ-
ment ever since it was first used.
This process threatens both these particular spe-
cies and also all the other species above and below
in the food chain. All these combining with the
massive extinctions of certain species – primarily
because of the disturbance of their habitat –
induce also massive reductions in biodiversity.

Control Measures for Land Pollution

Consequences of Land Pollution
Land pollution, along with other types of pollu-
All types of pollution are interrelated and their
tion, poses a threat to the sustainability of world
consequences cannot be restricted to the place
resources. However, while others can have self-
where the pollution is first discharged. This is
purification opportunities through the help of nat-
particularly because of the atmospheric deposition
ural events, it can stay as polluted till to be cleaned
in which existing pollution in the air (atmosphere)
up. Given the time necessary to pass for the dis-
creating pollution in water or land as well.
appearance of plastics in nature (hundreds of
Since they are interrelated to each other, their
years) and the radioactive waste (almost forever),
impacts are similar to each other as well. Like the
this fact can be understood better. So then land
others, land pollution has also serious conse-
pollution becomes one of the serious concerns of
quences on both humans, animals and other living
the humankind.
organisms, and environment. First of all, all living
When the question is asked what should be
things depend on the resources of the earth to
done to deal with it, first of all, it is essential to
survive and on the plants growing from the land,
remind that it is a global problem having no
so anything that damages or destroys the land
boundaries, so requires to be handled with collec-
ultimately has an impact on the survival of
tively. While working collectively, it is first of all
humankind itself and all other living things on
necessary to set serious environmental objectives
the earth. Damages on the land also lead to some
and best-practice measures. A wide range of
problems in relation to health like respiratory
measures – changing according to the cause of
problems, skin problems, and various kinds of
the pollution – can be thought to prevent, reduce,
cancers.
or stop land pollution, such as adopting and
Its effects on environment also require to take
encouraging organic farming instead of using
attention as it forms one of the most important
chemicals herbicides, and pesticides, restricting
reasons of the global warming which has started to
or forbidding their usage, developing the effective
be a very popular but still not adequately under-
methods of recycling and reusing of waste mate-
stood phenomena. This emerges from a natural
rials, constructing proper disposal of all wastes
circulation, in turn, land pollution leads to the
(domestic, industrials, etc.) into secured landfill
deforestation, it leads to less rain, eventually to
sites, and creating public awareness and support
problems such as the greenhouse effect and global
towards all environmental issues.
warming/climate change. Biomagnification is the
Apart from all those measures, the use of Big
other major concern stemming from land pollu-
Data technologies can also be thought as a way of
tion. It occurs when certain substances, such as
addressing rapidly increasing and wide-ranging
pesticides or heavy metals, gained through eating
consequences of land pollution.
by aquatic organisms such as fish, which in turn
Some of the cases in which Big Data technol-
are eaten by large birds, animals, or humans. They
ogies are used in relation to one or more aspects of
become concentrated in internal organs as they
land pollution can be illustrated as follows (ELI
move up the food chain, and then the concentra-
2014):
tion of these toxic compounds tends to increase.
Pollution, Land
• Located under US Department of the Interior
(DOI), the National Integrated Land System
(NILS) aims to provide the principal data
source for land surveys and status by combin-
ing Bureau of Land Management (BLM) and
Forest Service data into a joint system.
• New York City Open Accessible Space Infor-
mation System (OASIS) is another sample
case; as being an online open space mapping
tool, it involves a huge amount of data
concerning public lands, parks, community
gardens, coastal storm impact areas, and zon-
ing and land use patterns.
• Providing online accession of the state Depart-
ments of Natural Resources (DNRs) and other
agencies to the data of Geographic Information
Systems (GIS) on environmental concerns,
while contributing to the effective manage-
ment of land, water, forest, and wildlife, it
essentially requires the use of Big Data to
make this contribution.
• Alabama’s State Water Program is another
example ensuring geospatial data related to
hydrologic, soil, geological, land use, and
land cover issues.
• The National Ecological Observatory Network
(NEON) is an environmental organization pro-
viding the collection of the site-based data
related to the effects of climate change, inva-
sive species from 160 sites and also land use
throughout the USA.
• The Tropical Ecology Assessment and Moni-
toring Network (TEAM) is also a global net-
work facilitating the collection and integration
of publicly shared data related to patterns of
biodiversity, climate, ecosystems, and also
land use.
• The Danger Maps is another sample case for
the use of Big Data, as it also provides the
mapping of government-collected data on
over 13,000 polluting facilities in China to
allow users to search by area or type of pollu-
tion (water, air, radiation, soil).
The US Environmental Protection Agency
(EPA) and the Environmental Performance Index
(EPI) are also other platforms using Big Data
3
compiled from a great number of sensors regard-
ing environmental issues, on land pollution and on
other types of pollution. That is, Big Data tech-
nologies can be thought as a way of addressing
consequences of all types of pollution, not just of
land pollution. This is particularly because, all
types of pollution are deeply interconnected with
another type, so their consequences cannot be
restricted to the place where the pollution is first
discharged as mentioned above. Therefore, actu-
ally, for all types of pollution, relying on satellite
technology and data and data visualization is
essentially required to monitor them regularly, to
forecast and reduce their possible impacts, and to
mitigate the pollution itself. Nonetheless, there are
serious concerns raised about different aspects of
the use of Big Data in general (boyd 2010; boyd
and Crawford 2012; De Mauro et al. 2016; Forte
Wares; Keeso 2014; Mayer-Schönberger and
Cukier 2013; Simon 2013; Sowe and Zettsu
2014). So, further investigation and analysis are
needed to clarify the relevant gaps and challenges
regarding the use of Big Data for specifically land
pollution.
Cross-References
▶ Climate Change
▶ Earth Sciences
▶ Environment
▶ Natural Sciences
▶ Pollution, Air
▶ Pollution, Water
Further Readings
Alloway, B. J. (2001). Soil pollution and land contamina-
tion. In R. M. Harrison (Ed.), Pollution: Causes, effects
and control (pp. 352–377). Cambridge: The Royal
Society of Chemistry.
Boyd, D. (2010). Privacy and publicity in the context of big
data. WWW Conference, Raleigh, 29 Apr 2010.
Retrieved from http://www.danah.org/papers/talks/
2010/WWW2010.html. Accessed 3 Feb 2017.
Boyd, D., & Crawford, K. (2012). Critical questions for big
data, information, communication & society. 15(5),
662–679. Retrieved from http://www.tandfonline.com/
4 Pollution, Land
doi/abs/10.1080/1369118X.2012.678878. Accessed 3 Feb
2017.
De Mauro, A., Greco, M., & Grimaldi, M. (2016). A formal
definition of big data based on its essential features.
Retrieved from https://www.researchgate.net/publica
tion/299379163_A_formal_definition_of_Big_Data_
based_on_its_essential_features. Accessed 3 Feb 2017.
Environmental Law Institute (ELI). (2014). Big data and
environmental protection: An initial survey of public
and private initiatives. Washington, DC: Environmen-
tal Law Institute. Retrieved from https://www.eli.org/
sites/default/files/eli-pubs/big-data-and-environmental-prot
ection.pdf. Accessed 3 Feb 2017.
Environmental Performance Index (EPI). Available at:
http://epi.yale.edu/. Accessed 3 Feb 2017.
Forte Wares. Failure to launch: From big data to big deci-
sions why velocity, variety and volume is not improv-
ing decision making and how to fix it. White Paper.
A Forte Consultancy Group Company. Retrieved from
http://www.fortewares.com/Administrator/userfiles/Ban
ner/forte-wares–pro-active-reporting_EN.pdf. Accessed
3 Feb 2017.
Hill, M. K. (2004). Understanding environmental pollu-
tion. New York: Cambridge University Press.
Keeso, A. (2014). Big data and environmental sustainabil-
ity: A conversation starter. Smith School Working Paper
Series, Dec 2014, Working paper 14-04. Retrieved from
http://www.smithschool.ox.ac.uk/library/working-paper
s/workingpaper%2014-04.pdf. Accessed 3 Feb 2017.
Mayer-Schönberger, V., & Cukier, K. (2013). Big data:
A revolution that will transform how we live, work and
think. London: John Murray.
Mirsal, I. A. (2008). Soil pollution, origin, monitoring &
remediation. Berlin/Heidelberg: Springer.
Raven, P. H., & Berg, L. R. (2006). Environment. Danvers:
Wiley.
Simon, P. (2013). Too big to ignore: The business case for
big data. Hoboken: Wiley.
Sowe, S. K., & Zettsu, K. (2014). Curating big data made
simple: Perspectives from scientific communities. Big
Data, 2(1), 23–33. Mary Ann Liebert, Inc.
Withgott, J., & Brennan, S. (2011). Environment. Cornell
University: Pearson.
P
Pollution, Water
Zerrin Savaşan
Faculty of Economics and Administrative
Sciences, Department of International Relations,
Sub-Department of International Law, Selçuk
University, Konya, Turkey
Water pollution can be defined as the contamina-
tion of water bodies by the entrance of large
amounts of materials/substances to those bodies,
resulting in physical or chemical change in water,
modifying the natural features of the water,
degrading the water quality, and adversely affect-
ing the humans and the environment.
Particularly in recent decades, it is highly
accepted that water pollution is a global environ-
mental problem which is interrelated to all other
environmental challenges. Water pollution con-
trol, at national level, generally should involve
financial resources, technology improvement,
policy measures, and necessary legal and admin-
istrative framework and institutional/staff capac-
ity for implementing these policy measures in
practice. However, more importantly, at global
level, it should involve cooperation of all related
sides at all levels. Despite the efforts at both
national and global levels, reducing pollution sub-
stantially still continues to pose a challenge. This
is particularly because even though the world is
becoming increasingly globalized, it is still mostly
regarded as having with unlimited resources.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_169-1
Hence, it becomes essential to explain that it is
limited and so its resources should not be polluted.
Here, it also becomes essential to have adequate
information on all types of pollution resulting in
environmental deterioration and on water
pollution.
What Causes Water Pollution?
This question has many responses, but basically it
is possible to mention two main reasons: natural
reasons and human-driven reasons. All waters are
subject to some degree of natural (or ecological)
pollution caused by nature rather than by human
activity, through algal blooms, forest fires, floods,
sedimentation stemming from rainfalls, volcanic
eruptions, and other natural events. However, a
greater part of the instances of water pollution
arises from humans’ activities, particularly from
massive industrialization. Accidental spills (e.g., a
disaster like the wreck of an oil tanker, as different
from others, is unpredictable); domestic dis-
charges; industrial discharges; the usage of large
amounts of herbicides, pesticides, chemical fertil-
izers; sediments in waterways of agricultural
fields; improper disposal of hazardous chemicals
down the sewages; and being not able to construct
adequate waste disposal systems can be expressed
as not all but just some of the human-made rea-
sons of water pollution.
The causes as abovementioned vary greatly
because a complex variety of pollutants, lying
2 Pollution, Water
suspended in the water or depositing beneath the
earth’s surface, get involved in water bodies and
result in water quality degradation. Indeed, there
are many different types of water pollutants spill-
ing into waterways causing water pollution. They
all can be divided up into various categories:
chemical, physical, pathogenic pollutants, radio-
active substances, organic pollutants, inorganic
fertilizers, metals, toxic pollutants, biological pol-
lutants, and so on. Conventional, non-
conventional, and toxic pollutants are some of
these divisions which are regulated by the US
Clean Water Act. The conventional pollutants
are as follows: dissolved oxygen, biochemical
oxygen demand (BOD), temperature, pH (acid
deposition), sewage, pathogenic agents, animal
wastes, bacteria, nutrients, turbidity, sediment,
total suspended solids (TSS), fecal coliform, oil,
and grease. Nonconventional (or nontoxic) pollut-
ants are not identified as either conventional or
priority, like aluminum, ammonia, chloride, col-
ored effluents, exotic species, instream flow, iron,
radioactive materials, and total phenols. Toxic
pollutants, metals, dioxin, and lead can be counted
as examples of priority pollutants. Each group of
these pollutants has its own specific ways of enter-
ing the water bodies and its own specific risks.
Water Pollution Control
In order to control all these pollutants, it is bene-
ficial to determine from where they are
discharged. So, the following categories can be
identified to find out where they originate from:
point and nonpoint sources of pollution. If the
sources causing pollution come from single iden-
tifiable points of discharge, they are point sources
of pollution, e.g., domestic discharges, ditches,
pipes of industrial facilities, and ships discharging
toxic substances directly into a water body. Non-
point sources of pollution are characterized by
dispersed, not easily identifiable discharge points,
e.g., runoff of pollutants into a waterway, like
agricultural runoff, stormwater runoff. As it is
harder to identify them, it is nearly impossible to
collect, trace, and control them precisely, whereas
point sources can be easily controlled.
Water pollution, like other types of pollution,
has serious widespread effects. In fact, adverse
alteration of water quality produces costs both to
humans (e.g., large-scale diseases and deaths) and
to environment (e.g., biodiversity reduction, spe-
cies mortality). Its impact differs depending on the
type of water body affected (groundwater, lakes,
rivers, streams, and wetlands). However, it can be
prevented, lessened, and even eliminated in many
different ways. Some of these different treatment
methods, aiming to keep the pollutants from dam-
aging the waterways, can be relied on the use of
techniques reducing water use, reducing the usage
of highly water soluble pesticide and herbicide
compounds, and reducing their amounts, control-
ling rapid water runoff, physical separation of
pollutants from the water, or on the management
practices in the field of urban design and
sanitation.
There are also some other attempts to measure,
reduce, and address rapidly growing impacts of
water pollution, such as the use of Big Data. Big
Data technologies can provide ways of achieving
better solutions for the challenges of water pollu-
tion. To illustrate, EPA databases can be accessed
and maps can be generated from them including
information on environmental activities affecting
water and also on air and land in the context of
EnviroMapper. Under US Department of the Inte-
rior (DOI), National Water Information System
(NWIS) monitors surface and underground water
quantity, quality, distribution, and movement.
Under National Oceanic and Atmospheric
Administration (NOAA), California Seafloor
Mapping Program (CSMP) works for creating a
comprehensive base map series of coastal/marine
geology and habitat for all waters of the USA.
Additionally, the Hudson River Environmental
Conditions Observing System comprises 15 mon-
itoring stations – located between Albany and the
New York Harbor – automatically collecting sam-
ples every 15 min that are used to monitor water
quality, assess flood risk, and assist in pollution
cleanup and fisheries management. Contamina-
tion Warning System Project, conducted by the
Philadelphia Water Department, is a combination
of new data technologies with existing manage-
ment systems. It provides a visual representation
Pollution, Water
of data streams containing geospatial, water qual-
ity, customer concern, operations and public
health information. Creek Watch is another sam-
ple case of the use of Big Data in the field of water
pollution. It is developed by IBM and the Califor-
nia State Water Resources Control Board’s Clean
Water Team as a free app to allow users to rate the
waterway on three criteria: amount of water, rate
of flow, and amount of trash. The collected data is
in large enough to track pollution and manage
water resources. The Danger Maps is another
project mapping government-collected data on
over 13,000 polluting facilities in China. It ren-
ders users to search by area or type of pollution
(water, air, radiation, soil). Developing technol-
ogy on farm performance can also be shown as
another sample on the use of Big Data compiled
from yield information, sensors, high-resolution
maps, and databases for water pollution issue. For
example, machine-to-machine (M2M) agricul-
tural technology produced by a Canadian startup
company Semios allows farmers to improve
yields and their farm operations’ efficiency but
also it provides information for reducing polluted
runoff through increasing the efficient use of
water, pesticides, and fertilizers (ELI 2014).
The Environmental Performance Index (EPI)
is also another platform using Big Data to display
how each country manages environmental issues
and to allow users to investigate data through
comparing environmental performance with
GDP, population, land area, or other variables.
As shown above by example cases, the use of
Big Data technologies is increasingly applied in
the water field, in its different aspects from man-
agement to pollution. However, it is still required
to make further research for their effective use in
order to eliminate related concerns. This is partic-
ularly because there is still debate on the use of
Big Data even regarding its general scope and
terms (Boyd 2010; Boyd and Crawford 2012; De
Mauro et al. 2016; Forte Wares, - ; Keeso 2014;
Mayer-Schönberger and Cukier 2013; Simon
2013; Sowe and Zettsu 2014).
3
Cross-References
▶ Climate Change
▶ Earth Sciences
▶ Environment
▶ Natural Sciences
▶ Pollution, Land
▶ Pollution, Water
Further Readings
Boyd, D. (2010). Privacy and publicity in the context of big
data. WWW Conference, Raleigh, 29 Apr 2010.
Retrieved from http://www.danah.org/papers/talks/
2010/WWW2010.html. Accessed 3 Feb 2017.
Boyd, D., & Crawford, K. (2012). Critical questions for big
data, information, communication &society. 15(5),
662–679. Retrieved from http://www.tandfonline.
com/doi/abs/10.1080/1369118X.2012.678878.
Accessed 3 Feb 2017.
De Mauro, A., Greco, M., & Grimaldi, M. (2016). A formal
definition of big data based on its essential features.
Retrieved from https://www.researchgate.net/publica
tion/299379163_A_formal_definition_of_Big_Data_
based_on_its_essential_features. Accessed 3 Feb 2017.
Environmental Law Institute (ELI). (2014). Big data and
environmental protection: An initial survey of public
and private initiatives. Washington, DC: Environmen-
tal Law Institute. Retrieved from https://www.eli.org/
sites/default/files/eli-pubs/big-data-and-environmental-
protection.pdf. Accessed 3 Feb 2017.
Environmental Performance Index (EPI). Available at:
http://epi.yale.edu/. Accessed 3 Feb 2017.
Forte Wares. Failure to launch: From big data to big deci-
sions why velocity, variety and volume is not improv-
ing decision making and how to fix it. White Paper.
A Forte Consultancy Group Company. Retrieved from
http://www.fortewares.com/Administrator/userfiles/Ban
ner/forte-wares–pro-active-reporting_EN.pdf. Accessed
3 Feb 2017.
Hill, M. K. (2004). Understanding environmental pollu-
tion. New York: Cambridge University Press.
Keeso, A. (2014). Big data and environmental sustainabil-
ity: A conversation starter. Smith School Working Paper
Series, Dec 2014, Working paper 14-04. Retrieved from
http://www.smithschool.ox.ac.uk/library/working-papers/
workingpaper%2014-04.pdf. Accessed 3 Feb 2017.
Mayer-Schönberger, V., & Cukier, K. (2013). Big data:
A revolution that will transform how we livework and
think. London: John Murray.
Raven, P. H., & Berg, L. R. (2006). Environment. Danvers:
Wiley.
4 Pollution, Water

Simon, P. (2013). Too big to ignore: The business case for
big data. Hoboken: Wiley.
Sowe, S. K., & Zettsu, K. (2014). Curating big data made
simple: Perspectives from scientific communities. Big
Data, 2(1), 23–33. Mary Ann Liebert, Inc.
The Open University. (2007). T210 – Environmental con-
trol and public health. The Open University.
Vaughn, J. (2007). Environmental politics. Thomson
Wadsworth.
Vigil, K. M. (2003). Clean water, An introduction to water
quality and water pollution control. Oregon State Uni-
versity Press.
Withgott, J., & Brennan, S. (2011). Environment. Pearson.
P
Predictive Analytics
Anamaria Berea
Center for Complexity in Business, University of
Maryland, College Park, MD, USA
Predictive analytics is a methodology in data min-
ing that uses a set of computational and statistical
techniques to extract information from data with
the purpose to predict trends and behavior pat-
terns. Often, the unknown event of interest is in
the future, but predictive analytics can be applied
to any type of unknown data, whether it is in the
past, present, or future (Siegel 2013). In other
words, predictive analytics can be applied not
only to time series data but to any data where
there is some unknown that can be inferred.
Therefore prediction analytics is a powerful set
of tools for inferring lost past data as well.
The core of predictive analytics in data science
relies on capturing relationships between explan-
atory variables and the predicted variables from
past occurrences, and exploiting them to predict
the unknown outcome. It is important to note,
however, that the accuracy and usability of results
will depend greatly on the level of data analysis
and the quality of assumptions (Tukey 1977).
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_170-1
Predictive Analytics and Forecasting
Prediction, in general, is about forecasting the
future or forecasting the unknown. In the past,
before the scientific method was invented, pre-
dictions were based on astrological observations,
witchcraft, foretelling, oral history folklore, and,
in general, on random observations or associa-
tions of observations that happened at the same
time. For example, if a conflict happened during
an eclipse, then all eclipses would become
“omens” of wars and, in general, bad things. For
a long period of time in our civilization, the events
were merely separated in two classes: good or
bad. And thus the associations of events that
would lead to a major conflict or epidemics or
natural catastrophe would be categorized as
“bad” omens from there on, while any associa-
tions of events that would lead to peace, prosper-
ity, and, in general, “good” major events would be
categorized as “good” omens or good predictors
from there on.
The idea of associations of events as predictive
for another event is actually at the core of some of
the statistical methods we are using today, such as
correlation. But the fallacy of using these methods
metaphorically instead of in a quantitative system-
atic analysis is that only one set of observations
cannot be predictive for the future. That was true
in the past and it is true now as well, no matter
how sophisticated the techniques we are using.
Predictive analytics uses a series of events or
associations of events, and the longer the series,
2 Predictive Analytics
the more informative the predictive analysis
can be.
Unlike past good or bad omens, the results of
predictive analytics are probabilistic. This means
that predictive analytics informs the probability of
a certain data point or the probability of a hypoth-
esis to be true.
While true prediction can be achieved only by
determining clearly the cause and the effect in a
set of data, a task that is usually hard to do, most of
the predictive analytics techniques are outputting
probabilistic values and error term analyses.
Predictive Modeling Methods
Predictive modeling statistically shows the under-
lying relationships in historical, time series data in
order to explain the data and make predictions,
forecasts, or classifications about future events.
In general, predictive analytics uses a series of
statistical and computational techniques in order
to forecast future outcomes from past data. Tradi-
tionally, the most used method has been the linear
regression, but lately, with the emergence of the
Big Data phenomenon, there have been developed
many other techniques aiming to support busi-
nesses and forecasters, such as machine learning
algorithms or probabilistic methods.
Some classes of techniques include:
1. Applications of both linear and nonlinear
mathematical programming algorithms, in
which one objective is optimized within a set
of constraints.
2. Advanced “neural” systems, which learn com-
plex patterns from large datasets to predict the
probability that a new individual will exhibit
certain behaviors of business interest. Neural
networks (also known as deep learning) are
biologically inspired machine learning models
that are being used to achieve the recent
record-breaking performance on speech recog-
nition and visual object recognition.
3. Statistical techniques for analysis and pattern
detection within large datasets.
Some techniques in predictive analytics are
borrowed from traditional forecasting techniques,
such as moving average, linear regressions, logis-
tic regressions, probit regressions, multinomial
regressions, time series models, or random forest
techniques. Other techniques, such as supervised
learning, A|B testing, correlation ranking,
k-nearest neighbor algorithm are closer to
machine learning and newer computational
methods.
One of the most used techniques in predictive
analytics today though is supervised learning or
supervised segmentation (Provost and Fawcett
2013). Supervised segmentation includes the fol-
lowing steps:
– Selection of informative attributes – particu-
larly in large datasets, the selection of the vari-
ables that are more likely to be informative to
the goal of prediction is crucial; otherwise the
prediction can render spurious results.
– Information gain and entropy reduction – these
two techniques measure the information in the
selected attributes.
– Selection is done based on tree induction,
which fundamentally represents subsetting
the data and searching for these informative
attributes.
– The resulting tree-structured model partitions
the space of all data into possible segments
with different predicted values.
The supervised learning/segmentation has
been popular because it is computationally and
algorithmically simple.
Visual Predictive Analytics
Data visualization and predictive analytics com-
plement each other nicely and together they are an
even more powerful methodology for the analysis
and forecasting of complex datasets that comprise
a variety of data types and data formats.
Visual predictive analytics is a specific set of
techniques of predictive analytics that is applied
to visual and image data. Just as in the case of
Predictive Analytics
predictive analytics in general, temporal data is
required for the visual (spatial) data (Maciejewski
et al. 2011). This technique is particularly useful
in determining hotspots and areas of conflict with
a high dynamics. Some of the techniques used in
spatiotemporal analysis are kernel density estima-
tion for event distribution and seasonal trend
decomposition by loess smoothing (Maciejewski
et al. 2011).
Predictive Analytics Example
A good example for using predictive analytics is
in healthcare. The problem of understanding the
probability of an upcoming epidemics or the prob-
ability of increase in incidence of various dis-
eases, from flu to heart disease and cancer.
For example, given a dataset that contains data
with respect to the past incidence of heart disease
in the USA, demographic data (gender, average
income, age, etc.), exercise habits, eating habits,
traveling habits, and other variables, a predictive
model would follow these steps:
1. Descriptive statistics – the first step in doing
predictive analytics or building a predictive
model is always an understanding of the data
with respect to what the variables represent,
what ranges they fall into, how long is the
time series, ASO, essentially a summary statis-
tics of the data.
2. Data cleaning and treatment – it is very impor-
tant to understand not what the data is or has
but also what the data is missing.
3. Build the model/s – in this step, several tech-
niques can be explored and used comparatively
and based on their results; the best one should
be chosen. For example, both a general regres-
sion and a random forest can be used and
compared, or supervised segmentation based
on demographics and then the segments
compared.
4. Performance and accuracy estimation – in this
final step, the probabilities or measurements of
forecasting accuracy are computed and
interpreted.
3
In any predictive model or analytics technique,
the model can do only what the data is. In other
words, it is impossible to assess a predictive
model of the heart disease incidence based on
the travel habits if no data regarding travel is
included.
Another important point to remember is that
the accuracy of the model also depends on the
accuracy measure, and using multiple accuracy
measures is desired (i.e., mean squared error,
p-value, R-squared).
In general, any predictive analytic technique
will output a dataset of created variables, called
predictive values, and the newly created dataset.
Therefore a good technique for verification and
validation of the methods used is to partition the
real dataset in two sets and use one to “train” the
model and the second one to validate the model’s
results.
The success of the model ultimately depends
on how real events will unfold and that is one of
the reasons why longer time series are better at
informing predictive modeling and giving better
accuracy for the same set of techniques.
Predictive Analytics Fallacies
Cases of “spurious correlations” tend to be quite
famous, such as the correlation between the num-
ber of people who dies tangled in their bed sheets
and the consumption of cheese per capita (http://
www.tylervigen.com/spurious-correlations). These
examples fall on the same fallacy as the “bad”/
“good” omen one, as the observations of the events
at the same time does not imply that there is a
causal relationship between the two events.
Another classic example is to think, in general,
that correlations show a causal relationship; there-
fore predictions based on correlation analyses
alone tend to fail often.
Some other fallacies of predictive analytics
techniques include an insufficient analysis of the
errors, relying on the p-value alone, relying on a
Poisson distribution of the current data, and
many more.
4 Predictive Analytics
Predictive/Descriptive/Prescriptive
There is a clear distinction between descriptive
vs. predictive vs. prescriptive analytics in Big
Data (Shmueli 2010). Descriptive analytics
shows how past or current data can be analyzed
in order to determine patterns and extract mean-
ingful observations out of the data. Predictive
analytics is generally based on a model that is
informed by descriptive analytics and gives vari-
ous outcomes based on past data and the model.
Prescriptive analytics is closely related to predic-
tive analytics, as it takes the predictive values,
puts them in a decision model, and informs the
decision-makers about the future course of action
(Shmueli and Koppius 2010).
Predictive Analytics Applications
In practice, predictive analytics can be applied to
almost all disciplines – from predicting the failure
of mechanical engines in hard sciences, to pre-
dicting customers’ buying power in social sci-
ences and business (Gandomi and Haider 2015).
Predictive analytics is especially used in busi-
ness and marketing forecasting. Hair Jr. (2007)
shows the importance of predictive analytics for
marketing and how it has become more relevant
with the emergence of the Big Data phenomenon.
He argues that survival in a knowledge-based
economy is derived from the ability to convert
information to knowledge. Data mining identifies
and confirms relationships between explanatory
and criterion variables. Predictive analytics uses
confirmed relationships between variables to pre-
dict future outcomes. The predictions are most
often values suggesting the likelihood a particular
behavior or event will take place in the future.
Hair also argues that, in the future, we can
expect predictive analytics to increasingly be
applied to databases in all fields and revolutionize
the ability to identify, understand, and predict
future developments; data analysts will increas-
ingly rely on mixed-data models that examine
both structured (numbers) and unstructured (text
and images) data; statistical tools will be more
powerful and easier to use; future applications
will be global and real time; demand for data
analysts will increase as will the need for students
to learn data analysis methods; and scholarly
researchers will need to improve their quantitative
skills so the large amounts of information avail-
able can be used to create knowledge instead of
information overload.
Predictive Modeling and Other
Forecasting Techniques
Some predictive modeling techniques do not nec-
essarily involve Big Data. For example, Bayesian
networks and Bayesian inference methods, while
they can be informed by Big Data, they cannot be
applied granularly to each data point due to the
computational complexity that can arise from cal-
culating thousands of conditional probability
tables. But Bayesian models and inferences can
certainly be used in combination with statistical
predictive modeling techniques in order to bring
the analysis closer to a cause-effect type of infer-
ence (Pearl 2009).
Another forecasting technique, that does not
rely on Big Data, but harnesses the power of the
crowds, is the prediction market. Just like Bayes-
ian modeling, prediction markets can be used as a
complement to Big Data and predictive modeling
in order to augment the likelihood value of the
predictions (Arrow et al. 2008).
References
Arrow, K.J., et al. (2008). The promise of prediction mar-
kets. Science-New York then Washington-320.5878:
877.
Gandomi, A., & Haider, M. (2015). Beyond the hype: Big
data concepts, methods, and analytics. International
Journal of Information Management, 35(2), 137–144.
Hair Jr., J. F. (2007). Knowledge creation in marketing:
The role of predictive analytics. European Business
Review, 19(4), 303–315.
Maciejewski, R., et al. (2011). Forecasting hotspots –
A predictive analytics approach. IEEE Transactions
on Visualization and Computer Graphics, 17(4),
440–453.
Pearl, J. (2009). Causality. Cambridge: Cambridge univer-
sity press.
Predictive Analytics
Provost, F., & Fawcett, T. (2013). Data science for busi-
ness: What you need to know about data mining and
data-analytic thinking. Sebastopol: O’Reilly Media.
Shmueli, G. (2010) To Explain or to Predict?. Statistical
Science 25(3):289–310.
5
Shmueli, G., & Koppius, O. (2010). Predictive analytics in
information systems research. Robert H. Smith School
Research Paper No. RHS, 06-138.
Siegel, E. (2013). Predictive analytics: The power to pre-
dict who will click, buy, lie, or die. Hoboken: Wiley.
Tukey, J. (1977). Exploratory data analysis. New York:
Addison-Wesley.
P
Privacy
Joanna Kulesza
Department of International Law and
International Relations, University of Lodz, Lodz,
Poland
Origins and Definition
Privacy is a universally recognized human right,
subject to state protection from arbitrary or unlaw-
ful interference and unlawful attacks. The age of
Big Data has brought it to the foreground of all
technology-related debates as the amount of infor-
mation aggregated online, generated by various
sources together with the computing capabilities
of modern networks, makes it easy to connect an
individual to a particular piece of information
about them, possibly causing a direct threat to
their privacy. Yet international law grants every
person the right to legal safeguards against any
interference with one’s right or attacks upon
it. The right to privacy covers, although is not
limited to, one’s identity, integrity, intimacy,
autonomy, communication, and sexuality and
results in legal protection for one’s physical integ-
rity; health information, including sex orientation
and gender; reputation; image; personal develop-
ment; personal autonomy; and self-determination
as well as family, home, and correspondence that
are to be protected by state from arbitrary or
unlawful interferences by its organs or third
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_172-1
parties. This catalogue is meant to remain an
open one, enabling protection of forever new cat-
egories of data, such as geographical location data
or arguably to a “virtual personality.” As such, the
term covers also information about an individual
that is produced, generated, or needed for the
purpose of rendering electronic services, such as
a telephone, an IMEI or an IP number, e-mail
address, a website address, geolocation data, or
search terms, as long as such information may be
linked to an individual and allows for their identi-
fication. Privacy is not an absolute right and may
be limited for reasons considered necessary in a
democratic society. While there is no numerus
clausus of such limitative grounds, they usually
include reasons of state security and public order
or the rights of others, such as their freedom of
expression. States are free to introduce certain
limitations on individual privacy right as long as
those are introduced by specific provisions of law,
communicated to the individuals whose privacy is
impacted, and applied solely when necessary in
particular circumstances. This seemingly clear and
precise concept suffers practical limitations as
states differ in their interpretations of “necessity”
of interference as well as the “specificity” of legal
norms required and scope of their application. As a
consequence the concept of privacy strongly
varies throughout the world’s regions and coun-
tries. This is a particular challenge at the time of
Big Data as various national and regional percep-
tions of privacy need to be applied to the very same
vast catalogue of online information.
2 Privacy
This inconsistency in privacy perceptions
results from varied cultural and historical back-
ground of individual states as well as their differ-
ing political and economic situation. In countries
recognizing values reflected in universal human
rights treaties, including Europe, large parts of
Americas, and some Asian states, the right to
privacy covers numerous elements of individual
autonomy and is strongly protected by compre-
hensive legal safeguards. On the other hand in
countries rapidly developing, as well as in ones
with unstable political or economic situation, pri-
marily located in Asia and Africa, the significance
of the right to one’s private life subsides to urgent
needs of protecting life and personal or public
security. As a consequence the undisputed right
to privacy, subject to numerous international
treaties and rich international law jurisprudence,
remains highly ambiguous, an object of
conflicting interpretations by national authorities
and their agents. This is one of the key challenges
to finding the appropriate legal norms governing
Big Data. In the unique Big Data environment, it
is not only the traditional jurisdictional chal-
lenges, specific to all online interactions, that
must be faced but also the tremendously varying
perceptions of privacy all finding their application
to the vast and varied Big Data resource.
History
The idea of privacy rose simultaneously in various
cultures. Contemporary authors most often refer
to the works of American and European legal
writers of late nineteenth century to identify its
origins. In US doctrine it were Warren and Bran-
deis who introduced in their writings “the right to
be let alone,” a notion still often used to describe
the essential content of privacy. Yet at remotely
the same time, German legal scholar Kohler
published a paper covering a similar concept. It
was also in mid nineteenth century that French
courts issued their first decisions protecting the
right to private life. The right to privacy was
introduced to grant individuals protection from
undesired intrusions into their private affairs and
home life, be it by nosy journalists or
governmental agents. Initially the right was used
to limit the rapidly evolving press industry, with
time, as individual awareness and recognition of
the right increased; the right to privacy primarily
introduced limits of individual information that
state or local authorities may obtain and process.
As any new idea, the right to privacy initially
provoked much skepticism, yet by mid twentieth
century became a necessary element of the rising
human rights law. In the twenty-first century, it
gained increased attention as a side effect of the
growing, global information society. International
online communications allowed for easy and
cheap mass collection of data, creating the
greatest threat to privacy so far. What followed
was an eager debate on the limits of allowed
privacy intrusions and actions required from states
aimed at safeguarding the rights of an individual.
A satisfactory compromise is not easy to find as
states and communities view privacy differently,
based on their history, culture, and mentality. The
existing consensus on human rights seems to be
the only starting point of a successful search for an
effective privacy compromise, much needed in
the era of transnational companies operating on
Big Data. With the modern notions of “the right to
be forgotten” or “data portability” referring to new
facets of the right to protect one’s privacy, the Big
Data phenomenon is one of the deciding factors of
this ongoing evolution.
Privacy as a Human Right
The first document of international human rights
law, recognizing the right to privacy, was the 1948
Universal Declaration on Human Rights (UDHR).
The nonbinding political middle ground was not
too difficult to find with the greatest horrors in
human history of World War II still vividly in the
mind of world’s politicians and citizens alike.
With horrid memories fading away and the Iron
Curtain drawing a clear line between differing
values and interests, a binding treaty on the very
issue took almost 20 more years. Irreconcilable
differences between communist and capitalist
countries covered the scope and implementation
of individual property, free speech, or privacy.
Privacy
The eventual 1966 compromise in the form of the
two fundamental human rights treaties: the Inter-
national Covenant on Civil and Political Rights
(ICCPR) and the International Covenant on Eco-
nomic Social and Cultural Rights (ICESCR) allo-
wed for a conciliatory wording on hard law
obligations for different categories of human
rights, yet left the crucial details to future state
practice and international jurisprudence. Among
the right to be put into detail by future state prac-
tice, international courts, and organizations was
the right to privacy, established as a human right
in Article 12 UDHR and Article 17 ICCPR. They
both granted every individual freedom from “arbi-
trary interference” with their “privacy, family,
home, or correspondence” as well as from any
attacks upon their honor and reputation. While
neither document defines “privacy,” the UN
Human Rights Committee (HRC) has gone into
much detail on delimitating its scope for the inter-
national community. All 168 ICCPR state parties
are obliged per the Covenant to reflect HRC rec-
ommendations on the scope and enforcement of
the treaty in general and privacy in particular.
Over time the HRC produced detailed instruction
on the scope of privacy protected by international
law, discussing the thin line with state sover-
eignty, security, and surveillance. According to
Article 12 UDHR and Article 17 ICCPR, privacy
must be protected against “arbitrary or unlawful”
intrusions or attacks through national laws and
their enforcement. Those laws are to detail limits
for any justified privacy invasions. Those limits of
individual privacy right are generally described in
Article 29 para. 2 which allows for limitations of
all human rights determined by law solely for the
purpose of securing due recognition and respect
for the rights and freedoms of others and of meet-
ing the just requirements of morality, public order,
and the general welfare in a democratic society.
Although proposals for including a similar
restraint in the text of the ICCPR were rejected
by negotiating parties, the right to privacy is not
an absolute one. Following HRC guidelines and
state practice surrounding the ICCPR, privacy
may be restrained according to national laws
which meet the general standards present in
human rights law. The HRC confirmed this
3
interpretation in its 1988 General Comment
No. 16 as well as recommendations and observa-
tions issued thereafter. Before Big Data became,
among its other functions, an effective tool for
mass surveillance, the HRC took a clear stand on
the question of legally permissible limits of state
inspection. It clearly stated that any surveillance,
whether electronic or otherwise; interceptions of
telephonic, telegraphic, and other forms of com-
munication; wiretapping; and recording of con-
versations should be prohibited. It confirmed that
individual limitation upon privacy must be
assessed on a case-by-case basis and follow a
detailed legal guideline, containing precise cir-
cumstances when privacy may be restricted by
actions of local authorities or third parties. The
HRC specified that even interference provided for
by law should be in accordance with the provi-
sions, aims, and objectives of the Covenant and
reasonable in the particular circumstances, where
“reasonable” means justified by those particular
circumstances. Moreover, as per the HRC inter-
pretation, states must take effective measures to
guarantee that information about individual’s life
does not reach ones not authorized by law to
obtain, store, or process it. Those general guide-
lines are to be considered the international stan-
dard of protecting the human right to privacy and
need to be respected regardless of the ease that Big
Data services offer in connecting pieces of infor-
mation available online with individuals they
relate to. Governments must ensure that Big
Data is not to be used in a way that infringes
individual privacy, regardless of the economic
benefits and technical accessibility of Big Data
services.
The provisions of Article 17 ICCPR resulted in
similar stipulations of other international treaties.
Those include Article 8 of the European Conven-
tion on Human Rights (ECHR) binding upon its
48 member states or Article 11 of the American
Convention on Human Rights (ACHR) agreed
upon by 23 parties to the treaty. The African
Charter on Human and Peoples’ Rights (Banjul
Charter) does not contain a specific stipulation
regarding privacy, yet its provisions of Article
4 on the inviolability of human rights, Article
5 on human dignity, and Article 16 on the right
4 Privacy
to health serve as basis to grant individuals within
the jurisdiction of 53 state parties the protection
recognized by European or American states as
inherent to the right of privacy. While no general
human rights document exists among Austral-
asian states, the general guidelines provided by
the HRC and the work of the OECD are often
reflected in national laws on privacy, personal
rights, and personal data protection.
Privacy and Personal Data
The notion of personal data is closely related to
that of privacy, yet their scopes differ. While per-
sonal data is a term relatively well defined, pri-
vacy is a more broad and ambiguous notion. As
Kuner rightfully notes, the concept of privacy
protection is a broader one than personal data
regulations, where the latter provides a more
detailed framework for individual claims. The
influential Organization for Economic
Co-operation and Development (OECD) Forum
identified personal data as a component of the
individual right to privacy, yet its 34 members
differ on the effective methods of privacy protec-
tion and the extent to which such protection
should be granted. Nevertheless, the nonbinding
yet influential 1980 OECD Guidelines on the Pro-
tection of Privacy and Transborder Flow of Per-
sonal Data (Guidelines) together with their 2013
update have so far encouraged over data protec-
tion laws in over 100 countries, justifying the
claim that, thanks to its detailed yet unified char-
acter and national enforceability personal data
protection, is the most common and effective
legal instrument safeguarding individual privacy.
The Guidelines identify the universal privacy pro-
tection through eight personal data processing
principles. The definition of “personal data”
contained in the Guidelines is usually directly
adopted by national legislations which cover any
information relating to an identified or identifiable
individual, referred to as “data subject.” The basic
eight principles of privacy and data protection
include (1) the collection limitation principle,
(2) the data quality principle, (3) the individual
participation principle, (4) the purpose
specification principle, (5) the use limitation prin-
ciple, (6) the security safeguards principle, (7) the
openness principle, and (8) the accountability
principle. They introduce certain obligations
upon “data controllers” that is parties “who,
according to domestic law, are competent to
decide about the contents and use of personal
data regardless of whether or not such data are
collected, stored, processed or disseminated by
that party or by an agent on their behalf.” They
oblige data controllers to respect limits made by
national laws pertaining to the collection of per-
sonal data. As already noted this is of particular
importance to Big Data operators, who must be
aware and abide by the varying national regimes.
Personal data must be obtained by “lawful and
fair” means and with the knowledge or consent
of the data subject, unless otherwise provided by
relevant law. Collecting or processing personal
data may only be done when it is relevant to the
purposes for which it will be used. Data must be
accurate, complete, and up to date. The purposes
for data collection ought to be specified no later
than at the time of data collection. The use of the
data must be limited to the purposes so identified.
Data controllers, including those operating on Big
Data, are not to disclose personal data at their
disposal for purposes other than those initially
specified and agreed upon by the data subject,
unless such use or disclosure is permitted by law.
All data processors are to show due diligence in
protecting their collected data, by introducing rea-
sonable security safeguards against the loss or
unauthorized data access and its destruction, use,
modification, or disclosure. This last obligation
may prove particularly challenging for Big Data
operators, with regard to the multiple locations of
data storage and their continuously changeability.
Consequently each data subjects enjoys the right
to obtain information on the fact of the data con-
troller having data relating to him, to have any
such data communicated within a reasonable time,
to be given reasons if a request for such informa-
tion is denied, as well as to be able to challenge
such denial and any data relating to him.
Followingly each data subject enjoys the right to
have their data erased, rectified, completed, or
amended, and data controller is to be held
Privacy
accountable to national laws for lack of effective
measures ensuring all of those personal data
rights.
Therewith the OECD principles form a practi-
cal standard for privacy protection represented in
the human rights catalogue, applicable also to Big
Data operators, given the data in their disposal
relates directly or indirectly to an individual.
While their effectiveness may come to depend
upon jurisdictional issues, the criteria for identifi-
cation of data subjects and obligations of data
processors are clear.
Privacy as a Personal Right
Privacy is recognized not only by international
law treaties and international organizations but
also by national laws, from constitutions to civil
and criminal law codes and acts. Those regula-
tions hold great practical significance, as they
allow for direct remedies against privacy infrac-
tions from private parties, rather than those
enacted by state authorities. Usually privacy is
considered an element of the larger catalogue of
personal rights and granted equal protection. It
allows individuals whose privacy is under threat
for the threatening activity to be seized (e.g.,
infringing information be deleted or a press
release be stopped). It also allows for pecuniary
compensation or damages should a privacy
infringement already take place.
Originating from German-language civil law
doctrine, privacy protection may be well
described by the theory of concentric spheres.
Those include the public, private, and intimate
sphere, with different degrees of protection from
interference granted to each of them. The stron-
gest protection is granted to intimate information;
activities falling within the public sphere are not
protected by law and may be freely collected and
used. All individual information may be qualified
as falling into one of the three spheres, with the
activities performed in the public sphere being
those performed by an individual as a part of
their public or professional duties and obligations
and deprived of privacy protection. This sphere
would differ as per individual, with “public
5
figures” enjoying least protection. An assessment
of the limits of one’s privacy when compared with
their public function would always be made on
case-by-case basis. Any information that may not
be considered public is to be granted privacy
protection and may only be collected or processed
with permission granted by the one it concerns.
The need to obtain consent from the individual the
information concerns is also required for the inti-
mate sphere, where the protection is even stron-
ger. Some authors argue that information on one’s
health, religious beliefs, sexual orientation, or his-
tory should only be distributed in pursuit of a
legitimate aim, even when permission for its dis-
tribution was granted by the one it concerns.
With the civil law scheme for privacy protec-
tion being relatively simple, its practical applica-
tion relies on case-by-case basis and therefore
may show challenging and unpredictable in prac-
tice, especially when international court practice
is of issue.
Privacy and Big Data
Big Data is a term that directly refers to informa-
tion about individuals. It may be defined as gath-
ering, compiling, and using large amounts of
information enabling for marketing or policy deci-
sions. With large amounts of data being collected
by international service providers, in particular
ones offering telecommunication services, such
as Internet access, the scope of data they may
collect and the use to which they may put it is of
crucial concern to all their clients but also to their
competitors and state authorities interested in par-
ticipating in this valuable resource. In the light of
the analysis presented above, any information
falling within the scope of Big Data that is col-
lected and processed while rendering online ser-
vices may be considered subject to privacy
protection when it refers to identified or identifi-
able individual that is a physical person who may
either be directly identified or whose identification
is possible. When determining whether particular
category or a piece of information constitutes
private data, account must be taken of means
likely reasonably to be used either by any person
6 Privacy
to identify the individual, in particular costs, time,
and labor needed to identify such person. When
private information has been identified, the pro-
cedures required for privacy protection described
above ought to be applied by entities dealing with
such information. In particular the guidelines
described by the HRC in their comments and
observations may serve as a guideline for han-
dling personal data falling within the Big Data
resource. Initiatives such as Global Network Ini-
tiative, a bottom-up initiative of the biggest online
service providers aimed at identifying and apply-
ing universal human rights standards for online
services, or the UN Protect Respect and Remedy
Framework for business, defining the human
rights obligations of private parties, present a use-
ful tool for introducing enhanced privacy safe-
guards for all Big Data resources. With the
users’ growing awareness of the value of their
privacy, company privacy policies prove to be a
significant element of the marketing game, incit-
ing Big Data operators to convince forever more
users to choose their privacy-oriented services.
Summary
Privacy recognized as a human right requires cer-
tain precautions to be taken by state authorities
and private business alike. Any information that
may allow for the identification of an individual
ought to be subjected to particular safeguards
allowing for its collection or processing solely
based on the consent of the individual in question
or a particular norm of law applicable in a case
where the inherent privacy invasion is reasonable
and necessary to achieve a justifiable aim. In no
case may private information be collected or
processed in bulk, with no judicial supervision
or without the consent of the individual it refers
to. Big Data offer new possibilities for collecting
and processing personal data. When designing
Big Data services or using information they pro-
vide, all business entities must address the inter-
national standards of privacy protection, as
identified by international organizations and
good business practice.
Cross-References
▶ Data Processing
▶ Data Profiling
▶ Data Quality Management
▶ Data Security
▶ Data Security Management
Further Readings
Kuner, C. (2009). An international legal framework for
data protection: Issues and prospects. Computer Law
and Security Review, 25(263), 307.
Kuner, C. (2013). Transborder data flows and data privacy
law. Oxford: Oxford University Press.
UN Human Rights Committee. General Comment No. 16:
Article 17 (Right to Privacy), The Right to Respect of
Privacy, Family, Home and Correspondence, and Pro-
tection of Honour and Reputation. 8 Apr 1988. http://
www.refworld.org/docid/453883f922.html.
UN Human Rights Council. Report of the Special Rappor-
teur on the promotion and protection of human rights
and fundamental freedoms while countering terrorism,
Martin Scheinin. U.N. Doc. A/HRC/13/37.
Warren, S.D., & Brandeis, L.D. (1980). The right to pri-
vacy. Harvard Law Review, v. 4/193.
Weber, R.H. (2013). Transborder data transfers: Concepts,
regulatory approaches and new legislative initiatives.
International Data Privacy Law v. 1/3–4.
P
Psychology
Daniel N. Cassenti and Katherine R. Gamble
U.S. Army Research Laboratory, Adelphi, MD,
USA
Wikipedia introduces big data as “a blanket term
for any collection of data sets so large and com-
plex that it becomes difficult to process using
on-hand data management tools or traditional
data processing applications.” The field of psy-
chology is interested in big data in two ways:
(1) at the level of the data, that is, how much
data there are to be processed and understood,
and (2) at the level of the user, or how the
researcher analyzes and interprets the data. Thus,
psychology can serve the role of helping to
improve how researchers analyze big data and
provide data sets that can be examined or analyzed
using big data principles and tools.
Psychology
Psychology may be divided into two overarching
areas: clinical psychology with a focus on indi-
viduals, and the fields of experimental psychology
with foci on the more general characteristics that
apply to the majority of people. Allen Newell
classifies the fields of experimental psychology
by time scale, to include biological at the smallest
time scale, cognitive (the study of mental
# Springer International Publishing AG (outside the USA) 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_173-1
processes) at the scale of hundreds of milliseconds
to tens of seconds, rational (the study of decision
making and problem solving) at minutes to hours,
and social at days to months. The cognitive, ratio-
nal, and social bands can all be related to big data
in terms of both the researcher analyzing data and
the data itself. Here, we describe how psycholog-
ical principles can be applied to the researcher to
handle data in the cognitive and rational fields and
demonstrate how psychological data in the social
field can be big data.
Cognitive and Rational Fields
One of the greatest challenges of big data is its
analysis. The principles of cognitive and rational
psychology can be applied to improve how the big
data researcher evaluates and makes decisions
about the data. The first step in analysis is atten-
tion to the data, which often involves filtering out
irrelevant from relevant data. While many soft-
ware programs can provide an automated filtering
of data, the researcher must still give attention and
critical analysis to the data as a check on the
automated system, which operates within rigid
criteria preset by the researcher that is not sensi-
tive to the context of the data. At this early level of
analysis, the researcher’s perception of the data,
ability to attend and retain attention, and working
memory capacity (i.e., the quantity of information
that an individual can store while working on a
task) are all important to success. That is, the
2 Psychology
researcher must efficiently process and highlight
the most important information, stay attentive
enough to do this for a long period of time, and
because of limited working memory capacity and
a lot of data to be processed, effectively manage
the data, such as by chunking information, so that
it is easier to filter and store in memory.
The goal of analysis is to lead to decisions or
conclusions about data, the scope of the rational
field. If all principles from cognitive psychology
have been applied correctly (e.g., only the most
relevant data are presented and only the most
useful information stored in memory), tenets of
rational psychology must next be applied to make
good decisions about the data. Decision making
may be aided by programming the analysis soft-
ware to present decision options to the researcher.
For example, in examining educational outcomes
of children who come from low income families,
the researcher’s options may be to include chil-
dren who are or are not part of a state-sponsored
program, or are of a certain race. Statistical soft-
ware could be designed to present these options to
the researcher, which may reveal results or rela-
tionships in the data that the researcher may not
have otherwise discovered. Option presentation
may not be enough, however, as researchers
must also be aware of the consequences of their
decisions. One possible solution is the implemen-
tation of associate systems for big data software.
An associate system is automation that attempts to
advise the user, in this case to aid decision mak-
ing. Because these systems are knowledge based,
they have situational awareness and are able to
recommend courses of action and the reasoning
behind those recommendations. Associate sys-
tems do not make decisions themselves, but
instead work semiautonomously, with the user
imposing supervisory control. If the researcher
deems recommended options to be unsuitable,
then the associate system can present what it
judges to be the next best options.
Social Field
The field of social psychology provides good
examples of methods of analysis that can be
used with big data, especially with big data sets
that include groups of individuals and their rela-
tionships with one another, the scope of social
psychology. The field of social psychology is
able to ask questions and collect large amounts
of data that can be examined and understood using
these big data-type analyses including, but not
limited to, the following types of analyses.
Linguistic analysis offers the ability to process
transcripts of communications between individ-
uals, or to groups as in social media applications,
such as tweets from a Twitter data set. A linguistic
analysis may be applied in a multitude of ways,
including analyzing the qualities of relationship
between individuals or how communications to
groups may differ based on the group. These
analyses can determine qualities of these commu-
nications, which may include trust, attribution of
personal characteristics, or dependencies, among
other considerations.
Sentiment analysis is a type of linguistic anal-
ysis that takes communications and produces rat-
ings of the emotional valence individuals direct to
the topic. This is of value for considerations of
social data researchers who must find those with
whom alliances may be formed and who to avoid.
A famous example is the strategy shift taken by
United Stated Armed Forces commanders to ally
with Iraqi residents. Sentiment analysis indicated
which residential leaders would give their coop-
eration for short-term goals of mutual interest.
The final social psychological big data analysis
technique under consideration here is social-
network analysis or SNA. With SNA, special
emphasis is not with the words spoken as in lin-
guistic and sentiment analysis but on the direc-
tionality and frequency of communication
between individuals. SNA created a type of net-
work map that uses nodes and ties to connect
members of groups or organizations to one
another. This visualization tool allows a
researcher to see how individuals are connected
to one another with factors like the thickness of a
line to determine frequency of communication, or
the number of lines coming from a node deter-
mining the number of nodes to which they are
connected.
Psychology
Psychological Data as Big Data
Each field of psychology potentially includes big
data sets for analysis by a psychological
researcher. Traditionally, psychologists have col-
lected data on a smaller scale using controlled
methods and manipulations analyzable with tradi-
tional statistical analyses. However, with the
advent of big data principles and analysis tech-
niques, psychologists can expand the scope of
data collection to examine larger data sets that
may lead to new and interesting discoveries. The
following section discusses each of the aforemen-
tioned fields.
In clinical psychology, big data may be used to
diagnose an individual. In understanding an indi-
vidual or attempting to make a diagnosis, the
person’s writings and interview transcripts may
be analyzed in order to provide insight to his or
her state of mind. To thoroughly analyze and treat
a person, a clinical psychologist’s most valuable
tool may be this type of big data set.
Biological psychology includes the subfields
of psychophysiology and neuropsychology. Psy-
chophysiological data may include hormone col-
lection (typically salivary), blood flow, heart rate,
skin conductance, and other physiological
responses. Neuropsychology includes multiple
technologies for collecting information about the
brain, including electroencephalography (EEG),
functional magnetic resonance imaging (fMRI),
functional near infrared spectroscopy (fNIRS),
among other lesser used technologies. Measures
in biological psychology are generally taken near-
continuously across a certain time range, so much
of the data collected in this field could be consid-
ered big data.
Cognitive psychology covers all mental pro-
cessing. That is, this field includes the initiation of
mental processing from internal or external stim-
uli (e.g., seeing a stoplight turn yellow), the actual
processing of this information (e.g., understand-
ing that a yellow light means to slow down), and
the initiation of an action (e.g., knowing that you
must step on the brake in order to slow your car).
For each action that we take, and even actions that
may be involuntary (e.g., turning your head
toward an approaching police siren as you begin
3
to slow your car), cognitive processing must take
place at the levels of perception, information pro-
cessing, and initiation of action. Therefore, any
behavior or thought process that is measured in
cognitive psychology will yield a large amount of
data for even the simplest of these, such that
complex processes or behaviors measured for
their cognitive process will yield data sets of the
magnitude of big data.
Another clear case of a field with big data sets
is rational psychology. In rational psychological
paradigms, researchers who limit experimental
participants to a predefined set of options often
find themselves limiting their studies to the point
of not capturing naturalistic rational processing.
The rational psychologist, instead typically con-
fronts big data as imaginative solutions to prob-
lems, and many forms of data, such as verbal
protocols (i.e., transcripts of participants
explaining their reasoning), require big data anal-
ysis techniques.
Finally, with the large time band under consid-
eration, social psychologists must often consider
days’ worth of data in their studies. One popular
technique is to have participants use wearable
technology to periodically remind them to record
how they are doing, thinking, and feeling during
the day. These types of studies lead to big data sets
not just because of the frequency with which the
data is collected, but also due to the enormous
number of possible activities, thoughts, and feel-
ing that participants may have experienced and
recorded at each prompted time point.
The Unique Role of Psychology in
Big Data
As described above, big data plays a large role in
the field of psychology, and psychology can play
an important role in how big data are analyzed and
used. One aspect of this relationship is the neces-
sity of the role of the psychology researcher on
both ends of big data. That is, psychology is a
theory-driven field, where data are collected in
light of a set of hypotheses, and analyzed as either
supporting or rejecting those hypotheses. Big data
offers endless opportunities for exploration and
4 Psychology

discovery in other fields, such as creating word Further Readings

clouds from various forms of social media to
determine what topics are trending, but solid psy- Cowan, N. (2004). Working memory capacity. New York:
Psychology Press.
chological experiments are driven by a priori
Endsley, M. R. (2000). Theoretical underpinnings of situ-
ideas, rather than data exploration. Thus, psychol- ation awareness: A critical review. In Situation aware-
ogy is important to help big data researchers learn ness analysis and measurement. Mahwah, NJ:
how to best process their data, and many types of Lawrence Erlbaum Associates.
Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis.
psychological data can be big data, but the impor-
Cambridge, MA: MIT-press.
tance of theory, hypotheses, and the role of the Lewis, T. G. (2011). Network science: Theory and appli-
researcher will always be integral in how psychol- cations. Hoboken: Wiley.
ogy and big data interact. Neisser, U. (1976). Cognition and reality: Principles and
implications of cognitive psychology. San Francisco:
W.H. Freeman and Co.
Newell, A. (1990). Unified theories of cognition. Cam-
bridge, MA: Harvard University Press.
Cross-References Newell, A., & Simon, H. (1972). Human problem solving.
Englewood Cliffs: Prentice-Hall.
▶ Artificial Intelligence Pang, B., & Lee, L. (2008). Opinion mining and sentiment
▶ Communications analysis. Foundations and Trends in Information
Retrieval, 2(1–2), 1–35.
▶ Decision Theory Pentland, A. (2014). Social physics: How good ideas
▶ Social Media spread – The lessons from a new science. New York:
▶ Social Network Analysis Penguin Press.
▶ Social Sciences Yarkoni, T. (2012). Psychoinformatics new horizons at the
interface of the psychological and computing sciences.
▶ Spatial Analytics Current Directions in Psychological Science, 21(6),
▶ Visualization 391–397.
R
Regression
Qinghua Yang
Department of Communication Studies, Texas
Christian University, Fort Worth, TX, USA
Regression is a statistical tool to estimate the
relationship(s) between a dependent variable
(y or outcome variable) and one or more indepen-
dent variables (x or predicting variables; Fox
2008). More specifically, regression analysis
helps in understanding the variation in a depen-
dent variable using the variation in independent
variables with other confounding variable(s)
controlled. Regression analysis is widely used to
make prediction and estimation of the conditional
expectation of the dependent variable given the
independent variables, where its use overlaps with
the field of machine learning. Figure 1 shows how
crime rate is related to residents’ poverty level and
predicts the crime rate of a specific community.
We know from this regression that there is a
positive linear relationship between the crime
rate (y axis) and residents’ poverty level (x axis).
Given the poverty index of a specific community,
we are able to make a prediction of the crime rate
at that area.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_174-1
Linear Regression
The estimation target of regression is a function
that predicts the dependent variable based upon
values of the independent variables, which is
called the regression function. For simple linear
regressions, the function can be represented as
yi = a + bxi + ei. The function of multiple lin-
ear regressions is yi = b0 + b1x1 + b2x2 þ   
þ bkxk + ei where k is the number of independent
variables. The regression estimation using ordi-
nary least squares (OLS) selects the line with the
lowest total sum of squared residuals. The propor-
tion of total variation (SST) that is explained by
the regression (SSR) is known as the coefficient
of determination, often referred to as R2, a value
ranging between 0 and 1 with a higher value
indicating a better regression model (Keith 2015).
Nonlinear Regression
In the real world, there are much more nonlinear
functions than linear ones. For example, the rela-
tionship between x and y can be fitted in a qua-
dratic function shown in Figure 2. There are in
general two ways to deal with nonlinear models.
First, nonlinear models can be approximated with
linear functions. Both nonlinear functions in
Figure 2 can be approximated by two linear func-
tions according to the slope: the first linear regres-
sion function is from the beginning of the
semester to the final exam, and the second
2 Regression
50
25
crime
–25
–50
–1.00 –.50
Regression, Figure 1 Linear regression of crime rate and residents’ poverty level
function is from the final to the end of the semes-
ter. Similarly, regarding cubic, quartic, and more
complicated regressions, they can also be approx-
imated with a sequence of linear functions. How-
ever, analyzing nonlinear models in this way can
produce much residual and leave considerable
variance unexplained. The second way is consid-
ered better than the first one from this aspect, by
including nonlinear terms in the regression func-
tion as ^y = a þ b1x þ b2x2. As the graph of a
quadratic function is a parabola, if b2 < 0, the
parabola opens downward, and if b2 > 0, the
parabola opens upward. Instead of having x2 in
the model, the nonlinearity can also be presented
pffiffiffi
in many other ways, such as x, ln(x), sin(x),
cos(x), and so on. However, which nonlinear
model to choose should be based on both theory
or former research and the R2.
Logistic Regression
When the outcome variable is dichotomous (e.g.,
yes/no, success/failure, survived/died, accept/
reject), logistic regression is applied to make
–.00 .50 1.00 1.50
poverty_sqrt
prediction of the outcome variable. In logistic
regression, we predict the odds or log-odds
(logit) that a certain condition will or will not
happen. Odds range from 0 to infinity and are a
ratio of the chance of an event (p) divided by the
chance of the event not happening, that is, p/
(1p). Log-odds (logits) are transformed odds,
ln[p/(1p)], and range from negative to positive
infinity. The relationship predicting probability
using x follows an S-shaped curve as shown in
Figure 3. The shape of curve above is called a
“logistic curve.” This is defined as
expðb0 þb1 xi þei Þ
pð y i Þ ¼ . In this logistic regression,
1þexpðb0 þb1 xi þei Þ
the value predicted by the equation is a log-odds
or logit. This means when we run logistic regres-
sion and get coefficients, the values the equation
produces are logits. Odds is computed as exp
expðlogitÞ
(logit), and probability is computed as 1þexp ðlogitÞ.
Another model used to predict binary outcome is
the probit model, with the difference between
logistic and probit models lying in the assumption
about the distribution of errors: while the logit
model assumes standard logistic distribution of
errors, probit model assumes normal distribution
Regression
Regression, Anxiety
Figure 2 Nonlinear
regression models
Confidence in
the Subject
of errors (Chumney & Simpson 2006). Despite
the difference in assumption, the predictive results
using these two models are very similar. When the
outcome variable has multiple categories, multi-
nomial logistic regression or ordered logistic
regression should be implemented depending on
whether the dependent variable is nominal or
ordinal.
Regression in Big Data
Due to the advanced technologies that have been
increasingly used in data collection and the vast
amount of user-generated data, the amount of data
will continue to increase at a rapid pace, along
with a growing accumulation of scholarly works.
The explosion of knowledge makes big data one
of new research frontiers with an extensive num-
ber of application areas affected by big data, such
as public health, social science, finance, geogra-
phy, and so on. The high volume and complex
structure of big data bring statisticians both
3
Semester Mid-term Final Semester
begins ends
Semester Mid-term Final Semester
begins ends
opportunities and challenges. Generally speaking,
big data is a collection of large-scale and complex
data sets that are difficult to be processed and
analyzed using traditional data analytic tools.
Inspired by the advent of machine learning and
other disciplines, statistical learning has
emerged as a new subfield in statistics, including
supervised and unsupervised statistical learn-
ing (James, Witten, Hastie, & Tibshirani, 2013).
Supervised statistical learning refers to a set of
approaches for estimating the function f based on
the observed data points, to understand the rela-
tionship between Y and X = (X1, X2, . . . , XP),
which can be represented as Y = f(X) þ e. Since
the two main purposes for the estimation are to
make prediction and inference, which regression
modeling is widely used for, many classical sta-
tistical learning methods use regression models,
such as linear, nonlinear, and logistic regression,
with the selection of specific regression model
based on research question and data structure. In
contrast, for unsupervised statistical learning,
there is no response variable to predict for every
4 Regression

Regression,
Figure 3 Logistic 1.00
regression models

0.80

0.60

pass
0.40

0.20

0.00

0 2 4 6 8 10
X

observation that can supervise our analysis (James ▶ Statistics

et al. 2013). Additionally, more methods have
been recently developed, such as Bayesian and
Markov chain Monte Carlo (MCMC). Bayes-
ian approach, distinct from the frequentist
approach, treats model parameters as random
and models them via distributions. MCMC is
statistical sampling investigations that involve
sample data generation to obtain empirical sam-
pling distributions based on constructing a Mar-
kov chain that has the desired
distribution (Bandalos & Leite 2013).
Cross-References
▶ Data Mining Algorithms
▶ Machine Learning
▶ Statistical Analysis
Further Readings
Bandalos, D. L., & Leite, W. (2013). Use of Monte Carlo
studies in structural equation modeling research. In
G. R. Hancock & R. O. Mueller (Eds.), Structural
equation modeling: A second course (pp. 625-666).
Charlotte, NC: Information Age Publishing.
Chumney, E. C., & Simpson, K. N. (2006). Methods and
designs for outcomes research. Bethesda, MD: ASHP.
Fox, J. (2008). Applied regression analysis and general-
ized linear models. Thousand Oaks, CA: Sage.
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013).
An introduction to statistical learning (Vol. 6).
New York, NY: Springer.
Keith, T. Z. (2015). Multiple regression and beyond: An
introduction to multiple regression and structural
equation modeling. New York, NY: Routledge.
R
Religion
Matthew Pittman and Kim Sheehan
School of Journalism & Communication,
University of Oregon, Eugene, OR, USA
In his work on the changing nature of religion in
our modern mediated age, Stewart Hoover notes
that religion today is much more commodified,
therapeutic, public, and personalized than it has
been for most of history. He also notes that,
because media are coming together to create an
environment in which our personal projects of
identity, meaning, and self are worked out, reli-
gion and media are actually converging. As more
people around the globe obtain devices capable of
accessing the Internet, their everyday religious
practices are leaving digital traces for interested
companies and institutions to pick up on. The age
of big data is usually thought to affect institutions
like education, mass media, or law, but religion is
undergoing dynamic shifts as well.
Though religious practice was thought to be in
decline through the end of the twentieth century,
there has been a resurgence of interest through the
beginning of the 21st. A Google NGram viewer
(which keeps track of a word’s frequency in
published books and general literature over time)
shows that “data” surpassed “God” for the first
time in 1973. Yet, by about 2004, God once again
overtook data (and its synonym “information”),
indicating that despite incredible scientific and
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_175-1
technological advances, people still wrestle with
spiritual or existential matters.
While the term “big data” seems commonplace
now, it is a fairly recent development. Several
researchers and authors claim to have coined the
term, but its modern usage took off in the mid-
1990s and only really became mainstream in 2012
when the White House and the Davos World
Economic Forum identified it as a serious issue
worth tackling. Big data is a broad term, but
generally has two main precepts: humans are
now producing information at an unprecedented
rate, and new methods of analysis are needed to
make sense of that information. Religious prac-
tices are changing in both of these areas. Faith-
based activity is creating new data streams even as
churches, temples, and mosques are figuring out
what to do with all that data. On an institutional
level, the age of big data is giving religious groups
new ways to learn about the individuals who
adhere to their teachings. On an individual level,
technology is changing how people across the
globe learn about, discuss, and practice their
faiths.
Institutional Religion
It is now common for religious institutions to
using digital technology to reach their believers.
Like any other business or group that needs mem-
bers to survive, most seek to utilize or leverage
new devices and trends into opportunities to
2 Religion
strengthen existing members or recruit potential
new ones. Of course, depending on a religion’s
stance toward culture, they may (like the Amish)
eschew some technology. However, for most
mosques, churches, and synagogues, it has
become standard for each to have its own website
or Facebook page. Email newsletters and Twitter
accounts feeds have replaced traditional newslet-
ters and event reminders.
New opportunities are constantly emerging
that create novel space for leaders to engage prac-
titioners. Religious leaders can communicate
directly with followers through social media,
adding a personal touch to digital messages,
which can sometimes feel distant or cold. Rabbi
SchmuleyBoteach, “America’s Rabbi,” has
29 best-selling books but often communicates
daily though his Twitter account, which has over
a hundred thousand followers. On the flip side,
people can thoroughly vet potential religious
leaders or organizations before committing to
them. If concerned that a particular group’s ideol-
ogy might not align with one’s own, a quick
Internet search or trip to the group’s website
should identify any potential conflicts. In this
way, providing data about their identity and
beliefs helps religious groups differentiate
themselves.
In a sense, big data makes it possible for reli-
gious institutions to function more like – and take
their cues from – commercial enterprises. Track-
ing streams of information about its followers can
help religious groups be more in tune with the
wants and needs of these “customers.” Some reli-
gious organizations implement the retail practice
of “tweets and seats”: by ensuring that members
always have available places to sit, rest, or hang
out, and that wifi (wireless Internet connectivity)
is always accessible, they hope to keep people
present and engaged. Not all congregations
embrace this change, but the clear cultural trend
is toward ubiquitous smart phone connectivity.
Religious groups that take advantage of this may
provide several benefits to their followers: mem-
bers could immediately identify and download
any worship music being played; interested mem-
bers could look up information about a local reli-
gious leader; members could sign up for events
and groups as they are announced in the service;
or those using online scripture software can access
texts and take notes. There are just a few
possibilities.
There are other ways religious groups can har-
ness big data. Some churches have begun analyz-
ing liturgies to assess and track length and content
over time. For example, a dip in attendance during
a given month might be linked to the sermons
being 40% longer in that same time frame. Many
churches make their budgets available to mem-
bers for the sake of transparency, and in a digital
age it is not difficult to create financial records that
are clear and accessible to laypeople. Finally,
learning from a congregant’s social media profiles
and personal information, a church might remind
a parishioner of her daughter’s upcoming birth-
day, the approaching deadline for an application
to a family retreat, or when other congregants are
attending a sporting event of which she is a fan.
The risk of overstepping boundaries is real and,
just like with Facebook or similar entities, privacy
settings should be negotiated beforehand.
As with other commercial entities, religious
institutions utilizing big data must learn to differ-
entiate information they need from information
they don’t. The sheer volume of available data
makes distinguishing desired signal from irrele-
vant noise an increasingly important task. Ran-
dom correlations may lead to false positive
causation. A mosque may benefit from learning
that members with the highest income are not
actually its biggest givers, or testing for a relation-
ship between how far away its members live and
how often they attend. Each religious group must
determine how big data may or may not benefit its
operation in any given endeavor, and the oppor-
tunities are growing.
Individual Religion
The everyday practice of religion is becoming
easier to track as it increasingly utilizes digital
technology. A religious individual’s personal
blog, Twitter feed, Facebook profile keep a record
of his or her activity or beliefs, making it relatively
easy for any interested entity to track online
Religion
behavior over time. Producers and advertisers use
this data to promote products, events, or websites
to people who might be interested. Currently com-
panies like Amazon have more incentive than,
say, a local synagogue in keeping tabs on what
websites one visits, but the potential exists for
religious groups to access the same data that
Facebook, Amazon, Google, etc. already possess.
Culturally progressive religious groups antici-
pate mutually beneficial scenarios: they provide a
data service that benefits personal spiritual
growth, and in turn the members generate fields
of data that are of great value to the group. A Sikh
coalition created the FlyRights app in 2012 to help
with quick reporting of discriminatory TSA pro-
filing while travelling. The Muslim’s Prayer
Times app provides a compass, calendar (with
moon phases), and reminders for Muslims about
when and in what direction to pray. Apple’s
app store has also had to ban other apps from
fringe religious groups or individuals for being
too irreverent or offensive.
The most popular religious app to date simply
provides access to scripture. In 2008 LifeChurch.
tv launched “the Bible app,” also called
YouVersion, and it currently has over 151 million
installations worldwide on smartphones and tab-
lets. Users can access scripture (in over 90 differ-
ent translations) while online or download it for
access offline. An audio recording of each chapter
being read aloud can also be downloaded for some
of the translations. A user can search through
scripture by keyword, phrase, or book of the
Bible, or there are reading plans of varying levels
of intensity and access to related videos or
movies. A “live” option lets users search out
churches and events in surrounding geographic
areas, and a sharing option lets users promote the
app, post to social media what they have read, or
share personal notes directly to friends. The digi-
tal highlights or notes made, even when using the
app offline, will later upload to one’s account and
remain in one’s digital “bible” permanently.
All this activity has generated copious amounts
of data for YouVersion’s producers. In addition to
using the data to improve their product they also
released it to the public. This kind of insight into
the personal religious behavior of so many
3
individuals is unprecedented. With over a billion
opens and/or uses, YouVersion statistically pro-
ved several phenomena. The data demonstrated
the most frequent activity for users is looking up a
favorite verse for encouragement. Despite the ste-
reotype of shirtless men at football games, the
most popular verse was not John 3:16, but Philip-
pians 4:13: “I can do all things through him who
gives me strength.” Religious adherents have
always claimed that their faith gives them strength
and hope, but big data has now provided a brief
insight into one concrete way this actually
happens.
The YouVersion data also reveal that people
used the bible to make a point in social media.
Verses were sought out and shared in an attempt to
support views on marriage equality, gender roles,
or other divisive topics. Tracking how individuals
claim to have their beliefs supported by scripture
may help religious leaders learn more about how
these beliefs are formed, how they change over
time, and which interpretations of scripture are
most influential. Finally, YouVersion data reveal
that Christian users like verses with simple mes-
sages, but chapters with profound ideas. Verses
are easier to memorize when they are short and
unique, but when engaging in sustained reading,
believers prefer chapters with more depth.
Whether large data sets confirm suspicions or
shatter expectations, they continue to change the
way religion is practiced and understood.
Numerous or Numinous
In the past, spiritual individuals had a few reli-
gions to choose from, but the globalizing force of
technology has dramatically increased the avail-
able options. While the three big monotheisms
(Christianity, Judaism, and Islam) and
pan/polytheisms (Hinduism and Buddhism) are
still the most popular, the Internet has made it
possible for people of any faith, sect, or belief to
find each other and validate their practice. Though
pluralism is not embraced in every culture, there is
at least increasing awareness of the many ways
religion is practiced across the globe.
4 Religion
Additionally, more and more people are iden-
tifying themselves as “spiritual but not religious,”
indicating a desire to seek out spiritual experi-
ences and questions outside the confines of a
traditional religion. Thus for discursive activities
centered on religion, Daniel Stout advocates the
use of another term in addition to “religion”:
numinous. Because “religious” can have negative
or limiting connotations, looking for the “numi-
nous” in cultural texts or trends can broaden the
search for and dialogue about a given topic. To be
numinous, something must meet several criteria:
stir deep feeling (affect), spark belief (cognition),
include ritual (behavior), and be done with fellow
believers (community). This four-part framework
is a helpful tool for identification of numinous
activity in a society where it once might have
been labeled “religious.”
By this definition, the Internet (in general) and
entertainment media (in particular) all contain
numinous potential. The flexibility of the Internet
makes it relevant to the needs of most; while
authority of some of its sources can be dubious,
the ease of social networking and multi-mediated
experiences provides all the elements of tradi-
tional religion (community, ritual, belief, feeling).
Entertainment media, which produce at least as
much data as – and may be indistinguishable
from – religious media, emphasize universal
truths through storytelling. The growing opportu-
nities of big data (and its practical analysis) will
continue to offer for those who engage in numi-
nous and religious behavior.
Cross-References
▶ Data Monetization
▶ Digitization
▶ Entertainment
▶ Internet
▶ Text Analytics
Further Readings
Campbell, H. A. (Ed.). (2012). Digital religion: Under-
standing religious practice in new media worlds.
Abingdon: Routledge.
Hjarvard, S. (2008). The mediatization of religion:
A theory of the media as agents of religious change.
Northern Lights: Film & Media Studies Yearbook, 6(1),
9–26.
Hoover, S. M., & Lundby, K. (Eds.). (1997). Rethinking
media, religion, and culture (Vol. 23). Thousand Oaks:
Sage.
Kuruvilla, C. Religious mobile apps changing the faith-
based landscape in America. Retrieved from http://
www.nydailynews.com/news/national/gutenberg-
moment-mobile-apps-changing-america-religious-
landscape-article-1.1527004. Accessed Sep 2014.
Mayer-Schönberger, V., & Cukier, K. (2013). Big data:
A revolution that will transform how we live, work, and
think. Houghton Mifflin Harcourt.
Taylor, B. (2008). Entertainment theology (cultural exege-
sis): New-edge spirituality in a digital democracy.
Baker Books.
R
Risk Analysis
Jonathan Z. Bakdash
Human Research and Engineering Directorate, U.
S. Army Research Laboratory, Aberdeen Proving
Ground, MD, USA
Definition and Introduction
Society is becoming increasingly interconnected
with networks linking people, the environment,
information, and technology. This rising com-
plexity is a challenge for risk analysis. Risk anal-
ysis is the identification and evaluation of the
probability of an adverse outcome, its associated
risk factors, and the potential impact if that out-
come occurs. Successfully modeling risk within
interdependent and complex systems requires
access to considerably more data than traditional,
simple risk models. The increasing availability of
big data offers enormous promise for improving
risk analysis through more detailed, comprehen-
sive, faster, and accurate predictions of risks and
their impacts than small data alone.
However, risk analysis is not purely a compu-
tational challenge that can be solved by more data.
Big data does not eliminate the importance of data
quality and modeling assumptions; it is not nec-
essarily a replacement for small data. Further-
more, traditional risk analysis methods typically
underestimate the probability and impact of risks
(e.g., terrorist attacks, power failures, and natural
# Springer International Publishing AG (outside the USA) 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_176-1
disasters such as hurricanes) because normal data
and independent observations are assumed. Tra-
ditional methods also typically do not account for
cascading failures, which are not uncommon in
complex systems. For example, a hurricane may
cause a power failure, which in turn results in
flooding.
The blessing and curse of risk analysis with big
data are illustrated by the example of Google Flu
Trends (GFT). Initially, it was highly successful in
estimating flu rates in real time, but over time it
became inaccurate due to external factors, lack of
continued validation, and incorrect modeling
assumptions.
Interdependencies
Globalization and advances in technology have
led to highly networked and interdependent
social, economic, political, natural, and techno-
logical systems (Helbing 2013). Strong interde-
pendencies are potentially dangerous because
small or gradual changes in a single system can
cause cascading failures throughout multiple sys-
tems. For example, climate change is associated
with food availability, food availability with eco-
nomic disparity, and economic disparity with war.
In interconnected systems, risks often spread
quickly in a cascading process, so early detection
and mitigation of risks is critical to stopping fail-
ures before they become uncontrollable. Helbing
(2013) contends that big data is necessary to
2 Risk Analysis
model risks in interconnected and complex sys-
tems: Capturing interdependent dynamics and
other properties of systems requires vast amounts
of heterogeneous data over space and time.
Interdependencies are also critical to risk anal-
ysis because even when risks are mitigated, they
may still cause amplifying negative effects
because of human risk perception. Perceived risk
is the public social, political, and economic
impacts of unrealized (and realized) risks. An
example of the impact of a perceived risk is the
nuclear power accident at Three-Mile Island. In
this accident, minimal radiation was released so
the real risk was mitigated. Nevertheless, the near
miss of a nuclear meltdown had immense social
and political consequences that continue to nega-
tively impact the nuclear power industry in the
United States. The realized consequences of per-
ceived risk mean that “real” risk should not nec-
essarily be separated from “perceived” risk.
Data: Quality and Sources
Many of the analysis challenges for big data are
not unique but are pertinent to analysis of all data
(Lazer et al. 2014). Regardless of the size of the
dataset, it is important for analysts and
policymakers to understand how, why, when,
and where the data were collected and what the
data contain and do not contain. Big data may be
“poor data” because rules, causality, and out-
comes are far less clear compared to small data.
More specifically, Vose (2008) describes the
quality of data characteristics for risk analysis.
The highest quality data are obtained using a
large sample of direct and independent measure-
ments collected and analyzed using established
best practices over a long period of time and
continually validated to correct data for errors.
The second highest quality data use proxy mea-
sures, a widely used method for collection, anal-
ysis, and some validation. Other characteristics of
decreasing data quality are: A smaller sample of
objective data, agreement among multiple
experts, a single expert opinion, and is weakest
with speculation. While there may be some situa-
tions in which expert opinions are the only data
source, general findings indicate this type of data
has poor predictive accuracy. Additional reasons
to question experts are situations or systems with a
large number of unknown factors and potentially
catastrophic impacts for erroneous estimations.
Big data can be an improvement over small data
and one or several expert opinions. However,
volume is not necessarily the same as quality.
Multidimensional aspects of data quality, whether
the data are big or small, should always be
considered.
Risk Analysis Methods
Vose (2008) explains the general techniques for
conducting risk analysis. A common, descriptive
method for risk analysis is Probability-Impact (P-
I). P-I is the probability of a risk occurring multi-
plied by the impact of the risk if it materializes:
Probability  Impact = Weighted Risk. All values
may be either qualitative (e.g., low, medium, and
high likelihood or severity) or quantitative (e.g.,
10% or one million dollars). The Probability may
be a single value or multiple values such as a
distribution of probabilities. The Impact may
also be a single value or multiple values and is
usually expressed as money. A similar weighted
model to P-I, Threat  Vulnerability  Conse-
quence = Risk, is frequently used in risk analysis.
However, a significant weakness with P-I and
related models with fixed values is that they tend
to systematically underestimate the probability
and impact of rare events that are interconnected,
such as natural hazards (e.g., floods), protection of
infrastructure (e.g., power grid), and terrorist
attacks. Nevertheless, the P-I method can be effec-
tive for quick risk assessments.
Probabilistic Risk Assessment
P-I is a foundation for Probabilistic Risk Assess-
ment (PRA), an evaluation of the probabilities for
multiple potential risks and their respective
impacts. The US Army’s standardized risk matrix
is an example of qualitative PRA, see Fig. 1 (also
see Level 5 of risk analysis below).
The risk matrix is constructed by:
Risk Analysis 3

Risk Analysis, Fig. 1 Risk analysis (Source: Safety Risk Management, Pamphlet 385-30 (Headquarters, Department of
the Army, 2014, p. 8): www.apd.army.mil/pdffiles/p385_30.pdf)

Step 1: Identifying possible hazards (i.e., potential
risks)
Step 2: Estimating the probabilities and impacts of
each risk and using the P-Is to categorize
weighted risk
Risk analysis informs risk reduction, but they
are not one and the same. After the risk matrix is
constructed, appropriate risk tolerance and miti-
gation strategies are considered. The last step is
ongoing supervision and evaluation of risk as
conditions and information change, updating the
risk matrix as needed, and provided feedback to
improve the accuracy of future risk matrix.
Other widely used techniques include inferen-
tial statistical tests (e.g., regression) and the more
comprehensive approach of what-if data simula-
tions, which are also used in catastrophe model-
ing. Big data may improve the accuracy of
probability and impact estimates, particularly the
upper bounds in catastrophe modeling, leading to
more accurate risk analysis.
4 Risk Analysis
From a statistical perspective, uncertainty and
variability tend to be interchangeable. If uncer-
tainty can be attributed to random variability,
there is no distinction. However, in risk analysis,
uncertainty can arise from incomplete knowledge
(Paté-Cornell 1996). Uncertainty in risk may be
due to a lack of data (particularly for rare events),
not knowing relevant risks and/or impacts and
unknown interdependencies among risks and/or
impacts.
Levels of Risk Analysis
There are six levels for understanding uncertainty,
ranging from qualitative identification of risk fac-
tors (Level 0) to multiple risk curves constructed
using different PRAs (Level 5) (Paté-Cornell
1996). Big data are relevant to Level 2 and
beyond. The specific levels are as follows
(adapted Paté-Cornell 1996):
Level 0: Identification of a hazard or failure
modes. Level 0 is primarily qualitative. For
example, does exposure to a chemical increase
the risk of cancer?
Level 1: Worst case. Level 1 is also qualitative,
with no explicit probability. For example, if
individuals are exposed to a cancer-causing
chemical, what is the highest number that
could develop cancer?
Level 2: Quasi-worst case (probabilistic upper-
bound). Level 2 introduces subjective estima-
tion of probability based on reasonable expec-
tation(s). Using the example from Level 1, this
could be the 95th percentile for the number of
individuals developing cancer.
Level 3: Best and central estimates. Rather than a
worst case, Level 3 aims to model the most
likely impact using central values (e.g., mean
or median).
Level 4: Single-curve PRA. Previous levels were
point estimates of risk; Level 4 is a type of
PRA. For example, what is the number of
individuals that will develop cancer across a
probability distribution?
Level 5: Multiple-curve PRA. Level 5 has more
than one probabilistic risk curve. Using the
cancer risk example, different probabilities
from distinct data can be represented using
multiple curves, which are then combined
using the average or another measure. A
generic example of Level 5, for qualitative
values, was illustrated with the above risk
matrix. When implemented quantitatively,
Level 5 is similar to what-if simulations in
catastrophe modeling.
Catastrophe Modeling
Big data may improve risk analysis at Level 2 and
above but may be particularly informative for
modeling multiple risks at Level 5. Using catas-
trophe modeling, big data can allow for a more
comprehensive analysis of the combinations of P-
Is while taking into account interdependences
among systems. Catastrophe modeling involves
running a large number of simulations to construct
a landscape of risk probabilities and their impacts
for events such as terrorist attacks, natural disas-
ters, and economic failures. Insurance, finance,
other industries, and governments are increas-
ingly relying on big data to identify and mitigate
interconnected risks using catastrophe modeling.
Beiser (2008) describes the high level of data
detail in catastrophe modeling. For risk analysis of
a terrorist attack in a particular location,
interconnected variables taken into account may
include the proximity to high-profile targets (e.g.,
government buildings, airports, and landmarks),
the city, and details of the surrounding buildings
(e.g., construction materials), as well as the poten-
tial size and impact of an attack. Simulations are
run under different assumptions, including the
likelihood of acquiring materials to carry out a
particular type of attack (e.g., a conventional
bomb versus a biological weapon) and the proba-
bility of detecting the acquisition of such mate-
rials. Big data is informative for the wide range of
possible outcomes and their impacts in terms of
projected loss of life and property damage. How-
ever, risk analysis methods are only as good as
their assumptions, regardless of the amount of
data.
Assumptions: Cascading Failures
Even with big data, risk analysis can be flawed
due to inappropriate model assumptions. In the
case of Hurricane Katrina, the model assumptions
Risk Analysis
for a Category 3 hurricane did specify a large,
slow-moving storm system with heavy rainfall
nor did they account for the interdependencies in
infrastructure systems. This storm caused early
loss of electrical power, so many of the pumping
stations for levees could not operate. Conse-
quently, water overflowed, causing breaches,
resulting in widespread flooding. Because of cas-
cading effects in interconnected systems, risk
probabilities and impacts are generally far greater
than in independent systems and therefore will be
substantially underestimated when incorrectly
treated as independent.
Right Then Wrong: Google Flu Trends
GFT is an example of both success and failure for
risk analysis using big data. The information pro-
vided by an effective disease surveillance tool can
help mitigate disease spread by reducing illnesses
and fatalities. Initially, GFT was a successful real-
time predictor of flu prevalence, but over time, it
becomes inaccurate. This is because the model
assumptions did not hold over time, validation
with small data was not on-going, and it lacked
transparency. GFT used a data-mining approach
to estimate real-time flu rates: Hundreds of mil-
lions of possible models were tested to determine
the best fit of millions of Google searches to
traditional weekly surveillance data. The tradi-
tional weekly surveillance data consisted of the
proportion of reported doctor visits for flu-like
systems. At first, GFT was a timely and accurate
predictor of flu prevalence, but it began to produce
systematic overestimates, sometimes by a factor
of two or greater compared with the gold-standard
of traditional surveillance data. The erroneous
estimates from GFT resulted from a lack of con-
tinued validation, thus assuming relevant search
terms only changed as a result of flu symptoms
and transparency in the data and algorithms used.
Lazer et al. (2014) called the inaccuracy of
GFT a parable for big data, highlighting several
key points. First, a key cause for the misestimates
was that the algorithm assumed that influences on
search patterns were the same over time and pri-
marily driven by the onset of flu symptoms. In
5
reality, searches were likely influenced by exter-
nal events such as media reporting of a possible
flu pandemic, seasonal increases in searches for
cold symptoms that were similar to flu symptoms,
and the introduction of suggestions in Google
Search. Therefore, GFT wrongly assumed the
data were stationary (i.e., no trends or changes in
the mean and variance of data over time). Second,
Google did not provide sufficient information for
understanding the analysis, such as all selected
search terms and access to the raw data and algo-
rithms. Third, big data is not necessarily a replace-
ment for small data. Critically, the increased
volume of data does not necessarily make it the
highest quality source. Despite these issues, GFT
was at the second highest level of data quality
using criteria from Vose (2008) because GFT ini-
tially used:
1. Proxy measures: search terms originally corre-
lated with local flu reports over a finite period
of time
2. A common method: search terms used for
Internet advertising, disease surveillance was
novel (with limited validation)
In the case of GFT, the combination of big and
small data, by continuously recalibrating the algo-
rithms for the big data using the small (surveil-
lance) data, would have been much more accurate
than either alone. Moreover, big data can make
powerful predictions that are impossible with
small data alone. For example, GFT could provide
estimates of flu prevalence in local geographic
areas using detailed spatial and temporal informa-
tion from searches; this would be impossible with
only the aggregated traditional surveillance data.
Conclusions
Similar to GFT, many popular techniques for ana-
lyzing big data use data mining to automatically
uncover hidden structures. Data mining tech-
niques are valuable for identifying patterns in
big data but should be interpreted with caution.
The dimensions of big data do not obviate con-
siderations of data quality, the need for continuous
6 Risk Analysis

validation, and the importance of modeling References

assumptions (e.g., non-normality, non-
stationarity, and non-independence). While big Beiser, V. (2008). Pricing terrorism: Insurers gauge risks,
costs, Wired. Permanent link: http://web.archive.org/
data has enormous potential to improve the accu-
save/_embed/http://www.wired.com/2008/06/pb-
racy and insights of risk analysis, particularly for terrorism/
interdependent systems, it is not necessarily a Helbing, D. (2013). Globally networked risks and how to
replacement for small data. respond. Nature, 497(7447), 51–59. doi:10.1038/
nature12047.
Lazer, D. M., Kennedy, R., King, G., & Vespignani, A.
(2014). The parable of Google flu: Traps in big data
Cross-References analysis. Science, 343(6176), 1203–1206. doi:10.1126/
science.1248506.
Paté-Cornell, M. E. (1996). Uncertainties in risk analysis:
▶ Complex Networks
Six levels of treatment. Reliability Engineering & Sys-
▶ Google Flu tem Safety, 54(2), 95–111. doi:10.1016/S0951-8320
▶ Military Operations (Counter-Intelligence and (96)00067-1.
Counter-Terrorism) Vose, D. (2008). Risk analysis: A quantitative guide (3rd
ed.). West Sussex: Wiley.
▶ Small Data
▶ Statistical Analysis
U
Upturn
Katherine Fink
Department of Media, Communications, and
Visual Arts, Pace University, Pleasantville, NY,
USA
Introduction
Upturn is a think tank that focuses on the impact
of big data on civil rights. Founded in 2011 as
Robinson þ Yu, the organization announced a
name change in 2015 and expansion of its staff
from two to five people. The firm’s work
addresses issues such as criminal justice, lending,
voting, health, free expression, employment, and
education. Upturn recommends policy changes
with the aim of ensuring that institutions use tech-
nology in accordance with shared public values.
The firm has published white papers, academic
articles, and an online newsletter targeting
policymakers and civil rights advocates.
Background
Principals of Upturn include experts in law, public
policy, and software engineering. David Robinson
was formerly the founding Associate Director of
Princeton University’s Center for Information
Technology Policy, which conducts interdisci-
plinary research in computer science and public
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_177-1
policy. Robinson holds a JD from Yale
University’s Law School and has reported for the
Wall Street Journal and The American, an online
magazine published by the American Enterprise
Institute. Harlan Yu holds a PhD in Computer
Science from Princeton University, where he
developed software to make court records more
accessible online. He has also advised the US
Department of Labor on open government poli-
cies and analyzed privacy, advertising, and broad-
band access issues for Google. Aaron Rieke has a
JD from the University of California Berkeley’s
Law School and has worked for the Federal Trade
Commission and the Center for Democracy and
Technology on data security and privacy issues.
Cofounders Robinson and Yu began their col-
laboration at Princeton University as researchers
on government transparency and civic engage-
ment. They were among four coauthors of the
2009 Yale Journal of Law & Technology article
“Government Data and the Invisible Hand,”
which argued that the government should priori-
tize opening access to more of its data rather than
creating websites. The article suggested that “pri-
vate parties in a vibrant marketplace of engineer-
ing ideas” were better suited to develop websites
that could help the public access government data.
In 2012, Robinson and Yu coauthored the UCLA
Law Review article “The New Ambiguity of
‘Open Government,’” in which they argued that
making data more available to the public did not
by itself make government more accountable. The
article recommended separating the notion of
2 Upturn
open government from the technologies of open
data in order to clarify the potential impacts of
public policies on civic life.
Criminal Justice
Upturn has worked with the Leadership Confer-
ence, a coalition of civil rights and media justice
organizations, to evaluate police department pol-
icies on the use of body-worn cameras. The orga-
nizations, noting increased interest in the use of
such cameras following police-involved deaths in
communities such as Ferguson (Missouri), New
York City, and Baltimore, also cautioned that
body-worn cameras could be used for surveil-
lance, rather than protection, of vulnerable indi-
viduals. The organizations released a scorecard on
body-worn camera policies of 25 police depart-
ments in November 2015. The scorecard included
criteria such as whether body-worn camera poli-
cies were publicly available, whether footage was
available to people who file misconduct com-
plaints, and whether the policies limited the use
of biometric technologies to identify people in
recordings.
Lending
Upturn has warned of the use of big data by
predatory lenders to target vulnerable consumers.
In a 2015 report, “Led Astray,” Upturn explained
how businesses used online lead generation to sell
risky payday loans to desperate borrowers. In
some cases, Upturn found that the companies
violated laws against predatory lending. Upturn
also found some lenders exposed their customers’
sensitive financial data to identity thieves. The
report recommended that Google, Bing, and
other online platforms tighten restrictions on pay-
day loan ads. It also called on the lending industry
to promote best practices for online lead genera-
tion and for greater oversight of the industry by
the Federal Trade Commission and Consumer
Financial Protection Bureau.
Robinson þ Yu researched the effects of the
use of big data in credit scoring in a guide for
policymakers titled “Knowing the Score.” The
guide endorsed the most widely used credit scor-
ing methods, including FICO, while acknowledg-
ing concerns about disparities in scoring among
racial groups. The guide concluded that the scor-
ing methods themselves were not discriminatory,
but that the disparities rather reflected other under-
lying societal inequalities. Still, the guide advo-
cated some changes to credit scoring methods.
One recommendation was to include “mainstream
alternative data” such as utility bill payments in
order to allow more people to build their credit
files. The guide expressed reservations about
“nontraditional” data sources, such as social net-
work data and the rate at which users scroll
through terms of service agreements. Robinson
þ Yu also called for more collaboration among
financial advocates and the credit industry, since
much of the data on credit scoring is proprietary.
Finally, Robinson þ Yu advocated that govern-
ment regulators more actively investigate “mar-
keting scores,” which are used by businesses to
target services to particular customers based on
their financial health. The guide suggested that
marketing scores appeared to be “just outside the
scope” of the Fair Credit Reporting Act, which
requires agencies to notify consumers when their
credit files have been used against them.
Voting
Robinson þ Yu partnered with Rock the Vote in
2013 in an effort to simplify online voter registra-
tion processes. The firm wrote a report,
“Connected OVR: a Simple, Durable Approach
to Online Voter Registration.” At the time of the
report, nearly 20 states had passed online voter
registration laws. Robinson þ Yu recommended
that all states allow voters to check their registra-
tion statuses in real time. It also recommended that
online registration systems offer alternatives to
users who lack state identification, and that the
systems be responsive to devices of various sizes
Upturn
and operating systems. Robinson þ Yu also
suggested that states streamline and better coordi-
nate their online registration efforts. Robinson þ
Yu recommended that states develop a simple,
standardized platform for accepting voter data
and allow third-party vendors (such as Rock the
Vote) to design interfaces that would accept voter
registrations. Outside vendors, the report
suggested, could use experimental approaches to
reach new groups of voters while still adhering to
government registration requirements.
Big Data and Civil Rights
In 2014, Robinson þ Yu advised The Leadership
Conference on “Civil Rights Principles for the Era
of Big Data.” Signatories of the document
included the American Civil Liberties Union,
Free Press, and NAACP. The document offered
guidelines for developing technologies with
social justice in mind. The principles included an
end to “high-tech profiling” of people through the
use of surveillance and sophisticated data-gather-
ing techniques, which the signatories argued
could lead to discrimination. Other principles
included fairness in algorithmic decision-making;
the preservation of core legal principles such as
the right to privacy and freedom of association;
individual control of personal data; and protec-
tions from data inaccuracies.
The “Civil Rights Principles” were cited by the
White House in its report, “Big Data: Seizing
Opportunities, Preserving Values.” John Podesta,
Counselor to President Barack Obama, cautioned
in his introduction to the report that big data had
the potential “to eclipse longstanding civil rights
protections in how personal information is used.”
Following the White House report, Robinson þ
Yu elaborated upon four areas of concern in the
white paper “Civil Rights, Big Data, and Our
Algorithmic Future.” The paper included four
chapters: Financial Inclusion, Jobs, Criminal Jus-
tice, and Government Data Collection and Use.
The Financial Inclusion chapter argued the era
of big data could result in new barriers for low-
3
income people. The automobile insurance com-
pany Progressive, for example, installed devices
in customers’ vehicles that allowed for the track-
ing of high-risk behaviors. Such behaviors
included nighttime driving. Robinson þ Yu
argued that many lower-income workers com-
muted during nighttime hours and thus might
have to pay higher rates, even if they had clean
driving records. The report also argued that mar-
keters used big data to develop extensive profiles
of consumers based on their incomes, buying
habits, and English-language proficiency, and
such profiling could lead to predatory marketing
and lending practices. Consumers often are not
aware of what data has been collected about
them and how that data is being used, since such
information is considered to be proprietary. Rob-
inson þ Yu also suggested that credit scoring
methods can disadvantage low-income people
who lack extensive credit histories.
The report found that big data could impair job
prospects in several ways. Employers used the
federal government’s E-Verify database, for
example, to determine whether job applicants
were eligible to work in the United States. The
system could return errors if names had been
entered into the database in different ways. For-
eign-born workers and women have been dispro-
portionately affected by such errors. Resolving
errors can take weeks, and employers often lack
the patience to wait. Other barriers to employment
arise from the use of automated questionnaires
some applicants must answer. Some employers
use the questionnaires to assess which potential
employees will likely stay in their jobs the lon-
gest. Some studies have suggested that longer
commute times correlate to shorter-tenured
workers. Robinson þ Yu questioned whether ask-
ing the commuting question was fair, particularly
since it could lead to discrimination against appli-
cants who lived in lower-income areas. Finally,
Robinson þ Yu raised concerns about “sublimi-
nal” effects on employers who conducted web
searches for job applicants. A Harvard researcher,
they noted, found that Google algorithms were
more likely to show advertisements for arrest
4 Upturn
records in response to web searches of “black-
identifying names” rather than “white-identifying
names.”
Robinson þ Yu found that big data had
changed approaches to criminal justice. Munici-
palities used big data in “predictive policing,” or
anti-crime efforts that targeted ex-convicts and
victims of crimes as well as their personal net-
works. Robinson þ Yu warned that these systems
could lead to police making “guilt by association”
mistakes, punishing people who had done nothing
wrong. The report also called for greater transpar-
ency in law enforcement tactics that involved
surveillance, such as the use of high-speed cam-
eras that can capture images of vehicle license
plates, and so-called stingray devices, which inter-
cept phone calls by mimicking cell phone towers.
Because of the secretive nature with which police
departments procure and use these devices, the
report contended that it was difficult to know
whether they were being used appropriately. Rob-
inson þ Yu also noted that police departments
were increasingly using body cameras and that
early studies suggested the presence of the cam-
eras could de-escalate tension during police
interactions.
The Data Government and Use chapter
suggested that big data tools developed in the
interest of national security were also being used
domestically. The DEA, for example, worked
closely with AT&amp;T to develop a secret data-
base of phone records for domestic criminal
investigations. To shield the database’s existence,
agents avoided mentioning it by name in official
documents. Robinson þ Yu warned that an abun-
dance of data and a lack of oversight could result
in abuse, citing cases in which law enforcement
workers used government data to stalk people
they knew socially or romantically. The report
also raised concerns about data collection by the
US Census Bureau, which sought to lower the
cost of its decennial count by collecting data
from government records. Robinson þ Yu cau-
tioned that the cost-cutting measure could result in
undercounting some populations.
Newsletter
Equal Future, Upturn’s online newsletter, began
in 2013 with support from the Ford Foundation.
The newsletter has highlighted news stories
related to social justice and technology. For
instance, Equal Future has covered privacy issues
related to the FBI’s Next Generation Identification
system, a massive database of biometric and other
personal data. Other stories have included a legal
dispute in which a district attorney forced
Facebook to grant access to the contents of nearly
400 user accounts. Equal Future also wrote about
an “unusually comprehensive and well-consid-
ered” California law that limited how technology
vendors could use educational data. The law was
passed in response to parental concerns about
sensitive data that could compromise their chil-
dren’s privacy or limit their future educational and
professional prospects.
Cross-References
▶ American Civil Liberties Union
▶ Biometrics
▶ Criminology and Law Enforcement
▶ Data-Driven Marketing
▶ e-commerce
▶ Federal Bureau of Investigation (FBI)
▶ Financial Services
▶ Google
▶ Governance
▶ Marketing/Advertising
▶ National Association for the Advancement of
Colored People
▶ Online Advertising
Further Readings
Civil Rights Principles for the Era of Big Data. (2014,
February). http://www.civilrights.org/press/2014/civil-
rights-principles-big-data.html
Upturn
Robinson, D., & Yu, H. (2014, October). Knowing the score:
New data, underwriting, and marketing in the consumer
credit marketplace. https://www.teamupturn.com/static/
files/Knowing_the_Score_Oct_2014_v1_1.pdf
Robinson þ Yu. (2013). Connected OVR: A simple, durable
approach to online voter registration. Rock the Vote.
http://www.issuelab.org/resource/connected_ovr_a_simp
le_durable_approach_to_online_voter_registration
Robinson, D., Yu, H., Zeller, W. P., & Felten, E. W. (2008).
Government data and the invisible hand. Yale JL &
Tech., 11, 159.
5
The Leadership Conference on Civil and Human Rights &
Upturn. (2015, November). Police body worn cameras:
A policy scorecard. https://www.bwcscorecard.org/
static/pdfs/LCCHR_Upturn-BWC_Scorecard-v1.04.pdf
Upturn. (2014, September). Civil rights, big data, and our
algorithmic future. https://bigdata.fairness.io/
Upturn. (2015, October). Led Astray: Online lead genera-
tion and payday loans. https://www.teamupturn.com/
reports/2015/led-astray
Yu, H., & Robinson, D. G. (2012). The new ambiguity of
‘open government’. UCLA L. Rev. Disc. 59, 178.
S
Salesforce
Jason Schmitt
Communication and Media, Clarkson University,
Potsdam, NY, USA
Salesforce is a global enterprise software com-
pany, with Fortune 100 standing, most well-
known for its role in linking cloud computing to
on-demand customer relationship management
(CRM) products. Salesforce CRM and marketing
products work together to make corporations
more functional and ultimately more efficient.
Founded in 1999 by Marc Benioff, Parker Harris,
Dave Moellenhoff, and Frank Domingues,
Salesforce’s varied platforms allow organizations
to understand the consumer and the varied media
conversations revolving around a business or
brand. According to Forbes (April 2011) which
conducted an assessment of businesses focused on
value to shareholders, Marc Benioff of Salesforce
was the most effective CEO in the world.
Salesforce provides a cloud-based centralized
location to track data. Contacts, accounts, sales
deals, and documents as well as corporate mes-
saging and the varied social media conversations
are all archived and retrievable within the
Salesforce architecture from any web or mobile
device without the use of any tangible software.
Salesforce’s quickly accessible information has an
end goal to optimize profitability, revenue, and
customer satisfaction by orientating the
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_179-1
organization around the customer. This ability to
track and message correctly highlights
Salersforce’s unique approach to management
practice known in software development as
Scrum.
Scrum is an incremental software development
framework for managing product development by
a development team that works as a unit to reach a
common goal. A key principle of Salesforce’s
Scrum direction is the recognition that during a
project the customers can change their minds
about what they want and need, often called
churn, and predictive understanding is hard to
accomplish. As such, Salesforce takes an empiri-
cal approach in accepting that an organization’s
problem cannot be fully understood or defined
and instead focuses on maximizing the team’s
ability to deliver messaging quickly and respond
to emerging requirements.
Salesforce provides a fully customizable user
interface for custom adoption and access for a
diverse array of organization employees. Further,
Salesforce has the ability to integrate into existing
websites and allows for building additional web
pages through the cloud-based service. Salesforce
has the ability to link with Outlook and other mail
clients to sync calendars and associate emails with
the proper contact and provides the functionality
to keep a record every time a contact or data entry
is accessed or amended. Similarly, Salesforce
keeps track and organizes customer support issues
and tracks them through to resolution with the
ability to escalate individual cases based on time
2 Salesforce
sensitivity and the hierarchy of various clients.
Extensive reporting is a value of Salesforce’s
offerings, which provides management an ability
to track problem areas within an organization to a
distinct department, area, or tangible product
offering.
Salesforce has been a key leader in evolving
marketing within this digital era through the use of
specific marketing strategy aimed at creating and
tracking marketing campaigns as well as measur-
ing the success of online campaigns. These
services are part of another growing segment
available within Salesforce offerings in addition
to the CRM packaging. Marketing departments
leveraging Salesforce’s Buddy Media, Radian6,
or ExactTarget obtain the ability of users to con-
duct demographic, regional, or national searches
on keywords and themes across all social net-
works, which create a more informed and accurate
marketing direction. Further, Salesforce’s dash-
board, which is the main user interactive page,
allows the creation of specific marketing directed
tasks that can be customized and shared for dif-
fering organizational roles or personal
preferences.
Salesforce marketing dashboard utilizes wid-
gets that are custom, reusable page elements,
which can be housed on individual users’ pages.
When a widget is created, it is added to a widgets
view where all team members can easily be
assigned access. This allows companies and orga-
nizations to share appropriate widgets defined and
created to serve the target market or industry-
specific groups. The shareability of widgets
allows the most pertinent and useful tasks to be
replicated by many users within a single
organization.
Types of Widgets
The Salesforce Marketing Cloud “River of News”
is a widget that allows users to scroll through
specific search results, within all social media
conversations, and utilizes user-defined key-
words. Users have the ability to see original
posts that were targeted from keyword searches
and provided a source link to the social media
platform the post or message originated from.
The “River of News” displays posts with many
different priorities, such as newest post first, num-
ber of Twitter followers, social media platform
used, physical location, and Klout score. This
tool provides strong functionality for marketers
or corporations wishing to hone in, or take part
in, industry, customer, or competitor
conversations.
“Topic analysis” is a widget that is most often
used to show share of voice or the percentage of
conversation happening about your brand or orga-
nization in relation to competitor brands. It is
displayed as a pie chart and can be segmented
multiple ways based on user configuration.
Many use this feature as a quick visual assessment
to see the conversations and interest revolving
around specific initiatives or product launches.
“Topic trends” is a widget that provides the
ability to display the volume of conversation
over time through graphs and charts. This feature
can be used to understand macro day, week, or
month data. This widget is useful when tracking
crisis management or brand sentiment. With a line
graph display, users can see spikes of activity and
conversation around critical areas. Further, users
then can click and hone in on spikes, which can
open a “Conversation Cloud” or “River of News”
that allows users to see the catalyst behind the
spike of social media activity. This tool is used
as a way to better understand reasons for increased
interest or conversation across broad social media
platforms.
Salesforce Uses
Salesforce offers wide ranging data inference
from its varied and evolving products. As CRM
integration within the web and mobile has
increased, the broad interest to better understand
and leverage social media marketing campaigns
has risen as well, allowing Salesforce a leading
push within this industry’s market share. The
diverse array of businesses, nonprofits, munici-
palities, and other organizations that utilize
Salesforce illustrates the importance of this soft-
ware within daily business and marketing
Salesforce
strategy. Salesforce clients include the American
Red Cross, the City of San Francisco,
Philadelphia’s 311 system, Burberry, H&R
Block, Volvo, and Wiley Publishing.
Salesforce Service Offerings
Salesforce is a leader within other CRM and
media marketing-orientated companies such as
Oracle, SAP, Microsoft Dynamics CRM, Sage
CRM, Goldmine, Zoho, Nimble, Highrise,
Insight.ly, and Hootsuite. Salesforce’s offerings
can be purchased individually or as a complete
bundle. It offers current breakdowns of services
and access in its varied options that are referred to
as Sales Cloud, Service Cloud, ExactTarget Mar-
keting Cloud, Salesforce1 Platform, Chatter, and
Work.com.
Sales Cloud allows businesses to track cus-
tomer inquiries, escalate issues requiring special-
ized support, and monitor employee productivity.
This product provides customer service teams
with the answers to customers’ questions and the
ability to make the answers available on the web
so consumers can find answers for themselves.
Service Cloud offers active and real-time infor-
mation directed toward customer service. This
service provides functionality such as Agent Con-
sole which offers relevant information about cus-
tomers and their media profiles. This service also
provides businesses the ability to give customers
access to live agent web chats from the web to
ensure customers can have access to information
without a phone call.
ExactTarget Marketing Cloud focuses on cre-
ating closer relationships with customers through
directed email campaigns, in-depth social market-
ing, data analytics, mobile campaigns, and mar-
keting automation.
Sales1Platform is geared toward mobile app
creation. Sales1Platform gives access to create
and promote mobile apps with over four million
apps created utilizing this service.
3
Chatter is a social and collaborative function
that relates to the Salesforce platform. Similar to
Facebook and Twitter, Chatter allows users to
form a community within their business that can
be used for secure collaboration and knowledge
sharing.
Work.com is a corporate performance manage-
ment platform for sales representatives. The plat-
form targets employee engagement in three areas:
alignment of team and personal goals with busi-
ness goals, motivation through public recognition,
and real-time performance feedback.
Salesforce has more than 5,500 employees,
revenues of approximately $1.7 billion, and a
market value of approximately $17 billion. The
company regularly conducts over 100 million
transactions a day and has over 3 million
subscribers.
Headquartered in San Francisco, California,
Salesforce also maintains regional offices in Dub-
lin, Singapore, and Tokyo with secondary loca-
tions in Toronto, New York, London, Sydney, and
San Mateo, California. Salesforce operates with
over 170,000 companies and 17,000 nonprofit
organizations. In June 2004, Salesforce was
offered on the New York Stock Exchange under
the symbol CRM.
Cross-References
▶ Customer Service
▶ Data Aggregation
▶ Social Media
▶ Streaming Data
Further Readings
Denning, S. (2011). Successfully implementing radical
management at Salesforce.com. Strategy & Leader-
ship, 39(6), 4.
S
Scientometrics
Jon Schmid
Georgia Institute of Technology, Atlanta, GA,
USA
Scientometrics refers to the study of science
through the measurement and analysis of
researchers’ productive outputs. These outputs
include journal articles, citations, books, patents,
data, and conference proceedings. The impact of
big data analytics on the field of scientometrics
has primarily been driven by two factors: the
emergence of large online bibliographic databases
and a recent push to broaden the evaluation of
research impact beyond citation-based measures.
Large online databases of articles, conferences
proceedings, and books allow researchers to
study the manner in which scholarship develops
and measure the impact of researchers, institu-
tions, and even countries on a field of scientific
knowledge. Using data on social media activity,
article views, downloads, social bookmarking,
and the text posted on blogs and other websites,
researchers are attempting to broaden the manner
in which scientific output is measured.
Bibliometrics, a subdiscipline of
scientometrics that focuses specifically on the
study of scientific publications, witnessed a boon
in research due to the emergence of large digital
bibliographic databases such as Web of Science,
Scopus, Google Scholar, and PubMed. The utility
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_180-1
of increased digital indexing is enhanced by the
recent surge in total scientific output. Lutz
Bornmann and Ruediger Mutz find that global
scientific output has grown at a rate of 8–9% per
year since World War II (equivalent to a doubling
every 9 years) (Bornmann and Mutz 2015).
Bibliometric analysis using large data sets has
been particularly useful in research that seeks to
understand the nature of research collaboration.
Because large bibliographic databases contain
information on coauthorships, the institutions
that host authors, journals, and publication dates,
text mining software can be used in combination
with social network analysis to understand the
nature of collaborative networks. Visualizations
of these networks are increasingly used to show
patterns of collaboration, ties between scientific
disciplines, and the impact of scientific ideas. For
example, Hanjun Xian and Krishna Madhavan
analyzed over 24,000 journal articles and confer-
ence proceedings from the field of engineering
education in effort to understand how the litera-
ture was produced (Xian and Madhaven 2014).
These data were used to map the network of
collaborative ties in the discipline. The study
found that cross-disciplinary scholars played a
critical role in linking isolated network segments.
Besides studying authorship and collaboration,
big data analytics have been used to analyze cita-
tions to measure the impact of research,
researchers, and research institutions. Citations
are a common proxy for the quality of research.
Important papers will generally be highly cited as
2 Scientometrics
subsequent research relies on them to advance
knowledge.
One prominent metric used in scientometrics is
the h-index, which was proposed by Jorge Hirsch
in 2005. The h-index considers the number of
publications produced by an individual or organi-
zation and the number of citations these publica-
tions receive. An individual can be said to have an
h-index of h when she produces h publications
each of which receives at least h citations and no
other publication receives more than h citations.
The advent of large databases and big data
analytics has greatly facilitated the calculation of
the h-index and similar impact metrics. For exam-
ple, in a 2013 study, Filippo Radicchi and Claudio
Castellano utilized the Google Scholar Citations
data set to evaluate the individual scholarly con-
tribution of over 35,000 scholars (Radicchi and
Castellano 2013). The researchers found that the
number of citations received by a scientist is a
strong proxy for that scientist’s h-index, whereas
the number of publications is a less precise proxy.
The same principles behind citation analysis can
be applied to measure the impact or quality of
patents. Large patent databases such as PATSTAT
allow researchers to measure the importance of indi-
vidual patents using forward citations. Forward cita-
tions come from the “prior art” section of the patent
documents, which describes the technologies that
were deemed critical to their innovation by the
patent applicants. Scholars use patent counts,
weighed by forward citations, to derive measures
of national innovative productivity.
Until recently, measurement of research impact
has been almost exclusively based on citation-based
measures. However, citations are slow to accumu-
late and ignore the influence of research on the
broader public. Recently there has been a push to
include novel data sources in the evaluation of
research impact. Gunther Eysenbach has found
that tweets about a journal article within the first
3 days of publication are a strong predictor of even-
tual citations for highly cited research articles
(Eysenbach 2011). The direction of causality in
this relationship – i.e., whether strong papers lead
to a high volume of tweets or whether the tweets
themselves cause subsequent citations – is unclear.
However, the author suggests that the most
promising use of social media data lies not in its
use as a predictor of traditional impact measures but
as means of creating novel metrics of the social
impact of research.
Indeed the development of an alternative set of
measurements – often referred to as “altmetrics” –
based on data gleaned from the social web repre-
sents a particularly active field of scientometrics
research. Toward this end, services such as PLOS
Article-Level Metrics use big data techniques to
develop metrics of research impact that consider
factors other than citations. PLOS Article-Level
Metrics pulls in data on article downloads,
commenting and sharing via services such
CiteuLike, Connotea, and Facebook, to broaden
the way in which a scholar’s contribution is
measured.
Certain academic fields, such as the humanities,
that rely on under-indexed forms of scholarship
such as book chapters and monographs have proven
difficult to study using traditional scientometrics
techniques. Because they do not depend on online
bibliographic databases, altmetrics may prove useful
in studying such fields. Björn Hammarfelt uses data
from Twitter and Mendeley – a web-based citation
manager that has a social networking component –
to study scholarship in the humanities (Hammarfelt
2014). While his study suggests that coverage gaps
still exist using altmetrics, as these applications
become more widely used, they will likely become
a useful means of studying neglected scientific
fields.
See Also
▶ Bibliometrics
▶ Social Media
▶ Text Analytics
▶ Thomson Reuters
Further Readings
Bornmann, L., & Mutz, R. (2015). Growth rates of modern
science: A bibliometric analysis based on the number of
publications and cited references. Journal of the Asso-
ciation for Information Science and Technology,
66(11), 2215–2222. arXiv:1402.4578 [Physics, Stat].
Scientometrics
Eysenbach, G. (2011). Can tweets predict citations? Met-
rics of social impact based on Twitter and correlation
with traditional metrics of scientific impact. Journal of
Medical Internet Research, 13, e123.
Hammarfelt, B. (2014). Using altmetrics for assessing
research impact in the humanities. Scientometrics,
101, 1419–1430.
3
Radicchi, F., & Castellano, C. (2013). Analysis of
bibliometric indicators for individual scholars in a
large data set. Scientometrics, 97(3), 627–637. https://
doi.org/10.1007/s11192-013-1027-3.
Xian, H., & Madhavan, K. (2014). Anatomy of scholarly
collaboration in engineering education: A big-data
bibliometric analysis. Journal of Engineering Educa-
tion, 103, 486–514.
S
Semantic/Content Analysis/Natural
Language Processing
Paul Nulty
Centre for Research in Arts Social Science and
Humanities, University of Cambridge,
Cambridge, United Kingdom
Introduction
One of the most difficult aspects of working with
big data is the prevalence of unstructured data,
and perhaps the most widespread source of
unstructured data is the information contained in
text files in the form of natural language. Human
language is in fact highly structured, but although
major advances have been made in automated
methods for symbolic processing and parsing of
language, full computational language under-
standing has yet to be achieved, and so a combi-
nation of symbolic and statistical approaches to
machine understanding of language are com-
monly used. Extracting meaning or achieving
understanding from human language through sta-
tistical or computational processing is one of the
most fundamental and challenging problems of
artificial intelligence. From a practical point of
view, the dramatic increase in availability of text
in electronic form means that reliable automated
analysis of natural language is an extremely useful
source of data for many disciplines.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_182-1
Big data is an interdisciplinary field, of which
natural language processing (NLP) is a
fragmented and interdisciplinary subfield.
Broadly speaking, researchers use approaches
somewhere on a continuum between representing
and parsing the structures of human language in a
symbolic, rule-based fashion, or feeding large
amounts of minimally preprocessed text into
more sophisticated statistical machine learning
systems. In addition, various substantive research
areas have developed overlapping but distinct
methods for computational analysis of text.
The question of whether NLP tasks are best
approached with statistical, data-driven methods
or symbolic, theory-driven models is an old
debate. In 1957, Noam Chomsky wrote:
it must be recognized that the notion of “probability
of a sentence” is an entirely useless one, under
any known interpretation of this term.
However, at present the best methods we have
for translating, searching, and classifying natural
language text use flexible machine-learning algo-
rithms that learn parameters probabilistically from
relatively unprocessed text. On the other hand,
some applications, such as the IBM Watson ques-
tion answering system (Ferruci et al. 2010), make
good use of a combination of probabilistic learn-
ing and modules informed by linguistic theory to
disambiguate nuanced queries.
The field of computational linguistics origi-
nally had the goal of improving understanding of
2 Semantic/Content Analysis/Natural Language Processing
human language using computational methods.
Historically, this meant implementing rules and
structures inspired by the cognitive structures pro-
posed by Chomskyan generative linguistics. Over
time, computational linguistics has broadened to
include diverse methods for machine processing
of language irrespective of whether the computa-
tional models are plausible cognitive models of
human language processing. As practiced today,
computational linguistics is closer to a branch of
computer science than a branch of linguistics. The
branch of linguistics that uses quantitative analy-
sis of large text corpora is known as corpus
linguistics.
Research in computational linguistics and nat-
ural language processing involves finding solu-
tions for the many subproblems associated with
understanding language, and combining advances
in these modules to improve performance on gen-
eral tasks. Some of the most important NLP sub-
problems include part-of-speech tagging,
syntactic parsing, identifying the semantic roles
played by verb arguments, recognizing named
entities, and resolving references. These feed
into performance on more general tasks like
machine translation, question answering, and
summarization.
In the social sciences, the terms quantitative
content analysis, quantitative text analysis, or
“text as data” are all used. Content analysis may
be performed by human coders, who read and
mark-up documents. This process can be stream-
lined with software. Fully automated content anal-
ysis, or quantitative text analysis, typically
employs statistical word-frequency analysis to
discover latent traits from text, or scale documents
of interest on a particular dimension of interest in
social science or political science.
Tools and Resources
Text data does not immediately challenge compu-
tational resources to the same extent as other big
data sources such as video or sensor data. For
example, the entire proceedings of the European
parliament from 1996 to 2005, in 21 languages,
can be stored in 5.4 gigabytes – enough to load
into main memory on most modern machines.
While techniques such as parallel and distributed
processing may be necessary in some cases, for
example, global streams of social media text or
applying machine learning techniques for classi-
fication, typically the challenge of text data is to
parse and extract useful information from the idi-
osyncratic and opaque structures of natural lan-
guage, rather than overcoming computational
difficulties simply to store and manipulate the
text. The unpredictable structure of text files
means that general purpose programming lan-
guages are commonly used, unlike in other appli-
cations where the tabular format of the data allows
the use of specialized statistical software.
Original Unix command line tools such as
grep, sed, and awk are still extremely useful for
batch processing of text documents. Historically,
Perl has been the programming language of
choice for text processing, but recently Ruby and
Python have become more widely used. These are
scripting languages, designed for ease of use and
flexibility rather than speed. For more computa-
tionally intensive tasks, NLP tools are
implemented in Java or C/Cþþ.
The python libraries spaCy and gensim and the
Java-based Stanford Core NLP software are
widely used in industry and academia. They pro-
vide implementations and guides for the most
widely used text processing and statistical docu-
ment analysis methods.
Preprocessing
The first step in approaching a text analysis
dataset is to successfully read the document for-
mats and file encodings used. Most programming
languages provide libraries for interfacing with
Microsoft Word and pdf documents. The ASCII
coding system represents unaccented English
upper and lowercase letters, numbers, and punc-
tuation, using one byte per character. This is no
longer sufficient for most purposes, and modern
documents are encoded in a diverse set of charac-
ter encodings. The Unicode system defines code
points which can represent characters and sym-
bols from all writing systems. The UTF-8 and
Semantic/Content Analysis/Natural Language Processing
UTF-16 encodings implement these code points
in 8 bit or 16 bit encoded files.
Words are the most apparent units of written
text, and most text processing tasks begin with
tokenization – dividing the text into words. In
many languages, this is relatively uncomplicated:
whitespace delimits words, with a few ambiguous
cases such as hyphenation, contraction, and the
possessive marker. Within languages written in
the Roman alphabet there is some variance, for
example, agglutinative languages like Finnish and
Hungarian tend to use long compound terms dis-
ambiguated by case markers, which can make the
connection between space-separated words and
dictionary-entry meanings tenuous. For languages
with a different orthographic system, such as Chi-
nese, Japanese, and Arabic, it is necessary to use a
customized tokenizer to split text into units suit-
able for quantitative analysis.
Even in English, the correspondence between
space-separated word and semantic unit is not
exact. The fundamental unit of vocabulary –
sometimes called the lexeme – may be modified
or inflected by the addition of morphemes indicat-
ing tense, gender, or number. For many applica-
tions, it is not desirable to distinguish between the
inflected forms of words, rather we want to sum
together counts of equivalent words. Therefore, it
is common to remove the inflected endings of
words and count only the root, or stem. For exam-
ple, a system to judge the sentiment of a movie
review need not distinguish between the words
“excite,” “exciting,” “excites,” and “excited.”
Typically the word ending is removed and the
terms are treated equivalently.
The Porter stemmer (Porter 1980) is one of the
most frequently used algorithms for this purpose.
A slightly more sophisticated method is
lemmatization, which also normalizes inflected
words, but uses a dictionary to match irregular
forms such as “be”/“is”/“are”. In addition to stem-
ming and tokenizing, it may be useful to remove
very common words that are unlikely to have
semantic content related to the task. In English,
the most common words are function words such
as “of,” “in,” and “the.” These “stopwords”
largely serve a grammatical rather than semantic
3
function, and some NLP systems simply remove
them before proceeding with a statistical analysis.
After the initial text preprocessing, there are
several simple metrics that may be used to assess
the complexity of language used in the docu-
ments. The type-token ratio, a measure of lexical
diversity, gives an estimate of the complexity of
the document by comparing the total number of
words in the document to the number of unique
words (i.e., the size of the vocabulary). The
Fleisch-Kincaid readability metric uses the aver-
age sentence length and the average number of
syllables per word combined with coefficients
calibrated with data from students to give an esti-
mate of the grade-level reading difficulty of a text.
Document-Term Matrices
After tokenization and other preprocessing steps,
most text analysis methods work with a matrix
that stores the frequency with which each word in
the vocabulary occurs in each document. This is
the simplest case, known as the “bag-of-words”
model, and no information about the ordering of
the words in the original texts is retained. More
sophisticated analysis might involve extracting
counts of complex features from the documents.
For example, the text may be parsed and tagged
with part-of-speech information as part of the
preprocessing stage, which would allow for the
words with identical spellings but different part-
of-speech categories or grammatical roles to be
counted as separate features.
Often, rather than using only single words,
counts of phrases are used. These are known as
n-grams, where n is the number of words in the
phrase, for example, trigrams are three-word
sequences. N-gram models are especially impor-
tant for language modeling, used to predict the
probability of a word or phrase given the preced-
ing sequence of words. Language modeling is
particularly important for natural language gener-
ation and speech recognition problems.
Once each document has been converted to a
row of counts of terms or features, a wide range of
automated document analysis methods can be
employed. The document-term matrix is usually
4 Semantic/Content Analysis/Natural Language Processing
sparse and uneven – a small number of words
occur very frequently in many documents, while
a large number of words occur rarely, and most
words do not occur at all in a given document.
Therefore, it is common practice to smooth or
weight the matrix, either using the log of the
term frequency or with a measure of term impor-
tance like tf-idf (term frequency x inverse docu-
ment frequency) or mutual information.
Matrix Analysis
Supervised classification methods attempt to auto-
matically categorize documents based on the doc-
ument-term matrix. One of the most familiar of
such tasks is the email spam detection problem.
Based on the frequencies of words in a corpus of
emails, the system must decide if an email is spam
or not. Such a system is supervised in the sense
that it requires as a starting point a set of docu-
ments that have been correctly labeled with the
appropriate category, in order to build a statistical
model of which terms are associated with each
category. One simple and effective algorithm for
supervised document classification is Naive
Bayes, which gives a new document the class
that has the maximum a posteriori probability
given the term counts and their independent asso-
ciation between the terms and the categories in the
training documents. In political science,a similar
algorithm – “wordscores” – is widely used, which
sums Naive-Bayes-like word parameters to scale
new documents based on reference scores
assigned to training texts with extreme posi-
tions (Laver et al. 2003).
Other widely used supervised classifiers
include support vector machines, logistic regres-
sion, and nearest neighbor models. If the task is to
predict a continuous variable rather than a class
label, then a regression model may be used. Sta-
tistical learning and prediction systems that oper-
ate on text data very often face the typical big data
problem of having more features (word types)
than observed or labeled documents. This is a
high dimensional learning problem, where p (the
number of parameters) is much larger than n (the
number of observed examples).
In addition, word frequencies are extremely
unevenly distributed (an observation known as
Zipf’s law) and are highly correlated with one
another, resulting in parameter vectors that make
less than ideal examples for regression models. It
may therefore be necessary to use regression
methods designed to mitigate this problem, such
as lasso and ridge regression, or to prune the
feature space to avoid overtraining, using feature
subset selection or a dimensionality reduction
technique like principal components analysis or
singular value decomposition. With recent
advances in neural network research, it has
become more common to use unprocessed counts
of n-grams, tokens, or even characters as input to a
neural network with many intermediate layers.
With sufficient training data, such a network can
learn the feature extraction process better than
hand-curated feature extraction systems, and
these “deep learning” networks have improved
the state of the art in machine translation and
image labeling.
Unsupervised methods can cluster documents
or reveal the distribution of topics in documents in
a data-driven fashion. For unsupervised scaling
and clustering of documents, methods include k-
means clustering, or the Wordfish algorithm, a
multinomial Poisson scaling model for political
documents.
Another goal of unsupervised analysis is to
measure what topics comprise the text corpus,
and how these topics are distributed across docu-
ments. Topic modeling (Blei 2012) is a widely
used generative technique to discover a set of
topics that influence the generation of the texts,
and explore how they are associated with other
variables of interest.
Vector Space Semantics and Machine
Learning
In addition to retrieving or labeling documents, it
can be useful to represent the meaning of terms
found in the documents. Vector space semantics,
or distributional semantics, aims to represent the
meaning of words using counts of their co-occur-
rences with other words. The “distributional
Semantic/Content Analysis/Natural Language Processing
hypothesis,” as described by JR Firth (Firth 1957),
is the idea that “you shall know a word by the
company it keeps.” The co-occurrence vectors of
words have been shown to be useful for noun
phrase disambiguation, semantic relation extrac-
tion, and analogy resolution. Many systems now
use the factorization of the co-occurrence matrices
as the initial input to statistical learners, allowing a
fine-grained representation of lexical semantics.
Vector semantics also allows for word sense dis-
ambiguation – it is possible to distinguish the
different senses of a word by clustering the vector
representations of its occurrences.
These vectors may count instances of words
co-occurring with the same context (syntagmatic
relations) or compare the similarity of the contexts
of words as a measure of their substitutability
(paradigmatic relations) (Turney and Pantel
2010). The use of neural networks or dimension-
ality reduction techniques allows researchers to
produce a relatively low dimensional space in
which to compare word vectors, sometimes called
word embeddings.
Machine learning has long been used to per-
form classification of documents or to aid the
accuracy of NLP subtasks described above. How-
ever, as in many other fields, the recent application
of neural networks with many hidden layers
(Deep Learning) has led to large improvements
in accuracy rates on many tasks. These opaque but
computationally powerful techniques require only
a large volume of training data and a differentiable
target function to model complex linguistic
behavior.
Conclusion
Natural language processing is a complex and
varied problem that lies at the heart of artificial
intelligence. The combination of statistical and
symbolic methods has led to huge leaps forward
5
over the last few decades, and with the prepon-
derance of online training data and advances in
machine learning methods, it is likely that further
gains will be made in the coming years. For
researchers intending to make use of rather than
advance these methods, a fruitful approach is a
good working knowledge of a general purpose
programming language, combined with the ability
to configure and execute off-the-shelf machine
learning packages.
Cross-References
▶ Artificial Intelligence
▶ Biomedical Natural Language Processing
▶ Python Scripting Language
▶ Supervised Machine Learning
▶ Text Analytics
▶ Unstructured Data
References
Blei, D. M. (2012). Probabilistic topic models. Communi-
cations of the ACM, 55(4), 77–84.
Chomsky, N. (2002). Syntactic structures. Berlin: Walter
de Gruyter.
Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek,
D., Kalyanpur, A., Lally, A., Murdock, J., Nyberg, E.,
Prager, J., Schlaefer, N., & Welty, C. A. (2010). Build-
ing Watson: An overview of the deep QA project. AI
Magazine, 31(3), 59–79.
Firth, J. R. (1957). A synopsis of linguistic theory. In
Studies in linguistic analysis. Blackwell: Oxford.
Laver, M., Benoit, K., & Garry, J. (2003). Extracting policy
positions from political texts using words as data.
American Political Science Review, 97(02), 311–331.
Porter, MF. "An algorithm for suffix stripping." Pro-
gram 14.3 (1980): 130–137.
Slapin, J. B., & Proksch, S.-O. (2008). A scaling model for
estimating time-series party positions from texts. Amer-
ican Journal of Political Science, 52(3), 705–722.
Turney, P. D., & Pantel, P. (2010). From frequency to
meaning: Vector space models of semantics. Journal
of Artificial Intelligence Research, 37(1), 141–188.
S
Semi-structured Data
Yulia A. Strekalova1 and Mustapha Bouakkaz2
1
College of Journalism and Communications,
University of Florida, Gainesville, FL, USA
2
University Amar Telidji Laghouat, Laghouat,
Algeria
More and more data become available electroni-
cally every day, and they may be stored in a
variety of data systems. Some data entries may
reside in unstructured document file systems, and
some data may be collected and stored in highly
structured relational databases. The data itself
may represent raw images and sounds or come
with a rigid structure as strictly entered entities.
However, a lot of data currently available through
public and proprietary data systems is semi-
structured.
Definition
Semi-structured data is data that resembles struc-
tured data by its format but is not organized with
the same restrictive rules. This flexibility allows
collecting data even if some data points are miss-
ing or contain information that is not easily trans-
lated in a relational database format. Semi-
structured data carries the richness of human
information exchange, but most of it cannot be
automatically processed and used. Developments
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_183-1
in markup languages and software applications
allow the collection and evaluation of semi-
structured data, but the richness of natural text
contained in semi-structured data still presents
challenges for analysts.
Structured data has been organized into a for-
mat that makes it easier to access and process such
as databases where data is stored in columns,
which represent the attribute of the database. In
reality, very little data is completely structured.
Conversely, unstructured data has been not
reformatted, and its elements are not organized
into a data structure. Semi-structured data com-
bines some elements of both data types. It is not
organized in a complex manner that supports
immediate analyses; however, it may have infor-
mation associated with it, such as metadata tag-
ging, that allows elements contained to be
addressed through more sophisticated access
queries. For example, a word document is gener-
ally considered to be unstructured data. However,
when metadata tags in the form of keywords that
represent the document content are added, the data
becomes semi-structured.
Data Analysis
The volume and unpredictable structure of the
available data present challenges in analysis. To
get meaningful insights from semi-structured
data, analysts need to pre-analyze it to ask ques-
tions that can be answered with the data. The fact
2 Semi-structured Data
that a large number of correlations can be found
does not necessarily mean that analysis is reliable
and complete. One of the preparation measures
before the actual data analysis is data reduction.
While a large number of data points may be avail-
able for collection, not all these data points should
be included in an analysis to every question.
Instead, a careful consideration of data points is
likely to produce a more reliable and explainable
interpretation of observed data. In other words,
just because the data is available, it does not mean
it needs to be included in the analysis. Some
elements may be random and will not add sub-
stantively to the answer to a particular questions.
Some other elements may be redundant and not
add any new information compared to the one
already provided by other data points.
Jules Berman suggests nine steps to the analy-
sis of semi-structured data. Step 1 includes formu-
lation of a question which can and will be
subsequently answered with data. A Big Data
approach may not be the best strategy for ques-
tions that can be answered with other traditional
research methods. Step 2 evaluates data resources
available for collection. Data repositories may
have “blind spots” or data points that are system-
atically excluded or restricted for public access.
At step 3, a question is reformulated to adjust for
the resources identified in step 2. Available data
may be insufficient to answer the original question
despite the access to large amounts of data. Step
4 involved evaluation of possible query outputs.
Data mining may return a large number of data
points, but these data points most frequently need
to be filtered to focus on the analysis of the ques-
tion at hand. At step 5, data should be reviewed
and evaluated for its structure and characteristics.
Returned data may be quantitative or qualitative,
or it may have data points which are missing for a
substantial number of records, which will impact
future data analysis. Step 6 requires a strategic and
systematic data reduction. Although it may sound
counterintuitive, Big Data analysis can provide
most powerful insights when the data set is con-
densed to bare essentials to answer a focused
question. Some collected data may be irrelevant
or redundant to the problem at hand and will not
be needed for the analysis. Step 7 calls for the
identification of analytic algorithms, should they
be deemed necessary. Algorithms are analytic
approaches to data, which may be very sophisti-
cated. However, establishing a reliable set of
meaningful metrics to answer a question may be
a more reliable strategy. Step 8 looks at the results
and conclusions of the analysis and calls for con-
servative assessment of possible explanations and
models suggested by the data, assertions for cau-
sality, and possible biases. Finally, step 9 calls for
validation of results in step 8 using comparable
data sets. Invalidation of predictions may suggest
necessary adjustments to any of the steps in the
data analysis and make conclusions more robust.
Data Management
Semi-structured data includes both database char-
acteristics and incorporates documents and other
files types, which cannot be fully described by a
standard database entry. Data entries in structured
data sets follow the same order; all entries in a
group have the same descriptions, defined format,
and predefined length. In contrast, semi-structured
data entries are organized in semantic entities,
similar to the structured data, which may not
have same attributes in the same order or of the
same length. Early digital databases were orga-
nized based on the relational model of data, where
data is recorded into one or more tables with a
unique identifier for each entry. The data for such
databases needs to be structured uniformly for
each record. Semi-structured data but relies on
tag or other markers to separate data elements.
Semi-structured data may miss data elements or
have more than one data point in an element.
Overall, while semi-structured data has a pre-
defined structure, the data within this structure is
not entered with the same rigor as in the traditional
relational databases. This data management situa-
tion arises from the practical necessity to handle
user-generated and widely interactional data
brought up by the Web 2.0. The data contained
in emails, blog posts, PowerPoint presentation
files, images, and videos may have very different
sets of attributes, but they also offer a possibility
to assign metadata systematically. Metadata may
Semi-structured Data
include information about author and time and
may create the structure to assign the data to
semantic groups. Unstructured data, on the other
hand, is the data that cannot be readily organized
in tables to capture the full extent of it. Semi-
structured data, as the name suggests, carries
some elements of structured data. These elements
are metadata tags that may list the author or
sender, entry creation and modification times,
the length of a document, or the number of slides
in a presentation. Yet, these data also have ele-
ments that cannot be described in a traditional
relational database. For example, traditional data-
base structure which would require initial infra-
structure design will not be able to handle
information as a sent email, and all response that
were received as it is unknown if an email respon-
dents will use one or all names in response, if
anyone will get added or omitted, if original mes-
sage will be modified, if attachments will be
added to subsequent messages, etc.
Semi-structured data allows programmers to
nest data or create hierarchies that represent com-
plex data models and relationships among entries.
However, robustness of the traditional relational
data model forces more thoughtful implementa-
tion of data applications and possible subsequent
ease in analysis. Handling of semi-structured data
is associated with some challenges. The data itself
may present a problem by being embedded in
natural text, which cannot always be extracted
automatically with precision. Natural text is
based on sentences which may not have easily
identifiable relationships and entities which are nec-
essary for data collection. Natural text is based on
sentences that may not have easily identifiable rela-
tionships and entities, which are necessary for data
collection, and the lack of widely accepted stan-
dards for vocabularies. A communication process
may involve different models to transfer the same
information or require richer data transfer available
through natural text and not through a structured
exchange of keywords. For example, email
exchange can capture the data about senders and
recipients, but automated filtering and analysis of
the body of email are limited.
3
Two main types of semi-structured data for-
mats are Extensible Markup Language (XML)
and JavaScript Object Notation (JSON). XML,
developed in the mid-1990s, is a markup language
that sets rules for the data interchange. XML,
although being an improvement to earlier markup
languages, has been critiqued for being bulky and
cumbersome in implementation. JSON is viewed
as a possible successor format for digital architec-
ture and database technologies. JSON is an open
standard format that transmits data between an
application and a server. Data objects in JSON
format consist of attribute-value pairs stored in
databases like MongoDB and Couchbase. The
data, which is stored in a database like MongoDB,
can be pulled with a software network for more
efficient and faster processing. Apache Hadoop is
an example of an open-source framework that
provides both storage and processing support.
Other multi-platform query processing applica-
tions suitable for enterprise-level use are Apache
Spark and Presto.
See Also
▶ Big Data Storytelling, Digital Storytelling
▶ Discovery Analytics
▶ Hadoop
▶ MongoDB
▶ Text Analytics
Further Readings
Abiteboul, S., et al. (2012). Web data management.
New York: Cambridge University Press.
Foreman, J. W. (2013). Data smart: Using data science to
transform information into insight. Indianapolis: Wiley.
Miner, G., et al. (2012). Practical text mining and statisti-
cal analysis for non-structured text data applications.
Waltham: Academic.
S
Sentiment Analysis
Francis Dalisay1, Matthew J. Kushin2 and
Masahiro Yamamoto3
1
Communication & Fine Arts, College of Liberal
Arts & Social Sciences, University of Guam,
Mangilao, GU, USA
2
Department of Communication, Shepherd
University, Shepherdstown, WV, USA
3
Department of Communication, University at
Albany – SUNY, Albany, NY, USA
Sentiment analysis is defined as the computational
study of opinions, or sentiment, in text. Sentiment
analysis typically intends to capture an opinion
holder’s evaluative response (e.g., positive, nega-
tive, or neutral, or a more fine-grained classifica-
tion scheme) toward an object. The evaluative
response reflects an opinion holder’s attitudes, or
affective feelings, beliefs, thoughts, and
appraisals.
According to scholars Erik Cambria, Bjorn
Schuller, Yunging Xia, and Catherine Havasi,
sentiment analysis is a term typically used inter-
changeably with opinion mining to refer to the
same field of study. The scholars note, however,
that opinion mining generally involves the detec-
tion of the polarity of opinion, also referred to as
the sentiment orientation of a given text (i.e.,
whether the expressed opinion is positive, nega-
tive, or neutral). Sentiment analysis focuses on the
recognition of emotion (e.g., emotional states
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_184-1
such as “sad” or “happy”), but also typically
involves some form of opinion mining. For this
reason, and since both fields rely on natural lan-
guage processing (NLP) to analyze opinions from
text, sentiment analysis is often couched under the
same umbrella as opinion mining.
Sentiment analysis has gained popularity as a
social data analytics tool. Recent years have
witnessed the widespread adoption of social
media platforms as outlets to publicly express
opinions on nearly any subject, including those
relating to political and social issues, sporting and
entertainment events, weather, and brand and con-
sumer experiences. Much of the content posted on
sites such as Twitter, Facebook, YouTube, cus-
tomer review pages, and news article comment
boards is public. As such, businesses, political
campaigns, universities, and government entities,
among others, can collect and analyze this infor-
mation to gain insight into the thoughts of key
publics.
The ability of sentiment analysis to measure
individuals’ thoughts and feelings has a wide
range of practical applications. For example, sen-
timent analysis can be used to analyze online news
content and to examine the polarity of news cov-
erage of particular issues. Also, businesses are
able to collect and analyze the sentiment of com-
ments posted online to assess consumers’ opin-
ions toward their products and services, evaluate
the effectiveness of advertising and PR cam-
paigns, and identify customer complaints. Gath-
ering such market intelligence helps guide
2 Sentiment Analysis
decision-making in the realms of product research
and development, marketing and public relations,
crisis management, and customer relations.
Although businesses have traditionally relied on
surveys and focus groups, sentiment analysis
offers several unique advantages over such con-
ventional data collection methods. These advan-
tages include reduced cost and time, increased
access to much larger samples and hard-to-reach
populations, and real-time intelligence. Thus, sen-
timent analysis can be a useful market research
tool. Indeed, sentiment analysis is now commonly
offered by many commercial social data analysis
services.
Approaches
Broadly speaking, there exist two approaches in
the automatic extraction of sentiment from textual
material: the lexicon-based approach and the
machine learning-based approach. In the
lexicon-based approach, a sentiment orientation
score is calculated for a given text unit based on
a predetermined set of opinion words with posi-
tive (e.g., good, fun, exciting) and negative (e.g.,
bad, boring, poor) sentiments. In a simple form, a
list of words, phrases, and idioms with known
sentiment orientations is built into a dictionary,
or an opinion lexicon. Each word is assigned
specific sentiment orientation scores. Using the
lexicon, each opinion word extracted receives a
predefined sentiment orientation score, which is
then aggregated for a text unit.
The machine learning-based approach, also
called the text classification approach, builds a
sentiment classifier to determine whether a given
text about an object is positive, negative, or neu-
tral. Using the ability of machines to learn, this
approach trains a sentiment classifier to use a large
set of examples, or training corpus, that have
sentiment categories (e.g., positive, negative, or
neutral). The sentiment categories are manually
annotated by humans according to predefined
rules. The classifier then applies the properties of
the training corpus to classify data into sentiment
categories.
Levels of Analysis
The classification of an opinion in text as positive,
negative, or neutral (or a more fine-grained clas-
sification scheme) is impacted by and thus
requires consideration of the level at which the
analysis is conducted. There are three levels of
analysis: document, sentence, and aspect and/or
entity. First, the document-level sentiment classi-
fication addresses a whole document as the unit of
analysis. The task of this level of analysis is to
determine whether an entire document (e.g., a
product review, a blog post, an email, etc.) is
positive, negative, or neutral about an object.
This level of analysis assumes that the opinions
expressed on the document are targeted toward a
single entity (e.g., a single product). As such, this
level is not particularly useful to documents that
discuss multiple entities.
The second, sentence-level sentiment classifi-
cation, focuses on the sentiment orientation of
individual sentences. This level of analysis is
also referred to as subjectivity classification and
comprised of two tasks: subjective classification
and sentence-level classification. In the first task,
the system determines whether a sentence is sub-
jective or objective. If it is determined that the
sentence expresses a subjective opinion, the anal-
ysis moves to the second task, sentence-level clas-
sification. This second task involves determining
whether the sentence is positive, negative, or
neutral.
The third type of classification is referred to as
entity and aspect-level sentiment analysis. Also
called feature-based opinion mining, this level of
analysis focuses on sentiments directed at entities
and/or their aspects. An entity can include a prod-
uct, service, person, issue, or event. An aspect is a
feature of the entity, such as its color or weight.
For example, in the sentence “the design of this
laptop is bad, but its processing speed is excel-
lent,” there are two aspects stated – “design” and
Sentiment Analysis
“processing speed.” This sentence is negative
about one aspect, “design,” and positive about
the other aspect, “processing speed.” Entity- and
aspect-level sentiment analysis is not limited to
analyzing documents or sentences alone. Indeed,
although a document or sentence may contain
opinions regarding multiple entities and their
aspects, the entity- and aspect-level sentiment
analysis has the ability to identify the specific
entities and/or aspects that the opinions on the
document or sentence are referring to and then
determine whether the opinions are positive, neg-
ative, or neutral.
Challenges and Limitations
Extracting opinions from texts is a daunting task.
It requires a thorough understanding of the seman-
tic, syntactic, explicit, and implicit rules of a lan-
guage. Also, because sentiment analysis is carried
out by a computer system with a typical focus on
analyzing documents on a particular topic, off-
topic passages containing irrelevant information
may also be included in the analyses (e.g., a doc-
ument may contain information on multiple
topics). This could result in creating inaccurate
global sentiment polarities about the main topic
being analyzed. Therefore, the computer system
must be able to adequately screen and distinguish
opinions that are not relevant to the topic being
analyzed. Relatedly, for the machine learning-
based approach, a sentiment classifier trained on
a certain domain (e.g., car reviews) may perform
well on the particular topic, but may not when
applied to another domain (e.g., computer
review). The issue of domain independence is
another important challenge.
Also, the complexities of human communica-
tion limit the capacity of sentiment analysis to
capture nuanced, contextual meanings that opin-
ion holders actually intend to communicate in
their messages. Examples include the use of sar-
casm, irony, and humor in which context plays a
key role in conveying the intended message, par-
ticularly in cases when an individual says one
thing but means the opposite. For example, some-
one may say “nice shirt,” which implies positive
3
sentiment if said sincerely but implies negative
sentiment if said sarcastically. Similarly, words
such as “sick,” “bad,” and “nasty” may have
reversed sentiment orientation depending on con-
text and how they are used. For example, “My
new car is sick!” implies positive sentiment
toward the car. These issues can also contribute
to inaccuracies in sentiment analysis.
Altogether, despite these limitations, the com-
putational study of opinions provided by senti-
ment analysis can be beneficial for practical
purposes. So long as individuals continue to
share their opinions through online user-generated
media, the possibilities for entities seeking to gain
meaningful insights into the opinions of key pub-
lics will remain. Yet, challenges to sentiment,
analysis such as those discussed above, pose sig-
nificant limitations to its accuracy and thus its
usefulness in decision-making.
Cross-References
▶ Competitive Monitoring
▶ Consumer Products
▶ Data Mining
▶ Facebook
▶ Internet
▶ LinkedIn
▶ Marketing/Advertising
▶ Online Identity
▶ Real-Time Analytics
▶ SalesForce
▶ Social Media
▶ Twitter
Further Reading
Cambria, E., Schuller, B., Xia, Y., & Havasi, C. (2013).
New avenues in opinion mining and sentiment analysis.
IEEE Intelligent Systems, 28, 15–21.
Liu, B. (2011). Sentiment analysis and opinion mining. San
Rafael: Morgan & Claypool.
Pang, B., & Lee, L. (2008). Opinion mining and sentiment
analysis. Foundations and Trends in Information
Retrieval, 2(1–2), 1–135.
Pang, B., Lee, L., & Vaithyanathan S. (2002). Thumbs up?
Sentiment classification using machine learning tech-
niques. In Proceedings of the Conference on Empirical
4 Sentiment Analysis

Methods in Natural Language Processing (EMNLP) washingtonpost.com/politics/the-secret-service-wants-

(pp. 79–86). software-that-detects-sarcasm-yeah-good-luck/2014/
Zezima, K. The secret service wants software that detects 06/03/35bb8bd0-eb41-11e3-9f5c-9075d5508f0a_
sarcasm (Yeah, good luck.) The Washington Post. story.html
Retrieved 11 Aug 2014 from http://www.
S
Smart Cities
Jan Lauren Boyles
Greenlee School of Journalism and
Communication, Iowa State University, Ames,
IA, USA
Definition/Introduction
Smart cities are built upon aggregated, data-
driven insights that are obtained directly from
the urban infrastructure. These data points trans-
late into actionable information that can guide
municipal development and policy (Albino et al.
2015). Building on the emergent Internet of
Things movement, networked sensors (often
physically embedded into the built environment)
create rich data streams that uncover how city
resources are used (Townsend 2013; Komninos
2015; Sadowski and Pasquale 2015). Such intel-
ligent systems, for instance, can send alerts to city
residents when demand for urban resources
outpaces supply or when emergency conditions
exist within city limits. By analyzing these data
flows (often in real time), elected officials, city
staff, civic leaders, and average citizens can more
fully understand resource use and allocation,
thereby optimizing the full potential of municipal
services (Hollands 2008; de Lange and de Waal
2013; Campbell 2013; Komninos 2015). Over
time, the integration of such intelligent systems
into metropolitan life acts to better inform urban
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_185-1
policy making and better direct long-term munic-
ipal planning efforts (Batty 2013; Komninos
2015; Goldsmith and Crawford 2014). Despite
this promise of more effective and responsive
governance, however, achieving a truly smart
city often requires the redesign (and in many
cases, the physical rebuilding) of structures to
harvest and process big data from the urban envi-
ronment (Campbell 2013). As a result, global
metropolitan leaders continue to experiment with
cost-effective approaches to constructing smart
cities in the late-2010s.
Heralded as potentially revolutionizing citizen-
government interactions within cities, the initial
integration of Internet Communication Technolo-
gies (ICTs) into the physical city in the late 1990s
was viewed as the first step toward today’s smart
cities (Caragliu et al. 2011; Albino et al. 2015). In
the early 2000s, the burgeoning population
growth of global cities mandated the use of more
sophisticated computational tools to effectively
monitor and manage metropolitan resources
(Campbell 2013; Meijer and Bolivar 2015). The
rise of smart cities in the early 2010s can, in fact,
be traced to a trio of technological advances: the
adoption of cloud computing, the expansion of
wireless networks, and the acceleration of pro-
cessing power. At the same time, the societal
uptick in mobile computing by everyday citizens
enables more data to be collected on user habits
and behaviors of urban residents (Batty 2013).
The most significant advance in smart city adop-
tion rests, however, in geolocation – the concept
2 Smart Cities
that data can be linked to physical space (Batty
2013; Townsend 2013). European metropolises,
in particular, have been early adopters of intelli-
gent systems (Vanolo 2013).
The Challenges of Intelligent
Governance
Tactically, most smart cities attempt to tackle
wicked problems – the types of dilemmas that
have historically puzzled city planners
(Campbell 2013; Komninos 2015). The integra-
tion of intelligent systems into the urban environ-
ment has accelerated the time horizon for
policymaking for these issues (Batty 2013). Data
that once took years to gather and assess can now
be accumulated and analyzed in mere hours, or in
some cases, in real time (Batty 2013). Within
smart cities, crowdsourcing efforts often also
enlist residents, who voluntarily provide data to
fuel collective and collaborative solutions (Batty
2013). Operating in this environment of height-
ened responsiveness, municipal leaders within
smart cities are increasingly expected to integrate
open data initiatives that provide public access to
the information gathered by the data-driven
municipal networks (Schrock 2016). City plan-
ners, civic activists, and urban technologists
must also jointly consider the needs of city
dwellers throughout the process of designing
smart cities, directly engaging residents in the
building of smart systems (de Lange and de
Waal 2013). At the same time, urban officials
must be increasingly cognizant that as more user
behaviors within city limits are tracked with data,
the surveillance required to power smart systems
may also concurrently challenge citizen notions of
privacy and security (Goldsmith and Crawford
2014; Sadowski and Pasquale 2015). Local gov-
ernments must also ensure that the data collected
will be safe and secure from hackers, who may
wish to disrupt essential smart systems within
cities (Schrock 2016).
Conclusion
The successful integration of intelligent systems
into the city is centrally predicated upon financial
investment in overhauling aging urban infrastruc-
ture (Townsend 2013; Sadowski and Pasquale
2015). Politically, investment decisions are fur-
ther complicated by fragmented municipal lead-
ership, whose priorities for smart city
implementation may shift between election cycles
and administrations (Campbell 2013). Rather than
encountering these challenges in isolation, munic-
ipal leaders are beginning to work together to
develop global solutions to shared wicked prob-
lems. Intelligent system advocates argue that devel-
oping collaborative approaches to building smart
cities will drive the growth of smart cities into the
next decade (Goldsmith and Crawford 2014).
Cross-References
▶ Internet of Things
▶ Open Data
▶ Semantic Web
Further Reading
Albino, V., Berardi, U., & Dangelico, R. M. (2015). Smart
cities: Definitions, dimensions, performance, and ini-
tiatives. Journal of Urban Technology, 22(1), 3–21.
Batty, M. (2013). Big data, smart cities and city planning.
Dialogues in Human Geography, 3(3), 274–279.
Campbell, T. (2013). Beyond smart cities: How cities net-
work, learn and innovate. New York: Routledge.
Caragliu, A., Del Bo, C., & Nijkamp, P. (2011). Smart
cities in Europe. Journal of Urban Technology, 18(2),
65–82.
de Lange, M., & de Waal, M. (2013). Owning the city: New
media and citizen engagement in urban design. First
Monday, 18(11). doi:10.5210/fm.v18i11.4954.
Goldsmith, S., & Crawford, S. (2014). The responsive city:
Engaging communities through data-smart gover-
nance. San Francisco: Jossey-Bass.
Hollands, R. G. (2008). Will the real smart city please stand
up? Intelligent, progressive or entrepreneurial? City,
12(3), 303–320.
Komninos, N. (2015). The age of intelligent cities: Smart
environments and innovation-for-all strategies. New
York: Routledge.
Smart Cities
Meijer, A., & Bolívar, M. P. R. (2015). Governing the
smart city: A review of the literature on smart urban
governance. International Review of Administrative
Sciences. doi:10.1177/0020852314564308.
Sadowski, J., & Pasquale, F. A. (2015). The spectrum
of control: A social theory of the smart city.
First Monday, 20(7). doi:10.5210/fm.v20i7.5903.
3
Schrock, A. R. (2016). Civic hacking as data activism and
advocacy: A history from publicity to open government
data. New Media & Society, 18(4), 581–599.
Townsend, A. (2013). Smart cities: Big data, civic hackers,
and the quest for a new utopia. New York:
W.W. Norton.
Vanolo, A. (2013). Smartmentality: The smart city as
disciplinary strategy. Urban Studies, 51(5), 883–898.
S
Social Media
Dimitra Dimitrakopoulou
School of Journalism and Mass Communication,
Aristotle University of Thessaloniki,
Thessaloniki, Greece
Social media and networks are based on the tech-
nological tools and the ideological foundations of
Web 2.0 and enable the production, distribution,
and exchange of user-generated content. They
transform the global media landscape by transpos-
ing the power of information and communication
to the public that had until recently a passive role
in the mass communication process.
Web 2.0 tools refer to the sites and services that
emerged during the early 2000s, such as blogs (e.
g., Blogspot, Wordpress), wikis (e.g., Wikipedia),
microblogs (e.g., Twitter), social networking sites
(e.g., Facebook, LinkedIn), video (e.g.,
YouTube), image (e.g., Flickr), file-sharing plat-
forms (e.g., We, Dropbox), and related tools that
allow participants to create and share their own
content. Though the term was originally used to
identify the second coming of the Web after the
dotcom burst and restore confidence in the indus-
try, it became inherent in the new WWW applica-
tions through its widespread use.
The popularity of Web 2.0 applications dem-
onstrates that, regardless of their levels of techni-
cal expertise, users can wield technologies in
more active ways than had been apparent
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_186-1
previously to traditional media producers and
technology innovators. In addition to referring to
various communication tools and platforms,
including social networking sites, social media
also hint at a cultural mindset that emerged in
the mid-2000s as part of the technical and busi-
ness phenomenon referred to as Web 2.0.
It is important to distinguish between social
media and social networks. Whereas often both
terms are used interchangeably, it is important to
understand that social media are based on user-
generated content produced by the active users
who now can act as producers as well. Social
media have been defined on multiple levels,
starting from more operational definitions that
underline that social media indicate a shift from
HTML-based linking practices of the open Web to
linking and recommendation, which happen
inside closed systems. Web 2.0 has three
distinguishing features: it is easy to use, it facili-
tates sociality, and it provides users with free
publishing and production platforms that allow
them to upload content in any form, be it pictures,
videos, or text. Social media are often contrasted
to traditional media by highlighting their
distinguishing features, as they refer to a set of
online tools that supports social interaction
between users. The term is often used to contrast
with more traditional media such as television and
books that deliver content to mass populations but
do not facilitate the creation or sharing of content
by users as well as their ability to blur the
2 Social Media
distinction between personal communication and
the broadcast model of messages.
Theoretical Foundations of Social Media
Looking into the role of the new interactive and
empowering media, it is important to study their
development as techno-social systems, focusing
on the dialectic relation of structure and agency.
As Fuchs (2014) describes, media are techno-
social systems, in which information and commu-
nication technologies enable and constrain human
activities that create knowledge that is produced,
distributed, and consumed with the help of tech-
nologies in a dynamic and reflexive process that
connects technological structures and human
agency. The network infrastructure of the Internet
allows multiple and multi-way communication
and information flow between agents, combining
both interpersonal (one-to-one), mass (one-to-
many), and complex, yet dynamically equal com-
munication (many-to-many).
The discussion on the role of social media and
networks finds its roots in the emergence of the
network society and the evolvement of the Inter-
net as a result of the convergence of the audiovi-
sual, information technology, and
telecommunications sector. Contemporary society
is characterized by what can be defined as conver-
gence culture (Jenkins 2006) in which old and
new media collide, where grassroots and corpo-
rate media intersect, where the power of the media
producer and the power of the media consumer
interact in unpredictable ways.
The work of Manuel Castells (2000) on the
network society is central, emphasizing that the
dominant functions and processes in the Informa-
tion Age are increasingly organized around net-
works. Networks constitute the new social
morphology of our societies, and the diffusion of
networking logic substantially modifies the oper-
ation and outcomes in processes of production,
experience, power, and culture. Castells (2000)
introduces the concept of “flows of information,”
underlining the crucial role of information flows
in networks for the economic and social
organization.
In the development of the flows of information,
the Internet holds the key role as a catalyst of a
novel platform for public discourse and public
communication. The Internet consists of both a
technological infrastructure and (inter)acting
humans, in a technological system and a social
subsystem that both have a networked character.
Together these parts form a techno-social system.
The technological structure is a network that pro-
duces and reproduces human actions and social
networks and is itself produced and reproduced by
such practices.
The specification of the online platforms, such
as Web 1.0, Web 2.0, or Web 3.0, marks distinc-
tively the social dynamics that define the evolu-
tion of the Internet. Fuchs (2014) provides a
comprehensive approach for the three “genera-
tions” of the Internet, founding them on the idea
of knowledge as a threefold dynamic process of
cognition, communication, and cooperation. The
(analytical) distinction indicates that all Web 3.0
applications (cooperation) and processes also
include aspects of communication and cognition
and that all Web 2.0 applications (communica-
tion) also include cognition. The distinction is
based on the insight of knowledge as threefold
process that all communication processes require
cognition, but not all cognition processes result in
communication, and that all cooperation pro-
cesses require communication and cognition, but
not all cognition and communication processes
result in cooperation.
In many definitions, the notions of collabora-
tion and collective actions are central, stressing
that social media are tools that increase our ability
to share, to cooperate, with one another, and to
take collective action, all outside the framework
of traditional institutional institutions and organi-
zations. Social media enable users to create their
own content and decide on the range of its dis-
semination through the various available and eas-
ily accessible platforms. Social media can serve as
online facilitators or enhancers of human net-
works – webs of people that promote connected-
ness as a social value.
Social network sites (SNS) are built on the
pattern of online communities of people who are
connected and share similar interests and
Social Media
activities. Boyd and Ellison (2007) provide a
robust and articulated definition of SNS, describ-
ing them as Web-based services that allow indi-
viduals to (1) construct a public or semipublic
profile within a bounded system, (2) articulate a
list of other users with whom they share a con-
nection, and (3) view and traverse their list of
connections and those made by others within the
system. The nature and nomenclature of these
connections may vary from site to site. As the
social media and user-generated content phenom-
ena grew, websites focused on media sharing
began implementing and integrating SNS features
and becoming SNSs themselves.
The emancipatory power of social media is
crucial to understand the importance of network-
ing, collaboration, and participation. These con-
cepts, directly linked to social media, are key
concepts to understand the real impact and dimen-
sions of contemporary participatory media cul-
ture. According to Jenkins (2006), the term
participatory culture contrasts with older notions
of passive media consumption. Rather than
talking about media producers and consumers
occupying separate roles, we might now see
them as participants who interact with each other
and contribute actively and prospectively equally
to social media production.
Participation is a key concept that addresses the
main differences between the traditional (old)
media and the social (new) media and focuses
mainly on the empowerment of the audience/
users of media toward a more active information
and communication role. The changes transform
the relation between the main actors in political
communication, namely, political actors, journal-
ists, and citizens. Social media and networks
enable any user to participate in the mediation
process by actively searching, sharing, and
commenting on available content. The distrib-
uted, dynamic, and fluid structure of social
media enables them to circumvent professional
and political restrictions on news production and
has given rise to new forms of journalism defined
as citizen, alternative, or participatory journalism,
but also new forms of propaganda and
misinformation.
3
The Emergence of Citizen Journalism
The rise of social media and networks has a direct
impact on the types and values of journalism and
the structures of the public sphere. The transfor-
mation of interactions between political actors,
journalists and citizens through the new technol-
ogies has created the conditions for the emergence
of a distinct form from professional journalism,
often called citizen, participatory, or alternative
journalism. The terms used to identify the new
journalistic practices on the Web range from inter-
active or online journalism to alternative journal-
ism, participatory journalism, citizen journalism,
or public journalism. The level and the form of
public’s participation in the journalistic process
determine whether it is a synergy between jour-
nalists and the public or exclusive journalistic
activities of the citizens.
However, the phenomenon of alternative jour-
nalism is not new. Already in the nineteenth cen-
tury, the first forms of alternative journalism made
their appearance with the development of the rad-
ical British press. The radical socialist press in the
USA in the early twentieth century followed as
did the marginal and feminist press between 1960
and 1970. Fanzines and zines appeared in the
1970s and were succeeded by pirate radio sta-
tions. At the end of the twentieth century, how-
ever, the attention has moved to new media and
Web 2.0 technologies.
The evolution of social networks with the new
paradigm shift is currently defining to a great
extent the type, the impact, and the dynamics of
action, reaction, and interaction of the involved
participants in a social network. According to
Atton (2003), alternative journalism is an ongoing
effort to review and challenge the dominant
approaches to journalism. The structure of this
alternative journalistic practice appears as the
counterbalance to traditional and conventional
media production and disrupts its dominant
forms, namely, the institutional dimension of
mainstream media, the phenomena of capitaliza-
tion and commercialization, and the growing con-
centration of ownership.
Citizen journalism is based on the assumption
that the public space is in crisis (institutions,
4 Social Media
politics, journalism, political parties). It appears as
an effort to democratize journalism and thereby is
questioning the added value of objectivity, which
is supported by professional journalism.
The debate on a counterweight to professional,
conventional, mainstream journalism was intensi-
fied around 1993, when the signs of fatigue and
the loss of public’s credibility in journalism
became visible and overlapped with the innova-
tive potentials of the new interactive technologies.
The term public journalism (public journalism)
appeared in the USA in 1993 as part of a move-
ment that expressed concerns for the detachment
of journalists and news organizations from the
citizens and communities, as well as of US citi-
zens from public life. However, the term citizen
journalism has defined on various levels. If both
its supporters and critics agree on one core thing,
it is that it means different things to different
people.
The developments that Web 2.0 has introduced
and the subsequent explosive growth of social
media and networks mark the third phase of public
journalism and its transformation to alternative
journalism. The field of information and commu-
nication is transformed into a more participatory
media ecosystem, which evolves the news as
social experiences. News are transformed into a
participatory activity to which people contribute
their own stories and experiences and their reac-
tions to events.
Citizen journalism proposes a different model
of selection and use of sources and of news prac-
tices and redefinition of the journalistic values.
Atton (2003) traces the conflict with traditional,
mainstream journalism in three key points: (a)
power does not come exclusively from the official
institutional institutions and the professional cat-
egory of journalists, (b) reliability and validity can
derive from descriptions of lived experience and
not only objectively detached reporting, and (c) it
is not mandatory to separate the facts from sub-
jective opinion. Although Atton (2003) does not
consider lived experiences as an absolute value,
he believes it can constitute the added value of
alternative journalism, combining it with the
capability of recording it through documented
reports.
The purpose of citizen journalism is to reverse
the “hierarchy of access” as it was identified by
Glasgow University Media Group, giving voice to
the ones marginalized by the mainstream media.
While mainstream media rely extensively on elite
groups, alternative media can offer a wider range
of “voices” that wait to be heard. The practices of
alternative journalism provide “first-hand” evi-
dences, as well as collective and anti-hierarchical
forms of organizations and a participatory, radical
approach of citizen journalism. This form of jour-
nalism is identified by Atton as native reporting.
To determine the moving boundary between
news producers and the public, Bruns (2005)
used the term produsers, combining the words
and concepts of producers and users. These
changes determine the way in which power rela-
tions in the media industry and journalism are
changing, shifting the power from journalists to
the public.
Social Movements
In the last few years, we have witnessed a growing
heated debate among scholars, politicians, and
journalists regarding the role of the Internet in
contemporary social movements. Social media
tools such as Facebook, Twitter, and YouTube
which facilitate and support user-generated con-
tent have taken up a leading role in the develop-
ment and coordination of a series of recent social
movements, such as the student protests in Britain
at the end of 2010 as well as the outbreak of
revolution in the Arab world, the so-called Arab
Spring.
The open and decentralized character of the
Internet has inspired many scholars to envisage a
rejuvenation of democracy, focusing on the
(latent) democratic potentials of the new media
as interactive platforms that can motivate and
fulfill the active participation of the citizens in
the political process. On the other hand, Internet
skeptics suggest that the Internet will not itself
alter traditional politics. On the contrary, it can
generate a very fragmented public sphere based
on isolated private discussions while the abun-
dance of information, in combination with the
Social Media
vast amounts of offered entertainment and the
options for personal socializing, can lead people
to restrain from public life. The Internet actually
offers a new venue for information provision to
the citizen-consumer. At the same time, it allows
politicians to establish direct communication with
the citizens free from the norms and structural
constraints of traditional journalism.
Social media aspire to create new opportunities
for social movements. Web 2.0 platforms allow
protestors to collaborate so that they can quickly
organize and disseminate a message across the
globe. By enabling the fast, easy, and low-cost
diffusion of protest ideas, tactics, and strategies,
social media and networks allow social move-
ments to overcome problems historically associ-
ated with collective mobilization.
Over the last years, the center of attention was
not the Western societies, which were used in
being the technology literate and information-
rich part of the world, but the Middle Eastern
ones. Especially after 2009, there is considerable
evidence advocating in favor of the empowering,
liberating, and yet engaging potentials of the
online social media and networks as in the case
of the protesters in Iran who have actively used
Web services like Facebook, Twitter, Flickr, and
YouTube to organize, attract support, and share
information about street protests after the June
2009 presidential elections. More recently, a rev-
olutionary wave of demonstrations has swept the
Arab countries as the so-called Arab Spring, using
again the social media as means for raising aware-
ness, communication, and organization, facing at
the same time strong Internet censorship. Though
neglecting the complexity of these transforma-
tions, the uprisings were largely quoted as “the
5
Facebook revolution,” demonstrating the power
of networks.
In the European continent, we have witnessed
the recent development of the Indignant Citizens
Movement, whose origin was largely attributed to
the social movements that started in Spain and
then spread to Portugal, the Netherlands, the
UK, and Greece. In these cases, the digital social
networks have proved powerful means to convey
demands for a radical renewal of politics based on
a stronger and more direct role of citizens and on a
critique of the functioning of Western democratic
systems.
See Also
▶ Digital Literacy
▶ Open Data
▶ Social Network Analysis
▶ Twitter
Further Reading
Atton, C. (2003). What is ‘alternative’ journalism? Jour-
nalism: Theory Practice and Criticism, 4(3), 267–272.
Boyd, D. M., & Ellison, N. B. (2007). Social network sites:
Definition, history, and scholarship. Journal of Com-
puter-Mediated Communication, 13(1), 210–230.
Bruns, A. (2005). Gatewatching: Collaborative online
news production. New York: Peter Lang.
Castells, M. (2000). The rise of the network society, the
information age: Economy, society and culture vol. I.
Oxford: Blackwell.
Fuchs, C. (2014). Social media: A critical introduction.
London: Sage.
Jenkins, H. (2006). Convergence culture: Where old and
new media collide. New York: New York University
Press.
S
Social Sciences
Ines Amaral
University of Minho, Minho, Portugal
Autonomous University of Lisbon, Lisbon,
Portugal
Social Science is an academic discipline
concerned with the study of humans through
their relations with society and culture. Social
Science disciplines analyze the origins, develop-
ment, organization, and operation of human soci-
eties and cultures. The technological evolution
has strengthened Social Sciences since it enables
empirical studies developed through quantitative
means, allowing the scientific reinforcement of
many theories about the behavior of man as a
social actor. The rise of big data represents an
opportunity for the Social Sciences to advance
the understanding of human behavior using mas-
sive sets of data.
The issues related to Social Sciences began to
have a scientific nature in the eighteenth century
with the first studies on the actions of humans in
society and their relationships with each other. It
was by this time that Political Economy emerged.
Most of the subjects belonging to the fields of
Social Sciences, such as Anthropology, Sociol-
ogy, and Political Science arisen in the nineteenth
century.
Social Sciences can be divided in disciplines
that are dedicated to the study of the evolution of
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_188-1
societies (Archeology, History, Demography),
social interaction (Political Economy, Sociology,
Anthropology), or cognitive system (Psychology,
Linguistics). There are also applied Social Sci-
ences (Law, Pedagogy) and other Social Sciences
classified in the generic group of Humanities
(Political Science, Philosophy, Semiotics, Com-
munication Sciences). The anthropologist Claude
Lévi-Strauss, the philosopher and political scien-
tist Antonio Gramsci, the philosopher Michel
Foucault, the economist and philosopher Adam
Smith, the economist John Maynard Keynes, the
psychoanalyst Sigmund Freud, the sociologist
Émile Durkheim, the political scientist and soci-
ologist Max Weber, and the philosopher, sociolo-
gist, and economist Karl Marx are some of the
leading social scientists of the last centuries.
The social scientist studies phenomena, struc-
tures, and relationships that characterize the social
and cultural organizations; analyzes the move-
ments and population conflicts, the construction
of identities, and the formation of opinions;
researches behaviors and habits and the relation-
ship between individuals, families, groups, and
institutions; and develops and uses a wide range
of techniques and research methods to study
human collectivities and understand the problems
of society, politics, and culture.
The study of humans through their relations
with society and culture relied on “surface data”
and “deep data.” “Surface data” was used in the
disciplines that adapted quantitative methods, like
Economics. “Deep data” about individuals or
2 Social Sciences
small groups was used in disciplines that analyze
society through qualitative methods, such
Sociology.
Data collection has always been a problem for
social research because of its inherent subjectivity
as Social Sciences have traditionally relied on
small samples using methods and tools gathering
information based on people. In fact, one of the
critical issues of Social Science is the need to
develop research methods that ensure the objec-
tivity of the results. Moreover, the objects of study
of Social Sciences do not fit into the models and
methods used by other sciences and do not allow
the performance of experiments under controlled
laboratory conditions. The quantification of infor-
mation is possible because there are several tech-
niques of analysis that transform ideas, social
capital, relationships, and other variables from
social systems into numerical data. However, the
object of study always interacts with the culture of
the social scientist, making it very difficult to have
a real impartiality.
Big data is not self-explanatory. Consequently,
it requires new research paradigms across multi-
ple disciplines, and for social scientists, it is a
major challenge as it enables interdisciplinary
studies and the intersection between computer
science, statistics, data visualization, and social
sciences. Furthermore, big data empowers the
use real-time data on the level of whole
populations, to test new hypotheses and study
social phenomena on a larger scale. In the context
of modern Social Sciences, large datasets allow
scientists to understand and study different social
phenomena, from the interactions of individuals
and the emergence of self-organized global move-
ments to political decisions and the reactions of
economic markets.
Nowadays, social scientists have more infor-
mation on interaction and communication pat-
terns than ever. The computational tools allow
understanding the meaning of what those patterns
reveal. The models build about social systems
within the analysis of large volumes of data must
be coherent with the theories of human actors and
their behavior. The advantages of large datasets
and of the scaling up the size of data are that it is
possible to make sense of the temporal and spatial
dimensions. What makes big data so interesting to
Social Sciences is the possibility to reduce data,
apply filters that allow to identify relevant patterns
of information, aggregate sets in a way that helps
identify temporal scales and spatial resolutions,
and segregate streams and variables in order to
analyze social systems.
As big data is dynamic, heterogeneous, and
interrelated, social scientists are facing new chal-
lenges due to the existence of computational and
statistical tools, which allow extracting and ana-
lyzing large datasets of social information. Big
data is being generated in multiple and
interconnecting disciplinary fields. Within the
social domain, data is being collected from trans-
actions and interactions through multiple devices
and digital networks. The analysis of large
datasets is not within the field of a single scientific
discipline or approach. In this regard, big data can
change Social Science because it requires an inter-
section of sciences within different research tradi-
tions and a convergence of methodologies and
techniques. The scale of the data and the methods
required to analyze them need to be developed
combining expertise with scholars from other sci-
entific disciplines. Within this collaboration with
data scientists, social scientists must have an
essential role in order to read the data and under-
stand the social reality.
The era of big data implies that Social Sciences
rethink and update theories and theoretical ques-
tions such as small world phenomenon, complex-
ity of urban life, relational life, social networks,
study of communication and public opinion for-
mation, collective effervescence, and social influ-
ence. Although computerized databases are not
new, the emergence of an era of big data is critical
as it creates a radical shift of paradigm in social
research. Big data reframes key issues on the
foundation of knowledge, the processes and tech-
niques of research, the nature of information, and
the classification of social reality.
The new forms of social data have interesting
dimensions: volume, variety, velocity, exhaustive,
indexical, relational, flexible, and scalable. Big
data consists of relational information in large
scale that can be created in or near real time with
different structures, extensive in scope, capable of
Social Sciences
identifying and indexing information distinc-
tively, flexible, and able to expand in size quickly.
The datasets can be created by personal data or
nonpersonal data. Personal data can be defined as
information relating to an identified person. This
definition includes online user-generated content,
online social data, online behavioral data, location
data, sociodemographic data, and information
from an official source (e.g., police records). All
data collected that do not directly identify individ-
uals are considered nonpersonal data. Personal
data can be collected from different sources with
three techniques: voluntary data that is created
and shared online by individuals; observed data,
which records the actions of the individual; and
data inferred about individuals based on voluntary
information or observed.
The disciplinary outlines of Social Sciences in
the age of big data are in constant readjustment
because of the speed of change in the data land-
scape. Some authors argued that the new data
streams could reconfigure and constitute social
relations and populations. Academic researchers
attempt to handle the methodological challenges
presented by the growth of big social data, and
new scientific trends arise, although the diversity
of the philosophical foundations of Social Science
disciplines. Objectivity of the data does not result
directly in their interpretation. The scientific
method postulated by Durkheim attempts to
remove itself from the subjective domain. Never-
theless, the author stated that objectivity is made
by subjects and is based on subjective observa-
tions and selections of individuals.
A new empiricist epistemology emerged in
Social Sciences and goes against the deductive
approach that is hegemonic within modern sci-
ence. According to this new epistemology, big
data can capture an entire social reality and pro-
vide their full understanding. Therefore, there is
no need for theoretical models or hypotheses. This
perspective assumes that patterns and relation-
ships within big data are characteristically signif-
icant and accurate. Thus, the application of data
analytics transcends the context of a single scien-
tific discipline or a specific domain of knowledge
and can be interpreted by those who can interpret
statistics or data visualization.
3
Several scholars, who believe that the new
empiricism operates as a discursive rhetorical
device, criticize this approach. Kitchin argues
that whereas data can be interpreted free of con-
text and domain-specific expertise, such an epis-
temological interpretation is probable to be
unconstructive as it absences to be embedded in
broader discussions.
As large datasets are highly distributed and
present complex data, a new model of data-driven
science is emerging within the Social Science
disciplines. The data-driven science uses a hybrid
combination of abductive, inductive, and deduc-
tive methods to the understanding of a phenome-
non. This approach assumes theoretical
frameworks and pursues to generate scientific
hypotheses from the data by incorporating a
mode of induction into the research design. There-
fore, the epistemological strategy adopted within
this model is to detect techniques to identify
potential problems and questions, which can be
worth of further analysis, testing, and validation.
Although big data enhance the set of data
available for analysis and enable new approaches
and techniques, it does not replace the traditional
small data studies. Due to the fact that big data
cannot answer specific social questions, more
targeted studies are required. Computational
Social Sciences can be the interface between com-
puter science and the traditional social sciences.
This interdisciplinary and emerging scientific
from Social Sciences uses computationally
methods to model social reality and analyze phe-
nomena, as well as social structures and collective
behavior. The main computational approaches
from Social Sciences to study big data are social
network analysis, automated information extrac-
tion systems, social geographic information sys-
tems, complexity modeling, and social simulation
models.
Computational Social Science is an intersec-
tion of Computer Science, Statistics, and the
Social Sciences, which uses large-scale demo-
graphic, behavioral, and network data to analyze
individual activity, collective behaviors, and rela-
tionships. Computational Social Sciences can be
the methodological approach to Social Sciences
study big data because of the use of mathematical
4 Social Sciences
methods to model social phenomena and the abil-
ity to handle with large datasets.
The analysis of big volumes of data opens up
new perspectives of research and makes it possi-
ble to answer questions that were previously
incomprehensible. Though big data itself is rela-
tive, its analysis within the theoretical tradition of
Social Sciences to build a context for information
will enable its understanding and the intersection
with the smaller studies to explain specific data
variables.
Big data may have a transformational impact as
it can transform policy making, by helping to
improve communication and governance in sev-
eral policy domains. Big social data also raise
significant ethical issues for academic research
and request an urgent debate for a wider critical
reflection on the epistemological implications of
data analytics.
Cross-References
▶ Anthropology
▶ Communications
▶ Complex Networks
▶ Computational Social Sciences
▶ Computer Science
▶ Data Science
▶ Network Analytics
▶ Network Data
▶ Psychology
▶ Social Network Analysis (SNA)
▶ Sociology
▶ Visualization
Further Readings
Allison, P. D. (2002). Missing data: Quantitative applica-
tions in the social sciences. British Journal of Mathe-
matical and Statistical Psychology, 55(1), 193–196.
Berg, B. L., & Lune, H. (2004). Qualitative research
methods for the social sciences (Vol. 5). Boston:
Pearson.
Boyd, D., & Crawford, K. (2012). Critical questions for big
data: Provocations for a cultural, technological, and
scholarly phenomenon. Information, Communication
& Society, 15(5), 662–679.
Coleman, J. S. (1990). Foundations of social theory. Cam-
bridge, MA: Belknap Press of Harvard University
Press.
Floridi, L. (2012). Big data and their epistemological chal-
lenge. Philosophy & Technology, 25, 435–437.
González-Bailón, S. (2013). Social science in the era of big
data. Polymer International, 5(2), 147–160.
Lohr, S. (2012). The age of big data. New York Times 11.
Lynch, C. (2008). Big data: How do your data grow?
Nature, 455(7209), 28–29.
Oboler, A., et al. (2012). The danger of big data: Social
media as computational social science. First Monday,
17(7-2). Retrieved from http://firstmonday.org/ojs/
index.php/fm/article/view/3993/3269
S
Spatial Data
Xiaogang Ma
Department of Computer Science, University of
Idaho, Moscow, ID, USA
Synonyms
Geographic information; Geospatial
Geospatial information
Introduction
Spatial property is almost a pervasive component
in the big data environment because everything
happening on the Earth happens somewhere. Spa-
tial data can be grouped into raster or vector
according to the methods used in representations.
Web-based services facilitate the publication and
use of spatial data legacies, and the crowdsourcing
approaches enable people to be both contributors
and users of spatial data. Semantic technologies
further enable people to link and query the spatial
data available on the Web, find patterns of interest,
and to use them to tackle scientific and business
issues.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_192-1
Raster and Vector Representations
Spatial data are representations of facts that con-
tain positional values, and geospatial data are
spatial data that are about facts happening on the
surface of the Earth. Almost everything on the
Earth has location properties, so geospatial data
and spatial data are regarded as synonyms. Spatial
data can be seen almost everywhere in the big data
data; deluge, such as social media data stream, traffic
control, environmental sensor monitoring, and
supply chain management, etc. Accordingly,
there are various applications of spatial data in
the actual world. For example, one may find a
preferred restaurant based on the grading results
on Twitter. A driver may adjust his route based on
the real-time local traffic information. An engi-
neer may identify the best locations for new build-
ings in an area with regular earthquakes. A forest
manager may optimize timber production using
data of soil and tree species distribution and con-
sidering a few constraints such as the requirement
of biodiversity and market price.
Spatial data can be divided into two groups:
raster representations and vector representations.
A raster representation can be regarded as a group
of mutually exclusive cells which form the repre-
sentation of a partition of space. There are two
types of raster representations: regular and irreg-
ular. The former has cells with same shape and
size and the latter with cells of varying shape and
size. Raster representations do not store coordi-
nate pairs. In contrast, vector representations use
2 Spatial Data
coordinate pairs to explicitly describe a geo-
graphic phenomenon. There are several types of
vector representations, such as points, lines, areas,
and the triangulated irregular networks. A point is
a single coordinate pair in a two-dimensional
space or a coordinate triplet in a three-dimensional
space. A line is defined by two end points and zero
to more internal points to define the shape. An
area is a partition of space defined by a boundary
(Huisman and de By 2009).
The raster representations have simple but less
compact data structures. They enable simple
implementation of overlays but pose difficulties
for the representation of interrelations among geo-
graphic phenomena, as the cell boundaries are
independent of feature boundaries. However, the
raster representations are efficient for image pro-
cessing. In contrast, the vector representations
have complex data structures but are efficient for
representing spatial interrelations. The vector rep-
resentations work well in scale changes but are
hard to implement overlays. Also, they allow the
representation of networks and enable easy asso-
ciation with attribute data.
The collection, processing, and output of spa-
tial data are often relevant to a number of plat-
forms and systems, among them the most well-
known are the geographic information system,
remote sensing, and the global positioning sys-
tem. A geographic information system is a com-
puterized system that facilitates the phases of data
collection, data processing, and data output, espe-
cially for spatial data. Remote sensing is the use of
satellites to capture information about the surface
and atmosphere of the Earth. Remote sensing data
are normally stored in raster representations. The
global positioning system is a space-based satel-
lite navigation system that provides direct mea-
surement of position and time on the surface of the
Earth. Remote sensing images and global posi-
tioning system signals can be regarded as primary
data sources for the geographic information
system.
Spatial Data Service
Various proprietary and public formats for raster
and vector representations have been introduced
since computers were used for spatial data collec-
tion, analysis, and presentation. Plenty of remote
sensing images, digital maps, and sensor data
form a massive spatial data legacy. On the one
hand, they greatly facilitate the progress of using
spatial data to tackle scientific and social issues.
On the other hand, the heterogeneities caused by
the numerous data formats, conceptual models,
and software platforms bring huge challenges for
data integration and reuse from multiple sources.
The Open Geospatial Consortium (OGC) (2016)
was formed in 1994 to promote a worldwide con-
sensus process for developing publicly available
interface standards for spatial data. By early 2015,
the consortium consists of more than 500 mem-
bers from industry, government agencies, and aca-
demia. Standards developed by OGC have been
implemented for promoting interoperability in
spatial data collection, sharing, service, and pro-
cessing. Well-known standards include the Geog-
raphy Markup Language, Keyhole Markup
Language, Web Map Service, Web Feature Ser-
vice, Web Processing Service, Catalog Service for
the Web, Observations and Measurements, etc.
Community efforts such as the OGC service
standards offer a solution to publish multisource
heterogeneous spatial data legacy on the Web. A
number of best practices have emerged in recent
years. The OneGeology is an international initia-
tive among the geological surveys across the
world. It was launched in 2007, and by early
2015, it has 119 participating member nations.
Most members in OneGeology share national
and/or regional geological maps through the
OGC service standards, such as Web Map Service
and Web Feature Service. The OneGeology Portal
provides a central node for the various distributed
data services. The Portal is open and easy to use.
Anyone with an internet browser can view the
maps registered on the portal. People can also
use the maps in their own applications as many
software programs now provide interfaces to
access the spatial data services. Another more
comprehensive project is the GEO Portal of the
Spatial Data
Global Earth Observation System of Systems,
which is coordinated by the Group on Earth
Observations. It acts as a central portal and clear-
inghouse providing access to spatial data in sup-
port of the whole system. The portal provides
registry for both data services and standards used
in data services. It allows users to discover,
browse, edit, create, and save spatial data from
members of the Group on Earth Observations
across the world.
Another popular spatial data service is the vir-
tual globe, which provides three-dimensional rep-
resentation of the Earth or another world. It allows
users to navigate in a virtual environment by
changing the position, viewing angle, and scale.
A virtual globe has the capability to represent
various different views on the surface of the
Earth by adding spatial data as layers on the sur-
face of a three-dimensional globe. Well-known
virtual globes include Google Earth, NASA
World Wind, ESRI ArcGlobe, etc. Besides spatial
data browsing, most virtual globe programs also
enable the functionality of interactions with users.
For example, the Google Earth can be extended
with many add-ons encoded in the Keyhole
Markup Language, such as geological map layers
exported from OneGeology.
Open-Source Approaches
There are already widely used free and open-
source software programs serving different pur-
poses in spatial handling (Steiniger and Hunter
2013). Those programs can be grouped into a
number of categories:
(1) Standalone desktop geographic information
systems such as GRASS GIS, QGIS, and
ILWIS
(2) Mobile and light geographic information sys-
tems such as gvSIG Mobile, QGIS for
Android, and tangoGPS
(3) Libraries with capabilities for spatial data pro-
cessing, such as GeoScript, CGAL, and
GDAL
(4) Data analysis and visualization tools such as
GeoVISTA Studio and R and PySAL;
3
(5) Spatial database management systems such as
PostgreSQL, Ingres Geospatial, and JASPA
(6) Web-based spatial data publication and pro-
cessing servers such as GeoServer,
MapServer, and 52n WPS
(7) Web-based spatial data service development
frameworks such as OpenLayers, GeoTools,
and Leaflet
An international organization, the Open
Source Geospatial Foundation, was formed in
2006 to support the collaborative development
of open-source geospatial software programs and
promote their widespread use.
Companies such as Google, Microsoft, and
Yahoo! already provide free map services. One
can browse maps on the service website, but the
spatial data behind the service is not open. In
contrast, the free and open-source spatial data
approach requires not only freely available
datasets but also details about the data, such as
format, conceptual structure, vocabularies used,
etc. A well-known open-source spatial data pro-
ject is the OpenStreetMap, which aims at creating
a free editable map of the world. The project was
launched in 2004. It adopts a crowdsourcing
approach, that is, to solicit contributions from a
large community of people. By the middle of
2014, the OpenStreetMap project has more than
1.6 million contributors. Comparing with the
maps, the data generated by the OpenStreetMap
are considered as the primary output. Due to the
crowdsourcing approach, the current data quali-
ties vary across different regions. Besides the
OpenStreetMap, there are numerous similar
open-source and collaborative spatial data pro-
jects addressing the needs of different communi-
ties, such as the GeoNames for geographical
names and features, the OpenSeaMap for a world-
wide nautical chart, and the eBird project for real-
time data about bird distribution and abundance.
Open-source spatial data formats have also
received increasing attention in recent years, espe-
cially Web-based formats. A typical example is
GeoJSON, which enables the encoding of simple
geospatial features and their attributes using
JavaScript Object Notation (JSON). GeoJSON is
now supported by various spatial data software
4 Spatial Data
packages and libraries, such as OpenLayers,
GeoServer, and MapServer. Map services of Goo-
gle, Yahoo!, and Microsoft also support
GeoJSON in their application programming
interfaces.
Spatial Intelligence
The Semantic Web brings innovative ideas to the
geospatial community. The Semantic Web is a
web of data compared to the traditional web of
documents. A solid enablement of the Semantic
Web is the Linked Data, which is a group of
methodologies and technologies to publish struc-
tured data on the Web so they can be annotated,
interlinked, and queried to generate useful infor-
mation. The Web-based capabilities of linking and
querying are specific features of the Linked Data,
which help people to find patterns from data and
use them in scientific or business activities. To
make full use of the Linked Data, the geospatial
community is developing standards and technol-
ogies to (1) transform spatial data into Semantic
Web compatible formats such as the Resource
Description Framework (RDF), (2) organize and
publish the transformed data using triple stores,
and (3) explore patterns in the data using new
query languages such as GeoSPARQL.
The RDF uses a simple triple structure of sub-
ject, predicate, and object. The structure is robust
enough to support the linked spatial data
consisting of billions of triples. Building on the
basis of the RDF, there are a number of specific
schemas for representing locations and spatial
relationships in triples, such as the GeoSPARQL.
Triple stores offer functionalities to manage spa-
tial data RDF triples and query them, which are
very similar to what the traditional relational data-
bases are capable. As mentioned above, spatial
data have two major sources: conventional data
legacy and crowdsourcing data. While technolo-
gies are being mature for transforming both of
them into triples, the crowdsourcing data provide
a more flexible mechanism for the Linked Data
approach and data exploration as they are fully
open. For example, there are already works done
to transform data of the OpenStreetMap and
GeoNames into RDF triples. For the pattern
exploration, there are already initial results, such
as those in the GeoKnow project (Athanasiou et
al. 2014). The project built a prototype called
GeoKnow Generator which provides functions
to link, enrich, query, and visualize RDF triples
of spatial data and build lightweight applications
addressing specific requests in the actual world.
The linked spatial data is still far from mature
yet. More efforts are needed on the annotation and
accreditation of shared spatial RDF data, the inte-
gration and fusion of them, the efficient RDF
query in a big data environment, and innovative
ways to visualize and present the results.
Cross-References
▶ Geography
▶ Location Data
▶ Spatial Analytics
▶ Spatio-Temporal Analytics
References
Athanasiou, S., Hladky, D., Giannopoulos, G., Rojas, A.
G., Lehmann, J. (2014). GeoKnow: Making the web an
exploratory place for geospatial knowledge. ERCIM
News, 96. http://ercim-news.ercim.eu/en96/special/
geoknow-making-the-web-an-exploratory-place-for-geo
spatial-knowledge. Accessed 29 Apr 2016.
Huisman, O., & de By, R. A. (Eds.). (2009). Principles of
geographic information systems. Enschede: ITC Edu-
cational Textbook Series.
Open Geospatial Consortium (2016). About OGC. http://
www.opengeospatial.org/ogc. Accessed 29 Apr 2016.
Steiniger, S., & Hunter, A. J. S. (2013). The 2012 free and
open source GIS software map: A guide to facilitate
research, development, and adoption. Computers,
Environment and Urban Systems, 39, 136–150.
T
Transparency
Anne L. Washington
George Mason University, Fairfax, VA, USA
Transparency is a policy mechanism that encour-
ages organizations to disclose information to the
public. Scholars of big data and transparency rec-
ognize the inherent powerpower of information
and share a common intellectual history. Govern-
ment and corporate transparency, which is often
implemented by releasing open dataopen data,
increases the amount of material available for
big data projects. Furthermore, big data has its
own need for transparency as data-driven algorith-
malgorithms support essential decisions in society
with little disclosure about operations and proce-
dures. Critics question whether information can
be used as a control mechanism in an industry that
functions as a distributed network.
Definition
Transparency is defined as a property of glass or
any object that lets in light. As a governance
mechanism, transparency discloses the inner
mechanisms of an organization. OrganizationsOr-
ganization implement or are mandated to abide by
transparency policies that encourage the release of
information about how they operate. Hood and
Heald (2006) uses a directional typology to define
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_199-1
transparency. Upward and downward transpar-
ency refers to disclosure within an organization.
Supervisors observing subordinates is upward
transparency, while subordinates observing the
hierarchy above is downward transparency.
Inward and outward transparency refers to disclo-
sure beyond organizational boundaries. An orga-
nization aware of its environment is outward
transparency, while citizen awareness of govern-
ment activity is inward transparency. Transpar-
ency policies encourage the visibility of
operating status and standard procedures.
First, transparency may compel information on
operating status. When activities may impact
others, organizations disclose what they are
doing in frequent updates. For example, the US
government required regular reports from stock
exchanges and other financial markets after the
stock market crash in 1929. Operating status
information gives any external interest an ability
to evaluate the current state of the organization
and auditing.
Second, transparency efforts may distribute
standard procedures in order to enforce ideal
behaviors. This type of transparency holds people
with the public trust accountable. For example,
cities release open data with transportation sched-
ules and actual arrival times. The planned infor-
mation is compared to the actual information to
evaluate behaviors and resource distribution. Pro-
cedural transparency assumes that organizations
can and should operate predictably.
2 Transparency

Disclosures allow comparison and review. government projects developed technology capa-
Detailed activity disclosure of operations answers bilities in public sector organizations.
questions of who, what, when, and where. Con- Advances in computing have increased the use
versely, disclosures can also answer questions of big data techniques to automatically review
about influential people or wasteful projects. Dis- transparency disclosures. Transparency can be
closure may emphasize predictive trends and ret- implemented without technology, but often the
rospective measurement, while other disclosures two are intrinsically linked. One impact technol-
may emphasize narrative interpretation and ogy has on transparency is that information now
explanation. comes in multiple forms. Disclosure before tech-
nology was the static production of documents
and regularly scheduled reports that could be
released on paper by request. Disclosure with
Implementation
technology is the dynamic streaming of real-time
data available through machine-readable search
Transparency is implemented by disclosing
and discovery. Transparency is often implemented
timely information to meet specific needs. This
by releasing digital material as open data that can
assumes that stakeholders will discover the
be reused with few limitations. Open data trans-
disclosed information, comprehend its impor-
parency initiatives disclose information in formats
tance, and subsequently use it to change behavior.
that can be used with big data methods.
Organizations, including corporations and gov-
ernment, often implement transparency using
technology which creates digital material used in
Intellectual History
big data.
Corporations release information about how
Transparency has its origins in economic and
their actions impact communities. The goal of
philosophical ideas about disclosing the activities
corporate transparency is to improve services,
of those in authority. In Europe, the intellectual
share financial information, reduce harm to the
history spans from Aristotle in fifth-century
public, or reduce reputation risks. The veracity
Greece to Immanuel Kant in eighteenth-century
of corporate disclosures has been debated by man-
Prussia. Debates on big data can be positioned
agement science scholars (Bennis et al. 2008). On
within these conversations about the dynamics
the one hand, mandatory corporate reporting fails
of information and power. An underlying assump-
if the information provided does not solve the
tion of transparency is that there are hidden and
target issue (Fung et al. 2007). On the other
visible power relationships in the exchange of
hand, organizations that are transparent to
information. Transparency is often an antidote to
employees, management, stockholders, regulators,
situations where information is used as power to
and the public may have a competitive advantage.
control others.
In any case, there are real limits to what corpora-
Michel Foucault, the twentieth-century French
tions can disclose and still remain both domesti-
philosopher, considered how rulers used statistics
cally and internationally competitive.
to control populations in his lecture on
Governments release information as a form of
Governmentality. Foucault engaged with Jeremy
accountability. From the creation of the postal
Bentham’s eighteenth-century descriptions of the
code system to social security numbers, govern-
ideal prison and the ideal government, both of
ments have inadvertently provided core categories
which require full visibility. This philosophical
for big data analytics (Washington 2014). Starting
position argues that complete surveillance will
in the mid-twentieth century, legislatures around
result in complete cooperation. While some
the world began to write freedom of information
research suggests that people will continue bad
laws that supported the release of government
behavior under scrutiny, transparency is still seen
materials on request. Subsequently, electronic
as a method of enforcing good behavior.
Transparency
Big data extends concerns about the balance of
authority, power, and information. Those who
collect, store, and aggregate big data have more
control than those generating data. These concep-
tual foundations are useful in considering both the
positive and negative aspects of big data.
Big Data Transparency
Big data transparency discloses the transfer and
transformation of data across networks. Big data
transparency brings visibility to the embedded
power dynamic in predicting human behavior.
Analysis of digital material can be done without
explicit acknowledgment or agreement. Further-
more, the industry that exchanges consumer data
is easily obscured because transactions are all
virtual. While a person may willingly agree to
free services from a platform, it is not clear if
users know who owns, sees, collects, or uses
their data. The transparency of big data is
described from three perspectives: sources, orga-
nizations, and the industry.
Transparency of sources discloses information
about the digital material used in big data. Disclo-
sure of sources explains which data generated on
which platforms were used in which analysis. The
flip side of this disclosure is that those who create
user-generated content would be able to trace their
digital footprint. User-generated content creators
could detect and report errors and also be aware of
their overall data profile. Academic big data
research on social media was initially questioned
because of opaque sources from privacy compa-
nies. Source disclosure increases confidence in
data quality and reliability.
Transparency of platforms considers organiza-
tions that provide services that create user-gener-
ated content. Transparency within the
organization allows for internal monitoring.
While part of normal business operations, some-
one with command and control is able to view
personally identifiable information about the
activities of others. The car ride service Uber
was fined in 2014 because employees used the
internal customer tracking system inappropriately.
Some view this as a form of corporate surveillance
3
because it includes monitoring customers and
employees.
Transparency of the analytics industry dis-
closes how the big data market functions. Industry
transparency of operations might establish techni-
cal standards or policies for all participating orga-
nizations. The World Wide Web Consortium’s
data provenance standard provides a technical
solution by automatically tracing where data orig-
inated. Multi-stakeholder groups, such as those
for Internet Governance, are a possible tool to
establish self-governing policy solutions. The
intent is to heighten awareness of the data supply
chaindata supply chain from upstream content
quality to downstream big data production. Indus-
try transparency of procedure might disclosure
algorithms and research designs that are used in
data-driven decisions.
Big data transparency makes it possible to
compare data-driven decisions to other methods.
It faces particular challenges because its produc-
tion process is distributed across a network of
individuals and organizations. The process flows
from an initial data capture to secondary uses and
finally into large-scale analytic projects. Transpar-
ency is often associated with fighting potential
corruption or attempts to gain unethical power.
Given the influence of big data in many aspects
of society, the same ideas apply to the transpar-
ency of big data.
Criticism
A frequent criticism of transparency is that its
unintended consequences may thwart the antici-
pated goals. In some cases, the trend toward
visibility is reversed as those under scrutiny stop
creating findable traces and turn to informal mech-
anisms of communication.
It is important to note that a transparency label
may be used to legitimize authority without any
substantive information exchange. Large amounts
of information released under the name of trans-
parency may not, in practice, provide the intended
result. Helen Margetts (1999) questions whether
unfiltered data dumps obscure more than they
reveal. Real-time transparency may lack
4 Transparency
meaningful engagement because it requires inter-
mediary interpretation. This complaint has been
lodged at open data transparency initiatives which
did not release crucial information.
Implementation of big data transparency is
constrained by complex technical and business
issues. Algorithms and other technology are lay-
ered together, each with its own embedded
assumptions. Business agreements about the
exchange of data may be private, and release
may impact market competition. Scholars ques-
tion how to analyze and communicate what drives
big data, given these complexities.
Other critics question whether what is learned
through disclosure is looped back into the system
for reform or learning. Information disclosed for
transparency may not be channeled to the right
places or people. Without any feedback mecha-
nism, transparency can be a failure because it does
not drive change. Ideally, either organizations
improve performance or individuals make new
consumer choices.
Summary
Transparency is a governance mechanism for dis-
closing activities and decisions that profoundly
enhances confidence in big data. It builds on
existing corporate and government transparency
efforts to monitor the visibility of operations and
procedures. Transparency scholarship builds on
earlier research that examines the relationship
between power and information. Transparency
of big data evaluates the risks and opportunities
of aggregating sources for large-scale analytics.
Cross-References
▶ Algorithmic Accountability
▶ Business Process
▶ Data Governance
▶ Economics
▶ Enterprise Data
▶ Privacy
▶ Standardization
Further Readings
Bennis, W. G., Goleman, D., & O’Toole, J. (2008). Trans-
parency: How leaders create a culture of candor. San
Francisco: Jossey-Bass.
Fung, A., Graham, M., & Weil, D. (2007). Full disclosure:
The perils and promise of transparency. New York:
Cambridge University Press.
Hood, C., & Heald, D. (Eds.). (2006). Transparency: The
key to better governance? Oxford. New York: Oxford
University Press.
Margetts, H. (1999). Information technology in govern-
ment: Britain and America. London: Routledge.
Washington, A. L. (2014). Government information policy
in the era of big data. Review of Policy Research, 31(4).
doi:10.1111/ropr.12081.
U
United Nations Educational,
Scientific and Cultural Organization
(UNESCO)
Jennifer Ferreira
Centre for Business in Society, Coventry
University, Coventry, UK
United Nations Educational, Scientific and Cul-
tural Organization (UNSCO), founded in 1945, is
an agency of the United Nations (UN) which
specializes in education, natural sciences, social
and human sciences, culture, and communications
and information. With 195 members, 9 associate
members, and 50 field offices, working with over
300 international NGOs, UNESCO carries out
activities in all of these areas, with the post-2015
development agenda underpinning their overall
agenda.
As the only UN agency with a mandate to
address all aspects of education, it proffers that
education is at the heart of development, with a
belief that education is fundamental to human,
social, and economic development. It coordinates
“Education for All” movement, a global commit-
ment to provide quality basic education for all
children, youth, and adults, monitoring trends in
education and where possible make attempts to
raise the profile of education on the global devel-
opment agenda. For the natural sciences,
UNESCO acts as an advocate for science as it
focuses on encouraging international cooperation
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_201-1
in science as well as promoting dialogue between
scientists and policy-makers. In doing so, it acts as
a platform for dissemination of ideas in science
and encourages efforts on crosscutting themes
including disaster risk reduction, biodiversity,
engineering, science education, climate change,
and sustainable development. Within the social
and human sciences, UNESCO plays a large role
in promoting heritage as a source of identity and
cohesion for communities. It actively contributes
by developing cultural conventions that provide
mechanisms for international cooperation. These
international agreements are designs to safeguard
natural and cultural heritage across the globe, for
example, through designation as UNESCO World
Heritage sites. The development of communica-
tion and sharing information is embedded in all
their activities.
UNESCO has five key objectives: to attain
quality education for all and lifelong learning;
mobilize science knowledge and policy for sus-
tainable development; address emerging social
and ethical challenges; foster cultural diversity,
intercultural dialogue, and culture of peace; and
build inclusive knowledge societies through
information and communication. Like other UN
agencies, UNESCO has been involved in debates
about the data revolution for development and the
role that big data can play.
The data revolution for sustainable develop-
ment is an international initiative designed to
improve the quality of data and information that
is generated and made available. It recognizes that
2 United Nations Educational, Scientific and Cultural Organization (UNESCO)
societies need to take advantage of new technolo-
gies and crowd-sourced data and improve digital
connectivity in order to empower citizens with
information that can contribute towards progress
towards wider development goals. While there are
many data sets available about the state of global
education, it is argued that better data could be
generated, even around basic measures such as the
number of schools. In fact rather than focus on
“big data” which has captured the attention of
many leaders and policy-makers, instead more
efforts should focus on “little data,” i.e., focus
on data that is both useful and relevant to partic-
ular communities. Now discussions are shifting to
identify which indicators and data should be
prioritized.
UNESCO Institute for Statistics is the organi-
zation’s own statistic arm; however, much of the
data collection and analysis that takes place here
relies on much more conventional management
and information systems which in turn relies on
national statistical agencies which in many devel-
oping countries are often unreliable or heavily
focused on administrative data (UNESCO 2012).
This means that the data used by UNESCO is
often out of date, or not detailed enough. While
digital technologies have become widely used in
many societies, more potential sources of data are
generated (Pentland 2013). For example, mobile
phones are now used as banking devices as well as
for standard communications. Official statistics
organizations are still behind in many countries
and international organizations in that they have
not developed ways to adapt and make use of this
data alongside the standard administrative data
already collected.
There are a number of innovative initiatives to
make better use of survey data and mobile phone-
based applications to collect data more efficiently
and prove more timely feedback to schools, com-
munities, and ministries on target areas such as
enrolment, attendance, and learning achievement.
UNESCO could make a significant contribution to
a data revolution in education by investing in
resources in collecting these innovations and
making them more widely available to countries.
Access to big data for development, as with all
big data sources, presents a number of ethical
considerations based around the ownership of
data and privacy. This is an area the UN recog-
nizes that policy-makers will need to address to
ensure that data will be used safely to address their
objectives while still protecting the rights of peo-
ple whom the data is about or generated from.
Furthermore, there are a number of critiques of
big data which make more widespread use of big
data for UNESCO problematic: first that claims
that big data are objective and accurate represen-
tations are misleading; not all data produced can
be used comparably; there are important ethical
considerations necessary about the use of big data;
limited access to big data is exacerbating existing
digital divides.
The Scientific Advisory Board of the Secre-
tary-General of the United Nations which is
hosted by UNESCO provided comments on the
report on data revolution in sustainable develop-
ment. It highlighted concerns over equity and
access to data noting that the data revolution
should lead to equity in access and use of data
for all. Furthermore, it suggested that a number of
global priorities should be included in any agenda
related to the data revolutions: first that countries
should seek to avoid contributing to a data divide
between the rich and poor countries and secondly
that there should be some form of harmonization
and standardization of data platform to increase
accessibility internationally, there should be
national and regional capacity building efforts,
and there should be a series of training institutes
and training programs in order to develop skills
and innovation in areas related to data generation
and analysis (Manyika et al. 2011). A key point
made here is that the quality and integrity of the
data generated needs to be addressed, as it is
recognized that big data often plays an important
role in political and economic decision-making.
Therefore a series of standards and methods for
analysis and evaluation of data quality should be
developed.
In the journal Nature, Hubert Gijzen,
UNESCO Regional Science Bureau for Asia and
the Pacific, calls for more big data to help secure a
sustainable future (Gijzen 2013). He argues that
more data should be collected which can be used
to model different scenarios for sustainable
United Nations Educational, Scientific and Cultural Organization (UNESCO)
societies concerning a range of issues from energy
consumption, improving water conditions, and
poverty eradication. Big data, according to
Gijzen, has the potential if coordinated globally
between countries, regions, and relevant institu-
tions to have a big impact on the way societies
address some of these global challenges. The
United Nations has begun to take actions to do
this through the creation of the Global Pulse ini-
tiative bringing together experts from the govern-
ment, academic, and private sectors to consider
new ways to use big data to support development
agendas. Global Pulse, a network of innovation
labs which conduct research on Big Data for
Development via collaborations between the gov-
ernments, academic, and private sectors. The ini-
tiative is designed especially to make use of the
digital data flood that has developed in order to
address the development agendas that are at the
heart of UNESCO, and the UN more broadly.
The UN Secretary-General’s Independent
Expert Advisory Group on the Data Revolution
for Sustainable Development produced the report
“A World That Counts” UN Secretary-General’s
Export Advisory Group on Data Revolution
report in November 2014 suggested a number of
key principles which should be sought regards to
the use of data: data quality and integrity to ensure
clear standards for use of data, data disaggregation
to provide a basis for comparison, data timeliness
to encourage a flow of high quality data for used in
evidence-based policy-making, data transparency
to encourage systems which allow data to make
freely available, data usability to ensure data can
be made user-friendly, data protection and pri-
vacy: to establish international and national poli-
cies and legal frameworks for regulating data
generation and use, data governance and indepen-
dence, data resources and capacity to ensure all
countries have effective national statistical agen-
cies, and finally data rights to ensure human rights
remains a core part of any legal or regulatory
3
mechanisms that are developed with respect to
big data (United Nations 2014). These principles
are likely to influence UNESCOs engagement
with big data in the future.
UNESCO, and the UN more broadly, acknowl-
edge that technology has been, and will continue
to be, a driver of the data revolution and a wider
variety of data sources. For big data that is derived
from this technology to have an impact, these data
sources need to be leveraged in order to develop a
greater understanding of the issues related to the
development agenda.
Cross-References
▶ History
▶ International Development
▶ United National Global Pulse
▶ United Nations
▶ World Bank
Further Readings
Gijzen, H. (2013). Development: Big data for a sustainable
future. Nature, 52, 38.
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R.,
Roxburgh, C., Byers, A. (2011). Big data: The next
frontier for innovation, competition, and productivity.
McKinsey Global Institute. New York. http://www.
mckinsey.com/insights/mgi/research/technology_and_
innovation/big_data_the_next_frontier_for_innova
tion. Accessed 12 Nov 14.
Pentland, A. (2013). The data driven society. Scientific
American, 309, 78–83.
UNESCO (2012). Learning analytics. UNESCO Institute
for Information Technologies Policy Brief. Available
from http://iite.unesco.org/pics/publications/en/files/
3214711.pdf. Accessed 11 Nov 14.
United Nations (2014). A world that counts. United
Nations. United Nations. Available from http://www.
unglobalpulse.org/IEAG-Data-Revolution-Report-A-
World-That-Counts. Accessed 28 Nov 14.
V
Visualization
Xiaogang Ma
Department of Computer Science, University of
Idaho, Moscow, ID, USA
Synonyms
Data visualization; Information visualization;
Visual representation
Introduction
People use visualization for information commu-
nication. Data visualization is the study of creat-
ing visual representations of data, which bears two
levels of meaning: the first is to make information
visible and the second is to make it obvious for
understand. Visualization is a pervasive existence
in the data life cycle and recent trends is to pro-
mote the use of visualization in data analysis
rather than use it only as a way to present the
result. Community standards and open source
libraries set the foundation for visualization of
Big Data, and domain expertise and creative
ideas are needed to put standards into innovative
applications.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_202-1
Visualization and Data Visualization
Visualization, in its literal meaning, is the proce-
dure to form a mental picture of something that is
not present to the sight (Cohen et al. 2002). People
can also illustrate such kind of mental pictures by
using various visible media such as papers and
computer screens. Seen as a way to facilitate
information communication, the meaning of visu-
alization can be understood at two levels. The first
level is to make something to be visible and the
second level is to make it obvious so it is easy to
understand (Tufte 1983). People’s daily experi-
ence shows that graphics are easier to read and
understand than words and numbers, such as the
use of maps in automotive navigation systems to
show the location of an automobile and the road to
the destination. The daily experience is approved
by scientific discoveries. Studies on visual object
perceptions explain such differentiation in reading
graphics and texts/numbers: the human brain
deciphers image elements simultaneously and
decodes language in a linear and sequential man-
ner, where the linear process takes more time than
the simultaneous process.
Data are representations of facts and informa-
tion is the meaning worked out from data. In the
context of the Big Data, visualization is a crucial
method to tackle the considerable needs of extra-
cting information from data and presenting
it. Data visualization is the study of creating visual
representations of data. In practice, data visuali-
zation means to visually display one or more
2 Visualization
objects by combined use of words, numbers, sym-
bols, points, lines, color, shading, coordinate sys-
tems, and more. While there are various choices of
visual representations for a same piece of data,
there are a few general guidelines that can be
applied to establish effective and efficient data
visualization. This first is to avoid distorting
what the data have to say. That is, the visualization
should not give a false or misleading account of
the data. The second is to know the audience and
serve a clear purpose. For instance, the visualiza-
tion can be a description of the data, a tabulation
of the records, or an exploration of the information
that is of interest to the audience. The third is to
make large datasets coherent. A few artistic
designs will be required to present the data and
information in an orderly and consistent way. The
presidential, Senate, and House elections of the
United States have been reported with well-
presented data visualization, such as those on the
website of The New York Times. The visualization
on that website is underpinned by dynamic
datasets and can show the latest records
simultaneously.
Visualization in the Data Life Cycle
Visualization is crucial in the process from data to
information. However, information retrieval is
just one of the many steps in the data life cycle,
and visualization is useful through the whole data
life cycle. In conventional understanding, a data
life cycle begins with data collection and con-
tinues with cleansing, processing, archiving, and
distribution. Those are from the perspective of
data providers. Then, from the perspective of
data users, the data life cycles continues with
data discovery, access, analysis, and then
repurposing. From repurposing, the life cycle
may go back to the collection or processing step
restarting the cycle. Recent studies show that there
is another step called concept before the step of
data collection. The concept step covers works
such as conceptual models, logical models, and
physical models for relational databases, and
ontologies and vocabularies for Linked Data in
the Semantic Web.
Visualization or more specifically data visual-
ization provides support to different steps in the
data life cycle. For example, the Unified Modeling
Language (UML) provides a standard way to
visualize the design of information systems,
including the conceptual and logical models of
databases. Typical relationships in UML include
association, aggregation, and composition at the
instance level, generalization and realization at the
class level, and general relationships such as
dependency and multiplicity. For ontologies and
vocabularies in the Semantic Web, concept maps
are widely used for organizing concepts in a sub-
ject domain and the interrelationships among
those concepts. In this way a concept map is the
visual representation of a knowledge base. The
concept maps are more flexible than UML
because they cover all the relationships defined
in UML and allow people to create new relation-
ships that apply to the domain under working
(Ma et al. 2014). For example, there are concept
maps for the ontology of the Global Change Infor-
mation System led by the US Global Change
Research Program. The concept maps are able to
show that report is a subclass of publication, and
there are several components in a report, such as
chapter, table, figure, array, and image. Recent
work in information technologies also enable
online visualized tools to capture and explore
concepts underlying collaborative science activi-
ties, which greatly facilitate the collaboration
between domain experts and computer scientists.
Visualization is also used to facilitate data
archive, distribution, and discovery. For instance,
the Tetherless World Constellation at Rensselaer
Polytechnic Institute recently developed the Inter-
national Open Government Dataset Catalog,
which is a Web-based faceted browsing and
search interface to help users find datasets of
interest. A facet represents a part of the properties
of a dataset, so faceted classification allows the
assignment of the dataset to multiple taxonomies,
and then datasets can be classified and ordered in
different ways. On the user interface of a data
center the faceted classification can be visualized
as a number of small windows and options, which
allows the data center to hide the complexity of
Visualization
data classification, archive and search on the
server side.
Visual Analytics
The pervasive existence of visualization in the
data life cycle shows that visualization can be
applied broadly in data analytics. Yet, in actual
practices visualization is often treated as a method
to show the result of data analysis rather than as a
way to enable the interactions between users and
complex datasets. That is, the visualization as a
result is separated from the datasets upon which
the result is generated. Many of the data analysis
and visualization tools scientists use in nowadays
do not allow dynamic and live linking between
visual representations and datasets, and when
dataset changes, the visualization is no longer
updated to reflect the changes. In the context of
Big Data, many socioeconomic challenges and
scientific problem facing the world are increas-
ingly linked to the interdependent datasets from
multiple fields of research, organizations, instru-
ments, dimensions, and formats. Interactions are
becoming an inherent characteristic of data ana-
lytics with the Big Data, which requires new
methodologies and technologies of data visuali-
zation to be developed and deployed.
Visual analytics is a field of research to address
the requests of interactive data analysis. It com-
bines many existing techniques from data visual-
ization with those from computational data
analysis, such as those from statistics and data
mining. Visual analytics is especially focused on
the integration of interactive visual representa-
tions with the underlying computational process.
For example, the IPython Notebook provides an
online collaborative environment for interactive
and visual data analysis and report drafting.
IPython Notebook uses JavaScript Object Nota-
tion (JSON) as the scripting language, and each
notebook is a JSON document that contains a
sequential list of input/output cells. There are
several types of cells to contain different contents,
such as text, mathematics, plots, codes, and even
rich media such as video and audio. Users can
design a workflow of data analysis through the
3
arrangement and update of cells in a notebook.
A notebook can be shared with others as a normal
file, or it can also be shared with the public using
online services such as the IPython Notebook
Viewer. A completed notebook can be converted
into a number of standard output formats, such as
HyperText Markup Language (HTML), HTML
presentation slides, LaTeX, Portable Document
Format (PDF), and more. The conversion is
done through a few simple operations, so that
means once a notebook is complete, a user only
needs to press a few buttons to generate a scien-
tific report. The notebook can be reused to analyze
other datasets, and the cells inside it can also be
reused in other notebooks.
Standards and Best Practices
Any applications of Big Data will face the chal-
lenges caused by the four dimensions of Big Data:
volume, variety, velocity, and veracity. Com-
monly accepted standards or communities con-
sensus are a proved way to reduce the
heterogeneities between datasets under working.
Various standards have already been used in appli-
cation tackling scientific, social, and business
issues, such as the aforementioned JSON for
transmitting data with human-readable text, the
Scalable Vector Graphics (SVG) for two-
dimensional vector graphics, and the GeoJSON
for representing collections of georeferenced fea-
tures. There are also organizations coordinating
the works on community standards. The World
Wide Web Consortium (W3C) coordinates the
development of standards for the Web. For exam-
ple, the SVG is an output of the W3C. Other W3C
standards include the Resource Description
Framework (RDF), the Web Ontology Language
(OWL), and the Simple Knowledge Organization
System (SKOS). Many of them are used for data
in the Semantic Web. The Open Geospatial Con-
sortium (OGC) coordinates the development of
standards relevant to geospatial data. For exam-
ple, the Keyhole Markup Language (KML) is
developed for presenting geospatial features in
Web-based maps and virtual globes such as Goo-
gle Earth. The Network Common Data Form
4 Visualization
(netCDF) is developed for encoding array-
oriented data. Most recently, the GeoSPARQL is
developed for encoding and querying geospatial
data in the Semantic Web.
Standards just enable the initial elements for
data visualization, and domain expertise and
novel ideas are needed to put standards into prac-
tice (Fox and Hendler 2011). For example, Goo-
gle Motion Chart adapts the fresh idea of motion
charts to extend the traditional static charts, and
the aforementioned IPython Notebook allows the
use of several programming languages and data
formats through the use of cells. There are various
programming libraries developed for data visual-
ization, and many of them are made available on
the Web. The D3.js is a typical example of such
open source libraries (Murray 2013). The D3 here
represents Data-Driven Documents. It is a
JavaScript library using digital data to drive the
creation and running of interactive graphics in
Web browsers. D3.js based visualization uses
JSON as the format of input data and SVG as
the format for the output graphics. The
OneGeology data portal provides a platform to
browse geological map services across the
world, using standards developed by both OGC
and W3C, such as SKOS and Web Map Service
(WMS). GeoSPARQL is a relatively newer stan-
dard for geospatial data but there are already fea-
ture applications. The demo system of the Dutch
Heritage and Location shows the linked open
dataset of the National Cultural Heritage with
more than 13 thousand archaeological monu-
ments in the Netherlands. Besides the
GeoSPARQL, GeoJSON and few other standards
and libraries are also used in that demo system.
Cross-References
▶ Data Visualization
▶ Data-Information-Knowledge-Action Model
▶ Interactive Data Visualization
▶ Pattern Recognition
References
Cohen, L., Lehericy, S., Chochon, F., Lemer, C., Rivaud,
S., & Dehaene, S. (2002). Language-specific tuning of
visual cortex? Functional properties of the visual word
form area. Brain, 125(5), 1054–1069.
Fox, P., & Hendler, J. (2011). Changing the equation on
scientific data visualization. Science, 331(6018),
705–708.
Ma, X., Fox, P., Rozell, E., West, P., & Zednik, S. (2014).
Ontology dynamics in a data life cycle: Challenges and
recommendations from a geoscience perspective. Jour-
nal of Earth Science, 25(2), 407–412.
Murray, S. (2013). Interactive data visualization for the
web. Sebastopol: O’Reilly.
Tufte, E. (1983). The visual display of quantitative infor-
mation. Cheshire: Graphics Press.
W
White House Big Data Initiative
Gordon Alley-Young
Department of Communications & Performing
Arts, Kingsborough Community College,
Kingsborough Community College – City
University of New York, New York, NY, USA
Synonyms
The Big Data Research and Development Initia-
tive (TBDRDI)
Introduction
On March 29, 2012, the White House introduced
The Big Data Research and Development Initia-
tive (TBDRDI) at a cost of $200 million. Big data
(BD) refers to the collection and interpretation of
enormous datasets, using supercomputers running
smart algorithms to rapidly uncover important
features (e.g., interconnections, emerging trends,
anomalies, etc.). The Obama Administration
developed TBDRDI because having the large
amounts of instantaneous data that is continually
being produced by research and development
(R&D) and emerging technology go unprocessed
hurts the US economy and society. President
Obama requested an all-hands-on-deck for
TBDRDI including the public (i.e., government)
and private (i.e., business) sectors to maximize
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_204-1
economic growth, education, health, clean energy,
and national security (Raul 2014; Savitz 2012).
The administration stated that the private sector
would lead by developing BD while the govern-
ment will promote R&D, facilitate private sector
access to government data, and shape public pol-
icy. Several government agencies made the initial
investment in this initiative to advance the tools/
techniques required to analyze and capitalize on
BD. TBDRDI has been compared by the Obama
Administration to previous administrations’
investments in science in technology that lead to
innovations such as the Internet. Critics of the
initiative argue that administration BD efforts
need to be directed elsewhere.
History of the White House Big Data
Initiative
TBDRDI is the White House’s $200 million fed-
eral agency funded initiative that seeks to secure
the US’s position as the world’s most powerful
and influential economy by channeling the infor-
mation power of BD into social and economic
development (Raul 2014). BD is an all-inclusive
name for the nonstop supply of sophisticated elec-
tronic data that is being produced by a variety of
technologies and by scientific inquiry. In short,
BD includes any digital file, tag or data that is
created whenever we interact with technology, no
matter how briefly (Carstensen 2012). The
dilemma posed by BD to the White House, as
2 White House Big Data Initiative
well as to other countries, organizations, and busi-
nesses worldwide, is that so much of it goes
unanalyzed due to its sheer volume and the limits
of our current technological tools to effectively
store, organize, and analyze. Processing BD is not
so simple because it requires supercomputing
capabilities, some of which are still emerging.
Experts argue that up until 2003, only
5-exabytes (EB) of data were produced; that num-
ber has since exploded to over five quintillion
bytes of data (approximately 4.3 EB) every
2 days.
The White House Office of Science and Tech-
nology Policy (WHOSTP) announced TBDRDI
in March 2012 in conjunction with the National
Science Foundation (NSF), National Institutes of
Health (NIH), US Geological Survey (USGS),
and the Department of Defense (DoD) and
Department of Energy (DoE). Key concerns to
be addressed by TBDRDI are to manage BD by
significantly increasing the speed of scientific
inquiry and discovery, bolstering national secu-
rity, and overhauling US education. TBDRDI is
the result of recommendations in 2011 by the
President’s Council of Advisors on Science and
represents the US government’s wish to get ahead
of the wave BD and prevent a cultural lag by
revamping its BD practices (Executive Office of
the President 2014). John Holdren, Director of
WHOSTP, compared the $200 million being
invested in BD to prior federal investments in
science and technology that are responsible for
our current technological age (Scola 2013). The
innovations of the technology age ironically have
created the BD that makes initiatives such as these
necessary.
In addition to the US government agencies that
helped to unveil TBDRDI, several other federal
agencies had been requested to develop BD man-
agement strategies in the time leading up to and
following this initiative. A US government fact
sheet listed between 80 and 85 BD projects across
a dozen federal agencies including, in addition to
the departments previously mentioned, the
Department of Homeland Security (DHS),
Department of Health and Human Services
(DHHS), and the Food and Drug Administration
(FDA) (Henschen 2012). The White House
referred to TBDRDI as representing it placing its
bet on BD meaning that the financial investment
in this initiative is expected to yield a significant
return for the country in coming years. To this
end, President Obama has sought the involvement
of public, private, and other (e.g., academia, non-
governmental organizations) experts and organi-
zations to work in a way that emphasizes
collaboration. For spearheading TBDRDI and
for choosing to stake the future of the country on
BD, President Barack Obama has been dubbed the
BD president by the media.
Projects of the White House Big Data
Initiative
The projects included under the umbrella of
TBDRDI are diverse, but they share common
themes of emphasizing collaboration (i.e., to max-
imize resources and eliminate data overlap) and
making data openly accessible for its social and
economic benefits. One project undertaken with
the co-participation of NIH and Amazon, the
world’s largest online retailer, aims to provide
public access to the 1,000 Genomes Project
using cloud computing (Smith 2012). The 1,000
Genomes Project involved scientists and
researchers sequencing the genomes of over
1,000 anonymous and ethnically diverse people
between 2008 and 2012 in order to better treat
illness and predict medical conditions that are
genetically influenced. The NIH will deposit
200 terabytes (TB) of genomic data into Ama-
zon’s Web Services. According to the White
House, this is currently the world’s largest collec-
tion of human genetic data. In August 2014, the
UK reported that it would undertake a 100,000
genomes project that is slated to finish in 2017.
The NIH and NSF will cooperate to fund 15–20
research projects for a cost of $25 million. Other
collaborations include the DoE’s and University
of California’s creation of a new facility as part of
their Lawrence Berkeley National Laboratory
called the Scalable Data Management, Analysis,
and Visualization Institute ($25 million) and the
NSF and University of California, Berkeley’s
geosciences Earth Cube BD project ($10 million).
White House Big Data Initiative
The CyberInfrastructure for Billions of Elec-
tronic Records (CIBER) project is a co-initiative
of the National Archives and Records Adminis-
tration (NARA), the NSF, and the University of
North Carolina Chapel Hill. The project will
assemble decades of historical and digital-era doc-
uments on demographics and urban development/
renewal. The project draws on citizen-led sourc-
ing or citizen sourcing meaning that the project
will build a participative archive fueled by
engaged community members and not just by
professional archivists and/or governmental
experts. Elsewhere the NSF will partner with
NASA’s on its Global Earth Observation System
of Systems (GEOSS), an international project to
share and integrate Earth observation data. Simi-
larly, the National Oceanic and Atmospheric
Administration (NOAA) and NASA, who collec-
tively oversee hundreds of thousands of environ-
mental sensors producing reams of climate data,
have partnered with Computer Science Corpora-
tion (CSC) to manage this climate data using their
ClimatEdge™ risk management suite of tools.
CSC will collect and interpret the climate data
and make it available to subscribers in the forms
of monthly reports that anticipate how climate
changes could affect global agriculture, global
energy demand/production, sugar/soft commodi-
ties, grain/oilseeds, and energy/natural gas. These
tools are promoted to help companies and con-
sumers make better decisions. For example, fluc-
tuating resource prices caused by climate changes
will allow a consumer/business to find new sup-
plies/suppliers in advance of natural disasters and
weather patterns. Future goals include providing
streaming data to advanced users of the service
and expanding this service to other sectors includ-
ing disease and health trends (Eddy 2014).
The DoD argues that it will spend $250 million
annually on BD. Several of its initiatives promote
cybersecurity like its Cyber-Insider Threat pro-
gram quick and precise targeting of cyber espio-
nage threats to military computer networks. The
DoD’s cybersecurity projects also include devel-
oping cloud-computing capabilities that would
retain function in the midst of an attack, program-
ming languages that stay encrypted whenever in
use, and security programs suitable for BD
3
supercomputer networks. In keeping with
TBDRDI maxim to collaborate and share, the
DoD has partnered with Lockheed Martin Corpo-
ration to provide the military and its partners with
time-sensitive intelligence, surveillance, and
reconnaissance data in what is being called a
Distributed Common Ground System (DCGS).
This project is touted as having the potential to
save individual soldier’s lives on the battlefield.
Other defense-oriented initiatives under TBDRDI
include how the Pentagon is working to increase
its ability to extract information from texts over
100 times its current rates and Defense Advanced
Research Projects Agency’s (DARPA) develop-
ment of XDATA (Raul 2014), a $100 million
program for sifting BD.
Influences of the Initiative and Expected
Outcomes
The United Nations’ (UN) Global Pulse Initiative
(GPI) may have shaped TBDRDI (UN Global
Pulse 2012). Realizing in 2009–2010 that the
data it relied upon to respond to global crises
was outdated, the UN created its GPI to provide
real-time data. In 2011, the proof of concept (i.e.,
primary project) phase began with the analysis of
2 years’ worth of US and Irish social media data
for mood scores/conversation indicators that
could, in some cases, predict economic downturns
5 months out and economic upturns 2 months out.
Success in this project justified opening GPI labs
in Jakarta, Indonesia, and Kampala, Uganda.
Similarly in 2010, President Obama’s Council of
Advisors on Science and Technology urged focused
investment in information technology (IT) to avoid
overlapping efforts (Henschen 2012). This advice fit
with 2010’s existing cost-cutting efforts that were
moving government work to less expensive
Internet-based applications. TBDRDI, emerging
from IT recommendations and after a period of
economic downturn, differs from the so-called
reality-based community (i.e., studying what has
happened) of the Bush Administration by focusing
instead on what will happen in the future. Some also
argue that an inkling of TBDRDI can be seen as
early as 2008 when then Senator Obama
4 White House Big Data Initiative
cosponsored a bipartisan online federal spending
database bill (i.e., for USAspending.gov) and as a
presidential candidate who actively utilized BD
techniques (Scola 2013).
TBDRDI comes at a time when International
Data Corporation (IDC) predicts that by 2020,
over a third of digital information will generate
value if analyzed. Making BD open and accessi-
ble will bring businesses an estimated three tril-
lion dollars in profits. Mark Weber, President of
US Public Sector for NetApp and government IT
commentator, argues that the value of BD lies in
transforming it into quality knowledge for
increasing efficiency better informed decision-
making (CIO Insight 2012). TBDRDI is also
said to national security. Kaigham Gabriel, a Goo-
gle executive and the next CEO and President of
Draper Laboratory, argued that the cluttered
nature of the BD field allows America’s adversar-
ies to hide and that field is becoming increasingly
cluttered as it is estimated that government agen-
cies generated one petabyte (PB) or one quadril-
lion bytes of data from 2012 to 2014 (CIO Insight
2012). One would need almost 14,552
64-gigabyte (GB) iPhones in order to store this
amount of data. Experts argue that the full extent
of technology/applications required to success-
fully manage the amounts BD that TBDRDI
could produce now and in the future remains to
be seen.
President Obama promised that TBDRDI
would stimulate the economy and save taxpayer
money, and there is evidence to indicate this. The
employment outlook for individuals trained in
mathematics, science, and technology is strong
as the US government attempts to hire sufficient
staff to carry out the work of TBDRDI. Hiring
across governmental agencies requires the skilled
work of deriving actionable knowledge from
BD. This responsibility falls largely on a subset
of highly trained professionals known as quanti-
tative analysts or the quants for short. Currently
these employees are in high demand and thus can
be difficult to source as the US government must
compete alongside private sector businesses for
talent when the latter may be able to provide larger
salaries and higher profile positions (e.g., Wall
Street firms). Some have argued for the
government to invest more money in the training
of quantitative analysts to feed initiatives such
as this (Tucker 2012).
In terms of cutting overspending, cloud com-
puting (platform-as-a-service technologies) has
been identified under TBDRDI as a means to
consolidate roughly 1,200 unneeded federal data
centers (Tucker 2012). The Obama Administra-
tion has stated that it will eliminate 40 % of federal
data centers by 2015. This is estimated to generate
a $5 billion in savings. Some in the media applaud
the effort and corresponding savings while some
critics of the plan argue that the data centers be
streamlined and upgraded instead. As of 2014, the
US government reports that 750 data centers have
been eliminated.
In January 2014, after classified information
leaks by former NSA contractor Edward Snowden,
President Obama asked the White House for a com-
prehensive review of BD that some argue dampened
the enthusiasm for TBDRDI (Raul 2014). The US
does not have a specific BD privacy law leading
critics to claim a policy deficit. Others point to the
Federal Trade Commission (FTC) Act, Section 5
that prohibits unfair or deceptive acts or practices in
or affecting commerce as being firm enough to
handle any untoward business practices that might
emerge from BD while flexible enough to not hinder
the economy (Raul 2014). Advocates note that the
European Union (EU) has adopted a highly detailed
privacy policy that has done little to foster commer-
cial innovation and economic growth (Raul 2014).
Conclusion
Other criticism argues that TBDRDI, and the
Obama Administration by default, actually serves
big business instead of individual consumers and
citizens. In support of this argument, critics argue
that the administration pressured communications
companies to provide more affordable and higher
speeds of mobile broadband. As of the summer of
2014, Hong Kong has the world’s fastest mobile
broadband speeds that are also some of the most
affordable with South Korea second and Japan
third; the US and its neighbor Canada are not
even in the top ten list of fastest mobile broadband
White House Big Data Initiative
speed countries. Supporters of the administration
cite that the Obama Administration has instead
chosen to emphasize its unprecedented open data
initiatives under TBDRDI. The US Open Data
Action Plan emphasizes making high-priority
US government data both mobile and publically
accessible while Japan is reported to have fallen
behind in open-sourcing its BD, specifically in
providing access to their massive stores of state/
local data, costing its economy trillions of yen.
Cross-References
▶ Big Data
▶ Cloud or Cloud Computing
▶ Cyberinfrastructure
▶ Defense Advanced Research Projects Agency
(DARPA)
▶ Department of Homeland Security
▶ Food and Drug Administration (FDA)
▶ NASA
▶ National Oceanic and Atmospheric
Administration
▶ National Science Foundation
▶ Office of Science and Technology Policy
▶ United Nations Global Pulse (Development)
▶ United States Geological Survey (USGS)
References
Carstensen, J. (2012). Berkeley group digs in to challenge of
making sense of all that data. Retrieved from http://
www.nytimes.com/2012/04/08/us/berkeley-group-tries-
to-make-sense-of-big-data.html?_r=0.
5
CIO Insight (2012). Can government IT meet the big data
challenge? Retrieved from http://www.cioinsight.com/
c/a/Latest-News/Big-Data-Still-a-Big-Challenge-for-G
overnment-IT-651653/.
Eddy, N. (2014). Big data proves alluring to federal IT pros.
Retrieved from http://www.eweek.com/enterprise-apps/
big-data-proves-alluring-to-federal-it-pros.html.
Executive Office of the President (2014). Big data: Seizing
opportunities, preserving values. Retrieved from https://
www.whitehouse.gov/sites/default/files/docs/big_data_
privacy _report_may_1_2014.pdf.
Henschen, D. (2012). Big data initiative or
big government boondoggle? Retrieved from http://
www.informationweek.com/software/information-man
agement/big-data-initiative-or-big-government-boondog
gle/d/d-id/1103666?
Raul, A.C. (2014). Don’t throw the big data out with the bath
water. Retrieved from http://www.politico.com/maga
zine/story/2014/04/dont-throw-the-big-data-out-with-the-
bath-water-106168_full.html?print#.U_PA-lb4bFI.
Savitz, E. (2012). Big data in the enterprise: A lesson or
two from big brother. Retrieved from http://www.
forbes.com/sites/ciocentral/2012/12/26/big-data-in-the-
enterprise-a-lesson-or-two-from-big-brother/.
Scola, N. (2013). Obama, the ‘big data’ president. Retrieved
from http://www.washingtonpost.com/opinions/obama-
the-big-data-president/2013/06/14/1d71fe2e-d391-11e2-
b05f-3ea3f0e7bb5a_story.html.
Smith, J. (2012). White House aims to tap power of gov-
ernment data. Retrieved from https://www.yahoo.
com/news/white-house-aims-tap-power-government-
data-093701014.html?ref=gs.
Tucker, S. (2012). Budget pressures will drive government
IT change. Retrieved from http://www.washingtonpost.
com/business/capitalbusiness/budget-pressures-will-dri
ve-government-it-change/2012/08/24/ab928a1e-e898-
11e1-a3d2-2a05679928ef_story.html.
UN Global Pulse. (2012). Big data for development: Chal-
lenges & opportunities. Retrieved from UN Global Pulse,
Executive Office of the Secretary-General United
Nations, New York, NY at http://www.unglobalpulse.
org/sites/default/files/BigDataforDevelopment-UNGl
obalPulseJune2012.pdf.
W
White House BRAIN Initiative
Gordon Alley-Young
Department of Communications & Performing
Arts, Kingsborough Community College, City
University of New York, New York, NY, USA
Synonyms
Brain Research Through Advancing Innovative
Neurotechnologies
Introduction
The White House BRAIN Initiative (TWHBI)
includes an acronym where BRAIN stands for
the Brain Research Through Advancing Innova-
tive Neurotechnologies. The goal of the initiative
is to spur brain research, such as mapping the
brain’s circuitry, and technology that will lead to
treatments and preventions for common brain dis-
orders. President Barack Obama first announced
the initiative in his February 2013 State of the
Union Address (SOTHA). More than 200 leaders
from universities, research institutes, national lab-
oratories, and federal agencies were invited to
attend when President Obama formally unveiled
TWHBI on April 2, 2013. The Obama adminis-
tration identified this initiative as one of the grand
challenges of the twenty-first century. The $100
million initiative is funded via The National
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_205-1
Institutes of Health (NIH), the Defense Advanced
Research Projects Agency (DARPA), and the
National Science Foundation (NSF) with
matching support for the initiative reported to
come from private research institutions and foun-
dations. TWHBI has drawn comparisons to the
Human Genome Project (HGP) for the potential
scientific discovery that the project is expected to
yield. The HGP and TWHBI are also big data
projects for the volume of data that they have
already produced and will produce in the future.
History and Aims of the Initiative
TWHBI aims to provide opportunities to map,
study, and thus treat brain disorders including
Alzheimer’s disease, epilepsy, autism, and trau-
matic brain injuries. The NIH will lead efforts
under the initiative to map brain circuitry, measure
electrical/chemical activity along those circuits,
and understand the role of the brain in human
behavioral and cognitive output. The initiative is
guided by eight key goals. The first is to make
various types of brain cells available for experi-
mental researchers to study their role in illness and
well-being. The second is to create multilayered
maps of the brain’s different circuitry levels as
well as a map of the whole organ. The third
would see the creation of a dynamic picture of
the brain through large-scale monitoring of neural
activity. Fourth is to link brain activity to behavior
with tools that could intervene in and change
2 White House BRAIN Initiative
neural circuitry. A fifth goal is to increase under-
standing of the biological basis for mental pro-
cesses by theory building and developing new
data analysis tools. The sixth is to innovate tech-
nology to better understand the brain so as to
better treat disorders. The seventh is to establish
and sustain interconnected networks of brain
research. Finally, the last goal is to integrate the
outcomes of the other goals to discover how
dynamic patterns of neural activity get translated
into human thought, emotion, perception, and
action in illness and in health.
NIH Director Dr. Francis Collins echoed Pres-
ident Obama in publically stating that TWHBI
will change the way we treat the brain and grow
the economy (National Institutes of Health 2014).
During his 2013 SOTUA, President Obama drew
an analogy to the Human Genome Project (HGP)
arguing that for every dollar the USA invested in
the project, the US economy gained $140. Esti-
mates suggest that the HGP created $800 billion in
economic activity. The HGP was estimated to cost
$3 billion and take 15 years (i.e., 1990–2005). The
project finished 2 years early and under cost at
$2.7 billion in 1991 dollars. The HGP project is
estimated to have cost $3.39–$5 billion in 2003
dollars. TWHBI has a budget of $100 million
allocated in budget year 2014 with comparable
funds ($122 million) contributed by private inves-
tors. A US federal report calls for $4.5 billion in
funding for brain research over the next 12 years.
Projects Undertaken by the Initiative
The first research paper believed to be produced
under TWHBI initiative came from a paper
published on June 19, 2014, by principal investi-
gator Dr. Karl Deisseroth of Stanford University.
The research described Deisseroth and his team’s
innovation of the CLARITY technique that can
remove fat from the brain without damaging its
wiring and enable the imaging of a whole trans-
parent brain. Data from the study is being used by
international biomedical research projects.
TWHBI was undertaken because it addresses
what the science, society, and government con-
siders one of the grand challenges of the twenty-
first century (i.e., The HGP was previously
deemed a grand challenge). Unlocking the secrets
of the brain will tell us how the brain can record,
process, utilize, retain, and recall large amounts of
information. Dr. Geoffrey Ling, deputy director of
the Defense Sciences Office at Defense Advanced
Research Projects Agency (DARPA), states that
TWHBI is needed to attract young and intelligent
people into the scientific community. Ling cites a
lack of available funding as a barrier to persuading
students to pursue research careers (Vallone
2013). Current NIH director and former HGP
director Dr. Francis Sellers Collins notes the
potential of TWHBI to create jobs while poten-
tially curing diseases of the brain and the nervous
system, for instance, Alzheimer’s disease (AD). In
2012 Health and Human Services Secretary
Kathleen Sebelius stated the Obama administra-
tion’s goal to cure AD by 2025. The Alzheimer’s
Association (AA) estimates that AD/dementia
health and care cost $203 billion in 2013 ($142
billion by Medicare/Medicaid); this will reach
$1.2 trillion by 2050 (Alzheimer’s Association
2013).
Dr. Ling argues that for scientists to craft and
validate their hypotheses to build on their knowl-
edge that potentially lead to medical break-
throughs, they need access to the latest research
tools. Ling states that some of the today’s best
clinical brain research tools are nonetheless lim-
ited and outdated in light of TWHBI work that
remains to be done. To bolster his case for better
research tools, Ling uses an analogy whereby the
physical brain is hardware and the dynamic pro-
cesses across the brain’s circuits are software.
Ling notes that cutting-edge tools can help iden-
tify bugs in the brain’s software caused by a phys-
ical trauma (i.e., to the hardware) that once found
might be repairable. The tools necessary for med-
ical research will need to be high-speed tools with
a much greater capacity for record signals from
brain cells. TWHBI, by bringing together scien-
tists and researchers from a variety of fields such
as nanoscience, imaging, engineering, informat-
ics, has the greatest opportunity to develop these
tools.
White House BRAIN Initiative
Earlier Efforts and Influences
Brain research was emphasized prior to TWHBI
by the previous two administrations. The Clinton
administration held a White House conference on
early childhood development and leaning focused
on insights gleaned from the latest brain research
in 1997. In 2002 the Bush administration’s
National Drug Control Policy Director John
Walters donated millions of dollars of drug-war
money to purchase dozens of MRI machines.
Their goal was a decade long, $100 million
brain-imaging initiative to study the brain to better
understand addiction.
Publicity surrounding TWHBI brings attention
to how much science has learned about the brain
in relatively short period of time. In the nineteenth
century, brain study focused mostly on what hap-
pens when parts of the brain are damaged/
removed. For instance, Phineas Gage partially
lost his prefrontal cortex in an 1848 accident,
and scientists noted how Mr. Gage changed from
easygoing and dependable before to angry and
irresponsible afterward. From the late eighteenth
to mid-nineteenth centuries, pseudoscientists
practiced phrenology or reading a person’s mind
by handling a person’s skull.
Phillip Low, a director of San Diego-based
NeuroVigil Inc. (NVI), states that the White
House talked to many scientists and researchers
while planning TWHBI but did not reveal to these
individuals that they were talking to many others,
all of who potentially believed they were the par-
ent of TWHBI. However, the originators of the
idea that lead to TWHBI are said to be six scien-
tists, whose journal article in the June 2012 issue
of Neuron proposed a brain-mapping project. The
six are A. Paul Alivisatos (University of Califor-
nia Berkeley), Miyoung Chun (The Kavli Foun-
dation), George M. Church (Harvard University),
Ralph J. Greenspan (The Kavli Institute), Michael
L. Roukes (Kavli Nanoscience Institute), and
Rafael Yuste (Columbia University) (Alivisatos
et al. 2012). New York Times reporter Steve
Connor says the roots of TWHBI occur 10 years
earlier when Microsoft cofounder and philanthro-
pist Paul G. Allen established a brain science
institute in Seattle for a $300 million investment.
3
Similarly, with a $500 million investment, billion-
aire philanthropist Fred Kavli funded brain insti-
tutes at Yale, Columbia, and the University of
California (Broad 2014). It was primarily scien-
tists from these two institutes that crafted the
TWHBI blueprint. Connor states that there are
benefits and downsides to TWHBI’s connections
to private philanthropy. Connor acknowledges
that philanthropists are able to invest in risky
initiatives in a way that the government cannot
but that this can lead to a self-serving research
focus, the privileging of affluent universities at the
expense of poorer ones and a US government that
is following the lead of private interests rather
than setting the course itself (Connor 2013).
The $100 million for the first phase of TWHBI
in fiscal year 2014 comes from three government
agencies’ budgets specifically NIH, DARPA, and
NSF. The NIH Blueprint for Neuroscience
Research will lead with contributions specifically
geared to projects that would lead to the develop-
ment of cutting edge, high-speed tools, training,
and other resources. The next generation of tools
has designated as viewed as vital to the advance-
ment of this initiative. Contributor DARPA will
invest in programs that aim to understand the
dynamic functions of the brain, noted in
Dr. Ling’s analogy as the software of the brain,
showing breakthrough applications based on the
dynamic function insights gained. DARPA also
seeks to develop new tools for capturing and
processing dynamic neural and synaptic activities.
DARPA develops applications for improving the
diagnosis and treatment of post-traumatic stress,
brain injury, and memory loss sustained through
war and battle. Such applications would include
generating new information processing systems
related to the information processing system in
the brain and mechanisms of functional restora-
tion after brain injury. DARPA is mindful that
advances in neurotechnology, such as those
outlined above, will entail ethical, legal, and
social issues that it will oversee via its own
experts. Ethics are also at the forefront of
TWHBI. Specifically President Obama identified
adhering to the highest standards of research pro-
tections as a prime focus. Oversight of ethical
issues related to this as well as any other
4 White House BRAIN Initiative
neuroscience initiative will fall to the administra-
tion’s Commission for the Study of Bioethical
Issues.
The NSF’s strength as a contributor to TWHBI
is that it will sponsor interdisciplinary research
that spans the fields of biology, physics, engineer-
ing, computer science, social science, and behav-
ioral science. The NSF’s contribution to TWHBI
again emphasizes the development of tools and
equipment specifically molecular-scale probes
that can sense and record the activity of neural
networks. Additionally, the NSF will also seek to
address the innovations that will be necessary in
the field of big data in order to store, organize, and
analyze the enormous amounts of data that will be
produced. Finally, NSF projects under TWHBI
will see better understanding of how thoughts,
emotions, actions, and memories get represented
in the brain.
In addition to federal government agencies, at
least four private institutes and foundations have
pledged an estimated $122 million to support to
TWHBI: The Allen Institute (TAI), the Howard
Hughes Medical Institute (HHMI), The Kavli
Foundation (TKF), and The Salk Institute for Bio-
logical Studies (TSI). TAI’s strengths lie in large-
scale brain research, tools, and data sharing which
is necessary for a big data project like TWHBI
represents. Starting in March 2012, TAI under-
took a 10-year project to unlock the neural code
(i.e., how brain activity leads to perception,
decision-making, and action). HHMI by compar-
ison is the largest nongovernmental funder of
basic biomedical research and has long supported
neuroscience research. TKF anticipates drawing
on the endowments of existing Kavli Institutes
(KI) to fund its participation in TWHBI. This
includes funding new KIs. Finally the TSI, under
its dynamic BRAIN initiative, will support cross-
boundary research in neuroscience. For example,
TSI researchers will map brain’s neural networks
to determine their interconnections. TSI scientists
will lay the groundwork for solving neurological
puzzles such as Alzheimer’s/Parkinson’s by
studying age-related brain differences (The
White House 2013).
The work of TWHBI will be spread across
affiliated research institutions and laboratories
across the USA. The NIH is said to be establishing
a bicoastal cochaired working group under
Dr. Cornelia Bargmann, a former UCSF Profes-
sor, with the Rockefeller University in New York
City and Dr. William Newsome from California’s
Stanford University to specify goals for the NIH’s
investment and create a multiyear plan for achiev-
ing these goals with timelines and costs (Univer-
sity of California San Francisco 2013). On the east
coast of the USA, the NIH Blueprint for Neuro-
science Research, draws on 15 of its 27 NIH Insti-
tutes and Centers headquartered in Bethesda, MD,
will be a leading NIH contributor to TWHBI.
Research will occur in nearby Virginia at
HHMI’s Janelia Farm Research Campus that
focuses on developing new imaging technologies
and finding out how information is stored and
processed in neural networks. Imaging technol-
ogy furthers TWHBI’s goals of mapping the
brain’s structures by allowing researchers to cre-
ate dynamic brain pictures down to the level of
single brain cells as they interact with complex
neural circuits at the speed of thought.
Conclusion
Contributions to and extensions of TWHBI are
also happening on the US west coast and interna-
tionally. San Diego State University (SDSU) is
contributing to TWHBI via its expertise in clinical
and cognitive neuroscience specifically their
investigations to understand and treat brain-
based disorders like autism, aphasia, fetal alcohol
spectrum (FAS) disorders, and AD. San Diego’s
NVI, founded in 2007 and advised by Dr. Stephen
Hawking, and its founder, CEO, and Director
Dr. Phillip Low, helped to shape TWHBI initia-
tive. NVI’s is notable for its iBrain™ single-
channel electroencephalograph (EEG) device
that noninvasively monitors the brain (Keshavan
2013). Dr. Low has also taken the message of the
WBHI international as he was asked to go to Israel
and help them develop their own BRAIN initia-
tive. To this end Dr. Low delivered one of two
keynotes for Israel’s first International Brain
Technology Conference in Tel Aviv in October
2013. Australia also supports TWHBI through
White House BRAIN Initiative 5

neuroscience research collaboration and increased Alzheimer’s Association. (2013). Alzheimer’s Association
hosting of the NSF’s US research fellows for applauds White House Brain Mapping Initiative.
Retrieved from Alzheimer’s Association National
collaborating on relevant research projects. Office, Chicago, IL at http://www.alz.org/news_and_
events_alz_association_applauds_white_house.asp
Broad, W.J. (2014). Billionaires with big ideas are
privatizing American science. Retrieved from The
Cross-References New York Times, New York, NY http://www.nytimes.
com/2014/03/16/science/billionaires-with-big-ideas-
▶ Australia are-privatizing-american-science.html
Connor, S. (2013). One of the biggest mysteries in the
▶ Big Data universe is all in the head. Retrieved from Independent
▶ Data Sharing Digital News and Media, London, UK at http://www.
▶ Defense Advanced Research Projects Agency independent.co.uk/voices/comment/one-of-the-biggest-
(DARPA) mysteries-in-the-universe-is-all-in-the-head-8791565.
html
▶ Engineering Keshavan, M. (2013). BRAIN Initiative will tap our best
▶ Human Genome Project minds. San Diego Business Journal, 34(15), 1.
▶ Medicare National Institutes of Health. (2014). NIH embraces bold,
▶ Medical/Health Care 12-year scientific vision for BRAIN Initiative. Retrieved
from National Institutes of Health, Bethesda, MD at
▶ Medicaid http://www.nih.gov/news/health/jun2014/od-05.htm
▶ National Institutes of Health The White House. (2013). Fact sheet: BRAIN Initiative.
▶ National Science Foundation Retrieved from The White House Office of the Press
▶ Neuroscience Secretary, Washington, DC at http://www.whitehouse.
gov/the-press-office/2013/04/02/fact-sheet-brain-initiative
University of California San Francisco. (2013). President
Obama unveils brain mapping project. Retrieved from
the University Of California San Francisco at http://
References www.ucsf.edu/news/2013/04/104826/president-obama-
unveils-brain-mapping-project
Alivisatos, A. P., Chun, M., Church, G. M., Greenspan, Vallone, J. (2013). Federal initiative takes aim at treating
R. J., Roukes, M. L., & Yuste, R. (2012). The brain brain disorders. In Investors Business Daily, Los
activity map project and the challenge of functional Angeles, CA, (p. A04).
connectomics. Neuron, 74(6), 970–974.
W
WikiLeaks
Kim Lacey
Saginaw Valley State University, University
Center, MI, USA
WikiLeaks is a nonprofit organization devoted to
sharing classified, highly secretive, and otherwise
controversial documents to promote transparency
among global superpowers. These shared docu-
ments are commonly referred to as “leaks.”
WikiLeaks has received both highly positive and
negative attention for this project particularly
because of its mission to share leaked information.
WikiLeaks is operated by the Icelandic Sunshine
Press, and Julian Assange is often named the
founder of the organization.
WikiLeaks began in 2006, and its founding is
largely attributed to Australian Julian Assange,
often described as an Internet activist and hacker.
The project, which aims to share government doc-
uments usually kept from citizens, is a major
source of division between individuals and offi-
cials. The perspective on this division differs
depending on the viewpoint. From the perspective
of its opponents, the WikiLeaks documents are
obtained illegally, and their distribution is poten-
tially harmful for national security purposes.
From the perspective of its supporters, the docu-
ments point to egregious offenses perpetrated, and
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_206-1
ultimately stifled, by governments. On its website,
WikiLeaks notes that it is working toward what it
calls “open governance,” the idea that leaks are
not only for international, bureaucratic diplomacy
but more importantly for clarity of citizens’
consciousness.
In 2010, Chelsea (born Bradley) Manning
leaked a United States’ military cable containing
400,000 files regarding the Iraq War. According to
Andy Greenberg, this leak, which later became
known as Cablegate, marked the largest leak of
United States’ government information since
Daniel Ellsberg photocopied The Pentagon
Papers. After chatting for some time, Manning
confessed to former hacker Adrian Lamo. Even-
tually, Lamo turned Manning over to the army
authorities leading to her arrest. The United
States’ government officials were outraged by
the leak of classified documents and viewed Man-
ning as a traitor. This leak eventually led to Man-
ning’s detention, and officials kept her detained
for more than 1,000 days without a trial. Because
of this delay, supporters of WikiLeaks were out-
raged at Manning’s denial of a swift trial. Man-
ning was eventually acquitted of aiding the
enemy, but, in August 2013, was sentenced to
35 years for various crimes including violations
of the Espionage Act.
One of the most well-known documents Man-
ning shared put WikiLeaks on the map for many
who were previously unfamiliar with the
2 WikiLeaks
organization. This video, known familiarly as
“Collateral Murder,” shows a United States’
Apache helicopter shooting Reuters reporters,
individuals helping these reporters, and seriously
injuring two children. There have been two ver-
sions of the video that have been released: a
shorter, 17-min video and a more detailed
39-min video. Both videos were leaked by
WikiLeaks and remain on its website.
WikiLeaks uses a number of different drop
boxes in order to obtain documents and maintain
the anonymity of the leakers. Many leakers are
well versed in anonymity protective programs
such as Tor, which uses what they call “onion
routing”: several layers of encryption to avoid
detection. However, in order to make leaking
less complicated, WikiLeaks provides instruc-
tions on its website for users to skirt around reg-
ular detection through normal identifiers. Users
are instructed to submit documents in one of
many anonymous drop boxes to avoid detection.
In order to verify the authenticity of a docu-
ment, WikiLeaks performs several forensic tests
including weighing the price of forgery as well as
possible motives for falsifying information. On its
website, WikiLeaks explains that it verified the
now infamous “Collateral Murder” video by actu-
ally sending journalists to interview individuals
affiliated with the attack. WikiLeaks states that
simply when it publishes a document, the fact
that it has been published is verification enough.
By making information more freely available,
WikiLeaks aims to start a larger conversation
within the press about access to authentic docu-
ments and democratic information.
Funding for WikiLeaks has been a contentious
issue since its founding. Since 2009, Assange has
noted several times that WikiLeaks is in danger of
running out of funding. One of the major reasons
causing these funding shortages is the result of
many corporations (including Visa, MasterCard,
and PayPal) ceasing to allow its customers to
donate money to WikiLeaks. On the WikiLeaks
website, this action is described as the “banking
blockade.” To work around this banking block-
ade, many mirror sites (websites that are hosted
separately but contain the same information) have
appeared allowing users to access WikiLeaks doc-
uments and also donate with “blocked” payment
methods. WikiLeaks also sells paraphernalia on
its website, but it is unclear if these products fall
under the banking blockade restrictions.
Because of his affiliation with WikiLeaks,
Julian Assange has been granted political asylum
in Ecuador in 2012. Prior to his asylum, he had
been accused of molestation and rape in Sweden
but evaded arrest. In June 2013, Edward Snow-
den, a former employer of the National Security
Agency (NSA), leaked evidence of the United
States spying on its citizens to the UK’s The
Guardian. On many occasions, WikiLeaks has
supported Snowden, helping him apply for polit-
ical asylum, providing funding, and also provid-
ing him with escorts him on flights (most notably
Sarah Harrison accompanying Snowden from
Hong Kong to Russia).
WikiLeaks has been nominated for multiple
awards for reporting. Among the awards, it has
won including the Economist Index on Censor-
ship Freedom of Expression award (2008) and the
Amnesty International human rights reporting
award (2009, New Media). In 2011, Norwegian
citizen Snorre Valen publically announced that he
nominated Julian Assange for the Nobel Peace
Prize, although Assange did not win.
Cross-References
▶ Anonymization
▶ National Security Agency (NSA)
▶ Transparency
Further Readings
Dwyer, D. n.d. “WikiLeaks’ Assange for Nobel Prize?”
ABC News. Available at: http://abcnews.go.com/Poli
tics/wikileaks-julian-assange-nominated-nobel-peace-
prize/story?id=12825383. Accessed 28 Aug 2014.
Greenberg, A. This machine kills secrets: How
wikileakers, cypherpunks, and hacktivists aim to free
the world’s information. Dutton: New York, 2012.
WikiLeaks
Sifry, Micah L 2011. WikiLeaks and the age of transpar-
ency. O/R Books: New York, Wikileaks.org.
WikiLeaks. Available at: https://www.wikileaks.org/.
Accessed 28 Aug 2014.
Tate, J. n.d., “Bradley Manning Sentenced to 35 Years in
WikiLeaks Case.” Washington Post Available at: http://
www.washingtonpost.com/world/national-security/
3
judge-to-sentence-bradley-manning-today/2013/08/
20/85bee184-09d0-11e3-b87c-476db8ac34cd_story.
html. Accessed 26 Aug 2014.
WikiRebels: The Documentary. n.d.. Available at: https://
www.youtube.com/watch?v=z9xrO2Ch4Co.
Accessed 1 Sept 2012.
W
Wikipedia
Ryan McGrady
North Carolina State University, Raleigh, NC,
USA
Wikipedia is an open-access online encyclopedia
hosted and operated by the Wikimedia Founda-
tion (WMF), a San Francisco-based nonprofit
organization. Unlike traditional encyclopedias,
Wikipedia is premised on an open editing model
whereby everyone using the site is allowed and
encouraged to contribute content and make
changes. Since its launch in 2001, it has grown
to over 40 million articles across nearly three
hundred languages, constructed almost entirely
by unpaid pseudonymous and anonymous users.
Since its infancy, Wikipedia has attracted
researchers from many disciplines to its vast col-
lection of user-generated knowledge, unusual pro-
duction model, active community, and open
approach to data.
Wikipedia works on a type of software called a
wiki, a popular kind of web application designed
to facilitate collaboration. Wiki pages can be mod-
ified directly using a built-in text editor. When a
user saves his or her changes, a new version of the
article is created and immediately visible to the
next visitor. Part of what allows Wikipedia to
maintain standards for quality is the meticulous
record-keeping of changes provided by wiki soft-
ware, storing each version of a page permanently
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_207-1
in a way that is easily accessible. If someone
makes changes that are not in the best interest of
the encyclopedia, another user can easily see the
extent of those changes and if necessary restore a
previous version or make corrections. Each
change is timestamped and attributed to either a
username or, if made anonymously, an IP address.
Although Wikipedia is transparent about what
data it saves and draws little criticism on privacy
matters, any use of a wiki requires self-awareness
given that one’s actions will be archived
indefinitely.
Article histories largely comprise the
Wikipedia database, which the WMF makes
available to download for any purpose compatible
with its Creative Commons license, including
mirroring, personal and institutional offline use,
and data mining. The full English language data-
base download amounts to more than ten
terabytes, with several smaller subsets available
that, for example, exclude discussion pages and
user profiles or only include the most current
version of each page.
As with any big data project, there is a chal-
lenge in determining not just what questions to
ask but how to use the data to convey meaningful
answers. Wikipedia presents an incredible amount
of knowledge and information, but it is widely
dispersed and collected in a database organized
around articles and users, not structured data. One
way the text archive is rendered intelligible is
through visualization, wrangling the unwieldy
information by expressing statistics and patterns
2 Wikipedia
through visuals like graphs, charts, or histograms.
Given the multi-language and international nature
of Wikipedia, as well as the disproportionate size
and activity of the English version in particular,
geography is important in its critical discourse.
Maps are thus popular visuals to demonstrate
disparities, locate concentrations, and measure
coverage or influence. Several programs have
been developed to create visualizations using
Wikipedia data, as well. One of the earliest, the
IBM History Flow tool, produces images based on
stages of an individual article’s development over
time, giving a manageable, visual form to an
imposingly long edit history and the disagree-
ments, vandalism, and controversies it contains.
The Wikipedia database has been and con-
tinues to be a valuable resource, but there are
limitations to what can be done with its unstruc-
tured data. It is downloaded as a relational data-
base filled with text and markup, but machines
that researchers use to process data are not able to
understand text like a human, limiting what tasks
they can be given. It is for this reason there have
been a number of attempts to extract structured
data as well. DBPedia is a database project started
in 2007 to put as much of Wikipedia into the
Resource Description Framework (RDF) as pos-
sible. Whereas content on the web typically
employs HTML to display and format text, mul-
timedia, and links, RDF emphasizes not what a
document looks like but how its information is
organized, allowing for arbitrary statements and
associations which effectively make the items
meaningful to machines. The article for the film
Moonlight Kingdom may contain the textual
statement “it was shot in Rhode Island,” but a
machine would have a difficult time extracting
the desired meaning, instead preferring to see a
subject “Moonlight Kingdom” with a standard
property “filming location” set to the value
“Rhode Island.”
In 2012, WMF launched Wikidata, its own
structured database. In addition to Wikipedia,
WMF operates a number of other sites like
Wiktionary, Wikinews, Wikispecies, and
Wikibooks. Like Wikipedia, these sites are avail-
able in many languages, each more or less inde-
pendent from the others. To solve redundancy
issues and to promote resource sharing, the
Wikimedia Commons was introduced in 2004 as
a central location for images and other media for
all WMF projects. Wikidata works on a similar
premise with data. Its initial task was to centralize
inter-wiki links, which connect, for example, the
English article “Cat” to the Portuguese “Gato”
and Swedish “Katt.” Inter-language links had pre-
viously been handled locally, creating links at the
bottom of an article to its counterparts at every
other applicable version. Since someone adding
links to the Tagalog Wikipedia is not likely to
speak Swedish, and because someone who speaks
Swedish is not likely to actively edit the Tagalog
Wikipedia and vice versa, this process frequently
resulted in inaccurate translations, broken links,
one-way connections, and other complications.
Wikidata helps by acting as a single junction for
each topic.
A topic, or an item, on Wikidata is given its
own page which includes an identification num-
ber. Users can then add a list of alternative terms
for the same item and a brief description in every
language. Items also receive statements
connecting values and properties. For example,
The Beatles’s 1964 album A Hard Day’s Night is
item Q182518. The item links to the album’s
Wikipedia articles in 49 languages and includes
17 statements with properties and values. The
very common instance of property has the value
“album,” a property called record label has the
value “Parlophone Records,” and four statements
connect the property genre with “rock and roll,”
“beat music,” “pop music,” and “rock music.”
Other statements describe its recording location,
personnel, language, and chronology, and many
applicable properties are not yet filled in. Like
Wikipedia, Wikidata is an open community pro-
ject and anybody can create or modify statements.
Some of the other properties items are given
include names, stage names, pen names, dates,
birth dates, death dates, demographics, genders,
professions, geographic coordinates, addresses,
manufacturers, alma maters, spouses, running
mates, predecessors, affiliations, capitals, awards
won, executives, parent companies, taxonomic
orders, and architects, among many others. So as
to operate according to the core Wikipedia tenet of
Wikipedia
neutrality, multiple conflicting values are allowed.
Property-value pairs can furthermore be assigned
their own property-value pairs such that the
record sales property and its value can have the
qualifier as of and another value to reflect when
the sales figure was accurate. Each property-value
pair along the way can be assigned references akin
to cited sources on Wikipedia.
Some Wikipedia metadata is easy to locate and
parse as fundamental elements of wiki technol-
ogy: timestamps, usernames, and article titles, for
example. Other data is incidental, like template
parameters. Design elements that would other-
wise be repeated in many articles are frequently
copied into a separate template which can then be
invoked when relevant, using parameters to cus-
tomize it for the particular page on which it is
displayed. For example, in the top-right corner
of articles about books there is typically a neatly
formatted table called an infobox which includes
standardized information input as template
parameters like author, illustrator, translator,
awards received, number of pages, Dewey deci-
mal classification, and ISBN number.
A fundamental part of DBPedia and the second
goal for Wikidata is the collection of data based on
these relatively few structured fields that exist in
Wikipedia.
Standardizing the factual information in
Wikipedia holds incredible potential for research.
Wikidata and DBPedia, used in conjunction with
the Wikipedia database, make it possible to, for
example, assess article coverage of female musi-
cians as compared to male musicians in different
parts of the world. Since they use machine-
readable formats, they can also interface with
one another and with many other sources like
GeoNames, Library of Congress Subject Head-
ings, Internet Movie Database, MusicBrainz, and
Freebase, allowing for richer, more complex
queries. Likewise, just as these can be used to
3
support Wikipedia research, Wikipedia can be
used to support other forms of research and even
enhance commercial products. Google, Facebook,
IBM, and many others regularly make use of data
from Wikipedia and Wikidata in order to improve
search results or provide better answers to ques-
tions. By creating points of informational inter-
section and interpretation for hundreds of
languages, Wikidata also has potential for use in
translation applications and to enhance cultural
education. The introduction of Wikidata in 2012,
built on an already impressively large knowledge
base, and its ongoing development, have opened
many new areas for exploration and accelerated
the pace of experimentation, incorporating the
data into many areas of industry, research, educa-
tion, and entertainment.
Cross-References
▶ Anonymity
▶ Crowdsourcing
▶ Open Data
▶ Semantic Web
Further Reading
Jemielniak, D. (2014). Common knowledge: An ethnogra-
phy of wikipedia. Stanford: Stanford University Press.
Krötzscha, M., et al. (2007). Semantic Wikipedia. Web
Semantics: Science, Services and Agents on the World
Wide Web, 5(4), 251–261.
Leetaru, K. (2012). A bigdata approach to the humanities,
arts, and social sciences: Wikipedia’s view of the world
through supercomputing. Research Trends, 30, 17–30.
Stefaner, M., et al. Notability – Visualizing deletion dis-
cussions on Wikipedia. http://www.notabilia.net/
Viégas, F., et al. (2004). Studying cooperation and conflict
between authors with history flow visualizations. Paper
presented at CHI 2004, Vienna.
W
World Bank
Jennifer Ferreira
Centre for Business in Society, Coventry
University, Coventry, UK
The World Bank, part of the World Bank Group
established in 1944, is the international financial
institution responsible for promoting economic
development and reducing poverty. The World
Bank has two key objectives: to end extreme
poverty by reducing the proportion of the world’s
population living on less than $1.25 a day and
promoting shared prosperity by fostering income
growth in the lowest 40% of the population.
A core activity for the World Bank is the pro-
vision of low interest loans, zero- to low-interest
grants to developing countries. This could be to
support a wide range of activities from education
and health care to infrastructure, agriculture, or
natural resource management. In addition to the
financial support, the World Bank provides policy
advice, research, analysis, and technical assis-
tance to various countries in order to inform its
own investments and ultimately to work toward
its key objectives. Part of its activities relate to the
provision of tools to research and address devel-
opment challenges, some of which are in the form
of providing access to data, for example, the Open
Data website which includes a comprehensive
range of downloadable data sets related to differ-
ent issues. This shows its recognition of the
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_209-1
demand for access to quantitative data to inform
development strategies (Lehdonvirta and Ernkvist
2011).
A significant amount of the data hosted and
disseminated by the World Bank is drawn from
national statistical organizations, and it recognizes
that the quality of global data therefore is reliant
on the capacity and effectiveness of these national
statistical organizations. The World Bank has ten
key principles with respect to its statistical activ-
ities (in line with the Fundamental Principles of
Official Statistics and the Principles Governing
International Statistical Activities of the United
Nations Statistical Division): quality, innovation,
professional integrity, partnership, country own-
ership, client focus, results, fiscal responsibility,
openness, and good management.
The world is now experiencing unprecedented
capacity to generate, store, process, and interact
with data (McAfee and Brynjolfsson 2012), a
phenomenon that has been recognized by the
World Bank, like other international institutions.
For the World Bank, data is seen as critical for the
design, implementation, and evaluation of effi-
cient and effective development policy recom-
mendations. In 2014, Jim Yong Kim, the
President of the World Bank, discussed the impor-
tance of efforts to invest in infrastructure, includ-
ing data systems. Big data is recognized as a new
advancement which has the potential to enhance
efforts to address development, although it recog-
nizes there are a series of challenges associated
with this. In 2013, the World Bank hosted an event
2 World Bank
where over 150 experts, data scientists, civil soci-
ety groups, and development practitioners met to
analyze various forms of big data and consider
how it could be used to tackle development issues.
The event was a public acknowledgement of how
the World Bank viewed the importance of
expanding the awareness of how big data can
help combine various data sets to generate knowl-
edge which can in turn foster development
solutions.
A report produced in conjunction with the
World Bank, Big Data in Action for Development,
highlights some of the potential ways in which big
data can be used to work toward development
objectives and some of the challenges associated
with doing so. The report sets out a conceptual
framework for using big data in the development
sector highlighting the potential transformative
capacity of big data, particularly in relation to
raising awareness, developing understanding,
and contributing to forecasting.
Using big data to develop and enhance aware-
ness of different issues has been widely acknowl-
edged. Examples of this include: using
demographic data in Afghanistan to detect
impacts of small scale violence outbreaks, using
social media content to indicate unemployment
rises or crisis related stress, or using tweets to
recognize where cholera outbreaks were
appearing at a much faster rate than was recog-
nized in official statistics. This ability to gain
awareness of situations, experiences, and senti-
ments is seen to have the potential to reduce
reaction times and improve processes which deal
with such situations.
Big data can also be used to develop under-
standing of societal behaviors (LaValle et al.
2011). Examples include investigation of twitter
data to explore the relationship between food and
fuel price tweets and changes in official price
indexes in Indonesia; after the 2010 earthquake
in Haiti, mobile photo data was used to track
population displacement after the event, and sat-
ellite rainfall data was used in combination with
qualitative data sources to understand how rainfall
affects migration.
Big data is also seen to have potential for
contributing to modelling and forecasting.
Examples include: the use of GPS-equipped vehi-
cles in Stockholm, providing real-time traffic
assessments, which are used in conjunction with
other data sets such as weather which can then be
used to make traffic predictions, using mobile
phone data to predict mobility patterns.
The World Bank piloted some activities in
Central America to explore the potential of big
data to impact on development agendas. This
region has historically experienced low frequen-
cies of data collection for traditional data forms,
such as household surveys and so other forms of
data collection, were viewed as particularly
important. One of these pilot studies used google
trends data to explore the potential for the ability
to forecast price changes to commodities. Another
study, in conjunction with the UN Global Pulse,
explored the use of social media content to ana-
lyze public perceptions of policy reforms, in par-
ticular a gas subsidy reform in El Salvador,
highlighting the potential for this form of data to
complement other studies on public perception
(United Nations Global Pulse 2012).
The report from the World Bank, Big Data in
Action for Development, presents a matrix of dif-
ferent ways in which big data could be used in
transformational ways toward the development
agenda: using mobile data (e.g., reduced mobile
phone top ups as an indicator of financial stress),
financial data (e.g., increased understanding of
customer preferences), satellite data (e.g., to
crowd source information on damage after an
earthquake), internet data (e.g., to collect daily
prices), and social media data (e.g., to track par-
ents perception of vaccination). The example of
examining the relationship between food and fuel
prices and corresponding change in official price
index measures by using twitter data (by the UN
Global Pulse Lab) is outlined in detail explaining
how it was used to provide an indication of social/
economic conditions in Indonesia. This was done
by extracting tweets mentioning food and fuel
prices between 2011 and 2013 (around 100,000
relevant tweets after filtering for location and lan-
guage) and analyzing these with corresponding
changes from official data sets. The analysis indi-
cated a clear relationship between official food
inflation statistics and the number of tweets
World Bank
about food price increases. This study was cited as
an example of how big data could be used to
analyze public sentiment, in addition to objective
economic conditions. The examples mentioned
here are just some of the activities undertaken by
the World Bank to embrace the world of big data.
As with many other international institutions
which recognize the potential uses for big data,
the World Bank also recognizes there are a range
of challenges associated with the generation, anal-
ysis, and use of big data.
One of the most basic challenges for many
organizations (and individuals) is gaining access
to data, from both government institutions and the
private sector. A new ecosystem needs to be
developed where data is made openly available
and sharing incentives are in place. It is acknowl-
edged by the World Bank that international agen-
cies will need to address this challenge by not only
by promoting the availability of data but promot-
ing collaboration and mechanisms for sharing
data. In particular, a shift in business models will
be required in order to ensure the private sector is
willing to share data, and governments will need
to design policy mechanisms to ensure the value
of big data is captured and is shared across depart-
ments. Related to this, there need to be consider-
ations of how to engage the public with this data.
Thinking particularly about the development
agenda at the heart of the World Bank, there is a
paradox: countries where poverty is high or where
development agendas require the most attention
are often countries where data infrastructures or
technological systems are insufficient. Because
the generation of big data relies largely on tech-
nological capabilities, relying on those who use or
interact with digital sources may be systematically
unrepresentative of the larger population that
forms the focus of the research.
The ways in which data are recorded have
implications for the results which are interpreted.
Where data is passively recorded, there is less
potential for bias in the results generated, and
likewise where data is actively recorded, there is
greater potential for the results to be more
susceptive to selection bias. Furthermore, how
data is processed into a more structured from the
often very large and unstructured data sets
3
requires expertise to both clean the data and
where necessary aggregate it (e.g., if one set of
data collected every hour, and another every day).
Then the media through which data is collected is
also an important factor to consider. Mobile
phones, for example, producing highly sensitive
data, satellite images produce highly unstructured
data, and social media platforms produce a lot of
unstructured text which requires filtering and cod-
ifying which in itself requires specific analytic
capabilities.
Then in order to make effective use of big data,
those using it need to consider elements about the
data itself. The generation of big data has been
driven by advances in technology, yet these
advances are not alone sufficient to be able to
understand the results which can be gleaned
from big data. Transforming vast data sets into
meaningful results requires effective human capa-
bilities. Depending on how the data is generated,
and by whom, there is scope for bias and therefore
misleading conclusions. Then with large amounts
of data, there is a tendency for patterns to be
observed where there may be none; because of
its nature, big data can give rise to significant
statistical correlations. It is important to remember
that correlation does not imply causation. Then
just because there is large amount of data avail-
able, this does not necessarily mean this is the
right data for the question or issue being
investigated.
The World Bank acknowledges that for big
data to be made effective for development, there
will need to be collaboration between practi-
tioners, social scientists, and data scientists in
order to ensure the understanding of the real-
world conditions and data generation mecha-
nisms, and methods of interpretation are effec-
tively combined. Beyond this there will need to
be cooperation between public and private sector
bodies in order to foster greater data sharing and
incentivize the use of big data across different
sectors. Even when data has been accessed, in
nearly all occasions it needs to be filtered and
made suitable for analysis. Filters require human
input and need to be applied carefully as their use
may preclude information and affect the results.
Data needs to be cleaned. Mobile data is received
4 World Bank
in unstructured form of millions of files, which
requiring time-intensive processing to obtain data
suitable for analysis. Likewise, analysis of text
from social media requires a decision making
process to filter out suitable search terms.
Finally, there are a series of concerns about
how privacy is ensured with big data, given that
often there are elements of big data which can be
sensitive in nature (either to the individual or
commercially). This is made more complicated
as each country will have different regulations
about data privacy which poses particular chal-
lenges for institutions working across national
boundaries, like the World Bank.
For the World Bank, the use of big data is seen
to have potential for improving and changing the
international development sector. Underpinning
the ideas of the World Bank’s approach to big
data is the recognition that while the technological
capacities for generation, storage, and processing
of data continue to develop, this also needs to be
accompanied by institutional capabilities to
enable big data analysis to contribute to effective
actions that can contribute to development,
whether this is through strengthening of warning
systems, raising awareness, or developing under-
standing of social systems or behaviors.
The World Bank has begun to consider an
underlying conceptual framework around the use
of big data, in particular considering the chal-
lenges it presents in terms of using big data for
development. In the report Big Data in Action for
Development, it is acknowledged that there is
great potential for big data to provide a valuable
input for designing effective development policy
recommendation but also that big data is no pan-
acea (Coppola et al. 2014). The World Bank has
made clear efforts to engage with the use of big
data and has begun to explore areas of clear poten-
tial for big data use. However, questions remain
about how it can support countries to take owner-
ship and create, manage, and maintain their own
data, contributing to their own development
agendas in effective ways.
Cross-References
▶ Bank of America
▶ Citigroup Inc
▶ International Development
▶ United Nations
▶ United Nations Global Pulse
▶ World Health Organization
Further Reading
Coppola, A., Calvo-Gonzalez, O., Sabet, E., Arjomand, N.,
Siegel, R., Freeman, C., Massarat, N. (2014). Big data
in action for development. Washington, DC: World
Bank and Second Muse. Available at: http://live.
worldbank.org/sites/default/files/Big%20Data%20for
%20Development%20Report_final%20version.pdf.
LaValle, S., Lesser, E., Shockley, R., Hopkins, M., &
Kruschwitz, N. (2011). Big data, analytics and the
path from insights to value. MIT Sloan Management
Review, 52(2), 21–31.
Lehdonvirta, V., & Ernkvist, M. (2011). Converting the
virtual economy into development potential: Knowl-
edge map of the virtual economy. InfoDev/World Bank
White Paper, 1, 5–17.
McAfee, A., & Brynjolfsson, E. (2012). Big data: The
management revolution. Harvard Business Review,
90(10), 60–66.
United Nations Global Pulse. (2012). Big data for devel-
opment: Challenges & opportunities. New York: UN,
New York.
Z
Zappos
Jennifer J. Summary-Smith
Culver-Stockton College, Canton, MO, USA
As one of the largest online retailers of shoes,
Zappos (derived from the Spanish word zapatos
meaning shoes) is a company that is setting an
innovative trend in customer service and manage-
ment style. According to Zappos’ website, one of
its primary goals is to provide the best online
service. The company envisions a world where
online customers will make 30% of all retail trans-
actions in the United States. Zappos hopes to be
the company that leads the market in online sales,
setting itself aside from other online retail com-
petitors by offering the best customer service and
selection.
History of the Company
Zappos was founded in 1999 by Nick Swinmurn
who developed the idea for the company while
walking around a mall in San Francisco, Califor-
nia, looking for a pair of shoes. After spending an
hour in the mall searching from store to store for
the right color and shoe size, he left the mall
empty handed and frustrated. Upon arriving
home, Swinmurn turned to the Internet to con-
tinue his search for his preferred shoes, which
again was unsuccessful. Swinmurn realized that
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_210-1
there were no major online retailers specializing in
shoes. It was at this point that Swinmurn decided
to quit his full-time job and start an online shoe
retailer named Zappos. Overtime the company has
evolved, focusing on making the speed of its
customers’ online purchase central to its business
model. In order to achieve this, Zappos ware-
houses have everything it sells. As the company
grew, it reached new heights in 2009 when Zappos
and Amazon joined forces combining their pas-
sion for strong customer service. Since then,
Zappos has grown significantly and restructured
into ten separate companies.
Security Breach
Unfortunately, Zappos has not been without a few
missteps. In 2012, the company experienced a
security breach, compromising as many as 24 mil-
lion customers. Ellen Messmer reports that cyber-
hackers successfully gained access to the
company’s internal network and systems. To
address this security breach, Zappos CEO Tony
Hsieh announced that existing customer pass-
words would be terminated as a result of the
breach. Still yet, the cyberhackers likely gained
accessed to names, phone numbers, the last four
digits of credit card numbers, cryptographically
scrambled passwords, email, billing information,
and shipping addresses. After Zappos CEO Tony
Hsieh posted an open letter explaining the breach
and how the company would head off resulting
2 Zappos
problems, there were mixed responses to how the
company had handled the situation. As part of its
response to the breach, the company sent out
emails informing its customers of the problem
urging them to change their passwords. Zappos
also provided an 800-number phone service to its
customers helping them through the process of
choosing a new password.
However, some experts familiar with the
online industry have criticized the moves by
Zappos. In an article by Ellen Messmer, she
interviewed an Assistant Professor of Information
Technology from the University of Notre Dame,
who argued that the response strategy by Zappos
was not appropriate. Professor John D’Arcy posits
that the company’s decision to terminate cus-
tomers’ passwords promotes a panic mode, creat-
ing a sense of panic in its customers. In contrast,
other analysts claim that Zappos public response
to the situation was the right move, communicat-
ing to its customers publicly.
Nevertheless, Zappos is doing a good job of
getting the information out about the security
breach to the public as soon as possible, according
to Professor John D’Arcy. This typically benefits
the customers, creating favorable reactions. In
terms of the cost of the security breaches, the
Ponemon Institute estimates that on average, a
data breach costs $277 per compromised record.
Lawsuits
After the security breach, dozens of lawsuits were
filed. Zappos attempted to send the lawsuits to
arbitration, citing its user agreement. In the fall
of 2012, a federal court struck down Zappos.
com’s user agreement, according to Eric
Goldman. Eric Goldman is a professor of law at
Santa Clara University School of Law who writes
about Internet law, intellectual property, and
advertising law. He states that Zappos made mis-
takes that are easily avoidable. The courts typi-
cally divide user agreements into one of three
groups: “clickwraps” or “click-through agree-
ments,” “browsewraps,” and “clearly not a con-
tract.” Eric Goldman argues that the click-through
agreements are effective in courts unlike
browsewraps. Browsewraps are user agreements
that bind users simply for browsing the website.
The courts ruled that Zappos presented its user
agreement as a browsewrap. Furthermore, Zappos
claimed on its website that the company reserved
the right to amend the contract whenever it saw fit.
Despite other companies using this language
online, it is detrimental to a contract. The courts
ruled that Zappos can amend the terms of the user
agreement at any time, making the arbitration
clause susceptible to change as well. This makes
the clause unenforceable. Eric Goldman posits
that the court ruling left Zappos in a bad position
because all of the risk management provisions are
ineffective. In other words, losing the contract left
Zappos without the following: its waiver of con-
sequential damages, its disclaimer of warranties,
its clause restricting class actions in arbitration,
and its reduced statute of limitations. Conversely,
companies that use click-through agreements and
remove clauses that state they can amend the
contract unilaterally are in a better legal position,
according to Eric Goldman.
Holacracy
Zappos CEO Tony Hsieh announced in
November 2013 that his company would be
implementing the management style known as
Holacracy. With Holacracy, there are two key
elements that Zappos will follow: distributed
authority and self-organization. According to an
article by Nicole Leinbach-Reyhle, distribution
authority allows employees to evolve the organi-
zation’s structure by responding to real-word cir-
cumstances. In regard to self-organization,
employees have the authority to engage in useful
action to express their purpose as long as it does
not “violate of the domain of another role.” There
is a common misunderstanding that Holacracy is
nonhierarchical when in fact it is strongly hierar-
chical, distributing power within the organization.
This approach to management creates an atmo-
sphere where employees can speak up evolving
into leaders rather than followers. Zappos CEO
Tony Hsieh states that he is trying to structure
Zappos less like a bureaucratic corporation and
Zappos
more like a city, resulting in increased productiv-
ity and innovation. To date, with 1,500
employees, Zappos is the largest company to
adopt the management model – Holacracy.
Innovation
The work environment at Zappos has become
known for its unique corporate culture, which
incorporates fun and humor into daily work. As
stated on Zappos.com, the company has a total of
ten core values: “deliver WOW through service,
embrace and drive change, create fun and a little
weirdness, be adventurous, creative, and open-
minded, pursue growth and learning, build open
and honest relationships with communication,
build a positive team and family spirit, do more
with less, be passionate and determined, and be
humble.” Nicole Leinbach-Reyhle writes that
Zappos’ values help to encourage its employees
to think outside of the box.
To date, Zappos is a billion-dollar online
retailer, expanding beyond selling shoes. The
company is also making waves in its corporate
culture and hierarchy. Additionally, information
technology plays a huge role in the corporation,
serving its customers and the business. Based
upon the growing success of Zappos, it is keeping
true to its mission statement “to provide the best
customer service possible.” It evident that Zappos
will continue to make positive changes for the
corporation and its corporate headquarters in Las
Vegas. In 2013, Zappos CEO Tony Hsieh com-
mitted $350 million to rebuild and renovate the
3
downtown Las Vegas region. As Sara Corbett
notes in her article, he hopes to change the area
into a start-up fantasyland.
Cross-References
▶ Bureau of Consumer Protection: Data Breach
▶ Legal Issues
▶ Small Business Enterprises
Further Reading
Corbett, S. (n.d.). How Zappos’ CEO turned Las Vegas into
a startup fantasyland. http://www.wired.com/2014/01/
zappos-tony-hsieh-las-vegas/
Goldman, E. (n.d.). How Zappos’ user agreement Failed in
court and left Zappos legally naked. http://www.forbes.
com/sites/ericgoldman/2012/10/10/how-zappos-user-
agreement-failed-in-court-and-left-zappos-legally-
naked/. Accessed Jul 2014
Leinbach-Reyhle, N. (n.d.). Shedding hierarchy: Could
Zappos be setting an innovative trend? http://www.
forbes.com/sites/nicoleleinbachreyhle/2014/07/15/
shedding-hierarchy-could-zappos-be-setting-an-
innvoative-trend/. Accessed Jul 2014
Messmer, E. (n.d.). Zappos data breach response a good
idea or just panic mode? Online shoe and clothing
retailer Zappos has taken assertive steps after breach,
but is it enough? http://www.networkworld.com/arti
cle/2184860/malware-cybercrime/zappos-data-breach-
response-a-good-idea-or-just-panic-mode-.html.
Accessed Jul 2014
Ponemon Group. (n.d.). 2013 cost of data breach study:
Global analysis. http://www.ponemon.org. Accessed
Jul 2014
Zappos. (n.d.). http://www.zappos.com. Accessed Jul 2014
Z
Zillow
Matthew Pittman and Kim Sheehan
School of Journalism & Communication,
University of Oregon, Eugene, OR, USA
Overview and Business Model
Like most industries, real estate is undergoing
dynamic shifts in the age of big data.
Real estate information, once in the hands of a
few agents or title companies, is being democra-
tized for any and all interested consumers. What
were previously physical necessities – real estate
agents, showings, and physical homes – are being
obsolesced by digital platforms like Zillow. Real
estate developers can use technology to track how
communities flow and interact with one another,
which will help build smarter, more efficient
neighborhoods in the future. The companies that
succeed in the future will be the ones who, like
Zillow, find innovative, practical, and valuable
ways to navigate and harness the massive amounts
of data that are being produced in and around their
field.
Founded in Seattle in 2005, Zillow is a billion-
dollar real estate database that uses big data to
help consumers learn about home prices, rent
rates, market trends, and more. They provide esti-
mates for most housing units in the United States.
It acquired its closest competitor, Trulia, in 2014
for $3.5 billion. It is the most-viewed real estate
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_211-1
destination in the country. Now with Trulia, it
accounts for 48% of Web traffic for real estate
listings, though that number is diminished to
around 15% if you factor in individual realtor
sites and local MLS (multiple listing service) list-
ings. The company’s chief economist Stan
Humphries created a tool that processes 1.2 mil-
lion proprietary statistical models three times per
week on the county and state real estate data it is
constantly gathering. In 2011, they shifted from
an in-house computer cluster to renting space in
the Amazon cloud to help with the massive
computing load.
On the consumer side, Zillow is a web site or
mobile app that is free to use. Users can enter a
city or zip code and search, filtering out home
types, sizes, or prices that are undesirable. There
are options to see current homes for sale, recently
sold properties, foreclosures, rental properties,
and even Zillow “zestimates” (the company’s sig-
nature feature) of the home’s current value based
on similar homes in the area, square footage,
amenities, and more. Upon clicking on a house
of interest, the user can see a real estate agent’s
description of the home, how long it has been on
the market – along with any price fluctuations – as
well as photos, similarly priced nearby houses,
proposed mortgage rates on the home, the agents
associated with it, the home’s sale history, and
facts and features.
Zillow makes money on real estate firms and
agents that advertise through the site and by pro-
viding subscriptions to real estate professionals.
2 Zillow
They can charge more for ads that appear during a
search for homes in Beverly Hills than in Bis-
marck, South Dakota. Some 57,000 agents spend
an average of $4,000 every year for leads to get
new buyers and sellers. Zillow keeps a record of
how many times a listing has been viewed, which
may help negotiate the price with among agents,
buyers, and sellers. Real estate agents can sub-
scribe to silver, gold, or platinum programs to get
CRM (customer relationship management) tools,
their photo in listings, a web site, and more. Basic
plans start at 10 dollars a month.
Zillow’s mortgage marketplace also earns
them revenue. Potential homebuyers can find
and engage with mortgage brokers and firms.
The mortgage marketplace tells potential buyers
what their monthly payment would be, how much
they can afford, submit loan requests, and get
quotes from various lenders. In the third quarter
of 2013, Zillow’s mortgage marketplace received
5.9 million loan requests from borrowers (more
than all of 2011), which grew its revenue stream
120% to $5.7 million. A majority of Zillow’s
revenue comes from the real estate segment that
lets users browse homes for sale and for rent. This
earned them over $35 million in 2013’s third
quarter.
Analysts and shareholders have voiced some
concerns over Zillow’s business model. Zillow
now spends over 70% of its revenues on sales
and marketing, as opposed to 33% for LinkedIn
and between 21% and 23% for IBM and Micro-
soft. Spending money on television commercials
and online ads for its services seems to have
diminishing returns for Zillow, who is spending
more and more on marketing for the same net
profit. What once seemed like a sure-fire
endeavor – making money by connecting cus-
tomers to agents through relevant and concise
management of huge amounts of data – is no
longer a sure thing. Zillow will have to continu-
ally evolve its business model if it is to stay afloat.
Zillow and the Real Estate Industry
Zillow has transformed the real estate industry by
finding new and practical ways to make huge
amounts of data accessible to common people.
Potential buyers no longer need to contact a real
estate agent before searching for homes – they can
start a detailed search on just about any house in
the country from their own mobile or desktop
device. This is empowering for consumers, but it
shakes up an industry that has long relied on
human agents. These agents made it their business
to know specific areas, learn the ins and outs of a
given community, and then help connect inter-
ested buyers to the right home. Sites that give
users a tool peer into huge amounts of data (like
Zillow) are useful to a point, but some critics feel
only a human being who is local and present in a
community can really serve potential buyers.
Because it takes an aggregate of multiple
national and MLS listing sites, Zillow is rarely
perfect. Any big data computing service that
works with offline or subjective entities – and
real estate prices certainly fit this description –
will have to make logical (some would say illog-
ical) leaps where information is scarce. When
Zillow does not have exact or current data on a
house or neighborhood, it “guesses” when prices
come in too high, sellers have unrealistic expec-
tations of the potential price of their home.
Buyers, too, may end up paying for a home than
it is actually worth.
A human expert (real estate agent) has tradi-
tionally been the expert in this area, yet people are
still surprised when too much stock is put into an
algorithm. Zillow zestimates tend to work best for
midrange homes in an area where there are plenty
of comparable houses. Zestimates are less accu-
rate for low- and high-end homes because there
are fewer comps (comparable houses for sale or
recently sold). Similarly, zestimates of rural,
unique, or fixer-upper homes are difficult to
gauge. Local MLS sites may have more detail on
a specific area, but Zillow has broader, more gen-
eral information over a larger area. They estimate
their coverage of American homes to be around
57%.
Real estate data is more difficult to come by in
some areas. Texas doesn’t provide public records
of housing transaction prices, so Zillow had to
access sales data from property databases through
real estate brokers. Because of the high number of
Zillow
cooperative buildings, New York City is another
difficult area in which to gauge real estate prices.
Tax assessments are made on the co-ops, not the
individual units, which negates that factor in
zestimate calculations. Additional information,
like square footage or amenities, is also difficult
to come by, forcing Zillow to seek out alternative
sources.
Of course, zestimates can be accurate as well.
As previously noted, when the house is midrange
and in a neighborhood with plenty of comps (and
thus plenty of data), zestimates can be very good
indicators of the home’s actual worth. As Zillow
zestimates – and sources from which to draw
factoring information – continue to evolve, the
service may continue growing in popularity. The
more popular Zillow becomes, the more incentive
real estate agents will have to list all of their
housing database information with the service.
Agents know that, in a digital society, speed is
key: 74% of buyers and 76% of sellers will work
with the first agent with whom they talk.
Recently Zillow is recognizing a big shift to
mobile: about 70% of Zillow’s usage now occurs
on mobile platforms. This trend is concurrent with
other platforms’ shift to mobile usage; Facebook,
Instagram, Zynga, and others have begun to rec-
ognize and monetize users’ access from
smartphones and tablets. For real estate, this
mobile activity is about more than just conve-
nience: user can find information on homes in
real time as they drive around a neighborhood,
looking directly at the potential homes, and con-
tact the relevant agent before they get home. This
3
sort of activity bridges the traditional brick-and-
mortar house hunting of the past with the instant
big data access of the future (and increasingly, the
present). Zillow has emerged as a leader in its field
of real estate by connecting its customers, not just
to big data but the right data at the right time and
places.
Cross-References
▶ Data-Driven Marketing
▶ Digitization
▶ E-Commerce
▶ Real Estate/Housing
▶ Utilities Industry
Further Readings
Arribas-Bel, D. (2014). Accidental, open and everywhere:
Emerging data sources for the understanding of cities.
Applied Geography, 49, 45–53.
Cranshaw, J., Schwartz, R., Hong, J.I., Sadeh,
N.M. (2012). The livelihoods project: Utilizing social
media to understand the dynamics of a city. In ICWSM.
Hagerty, J. R.(2007). How good are Zillow’s estimates?
Wall Street Journal.
Huang, H., & Tang, Y. (2012). Residential land use regu-
lation and the US housing price cycle between 2000
and 2009. Journal of Urban Economics, 71(1), 93–99.
Wheatley, M. (n.d.). Zillow-Trulia merger will create bound-
less new big data opportunities. http://siliconangle.com/
blog/2014/07/31/zillow-trulia-merger-will-create-bound
less-new-big-data-opportunities/. Accessed on Sept
2014.
A
AgInformatics
Andrea De Montis1, Giuseppe Modica2 and
Claudia Arcidiacono3
1
Dipartimento di Agraria, University of Sassari,
Sassari, Sardinia, Italy
2
Dipartimento di Agraria, Università degli Studi
Mediterranea di Reggio Calabria, Reggio
Calabria, Italy
3
Dipartimento di Agricoltura, Alimentazione e
Ambiente, University of Catania, Catania, Italy
Synonyms
E-agriculture; Precision agriculture; Precision
farming
Definition
The term stems from the blending of the two
words agriculture and informatics and refers to
the application of informatics to the analysis,
design and development of agricultural activities.
It overarches expressions such as Precision Agri-
culture (PA), Precision Livestock Farming (PLF),
and Agricultural landscape analysis and planning.
The adoption of AgInformatics can accelerate
agricultural development by providing farmers
and decision makers with more accessible, com-
plete, timely, and accurate information. However,
it is still hindered by a number of important yet
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_218-1
unresolved issues including big data handling,
multiple data sources and limited standardization,
data protection, and lack of optimization models.
Development of knowledge-based systems in the
farming sector would require key components,
supported by Internet of things (IoT), data acqui-
sition systems, ubiquitous computing and net-
working, machine-to-machine (M2M)
communications, effective management of
geospatial and temporal data, and ICT-supported
cooperation among stakeholders.
Generalities
This relatively new expression derives from a
combination of the two terms agriculture and
informatics, hence alluding to the application of
informatics to the analysis, design, and develop-
ment of agricultural activities. It broadly involves
the study and practice of creating, collecting, stor-
ing and retrieving, manipulating, classifying, and
sharing information concerning both natural and
engineered agricultural systems. The domains of
application are mainly agri-food and environmen-
tal sciences and technologies, while sectors
include biosystems engineering, farm manage-
ment, crop production, and environmental moni-
toring. In this respect, it encompasses the
management of the information coming from
applications and advances of information and
communication technologies (ICTs) in agriculture
(e.g., global navigation satellite system, GNSS;
2 AgInformatics
remote sensing, RS; wireless sensor networks,
WSN; and radio-frequency identification, RFID)
and performed through specific agriculture infor-
mation systems, models, and methodologies (e.g.,
farm management information systems, FMIS;
GIScience analyses; Data Mining; decision sup-
port systems, DSS).
AgInformatics is an umbrella concept that
includes and overlaps issues covered in precision
agriculture (PA), precision livestock farming
(PLF), and agricultural landscape analysis and
planning, as follows.
Precision Agriculture (PA)
PA was coined in 1929 and later defined as “a
management strategy that uses information tech-
nologies to bring data from multiple sources to
bear on decisions associated with crop produc-
tion” (Li and Chung 2015). The concept evolved
since the late 1980s due to new fertilization equip-
ment, dynamic sensing, crop yield monitoring
technologies, and GNSS technology for auto-
mated machinery guidance.
Therefore, PA technology has provided
farmers with the tools (e.g., built-in sensors in
farming machinery, GIS tools for yield monitor-
ing and mapping, WSNs, satellite and low-alti-
tude RS by means of unmanned aerial systems
(UAS), and recently robots) and information (e.g.,
weather, environment, soil, crop, and production
data) needed to optimize and customize the
timing, amount, and placement of inputs includ-
ing seeds, fertilizers, pesticides, and irrigation,
activities that were later applied also inside closed
environments, buildings, and facilities, such as for
protected cultivation.
To accomplish the operational functions of a
complex farm, FMISs for PA are designed to
manage information about processes, resources
(materials, information, and services), procedures
and standards, and characteristics of the final
products (Sørensen et al. 2010). Nowadays dedi-
cated FMISs operate on networked online frame-
works and are able to process a huge amount of
data. The execution of their functions implies the
adoption of various management systems, data-
bases, software architectures, and decision
models. Relevant examples of information
management between different actors are supply
chain information systems (SCIS) including those
specifically designed for traceability and supply
chain planning.
Recently, PA has evolved to predictive and
prescriptive agriculture. Predictive agriculture
regards the activity of combining and using a
large amount of data to improve knowledge and
predict trends, whereas prescriptive agriculture
involves the use of detailed, site-specific recom-
mendations for a farm field. Today PA embraces
new terms such as precision citrus farming, preci-
sion horticulture, precision viticulture, precision
livestock farming, and precision aquaculture (Li
and Chung 2015).
Precision Livestock Farming (PLF)
The increase in activities related to livestock farm-
ing triggered the definition of the new term preci-
sion livestock farming (PLF), namely, the real-
time monitoring technologies aimed at managing
the smallest manageable production unit’s tempo-
ral variability, known as “the per animal
approach” (Berckmans 2004). PLF consists in
the real-time gathering of data related to livestock
animals and their close environment, applying
knowledge-based computer models, and extra-
cting useful information for automatic monitoring
and control purposes. It implies monitoring ani-
mal health, welfare, behavior, and performance
and the early detection of illness or a specific
physiological status and unfolds in several activ-
ities including real-time analysis of sounds,
images, and accelerometer data, live weight
assessment, condition scoring, and online milk
analysis. In PLF, continuous measurements and
a reliable prediction of variation in animal data or
animal response to environmental changes are
integrated in the definition of models and algo-
rithms that allow for taking control actions (e.g.,
climate control, feeding strategies, and therapeu-
tic decisions).
Agricultural Landscape Analysis and Planning
Agricultural landscape analysis and planning is
increasingly based on the development of inter-
operable spatial data infrastructures (SDIs) that
AgInformatics
integrate heterogeneous multi-temporal spatial
datasets and time-series information.
Nearly all agricultural data has some form of
spatial component, and GISs allow to visualize
information that might otherwise be difficult to
interpret (Pierce and Clay 2007).
Land use/land cover (LU/LC) change detection
methods are widespread in several research fields
and represent an important issue dealing with the
modification analysis of agricultural uses. In this
framework, RS imagery plays a key role and
involves several steps dealing with the classifica-
tion of continuous radiometric information
remotely surveyed into tangible information,
often exposed as thematic maps in GIS environ-
ments, and that can be utilized in conjunction with
other data sets. Among classification techniques,
object-based image analysis (OBIA) is one of the
most powerful techniques and gained popularity
since the early 2000s in extracting meaningful
objects from high-resolution RS imagery.
Proprietary data sources are integrated with
social data created by citizens, i.e., volunteered
geographic information (VGI). VGI includes
crowdsourced geotagged information from social
networks (often provided by means of smart
applications) and geospatial information on the
Web (GeoWeb). Spatial decision support systems
(SDSSs) are computer-based systems that help
decision makers in the solution of complex prob-
lems, such as in agriculture, land use allocation,
and management. SDSSs implement diverse
forms of multi-criteria decision analysis
(MCDA). GIS-based MCDA can be considered
as a class of SDSS. Implementing GIS-MCDA
within the World Wide Web environment can
help to bridge the gap between the public and
experts and favor public participation.
Conclusion
Technologies have the potential to change modes
of producing agri-food and livestock. ICTs can
accelerate agricultural development by providing
more accessible, complete, timely, or accurate
3
information at the appropriate moment to decision
makers. Concurrently, management concepts,
such as PA and PLF, may play an important role
in driving and accelerating adoption of ICT tech-
nologies. However, the application of PA solu-
tions has been slow due to a number of
important yet unresolved issues including big
data handling, limited standardization, data pro-
tection, and lack of optimization models and
depends as well on infrastructural conditions
such as availability of broadband internet in rural
areas. The adoption of FMISs in agriculture is
hindered by barriers connected to poor interfac-
ing, interoperability and standardized formats,
and dissimilar technological equipment adoption.
Development of knowledge-based systems in the
farming sector would require key components,
supported by IoT, data acquisition systems, ubiq-
uitous computing and networking, M2M commu-
nications, effective management of geospatial and
temporal data, traceability systems along the sup-
ply chain, and ICT-supported cooperation among
stakeholders. Recent designs and prototypes
using cloud computing and the future Internet
generic enablers for inclusion in FMIS have
recently been proposed and lay the groundwork
for future applications. A modification, which is
underway, from proprietary tools to Internet-
based open systems supported by cloud hosting
services will enable a more effective cooperation
between actors of the supply chain. One of the
limiting factors in the adoption of SCIS is a lack of
interoperability, which would require implemen-
tation of virtual supply chains based on the
virtualization of physical objects such as con-
tainers, products, and trucks. Recent and promis-
ing developments of the spatial decision-making
deal with the interaction and the proactive
involvement of the final users, implementing the
so-called collaborative or participative Web-based
GIS-MCDA systems. Computers science and IT
evolvements affect the developments of RS in
agriculture, leading to the need for new methods
and solutions to the challenges of big data in a
cloud computing environment.
4 AgInformatics

Cross-References Further Readings

▶ Agriculture, Forestry, Fishery, Hunting Berckmans, D. (2004). Automatic on-line monitoring of

animals by precision livestock farming. In Proceedings
▶ Cloud
of the ISAH conference on animal production in
▶ Data Processing Europe: The Way Forward in a Changing World.
▶ Information Technology Saint-Malo, pp. 27–31.
▶ Radio-Frequency Identification (RFID) Li, M., & Chung, S. (2015). Special issue on precision
agriculture. Computers and Electronics in Agriculture,
▶ Satellite Imagery/Remote Sensing
112, 1.
▶ Semantic Web Pierce, F. J., & Clay, D. (Eds.). (2007). GIS applications in
▶ Sensor Technologies agriculture. Boca Raton: CRC Press Taylor and Francis
▶ Spatial Analytics Group.
Sørensen, C. G., Fountas, S., Nash, E., Pesonen, L.,
▶ Spatial Data
Bochtis, D., Pedersen, S. M., Basso, B., & Blackmore,
▶ Volunteered Geographic Information (VGI) S. B. (2010). Conceptual model of a future farm man-
agement information system. Computers and Electron-
ics in Agriculture, 72(1), 37–47.
B
Big Data Quality
Subash Thota
Synectics for Management Decisions, Inc.,
Arlington, VA, USA
Introduction
Data is the most valuable asset for any organiza-
tion. Yet in today’s world of big and unstructured
data, more information is generated than can be
collected and properly analyzed. The onslaught of
data presents obstacles to create data-driven deci-
sions. Data quality is an essential characteristic of
data that determines the reliability of data for
making decisions in any organization or business.
Errors in data can cost a company millions of
dollars, alienate customers, and make
implementing new strategies difficult or impossi-
ble (Redman 1995).
In practically every business instance, project
failures and cost overruns are due to fundamental
misunderstanding about the data quality that is
essential to the initiative. A global data manage-
ment survey by PricewaterhouseCoopers of 600
companies across the USA, Australia, and Britain
showed that 75% of reported significant problems
were a result of data quality issues, with 33% of
those saying the problems resulted in delays in
getting new business intelligence (BI) systems
running or in having to scrap them altogether
(Capehart and Capehart 2005). The importance
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_240-1
and complexity related to data and its quality
compounds incrementally and could potentially
challenge the very growth of the business that
acquired the data. This paper is intended to show-
case challenges related to data quality and
approaches to mitigating data quality issues.
Data Defined
Data is “ . . . language, mathematical or other sym-
bolic surrogates which are generally agreed upon to
represent people, objects, events and concepts”
(Liebenau and Backhouse 1990). Vayghan et al.
(2007) argued that most enterprises deal with three
types of data: master data, transactional data, and
historical data. Master data are the core data enti-
ties of the enterprise, i.e., customers, products,
employees, vendors, suppliers, etc. Transactional
data describe an event or transaction in an organi-
zation, such as sales orders, invoices, payments,
claims, deliveries, and storage records. Transac-
tional data is time bound and changes to historical
data once the transaction has ended. Historical
data contain facts, as of certain point in time (e.g.,
database snapshots), and version information.
Data Quality
Data quality is the capability of data to fulfill and
satisfy the stated business, framework, system and
technical requirements of an enterprise. A classic
2 Big Data Quality

definition of data quality is “fitness for use,” or more value is derived from analyzing connec-
more specifically, the extent to which some data tivity and relationships, the inability to link
successfully serve the purposes of the user (Tayi related data instance together impedes this
and Ballou 1998; Cappiello et al. 2003; Lederman valuable analysis.
et al. 2003; Watts et al. 2009).
To be able to correlate data quality issues to
business impacts, we must be able to both classify
Causes and Consequences
our data quality expectations as well as our busi-
ness impact criteria. In order to do that, it is
The “Big Data” era comes with new challenges
valuable to understand these common data quality
for data quality management. Beyond volume,
dimensions (Loshin 2006):
velocity, and variety lies the importance of the
fourth “V” of big data: veracity. Veracity refers
– Completeness: Is all the requisite information
to the trustworthiness of the data. Due to the sheer
available? Are data values missing, or in an
volume and velocity of some data, one needs to
unusable state? In some cases, missing data is
embrace the reality that when data is extracted
irrelevant, but when the information that is
from multiple datasets at a fast and furious clip,
missing is critical to a specific business pro-
determining the semantics of the data – and under-
cess, completeness becomes an issue.
standing correlations between attributes –
– Conformity: Are there expectations that data
becomes of critical importance.
values conform to specified formats? If so, do
Companies that manage their data effectively
all the values conform to those formats?
are able to achieve a competitive advantage in the
Maintaining conformance to specific formats
marketplace (Sellar 1999). On the other hand, bad
is important in data representation, presenta-
data can put a company at a competitive disad-
tion, aggregate reporting, search, and
vantage comments (Greengard 1998). It is there-
establishing key relationships.
fore important to understand some of the causes of
– Consistency: Do distinct data instances pro-
bad data quality:
vide conflicting information about the same
underlying data object? Are values consistent
• Lack of data governance standards or valida-
across data sets? Do interdependent attributes
tion checks.
always appropriately reflect their expected
• Data conversion usually involves transfer of
consistency? Inconsistency between data
data from an existing data source to a new
values plagues organizations attempting to rec-
database.
oncile different systems and applications.
• Increasing complexity of data integration and
– Accuracy: Do data objects accurately repre-
enterprise architecture.
sent the “real-world” values they are expected
• Unreliable and inaccurate sources of
to model? Incorrect spellings of products, per-
information.
sonal names or addresses, and even untimely
• Mergers and acquisitions between companies.
or not current data can impact operational and
• Manual data entry errors.
analytical applications.
• Upgrades of infrastructure systems.
– Duplication: Are there multiple, unnecessary
• Multidivisional or line-of-business usage of data.
representations of the same data objects within
• Misuse of data for purposes different from the
your data set? The inability to maintain a single
capture reason.
representation for each entity across your sys-
tems poses numerous vulnerabilities and risks.
Different people performing the same tasks
– Integrity: What data is missing important rela-
tionship linkages? The inability to link related have a different understanding of the data being
processed, which leads to inconsistent data mak-
records together may actually introduce dupli-
ing its way into the source systems. Poor data
cation across your systems. Not only that, as
Big Data Quality
quality is a primary reason for 40% of all business
initiatives failing to achieve their targeted benefits
(Friedman and Smith 2011). Marsh (2005) sum-
marizes consequences in one of his article:
• Eighty-eight percent of all data integration pro-
jects either fail completely or significantly
overrun their budgets.
• Seventy-five percent of organizations have
identified costs stemming from dirty data.
• Thirty-three percent of organizations have
delayed or canceled new IT systems because
of poor data.
• $611B per year is lost in the USA to poorly
targeted bulk mailings and staff overheads.
• According to Gartner, bad data is the number
one cause of customer-relationship manage-
ment (CRM) system failure.
• Less than 50% of companies claim to be very
confident in the quality of their data.
• Business intelligence (BI) projects often fail due
to dirty data, so it is imperative that BI-based
business decisions are based on clean data.
• Only 15% of companies are very confident in
the quality of external data supplied to them.
• Customer data typically degenerates at 2% per
month or 25% annually.
To Marsh, organizations typically overestimate
the quality of their data and underestimate the cost
of data errors. Business processes, customer
expectations, source systems and compliance
rules are constantly changing – and data quality
management systems must reflect this. Vast
amounts of time and money are spent on custom
coding and “firefighting” to dampen an immediate
crisis rather than dealing with the long-term prob-
lems that bad data can present to an organization.
Data Quality: Approaches
Due to the large variety of sources from which
data is collected and integrated, for its sheer vol-
ume and changing nature, it is impossible to man-
ually specify data quality rules. Below are a few
approaches to mitigating data quality issues:
3
1. Enterprise Focus and Discipline
Enterprises should be more focused and
engaged toward data quality issues; views toward
data cleansing must evolve. Clearly defining roles
and outlining the authority, accountability and
responsibility for decisions regarding enterprise
data assets provides the necessary framework for
resolving conflicts and driving a business forward
as the data-driven organization matures. Data
quality programs are most efficient and effective
when they are implemented in a structured,
governed environment.
2. Implementing MDM and SOA
The goal of a master data management (MDM)
solution is to provide a single source of truth of
data, thus providing a reliable foundation for that
data across the organization. This prevents busi-
ness users across an organization from using dif-
ferent versions of the same data. Another
approach of big data and big data governance is
the deployment of cloud-based models and soft-
ware-oriented architecture (SOA). SOA enables
the tasks associated with a data quality program to
be deployed as a set of services that can be called
dynamically by applications. This allows business
rules for data quality enforcement to be moved
outside of applications and applied universally at
a business process level. These services can either
be called proactively by applications as data is
entered into an application system, or by batch
after the data has been created.
3. Implementing Data Standardization and Data
Enrichment
Data standardization usually covers
reformatting of user-entered data without any
loss of information or enrichment of information.
Such solutions are most suitable for applications
that integrate data. Data enrichment covers the
reformatting of data with additional enrichment
or addition of useful referential and analytical
information.
4 Big Data Quality
Data Quality: Methodology in Profiling
Data profiling provides a proactive way to manage
and comprehend an organization’s data. Data pro-
filing is explicitly about discovering and
reviewing the underlying data available to deter-
mine the characteristics, patterns, and essential
statistics about the data. Data profiling is an
important diagnostic phase that furnishes quanti-
fiable and tangible facts about the strength of the
organization’s data. These facts not only help in
establishing what data is available in the organi-
zation but also how accurate, valid, and usable the
data is. Data profiling covers numerous tech-
niques and processes:
– Data Ancestry: This covers the lineage of the
dataset. It describes the source from which the
data is acquired or derived and the method of
acquisition.
– Data Accuracy: This is the closeness of the
attribute data associated with an object or fea-
ture, to the true value. It is usually recorded as
the percentage correctness for each topic or
attribute.
– Data Latency: This is the level at which the
data is current or accurate to date. This can be
measured by having appropriate data reconcil-
iation procedures to gauge any unintended
delays in acquiring the data due to technical
issues.
– Data Consistency: This is the fidelity or integ-
rity of the data within data structures or
interfaces.
– Data Adherence: This is a measure of compli-
ance or adherence of the data to the intended
standards or logical rules that govern the stor-
age or interpretation of data.
– Data Duplicity: This is a measure of dupli-
cates records or fields in the system that can be
consolidated to reduce the maintenance costs
and efficiency of the system storage processes.
– Data Completeness: This is a measure of the
correspondence between the real world and the
specified dataset.
In assessing a dataset for veracity, it is impor-
tant to answer core questions about it:
• Do the patterns of the data match expected
patterns?
• Do the data adhere to appropriate uniqueness
and null value rules?
• Are the data complete?
• Are they accurate?
• Do they contain information that is easily
understood and unambiguous?
• Do the data adhere to specified required key
relationships across columns and tables?
• Are there inferred relationships across col-
umns, tables, or databases?
• Are there redundant data?
Data in an enterprise is often derived from
different sources, resulting in data inconsistencies
and nonstandard data. Data profiling helps ana-
lysts dig deeper to look more closely at each of the
individual data elements and establish which data
values are inaccurate, incomplete, or ambiguous.
Data profiling allows analysts to link data in dis-
parate applications based on their relationships to
each other or to a new application being devel-
oped. Different pieces of relevant data spread
across many individual data stores make it diffi-
cult to develop a complete understanding of an
enterprise’s data. Therefore, data profiling helps
one understand how data sources interact with
other data sources.
Metadata
Metadata is used to describe the characteristics of
a data field in a file or a table and contains infor-
mation that indicates the data type, the field
length, whether the data should be unique, and if
a field can be missing or null. Pattern matching
determines if the data values in a field are in the
likely format. Basic statistics about data such as
minimum and maximum values, mean, median,
mode, and standard deviation can provide insight
into the characteristics of the data.
Big Data Quality
Conclusion
Ensuring data quality is one of the most pressing
challenges today for most organizations. With
applications constantly receiving new data and
undergoing incremental changes, achieving data
quality cannot be a onetime event. As organiza-
tions’ appetite for big data grows daily in their
quest to satisfy customers, suppliers, investors,
and employees, the common obstacle of impedi-
ment is data quality. Improving data quality is the
lynchpin to a better enterprise, better decision-
making, and better functionality.
Data quality can be improved, and there are
methods for doing so that are rooted in logic and
experience. On the market are commercial off-
the-shelf (COTS) products which are simple, intu-
itive methods to manage and analyze data – and
establish business rules for an enterprise. Some
can implement a data quality layer that filters
any number of sources for quality standards;
provide real-time monitoring; and enable the pro-
filing of data prior to absorption and aggregation
with a company’s core data. At times, however,
it will be necessary to bring in objective,
third-party subject-matter experts for an impartial
analysis and solution of an enterprise-wide data
problem.
Whatever path is chosen, it is important for an
organization to have a master data management
(MDM) plan no differently than it might have a
recruiting plan or a business development plan. A
sound MDM creates an ever-present return
on investment (ROI) that saves time, reduces
operating costs, and satisfies both clients and
stakeholders.
Further Readings
Capehart, B. L., & Capehart, L. C. (2005). Web based
energy information and control systems: case studies
and applications, 436–437.
5
Cappiello, C., Francalanci, C., & Pernici, B. (2003). Time-
related factors of data quality in multi-channel infor-
mation systems. Journal of Management Information
Systems, 20(3), 71–91.
Friedman, T., & M. Smith. (2011). Measuring the business
value of data quality (Gartner ID# G00218962). Avail-
able at: http://www.data.com/export/sites/data/com
mon/assets/pdf/DS_Gartner.pdf
Greengard, S. (1998). Don’t let dirty data derail you. Work-
force, 77(11), 107–108.
Knolmayer, G., & Röthlin, M. (2006). Quality of material
master data and its effect on the usefulness of distrib-
uted ERP systems. Lecture Notes in Computer Science,
4231, 362–371.
Lederman, R., Shanks, G., Gibbs, M.R. (2003). Meeting
privacy obligations: the implications for information
systems development. Proceedings of the 11th Euro-
pean Conference on Information Systems. Paper pre-
sented at ECIS: Naples, Italy.
Liebenau, J., & Backhouse, J. (1990). Understanding
information: an introduction. Information systems.
Palgrave Macmillan, London, UK.
Loshin, D. (2006). The data quality business case: Pro-
jecting return on investment (White paper). Available
at: http://knowledge-integrity.com/Assets/data_qual
ity_business_case.pdf
Marsh, R. (2005). Drowning in dirty data? It’s time to sink
or swim: A four-stage methodology for total data qual-
ity management. Database Marketing & Customer
Strategy Management, 12(2), 105–112. Available at:
http://link.springer.com/article/10.1057/palgrave.dbm.
3240247.
Redman, T. C. (1995). Improve data quality for competi-
tive advantage. MIT Sloan Management Rev., 36(2),
pp. 99–109.
Sellar, S. (1999). Dust off that data. Sales and Marketing
Management, 151(5), 71–73.
Tayi, G. K., & Ballou, D. P. (1998). Examining data qual-
ity. Communications of the ACM, 41(2), 54–57.
Vayghan, J. A., Garfinkle, S. M., Walenta, C., Healy, D. C.,
& Valentin, Z. (2007). The internal information trans-
formation of IBM. IBM Systems Journal, 46(4),
669–684.
Watts, S., Shankaranarayanan, G., & Even, A. (2009). Data
quality assessment in context: A cognitive perspective.
Decision Support Systems, 48(1), 202–211.
C
Core Curriculum Issues (Big Data
Research/Analysis)
Rochelle E. Tractenberg
Collaborative for Research on Outcomes and
–Metrics, Washington, DC, USA
Departments of Neurology; Biostatistics,
Bioinformatics & Biomathematics; and
Rehabilitation Medicine, Georgetown University,
Washington, DC, USA
Definition
A curriculum is defined as the material and con-
tent that comprises a course of study within a
school or college, i.e., a formal teaching program.
The construct of “education” is differentiated
from “training” based on the existence of a cur-
riculum, through which a learner must progress in
an evaluable, or at least verifiable, way. In this
sense, a fundamental issue about a “big data cur-
riculum” is what exactly is meant by the expres-
sion. “Big data” is actually not a sufficiently
concrete construct to support a curriculum, nor
even the integration of one or more courses into
an existing curriculum. Therefore, the principal
“core curriculum issue” for teaching and learning
around big data is to articulate exactly what
knowledge, skills, and abilities are to be taught
and practiced through the curriculum. A second
core issue is how to appropriately integrate those
key knowledge, skills, and abilities (KSAs) into
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_285-1
the curricula of those who will not obtain degrees
or certificates in disciplines related to big data –
but for whom training or education in these KSAs
is still desired or intended. A third core issue is
how to construct the curriculum – whether the
degree is directly related to big data or some key
KSAs relating to big data are proposed for inte-
gration into another curriculum – in such a way
that it is evaluable. Since the technical attributes
of big data and its management and analysis are
evolving nearly constantly, any curriculum devel-
oped to teach about big data must be evaluated
periodically (e.g., annually) to ensure that what is
being taught is relevant; this suggests that core
underpinning constructs must be identified so that
learners in every context can be encouraged to
adapt to new knowledge rather than requiring
retraining or reeducation.
Role of the Curriculum in “Education”
Versus “Training”
Education can be differentiated from training by
the existence of a curriculum in the former and its
absence in the latter. The Oxford English Dictio-
nary defines education as “the process of educating
or being educated, the theory and practice of teach-
ing,” whereas training is defined as “teaching a
particular skill or type of behavior through regular
practice and instruction.” The United Nations
Educational, Scientific and Cultural Organization
(UNESCO) highlights the fact that there may be an
2 Core Curriculum Issues (Big Data Research/Analysis)

articulated curriculum (“intended”) but the curric- curriculum, it is important to understand that
ulum that is actually delivered (“implemented”) there is no uniform cognitive schema, nor other
may differ from what was intended. There are also contextual support, that the formal curriculum
the “actual” curriculum, representing what students typically provides. Thus, it can be helpful to con-
learn, and the “hidden” curriculum, which com- sider “training in big data” as appropriate for those
prises all the bias and unintended learning that any who have completed a formal curriculum in data-
given curriculum achieves (http://www.unesco. related domains. Otherwise, skills that are
org/new/en/education/themes/strengthening-educ acquired in such training, intended for deploy-
ation-systems/quality-framework/technical-notes ment currently and specifically, may actually
/different-meaning-of-curriculum/). These types limit the trainees’ abilities to adapt to new knowl-
of curricula are also described by the Netherlands edge, and thereby, lead to a requirement for
Institute for Curriculum Development (SLO, retraining or reeducation.
http://international.slo.nl/) and worldwide in mul-
tiple books and publications on curriculum devel-
opment and evaluation.
Determining the Knowledge, Skills, and
When a curriculum is being developed or eval-
Abilities Relating to Big Data That
uated with respect to its potential to teach about big
Should Be Taught
data, each of these dimensions of that curriculum
(intended, implemented, actual, hidden) must be
The principal core curricular issue for teaching
considered. These features, well known to instruc-
and learning around big data is to articulate
tors and educators who receive formal training to
exactly what knowledge, skills, and abilities are
engage in the kindergarten–12th grade (US) or
to be taught and practiced through the curriculum.
preschool/primary/secondary (UK/Europe) edu-
As big data has become an increasingly popular
cation, are less well known among instructors in
construct (since about 2010), different stake-
tertiary/higher education settings whose training
holders in the education enterprise have articu-
is in other domains – even if their main job will be
lated curricular objectives in computer science,
to teach undergraduate, graduate, postgraduate,
statistics, mathematics, and bioinformatics for
and professional students. It may be helpful, in
undergraduate (e.g., De Veaux et al. 2017) and
the consideration of curricular elements around
graduate students (e.g., Greene et al. 2016). These
big data, for those in the secondary education/
stakeholders include longstanding national or
college/university setting to consider what attri-
international professional associations and new
butes characterize the curricula that their incom-
groups seeking to establish either their own cred-
ing students have experienced relating to the same
ibility or to define the niche in “big data” where
content or topics.
they plan to operate. However, “big data” is not a
Many modern researchers in the learning
specific domain that is recognized or recogniz-
domains reserve the term “training” to mean
able; it has been described as a phenomenon
“vocational training.” For example, Gibbs et al.
(Boyd and Crawford 2012) and is widely consid-
(2004) identify training as specifically “skills
ered not to be a domain for training or education
acquisition” to be differentiated from instruction
on its own. Instead, knowledge, skills, and abili-
(“information acquisition”); together with social-
ties relating to big data are conceptualized as
ization and the development of thinking and prob-
belonging to the discipline of data science; this
lem-solving skills, this information acquisition is
discipline is considered as existing at the intersec-
the foundation of education overall. The voca-
tion of mathematics, computer science, and statis-
tional training is defined as a function of skills or
tics. This is practically implemented as the
behaviors to be learned (“acquired”) by practice in
articulation of foundational aspects of each of
situ. When considering big data trainees, defined
these disciplines together with their formal and
as individuals who participate in any training
purposeful integration into a formal curriculum.
around big data that is outside of a formal
Core Curriculum Issues (Big Data Research/Analysis)
With respect to data science, then, generally,
there is agreement that students must develop
abilities to reason with data and to adapt to a
changing environment, or changing characteris-
tics of data (preferably both). However, there is
not agreement on how to achieve these abilities.
Moreover, because existing undergraduate course
requirements are complex and tend to be compre-
hensive for “general education” as well as for the
content making up a baccalaureate, associate, or
other terminal degree in the postsecondary con-
text, in some cases just a single course may be
considered for incorporation into either required
or elective course lists. This would represent the
least coherent integration of big data into a col-
lege/university undergraduate curriculum. In the
construction of a program that would award a cer-
tificate, minor or major, if it seeks to successfully
prepare students for work in or with big data, or
statistics and data science, or analytics, or of other
programs intended to train or prepare people for
jobs that either focus on, or simply “know about,”
big data must follow the same curricular design
principles that every formal educational enterprise
should follow. If they do not, they risk
underperforming on their advertising and
promises.
It is important to consider the role of training in
the development, or consideration of develop-
ment, of curricula that feature big data. In addition
to the creation of undergraduate degrees and
minors, Master’s degrees, post-baccalaureate cer-
tificate programs, and doctoral programs, all of
which must be characterized by the curricula
they are defined and created to deliver, many
other “training” opportunities and workforce
development initiatives also exist. These are
being developed in corporate and other human
resource-oriented domains, as well as in more
open (open access) contexts. Unlike traditional
degree programs, training and education around
big data are unlikely to be situated specifically
within a single disciplinary context – at least not
exclusively. People who have specific skills, or
who have created specific tools, often create free
or easily accessible representations of the skills or
3
tool – e.g., instructional videos on YouTube or as
formal courses of varying lengths that can be read
(slides, documentation) or watched as webinars.
Examples can be found online at sites including
Big Data University (bigdatauniversity.com), cre-
ated by IBM and freely available, and Coursera
(coursera.org) which offers data science, analyt-
ics, and statistics courses as well as eight different
specializations, comprising curated series of
courses – but also many other topics. Coursera
has evolved many different educational opportu-
nities and some curated sequences that can be
completed to achieve “certification,” with differ-
ent costs depending on the extent of student
engagement/commitment. The Open University
(www.open.ac.uk) is essentially an online version
of regular university courses and curricula (and
so is closer to “education” than “training”) –
degree and certificate programs all have costs
associated and also can be considered to follow a
formal curriculum to a greater extent than any
other option for widely accessible training/learn-
ing around big data. These examples represent a
continuum that can be characterized by the atten-
tion to the curricular structure from minimal (Big
Data University) to complete (The Open Univer-
sity). The individual who selects a given training
opportunity, as well as those who propose and
develop training programs, must articulate exactly
what knowledge, skills, and abilities are to be
taught and practiced. The challenge for individ-
uals making selections is to determine how cor-
rectly an instructor or program developer has
described the achievements the training is
intended to provide. The challenge for those curat-
ing or creating programs of study is to ensure that
the learning objectives of the curriculum are met,
i.e., that the actual curriculum is as high a match
to the intended curriculum as possible. Basic prin-
ciples of curriculum design can be brought to bear
for acceptable results in this matching challenge.
The stronger the adherence to these basic princi-
ples, the more likely a robust and evaluable curric-
ulum, with demonstrable impact, will result. This is
not specific to education around big data, but with
all the current interest in data and data science,
4 Core Curriculum Issues (Big Data Research/Analysis)
these challenges rise to the level of “core curricu-
lum issues” for this domain.
Utility of Training Versus a Curriculum
Around Big Data
De Veaux et al. (2017) convened a consensus
panel to determine the fundamental requirements
for an undergraduate curriculum in “data sci-
ence.” They articulated that the main topical
areas that comprise – and must be leveraged for
appropriate baccalaureate-level training in – this
domain are as follows: data description and
curation, mathematical foundations, computa-
tional thinking, statistical thinking, data model-
ing, communication, reproducibility, and ethics.
Since computational and statistical thinking, as
well as data modeling, all require somewhat dif-
ferent mathematical foundations, this list shows
clearly the challenges in selecting specific “train-
ing opportunities” to support development of new
skills in “big data” for those who are not already
trained in quantitative sciences to at least some
extent. Moreover, arguments are arising in many
quarters (science and society, philosophy/ethics/
bioethics, and professional associations like the
Royal Statistical Society, American Statistical
Association, and Association of Computing
Machinery) that “ethics” is not a single entity
but, with respect to big data and data science, is
a complex – and necessary – type of reasoning
that cannot be developed in a single course or
training opportunity. The complexity of reasoning
that is required for competent work in the domain
referred to exchangeably as “data analytics,”
“data science,” and “big data”, which includes
this ability to reason ethically, underscores the
point that piecemeal training will be unsuccessful
unless the trainee possesses the ability to organize
the new material together with extant (high level)
reasoning abilities, or at least a cognitive/mental
schema within which the diverse training experi-
ences can be integrated for a comprehensive
understanding of the domain.
However, the proliferation of training opportu-
nities around big data suggests a pervasive sense
that a formal curriculum is not actually needed –
just training is. This may arise from a sense that
the technology is changing too fast to create a
whole curriculum around it. Training opportunity
creators are typically experts in the domain, but
may not necessarily be sufficiently expert in
teaching and learning theories, or the domains
from which trainees are coming, to successfully
translate their expertise into effective “training.”
This may lead to the development of new training
opportunities that appear to be relevant, but which
can actually contribute only minimally to an indi-
vidual trainee’s ability to function competently in
a new domain like big data, because they do not
also include or provide contextualization or sche-
matic links with prior knowledge.
An example of this problem is the creation of
“competencies” by subject matter expert consen-
sus committees, which are then used to create
“learning plans” or checklists. The subject matter
experts undoubtedly can articulate what compe-
tencies are required for functional status in their
domain. However, (a) a training experience devel-
oped to fill in a slot within a competency checklist
often fails to support teaching and learning around
the integration of the competencies into regular
practice; and (b) curricula created in alignment
with competencies often do not promote the actual
development and refinement of these competen-
cies. Instead, they may tend to favor the checking-
off of “achievement of competency X” from the
list.
Another potential challenge arises from the
opposite side of the problem, learner-driven train-
ing development. “What learners want and need
from training” should be considered together with
what experts who are actually using the target
knowledge, skills, and abilities believe learners
need from training. However, the typical trainee
will not be sufficiently knowledgeable to choose
the training that is in fact most appropriate for
their current skills and learning objectives. The
construct of “deliberate practice” is instructive
here. In their 2007 Harvard Business Review arti-
cle, “The making of an expert,” Ericsson, Prietula,
and Cokely summarize Ericsson’s prior work on
expertise and its acquisition, commenting that
“(y)ou need a particular kind of practice – delib-
erate practice - to develop expertise” (emphasis in
Core Curriculum Issues (Big Data Research/Analysis)
original, p. 3). Deliberate practice is practice
where weaknesses are specifically identified and
targeted – usually by an expert both in the target
skillset and perhaps more particularly in identify-
ing and remediating specific weaknesses. If a
trainee is not (yet) an expert, determining how
best to address a weakness that one has self-iden-
tified can be another limitation on the success of a
training opportunity, if it focuses on what the
learner wants or believes they need without appeal
to subject matter experts. This perspective argues
for the incorporation of expert opinion into the
development, descriptions, and contextualizations
of training, i.e., the importance of deliberate prac-
tice in the assurance that as much as possible of
the intended curriculum becomes the actual cur-
riculum. Training opportunities around big data
can be developed to support, or fill in gaps, in a
formal curriculum; without this context, training
in big data may not be as successful as desired.
Conclusions
A curriculum is a formal program of study, and
basic curriculum development principles are
essential for effective education in big data – as
in any other domains. Knowledge, skills, and
abilities, and the levels to which these will be
both developed and integrated, must be articulated
in order to structure a curriculum to optimize the
match between the intended and the actual curric-
ula. The principal core curricular issue for teach-
ing and learning around big data is to articulate
exactly what knowledge, skills, and abilities are to
be taught and practiced. A second core issue is
that the “big data” knowledge, skills, and abilities
may require more foundational support for
5
training of those who will not obtain, or have not
obtained, degrees or certificates in disciplines
related to big data. A third core issue is how to
construct the curriculum in such a way that the
alignment of the intended and the actual objec-
tives is evaluable and modifiable as appropriate.
Since the technical attributes of big data and its
management and analysis are evolving nearly
constantly, any curriculum developed to teach
about big data must be evaluated periodically to
ensure the relevance of the content; however the
alignment of the intended and actual curricula
must also be regularly evaluated to ensure learn-
ing objectives are achieved and achievable.
Further Readings
Boyd, D., & Crawford, K. (2012). Critical questions for big
data: Provocations for a cultural, technological, and
scholarly phenomenon. Information, Communication,
& Society, 15(5), 662–679.
De Veaux, R. D., Agarwal, M., Averett, M., Baumer, B. S.,
Bray, A., Bressoud, T. C., et al. (2017). Curriculum guide-
lines for undergraduate programs in data science. Annual
Review of Statistics and its Applications, 4, 2.1–2.16.
doi:10.1146/annurev-statistics-060116-053930. Down-
loaded from http://www.amstat.org/asa/files/pdfs/EDU-
DataScienceGuidelines.pdf. 2 Jan 2017.
Ericsson, K. A., Prietula, M. J., & Cokely, E. T. (2007). The
making of an expert. Harvard Business Review 85
(7–8):114–121, 193. Downloaded from https://hbr.
org/2007/07/the-making-of-an-expert. 5 June 2010.
Gibbs, T., Brigden, D., & Hellenberg, D. (2004). The
education versus training and the skills versus compe-
tency debate. South African Family Practice, 46(10),
5–6. doi:10.1080/20786204.2004.10873146.
Greene, A. C., Giffin, K. A., Greene, C. S., & Moore, J. H.
(2016). Adapting bioinformatics curricula for big data.
Briefings in Bioinformatics, 17(1), 43–50. doi:10.1093/
bib/bbv018.
D
Data Exhaust
Daniel E. O’Leary1 and Veda C. Storey2
1
Marshall School of Business, University of
Southern California, Los Angeles, CA, USA
2
J Mack Robinson College of Business, Georgia
State University, Atlanta, GA, USA
Overview
Data exhaust is a type of big data that is often
generated unintentionally by users from normal
Internet interaction. It is generated in large quan-
tities and appears in many forms, such as the
results from web searches, cookies, and tempo-
rary files. Initially, data exhaust has limited, or
no, direct value to the original data collector.
However, when combined with other data for
analysis, data exhaust can sometimes yield valu-
able insights.
Description
Data exhaust is passively collected and consists of
random online searches or location data that is
generated, for example, from using smart phones
with location dependent services or applications
(Gupta and George 2016). It is considered to
be “noncore” data that may be generated when
individuals use technologies that passively emit
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_303-1
information in daily life (e.g., making an online
purchase, accessing healthcare information, or
interacting in a social network). Data exhaust
can also come from information-seeking behavior
that is used to make inferences about an
individual’s needs, desires, or intentions, such as
Internet searches or telephone hotlines (George
et al. 2014).
Additional Terminology
Data exhaust is also known as ambient data, rem-
nant data, left over data, or even digital exhaust
(Mcfedries 2013). A digital footprint or a digital
dossier is the data generated from online activities
that can be traced back to an individual. The
passive traces of data from such activities are
considered to be data exhaust. The big data that
interests many companies is called “found data.”
Typically data is extracted from random Internet
searches and location data is generated from smart
or mobile phone usage. Data exhaust should not
be confused with community data that is gener-
ated by users in online social communities, such
as Facebook and Twitter.
In the age of big data, one can, thus, view data
as a messy collage of data points, which includes
found data, as well as the data exhaust extracted
from web searches, credit card payments, and
mobile devices. These data points are collected
for disparate purposes (Harford 2014).
2 Data Exhaust
Generation of Data Exhaust
Data exhaust is normally generated autonomously
from transactional, locational, positional, text,
voice, and other data signatures. It typically is
gathered in real time. Data exhaust might not be
purposefully collected, or is collected for other
purposes and then used to derive insights.
Example of Data Exhaust
An example of data exhaust is backend data.
Davidson (2016) provides an example from a
real-time information transit application called
Transit App (Davidson 2016). The Transit App
provides a travel service to users. The App shows
the coming departures of nearby transit services. It
also has information on bike share, car share, and
other ride services, which appear when the user
simply opens the app. The app is intended to be
useful for individuals who know exactly where
they are going and how to get there, but want real-
time information on schedules. The server, how-
ever, retains data on the origin, destination, and
device data for every search result. The usefulness
of this backend data was assessed by comparing
the results obtained from using the backend data
to predict trips, to a survey data of actual trips,
which revealed a very similar origin-destination
pattern.
Sources of Data Exhaust
The origin of data exhaust may be passive, digital,
or transactional. Specifically, data exhaust can be
passively collected as transactional data from peo-
ple’s use of digital services such as mobile
phones, purchases, web searches, etc. These dig-
ital services are then used to create networked
sensors of human behavior.
Potential Value
Data exhaust is accessed either directly in an
unstructured format or indirectly as backend
data. The value of data exhaust often is in its use
to improve online experiences and to make pre-
dictions about consumer behavior. However, the
value of the data exhaust can depend on the par-
ticular application and context.
Challenges
There are practical and research challenges
to deriving value from data exhaust (technical,
privacy and security, and managerial). A major
technical challenge is the acquisition of data
exhaust. Because it is often generated without
the user’s knowledge, this can lead to issues of
privacy and security. Data exhaust is often
unstructured data for which there is, technically,
no known, proven, way to consistently extract its
potential value from a managerial perspective.
Furthermore, data mining and other tools that
deal with unstructured data are still at a relatively
early stage of development.
From a research perspective, traditionally,
research studies of humans have focused on data
collected explicitly for a specific purpose. Com-
putational social science increasingly uses data
that is collected for other purposes. This can result
in the following (Altman 2014):
1. Access to “data exhaust” cannot easily be con-
trolled by a researcher. Although a researcher
may limit access to their own data, data exhaust
may be available from commercial sources or
from other data exhaust sources. This increases
the risk that any sensitive information linked
with a source of data exhaust can be
reassociated with an individual.
2. Data exhaust often produces fine-grained
observations of individuals over time. Because
of regularities in human behavior, patterns in
data exhaust can be used to “fingerprint” an
individual, thereby enabling potential
reidentification, even in the absence of explicit
identifiers or quasi-identifiers.
Evolution
As ubiquitous computing continues to evolve,
there will be a continuous generation of data
exhaust from sensors, social media, and other
sources (Nadella and Woodie 2014). Therefore,
the amount of unstructured data will continue to
grow and, no doubt, attempts to extract value from
data exhaust will grow as well.
Data Exhaust
Conclusion
As the demand for capture and use of real-time
data continues to grow and evolve, data exhaust
may play an increasing role in providing value
to organizations. Much communication, leisure,
and commerce occur on the Internet, which is now
accessible from smartphones, cars, and a multi-
tude of devices (Harford 2014). As a result, activ-
ities of individuals can be captured, recorded,
and represented in a variety of ways, most likely
leading to an increase in efforts to capture and use
data exhaust.
Further Readings
Altman, M. (2014). Navigating the changing landscape of
information privacy. http://informatics.mit.edu/blog/
2014/10/examples-big-data-and-privacy-problems
3
Bhushan, A. (2013). “Big data” is a big deal for develop-
ment. In Higgins, K. (Ed), International development
in a changing world, 34. The North-South Institute,
Ottawa, Canada.
Davidson, A. (2016). Big data exhaust for origin-destina-
tion surveys: Using mobile trip-planning data for sim-
ple surveying. Proceedings of the 95th Annual Meeting
of the Transportation Research Board.
George, G., Haas, M. R., & Pentland, A. (2014). Big data
and management. Academy of Management Journal,
57(2), 321–326.
Gupta, M., & George, J. F. (2016). Toward the develop-
ment of a big data analytics capability. Information
Management, 53(8), 1049–1064.
Harford, T. (2014). Big data: A big mistake? Significance,
11(5), 14–19.
Mcfedries, P. (2013). Tracking the quantified self [Techni-
cally speaking]. IEEE Spectrum, 50(8), 24–24.
Nadella, A., & Woodie, A. (2014). Data ‘exhaust’ leads
to ambient intelligence, Microsoft CEO says. https://
www.datanami.com/2014/04/15/data_exhaust_leads_
to_ambient_intelligence_microsoft_ceo_says/
D
Data Fusion
Carolynne Hultquist
Geoinformatics and Earth Observation
Laboratory, Department of Geography and
Institute for CyberScience, The Pennsylvania
State University, University Park, PA, USA
Definition/Introduction
Data fusion is a process that joins together differ-
ent sources of data. The main concept of using a
data fusion methodology is to synthesize data
from multiple sources in order to create collective
information that is more meaningful than if only
using one form or type of data. Data from many
sources can corroborate information, and, in the
era of big data, there is an increasing need to
ensure data quality and accuracy. Data fusion
involves managing this uncertainty and
conflicting data at a large scale. The goal of data
fusion is to create useful representations of reality
that are more complete and reliable than a single
source of data.
Integration of Data
Data fusion is a process that integrates data
from many sources in order to generate more
meaningful information. Data fusion is very
domain-dependent, and therefore, tasks and the
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_305-1
development of methodologies are dependent on
the field for diverse purposes (Bleiholder and
Naumann 2008). In general, the intention is to
fuse data from many sources in order to increase
value. Data from different sources can support
each other which decreases uncertainty in the
assessment or conflicts which raises questions of
validity. Castanedo (2013) groups the data fusion
field into three major methodological categories
of data association, state estimation, and decision
fusion. Analyzing the relationships between
multiple data sources can help to provide an
understanding of the quality of the data as well
as identify potential inconsistencies.
Modern technologies have made data easier to
collect and more accessible. The development of
sensor technologies and the interconnectedness
of the Internet of things (IoT) have linked together
an ever-increasing number of sensors and devices
which can be used to monitor phenomena. Data
is accessible in large quantities, and multiple sources
of data are sometimes available for an area of inter-
est. Fusing data from a variety of forms of sensing
technologies can open new doors for research and
address issues of data quality and uncertainty.
Multisensor data fusion can be done for data
collected for the same type of phenomena. For
example, environmental monitoring data such as
air quality, water quality, and radiation measure-
ments can be compared to other sources and models
to test the validity of the measurements that were
collected. Geospatial data is fused with data col-
lected in different forms and is sometimes also
2 Data Fusion

known in this domain as data integration. Geograph- Cross-References

ical information from such sources as satellite
remote sensing, UAVs (unmanned aerial vehicles), ▶ Big Data Quality
geolocated social media, and citizen science data ▶ Big Data Volume
can be fused to give a picture that any one source ▶ Big Variety Data
cannot provide. Assessment of hazards is an appli- ▶ Data Integration
cation area in which data fusion is used ▶ Data Veracity
to corroborate the validity of data from many ▶ Disaster Planning
sources. The data fusion process is often able to fill ▶ Internet of things ( IOT)
some of the information gaps that exist and could ▶ Sensor Technologies
assist decision-makers by providing an assessment
of real-world events.
Further Readings
Conclusion Bleiholder, J., & Naumann, F. (2008). Data fusion. ACM
Computing Surveys, 41, 1:1–1:41.
The process of data fusion directly seeks to address Castanedo, F. (2013). A review of data fusion techniques.
challenges of big data. The methodologies are The Scientific World Journal, 2013, 1–19, Article ID
704504.
directed at considering the veracity of large volumes
and many varieties of data. The goal of data fusion is
to create useful representations of reality that are
more complete and reliable than trusting data that
is only from a single source.
M
Middle East
Feras A. Batarseh
College of Science, George Mason University,
Farifax, VA, USA
Synonyms
Mid-East; The Middle East and North Africa
(MENA)
Definition
The Middle East is a transcontinental region in
Western Asia and North Africa. Countries of the
Middle East are ones extending from the shores of
the Mediterranean Sea, south towards Africa, and
east towards Asia, and sometimes beyond
depending on the context (political, geographical,
etc.). The majority of the countries of the region
speak Arabic.
Introduction
The term “Middle East” evolved with time. It was
originally referred to as the countries of the Otto-
man empire, but by the mid-twentieth century, a
more common definition of the Middle East
included the following states (countries): Turkey,
Jordan, Cyprus, Lebanon, Iraq, Syria, Israel, Iran,
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_400-1
the West Bank and the Gaza Strip (Palestine),
Egypt, Sudan, Libya, Saudi Arabia, Kuwait,
Yemen, Oman, Bahrain, Qatar, and United Arab
Emirates (UAE). Subsequent political and histor-
ical events have tended to include more countries
into the mix (such as: Tunisia, Algeria, Morocco,
Afghanistan, and Pakistan).
The Middle East is often referred to as the
cradle of civilization. By studying the history of
the region, it is clear why the first human civiliza-
tions were established in this part of the world
(particularly the Mesopotamia region around the
Tigris and Euphrates rivers). The Middle East is
where humans made their first transitions from
nomadic to agriculture, invented the wheel, cre-
ated basic agriculture, and where the beginnings
of the written-word first existed. It is well known
that this region is an active political, economic,
historic, and religious part of the world
(Encyclopedia Britannica 2017). For the purposes
of this encyclopedia, the focus of this entry is on
technology, data, and software of the Middle East.
The Digital Age in the Middle East
Since the beginning of the 2000s, the Middle East
was one of the highest regions in the world in
terms of adoption of social media; certain coun-
tries (such as the United Arab Emirates, Qatar, and
Bahrain) have adopted social technologies by
70% of its population (which is a higher percent-
age than the United States). While citizens are
2 Middle East
jumping on the wagon of social media, govern-
ments still struggle to manage, define, or guide the
usage of such technologies.
The McKinsey Middle East Digitization Index
is the one of the main metrics to assess the level
and impact of digitization across the Middle East.
Only 6% of Middle Eastern public lives under a
digitized smart or electronic government (The
UAE, Jordan, Israel, and Saudi Arabia are
among the few countries that have some form of
e-government) (Elmasri et al. 2016). However,
many new technology startups are coming from
the Middle East with great success. The most
famous technology startup companies coming
out of the Middle East include: (1) Maktoob
(from Jordan): is one that stands out. The com-
pany represents a major trophy on the list of
Middle Eastern tech achievements. It made global
headlines when it was bought by Yahoo, Inc. for
$80 million in 2009, symbolizing a worldwide
important step by a purely Middle Eastern com-
pany. (2) Yamli (from Lebanon): One of the most
popular web apps for Arabic speakers today.
(3) GetYou (from Israel): A famous social media
application. (4) Digikala (from Iran): An online
retailer application. (5) ElWafeyat (from Egypt):
An Arabic language social media site for honoring
deceased friends and family. (6) Project X (from
Jordan): A mobile application that allows for 3D
printing of prosthetics, inspired by wars in the
region. These examples are assembled from mul-
tiple sources; many other exciting projects exist as
well (such as Souq which was acquired by Ama-
zon in 2017, Masdar, Namshi, Sukar, and many
others).
Software Arabization: The Next Frontier
The first step towards invoking more technology
in a region is to localize the software, content, and
its data. Localizing a software system is accom-
plished by supporting a new spoken language
(Arabic Language in this context, hence the
name, Arabization). A new term is presented in
this entry of the Encyclopedia, Arabization: it is
the overall concept that includes the process of
making the software available and reliable across
the geographical borders of the Arab states. Dif-
ferent spoken languages have different orienta-
tions and fall into different groups. Dealing with
these groups is accomplished by using different
code pages and Unicode fonts. Languages fall into
two main families, single-byte (such as: French,
German, and Polish) and double-byte (such as:
Japanese, Chinese, and Korean). Another catego-
rization that is more relevant to Middle Eastern
Languages is based on their orientation. Most
Middle Eastern languages are right-to-left (RTL)
(such as: Arabic and Hebrew), while other world
languages are left-to-right (LTR) (such as: English
and Spanish). For all languages, however, a set of
translated strings should be saved in a bundle file
that indexes all the strings, assign them IDs so the
software program can locate them and display the
right string in the language of the user. Further-
more, to accomplish software Arabization, char-
acters encoding should be enabled. The default
encoding for a given system is determined by the
runtime locale set on the machine’s operating
system. The most commonplace character
encoding format is UTF (USC transformation for-
mat) USC is the universal character set. UTF is
designed to be compatible with ASCII. UTF has
three types: UTF-8, UTF-16, and UTF-32. UTF is
the international standard for ISO/IEC 10646. It is
important to note that the process of Arabization is
not a trivial process; engineers cannot merely
inject translated language strings into the system,
or hardcode cultural, date, or numerical settings
into the software, rather, the process is done by
obtaining different files based on the settings of
the machine, the desires of the user, and applying
the right locales. An Arabization package needs to
be developed to further develop the digital, soft-
ware, and technological evolution in the
Middle East.
Bridging the Digital Divide
Information presented in this entry showed how
the Middle East is speeding towards catching-up
with industrialized nations in terms of software
Middle East
Middle East,
Fig. 1 Middle Eastern
Investments in Technology
(Elmasri et al. 2016)
technology adoption and utilizations (i.e., bridge
the digital divide between third world and first
world countries). Figure 1 below shows which
countries are investing towards leading that trans-
formation; numbers in the figure illustrate venture
capital funding as share of GDP (Elmasri et al.
2016). However, According to Cisco’s 2015
visual networking index (VNI), the world is
looking towards a new digital divide, beyond
software and mobile apps. By 2019, the number
of people connecting to Internet is going to rise to
3.9 billion users, reaching over 50% of the global
population. That will accelerate the new wave of
big data, machine learning, and the Internet of
Things (IoT). That will be the main new challenge
for technology innovators in the Middle East.
Middle Eastern countries need to first lay the
“data” infrastructure (such as the principle of soft-
ware Arabization presented above) that would
enable the peoples of the Middle East towards
higher adoption rates of future trends (big data
and IoT). Such a shift would greatly influence
3
economic growth at countries all across the
region; however, the impacts of technology
require minimum adoption thresholds before
those impacts begin to materialize; the wider the
intensity and use of big data, Internet of things
(IoT), and machine learning, the greater the
impacts.
Conclusion
The Middle East is known for many historical and
political events, conflicts, and controversies; how-
ever, it is not often referred to as a technological
and software-startup hub. This entry of the Ency-
clopedia presents a brief introduction to the Mid-
dle East and draws a simple picture about its
digitization, and claims that Arabization of soft-
ware could lead to many advancements across the
region and eventually the world – for startups and
creativity, the Middle East is an area worth
watching (Forbes 2017).
4 Middle East
References
Elmasri, T., Benni, E., Patel, J., & Moore, J. (2016). Digital
Middle East: Transforming the region into a leading
digital economy. McKinsey and Company. https://www.
google.com/url?sa=t&rct=j&q=&esrc=s&source=web&
cd=2&ved=0ahUKEwiG2J2e55LTAhXoiVQKHfD8Cx
AQFggfMAE&url=http%3A%2F%2Fwww.mckinsey.
com%2F~%2Fmedia%2Fmckinsey%2Fglobal%2520
themes%2Fmiddle%2520east%2520and%2520africa
%2Fdigital%2520middle%2520east%2520transforming
%2520the%2520region%2520into%2520a%2520lead
ing%2520digital%2520economy%2Fdigital-middle-east-
finalupdated.ashx&usg=AFQjCNHioXhFY692mS_Qwa
6hkBT6UiXYVg&sig2=6udbc7EP-bPs-ygQ18KSLA&
cad=rja
Encyclopedia Britannica. (2017). Available at https://
www.britannica.com/place/Middle-East
Forbes reports on the Middle East. (2017). Available at
http://www.forbes.com/sites/natalierobehmed/2013/08/
22/forget-oil-tech-could-be-the-next-middle-east-
goldmine/
S
Sensor Technologies
Carolynne Hultquist
Geoinformatics and Earth Observation
Laboratory, Department of Geography and
Institute for CyberScience, The Pennsylvania
State University, University Park, PA, USA
Definition/Introduction
Sensors technologies are developed to detect spe-
cific phenomena, behavior, or actions. The origin
of the word sensor comes from the Latin root
“sentire” a verb defined as “to perceive”
(Kalantar-zadeh 2013). Sensors are designed to
identify certain phenomena as a signal but not
record anything else as it would create noise in
the data. Sensors are specified by purpose to iden-
tify or measure the presence or intensity of differ-
ent types of energy: mechanical, gravitational,
thermal, electromagnetic, chemical, and nuclear.
Sensors have become part of everyday life and
continue to grow in importance in modern
applications.
Prevalence of Sensors
Sensors are used in everyday life to detect phenom-
ena, behavior, or actions such as force, temperature,
pressure, flow, etc. The type of sensor utilized is
based on the type of energy that is being sensed, be
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_442-1
it gravitational, mechanical, thermal, electromag-
netic, chemical, or nuclear. The activity of interest
is typically measured by a sensor and converted by
a transducer into a signal as a quantity (McGrath
and Scanaill 2013). Sensors have been integrated
into daily life so that we use them without consid-
ering tactile sensors such as elevator buttons,
touchscreen devices, and touch sensing lamps.
Typical vehicles contain numerous sensors for driv-
ing functions, safety, and the comfort of the pas-
sengers. Mechanical sensors measure motion,
velocity, acceleration, and displacement through
such sensors as strain gauges, pressure, force, ultra-
sonic, acoustic wave, flow, displacement, acceler-
ometers, and gyroscopes (McGrath and Scanaill
2013). Chemical and thermal biometric sensors
are often used for healthcare from traditional
forms like monitoring temperature, blood pressure
cuffs to glucose meters, pacemakers, defibrillators,
and HIV testing.
New sensor applications are developing which
produce individual, home, and environmental
data. There are many sensor types that were devel-
oped years ago but are finding new applications.
Navigational aids, such sensors as gyroscopes,
accelerometers, and magnetometers, have existed
for many years in flight instruments for aircraft
and more modernly for smartphones. Sensors
internal to smartphone devices are intended to
monitor the device but can be repurposed to mon-
itor to monitor many things such as extreme expo-
sure to heat or movement for health applications.
The interconnected network of devices to promote
2 Sensor Technologies
automation and efficiency is often referred to as
the Internet of things (IoT). Sensors are becoming
more prevalent and cheap enough that the public
can make use of personal sensors that already
exist in their daily lives or can be easily acquired.
Personal Health Monitoring
Health-monitoring applications are becoming
increasingly common and produce very large vol-
umes of data. Biophysical processes such as heart
rate, breathing rate, sleep patterns, and restless-
ness can be recorded continuously using devices
kept in contact with the body. Health-conscious
and athletic communities, such as runners, have
particularly taken to personal monitoring by using
technology to track their current condition and
progress. Pedometers, weight scales, and ther-
mometers are commonplace. Heart rate, blood
pressure, and muscle fatigue are now monitored
by affordable devices in the form of bracelets,
rings, adhesive strips, and even clothing. Brands
of smart clothing are offering built-in sensors for
heart rate, respiration, skin temperature and mois-
ture, and electrophysiological signals that are
sometimes even recharged by solar panels. There
are even wireless sensors for the insole of shoes to
automatically adjust for the movements of the
user in addition to providing health and training
analysis.
Wearable health technologies are often used to
provide individuals with private personal informa-
tion; however, certain circumstances call for sys-
tem-wide monitoring for medical or emergency
purposes. Medical patients, such as those with
diabetes or hypertension, can use continuously
testing glucose meters or blood pressure monitors
(Kalantar-zadeh 2013). Bluetooth-enabled devices
can transmit data from monitoring sensors and
contact the appropriate parties automatically if
there are health concerns. Collective health infor-
mation can be used to have a better understanding
of such health concerns as cardiac issues, extreme
temperatures, and even crisis information.
Smart Home
Sensors have long been a part of modern house-
holds from smoke and carbon monoxide detec-
tors to security systems and motion sensors.
Increasingly, smart home sensors are being used
for everyday monitoring in order to have more
efficient energy consumption with smart lighting
fixtures and temperature controls. Sensors are
often placed to inform on activities in the house
such as a door or window being opened. This
integrated network of house monitoring prom-
ises efficiency, automation, and safety based on
personal preferences. There is significant invest-
ment in smart home technologies, and big data
analysis can play a major role in determining
appropriate settings based on feedback.
Environmental Monitoring
Monitoring of the environment from the surface to
the atmosphere is traditionally a function
performed by the government through remotely
sensed observations and broad surveys. Remote
sensing imagery from satellites and airborne
flights can create large datasets on global environ-
mental changes for use in such applications as
agriculture, pollution, water, climatic conditions,
etc. Government agencies also employ static sen-
sors and make on-site visits to check sensors
which monitor environmental conditions. These
sensors are sometimes integrated into networks
which can communicate observations to
form real-time monitoring systems.
In addition to traditional government sources
of environmental data, there are growing collec-
tions of citizen science data that are focused pri-
marily on areas of community concern such as air
quality, water quality, and natural hazards. Air
quality and water quality have long been moni-
tored by communities concerned about pollution
in their environment, but a recent development
after the 2011 Fukushima nuclear disaster is radi-
ation sensing. Safecast is a radiation monitoring
project that seeks to empower people with infor-
mation on environmental safety and openly dis-
tributes measurements under creative commons
rights (McGrath and Scanaill 2013). Radiation is
not visibly observable so it is considered a “silent”
environmental harm, and the risk needs to be
considered in light of validated data (Hultquist
and Cervone 2017). Citizen science projects for
sensing natural hazards from flooding, landslides,
earthquakes, wildfires, etc. have come online with
Sensor Technologies
support from both governments and communities.
Open-source environmental data is a growing
movement as people get engaged with their envi-
ronment and become more educated about their
health.
Conclusion
The development and availability of sensor tech-
nologies is a part of the big data paradigm. Sen-
sors are able to produce an enormous amount of
data, very quickly with real-time uploads, and
from diverse types of sensors. Many questions
still remain of how to use this data and if
connected sensors will lead to smart environments
that will be a part of everyday modern life. The
Internet of things (IoT) is envisioned to connect
communication across domains and applications
in order to enable the development of smart cities.
Sensor data can provide useful information for
individuals and generalized information from col-
lective monitoring. Services often offer personal-
ized analysis in order to keep people engaged using
the application. Yet, most analysis and interest
from researchers in sensor data is at a generalized
level. Despite mostly generalized data analysis,
there is public concern related to data privacy
from individual and home sensors. The privacy
level of the data is highly dependent on the system
used and the terms of service agreement if a service
is being provided related to the sensor data.
Analysis of sensor data is often complex, messy,
and hard to verify. Nonpersonal data can often be
checked or referenced to a comparable dataset to
see if it makes sense. However, large datasets pro-
duced by personal sensors for such applications as
health are difficult to independently verify at an
individual level. For example, an environmental
condition could have caused a natural reaction of
a rapid heartbeat which is medically safe given the
condition that the user awoke with a quick increase
in heart rate due to an earthquake. Individual
inspection of data for such noise is fraught with
problems as it is complicated to identify causes in
3
the raw data from an individual, but at a generalized
level, such data can be valuable for research and
can appropriately take into account variations in the
data.
Sensor technologies are integrated into every-
day life and are used in numerous applications to
monitor conditions. The usefulness of technolog-
ical sensors should be no surprise as every living
organism has biological sensors which serve sim-
ilar purposes to indicate the regulation of internal
functions and conditions of the external environ-
ment. The integration of sensor technologies is a
natural step that goes from individual measure-
ments to collective monitoring which highlights
the need for big data analysis and validation.
Cross-References
▶ AgInformatics
▶ Air Pollution
▶ Biometrics
▶ Biosurveillance
▶ Crowdsourcing
▶ Drones
▶ Environment
▶ Health Informatics
▶ Land Pollution
▶ Participatory Health and Big Data
▶ Patient-Centered (Personalized) Health
▶ Remote Sensing
▶ Water Pollution
Further Readings
Hultquist, C., & Cervone, G. (2017). Citizen monitoring
during hazards: Validation of Fukushima radiation
measurements. Geo Journal. http://doi.org/10.1007/
s10708-017-9767-x.
Kalantar-zadeh, K. (2013). Sensors: An introductory
course (1st ed.). Boston: Springer US.
McGrath, M. J., & Scanaill, C. N. (2013). Sensor technol-
ogies: Healthcare, wellness, and environmental appli-
cations. New York: Apress Open.
S
“Small” Data
Rochelle E. Tractenberg1,2 and
Kimberly F. Sellers3
1
Collaborative for Research on Outcomes and
Metrics, Washington, DC, USA
2
Departments of Neurology; Biostatistics,
Bioinformatics & Biomathematics; and
Rehabilitation Medicine, Georgetown University,
Washington, DC, USA
3
Department of Mathematics and Statistics,
Georgetown University, Washington, DC, USA
Synonyms
Data; Statistics
Introduction
Big data are often characterized by “the 3 Vs”:
volume, velocity, and variety. This implies that
“small data” lack these qualities, but that is an
incorrect conclusion about what defines “small”
data. Instead, we define “small data” to be simply
“data” – specifically, data that are finite but not
necessarily “small” in scope, dimension, or rate of
accumulation. The characterization of data as
“small” is essentially dependent on the context
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_445-1
and use for which the data are intended. In fact,
disciplinary perspectives vary on how large “big
data” need to be to merit this label, but small data
are not characterized effectively by the absence of
one or more of these “3 Vs.” Most statistical
analyses require some amount of vector and
matrix manipulation for efficient computation in
the modern context. Data sets may be considered
“big” if they are so large, multidimensional,
and/or quickly accumulating in size that the typi-
cal linear algebraic manipulations cannot con-
verge or yield true summaries of the full data set.
The fundamental statistical analyses, however, are
the same for data that are “big” or “small”; the true
distinction arises from the extent to which com-
putational manipulation is required to map and
reduce the data (Day and Ghemawat 2004) such
that a coherent result can be derived. All analyses
share common features, irrespective of the size,
complexity, or completeness of the data – the
relationship between statistics and the underlying
population; the association between inference,
estimation, and prediction; and the dependence
of interpretation and decision-making on statisti-
cal inference. To expand on the lack of distin-
guishability between “small” data and “big”
data, we explore each of these features in turn.
By doing so, we expound on the assertion that a
characterization of a dataset as “small” depends
on the users’ intention and the context in which
the data, and results from its analysis, will be used.
2 “Small” Data
Understanding “Big Data” as “Data”
An understanding of why some datasets are char-
acterized as “big” and/or “small” requires some
juxtaposition of these two descriptors. “Big data”
are thought to expand the boundary of data sci-
ence because innovation has been ongoing to
promote ever-increasing capacity to collect and
analyze data with high volume, velocity, and/or
variety (i.e., the 3 Vs). In this era of technological
advances, computers are able to maintain and
process terabytes of information, including
records, transactions, tables, files, etc. However,
the ability to analyze data has always depended on
the methodologies, tools, and technology avail-
able at the time; thus the reliance on computa-
tional power to collect or process data is not new
or specific to the current era and cannot be con-
sidered to delimit “big” from “small” data.
Data collection and analyses date back to
ancient Egyptian civilizations that collected cen-
sus information; the earliest Confucian societies
collected this population-spanning data as well.
These efforts were conducted by hand for centu-
ries, until a “tabulating machine” was used to
complete the analyses required for the 1890
United States Census; this is possibly the first
time so large a dataset was analyzed with a non-
human “computer.” Investigations that previously
took years to achieve were suddenly completed in
a fraction of the time (months!). Since then, tech-
nology continues to be harnessed to facilitate data
collection, management, and analysis. In fact,
when it was suggested to add “data science” to
the field of statistics (Bickel 2000; Rao 2001),
“big data” may have referred to a data set of up
to several gigabytes in size; today, petabytes of
data are not uncommon. Therefore, neither the
size nor the need for technological advancements
are inherent properties of either “big” or
“small” data.
Data are sometimes called “big” if the data
collection process is fast(-er), not finite in time
or amount, and/or inclusive of a wide range of
formats and quality. These features may be
contrasted with experimental, survey, epidemio-
logic, or census data where the data structure,
timing, and format are fixed and typically finite.
Technological advances allow investigators to
collect batches of experimental, survey, or other
traditional types of data in near-real or real time,
or in online or streaming fashion; such informa-
tion has been incorporated to ask and answer
experimental and epidemiologic questions,
including testing hypotheses in physics, climate,
chemistry, and both social and biomedical sci-
ences, since the technology was developed. It is
inappropriate to distinguish “big” from “small”
data along these characteristics; in fact, two ana-
lysts simultaneously considering the same data set
may each perceive it to be “big” or “small”; these
labels must be considered to be relative.
Analysis and Interpretation of “Big
Data” Is Based on Methods for “Small
Data”
Considering analysis, manipulation, and interpre-
tation of data can support a deeper appreciation
for the differences and similarities of “big” and
“small” data. Large(r) and higher-dimensional
data sets may require computational manipulation
(e.g., Day and Ghemawat 2004), including group-
ing and dimension reduction, to derive an inter-
pretable result from the full data set. Further,
whenever a larger/higher dimension dataset is
partitioned for analysis, the partitions or subsets
are analyzed using standard statistical methods.
The following sections explicate how standard
statistical analytic methods (i.e., for “small”
data) are applied to a dataset whether it is
described as “small” or “big”. These methods are
selected, employed, and interpreted specifically to
support the user’s intention for the results and do
not depend inherently on the size or complexity of
the data itself. This underscores the difficulty of
articulating any specific criterion/a for character-
izing data as “big” or “small.”
Sample Versus Population
Statistical analysis and summarization of “big”
data are the same as for data generally; the
description, confidence/uncertainty, and coher-
ence of the results may vary with the size and
completeness of the data set. Even the largest
“Small” Data
and most multidimensional dataset is presumably
an incomplete (albeit massive) representation of
the entire universe of values – the “population.”
Thus, the field of statistics has historically been
based on long-run frequencies or computed esti-
mates of the true population parameters. For
example, in some current massive data collection
and warehousing enterprises, the full population
can never be obtained because the data are con-
tinuously streaming in and collected. In other
massive data sets, however, the entire population
is captured; examples include the medical records
for a health insurance company, sales on Amazon.
com, or weather data for the detection of an evolv-
ing storm or other significant weather pattern. The
fundamental statistical analyses would be the
same for either of these data types; however,
they would result in estimates for the
(essentially) infinite data set, while actual
population-descriptive values are possible when-
ever finite/population data are obtained. Impor-
tantly, it is not the size or complexity of the data
that results in either estimation or population
description – it is whether or not the data are finite.
This underscores the reliance of any and all data
analysis procedures on statistical methodologies;
assumptions about the data are required for the
correct use and interpretation of these methodol-
ogies for data of any size and complexity. It fur-
ther blurs qualifications of a given data set as
“big” or “small.”
Inference, Estimation, and Prediction
Statistical methods are generally used for two
purposes: (1) to estimate “true” population param-
eters when only sample information is available,
and (2) to make or test predictions about either
future results or about relationships among vari-
ables. These methods are used to infer “the truth”
from incomplete data and are the foundations of
nearly all experimental designs and tests of quan-
titative hypotheses in applied disciplines (e.g.,
science, engineering, and business). Modern sta-
tistical analysis generates results (i.e., parameter
estimates and tests of inferences) that can be char-
acterized with respect to how rare they are given
the random variability inherent in the data set.
3
In frequentist statistical analysis (based on long
run results), this characterization typically
describes how likely the observed result would
be if there were, in truth, no relationship between
(any) variables, or if the true parameter value was
a specific value (e.g., zero). In Bayesian statistical
analysis (based on current data and prior knowl-
edge), this characterization describes how likely it
is that there is truly no relationship given the data
that were observed and prior knowledge about
whether such a relationship exists.
Whenever inferences are made about estimates
and predictions about future events, relationships,
or other unknown/unobserved events or results,
corrections must be made for the multitude of
inferences that are made for both frequentist and
Bayesian methods. Confidence and uncertainty
about every inference and estimate must accom-
modate the fact that more than one has been made;
these “multiple comparisons corrections” protect
against decisions that some outcome or result is
rare/statistically significant when, in fact, the var-
iability inherent in the data make that result far
less rare than it appears. Numerous correction
methods exist with modern (since the mid-
1990s) approaches focusing not on controlling
for “multiple comparisons” (which are closely
tied to experimental design and formal hypothesis
testing), but controlling the “false discovery rate”
(which is the rate at which relationships or esti-
mates will be declared “rare given the inherent
variability of the data” when they are not, in fact,
rare). Decisions made about inferences, estimates,
and predictions are classified as correct (i.e., the
event is rare and is declared rare, or the event is
not rare and is declared not rare) or incorrect (i.e.,
the event is rare but is declared not rare – a false
negative/Type II error; or the event is not rare but
is declared rare – a false positive/Type I error);
controls for multiple comparisons or false discov-
eries seek to limit Type I errors.
Decisions are made based on the data analysis,
which holds for “big” or “small” data. While
multiple comparisons corrections and false dis-
covery rate controls have long been accepted as
representing competent scientific practice, they
are also essential features of the analysis of big
4 “Small” Data
data, whether or not these data are analyzed for
scientific or research purposes.
Analysis, Interpretation, and Decision Making
Analyses of data are either motivated by theory or
prior evidence (“theory-driven”), or they are
unplanned and motivated by the data themselves
(“data-driven”). Both types of investigations can
be executed on data of any size, complexity, or
completeness. While the motivations for data
analysis vary across disciplines, evidence that
supports decisions is always important. Statistical
methods have been developed, validated, and uti-
lized to support the most appropriate analysis,
given the data and its properties, so that defensible
and reproducible interpretations and inferences
result. Thus, decisions that are made based on
the analysis of data, whether “big” or “small,”
are inherently dependent on the quality of the
analysis and associated interpretations.
Conclusion
As has been the case for centuries, today’s “big”
data will eventually be perceived as “small”; how-
ever, the statistical methodologies for analyzing
and interpreting all data will also continue to
evolve, and these will become increasingly
interdependent on the methods for collecting,
manipulating, and storing the data. Because of
the constant evolution and advancement in tech-
nology and computation, the notion of “big data”
may be best conceptualized as representing the
processes of data collection, storage, and manip-
ulation for interpretable analysis, and not the size,
utility, or complexity of the data itself. Therefore,
the characterization of data as “small” depends
critically on the context and use for which the
data are intended.
Further Reading
Bickel, P. J. (2000). Statistics as the information science.
Opportunities for the mathematical sciences, 9, 11.
Day, J., & Ghemawat, S (2004, December). MapReduce:
Simplified data processing on large clusters. In
OSDI’04: Sixth symposium on operating system design
and implementation. San Francisco. Downloaded from
https://research.google.com/archive/mapreduce.html on
21 Dec 2016.
Rao, C. R. (2001). Statistics: Reflections on the past and
visions for the future. Communications in Statistics –
Theory and Methods, 30(11), 2235–2257.
T
Time Series Analytics
Erik Goepner
George Mason University, Arlington, VA, USA
Synonyms
Time series analysis, Time series data
Introduction
Time series analytics utilize data observations
recorded over time at certain intervals. Subse-
quent values of time-ordered data often depend
on previous observations. Time series analytics is,
therefore, interested in techniques that can ana-
lyze this dependence (Box et al. 2015; Zois et al.
2015). Up until the second half of the twentieth
century, social scientists largely ignored the pos-
sibility of dependence within time series data
(Kirchgässner et al. 2012). Statisticians have
since demonstrated that adjacent observations
are frequently dependent in a time series and that
previous observations can often be used to accu-
rately predict future values (Box et al. 2015).
Time series data abound and are of importance
to many. Physicists and geologists investigating
climate change, for example, use annual tempera-
ture readings, economists study quarterly gross
domestic product and monthly employment
reports, and policy makers might be interested in
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_469-1
before and after annual traffic accident data to
determine the efficacy of safety legislation. Time
series analytics can be used to forecast, determine
the transfer function, assess the effects of unusual
intervention events, analyze the relationships
between variables of interest, and design control
schemes (Box et al. 2015). Preferably, observa-
tions have been recorded at fixed time intervals. If
the time intervals vary, interpolation can be used
to fill in the gaps (Zois et al. 2015).
Of critical importance is whether the variables
are stationary or nonstationary. Stationary vari-
ables are not time dependent (i.e., mean, variance,
and covariance remain constant over time). How-
ever, time series data are quite often non-
stationary. The trend of nonstationary variables
can be deterministic (e.g., following a time
trend), stochastic (i.e., random), or both.
Addressing nonstationarity is a key requirement
for those working with time series and is
discussed further under “Challenges” (Box et al.
2015; Kirchgässner et al. 2012).
Time series are frequently comprised of four
components. There is the trend over the long-term
and, often, a cyclical component that is normally
understood to be a year or more in length. Within
the cycle, there can be a seasonal variation. And
finally, there is the residual which includes all
variation not explained by the trend, cycle, and
seasonal components. Prior to the 1970s, only the
residual was thought to include random impact,
with trend, cycle, and seasonal change understood
to be deterministic. That has changed, and now it
2 Time Series Analytics
is assumed that all four components can be sto-
chastically modeled (Kirchgässner et al. 2012).
The Evolution of Time Series Analytics
In the first half of the 1900s, fundamentally dif-
ferent approaches were pursued by different dis-
ciplines. Natural scientists, mathematicians, and
statisticians generally modeled the past history of
the variable of interest to forecast future values of
the variable. Economists and other social scien-
tists, however, emphasized theory-driven models
with their accompanying explanatory variables. In
1970, Box and Jenkins published an influential
textbook, followed in 1974 by a study from
Granger and Newbold, that has substantially
altered how social scientists interact with time
series data (Kirchgässner et al. 2012).
The Box Jenkins approach, as it has been fre-
quently called ever since, relies on extrapolation.
Box Jenkins focuses on the past behavior of the
variable of interest rather than a host of explana-
tory variables to predict future values. The vari-
able of interest must be transformed so that it
becomes stationary and its stochastic properties
time invariant. At times, the terms Box Jenkins
approach and time series analysis have been used
interchangeably (Kennedy 2008).
Time Series Analytics and Big Data
Big Data has stimulated interest in efficient que-
rying of time series data. Both time series and Big
Data share similar characteristics relating to vol-
ume, velocity, variety, veracity, and volatility
(Zois et al. 2015). The unprecedented volume of
data can overwhelm computer memory and pre-
vent processing in real time. Additionally, the
speed at which new data arrives (e.g., from sen-
sors) has also increased. The variety of data
includes the medium from which it comes (e.g.,
audio and video) as well as differing sampling
rates, which can prove problematic for data anal-
ysis. Missing data and incompatible sampling
rates are discussed further in the “Challenges”
section below. Veracity includes issues relating
to inaccurate, missing, or incomplete data. Before
analysis, these issues should be addressed via
duplicate elimination, interpolation, data fusion,
or an influence model (Zois et al. 2015).
Contending with Massive Amounts of Data
Tremendous amounts of time series data exist,
potentially overwhelming computer memory. In
response, solutions are needed to lessen the effects
on secondary memory access. Sliding windows
and time series indexing may help. Both are com-
monly used; however, newer users may find the
learning curve unhelpfully steep for time series
indexing. Similarly, consideration should be
given to selecting management schemes and
query languages simple enough for common
users (Zois et al. 2015).
Analysis and Forecasting
Time series are primarily used for analysis and
forecasting (Zois et al. 2015). A variety of poten-
tial models exist, including autoregressive (AR),
moving average (MA), mixed autoregressive
moving average (ARMA), and autoregressive
integrated moving average (ARIMA). ARMA
models are used with stationary processes and
ARIMA models for nonstationary ones (Box et
al. 2015). Forecasting options include regression
and nonregression based models. Model develop-
ment should follow an iterative approach, often
executed in three steps: identification, estimation,
and diagnostic checking. Diagnostic checks
examine whether the model is properly fit, and
the checks analyze the residuals to determine
model adequacy. Generally, 100 or more observa-
tions are preferred. If fewer than 50 observations
exist, development of the initial model will
require a combination of experience and past
data (Box et al. 2015; Kennedy 2008).
Autoregressive, Moving Average, and Mixed
Autoregressive Moving Average Models
An autoregressive model predicts the value of the
variable of interest based on its values from one or
more previous time periods (i.e., its lagged value).
If, for instance, the model only relied on the value
Time Series Analytics
of the immediately preceding time period, then it
would be a first-order autoregression. Similarly, if
the model included the values for the prior two
time periods, then it would be referred to as a
second-order autoregression and so on. A moving
average model also uses lagged values, but of the
error term rather than the variable of interest (Ken-
nedy 2008). If neither an autoregressive nor mov-
ing average process succeeds in breaking off the
autocorrelation function, then a mixed auto-
regressive moving average approach may be pre-
ferred (Kirchgässner et al. 2012). AR, MA, and
ARMA models are used with stationary time
series, to include time series made stationary
through differencing. However, the potential loss
of vital information during differencing opera-
tions should be considered (Kirchgässner et al.
2012).
ARMA models produce unconditional fore-
casts, using only the past and current values of
the variable. Because such forecasts frequently
perform better than traditional econometric
models, they are often preferred. However,
blended approaches, which transform linear
dynamic simultaneous equation systems into
ARMA models or the inverse, are also available.
These blended approaches can retain information
provided by explanatory variables (Kirchgässner
et al. 2012).
Autoregressive Integrated Moving Average
(ARIMA) Models
In ARIMA models, also known as ARIMA (p,d,
q), p indicates the number of lagged values of Y*,
which represents the variable of interest after it
has been made stationary by differencing. d indi-
cates the number of differencing operations
required to transform Y into its stationary version,
Y*. The number of lagged values of the error term
is represented by q. ARIMA models can forecast
for univariate and multivariate time series (Ken-
nedy 2008).
Vector Autoregressive (VAR) Models
VAR models blend the Box Jenkins approach with
traditional econometric models. They can be quite
helpful in forecasting. VAR models express a
single vector (of all the variables) as a linear
3
function of the vector’s lagged values combined
with an error vector. The single vector is derived
from the linear function of each variable’s lagged
values and the lagged values for each of the other
variables. VAR models are used to investigate the
potential causal relationship between different
time series, yet they are controversial because
they are atheoretical and include dubious asser-
tions (e.g., orthogonal innovation of one variable
is assumed to not affect the value of any other
variable). Despite the controversy, many scholars
and practitioners view VAR models as helpful,
particularly VAR’s role in analysis and forecasting
(Kennedy 2008; Kirchgässner et al. 2012; Box et
al. 2015).
Error Correction Models
These models attempt to harness positive features
of both ARIMA and VAR models, accounting for
the dynamic feature of time series data while also
taking advantage of the contributions explanatory
variables can make. Error correction models add
theory-driven exogenous variables to a general
form of the VAR model (Kennedy 2008).
Challenges
Nonstationarity
Nonstationarity can be caused by deterministic
and stochastic trends (Kirchgässner et al. 2012).
To transform nonstationary processes into station-
ary ones, the deterministic and/or stochastic
trends must be eliminated. Measures to accom-
plish this include differencing operations and
regression on a time trend. However, not all non-
stationary processes can be transformed
(Kirchgässner et al. 2012).
The Box Jenkins approach assumes that
differencing operations will make nonstationary
variables stationary. A number of unit root tests
have been developed to test for nonstationarity,
but their lack of power remains an issue. Addi-
tionally, differencing (as a means of eliminating
unit roots and creating stationarity) comes with
the undesirable effect of eliminating any theory-
driven information that might otherwise contrib-
ute to the model.
4 Time Series Analytics
Granger and colleagues developed
cointegrated procedures to address this challenge
(Kirchgässner et al. 2012). When nonstationary
variables are cointegrated, that is, the variables
remain relatively close to each other as they wan-
der over time, procedures other than differencing
can be used. Examples of cointegrated variables
include prices and wages and short- and long-term
interest rates. Error correcting models may be an
appropriate substitute for differencing operations
(Kennedy 2008). Cointegration analysis has
helped shrink the gap between traditional econo-
metric methods and time series analytics, facili-
tating the inclusion of theory-driven explanatory
variables into the modeling process (Kirchgässner
et al. 2012).
Autocorrelation
Time series data are frequently autocorrelated
and, therefore, violate the assumption of ran-
domly distributed error terms. When autocorrela-
tion is present, the current value of a variable
serves as a good predictor of its next value. Auto-
correlation can disrupt models such that the anal-
ysis incorrectly concludes the variable is
statistically significant when, in fact, it is not
(Berman and Wang 2012). Autocorrelation can
be detected visually or with statistical techniques
like the Durbin-Watson test. If present, autocorre-
lation can be corrected with differencing or by
adding a trend variable, for instance (Berman
and Wang 2012).
Missing Data and Incompatible Sampling
Rates
Missing data occur for any number of reasons.
Records may be lost, destroyed, or otherwise
unavailable. At certain points, sampling rates
may fail to follow the standard time measurement
of the data series. Specialized algorithms may be
necessary. Interpolation can be used as a tech-
nique to fill in missing data or to smooth the
gaps between intervals (Zois et al. 2015).
Conclusion
Time series analytics utilizes data observations
recorded over time at certain intervals, observa-
tions which often depend on each other. Time
series analytics focuses on this dependence (Box
et al. 2015; Zois et al. 2015). A variety of models
exist for use in time series analysis (e.g., ARMA,
ARIMA, VAR, and ECM). Of critical importance
is whether the variables are stationary or non-
stationary. Stationary variables are not time
dependent (i.e., mean, variance, and covariance
remain constant over time). However, time series
data are quite often nonstationary. Addressing
nonstationarity is a key requirement for users of
time series (Box et al. 2015; Kirchgässner et al.
2012).
Cross-References
▶ Core Curriculum Issues (Big Data Research/
Analysis)
▶ Real-Time Analytics
▶ Spatio-Temporal Analytics
▶ Statistical Analysis
Further Readings
Berman, E., & Wang, X. (2012). Essential statistics for
public managers and policy analysts (3rd ed.). Los
Angeles: CQ Press.
Box, G., Jenkins, G., Reinsel, G., & Ljung, G. (2015). Time
series analysis: Forecasting and control. Hoboken:
Wiley.
Kennedy, P. (2008). A guide to econometrics (6th ed.).
Malden: Blackwell.
Kirchgässner, G., Wolters, J., & Hassler, U. (2012). Intro-
duction to modern time series analysis (2nd ed.). Hei-
delberg: Springer Science & Business Media.
Zois, V., Chelmis, C., & Prasanna, V. (2015). Querying of
time series for big data analytics. In L. Yan (Ed.),
Handbook of research on innovative database query
processing techniques (pp. 364–391). Hershey: IGI
Global.
W
Web Scraping
Bo Zhao
College of Earth, Ocean, and Atmospheric
Sciences, Oregon State University, Corvallis, OR,
USA
Web scraping, also known as web extraction or
harvesting, is a technique to extract data from the
World Wide Web (WWW) and save it to a file
system or database for later retrieval or analysis.
Commonly, web data is scrapped utilizing Hyper-
text Transfer Protocol (HTTP) or through a web
browser. This is accomplished either manually by
a user or automatically by a bot or web crawler.
Due to the fact that an enormous amount of het-
erogeneous data is constantly generated on the
WWW, web scraping is widely acknowledged as
an efficient and powerful technique for collecting
big data (Mooney et al. 2015; Bar-Ilan 2001). To
adapt to a variety of scenarios, current web scrap-
ing techniques have become customized from
smaller ad hoc, human-aided procedures to the
utilization of fully automated systems that are
able to convert entire websites into well-organized
data set. State-of-the-art web scraping tools are
not only capable of parsing markup languages or
JSON files but also integrating with computer
visual analytics (Butler 2007) and natural lan-
guage processing to simulate how human users
browse web content (Yi et al. 2003).
# Springer International Publishing AG (outside the USA) 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_483-1
The process of scraping data from the Internet
can be divided into two sequential steps; acquiring
web resources and then extracting desired infor-
mation from the acquired data. Specifically, a web
scraping program starts by composing a HTTP
request to acquire resources from a targeted
website. This request can be formatted in either a
URL containing a GET query or a piece of HTTP
message containing a POST query. Once the
request is successfully received and processed by
the targeted website, the requested resource will
be retrieved from the website and then sent back to
the give web scraping program. The resource can
be in multiple formats, such as web pages that are
built from HTML, data feeds in XML or JSON
format, or multimedia data such as images, audio,
or video files. After the web data is downloaded,
the extraction process continues to parse,
reformat, and organize the data in a structured
way. There are two essential modules of a web
scraping program – a module for composing
an HTTP request, such as Urllib2 or selenium
and another one for parsing and extracting infor-
mation from raw HTML code, such as Beautiful
Soup or Pyquery. Here, the Urllib2 module
defines a set of functions to dealing with HTTP
requests, such as authentication, redirections,
cookies, and so on, while Selenium is a web
browser wrapper that builds up a web browser,
such as Google Chrome or Internet Explorer, and
enables users to automate the process of browsing
a website by programming. Regarding data
extraction, Beautiful Soup is designed for
2 Web Scraping
scraping HTML and other XML documents. It
provides convenient Pythonic functions for navi-
gating, searching, and modifying a parse tree; a
toolkit for decomposing an HTML file and extra-
cting desired information via lxml or html5lib.
Beautiful Soup can automatically detect the
encoding of the parsing under processing and
convert it to a client-readable encode. Similarly,
Pyquery provides a set of Jquery-like functions to
parse xml documents. But unlike Beautiful Soup,
Pyquery only supports lxml for fast XML
processing.
Of the various types of web scraping programs,
some are created to automatically recognize the
data structure of a page, such as Nutch or Scrapy,
or to provide a web-based graphic interface that
eliminates the need for manually written web
scraping code, such as Import.io. Nutch is a robust
and scalable web crawler, written in Java. It
enables fine-grained configuration, paralleling
harvesting, robots.txt rule support, and machine
learning. Scrapy, written in Python, is an reusable
web crawling framework. It speeds up the process
of building and scaling large crawling projects. In
addition, it also provides a web-based shell to
simulate the website browsing behaviors of a
human user. To enable nonprogrammers to har-
vest web contents, the web-based crawler with a
graphic interface is purposely designed to mitigate
the complexity of using a web scraping program.
Among them, Import.io is a typical crawler for
extracting data from websites without writing any
code. It allows users to identify and convert
unstructured web pages into a structured format.
Import.io’s graphic interface for data identifica-
tion allows user to train and learn what to extract.
The extracted data is then stored in a dedicated
cloud server, and can be exported in CSV, JSON,
and XML format. A web-based crawler with a
graphic interface can easily harvest and visualize
real-time data stream based on SVG or WebGL
engine but fall short in manipulating a large data
set.
Web scraping can be used for a wide variety of
scenarios, such as contact scraping, price change
monitoring/comparison, product review collec-
tion, gathering of real estate listings, weather
data monitoring, website change detection, and
web data integration. For examples, at a micro-
scale, the price of a stock can be regularly scraped
in order to visualize the price change over time
(Case et al. 2005), and social media feeds can be
collectively scraped to investigate public opinions
and identify opinion leaders (Liu and Zhao 2016).
At a macro-level, the metadata of nearly every
website is constantly scraped to build up Internet
search engines, such as Google Search or Bing
Search (Snyder 2003).
Although web scraping is a powerful technique
in collecting large data sets, it is controversial and
may raise legal questions related to copyright
(O’Reilly 2006), terms of service (ToS) (Fisher
et al. 2010), and “trespass to chattels” (Hirschey
2014). A web scraper is free to copy a piece of data
in figure or table form from a web page without
any copyright infringement because it is difficult
to prove a copyright over such data since only a
specific arrangement or a particular selection of
the data is legally protected. Regarding the ToS,
although most web applications include some
form of ToS agreement, their enforceability usu-
ally lies within a gray area. For instance, the
owner of a web scraper that violates the ToS
may argue that he or she never saw or officially
agreed to the ToS. Moreover, if a web scraper
sends data acquiring requests too frequently, this
is functionally equivalent to a denial-of-service
attack, in which the web scraper owner may be
refused entry and may be liable for damages under
the law of “trespass to chattels,” because the
owner of the web application has a property inter-
est in the physical web server which hosts the
application. An ethical web scraping tool will
avoid this issue by maintaining a reasonable
requesting frequency.
A web application may adopt one of the fol-
lowing measures to stop or interfere with a web
scrapping tool that collects data from the given
website. Those measures may identify whether an
operation was conducted by a human being or a
bot. Some of the major measures include the fol-
lowing: HTML “fingerprinting” that investigates
the HTML headers to identify whether a visitor is
malicious or safe (Acar et al. 2013); IP reputation
determination, where IP addresses with a
recorded history of use in website assaults that
Web Scraping
will be treated with suspicion and are more likely
to be heavily scrutinized (Sadan and Schwartz
2012); behavior analysis for revealing abnormal
behavioral patterns, such as placing a suspiciously
high rate of requests and adhering to anomalous
browsing patterns; and progressive challenges
that filter out bots with a set of tasks, such as
cookie support, JavaScript execution, and
CAPTCHA (Doran and Gokhale 2011).
Further Readings
Acar, G., Juarez, M., Nikiforakis, N., Diaz, C., Gürses, S.,
Piessens, F., & Preneel, B. (2013). Fpdetective: Dusting
the web for fingerprinters. In Proceedings of the 2013
ACM SIGSAC conference on computer & communica-
tions security. New York: ACM.
Bar-Ilan, J. (2001). Data collection methods on the web for
infometric purposes – A review and analysis.
Scientometrics, 50(1), 7–32.
Butler, J. (2007). Visual web page analytics. Google
Patents.
Case, K. E., Quigley, J. M., & Shiller, R. J. (2005). Com-
paring wealth effects: The stock market versus the
housing market. The BE Journal of Macroeconomics,
5(1), 1.
Doran, D., & Gokhale, S. S. (2011). Web robot detection
techniques: Overview and limitations. Data Mining
and Knowledge Discovery, 22(1), 183–210.
3
Fisher, D., Mcdonald, D. W., Brooks, A. L., & Churchill,
E. F. (2010). Terms of service, ethics, and bias: Tapping
the social web for CSCW research. Computer
Supported Cooperative Work (CSCW), Panel
discussion.
Hirschey, J. K. (2014). Symbiotic relationships: Pragmatic
acceptance of data scraping. Berkeley Technology Law
Journal, 29, 897.
Liu, J. C.-E., & Zhao, B. (2016). Who speaks for climate
change in China? Evidence from Weibo. Climatic
Change, 140(3), 413–422.
Mooney, S. J., Westreich, D. J., & El-Sayed, A. M. (2015).
Epidemiology in the era of big data. Epidemiology,
26(3), 390.
O’Reilly, S. (2006). Nominative fair use and Internet
aggregators: Copyright and trademark challenges
posed by bots, web crawlers and screen-scraping tech-
nologies. Loyola Consumer Law Review, 19, 273.
Sadan, Z., & Schwartz, D. G. (2012). Social network
analysis for cluster-based IP spam reputation. Informa-
tion Management & Computer Security, 20(4),
281–295.
Snyder, R. (2003). Web search engine with graphic snap-
shots. Google Patents.
Yi, J., Nasukawa, T., Bunescu, R., & Niblack, W. (2003).
Sentiment analyzer: Extracting sentiments about a
given topic using natural language processing tech-
niques. Data Mining, 2003. ICDM 2003. Third IEEE
International Conference on, IEEE. Melbourne,
Florida, USA.
B

Big Geo-Data (uncertainty) of data, and the complex

interlinkages with (small) datasets that cover mul-
Song Gao tiple perspectives, topics, and spatiotemporal
Department of Geography, University of scales. It poses grand research challenges during
California, Santa Barbara, CA, USA the life cycle of large-scale georeferenced data
collection, access, storage, management, analysis,
modeling, and visualization.
Synonyms

Big georeferenced data; Big geospatial data;

Theoretical Aspects
Geospatial big data; Spatial big data
Geography has a long-standing tradition on
the duality of research methodologies: the law-
Definition/Introduction
seeking approach and the descriptive or explana-
tory approach. With the increasing popularity of
Big geo-data is an extension to the concept of
data-driven approaches in geography, a variety of
big data with emphasis on the geospatial compo-
statistical methods and machine learning methods
nent and under the context of geography or
have been applied in geospatial knowledge dis-
geosciences. It is used to describe the phenome-
covery and modeling for predictions. Miller and
non that large volumes of georeferenced data
Goodchild (2015) discussed the major challenges
(including structured, semi-structured, and
(i.e., populations not samples, messy not clean
unstructured data) about various aspects of the
data, and correlations not causality) and the
Earth environment and society are captured by
role of theory in the data-driven geographic
millions of environmental and human sensors
knowledge discovery and spatial modeling, with
in a variety of formats such as remote sensing
addressing the tensions between idiographic ver-
imageries, crowdsourced maps, geotagged videos
sus nomothetic knowledge in geography. Big geo-
and photos, transportation smart card transactions,
data is leading to new approaches to research
mobile phone data, location-based social media
methodologies in capturing complex spatiotem-
content, and GPS trajectories. Big geo-data is
poral dynamics of the Earth and the society
“big” not only because it involves a huge volume
directly at multiple spatial and temporal scales
of georeferenced data but also because of the high
instead of just snapshots. The data streams play a
velocity of generation streams, high dimensional-
driving-force role in data-driven methods rather
ity, high variety of data forms, the veracity
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_492-1
2 Big Geo-Data
than a test or calibration role behind the theory or
models in conventional geographic analyses.
While data-driven science and predictive ana-
lytics evolve in geographic and provide new
insights, sometimes it is still very challenging for
humans to interpret the meanings of machine
learning or analytical results or relate findings to
underlying theory. To solve this problem,
Janowicz et al. (2015) proposed a semantic cube
to illustrate the need for semantic technologies
and domain ontologies to address the role of
diversity, synthesis, and definiteness in big data
researches.
Social and Human Aspects
The emergence of big geo-data brings new oppor-
tunities for researchers to understand our socio-
economic and human environments. In the journal
of Dialogues in Human Geography (volume 3,
Issue 3, November 2013), several human geogra-
phers and GIScience researchers discussed a
series of theoretical and practical challenges and
risks to geographic scholarship and raised a num-
ber of epistemological, methodological, and ethi-
cal questions related to the studies of big data in
geography. With the advancements in location-
awareness technology, information and commu-
nication technology, and mobile sensing technol-
ogy, researchers employed emerging big geo-data
for investigating the geographical perspective of
human dynamics research within such contexts in
the special issue on Human Dynamics in the
Mobile and Big Data Era on the International
Journal of Geographical Information Science
(Shaw et al. 2016). By synthesizing multi-sources
of big data, those researches can uncover interest-
ing human behavioral patterns that are difficult or
impossible to uncover with the traditional
datasets. However, challenges still exist in the
scarcity of demographics and cross-validation or
getting the identity of individual behaviors rather
than aggregated patterns. Moreover, the location-
privacy concerns and discussions arise in both
academic world and the society. There exist social
tensions among big data accessibility and privacy
protection.
Technical Aspects
Cloud computing technologies and their distrib-
uted deployment models offer scalable computing
paradigms to enable big geo-data processing
for scientific researches and applications. In the
geospatial research world, cloud computing has
attracted increasing attention as a way of solving
data-intensive, computing-intensive, and access-
intensive geospatial problems and challenges,
such as supporting climate analytics, land-use
and land-cover change analysis, and dust storm
forecasting (Yang et al. 2017). Geocomputation
facilitates fundamental geographical science stud-
ies by synthesizing high-performance computing
capabilities with spatial analysis operations, with
providing a promising solution to aforementioned
geospatial research challenges.
There are a variety of big data analytics plat-
forms and parallelized database systems emerging
in the new era. They can be classified into two
categories: (1) the massively parallel processing
data warehousing systems like Teradata which
are designed for holding large-scale structured
data and support standard SQL queries and
(2) the distributed file storage systems and
cluster-computing framework like Apache
Hadoop and Apache Spark. The advantages of
Hadoop-based systems mainly lie in their high
flexibility, scalability, low cost, and reliability for
managing and efficiently processing a large volume
of structured and unstructured datasets, as well as
providing job schedules for balancing data,
resources, and task loads. A MapReduce computa-
tion paradigm on Hadoop takes the advantages
of divide-and-conquer strategy and improves the
processing efficiency. However, big geo-data has
its complexity on the spatial and temporal compo-
nents and requires new analytical framework
and functionalities compared with nonspatial big
data. Gao et al. (2017) built a scalable Hadoop-
based geoprocessing platform (GPHadoop) and
ran big geo-data analytical functions to solve
crowdsourced gazetteers harvesting problems.
Recently, more efforts have been made in
connecting traditional GIS analysis research com-
munity to the cloud computing research commu-
nity for the next frontier of big geo-data analytics.
Big Geo-Data
In one special issue on big data at the journal
of Annals of GIS (volume 20, Issue 4, 2014),
researchers further discussed several key techno-
logies (e.g., cloud computing, high-performance
geocomputation cyberinfrastructures) for dealing
with quantitative and qualitative dynamics of big
geo-data. Advanced spatiotemporal big data mining
and geoprocessing methods should be developed
by optimizing the elastic storage, balanced sched-
uling, and parallel computing resources in high-
performance geocomputation cyberinfrastructures.
Conclusion
With the advancements in location-awareness
technology and mobile distributed sensor net-
works, large-scale high-resolution spatiotemporal
datasets about the Earth and the society become
available for geographic research. The research on
big geo-data involves interdisciplinary collabora-
tive efforts. There are at least three research areas
that require further work: (1) the systematic inte-
gration of various big geo-data sources in
geospatial knowledge discovery and spatial
3
modeling, (2) the development of advanced
spatial analysis functions and models, and
(3) the advancement of quality assurance issues
on big geo-data. Finally, there will still be ongoing
comparisons between data-driven and theory-
driven research methodologies in geography.
Further Readings
Gao, S., Li, L., Li, W., Janowicz, K., & Zhang, Y. (2017).
Constructing gazetteers from volunteered big geo-data
based on Hadoop. Computers, Environment and Urban
Systems, 61, 172–186.
Janowicz, K., van Harmelen, F., Hendler, J., & Hitzler,
P. (2015). Why the data train needs semantic rails. AI
Magazine, Association for the Advancement of Artifi-
cial Intelligence (AAAI), pp. 5–14.
Miller, H. J., & Goodchild, M. F. (2015). Data-driven
geography. Geo Journal, 80(4), 449–461.
Shaw, S. L., Tsou, M. H., & Ye, X. (2016). Editorial:
Human dynamics in the mobile and big data era. Inter-
national Journal of Geographical Information Science,
30(9), 1687–1693.
Yang, C., Huang, Q., Li, Z., Liu, K., & Hu, F. (2017). Big
data and cloud computing: Innovation opportunities
and challenges. International Journal of Digital
Earth, 10(1), 13–53.
I
Integrated Data System
Ting Zhang
Department of Finance and Economics, Merrick
School of Business, University of Baltimore,
Baltimore, MD, USA
Definition/Introduction
Integrated Data Systems (IDS) typically link indi-
vidual level administrative records collected by
multiple agencies such as k–12 schools, commu-
nity colleges, other colleges and universities,
departments of labor, justice, human resources,
human and health services, police, housing, and
community services. The systems can be used for
quick knowledge-to-practice development cycle
(Actionable Intelligence for Social Policy 2017),
case management, program or service monitoring,
tracking, and evaluation (National Neighborhood
Indicators Partnership 2017), research and policy
analysis, strategic planning and performance
management, and so on. It can also help evaluate
how different programs, services, and policies
affect individual persons or individual geographic
units. The linkages between different agency
records are often made through a common indi-
vidual personal identification number, a shared
case number, or a geographic unit.
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_494-1
Purpose of an IDS
With the rising attraction of big data and the
exploding need to share existing data, the need
to link already collected various administrative
records rises. The systems allow government
agencies to integrate various databases and bridge
the gaps that have traditionally formed within
individual agency databases; it can be used for
quick knowledge-to-practice development cycle
to better address the often interconnected citizens’
needs efficiently and effectively (Actionable Intel-
ligence for Social Policy 2017), for case manage-
ment (National Neighborhood Indicators
Partnership 2017), program or service monitor-
ing, tracking, and evaluation, developing and test-
ing an intervention and monitoring the outcomes
(Davis et al. 2014), research and policy analysis,
strategic planning and performance management,
and so on. It can test social policy innovations
through high-speed, low-cost randomized control
trials and quasi-experimental approaches, can be
used for continuous quality improvement efforts
and benefit cost analysis, and can also help pro-
vide a complete account of how different pro-
grams, services, and policies affect individual
persons or individual geographic units to more
efficiently and effectively address the often
interconnected needs of the citizens (Actionable
Intelligence for Social Policy 2017).
2 Integrated Data System
Key Elements to Build an IDS
According to Davis et al. (2014) and Zhang and
Stevens (2012), typical crucial factors related to a
successful IDS include:
• A broad and steady institutional commitment
to administrate the system
• Individual-level data (no matter individual per-
sons or individual geographic units) to mea-
sure outcomes
• The necessary data infrastructure
• Linkable data fields, such as Social Security
numbers, business identifiers, shared case
number, and addresses
• The capacity to match various administrative
records
• A favorable state interpretation of the data
privacy requirements, consistent with federal
regulations
• The funding, knowhow, and analytical capac-
ity to work with and maintain the data
• Successfully obtaining participation from mul-
tiple data providing agencies with clearance to
use those data.
Maintenance
Administrative data records are typically col-
lected by public and private agencies. An IDS
often requires to extract, transform, clean, and
link information from various source administra-
tive databases and load it into a data warehouse.
Many data warehouses offer a tightly coupled
architecture that it usually takes little time to
resolve queries and extract information (Widom
1995).
Challenges
Identity Management and Data Quality
One challenge to build an IDS is to have effective
and appropriate individual record identity man-
agement diagnostics that include consideration
of the consequences of gaps in common identifier
availability and accuracy. This is the first key step
for data quality of IDS information. However,
some of the relevant databases, particularly stu-
dent records. do not include a universally linkable
personal identifier, that is, a Social Security num-
ber; some databases are unable to ensure that a
known to be valid Social Security number is
paired with one individual, and only that individ-
ual, consistently over time; and some databases
are unable to ensure that each individual is asso-
ciated with only one Social Security number over
time (Zhang and Stevens 2012). Zhang and Ste-
vens (2012) included ongoing collection of case
studies documenting how SSNs can be extracted,
validated, and securely stored offline. With the
established algorithms required for electronic
financial transactions, spreading adoption of elec-
tronic medical records and rising interest in big
data, there is an extensive, and rapidly growing,
literature illustrating probabilistic matching solu-
tions and various software designs to address the
identity management challenge. Often the
required accuracy threshold is application spe-
cific; assurance of an exact match may not be
required for some anticipated longitudinal data
system uses (Zhang and Stevens 2012).
Data Privacy
To build and use an IDS, issues related to privacy
of personal information within the system is
important. Many government agencies have rele-
vant regulations. For example, a nationally wide-
known law is the Family Educational Rights and
Privacy Act (FERPA) that defines when student
information can be disclosed and data privacy
practices (U.S. Department of Education 2017).
Similarly Health Insurance Portability and
Accountability Act of 1996 (HIPAA) addresses
the use and disclosure of health information (U.S.
Department of Health & Human Services 2017).
Ethics
Most IDS taps individual person’s information.
When using IDS information, in order not to mis-
use personal information, extra caution is needed.
Institutional review boards are often needed when
conducting research involving human subjects.
Integrated Data System
Data Sharing
To build an IDS, a favorable state interpretation of
the data privacy requirements, consistent with
federal regulations and clearance to use the data
for the IDS, is critical. For example, some state
education agencies have been reluctant to share
their education records, largely due to narrow
state interpretations of the confidentiality provi-
sions of FERPA and its implementing regulations
(Davis et al. 2014). Corresponding data sharing
agreements need to be in place.
Data Security
During the process of building, transferring,
maintaining, and using IDS information, the data
security issue in an IDS center is particularly
important. Measures to ensure data security and
information privacy and confidentiality becomes
the key factors for an IDS’ vigor and sustainabil-
ity. Fortunately, many of the US current IDS cen-
ters have had experience maintaining confidential
administrative records for years or even decades.
However, facing the convenience of web access to
maintain the continued data security and sustain-
ability often requires updated data protection tech-
nics. The federal, state, and local government has
important roles in safeguarding data and data use.
Examples
Example of IDS in the United States include:
Chapin Hall’s Planning for Human Service
Reform Using Integrated Administrative Data
Jacob France Institute’s database for education,
employment, human resources, and human
services
Juvenile Justice and Child Welfare Data Cross-
over Youth Multi-Site Research Study
Actionable Intelligence for Social Policy’s inte-
grated data systems initiatives for policy anal-
ysis and program reform
Florida’s Common Education Data Standards
(CEDS) Workforce Workgroup and the later
Florida Education & Training Placement Infor-
mation Program
3
Louisiana Workforce Longitudinal Data System
(WLDS) housed at the Louisiana Workforce
Commission
Minnesota’s iSEEK data. managed by an organi-
zation called iSEEK Solutions
Heldrich Center data at Rutgers University
Ohio State University’s workforce longitudinal
administrative database
University of Texas Ray Marshall Center database
Virginia Longitudinal Data System
Washington’s Career Bridge, managed by the
Workforce Training and Education Coordinat-
ing Board
Connecticut’s Preschool through Twenty and
Workforce Information Network (P-20 WIN)
Delaware Department of Education’s Education
Insight Dashboard
Georgia Department of Education’s Statewide
Longitudinal Data System and Georgia’s Aca-
demic and Workforce Analysis and Research
Data System (GA AWARDS)
Illinois Longitudinal Data System
Indiana Network of Knowledge (INK)
Maryland Longitudinal Data System
Missouri Comprehensive Data System
Ohio Longitudinal Data Archive (OLDA)
South Carolina Longitudinal Information Center
for Education (SLICE)
Texas Public Education Information Resource
(TPEIR) and Texas Education Research Center
(ERC)
Washington P-20W Statewide Longitudinal Data
System
Conclusion
Integrated Data Systems (IDS) typically link indi-
vidual level administrative records collected by
multiple agencies. The systems can be used for
case management, program or service monitoring,
tracking, and evaluation, research and policy anal-
ysis, etc. A successful IDS often requires a broad
and steady institutional commitment to adminis-
trate the system, individual-level data, the neces-
sary data infrastructure, linkable data fields,
capacity and knowhow to match various adminis-
trative records and maintain it, data access
4 Integrated Data System
permission, and data privacy procedures. Main
challenges to build a sustainable IDS include
identity management, data quality, data privacy,
ethics, data sharing, and data security. There are
many IDS in the United States.
Further Reading
Actionable Intelligence for Social Policy. (2017). Inte-
grated Data Systems (IDS). Retrieved in March 2017
from the World Wide Web at https://www.aisp.upenn.
edu/integrated-data-systems/.
Davis, S., Jacobson, L., & Wandner, S. (2014). Using
workforce data quality initiative databases to develop
and improve consumer report card systems.
Washington, DC: Impaq International.
National Neighborhood Indicators Partnership. (2017).
Resources on Integrated Data Systems (IDS), Retrieved
in March 2017 from the World Wide Web at http://www.
neighborhoodindicators.org/resources-integrated-data-
systems-ids.
U.S. Department of Education. (2017). Family Educational
Rights and Privacy Act (FERPA). Retrieved on May
14, 2017 from the World Wide Web https://ed.gov/
policy/gen/guid/fpco/ferpa/index.html.
U.S. Department of Health & Human Services. (2017).
Summary of the HIPAA Security Rule. Retrieved on
May 14, 2017 from the World Wide Web https://www.
hhs.gov/hipaa/for-professionals/security/laws-regula
tions/.
Widom, J. (1995). “Research problems in data
warehousing.” CIKM ’95 Proceedings of the fourth
international conference on information and knowl-
edge management (pp. 25–30). Baltimore.
Zhang, T., & Stevens, D. (2012). Integrated data system
person identification: Accuracy requirements and
methods. Jacob France Institute. Available at SSRN:
https://ssrn.com/abstract=2512590 or http://dx.doi.
org/10.2139/ssrn.2512590 and http://www.
workforcedqc.org/sites/default/files/images/JFI%20wdqi
%20research%20report%20January%202014.pdf.
S
State Longitudinal Data System
Ting Zhang
Department of Finance and Economics, Merrick
School of Business, University of Baltimore,
Baltimore, MD, USA
Definition
State Longitudinal Data Systems (SLDS) connect
databases across two or more of state-level agen-
cies of early learning, K–12, postsecondary, and
workforce. It is a state-level Integrated Data Sys-
tem and focuses on tracking individuals
longitudinally.
Purpose of the SLDS
SLDS are intended to enhance the ability of states
to capture, manage, develop, analyze, and use
student education records, to support evidence-
based decisions to improve student learning, to
facilitate research to increase student achievement
and close achievement gaps (National Center for
Education Statistics 2010), to address potential
recurring impediments to student learning, to
measure and document education long-term
return on investment, to support education
accountability systems, and to simplify the pro-
cesses used by state educational agencies to make
education data transparent through federal and
# Springer International Publishing AG 2017
L.A. Schintler, C.L. McNeely (eds.), Encyclopedia of Big Data,
DOI 10.1007/978-3-319-32001-4_495-1
public reporting (US Department of Education
2015). The Statewide Longitudinal Data Systems
Grant Program funds states’ efforts to develop and
implement these data systems in respond to legis-
lative initiatives (US Department of
Education 2015).
Information Offered
The data system aligns p-12 student education
records with secondary and postsecondary educa-
tion and the workforce records, with linkable stu-
dent and teacher identification numbers and
student and teacher information on student level
(National Center for Education Statistics 2010).
The student education records include informa-
tion on enrollment, demographics, program par-
ticipation, test records, transcript information,
college readiness test scores, successful transition
to postsecondary programs, enrollment in post-
secondary remedial courses, entries, and exits
from various levels of the education system
(National Center for Education Statistics 2010).
Statewide Longitudinal Data Systems
Grant Program
According to US Department of Education
(2015), the Statewide Longitudinal Data Systems
Program awards grants to State educational agen-
cies to design, develop, and implement SLDS to
2 State Longitudinal Data System
efficiently and accurately manage, analyze, disag-
gregate, and use individual student data. As autho-
rized by the Educational Technical Assistance Act
of 2002, Title II of the statute that created the
Institute of Education Sciences (IES), the SLDS
Grant Program has awarded competitive, cooper-
ative agreement grants to almost all states since
2005; in addition to the grants, the program offers
many services and resources to assist education
agencies with SLDS-related work
(US Department of Education 2016).
Challenges
In addition to the challenges an Integrated Data
System has, SLDS has the following main
challenges:
Training/Education Provider Participation
In spite of the recent years’ progress, participation
by training/education providers has not been uni-
versal. To improve the training and education
coverage, a few states have taken effective action.
For example, the Texas state legislature has tied a
portion of the funding of state technical colleges
to their ability to demonstrate high levels of pro-
gram completion and employment in occupations
related to training (Davis et al. 2014).
Privacy Issues and State Longitudinal Data
Systems
To ensure data privacy and protect personal infor-
mation, Family Educational Rights and Privacy
Act (FERPA), the Pupil Protection Rights Act
(PPRA), and Children’s Online Privacy Protec-
tion Act (COPPA) are issued (Parent Coalition
for Student Privacy 2017). However, the related
issues and rights are complex, and the privacy
rights provided by law are often not provided in
practice (National Center for Education Statistics
2010). For a sustained SLDS, a push in the
established privacy rights is important.
FERPA Interpretation
Another challenge is that some state education
agencies have been reluctant to share their educa-
tion records, largely due to narrow state
interpretations of the confidentiality provisions
of FERPA and its implementing regulations
(Davis et al. 2014). Many states have overcome
potential FERPA-related obstacles in their own
unique ways, for example: (1) obtaining legal
advice recognizing that the promulgation of
amended FERPA regulations was intended to
facilitate the use of individual-level data for
research purposes, (2) maintaining the workforce
data within the education state’s agency, and
(3) creating a special agency that holds both the
education and workforce data (Davis et al. 2014).
Maintaining Longitudinal Data
Many state’s SLDS already have linked student
records, but decision making based on a short-
term return on education investment is not neces-
sarily useful; the word “longitudinal” is the key-
stone needed for development of a strong business
case for sustained investment in a SLDS (Stevens
and Zhang 2014). “Longitudinal” means the capa-
bility to link information about individuals across
defined segments and through time. While there is
no evidence that the length of data retention
increases identity disclosure risk, public concern
about data retention is escalating (Stevens and
Zhang 2014).
Examples
Examples of US SLDS include:
Florida Education & Training Placement Informa-
tion Program
Louisiana Workforce Longitudinal Data System
(WLDS)
Minnesota’s iSEEK data.
Heldrich Center data at Rutgers University
Ohio State University’s workforce longitudinal
administrative database
University of Texas Ray Marshall Center
database,
Virginia Longitudinal Data System
Washington’s Career Bridge
Connecticut’s Preschool through Twenty and
Workforce Information Network
Delaware Education Insight Dashboard
State Longitudinal Data System
Georgia Statewide Longitudinal Data System and
Georgia Academic and Workforce Analysis
and Research Data System (GA AWARDS)
Illinois Longitudinal Data System
Indiana Network of Knowledge (INK),
Maryland Longitudinal Data System
Missouri Comprehensive Data System
Ohio Longitudinal Data Archive (OLDA)
South Carolina Longitudinal Information Center
for Education (SLICE)
Texas Public Education Information Resource
(TPEIR) and Texas Education Research Center
(ERC)
Washington P-20W Statewide Longitudinal Data
System.
Conclusion
SLDS connects databases across two or more of
agencies of p-20 and Workforce. It is a US state-
level Integrated Data System and focuses on
tracking individuals longitudinally. SLDS are
intended to enhance the ability of states to capture,
manage, design, develop, analyze, and use student
education records and to support data-driven deci-
sions to improve student learning and to facilitate
research to increase student achievement and
close achievement gaps. The Statewide Longitu-
dinal Data Systems (SLDS) Grant Program funds
states’ efforts to develop and implement these data
systems in respond to legislative initiatives. The
main challenges of SLDS include training/educa-
tion provider participation, privacy issues and
State Longitudinal Data Systems, and FERPA
interpretation, and maintaining longitudinal data.
There are many Nationwide SLDS examples.
3
Cross-References
▶ Integrated Data System
Further Readings
Davis, S., Jacobson, L., & Wandner, S. (2014). Using
workforce data quality initiative databases to develop
and improve consumer report card systems.
Washington, DC: Impaq International.
National Center for Education Statistics. (2010). “Data
stewardship: Managing personally identifiable infor-
mation in student education records.” SLDS technical
brief. Available at http://nces.ed.gov/pubsearch/
pubsinfo.asp?pubid=2011602
Stevens, D., & Zhang, T. (2014). “Toward a business case
for sustained investment in State Longitudinal Data
Systems.” Jacob France Institute. Available at http://
www.jacob-france-institute.org/wp-content/uploads/
JFI-WDQI-Year-Three-Research-Report1.pdf
US Department of Education. (2015). “Applications for
new awards; Statewide Longitudinal Data Systems
Program,” Federal register. Available at https://www.
federalregister.gov/documents/2015/03/12/2015-
05682/applications-for-new-awards-statewide-
longitudinal-data-systems-program
US Department of Education (2016). “Agency information
collection activities; Comment request; State Longitu-
dinal Data System (SLDS) Survey 2017–2019.” Fed-
eral Register. Available at https://www.federalregister.
gov/documents/2016/10/07/2016-24298/agency-
information-collection-activities-comment-request-
state-longitudinal-data-system-slds-survey
Parent Coalition for Student Privacy (2017). Federal Stu-
dent Privacy Rights: FERPA, PPRA AND COPPA,
retrieved on May 14, 2017 from the World Wide Web
https://www.studentprivacymatters.org/ferpa_ppra_
coppa/.