2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)
Big-Data/Analytics Projects Failure: A Literature Review
Gianna Reggio
DIBRIS - University of Genoa
Genoa, Italy
gianna.reggio@unige.it
Abstract—Nowadays, no one doubt that big data and analyt-
ics are fundamental tools for successfully running an enterprise.
However, big data/analytics projects do not seem to be all
sunshine and rainbows, and this is documented by many
sources providing startling figures of failed projects. Motivated
by such figures, we conducted a (grey and scientific) literature
review aimed to answer:
RQ1) Which are the documented cases of failed BD/A projects,
and which are the root causes of their failures?
RQ2) What is assumed useful to reduce the chance of failure
of BD/A projects?
We collected and examined 188 sources. The survey resulted
first in a list of hints for helping avoid the failure of the
BD/A projects (RQ2), and in a list of 21 cases of failed BD/A
projects documented in the literature (RQ1). The analysis of
those failures resulted in confirming the hints of RQ2 and
prompting other relevant ones.
The result of this study will be useful not only to people
developing a BDA project but also to researchers investigating
the challenges proposed by a BDA project to the traditional
methods of software engineering.
Keywords-big data project; data analytics project; big data
project failure; data analytics project failure
I. I NTRODUCTION
Nowadays, no one doubt that big data and analytics are
fundamental tools for successfully running an enterprise,
but big data/analytics projects (shortly BDA projects) do
not seem to be all sunshine and rainbows, as reported
below. [32] (2013) states that “55% of Big Data projects
don’t get completed, and many others fall short of their
objectives”. “through 2017, 60% of big data projects will
fail to go beyond piloting and experimentation, and will be
abandoned” [12] (2015). Later, on Twitter Nick Heudecker
about such statement says “We were too conservative. The
failure rate is closer to 85%. And the problem isn’t tech-
nology” [13]. “However, too many executives have assumed
that because they’ve made such big moves, the main chal-
lenges to becoming analytics-driven are behind them. But
frustrations are beginning to surface; it’s starting to dawn
on company executives that they’ve failed to convert their
analytics pilots into scalable solutions. (A recent McKinsey
survey found that only 8 percent of 1,000 respondents with
analytics initiatives engaged in effective scaling practices.)”
[11] (2018). “the technical factors of a successful analytics
initiative only account for 20-25% of all of the success
978-1-7281-9532-2/20/$31.00 ©2020 IEEE
DOI 10.1109/SEAA51224.2020.00050
Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
Egidio Astesiano
DIBRIS - University of Genoa
Genoa, Italy
astes@unige.it
factors.” [26] (2018). “Through 2020, 80% of AI projects
will remain alchemy, run by wizards whose talents will
not scale in the organization. Through 2022, only 20%
of analytic insights will deliver business outcomes” [35]
(2019).
The figures reported above may be just anecdotal evidence
without any scientific relevance, also because they have been
published in non-scientific venues, and thus cannot be a
definitive answer about the success of big data/analytics
projects. Perhaps, we may get completely different figures
concerning the success rate of projects of such kind.
There are notorious cases of failed BDA projects, such
as the Google Flu project for detecting the outbursting
of a flu epidemy by analysing the search made on the
web of flu-related terms, or the results of the USA 2016
Presidential Election (all the data analysis forecasted the
victory of Hilary Clinton with extremely high percentages)
to report two of them. But again, these may be just anecdotal
evidence, and these failures may be due to reasons not
related to big data/analytics. On the other side, there is a long
list of successful big data analytics projects; for example,
companies such as Netflix and Amazon thrive on big data
/analytics.
Since ”where there is smoke, there is fire,” we decided
to investigate the topic – the failure of Big Data/Analytics
projects – by conducting a literature review.
We use “big data/analytics project” in a quite general
sense encompassing any project involving big data or some
form of data analysis (e.g. business intelligence, data mining,
machine learning, artificial intelligence and deep learning).
Trying to precisely define what big data are and how to
distinguish between the different brands of data analysis is
outside the scope of this paper. That issue is thoroughly
discussed in [1], where it is stated that “there is no single
unified definition of big data.”
Our literature review aims to answer the following re-
search questions:
RQ1: Which are the documented cases of failed BD/A
projects, and which are the root causes of their failures?
RQ2: What is assumed useful to reduce the chance of failure
of BD/A projects?
We decided to consider both scientific and grey literature
because the first is still scarce (most conferences and jour-
246

Figure 1. Search strings and number of selected sources
nals started in the last five years) and to get the view of
the practitioners. We collected and examined 188 sources:
all agree that it is difficult to run successful BD/A projects
and that their failures are not related to technical issues. The
survey resulted in two lists:
i) one containing hints for helping avoid the failure of the
BD/A projects (answer to RQ2),
ii) and another one of 21 cases of failed BD/A projects,
each one documented in at least one source in the literature
(answer to RQ1), sorted by the failures genuinely related to
the management of big data/analytics and by those due to
other reasons (e.g. forgetting the basics of software testing).
The analysis of the cases in ii) resulted in confirming the
relevance of the hints in i) and in prompting other ones.
The result of this study will help people develop a BDA
project and researchers in the field of software engineering
interested in investigating the challenges proposed by a BDA
project to the traditional methods of software engineering.
We detail the sources collection process in Sect. II. Then,
we present and discuss the found hints in Sect. III (RQ2),
Library, Springer Open, Science Direct and ResearchGate1
(a professional network for scientists and researchers), again
considering only papers after 2010. Altogether, we found
3289 different papers; [25] reports the numbers of papers
found with the different searches.
Then, we examined those papers and selected the one
discussing failed BDA projects or presenting suggestions
for avoiding the failure of BDA projects (inclusion criteria).
At the end, we collected 188 papers (see [25] for their
bibliographic references). Fig. 1 also shows the numbers of
papers selected by the various searches (we counted each
paper only the first time it was selected), illustrating how
the scientific literature is still quite limited. The searches
and the selection procedure were performed during January
2020.
In our procedure for collecting sources, neither in defining
the search strings nor in the selection phase, we did not
look for hints for successful BDA projects nor for success
stories. In our opinion, the point of view of analysing the
failure could result in a sharper analysis of the underlying
problems, because we can learn more from a failed case than
from a success story.
Fig. 2 presents the kinds and the years of production of
the selected sources, showing the prominence of the grey
literature. The scientific production started later that the grey
literature, both increased in 2016/17, then the grey literature
seems stable (3 items in January 2002 suggest that 2020 will
reach at least the number of 2019).
All the sources support the view that BDA projects fail
1 https://www.researchgate.net/

and list and analyse the found failed BDA projects in

Sect. IV assessing if they are true failures and if some
lessons may be learned (RQ1). The related work and the
conclusion are in Sect. V and VI, respectively.

II. C OLLECTING THE SOURCES

Big data/analytics is a very new field, and thus the existing
scientific literature is scarce with respect to other fields (for
example, the main conferences and journals have appeared
in the last five years). Thus, we decided to consider also
the grey literature (e.g. technical web sites, online technical
journals, and blogs), to get the view of the practitioners.
We used the search strings reported in Fig. 1. Notice that
many of them are multi-word search strings. Indeed, using,
for example, “analytics” together with “failure” we got a
huge number of results (in some case a five-digit number),
but most of them concern the use of analytics to discover
the failures of something (e.g. a device) or the failure of
analytics algorithms.
First, we searched the web using Google and Google
Scholar (considering only web pages appearing after 2010
and the first ten pages of results, obviously disregarding the
advertised items). Later, we searched the main repositories
of scientific literature, precisely IEEE Explore, ACM Digital Figure 2. Selected Sources: Kinds and Production Dates

247

Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
and that the problems are not in the technicalities of big data
and analytics.
We have then read the selected sources to answer the two
research questions, collecting:
– any reference to cases of failed BDA projects (RQ1), see
Sect. IV,
– any proposed hint for reducing the chance of failure of
BDA projects (RQ2), see Sect. III.
III. W HAT IS CONSIDERED USEFUL TO REDUCE THE
CHANCE OF FAILURE OF BDA PROJECTS
A. Hints for avoiding the failure of BDA projects
To answer RQ2, we examined the selected sources looking
for hints on what to do when putting together a BDA project
to avoid its failure. Since many hints are supported by many
sources, we have also recorded which sources propose each
hint, to discover which are the most agreed upon (in [25]
there is the link to a spreadsheet showing the hints supported
by each of the 188 sources. Hints prompted by a unique
source are not considered here, except one involved in a
well-known failure case.
The discovered hints are presented below in alphabetical
order. In the following subsection, we will briefly discuss
their role in a project and their perceived relevance in the
survey.
• Agile Adhere to the agile approach in the BDA projects,
favouring iteration with short cycles and many interactions
with users to validate the results, and keep the data/analytics
activities synchronized with the business.
• Data Access Make accessible the data needed for a BDA
project.
• Data Culture Ensure a data-driven culture.
“A data-driven culture is a workplace environment that
employs a consistent, repeatable approach to tactical and
strategic decision-making through emphatic and empirical
data proof. Put simply, it is an organization that bases
decisions on data, not gut instinct.” [23]
• Data Governance Establish data governance.
• Data Production Design Whenever possible, design which
data collect and how, as well as which data should be
generated.
“The big data mantra is to store everything . . . Not every
data set is important. Just because you can collect data
doesn’t mean you should. You can’t just collect all the data
and expect someone to figure it out someday” [28]
• Data Quality Ensure the best level of quality of the used
data.
• Data Scientists, Business and IT People Together Bring to-
gether IT, data science, and line of business perspectives to
run a BDA project.
• Data Security Protect the data against internal and external
attacks.
• Deployment Carefully consider the deployment phase.
248
Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
The deployment phase of an analytics task is the set of
activities needed to use the result of the task, e.g., the choice
between on-line and off-line computation, visualization of
results, reporting, modification of some business processes
and related aspects.
• Domain Knowledge Get a good knowledge of the domain.
That is imperative in any project; but in the case of BDA
projects, it is especially important to relate the used data
to the domain to assess their context and their quality while
keeping in mind that the domain may change along the time.
• Executives’ Support Ensure the support of the executives,
and mainly of the top executives.
• Interaction with other Analytics Programs Look for possible
interactions with other analytics-based programs.
• No Data Silos Avoid data silos.
• Planning Carefully plan a BDA project, taking into account
all possible alternatives, the costs, the needed resources, and
all possible risks.
• Possible irrelevance of derived facts Keep in mind that facts
or knowledge derived from data may be useless.
• Problem Identification The problem to solve or the question
to answer by a BDA project must be not only known before
to start but also quite precisely defined and its relevance
assessed by appropriate key indicators).
“Begin your efforts with a defined goal that is measurable
and shared with others. Simply to improve efficiency or
enhance the customer experience are not measurable or
sharable.” [3]
• Short Time To Value Ensure that a BDA project provides
value in a short time.
• Specific Skills Analytics/big data skills must be available.
• Start Small Especially within a large organization, start
with one department or business unit, execute with focus and
resolve, get tangible results, and then use that momentum to
spread to other divisions.
Fig. 3 lists the above hints together with the number of
supporting sources sorted first by type of source (total, grey
and scientific), and then by chronology (before 2016, after
2017 and from 2016-2017).
B. Discussion
Some of the elicited hints apply to any software devel-
opment project, and the software engineering world largely
agrees on their value: Agile, Deployment, Domain Knowledge,
Problem Identification and Planning. They have been incor-
porated in the most known and used software development
process models and methods. The lack of attention to those
issues may be motivated by the fact that the novelty of
big data/analytics and the possibility to get awesome results
from the data resulted in projects organized and managed
(if they were) in an ad hoc way, completely forgetting the
wealth of results of the software engineering community.
However, Agile, Deployment, Domain Knowledge, Problem
Identification and Planning are, and should be, declined
 Figure 3. Hints for avoiding the failure of BDA projects: number of sources sponsoring them and production years
differently in the context of the BDA projects, as supported
by [22].
Surprisingly, at least in our opinion, testing and main-
tenance/evolution are never mentioned as means to reduce
the chance of failure of BDA projects. Now, we cannot
say whether these aspects are managed using traditional
approaches in a successful way or they are just ignored.
We only know that testing in this context poses new hard
challenges and has to be rethought completely, and that some
BDA projects require to answer some questions about data
only once (e.g. to make a strategic decision in a company)
and so maintenance/evolution is not an issue.
Executives’ Support is an essential issue for any project
based on very innovative techniques. In the case of the
BDA projects, it is more relevant and hard to grant since
executives must decide to trust the data more than their gut,
and that may result in self-esteem issues.
Data privacy is not present among the found hints, and
Data Security is one of the less supported hints (3 sources
prompt it, none after 2016). We can interpret this result that
now data privacy and security are standard issues in any
BDA project, and are agreed by anyone.
Now we can consider the hints specifically related to BDA
projects. The major hint emerging from the survey is Problem
Identification (mainly found in the grey literature and strongly
249
Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
prompted also after 2017). Namely, a BDA project should
be motivated by a clear (i.e. neither ambiguous nor generic)
question or problem to solve, and such question/problem
must be valuable for who (organization, company) pays for
the project. E.g., a project motivated by a generic “increasing
the income” or “what the data says about my clients” (which
is not sure will result into something actionable) is not
following that hint.
Possible irrelevance of derived facts (6 supporters, only one
after 2017) suggests being aware that a fact derived by data
may be useless, as in a famous case reported by [19]. A
retailer by mining his stock and purchase data had found
that a particular bottle of wine sold exceptionally well on a
Tuesday, and even more so if it was raining. But so what?
The issue is that shelf-space is pre-assigned and cannot be
increased for this brand for just this one day.
A related hint is Domain Knowledge (moderately sup-
ported, with an increase after 2017): the domain of a BDA
project should be well known and understood. In this case,
it is essential also to know well how it evolves and to keep
it under control during all the life of the project. Good
knowledge of the domain is also relevant for assessing the
quality of the data and for helping to consider also the “small
data (en.wikipedia.org/wiki/Small data).
 The hints Problem Identification, Possible irrelevance of
derived facts and Domain Knowledge may be summarized as
follows: – get a good knowledge of the domain, – define a
relevant problem whose solution has verifiable value, – recap
the available data and those that can be produced (big, small
and thick) and – assess their relationships with the domain.
Data Culture and Executives’ Support (moderately sup-
ported, with an increase after 2017) show that an organi-
zation embracing a BDA project should at any level, and
absolutely at the top management level, agree that decisions
should be based on data.
A group of hints concern directly with the data: – Data
Access, scarcely relevant; – No Data Silos, relevance is de-
creasing along the time (since it is agreed by anyone); –Data
Governance, scarcely relevant; – Data Quality, moderately
relevant and increasing along the time; – Data Production
Design, marginal (and this is surprising in our opinion).
Data Scientists, Business and IT People Together (highly
relevant and increasing along the time), and Specific Skills
(moderately relevant) concern the way a BDA project is
managed, and require to organize the team in a way to have
all the people together and obviously to have experts in data
science.
Start Small (moderately relevant) and Short Time To Value Figure 4. Documented failure cases (the last column is the number of
(scarcely relevant) again concern the BDA project manage- citing sources)
ment and should help successfully introduce the new big customers and the market; Blockbuster refused. A decade
data/analytics technologies in an organization. later, Blockbuster filed for bankruptcy. [9]
The scientific and the grey literature support more or less
That is one of the most referred failure cases, but it is
the various hints similarly; thus, both communities agree
not. We should read it oppositely. Indeed, it is a case where
about which are the most relevant hints.
big data/analytics technologies were mistakingly not used by
IV. FAILURE CASES Blockbusters, and conversely their use resulted in a “success
story” for Netflix. This case shows the relevance of Data
We summarize the found documented failure cases in
Culture because the refusal of using the data resulted in the
Fig. 4 (in [25] you can find the references to all the sources
company’s failure.
citing them).
“jobs” versus “Jobs” “A big data project was set out to use
We analysed each case to determine first if it is a real
failure related to big data/analytics, then we tried to deter- Twitter feeds to predict the U.S. unemployment rate. The
mine the root causes of the failure, listing the hints, among researchers devised a category of many words that pertained
those reported in Sect. III-A, that have been disregarded. to unemployment, including jobs. They culled tweets that
The analysis of the various cases also prompted new hints contained these words then looked for correlations between
that except one were never mentioned in any of the examined the total number of words per month in this category and
sources. We are aware that the list in Fig. 4 contains only the the monthly unemployment rate. The work crept along, when
cases that have become public either because a “victim” had all of a sudden there was a tremendous spike in the number
some connection with the media, or because the failed BDA of tweets containing the type of words that fell into this
project was heavily publicized before to start, e.g. cases category. Maybe the researchers were really onto something.
related to politics and sport, or because the failure of the What they hadn’t noticed was that Steve Jobs died.” [31]
involved company went public. However, we think they are The disregarded hint is Domain Knowledge, which also
interesting to derive suggestions for successfully developing suggests considering that the domain may change along the
BDA projects. time and how the analysed data are related to the domain
Blockbuster In 2000 Blockbuster was at the top of its game itself.
when Netflix proposed a strategic brand partnership where NHS UK National Health Service undertook a project to
they wanted to run Blockbuster’s online brand in exchange integrate all patient medical records into a central healthcare
for the latter promoting Netflix in their retail chains. Netflix database. The analytics this project predicted would have
offered the use of data analysis to gather insights about been significant and valuable to the NHS. The failure of the
project is directly related to several challenges. The effort

250

Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
to build this analytical platform began with a lack of clear
business objectives. Each health care provider within the
system had different requirements, yet they needed each
of their systems to work together. Because the leadership
team did not clearly understand, define or communicate
the benefit of this effort to those affected, health care
providers continued to work within their respective silos.
The program was also challenged by technical difficulties
stemming from a lack of expertise within the analytics
team. [15] In this case, the disregarded hints were: Problem
Identification, Specific Skills, and No Data Silos.
The Making of a Fly Price In April 2011, a biology book
was being sold by two book retailers under the Amazon
umbrella. Both were using big data solutions that would
change the prices of the books based on market demand
and book availability. Unfortunately, in this particular case,
both retailers added a rule by which if a book has a certain
demand and availability patterns, then they will charge for it
a little bit more than the market. Thus one system increased
the book price a little bit more, then the other system in
turn responded shortly after in the same way, and both
systems kept increasing the book price until it reached $23
million. [4] The disregarded hints are Interaction with other
Analytics Programs, which invites to consider the existence
of competing analytics programs, and Domain Knowledge.
That failure could have been avoided if the system was
designed to signal when the automatically generated price
was not sensible anymore, and this prompts a new hint:
• Sensible Results: The sensible results of an analytics task
should be determined, and any non-sensible outcome should
be signalled.
Canadian Bank The case involved an analytic model for
acquiring new customers for a new product of a well-known
Canadian bank. The model was built and worked very well
and then it was implemented and made operational within
a future marketing campaign. Unfortunately, the system
did not automatically generate the final result. Indeed, the
user had to perform various operations manually over the
equation resulting from the model, including multiplying
the entire equation by -1. Multiplying the equation by -1
was forgotten by the user when scoring the list of eligible
customers. As a result, clients with a high score were
considered the less eligible for the new product and vice-
versa, and the marketing campaign was a complete failure.
[5] This is just a plain software engineering failure: it is
not sensible not to have automatized and tested the entire
procedure.
JC Penny From a leading light in the retail consumer space
in 2012, JC Penney witnessed a mass exodus of customers in
under a year. The culprit? Poor deployment of a large scale
merchandising retail analytics solution. A few months into
a complete store revamp, a complete, open and integrated
suite of retail analytics solutions was implemented. The hope
was that JC Penney would have new customer shopping
251
Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
insights to respond faster to customer preferences. On paper,
this would simplify processes and capture structured and
unstructured data (e.g., customers sentiment) so it could
deliver the best possible customer experience. Nevertheless,
no meaningful insight from those analytics solutions was
gained. Instead, sales started to decline. [21] The Problem
Identification was disregarded in this case, and the result was
a big failure.
Romney’s Failed Bid During the 2012 US presidential elec-
tion campaign, Romney’s campaign utilized a big data
system called Orca. It was supposed to read data from
various sources, and from the campaign volunteers field
feedback to generate insights about voters who support Mr.
Romney and did not vote yet, so the campaign volunteers
can target them on the election day, and convince them to
participate. The Orca system failed in finding the right voters
and was useless in the election day, due to its complexity
that left volunteers confused about how to feed it with their
field data and how to use its outcome. [10]
That is a paradigmatic example of a software engineer-
ing&project management failure: no concern for possible
distributed denial of service attacks, no stress test, no re-
dundancy to avoid unavailability, no user training, 60 pages
of instructions sent the evening before the election, helpline
not working, system accessible only by a web site (not an
app) and available only the morning of the election day.
Hence the analytics part was not used in practice.
Target Target set off a maelstrom of outrage and privacy
concerns when the retail behemoth angered an unsuspecting
father by sending discount coupons for cribs and baby
clothes to his teenage daughter, who had not yet revealed her
pregnancy to her parents. Target clients purchase data were
mined to look for patterns (for example, pregnant women
buy unscented lotion at the start of their second trimester)
to send them special deals and coupons for baby items. [24]
Technically that is a big success of analytics because
expecting women were spotted precisely, but it also prompts
us to modify the hint Problem Identification by adding to its
definition
Moreover any possible side effect due to its solution
should be investigated.
In this case, it should have been considered that discov-
ering to be pregnant is a sensible issue and not always a
happy event.
Sears The Sears decline is also related to the failure to
exploit the data related to their membership program, see
[18] for a detailed presentation of the case. The hints not
followed in that case are Data Scientists, Business and IT
People Together and Data Culture.
Moreover, it seems that they forgot to consider also
thick data2 that could have helped to avoid the failure:
“The purchase data we used reflected a subset of the
entire customer experience, missing data undermined our
delivery of a complete customer value proposition. Our
data analytics team mostly focused on transactional data
and product-related behavior such as online browse paths.
However, delivery of a full retail business value proposition
depends on creating an experience for the customer, not
just understanding the products they buy/consider buying.
Our corporate team should have augmented our transactional
data with qualitative data from focus groups and surveys
to understand customer motives for shopping our owned
channels, as this may have helped us address key issues
in our business model.” [18] This last point prompts a new
hint:
• Thick Data: Whenever human behaviour is concerned,
consider complementing quantitative data with thick data.
Chicago Police The Chicago Police to reduce violent crime
produced a list of the 400 individuals most likely to break
the law. The index of violent individuals was the result
of a predictive analytics program that used a mathematical
algorithm to sift through crime data. However, the algorithm
ran into a firestorm of controversy in late 2013 when a
Chicago Tribune article told the story of a man on the list
who had no criminal arrests. [17].
That is a case of unethical use of data and analytics, which
is currently a rising topic for researchers and practitioners.
Concerning our hints, Domain Knowledge was disregarded
(remind that it also requires checking the consistency of the
data with the domain), and here the data were not truly
representing the crime in Chicago.
Keep Calm T-shirts “Amazon was caught up in a big data
fiasco when poor programming and analytics led it to sell
T-shirts on its site with offensive messages such as Keep
Calm and Rape a Lot and Keep Calm and Punch Her. The
seller company failed to check the batch phrases generated
via a scripted computer program and sent it for publishing.”
[8] This project is wrongly considered a failure related to
big data/analytics because it is a case where the results of
a non-analytic algorithm were assumed valuable and used
without a second thought. The hint Sensible Results perhaps
could have helped, driving the developers to think which
would be the acceptable combinations.
Google Flu The Google Flu Project was launched to pro-
vide real-time monitoring of flu cases. While the premise
seemed a good way to prove the power of big data, Google
got it wrong. In building its analytics, Google assumed that
there was a close correlation between people who search
for flu-related information on the web and the number of
people that had the flu. However, Google Flu overestimated
2 Thick data is qualitative information that provides insights into the
everyday emotional lives of consumers. It goes beyond big data to explain
why consumers have certain preferences, the reasons they behave the way
they do, why certain trends stick, and so on.
252
Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
the incidence of flu in 2011-2012 and 2012-2013 by more
than 50%. The reason for the failure was the data selected
for analytics. Google wrongly assumed that its search data
was an accurate measurement of flu infection. It was missing
what researchers call “small data,” i.e. traditional predictors
such as past flu seasons and regional factors, and by using
its own search analytics as a metric, Google was modifying
its own findings as it measured results, creating a feedback
loop that would skew the findings. [33]
Google Flu is a true big data/analytics failure. The disre-
garded hint is Domain Knowledge, which suggests controlling
the consistency of the data with the domain and keeping in
mind that the domain may change. Moreover, a good knowl-
edge of the domain may reduce the chance of overlooking
the small data.
Nokia Smart Phone Failure Tracy Wang, a technology ethno-
grapher, provides a clue on the Nokia failure, mainly due to
the fact that the company decided not to embrace the smart-
phone wave [34]. She tells how in 2009, her ethnographic
research in China suggested the rising interest for smart-
phones, and that Nokia dismissed the suggestion since their
selling data was confirming the growing trend for traditional
mobile phones. Thus she too suggests complementing big
data with thick data (see Sears case) whenever the project
is concerned with human behaviour.
Hotel Prices A hotel chain used some pretty sophisticated
mathematics, data mining, and time series analysis to coor-
dinate its yield management pricing and promotion efforts.
The system worked fine for about a third of the hotels but
was wildly, destructively off for another third. After long
and careful investigations, the most likely explanation was
that the analysts had priced against the hotel group’s peer
competitors, but hey had not weighted discount hotels into
either pricing or room availability. [29]
Not taking into account budget hotels is a clear case of
disregarding Domain Knowledge.
OfficeMax OfficeMax sent discount coupons to a customer,
and the envelope was addressed to Mike Seay, Daughter
Killed in Car Crash. Seay’s daughter Ashley had died in
a car crash a year before. Even after a company executive
apologized, they still did not explain why an office supply
company knew that his daughter had died. [24] The disre-
garded hint is Data Quality; indeed, OfficeMax likely used
data from an insurance company without evaluating their
quality.
Suicide-Prevention App Samaritans developed a free app to
notify people whenever someone they followed on Twitter
posted potentially suicidal phrases like “hate myself” or
“tired of being alone.” The group quickly removed the app
after complaints from people who warned that it could be
misused to harass users at their most vulnerable moments.
[16]
Here Problem Identification was disregarded; again, simi-
larly to the Target case, they should have carefully guessed
the effects of using the answer. Also, Domain Knowledge may
have helped, making aware of the potential stressing danger
in such situations.
Scottish Independence Vote During a report on the refer-
endum, CNN displayed a graphic that showed that 110%
of Scotland’s population had been polled. According to the
graphic, 58% of Scots had said Yes, and 52% voted No. The
hint Sensible Results could have saved this tiny failure.
Tay Chat Bot Tay, modeled to learn from conversations and
speak like a millennial, was promptly taken offline less than
a day after its debut. In less than 24 hours, Tay had gained
more than 50,000 followers and produced nearly 100,000
tweets, slurred with bigotry and racism. [14]
Technically, it has been a success, since the chatbot
learned from its users. However, a better understanding
of the domain (Domain Knowledge) and some preliminary
thoughts about the effect of echoing current contents on
social media (Problem Identification part on side effects) may
have avoided such a failure.
USA 2016 Election All data based forecasts for the 2016
presidential elections were in accord: Clinton will win, and
the percentages ranged from 70% to 95%, but the voters
chose Trump instead.
As in other cases considered before, this is not exactly a
failure since the analytics answers were accompanied with
percentages, as stated in [7] “The technology can be, and
is, enormously useful. But the key thing to understand is
that data science is a tool that is not necessarily going to
give you answers, but probabilities, . . . people often do not
understand that if the chance that something will happen is
70 percent, that means there is a 30 percent chance it will
not occur. The election performance is not really a shock to
data science and statistics. It’s how it works.” [7]. This case
study prompted us the new hint:
• Analytics’ Accuracy: State which is a suitable error margin
for analytics computations in a BDA project.
In our opinion, Data Culture should also include a clear
understanding of what a probability means, and that, in
general, an analytics computation has an estimated margin
of error, which may play a role in a BDA project. If a project
intendeds to find all customers suitable for an email-based
marketing campaign, an error of 10% may be ok, whereas
if it is about managing the safety of a nuclear plant, a very
low error margin is needed.
But, on the other hand, in this case, it seems that there
have also been technical errors (see e.g. the detailed analysis
in [36], and the use of flawed poll data [16] Data Quality).
A similar case is 2018 Football World Cup, where analytics
forecasted the victory of Brazil defeating Germany with a
relevant percentage, but in the end France won the cup. [6]
Project Shakti India National Congress Party launched the
Project Shakti gearing up for 2019 national elections, which
at the end lost. Shakti aimed to link party supporters to
their mobile numbers, through a simple SMS. The idea
253
Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
was to know who worked for the party and to be able to
connect with them. The biggest problem was data quality
since 50%-70% of the data depending on the regions were
fake [2] (party volunteers were encouraged to insert as much
as possible people in the system, and the result was that they
added random names). Then, the data were managed in a
silos style preventing the access to whom may get something
out of them. Finally, when the system was used to survey
the connected people, questions were poorly posed so that
responders echoed the view of the chiefs of the party.
The analysis presented above of the 21 projects, see Fig. 4,
documented in the (grey and scientific) literature as big
data/analytics failure led us to the following results. Six
of them cannot be considered big data/analytics failures (5
and 7 failed for software engineering issues; 1 and 11 can
hardly be considered BDA projects; 11 is a standard case
of automatic generation of combinations of words; and 19
and 20 are considered failures because people cannot un-
derstand percentages and probabilities). The most frequently
disregarded hints in the real failures are Domain Knowledge,
Data Quality and No Data Silos, which confirm their relevance
for the success of BDA projects.
Finally, the case analysis prompted three new hints:
Sensible Results, Analytics’ Accuracy and Thick Data, and the
refinement of Problem Identification.
V. R ELATED W ORK
Saltz and others in [27] present a systematic literature
survey run in 2016. Among other results, it provides a list
of success factors for big data projects, some of which are
included in our list (Data Scientists, Business and IT People
Together, Problem Identification, Data Culture, Specific Skills,
Executives’ Support, Data Quality and Data Governance), while
the others concern some tools.
In the literature, there are other papers investigating the
success factors for BDA projects by means of literature
surveys. In general, they examined a smaller set of papers,
and the found factors are included in our list of hints. For
example, [30] presents the results of a scientific literature
review related to success factors and methodologies for big
data projects. Differently from us, their search was driven
by success-related keywords; moreover, they do not detail
which repositories explored and consider papers after 2016,
10 altogether. Some of the reported success factors are
included in our list.
[20] reports on a limited personal opinion survey whose
results agree on the relevance of three of our hints.
[4] is an interesting (scientific and grey literature) review
primarily aimed to understand which are the “big data
challenges” for a company to get a competitive advantage.
It also partially investigates the causes of failures and agrees
on the relevance of Specific Skills, followed by issues about
collecting and integrating the data and the complexity of the
used technologies. There is only a tiny intersection of cases
with ours (two failed projects).
The review [1] presents a list of challenges for the big
data projects; most of those not related to the technology are
matched by some of our hints, precisely: Specific Skills, Data
Scientists, Business and IT People Together, Data Governance,
Data Quality, and Data Security.
VI. C ONCLUSION
This paper reports the results of a (scientific and grey)
literature review aimed to shed light on the high rate of
failure of BDA projects. We collected and analyzed docu-
mented failure cases to discover the root causes (RQ1), and
collected what is assumed to reduce the chance of failure
of those projects (RQ2). The survey at the end selected and
examined 188 sources dated from 2011 (140 grey and 48
scientific, this is not surprising since the field is really new).
The answer to RQ2 is a set of hints to decrease the chance
of failure for BDA projects distinguished in those highly/
moderately and scarcely supported by the examined sources;
perhaps surprisingly, they concern the project conception,
organization and management, but not technical issues.
The answer to RQ1 is a set of 21 projects documented as
big data/analytics failures. However, their analysis revealed
that six could not be considered failures at all or failed for
other reasons. The analysis of the remaining ones resulted
in improving one hint and in further hints, two of which
never proposed up to the best of our knowledge. The results
of this study could help people develop a BDA project and
researchers in the field of software engineering interested in
adapting current methods or devising new ones to cope with
the challenges of big data and analytics.
R EFERENCES
[1] Z. A. Al-Sai, R. Abdullah, and M. h. husin. Big data impacts
and challenges: A review. In IEEE JEEIT 2019, pages 150–
155, 2019.
[2] V. Ananth. When a project to empower left congress stranded.
The Economic Times Web Site, 2019.
[3] P. Bajis. Before starting, consider 5 reasons your big
data project will fail. SmartData Collective Web Site,
2018. www.smartdatacollective.com/before-starting-consider-
5-reasons-your-big-data-project-will-fail/.
[4] H. Barham. Achieving competitive advantage through big
data: A literature review. In PICMET 2017, pages 1–7, 2017.
[5] R. Boire and T. Senator. What not to do: Exam-
ples of business analytics failures. KDnuggets Web
site, 2012. www.kdnuggets.com/2012/06/business-analytics-
failures.html.
[6] K. Budek. When predictive analytics in football fall short (an
example). deepsense.ai Web Site, 2018. deepsense.ai/when-
predictive-analytics-in-football-fall-short-an-example/.
254
Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.
[7] S. Butt. Election 2016 forecast - a big data failure?
Linkedin Web Site, 2016. www.linkedin.com/pulse/election-
2016-forecast-big-data-failure-shahid-butt.
[8] Y. Chowdhury. 5 cases when big data failed. Linkedin Web
Site, 2016. www.linkedin.com/pulse/5-cases-when-big-data-
failed-cybyte-webmaster/.
[9] Y. Chowdhury. 5 places where big data projects failed.
CyByte Blog Web Site, 2016. blog.cybyte.com/5-places-
where-big-data-projects-failed/print/.
[10] J. Ekdahl. Mitt Romney’s “Project ORCA” was a disaster,
and it may have cost him the election. Business Insider Web
Site, 2012.
[11] O. Fleming, T. Fountaine, N. Henke, and T. Saleh. Ten
red flags signaling your analytics program will fail.
McKinsey Web Site, 2018. www.mckinsey.com/business-
functions/mckinsey-analytics/our-insights/ten-red-flags-
signaling-your-analytics-program-will-fail12/7/2018.
[12] Gartner. Gartner says business intelligence and ana-
lytics leaders must focus on mindsets and culture to
kick start advanced analytics. Gartner web site, 2015.
www.gartner.com/newsroom/id/3130017.
[13] N. Heudecker. Twit. Twitter web site, 2017. twit-
ter.com/nheudecker/status/928720268662530048.
[14] E. Khoury. 7 big data blunders of 2016. VCloud News
Web Site, 2017. www.vcloudnews.com/7-big-data-blunders-
of-2016/.
[15] Y. Liu, H. Han, and J. DeBello. The challenges of business
analytics: Successes and failures. In Proceedings of HICSS-
51, 2018.
[16] S. Lohr and N. Singer. How data failed us in calling an
election. The New York Times Web Site, 2016.
[17] K. Lum and W. Isaac. To predict and
serve? Significance Magazine Web site, 2016.
rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1740-
9713.2016.00960.x.
[18] mahylights. Can data save us all? lessons from
attempting to transform Sears and Kmart through data
& analytics. Harvard Business School Web Site, 2017.
digital.hbs.edu/platform-digit/submission/can-data-save-us-
all-lessons-from-attempting-to-transform-sears-and-kmart-
through-data-analytics/#.
[19] B. Marr. Where big data projects fail. Forbes Web Site, 2015.
hwww.forbes.com/sites/bernardmarr/2015/03/17/where-big-
data-projects-fail/#4ca727c4239f.
[20] G. J. Miller. Comparative analysis of big data analytics and
BI projects. In FedCSIS 2018, pages 701–705, 2018.
[21] S. Narayanan. From big data to botched data: 5
steps to total big data failure. MIS Asia Web Site,
2014. www.mis-asia.com/resource/applications/from-big-
data-to-botched-data-5-steps-to-total-big-data-failure/.
[22] F. Rabhi, M. Bandara, A. Namvar, and o. Demirors. Big
data analytics has little to do with analytics. In A. Beheshti,
M. Hashmi, H. Dong, and W. Zhang, editors, Service Re-
search and Innovation, pages 3–17. Springer, 2018.

[23] P. Ramaswamy. How to create a data culture. Cognizant Web

Site, 2015. www.cognizant.com/InsightsWhitepapers/how-to-
create-a-data-culture-codex1408.pdf.

[24] T. Reddy. 7 big data blunders you’re thankful

your company didn’t make. Umbel Web Site, 2014.
www.umbel.com/blog/big-data/7-big-data-blunders/.

[25] G. Reggio and E. Astesiano. Big data analytics projects fail-

ure literature review: Detailed information, 2020. Web page
available at https://sepl.dibris.unige.it/2020-FAILURE.php.

[26] S. Remington. How to avoid analytics failure: 10 factors

for analytics success from research and practice. Minerra
Web Site, 2018. www.minerra.net/business-analytics/how-to-
avoid-analytics-failure-10-factors-for-analytics-success-from-
research-and-practice/.

[27] J. S. Saltz and I. Shamshurin. Big data team process

methodologies: A literature review and the identification of
key factors for a project’s success. In Big Data 2016, pages
2872–2879, 2016.

[28] G. Satell. 4 things every leader should know be-

fore starting a big data project. Inc. Web Site,
2018. www.inc.com/greg-satell/4-things-every-leader-should-
know-before-starting-a-big-data-project.html.

[29] M. Schrage. Learn from your analytics failures. Harvard

Business Review web site, 2014.

[30] M. Soukaina, H. Anoun, M. Ridouani, and L. Hassouni. A

study of the factors and methodologies to drive successfully
a big data project. In ICDS 2019, pages 1–6, 2019.

[31] L. Tucci. Big data can mean bad analytics.

TechTarget – Search Cio Web Site, 2013.
searchcio.techtarget.com/opinion/Big-data-bad-
analytics10/09/2018.

[32] Unknown. Report: Cios & big data: What your IT

team wants you to know. Infochimps Web Site,
2013. www.infochimps.com/resources/report-cios-big-data-
what-your-it-team-wants-you-to-know-6/.

[33] Unknown. How to avoid big data project failures. Ingram Mi-
cro Inc. Web Site, 2014. www.ingrammicroadvisor.com/data-
center/how-to-avoid-big-data-project-failures.

[34] T. Wang. The human insights missing from big data. TED
Talk Web Site (video), 2016.

[35] A. White. Our top data and analytics

predicts for 2019. Gartner web site, 2019.
blogs.gartner.com/andrew white/2019/01/03/our-top-data-
and-analytics-predicts-for-2019/03/02/2020.

[36] A. Woodie. Six data science lessons from the

epic polling failure. Datanami Web Site, 2016.
www.datanami.com/2016/11/11/data-science-lessons-epic-
polling-failure/.

255

Authorized licensed use limited to: Rochester Institute of Technology. Downloaded on July 06,2023 at 00:47:03 UTC from IEEE Xplore. Restrictions apply.