You are on page 1of 39

State of the Industry Research Series :

Big Data in Retail
EKN is part of the Edgell Family
Premier Sponsor
Table of Contents
Executive Summary 3
A Pragmatic Approach to Big Data 6
Research Findings 11
Recommendations 21
Appendix A: EKN Big Data Need and Readiness Assessment Toolkit 28
Appendix B: EKN Big Data Vendor Landscape 34
Appendix C: Retail Honor Board 36

Executive Summary
In a short period of time Big Data has traversed a familiar hype cycle from exuberant optimism to disil-
lusionment. In EKNs view, retailers will do well to focus on pertinent issues it brings into focus: Namely, the
opportunity to improve customer-centric and operational decision-making by building deeper insight from
a massive stream of internal and external data.
However, retailers must approach Big Data as they would any other business strategy, which is to ask 3 im-
portant questions; Is this for us, how ready are we and how do we set ourselves up for success?
EKNs 2012 Big Data report was a frst of its kind, detailed look at Big Data in retail. This report is based on
an additional survey of 70+ retail industry executives, and includes new fndings as well as comparison with
those from last year. Key fndings include:
Big Data is 5 years away from becoming mainstream: For all the game-changer hype surround-
ing Big Data, the reality is that it is currently the domain of a few frst movers. Largely, retail contin-
ues to be an industry with relatively low analytics maturity in all areas of the business. EKN views
a natural progression to be one that frst focuses on advancing analytics capabilities before being
able to leverage Big Data efectively. Retailers investment plans in Big Data support this takeaway.
A billion dollar retailer will spend an average of $75,000 on Big Data in 2013: Retailers plan
to spend less than half a percent of their total IT budget on Big Data in 2013. This will increase to
1.2% in 2016. While this means large-scale Big Data initiatives will be few and far between, a major-
ity of retailers plan to enter an experimental phase with Big Data, focusing on executing Proof-of-
Concepts (POCs) that help them determine value and chart a course for more scalable investments
in the future.
Retailers biggest challenge in adopting Big Data is the variety of data, especially unstruc-
tured data: In 2012, 1 in 2 retailers said managing the volume of data was their biggest challenge.
At that time, we pointed out that retailers have always been used to managing large volumes of
data (they have access to decades of historical transactional data) and that we expected this issue
to diminish in comparison to others. In this years survey, retailers rate data volume to be the least
of their data management concerns. Instead, managing the variety of data emerges as the #1 chal-
There is no diference in the analytics maturity of large (over $1 billion in annual sales) and
smaller (less than $1 billion in annual sales) retailers: The largest retailers with deep pockets
and a culture of innovation, such as Target and Wal-Mart, will continue to make oversized invest-
ments in Big Data. In some cases, such as Sears, they may even approach it as an opportunity to
monetize investments in technology by ofering services to smaller retailers or even other indus-
tries. Yet, smaller retailers can take advantage of the lack of diference between how strategically
they leverage analytics as compared to their larger rivals, by making smart strategic choices and
investments of their own.
Executive Summary
State of the Industry Research Series: Big Data in Retail 5
It is easy to glorify or dismiss Big Data, but it is much more difcult to develop a measured approach focused
on exploring its value for your business. Like any strategy, EKN recommends identifying use cases and build-
ing a detailed plan on how you would make a move to Big Data analytics a successful one. EKNs recom-
mendations provided in this report are intended to help retailers get started with a pragmatic approach to
Big Data, once their need and readiness is established. A toolkit to help assess need and readiness is also
Executive Summary
Research Findings Fast Facts:
The percentage of retailers who fnd data
volume to be the biggest data management
challenge is down to 24% from 46% last year.
68% of retailers rate data organization as the
biggest challenge in managing unstructured
Big Data budgets will grow at 23.8% CAGR till
2016, yet will only account for 1.2% of total
IT budget.
Inside the report:
EKN Big Data Need and Readiness Assess-
ment Toolkit
EKN Big Data Vendor Landscape
Retail Honor Board
A Pragmatic Approach
to Big Data
State of the Industry Research Series: Big Data in Retail 7
EKNs 2012 report was one of the frst focused studies on the impact of Big Data on the retail industry. We
used insights from the study to evolve the areas of investigation this year and focused them specifcally on
How real is Big Data in retail in terms of current levels of adoption and planned investments?
What is the level of retailers preparedness for Big Data analytics in terms of analytics maturity?
How does Big Data impact retailers top analytics goal - customer insights?
What are the biggest impediments to retailers adoption of Big Data?
While fndings from EKNs survey follow this section, it is important to establish our point of view on what is
increasingly a contentious topic. Consistent with our raison dtre, our outlook towards Big Data is pragmatic
rather than dogmatic.
What is Big Data?
Big Data is a trendy buzzword. The hype around it is misleading and revealing at the same time. Misleading,
because it positions Big Data as a silver bullet that will resolve all of retails challenges. It is revealing because
it shines a light on how some technology marketers and analysts have taken a generally sound concept (that
data has grown beyond retailers current means of analysis and it is important to evaluate what they stand
to gain from building new capabilities) and turned it into a technology sales pitch.
Yet, Big Data is an important topic. There is more data (volume) being created by enterprises and consumers
than ever before, a greater number of sources and types (variety) of data, as well as an increased speed (ve-
locity) at which this data fows through a companys information systems. The opportunity to better leverage
data to build a better understanding of customers and improve business agility is very real. The importance
of doing so has also increased since traditional competitive forces of price, promotions, location and assort-
ment are becoming commoditized. The culture, skills and tools required to make sense of this data are very
diferent from the status quo at most retailers.
We must note an important distinction, which is that Big Data is not a new concept. Retailers have had access
to large sets of historical transaction data for decades. Some have even been able to perform predictive ana-
lytics by integrating other data source such as weather data, before Big Data as a term caught on. We illus-
trate this only to point out that advanced analytics and Big Data are diferent, and can be mutually exclusive.
In our 2012 report we defned Big Data as collectively referring to the strategy, business processes, tools and
technologies that pertain to datasets whose size and complexity is beyond the ability of typical database software
tools to capture, store, manage and analyze.
A Pragmatic Approach to Big Data
To clarify what is inherent in this defnition - Big Data is a relative concept. Think of it as a generational leap
from your organizations current analytics capability (from a data integration and management perspective)
to a state wherein it is able to derive strategic value from the analysis of:
A larger volume of data, at a level that represents exponential, not incremental, growth compared
to current capabilities
A greater variety of data, necessarily including unstructured and/or semi-structured data
A greater velocity of data, in terms of speed of input and output through enterprise systems
The analysis of larger data sets at greater speed and lower cost have been made possible by technological
advances such as in-memory computing, new information management architectures such as MapReduce,
Massively Parallel Processing (MPP) and NoSQL databases among others. It is very likely that a move towards
Big Data analytics will require an investment in such tools. However, thinking of Big Data only in terms of
such technologies is myopic.
Big Data must be thought of as a business strategy focused on upgrading the organizations capability to
take action based on insight gleaned from the availability of a much larger data set than it is currently ca-
pable of. Having the skills required to execute is an often ignored but critical component of such a strategy.
What patchwork of tools enables this strategy is incidental.
Once retailers defne Big Data in terms of their own analytics maturity, it becomes easier to ignore the philo-
sophical debates that distract them from more pertinent questions - do I need it, am I ready for it, and where
do I start?
Do I need it? Am I ready for it?
Part of the Big Data challenge is the perception that it is a natural progression of an organizations analytics
capability or maturity. That it is inherently good. Big Data is simply shorthand for the ability to analyze larger
sets of more varied data at greater speed than you are currently capable of. The nature of the analysis - from
reporting to basic analytics to predictive and investigative analytics - is what determines analytics maturity.
The existence of more data and the availability of tools that allow you to analyze this data should not be
the primary driver of interest in exploring Big Data. Unless driven by a clearly focused strategy, Big Data is a
solution in search of a problem.
A Pragmatic Approach to Big Data
State of the Industry Research Series: Big Data in Retail 9
Retailers need to evaluate how seriously they need to pursue Big Data and assess their current readiness
based on the factors below. These are available in expanded form as an assessment toolkit in Appendix A.
EKN is ofering retailers the ability to receive a complimentary assessment by flling out the digital version of
the toolkit. More details are available in the Appendix.
Strategic Intent
Innovation Will your Big Data strategy be linked to the companys innovation plans?
Competition Do competitive moves mandate an investment in Big Data?
Customer Engagement
Has the company identifed customer engagement as a key driver of business strategy and invest-
Opportunity Qualifcation
Storage / Data Warehouse Is your current data warehouse and storage capacity insufcient for projected data growth?
BI Tools Are your current BI tools insufcient in terms of speed of analysis?
Social and / or Mobile Do your plans include the need to analyze large sets of social media or mobile data?
Unstructured Data Do your plans include the need to analyze large sets of unstructured data?
Advanced Algorithms
Do you plan to develop and implement advanced analysis algorithms and apply them to large
data sets?
Organizational Readiness
Acceptance of Failure Is innovation engrained in the organization culture, including acceptance of failure?
Is the company culture insights-driven or based on intuition and leadership of strong personali-
Management Alignment Does Big Data have a C-level executive owner and champion?
Analytics Maturity
Advanced Analytics Capability Does your analytics maturity justify investments in Big Data?
Skills and Resources
Do you have or are willing to invest in acquiring the skills and resources required to elevate analyt-
ics maturity in the organization?
Data Management
Data Management Strategy How mature is your enterprise data management strategy?
Data Sources Which internal and external data sources are integrated into your analytics initiatives?
Investment Appetite
Technology Budget
What percentage of your analytics budget have you earmarked for spending on Big Data initia-
Technology Leadership
Does technology leadership come naturally to your organization, or is your strategy to adopt tried
and tested industry standard solutions?
Cloud or Investment Appetite
Is your organization ready to embrace public or hybrid cloud for data storage and analysis? If not,
is it ready for big investments in data management infrastructure such as private cloud or data
A Pragmatic Approach to Big Data
EKN 10
Where do I start?
Once need and readiness is established, EKN recommends retailers start with a clear defnition of their
goals, execution approach and success metrics. The recommendations section of this report is focused on
providing additional detail on the following steps EKN recommends retailers take to get started.
A Pragmatic Approach to Big Data
Identify a business function candi-
date for a POC
Start with a POC focused on no more than 1-2 business functions, such as Customer In-
sights, Marketing, Merchandising, Supply Chain, Stores or Multi-Channel.
Assess analytics maturity
Perform a detailed benchmark of your analytics maturity within the identifed business
Decisions and use cases defnition
Defne focused use cases including an identifcation of business decisions, insights required,
data needed to deliver those insights, type of analysis and expected action that will be taken
based on the analysis.
Data organization and integration
Based on the data requirements identifed from the step above, identify the sources of data
that will be tapped for the POC. Build an efort estimate for organizing and integrating the
Analysis models
Based on the insights required, identify whether existing statistical models are available or if
proprietary models will need to be built.
Systems and tools
Identify any new systems and tools that will be required as part of the POC. Focus on ones
that have a non-linear ability to process and analyze data.
Training and documentation
Clearly identify the team associated with the project and institute a short-term training plan
to get associates up to speed with Big Data concepts and analysis models. Ensure the POC
exercise is used to capture lessons learnt and as a process template for future projects.
State of the Industry Research Series: Big Data in Retail 11
The Consumerization of Retail Information Technology
State of the Industry Research Series:The Consumerization of IT: Personal Devices and Software in the Enterprise
Research Findings
EKN 12
Survey respondent distribution by segment
(Figures are percentage of total respondents)

Survey respondent distribution by annual revenue
(Figures are percentage of total respondents)

Survey respondent distribution by designation
(Figures are percentage of total respondents)
Research Findings
Apparel & Accessories
General Merchandise and Grocery
Less than $100 million
$100 million to $499 million
$500 million to $999 million
$1 billion to $4.9 billion
$5 billion or more
State of the Industry Research Series: Big Data in Retail 13
Research Findings
Big Data is 5 years away from becoming mainstream. 60% of retailers still assess them-
selves as having basic analytics capability.
Analytics maturity of retailers
(Figures are percentage of total respondents)

The one consistent theme across EKNs 2013 research has been the importance retailers have attached to im-
proving analytics maturity across the enterprise and the consistently low levels of current maturity they have
EKN believes retailers will frst move from basic to advanced on the analytics maturity curve (reporting, basic
analytics, predictive analytics, investigative analytics) before taking on large scale Big Data initiatives.
Compared to 2012, 9% fewer retailers rate themselves as only having basic analytics capabilities. However,
with more than 3 in 5 retailers still assessing themselves as having basic analytics capabilities, clearly the av-
erage retailer is data-rich and insights-poor. Tellingly, 83% of retailers consider themselves at par or behind
competition in terms of strategic use of analytics.
Even over the next 2 years, 1 in 2 retailers have no plans to implement Big Data initiatives in core retail func-
tions such as Stores or Supply Chain.
Given the current landscape of retailers analytics maturity and planned investments in Big Data over the next
2 years (see related fnding below), Big Data is still in an experimental stage in retail. A few retailers will make
big bets while others will continue to wait and watch or experiment cautiously.
Sources: EKNs Future of Analytics Report & EKNs Big Data in Retail 2012 Report
14% 15%
2012 2013
Reporting and basic analytics Investigative analytics Predictive analytics
Analytics Maturity Strategic Intent Organizational Readiness Opportunity Qualifcation
While 8 out of 10 retailers
are aware of Big Data, only
22% currently use Big Data
analytics solutions.
83% rate themselves as lag-
ging behind or at par with
competitors in their use of
customer analytics.
72% of retailers do not have
a C-level Big Data champion.
Less than 2 in 10 retailers
leverage predictive analytics
across business functions
EKN 14
Research Findings
The Big in Big Data is not the greatest challenge; organizing and integrating unstruc-
tured data is.
Big Data challenges that are the most difcult to manage
(Figures are percentage of total respondents)

Top data management challenges in managing unstructured data
(Figures are percentage of total respondents)

Timeframe for retailers for integrating data sources into analytics
(Figures are percentage of total respondents)
Handling the Variety (Types) of
data (48%)
Handling the Velocity of data (28%)
Handling the Volume of data (24%)
Handling the Volume of data (46%)
Handling the Variety (Types) of
data (34%)
Handling the Velocity of data (20%)
#1 #1
#3 #3
Data organization Data integration Data capture Data
Data access Data storage Data privacy
Currently integrated In the next 12 months In the next 12-24 months No plans to integrate
14% 24%
19% 35%
Store (POS
transaction data)
Online Loyalty or CRM
Syndicated data
(Nielsen, IRI)
Social media Mobile Census data
State of the Industry Research Series: Big Data in Retail 15
Research Findings
Importance vs. use of diferent data sources for Big Data initiatives focused on customer insights
(Y-axis: Figures are percentage explaining importance on a scale 0-100, 100 being the most important; X-axis: Figures are percentage of total
We know that retail is a data-intensive industry - serving millions of customers, moving thousands of items,
stocking thousands of SKUs in hundreds of stores. Retailers are accustomed to large swaths of data.
However, respondents in EKNs 2012 Big Data in Retail survey identifed the volume of data as their biggest
data management challenge. At that time, we highlighted that we did not expect this to be the case for long
and pointed out that retailers with higher analytics maturity found the variety of data to be a greater chal-
In 2013, as retailers understanding of Big Data has improved and they have gained a clearer understanding
of their own capabilities, managing the volume of data is rated the least of their challenges. Data variety has
clearly emerged as the #1 challenge.
More than anything else, the variety of data sources and types that enterprises have at their disposal best
defnes the need for a concept such as Big Data. Data residing in traditional data warehouses and data marts
is considered structured data. A greater area of growth and complexity (in terms of analysis) is unstructured
(such as documents, audio and video fles) and semi-structured (such as email, XML) data. Combining enter-
prise data with publicly available data from social networks, websites and public data source such as the
Department of Commerce can be a valuable strategy, but one that only adds to retailers data manage-
ment woes.
Transactional data
Operational data
Public data
Social media data
Web logs
Machine sensor logs
Textual data
Weather data
Maps/GIS data
Multimedia data
Consumer mobile geo
positioning data
Use of dierent data sources





Low High
EKN 16
Research Findings
To clarify, the challenge of data variety is driven primarily by the increasing amount of unstructured data
(some estimates suggest 80% of business relevant information originates in unstructured form). In terms of
managing unstructured data:
Organizing this data and integrating into their analytics initiatives poses the greatest challenge to
retailers, with 2 in 3 identifying these as the top data management issues related to unstructured
Data storage and access on the other hand are seen as relative non-issues, validating the fact that
retailers have moved on from viewing data volume as their top concern.
To put things into perspective, data privacy is rated as the least of retailers worries. This is a topic
that justifably gets a lot of airtime and attention, but retailers do not (yet) see it as holding them
back from exploring Big Data. We believe privacy will come to be viewed as more challenging in
years ahead once retailers are able to mature beyond the basic issues of organizing, integrating and
analyzing the data.
Interestingly, retailers have recognized the importance and complexity of the variety of data even as their
current data use for analytics indicates a singular focus on transactional and operational data. The gulf
between the percentage of retailers who integrate transactional (POS, eCommerce) data (2 in 3) and social
media or mobile data (1 in 4) is huge.
Location-based data is set to grow exponentially over the next few years. Over the next 2 years, retailers
plan to drive more GIS location-based promotions (to drive customers into stores) as well as use Wi-Fi or
other sensor-based technologies to engage customers at an aisle, shelf or even product level (once inside
the store).
EKN expects retailers to continue to face data variety related challenges over the next few years. We see data
organization and integration to be perceived as lesser challenges over the next 2 years, much like data vol-
ume challenges decreased from 2012 to 2013. We believe these are teething issues that are relatively easily
solved with a smart selection of tools and technologies. The real issue will be in developing domain and use
case specifc analytics models to make sense from unstructured data.
State of the Industry Research Series: Big Data in Retail 17
Research Findings
Big Data budgets are byte-sized and focused on Proof-of-Concepts (POCs).
Percentage of analytics budget allocated towards Big Data initiatives, 2013 vs. 2016
(Figures are average percentage of total analytics budget allocated towards Big Data initiatives for the corresponding year)

Timeframe for the deployment of technologies enabling Big Data initiatives
(Figures are percentage of total respondents)

2013 2016
Technologies Next 12 months Next 12-24 months
Specialized Big Data BI tools 19% 24%
Big Data Visualization 16% 32%
15% 16%
Software framework for data-intensive distributed applications (e.g. Hadoop)
12% 18%
BI Appliances 12% 19%
NoSQL databases 10% 19%
In-memory platforms (e.g. SAP HANA) 10% 22%
Unied Information Access Platforms 9% 11%
Traditional BI tools (eg: SAS, IBM) 8% 9%
Distributed Database Management System (e.g. Cassandra)
7% 12%
Semantic analytics tool
6% 13%
Column databases 6% 15%
Programming model for parallel-processing large datasets (e.g. MapReduce)
6% 16%
Traditional Database Management System 6% 6%
Statistical Computing (e.g. R programming language) 4% 19%
Traditional Data Storage (SAN, NAS) hardware 2% 6%
EKN 18
Research Findings
In our 2012 Big Data report, we strongly recommended retailers start with targeted POCs by identifying high
impact use cases where it makes sense for them to experiment with Big Data analytics. We suggested pricing
optimization, customer segmentation and marketing efectiveness as potential candidates.
The 2013 survey follows up with more details on retailers Big Data budgets, and it is a stark reality. The
spend on Big Data as a percentage of total IT budget is miniscule (0.49%), and is only expected to grow to
1.3% by 2016. In efect, a billion dollar retailer spent $75,000 on average on Big Data initiatives in 2013.
The reality is that other than a few large retailers such as Target, Wal-Mart and Nordstrom who have made
and are making big investments in building Big Data capabilities, retailers have adopted a wait and watch
approach. Their eforts range from do nothing to the targeted POCs we recommended in 2012.
Retailers plans over the next 2 years also do not indicate aggressive plans to upgrade their Big Data technol-
ogy infrastructure:
Less than 1 in 5 retailers plan to invest in any Big Data technology in the next 12 months. Their
plans are more aggressive in the 12-24 month timeframe, which begs the question as to whether
retailers are certain of those investments or are simply deferring intent to adopt to a time in the
future? The Big Data budget (above) suggests investments over the next few years will continue to
be circumspect.
Specialized Big Data BI tools and Big Data visualization will outpace other solutions in terms of
adoption over the next 2 years.
Retailers stated intent to invest in Big Data (in terms of budgets and solution adoption) does not match up
against the hype. Retailers will continue to invest in Proof-of-Concepts to establish ROI. Large-scale invest-
ments in building technology capabilities to store, manage or analyze data will be rare. They will predomi-
nantly be the domain of large retailers with deep pockets, or smaller retailers who will decide to make a
strategic bet on leapfrogging competition in terms of insights driven retailing.
The tools and solutions retailers will focus on in the short-term are ones that are quickly scalable, provide
them retail format or use case specifc data and analysis models, and have a low cost of failure (both mon-
etarily and in terms of impact on the business).
State of the Industry Research Series: Big Data in Retail 19
Research Findings
An opportunity for smaller retailers to improve competitiveness.
Comparison of large ($1 billion+) vs. smaller (less than $1 billion) retailers
(Figures are percentage of total respondents)

Analytics maturity is a great leveler in retail. Unlike other areas such as supply chain, merchandising, multi-
channel operations and marketing where larger retailers ($1 billion+ in annual revenue) display a greater
maturity of technology adoption and maturity compared to smaller retailers (less than $1billion in annual
revenue), there is little to no diference between them in terms of their ability to:
Conduct advanced analytics
Use insights to diferentiate against competition
Augment transactional data with other valuable sources of data

This is where larger retailers are better positioned. Data from EKNs research reveals a signifcant diference
between these two sets of retailers in terms of investment priorities and outlook towards Big Data:
Larger retailers topmost Big Data investment priority is training and team augmentation. More Tier
One retailers (by a degree of magnitude) plan to train their executives and existing analysts on Big
Data over the next 12 months.
Smaller retailers on the other hand identify Big Data tools or software as their topmost investment
In fact, whereas 83% of larger retailers consider data integration
as their top data management challenge, smaller retailers are
relatively less concerned, perhaps because the scale and com-
plexity of data in their enterprise systems is not as daunting.
Simply put, Big Data is not a technology challenge. It is a business
strategy that the organization must commit to, and building pro-
prietary analysis models based on use case and retail format are
central to an efective strategy.
While there is not much diference
in smaller retailers adoption of Big
Data or their analytics maturity
as compared to larger rivals,
they must look at Big Data more
strategically, rather than as a
technology silver bullet.
60% 83% 53% 50%
57% 17% 13%
Smaller Retailers
Investment priorities
Large Retailers
Smaller Retailers
Large Retailers
Outsourced analytics New tools or software New hardware
Have basic
analytics maturity
Data integration as a
data management challenge
Plan to train analysts
over the next 12 months
Plan to train the executive
team over the next 12 months
67% 51% 30% 32%
77% 30% 27%
EKN 20
Research Findings
Improving their analytics maturity may provide smaller retailers a rare opportunity to build a competitive
advantage against retailers - regardless of size - that do not.
To do so however, they need to evolve their approach towards Big Data. Look at it more strategically rather
than as a technology silver bullet. Invest aggressively in upgrading enterprise-wide analytics maturity regard-
less of any accompanying investments in Big Data tools. Defne use cases focused on customer engagement
and operational improvement and seek management buy-in to invest in 1-2 POCs.
Experimenting with Big Data, especially if the organization is willing to expose certain data sets to trusted
public cloud providers, does not need to be expensive.
The costs of storing, managing and analyzing data are decreasing.
Availability of cloud-based Software-as-a-Service solutions provides access to Big Data infrastruc-
A wide range of tools, from statistical computing to visualization, are available at an extremely low
cost (some are free).
Data from public sources such as social media networks, weather data, GIS data and data from the
US Census is also available at relatively low cost.
The biggest issues are defning where there is value, building strong use cases and developing analytical
models that allow you to extract value from the data being integrated.
State of the Industry Research Series: Big Data in Retail 21
Given how important ofering a unique in-store experience is to retailers ability to diferentiate against on-
line retailers, EKN asked respondents to rate the in-store experience they provide customers in their stores
against other brick and mortar competitors.
1 in 5 assess themselves as being industry leaders
1 in 2 consider the in-store experience they provide to be better than competitors
None of the respondents in EKNs survey rated the in-store experience they provide as lagging
The respondents view of the customer experience in-store is more favorable than EKNs view. Since the
response categories do not speak to the quality of experience, perhaps the response indicates one or many
of the following:
A bias introduced due to a competitive comparative question
A subtle acknowledgement of the low bar set by the industry as a whole in terms of in-store experi-
A variable understanding of in-store experience by the respondent set, given EKN did not describe
it in the context of Omni-channel retailing
Investments in store technologies account for nearly one-third of
the total IT budget.
Percentage of the total IT budget allocated towards store technologies
(Figures are average percentage of total IT budget allocated towards store technologies for the corresponding year)
EKNs 2012 report contained a set of detailed recommendations over a short, medium and long-term
horizon. Most of them still apply as retailers analytics maturity and Big Data adoption hasnt changed dra-
matically in the last year. We recommend readers of this report consult the recommendations section of
the 2012 report as well.
For this reason, we have focused our energies in the 2013 report on detailing out more specifc short-term
recommendations, providing retailers an actionable roadmap for 2014.
EKN 22
Establish if and why you need Big Data, and determine if you are ready for it.
Adopting a Big Data strategy is a long-term, resource-intensive strategic initiative that will likely change many
things about business-as-usual. As established previously, a majority of retailers are a few generations be-
hind achieving the type of analytics maturity that would lend itself to an efective Big Data strategy. The
industry, in general, is not racing to adopt Big Data for Big Datas sake, and neither should you.
As introduced earlier in the report, it is critical to frst establish a need for pursuing Big Data and subse-
quently assessing the organizations level of preparedness. We view these as foundational to a meaningful
approach to Big Data.
EKNs Big Data Need and Readiness Assessment Toolkit in Appendix A provides an easy reference toolkit
that retailers can use to conduct an assessment of whether they need Big Data. This is an excerpt and
retailers can download the toolkit from the EKN portal or reach out to EKN for an assessment exercise.
Start with a POC focused on 1-2 high priority Big Data use cases. Detail out the use cas-
es, including decisions, data inputs, sources, delivery and consuming audience and re-
source requirements.
Based on results from EKNs Big Data Need and Readiness Toolkit, or similar assessment templates, retail-
ers that fall in the Gently test the waters category should evaluate a focused POC or two to build strategic
capability and test ROI on a low-cost, low-risk basis.
The Big Data Opportunity-Impact Matrix is a decision framework that outlines Big Data opportunities for
retailers, taking into account time taken to implement and potential impact of outcome. This is an illustrative
one for a Billion dollar brick and mortar retailer. Things will looks diferent based on the size of the retailer
since impact and action can difer.
Highlighted opportunities correspond to the functional area that the EKN Big Data Use Case Template covers
Improve Customer
Improve Sourcing &
Dynamic Pricing
Real-time Segmentation
Optimize Promotions
Improve Demand
Improve Replenishment
Improve Infrastructure
Improve Transportation
Plan & Execute New
Store Opening
Improve Store
Manage Recalls
High Execution Time Low
State of the Industry Research Series: Big Data in Retail 23
Due to the subjective nature of what could be considered Big Data, we fnd an increasingly blurred line be-
tween advanced analytics (such as predictive and investigative analytics), data visualization and Big Data. For
this reason, a lot of reported Big Data activity is actually retailers improving their analytics maturity in terms
of being able to perform predictive or investigative analytics.
EKN recommends retailers detail out use cases, such as the one illustrated below, in order to develop a clear
understanding of all the parameters that qualify it as truly being a Big Data initiative vs. a business-as-usual
analytics project. We recommend focusing on one or more of the following areas:
Customer segmentation
Pricing optimization
Promotion optimization
Infrastructure availability optimization
EKN Big Data Use Case Template:
Functional area Marketing
Problem statement How can Big Data help us improve our campaign or promotion efectiveness?
Data Sources
POS data
eCommerce data

CRM or loyalty data

Promotions calendar and execution data
Public data (Census)
Syndicated data (Nielsen/IRI)
Social data (Facebook, Twitter (Firehose))
GIS Data

Consumer mobile data
Must have;
Good to have
An end-to-end Big Data platform or individual components
Data Management
Data-asa-service providers like FICO
Must have;
Good to have
Phase 1: Problem defnition
Boundary up the POC by deciding which product categories/SKUs and promotions to analyze.
Phase 2: Data review and standardization
Understand the data sources that are required for the analysis. Clean and standardize all data and
create templates for data inputs.
Phase 3: Analysis and model development
Analyze the data and build algorithms to model campaign performance. Build multiple models and
conduct scenario and sensitivity analysis.
Phase 4: Model optimization
Run tests to optimize the model on historical data and also on some live promotions. Finalize a set
of models for roll-out.
Phase 5: Roll-out
EKN 24
Base case scenario:
Do a complete analysis of your marketing campaigns, by product and by location using your entire
transactional and customer data instead of a subset. Integrate all transactional data from the POS
(sales, discounts), operational data like CRM or loyalty, marketing promotions calendar, public data,
syndicated data along with social data (Facebook, Twitter etc.) to segment customers and promotion
Conduct statistical analysis to understand campaign performance by customer segments and the
infuence of factors such as product type, time, device and past purchase. Create predictive models
to assess the campaign efectiveness and apply them on historical data to improve it. Setup new
campaigns based on customer segments, choose and adjust statistical models and conduct experi-
ments to test them. Use a library of models to drive more campaigns and use them to reduce of-price
promotions and thereby increase gross margin.

Next Case:
Integrate GIS data and overlay the analysis on a map to look at response rates of campaigns. Enable
interactivity on the map and allow users to run simulations. Integrate with marketing mix analysis to
help optimize spend.
Advanced Case:
Enable dynamic real-time promotions online and on the mobile device. Combine navigation path,
wish lists and customer location information to perform real-time customer segmentation and per-
sonalization. You can also combine social activity (that you have access to) to create a broader set of
interests that you might not have deciphered otherwise. Sensing the intensity and frequency of social
activity - recent tweets, Facebook likes - can help you position more relevant promotions.
A retailer can deliver real-time campaigns to customers while they are in the store targeting them as
a segment of one - Jill gets a 10% coupon, but Jennifer gets 15%.
What makes this Big Data?
High Volume (large data sets)
High Variety
Low Velocity*
*Base case is a point in time analysis. The advanced use case has high data velocity
Business Benefts
Increased conversion rates
Faster analysis
Improved accuracy
How much should you
spend on a POC?
Less than USD 75,000
State of the Industry Research Series: Big Data in Retail 25
Prioritize resourcing, training and team structuring.
Already, the #1 challenge facing retailers from an analytics perspective isnt the lack of available insights or a
lack of analytics tools; it is the inability to deliver the relevant insight to the right person at the right time. Big
Data wont address this issue; it will only serve to amplify it.
Further, EKN expects this issue to evolve from one focused on delivery of insights to one about inability to
take action. Retailers analytics investments over the next two years include solutions such as digital dash-
boards and mobile BI tools that should help bridge the delivery of insights gap. Improved visualization will
also help bridge the gap between statistical knowledge and business acumen.
But, two core issues remain that cannot be addressed by technology intervention alone. Retail employees
in general - merchants, buyers, store associates and managers, business executives, marketers, technology
professionals - are not statistical wizards. Also, insights-driven decision making may not come naturally to
them unless it has been part of their personal DNA. On the other hand, there are only as many analytical re-
sources retailers can hire, and they are unlikely to have the business acumen and retail expertise that other
retail employees do.
Retailers must take the following steps to ensure they are better prepared in terms of the ability to act on
insights as they prepare to make investments in Big Data analytics:
Training executives and analysts: For a large part, the creators and consumers of insights will
have difering skills and capabilities. Retailers must institute cross-competency training for execu-
tives to better understand analytical concepts and analytics resources to understand the most im-
portant insights their analysis enables.
Resourcing: In more than one EKN Peer Forum (a closed door meeting of EKNs retailer member-
ship), smaller (<$1 billion) retailers have identifed the lack of analysts, statisticians, economists and
other analytics resources as being a key limiting factor of their analytics maturity. Retailers must
project the resource requirements of any Big Data initiative and consider all options to shore up the
analytical capability available to them, including considering hiring contractors.
Restructuring: EKNs research reveals retailers consider their current analytics team structure to
be inefcient. 71% have a structure wherein each department is responsible for its own analytics
resources, but only 8% prefer such an arrangement. A majority want to move to a shared services
structure, and we agree that analytics as an enterprise-wide strategic function is a viable setup for
the future.
EKN 26
Data organization and integration will be a huge challenge. Strengthen your enterprise
data management strategy.
Respondents in EKNs survey identify data organization and integration as their two biggest data manage-
ment challenges. Countless anecdotal conversations come to mind from retailers and solution providers
alike that highlight how rampant this issue is. Stories of projects with a clear analytics goal and well-defned
use case getting mired in issues relating to data availability, quality and completeness must resonate with
readers of this report.
The importance of retailers developing or strengthening their enterprise data management strategy cannot
be overstated.
With the democratization of data - i.e. more freely sharing your data, using someone elses data and
making data available across the enterprise - stricter governance and standards are required. This
also holds true as retailers manage more data, of more types, from more sources, fowing through
information systems at greater speed.
The industry has spoken of a common view of the customer for decades. With cross-company shar-
ing of data along with retailers needing to tap into external customer data, the need for a common
customer data model (MDM) is amplifed.
The data architecture needs to support the need for real-time visibility and quicker decision mak-
ing, which means the ability to consume and provision data on the fy.
Data privacy and security will gain importance as a move to the cloud becomes inevita-
Even though privacy and security is an important topic of discussion in the retail industry, respondents in
EKNs survey rated it the least of their data management challenges. That being so, we believe privacy and
security will be important issues that will only increase in importance and complexity.
The use of cloud-based storage and Software-as-a-Service providers is inevitable. The cost structure
of IT and the expected ROI from Big Data initiatives does not support a move to in-network storage
expansion for all retailers. Even if retailers move to a private or hybrid cloud model for larger scale
Big Data investments, smaller POCs will require exposing their data to other providers or hosting it
in an environment not directly controlled by their own security policies.
Retailers are increasingly interested in rich customer data from partners and 3rd party sources.
These include telecom companies such as AT&T, consumer mobile applications such as ShopSav-
vy, data-as-a-service providers and industry data pools. As retailers explore quid-pro-quo data ex-
change arrangements or as such providers provide commercial services on sharing customer data,
both parties will need stronger privacy and security controls.
Privacy related action within the ecosystem would also have an impact on the data that is available
to retailers. Googles move to discontinue sharing keyword referrer data from being monitored in
analytics tools and Apples decision to stop supporting meta-referrer in the mobile version of Sa-
fari are two recent examples. Not only will retailers need to strengthen how they protect privacy of
their customer data, they will need to learn to operate in an environment where less may be avail-
able to them from consumerized web services.
State of the Industry Research Series: Big Data in Retail 27
EKN believes that in time an over-arching opt-in legislation and framework will allow consumers to have
deeper control over their personal information, providing them visibility and control over with whom it is
shared and for what purpose. In the meantime, retailers will beneft from developing a similarly transparent
process identifying sensitive customer information fowing through their systems, use cases that it is used
in, alternatives in case of legislation or other changes restricting or prohibiting it. More progressive retailers
may consider exposing a complete digital footprint of a customers information and how they use it to enrich
engagement with them via their loyalty programs.

Learn from the leaders and avoid common pitfalls as you build your Big Data strategy.
Along with retailers assessing their own analytics maturity, EKN recommends they benchmark themselves
against competitors within their retail format, similar sized retailers (in terms of revenue or store footprint)
and importantly against industry leaders such as Target and Wal-Mart.
In addition, retailers should develop a quick reference guide to avoid common pitfalls that plague Big Data
analytics strategies and initiatives. An illustrative list is below:
No use case Embarking on a Big Data project without a clear use case and ROI defnition.
Big Data = Technology Thinking of Big Data as a technology rather than as a strategy.
Data for datas sake
Retailers have access to more data than they know what to do with. Tracking Facebook likes or
integrating the Twitter Firehose API is easy. Doing so without a clear defnition of why will only in-
crease the data overload and could create more data silos.
Lack of data readiness
Proceeding with a Big Data project without investing time and efort in auditing the impacted data
sources for data availability, quality and completeness.
POCs will ultimately need to scale across the enterprise.
POCs should not be jigged to succeed, rather used to test readiness, uncover issues and help path
a roadmap to scalability.
Lack of management buy-in
Even a skunkworks project requires C-level alignment to ensure the ensuing results and lessons are
accepted, internalized and scaled quickly.
Lack of analytics resources
Related to the point about thinking of Big Data as a technology intervention. Without the right an-
alytical resources and analytics training for existing associates, projects can fail due to a lack of
proper interpretation and utilization of insights.
EKN 28
Appendix A: EKN Big
Data Need and
Readiness Assessment
The assessment toolkit that follows is an excerpt from EKNs detailed Big Data Need and Readiness
Assessment Toolkit. The scope of the toolkit is too large to provide in this report. However, EKN
invites retailers to customize the provided toolkit for their own use. EKN has also made available
a digital version of this assessment toolkit at the following URL, and invites retailers to receive a
complimentary Quick Assessment Score by submitting their responses on the online questionnaire.
Go to the digital version:
= Analytics Maturity
= Data Management
= Strategic Intent
= Opportunity Qualifcation
= Investment Appetite
= Organizational Readiness
State of the Industry Research Series: Big Data in Retail 29
Appendix A: EKN Big Data Need and Readiness Assessment ToolkitA
Area Question Options Score
How would you rate your organizations
analytics capabilities?
We do reporting -1
We do basic analytics 0
We do investigative analytics 1
We do predictive analytics 2
Compared to your competitors, how
would you describe your organizations
use of customer analytics?
Better than our competitors 2
At par with competitors 1
Lagging our competitors 0
Please rank the top 2 challenges that
prevent you from leveraging analytics
more strategically.
We dont have a clearly articulated analytics strategy linked with
specifc business outcomes
Our software and tools are outdated 1
We dont have adequate resources who can interpret the output of
analytics tools
Our data quality is a big stumbling block before we can analyze it
We have the insights but can do better in terms of delivering it to
the right resource at the right time
Our management style prevents us from making data-driven deci-
A previously failed analytics investment 3
Difculty in measuring analytics ROI 2
How much data do you currently store
across the enterprise?
< 10 GB 0
10 GB 100 GB 1
101 GB 1 TB 2
1 TB 5 TB 3
5 TB 100 TB 4
100 TB+ 5
What is the size of largest dataset that
you currently analyze or plan to analyze
in the next 12-24 months?
< 10 GB 0
10 GB 100 GB 1
101 GB 1 TB 2
1 TB 5 TB 3
5 TB 100 TB 4
100 TB+ 5
Our current Business Intelligence (BI) in-
frastructure is capable of handling your
analytics needs for the next 3 years:
Yes 1
No 0
EKN 30
Area Question Options Score
Which of the following data sources do
you currently integrate to do customer
Store (POS transactional data) 0
Loyalty or CRM data 1
Online 1
Mobile 4
Social media 2
Census data 3
Syndicated data (Nielsen, IRI) 3
Which of the following data sources do
you plan to integrate to do customer
analytics in the next 12-24 months?
Store (POS transactional data) 0
Loyalty or CRM data 1
Online 1
Mobile 4
Social media 2
Census data 3
Syndicated data (Nielsen, IRI) 3
When will you run out of your existing
capacity to store and process data?
I dont know 0
We are already out of capacity -1
Within the next 2 years 2
In the next 2-4 years 3
Beyond 4 Years 2
Never 2
What is the rate of growth (per year) of
your enterprise data?
< 19% 0
20% - 39% 1
40% - 59% 2
60%+ 3
What is your biggest data management
Handling data Volume 1
Handling data Variety 3
Handling data Velocity 2
In your organization what is the highest
level at which Big Data strategy is being
I am not aware 0
CXO Level 3
VP Level 2
Director Level 1
Manager Level 0
Its not being championed by anyone -1
What is your closest competition set
doing about Big Data?
They are actively pursuing Big Data 2
They are investigating Big Data 1
They are not doing anything 0
We dont know 0
Appendix A: EKN Big Data Need and Readiness Assessment Toolkit
State of the Industry Research Series: Big Data in Retail 31
Area Question Options Score
What impact will Big Data have on the
Retail industry?
Completey transform it 4
Have a very high impact 3
High impact 2
Medium impact 1
No or low impact 0
Do your plans over the next 12-24
months include the need to analyze
large sets of social media, mobile or
machine log data?
Yes 1
No 0
I dont know 0
Do your plans over the next 12-24
months include the need to analyze
large sets of unstructured data?
Yes 1
No 0
I dont know 0
Do you plan in the next 12-24 months
to develop and implement advanced
analytics algorithms and apply them to
large data sets?
Yes 1
No 0
I dont know 0
Is your current BI toolset sufcient to
handle your analytics needs over the
next 12-24 months?
Yes 1
No 0
I dont know 0
From a business perspective our com-
pany prefers to be a:
An innovator/leader 3
A close follower 2
A distant follower 1
A reactor to industry conditions and competitors moves 0
From a technical perspective our com-
pany prefers to be a:
An innovator/leader 3
A close follower 2
A distant follower 1
A reactor to industry conditions and competitors moves 0
What are your plans to specifcally
budget for Big Data projects?
I dont know 0
Already have a Big Data budget 3
Within the next 2 years 2
In the next 2-4 years 1
Beyond 4 Years 0
Never -1
Approximately what percentage of your
organizations total technology budget is
allocated towards in BI & analytics solu-
tions this year (2013)?
< 5% -3
5% - 10% -2
10% - 15% -1
15% - 20% 0
20% - 25% 1
25% - 30% 2
30% - 35% 3
35%+ 4
Appendix A: EKN Big Data Need and Readiness Assessment Toolkit
EKN 32
Area Question Options Score
Approximately what percentage of your
organizations total technology budget
will be allocated towards BI & analytics
solutions in 2016?
< 5% -2
5% - 10% -1
10% - 15% 0
15% - 20% 1
20% - 25% 2
25% - 30% 3
30% - 35% 4
35%+ 5
How much are you willing to invest on a
a Big Data Proof-of-Concept in the next
12 months? (Please select)
< 50k
50k - 100k
100k - 250k
250 k - 500k
500k - 1M
Is your organization ready to embrace
public or hybrid cloud for data storage
and analysis?
Yes 1
No 0
Dont know 0
What best describes the decision mak-
ing in your organization?
Complete data-driven 2
Mostly data-driven 1
Mostly intuition-driven 0
Completely intuition-driven -1
How many dedicated analytics re-
sources do you currently have within
the organization? (Please select)
I dont know
1 - 5
5 - 10
10 - 15
10 - 20
20 - 25
More than 25
Do you have plans to invest in acquiring
the analytics skills and resources over
the next 12-24 months?
Yes, we expect to make heavy investments 2
Yes, we expect to make some investments 1
No, we dont plan to make any investments -1
I dont know 0
How is the analytics team in your or-
ganization currently structured?
A dedicated analytics team across all business functions 1
Each department is primarily responsible for their own analytics
IT is primarily responsible for analytics 1
Use contract resources 1
We dont use analytics resources -1
Other model 1
Not aware 0
Appendix A: EKN Big Data Need and Readiness Assessment Toolkit
State of the Industry Research Series: Big Data in Retail 33
Area Question Options Score
What was your organizations annual
revenue in the last fscal year?
< $100 million 0
$100 million - $499 million 1
$500 million - $999 million 1
$1 billion - $4.999 billon 2
$ 5 billion + 3
Which primary retail format does your
organization belong to? (Please select)
Apparel & Accessories
Electronics & Appliance
General Merchandise Stores
Grocery & Food
Health & Personal Care
Home & Furniture
Sporting Goods, Hobby, Book, and Music Stores
*There is no score provided for these questions as the score will be based on the size of the organization and some of the other re-
sponses from the assessment kit.
Appendix A: EKN Big Data Need and Readiness Assessment Toolkit
EKN 34
Recommendations Recommendations
State of the Industry Research Series:The Consumerization of IT: Personal Devices and Software in the Enterprise
Appendix B: EKN Big
Data Vendor Landscape
State of the Industry Research Series: Big Data in Retail 35
Appendix B: EKN Big Data Vendor Landscape
Big Data
Big Data
Big Data
Big Data Specialized Analytics
Semantic Retail


# Also ofers Big Data Appliance
* Ofers with partners
* *
EKN 36
Appendix C:Retail
Honor Board
State of the Industry Research Series: Big Data in Retail 37
Appendix C: Retail Honor Board
Retailer Action uses real-time analytics to create custom cross-sell ofers based on a customers profle.
Using Big Data analytics the online retailer ofers customized content and dynamic pricing to custom-
ers. The retailer allows its competitors to post advertisements for the same product that the customer
is looking at on Data analytics allow it to monitor clicks those promotions produce, and
determine where, when and from what customer segments they are poaching sales. It is able to ac-
cordingly modify its own pricing, marketing and product mix. A dedicated Client Experience Analytics
Team runs customer simulations to measure website latency across the globe, identify trends or issues,
simulate website activity, and more. The simulations are done on a massive scale to mimic the 98 mil-
lion active customer accounts across more than 10 web properties.
Gilt Groupe uses a comprehensive solution consisting of advanced data analysis, visualization and re-
porting on both structured as well as unstructured data. The solution helps Gilt adjust inventory by
analyzing customer clicks, match that information with the characteristics of the days merchandise and
predict which products consumers might be interested in. It then sends members customized alerts
about product ofers based on software-driven recommendations. The retailer updates its predictions
daily and shares detailed demographic portraits with its brand suppliers of whos buying what.
Macys uses advanced analytics to analyze millions of terabytes of data each day, including data from
social media, enterprise systems and store POS systems. The use of best-of-breed technologies helps
Macys to generate deep insight into the data, align customer interactions to customers preferences
and automate processes. Using Big Data analytics it adjusts prices for the entire product range sev-
eral times a day. Time taken to analyze two terabytes of data reduced from 30 hours to less than 120
minutes; churn rate reduced by approximately 20 percent and Macys saved more than $500,000 in
productivity annually in terms of FTE time saved.
Nordstrom, an American upscale fashion retailer, provides customers a more personal shopping expe-
rience by using Big Data analytics to identify what products to promote to which customers when and
via what channel. Nordstrom Data Lab creates products that rely on a wide spectrum of data resources
from within the company and social media to reinforce a consistent brand experience to customers.
Apart from its website and POS, the retailer generates huge amount of data from its 2 million likes on
Facebook, 4.5 million followers on Pinterest, 300,000 followers on Twitter and the Fashion Rewards
Sears Holdings revamped its technology infrastructure by integrating a new software framework into
its data architecture to lower costs, deliver data faster, enhance the speed of processing cycles and
personalize marketing campaigns. The new integrated model allows the retailer to analyze marketing
campaigns for loyalty club members weekly, analyze certain online and mobile commerce scenarios
daily, while using 100% of the available data. It also uses Big Data to set prices - nearly in real-time - and
move inventory by giving loyalty shoppers customized coupons.
Target uses its predictive analytics capability to deduce whether an individual shopper possesses char-
acteristics that make them particularly good targets for a specifc marketing efort. It assigns each shop-
per a unique Guest ID that ties demographics with shopping behavior and preferences. The Guest Mar-
keting Analytics department helps Target to gain competitive advantage from knowing its customers
better than its competitors. An Active Data Warehouse efectively manages complex user queries on
large data volumes in a mixed workload environment across the enterprise.
Wal-Mart launched Wal-Mart Labs in 2011 to gain actionable insights from its digital data and cus-
tomer transactions data. It launched three special Big Data projects: Social Genome allows the retailer
to reach customers or their friends who have mentioned a product online to inform them that it has
the exact product at a discount. ShoppyCat recommends products to Facebook users based on the
hobbies and interests of their friends. Get on the Shelf, a crowd-sourcing solution that provides its vast
customer base the chance to pitch their product idea to a large online audience.
Premier Sponsor
About SAS
In retail, where companies can go quickly from rags to riches - or riches to rags - the customers perception
of, and experience with, your brand is critical not only to staying in business, but growing your business as
well. SAS enables retailers across all segments - apparel, grocery, specialty, general merchandise, online and
hardlines - to uncover in-depth customer insights that you can use to delight customers by engaging them
in proftable ways. With high-performance analytics at the core and backed by more than three decades of
retail experience, SAS solutions for retail enable you to fnd new and better ways of doing business.
About EKN
Our research agenda is developed using inputs from the end user community and the end user community exten-
sively reviews the research before it is published. This ensures that we inject a healthy dose of pragmatism into the
research and recommendations. This includes input of what research topics to pursue, incorporating heavy practi-
tioner input via interviews etc., and ensuring that the bend of research takeaways are oriented towards a real-world,
practical application of insights with community sign-of.
For more information, visit
Email us at
Copyright 2013 EKN
Registered Ofce: 4 Middlebury Blvd. Randolph, NJ 07869
Ph: (973) 607 1300
EKN is part of the Edgell Family
EKN does not make any warranties, express or implied, including, without limitation, those of merchantability and ft-
ness for a particular purpose. The information and opinions in research reports constitute judgments as at the date
indicated and are subject to change without notice. The information provided is not intended as fnancial or invest-
ment advice and should not be relied upon as such. The information is not a substitute for independent professional
advice before making any investment decisions.