2018 IEEE International Conference on Big Data (Big Data)
Big Data Augmentation with Data Warehouse: A Survey
Umar Aftab
Department of Computer Science
Quaid-i-Azam University
Islamabad, Pakistan
Email: umaraftab@cs.qau.edu.pk
Abstract – With dynamic changes in world’s technology, an
increasing growth and adoption observed in the usage of social
media, computer networks, internet of things, and cloud
computing. Research experiments are also generating huge
amount of data which are to be collected, managed and
analyzed. This huge data is known as “Big Data”. Research
analysts have perceived an increase in data that contains both
useful and useless entities. In extraction of useful information,
data warehouse finds difficulties in enduring with increasing
amount of data generated. With shifts in paradigm, big data
analytics emerged as promising area of research which
supports business intelligence in terms of decision making.
This paper provides a comprehensive survey on BigData,
BigData problems, BigData Analytics and Big Data
Warehouse. In addition, it also explains how the need for
augmentation of big data and data warehouse emerged in
perspective of decision making, comparing methods and
research problems. It also elaborates applications which
support Big Data, Data Warehouse, and its challenges.
Keywords—Data Warehouse, Big Data, Map Reduce,
Augmentation, Data Lake, OLAP, CMM, D&M.
I. INTRODUCTION
If we start by looking at the historical perspective of the
growth cycle of data, realizing the pattern how trend
changes in recent times e.g. 1) Relational databases, 2)
Online Analytical Processing (OLAP), 3) Column based
data storage and cloud computing, and 4) Big data
applications [1]. A need arises for handling high data
streaming. Initially, it was addressed by data management/
storage based systems. In recent years, the rate of data
streaming is increasing which is difficult to handle [2].
This paper is organized as follows: next section discusses
quality factors in data warehouse. In Section III, it’s about
origin of knowledge sources and its acquisition.
Furthermore, Section IV discusses performance evaluation
measures for data warehouse. However, Section V
discusses the data preprocessing in data warehouse.
Moreover, Section VI, VII, VIII, IX, and X discusses
analytics, its perspective in data warehouse over BigData,
and its paradigm. In Section XI, XII, it discusses the
competitive insights in perspective of industrial business
implications towards adopting BigData warehouse. Section
XIII concludes with conclusion and future work directions.
A. Introduction to Big data
The huge amount of data which is generated (structured
or unstructured data) is difficult to collect and analyze by
traditional data storages, is called “Big data” [1], [2], [4],
978-1-5386-5035-6/18/$31.00 ©2018 IEEE
Ghazanfar Farooq Siddiqui
Department of Computer Science
Quaid-i-Azam University
Islamabad, Pakistan
Email: ghazanfar@qau.edu.pk
[5]. The major reason behind the rise of data emergence are
tremendous amount of data and ease in its accessibility [2],
[3], [4], [5]. Reasons behind the emergence of big data are:
firstly, is social media (Twitter as the one of the biggest
social media platform) [2]. The most difficult and crucial
job is to build it over big data sets for observing and
perceiving different shifts in user’s behavioral patterns e.g.;
(sharing views, activities, opinions, likeness and
dislikeness) to address different customer needs with
requirements [2], [8]. Secondly, introduction to internet-of-
things (IoT), we are living in a fast, complex changing
environment with increasing number of connections
between humans and smart technologies, that leads us into a
digital world. Data can be generated from multiple sources
by multiple entities. This becomes worse when devices will
be interconnected for taking information [1], [2], [8].
Thirdly, Scientific Computing, in which researchers are
performing various experiments on data that results in data
generation. But extracting useful information to support
decision making from large data repositories looks
challenging in the context of traditional databases for
analysis [6]. Big data is creating a space with provisioning
of data which will strengthen different users to acquire
valuable data. With increase in the probability of making
opportunities by exploration to support decision making in
different social initiatives raised by the different institution
(business and government organizations) [5], [7]. Lastly,
Big data has become a part of the infrastructure for the
developed and under-developed countries in terms of
competing globally towards changing technology
paradigms [5]. In Industry, a question is raised ‘whether
this data is useful in obtaining valuable information?’,
which will be addressed by exploring (big data) and
exploiting (dark data, a data which is considered as useless
data initially) the data to get an in-depth understanding for
new insights [4]. Big data can be described either in terms
of features or challenges. These are, volume (‘data-at-rest’,
collection of data from different sources), variety (‘data-of-
many-forms’, data in multiple formats e.g. unstructured
data), veracity (‘data-in-doubt’, related to importance and
trustworthiness of data from different sources), and velocity
(‘data-in-motion’, data coming at an unprecedented speed
needs to be managed). But later on three more features were
identified which are variability (‘data-in-highlight’,
inconsistencies in data during collection), value (usefulness
of data in perspective of extraction and usage) and
visualization (a demonstration of data in different forms to
2785
get a clear understanding) [4], [8]. With respect to the Table 1: Difference b/w Traditional Data Warehouse and Data
Warehouse on Big Data
growth of data, different integrations and transformations Subject Traditional Data Warehouse Data warehouse in BigData
among data incur new research challenges, which receive a
lot of interest among the research communities [6]. Business • Understand user needs • Understand the
requirements • Access/calculate the architecture
B. Big Data Problems needs for requirements • Understand the user
of data for analysis needs
Big data, an unprecedented growth of data, makes it • Confirmation of the • Access the need for
difficult to store, process and manage. For analysis, we are data availability, data perspective
depending on effective analytical tools and technologies. security, and • Confirmation of data
But with such data growth, it seems difficult to manage by accessibility availability,
accessibility, security
traditional data management tools in terms of scalability Data Analysis • Analyze in terms of • Real-time interactive
[4], [7]. The problem arises in extracting useful information data types, business analysis
in accordance with fulfilling the business case requirements rules, quality, and • Integrated data
from an organizational perspective, which are i) data granularity. • Integrated analysis
volume, ii) data availability, iii) data format, iv) level of • Handle post-added history
requirements
granularity required, and business requirements [5]. Data Modeling • Construction of • Data architecture
(7V’s) relational tables built on the basis of
C. Big data Analytics information
• Define hierarchies
Data analytics are serving as emerging horizon in business • Physical database • Structural
intelligence. With its increasing importance, both in private • Staging schemas infrastructure based
construction. on data architecture
and public sectors regarding insights depends upon the • Any Data format
requirements [7]. Data visualization plays an important part (unstructured,
in big data analytics. It comprises of different techniques structured etc.)
and reports which will give a better picture and helps in Data Movement Data ETL Module ETL(e.g. Apache
Spark) , or ELT
analyzing data [8]. Big data, a data coming from numerous Data Quality Ensure data quality Ensure data and module
dimensions, allows you to adopt a business model or quality
thinking of building a new business model. The major goal Data Data aggregation, Module data aggregation
Transformation summarization, encryption and summarization
of adopting traditional business model or building a new Data Predefined interface for user, Interface based on user
business model, is to find and analyze new insights that will Visualization predefined rules requirements.
support different initiatives and incentives [7]. But
supporting effective decision making depends on market The art of defining data, in different dimensions according
situation and business requirements to get the advantage to different business models and implementing it with
over traditional business models [7]. In view of recent defined methods to execute analytical processes over
advancements in the Data Warehouse and Online BigData. But such analytics requires computational
Analytical Processing (OLAP) to support analytics in big resources such as 1) system with high processing
data has emerged as a new area of research. Complex query performance, 2) scalable enough to handle continuous data
analysis performed to extract valuable information from streaming, 3) elastic to support the management storage
variable data [6]. This focuses on finding and extracting handling 4) compatible infrastructures such as cloud
useful data insights among the researchers. computing with the powerful and flexible support of data
warehouse and OLAP analysis capabilities. But this needs a
D. Big Data Warehousing careful attention and in-depth understanding of platforms
Data analytics requires extraction of valuable information integration among different paradigms. Furthermore, it
to support decision making [7]. But there has been a supports the initial misconception regarding analysis
misapprehension regarding compatibility of data warehouse perceived by researchers which proves that there is no need
architecture with big data. The primary focus behind data of data warehouse with respect to current situation of
warehouse was to support the data processing on static data BigData and supporting the alternatives such as cloud
or data streaming at a very low rate. But this doesn’t mean computing (infrastructure as service supporting intensive
to support the real-time transactional data analysis and data streaming) in terms of scalability, flexibility and
supporting its alternatives [7]. In response to above, real- performance along with other attributes in supporting
time business intelligence is supported by real-time data common goals [6]. In response to this misapprehension,
warehouses. With changing environments, the researchers both architectures are depending upon each other via
have perceived that there will be no further need for data continuous data streaming. In addition, they need
warehouse and relational databases to support big data [4]. specialized sequential integration in which (i) cloud
Data Warehouse has its own importance regarding data infrastructure should act as staging layer in classical data
processing for some formats and it will be helpful in certain warehouse, (ii) staging layer will augment data
conditions. There are some data warehouse tools which management platforms for defining new insights in
provide support for big data [4]. To support the above analytics and enhancing its performance, scalability and
discussion, some points are shown in Table 1. adaptability among these paradigms [6]. But above
discussion doesn’t support all analytical requirements as a

2786
data warehouse is unable to answer maximum number of
questions. To discover and explore different new insights
from continuous data streaming, we need a new direction
[6].
Our survey proposes a new idea of BigData Augmentation
with Data Warehouse, “how big data could be transformed
and integrated with existing data sitting within a data
warehouse to provide an organization with the improved
capability to make strategic and tactical decisions?”.
II. QUALITY FACTORS IN DATA WAREHOUSE
Data Warehouse is an integral part of information
management system e.g. (heterogeneous data sources,
analytical tools). Furthermore, its practice for analysis is
gaining popularity across organizations. But a challenge
arises with the scale of complexity in terms of its
implementation and managing it with the shifting
paradigms in business to gain benefits across organizations.
In recent times, it was not always proved to be a significant
improvement to organizations [10]. Moreover, researchers
are working to provide guidance for its successful
implementation [11]. Also, some studies assessed on
development, implementation and critical success factors of
data warehouse [10]. However, some researchers pointed
out the impact on success factors in the implementation of a
data warehouse still unclear. The theoretical model
discussed here is DeLone and McLean Model (D&M)
updated model [11]. It comprises of measures which will
impact the performance of data warehouse. These measures
are described as follows,
A. Factors Influencing Data Warehouse Implementation
Success
Independent success factors regarding data warehouse
implementation are.
1) Organizational factors
Organizational factors, measures association between
stakeholders and their emphasis on certain factors during
implementation [11]. In recent times, it was supported by
researchers with multiple perspectives. Furthermore, based
on research, organizational factors proved to be successful
in data warehouse implementation. Therefore, current
research suggests precise organizational factors, which are
i) independent authority for taking decisions, ii) strategic
objectives and iii) non-confrontation within the
organization that affects the success of data warehouse
during its implementation.
2) Technical factors
Technical factors, measures limitation, resource
availability, and expertise to handle critical factors
regarding integration of heterogeneous data sources during
the implementation of data warehouse [11]. Therefore,
current research suggests precise technical factors, which
are structural base, data model, effective methodology, and
critical data archive that affects the success of data
warehouse during its implementation.
2787
3) Project factors
Project factors measure the complexity of tasks,
management of roles, and capabilities regarding the project
[11]. Furthermore, some tangible factors are associated with
the project. Therefore, current research suggests precise
project factors, which are planning and management,
human resource capabilities, technical experts, and external
support that affects the success of data warehouse during
implementation.
4) Environmental factors
Environmental factors measure the surroundings with
dynamic shifts in businesses, possibilities for uncertainties,
in order to reduce uncertainties, and utilize the new
information to get a competitive advantage. Therefore,
current research suggests precise environmental factors,
which are the competitive business environment, master
management vendor, and industry standards that affect the
success of data warehouse during its implementation.
5) Infrastructure factors
Infrastructure factors measure the dynamic environment
with shared and tangible resources of information to get a
competitive advantage in enabling present and future
business initiatives among organizations. Therefore,
researchers suggest infrastructure factors, which are reliable
information, advance development applications, and
hardware that affects the success of data warehouse during
its implementation.
B. Factors Influencing Data Warehouse Success
Dependent success factors regarding data warehouse are.
1) Information Quality
Information quality, measures the incoming information,
enabling different dimensions, to define new innovations
for getting a competitive advantage in business [11].
Furthermore, it will provide quantitative metrics to achieve
benchmark performance to measure success ratio.
Therefore, current research suggests information quality
factors, which are valuable information, adequate
information, and information integrity that will affect the
success of data warehouse.
2) System Quality
System quality measures the system competence and
strength of different entities to produce competitive results
[11]. Furthermore, its success is assessed on the basis of its
adoption, ease of use and efficient operations. In addition,
with the production of new information, to support decision
making for both end user and researchers. Therefore,
current research suggests precise factors, which are easy to
use, easily manageable, data positioning, and easy to
interact.
3) Service Quality
Service quality measures the fulfillment of assistance
required for implementation of the information system. But,
service quality will be an information effectiveness measure
[11]. Therefore, current research suggests service quality
factors, which are training manual, trustworthiness, and fast
response.
4) Relationship Quality
Relationship quality is a measure to strengthen already
build relationships and to transform the relation among the
inattentive customers into a devoted one [11]. Moreover,
relationship quality can be taken in different context in
different situations. In addition, the relationship between
business objectives and information technology impacts on
the progress of data warehouse in terms of getting a
competitive advantage. Therefore, current research suggests
relationship quality metrics which are effective trust,
commitment, and user participation.
A blend of implementation success factors and quality
success metrics had been acknowledged for the data
warehouse. In addition, with a better understanding, the
data warehouse can become more effective for analysis
purposes.
III. ORIGIN OF KNOWLEDGE SOURCES AND ITS
ACQUISITION
Data Warehouse is one of the sources for knowledge
acquisition. The objective to steer in the right direction for
analysis and to enhance effectiveness in decision making,
researchers are investigating the role of data warehouse and
its impact on an organization [12]. Research studies have
described the importance of data warehouse, a source for
valuable information to support decision making in
different business processes [12]. Recently, with technology
shift, business market is getting more competitive. As a
result, information system experts are counting on
progressive technology. In parallel with the evolving
environment, it tangibly impacts the nature of decisions.
Next, the organization's intents to acquire valuable
information ahead of time to get a competitive advantage.
To address the inline development described, integration of
information is considered valuable only, when its
availability, quality, pre-defined data sequence, and the
source is known. In addition, in order to support effective
decision making in the organization, it should be
presentable in a pre-defined understandable format to
organizations management so that most of the business
processes will be supported [12]. Moreover, it will be more
effective, if computational resources and their capability of
utilizing information meet business needs. The main
objectives were to acquire valuable information, to support
effective decision making, and gaining competitive
advantage across organizations [12]. However, some issues
have been highlighted in recent research studies regarding
technical issues in the data warehouse. In addition, the
major reasons behind information system employment
failures are e.g. 1) physiological issues, 2) environmental
2788
issues, and 3) organizational issues.
A survey was conducted for measuring adoption of data
warehouse across organizations. The result shows minor
adoption across companies [12].
IV. MEASURES FOR ASSESSING PERFORMANCE OF DATA
WAREHOUSE
In recent times, technology infrastructure has improved a
lot. There has been an increasing interest seen among the
community in contributing towards infrastructure building.
In line with the development, a lot of narratives emerged
from different organizations according to their needs. A
data warehouse is an important part of the infrastructure for
data management. Its adoption requires a huge financial
undertaking. Business organizations strength can be judged
on the basis of its financial standing. Therefore, competitive
business environment makes data warehouse adoption a
critical decision. Keeping in view, that it will help in
providing valuable information to different end users.
Despite these conditions, it can be adopted if it will remain
operational and its performance is accepted by the user.
Hence, concerned management requires a measure for
assessing data warehouse performance. In light of research
[13], balance scorecard approach will provide a
comprehensive view of performance regarding different
perspectives (customer, organizational, development,
innovation, and financial). It will address the critical factors
that will affect the success of data warehouse.
These factors are described as follows:
1) User Perspective
User perspective measures the factors that will strengthen
the relationship between the organization (DW managers,
developers) and analytical community [13]. Research
suggests critical quality dimensions that will influence the
decision making and contribute in achieving user
satisfaction, which are i) service effectiveness, ii) service
offering, iii) data consistency, iv) data compliance, and v)
robustness. With these, user satisfaction will be ensured in
the adoption of the data warehouse.
2) Organizational Business Significance Perspective
In line with development described above, business
organizations need financial undertakings in acquiring
resources or shifting technologies [13]. In view of financial
standing, adoption of a data warehouse requires a lot of
investment including different operational and maintenance
expenses (hardware, software, licenses). But, before
adoption, top management assesses whether it will be
beneficial for the organization in terms of lowering the
deficit in investment annually. To accomplish this, data
warehouse champions need to ensure its effectiveness in
achieving business goals. Defining the financial statement,
a measure to assess the organizational strategic role
contributing towards successful implementation and
execution to achieve a certain level of improvement. In
recent times, research suggests some factors, which are i)
data warehouse architects (evaluation and identification of
bottleneck and in-efficiencies in operations), data
warehouse champions, a good negotiator (chief technical
officer (CTO) and manager’s role in decreasing annual
expenses), risk aversion (hardware, software tools,
licenses)) that will play its part in successful data
warehouse adoption among organizations.
3) Internal Process Perspective
In line with the discussion above, top management needs
some measuring method to assess data warehouse
performance in the organization [13]. To accomplish it,
balance scorecard is regarded as a measure to provide a
comprehensive view of performance to managers.
Moreover, it will help them focus on critical operations to
gain more customers satisfaction level. But, some factors
regarding data warehouse internal operation needs to be
assessed. As they will be considered as critical factors,
which are i) Extract-Transform-Load (ETL) code
performance (stored procedure, views, report SQL), ii)
batch cycle execution, iii) analytical reporting, iv)
responsive progress, v) verification, vi) validation, vii)
distribution, viii) data warehouse scalability and stability
(Capability Maturity Model (CMM) provides useful
guidelines for improvement), and ix) data availability as per
DeLone and McLean Model (D&M) [14].
4) Business Innovation and Growth
In line with the development described above,
improvements in technology nowadays are getting attention
in research communities [13]. In recent times, technology
and business are changing quite fast. An organization
strength can be seen as its ability to learn the gaps, innovate
and improve itself to compete with the environment.
Talking about innovation and growth in data warehouse, it
will help in providing better reporting capabilities and
capabilities in exploring and providing better business
intelligence insights. Hence, research suggests some factors
(technology leadership, short project life cycle, efficient
test-driven development, process improvement, innovative
query writing, emerging technology learning, and increase
automation) that will play a part in the successful adoption
of data warehouse among organizations.
In line with the development described above, Nayem
Rahman [13] proposed a balanced method for measuring
effectiveness in different operational perspectives of data
management. This approach will provide a comprehensive
view of operations to the concerned management. With this
approach, key operational areas have been identified, which
are important regarding implementation. Furthermore, some
crucial measures for delivering operational excellence have
been identified for the data warehouse managers and
architects, which are i) industrial worth formation, ii)
budget loss prevention, iii) improvement, iv) requirement
fulfillment, v) in-house processes, vi) effectiveness and vii)
consistency that will be under consideration for
organizational benefit.
2789
V. DATA PRE-PROCESSING IN DATA WAREHOUSE
Data Warehouse is a platform designed for multiple
purposes. It comprises of different steps including pre-
processing as a pre-requisite before storing, scheduling and
management. Furthermore, complex queries are used for
analyzing and summarizing the organization's data in
Online Transaction Processing System (OLTP). Anosh
Fatima et.al. [15] explores different data pre-processing
techniques in data warehouse applicable to different
conditions and the problems raised.
A. Dynamic Study
The study presented in [15] comprises of industrial
experiment results. The motivation behind it was to
categorize the precise selection of information management
application platforms in terms of requirements, situation
and the benefits that can be gained from it [15]. Tools
considered here are Yale, Weka, and Alteryx.
1) Problems in Selection of Sources and Schemas
There are two types of sources and schemas
considered here
I. Single Sources Problems
In architecture, the absence of principle limitations, some
issues highlighted are i) uniqueness property is not
considered here, ii) referential integrity property is violated.
On instance level, errors highlighted are i) misspelled
words, ii) redundant words, iii) duplication, and iv)
contradictory values.
II. Multi-Source Problems
On schema level, problems involve heterogeneous data
models and schema design issues, one highlighted is
structural type conflict. On instance level, problems
involve overlapping and inconsistent data issues.
2) Data Pre-Processing
This study considers data pre-processing approaches i)
Classical Pre-Processing (cleaning, fusion, and structuring)
and ii) Advanced Pre-Processing (summarization) [15].
I. Data Cleaning Approaches
Data cleaning, a process that helps in preparation of data to
make it useful. This process includes data analysis, data
definition, transformation, verification, mapping rules and
back data flow analysis [15]. Research suggests two
appropriate approaches to data cleaning 1) Record
Matching, 2) Data Repairing to improve the performance of
data warehouse [16]. However, Query Optimization is
another factor for improving the performance of decision
making in data warehouse [16]. To make data warehouse
performance more effective, researchers suggest
Materialized View (pre-calculated end results of queries to
improve time efficiency) and Automated Child Views but
both approaches need more time to evolve enough to
optimize the execution.
 a) Materialized Views (MV’s)
Data analysis is considered to be an integral part of the data
warehouse. But, for analysis, some factors are considered
important, which are i) query processing time, ii) effective
query writing, and iii) fast pre-processing. To accomplish
all above, research suggests materialized views method.
This method comprises of estimated outcomes of analytical
queries to provide fast processing [16]. Usually, queries are
executed with joins and aggregation. But, materialized view
eliminates joins and aggregates. In addition, during
execution of queries, it stores results in cached tables which
are constantly updated till the processing remains for quick
access. The benefit of achieving development is to reduce
query processing time and frequent querying can be
executed in no time. Moreover, Cached tables can be
altered by indexing to increase execution speed.
Outlier detection and its elimination, are other factors that
play an important part in data cleaning [16]. In recent times,
past work in outlier detection is on numeric data. But
current study proposes a hybrid approach for outlier
detection [16], which comprises of data mining techniques
i) weighted k-means and ii) neural network for cluster
formation [16]. Moreover, another technique used for
outlier detection is integrated semantic knowledge (SOF),
which detects semantic outliers (a point which behaves
differently from others in a cluster) [16]. But, it also
requires improvement in symbol modification both in text
and numeric data.
II. Approaches for Missing Data
Data can be collected from heterogeneous data sources. It
may have missing attribute values while training the data.
Either the missing values are not recorded properly or
ignored in classification due to confidentiality constraints.
In recent times, research suggests three methods for data
pre-processing in missing data, which is the elimination of
constants, elimination with attribute mean, and elimination
of random values.
III. Framework
In recent times, research suggests some essential framework
and attributes that will help in its evaluation. Associations
between data acquisition, data quality, and data cleaning are
depending on each other. In light of research [16], some
data quality issues highlighted are i) completeness in data,
ii) accuracy in data, iii) validation of business procedures
and guidelines, iv) precision, v) non-duplication, vi)
derivation integrity, vii) accessibility, and viii) timeliness.
In addition, some data cleaning issues highlighted are i)
data entry error, ii) data integration error, and iii)
measurement error. Moreover, some data acquisition issues
highlighted are i) data conversion from heterogeneous data
sources, ii) redundant data removal, and iii) data
transformation.
In line with the above discussion, proposed frameworks are
Total Data Quality Management, Rule Base, and Log
2790
Management. Framework evaluation factors are i) integrity,
ii) precision, iii) validation, iv) repeated words elimination,
v) openness, and iv) data access time.
VI. ANALYTICS OVER BIG DATA
In line with the emerging advancements in the data
warehouse for improving analytics to support effective
decision making. Using analytics over big data, complex
procedures are executed to extract useful data. To execute
analytics over big data, it requires a system with high
performance, scalable, and elastic infrastructures. As a
consequence of supporting analytics, research challenges
arise at the convergence between a data warehouse and
cloud infrastructures [17]. Tools considered here are
Hadoop, Hive and Cloud Infrastructure.
A. Research Challenges
 Data preparation from variable data sources. As a
consequence, data can be populated using OLAP
[17].
 The issue of data cleaning and management in data
extractions to support Business Intelligence (BI)
with complex analytics view for effective decision
making [17].
B. Arguing Comments
In light of research challenges discussed here, a number of
questions are highlighted as follows;
 “Considering data-intensive cloud infrastructures
as today’s alternative of data warehouse”
 In response to above argument, research
community has given a few suggestions in [17],
which are as follows,
 Cloud will be considered as data staging
medium in data warehouse [17].
 “This staging area should populate the
subsequent Data Warehousing layer and
OLAP layer to define complex analytics
systems that are more and more powerful,
adaptable and scalable.”
C. Research Future Challenges
 Multi-Structural Support in Hadoop.
 Integration of live data streaming sources in
Hadoop
 Modification of Hive Query Language (HiveQL)
to support multi-structured data.
 The analysis in Hadoop over live streaming data.
 Multi-visualization representations supporting
multi-structural analysis.
VII. ANALYTICAL PERSPECTIVE OF DATA WAREHOUSE
OVER BIG DATA
Data Warehouse and Big Data, what separates them is the
size of data, data sources, data dimensions, data streaming
rate and analytical method. In order to make them a
compatible with each other, a framework needs to be
designed, Lihua Sun et.al. [18] analyzes and summarizes
three architectures i) Parallel Database, ii) Map Reduce
Framework and iii) Nested architecture, which makes data
warehouse suitable with big data analysis. Platform
considered here are Hadoop (MapReduce) and Parallel
Database (Relational DB).
Reasons behind considering architectures discussed above
are as follows,
 Data preparation and management of cost
expenses.
 Multiple Joins and Aggregations.
 Data re-positioning according to data model
 Static data splitting
A. Architectures Under Observation
First is a Parallel Database dominated architecture [18]. The
output of Map-Reduce is input to query as well as query
output can be input to the Map-Reduce algorithm. Problems
highlighted are fault tolerance and Scalability.
Second is a Map-Reduce dominated architecture [18].
HiveQL is an interface to analytical operations over data.
Whereas, PigLatin language uses data operatives in
providing the interface for the data stream.
Third is a Nested architecture, in which representative
systems are Hadoop DB, and Vertica [18]. It uses Hadoop
for task distribution over the network and relational
database as a medium for query processing. Its problems
include scalability issue in Vertica regarding structured data
and performance issue in Hadoop.
In general, problems highlighted are low preprocessing of
Map-Reduce because of its batch processing nature and
dynamic data redistribution of the relational database.
B. Research Directions
 Pre-computing of multi-dimensional data
 Parallel analysis implementation
 Query sharing
 User interface
 Multi-dimensional indexing
VIII. EXPLORING PARADIGM OF BIG DATA WAREHOUSE
Big Data Warehouse is a new software paradigm proposed
in [19], which seems to be quite different from traditional
data warehouses in terms of schema, design methodology,
principles, flexibility and factors regarding realization and
changing requirements adaption. The main objective of
exploring big data warehouse is to find new models
regarding new insights [19]. Furthermore, exploring and
exploiting (big data and dark data) requires the innovative
insight of data warehouse solutions. To do so, data analytics
must be devoted to improving decision making. The areas
discussed here are i) marketing analysis, ii) tweets and
blogs, iii) public sector (intelligence services), and iv)
public administration.
A. Contribution in Analytics
An innovative methodological solution for the
architecture of big data warehouse proposed in [19]. The
model is based on a key-value approach for the
representation of multidimensional definition at the
logical layer. It also discusses and motivates the approach
2791
of data analysis with new paradigms. But, considering the
incoming data management requirements, it performs
well in comparison to previous traditional method [19].
B. Proposed Approach
Francesco Di Tria et.al. [19] proposed a method called
GrHyMM Model. This is a key-value based model,
preferred with unstructured data, without joins. To do so,
data warehouse needs to extend itself to new data sources.
It comprises of following steps,
 Source Integration
 Semantic Ontology definition and its
representation
 Building predicate logic for vocabulary
 Entity description generation i.e. logic-
based description according to concepts
of database
 Similarity Comparison i.e. similarity
comparison rules for analyzing the logical
description
 The conceptual definition of rule
generation.
 Requirement Analysis i.e. adopting the I*
framework, represents business goals considering
the actors i) decision makers, ii) data warehouse
 Conceptual Design i.e. multidimensional model
for graphical oriented representation of relational
database
 Requirement analysis (attribute tree based
on facts, reduced on derived constraints)
 Integrated schema (detecting relations for
identifying facts)
 Re-engineering
 Logical Design
 Incremental step (exploring new facts with
changing requirements and data sources)
 Physical Design.
IX. DATA LAKE; DEFINITION
Data lake has emerged as a powerful architecture in a time
when most of the organizations are shifting to mobile,
cloud-based or internet of things (IoT’s) in context of an
increase in volumes of data and market growth [20]. Data
lake is a central repository for storing data regardless of its
source, size, and format. Data might be structured or
unstructured, a lot of tools regarding storage and processing
relating to extended big data ecosystem, are considered in
order to gain efficient access to important data and to
support organization decisions [20]. Its difference with
traditional data warehouse is shown in Table 2.
Table 2: Difference between Traditional Data Warehouse
and Data Lake
Traditional Data Warehouse Data Lake
1. Pre-defined Schema No specific Schema
1. Enterprise form Native form
2. Pre-Processing No Pre-Processing
3. Schema-on-Write Schema-on-Read
 4. Conformed Data Model No specific Data Model
5. Less Flexible More Flexible
6. Limited Questions More Questions Answered B. Architecture Review
Answered
Reference architecture of enterprise data warehouse is
shown in Figure 1, its factors proposed in [21] are described
A. Advantages of Data Lake in above sections.
 Ability to derive valuable insights from
heterogeneous data sources.
 Ability to acquire data in any format and store data
in a general format in data lake, e.g. from
Customer Relationship Management (CRM) data
to social media posts.
 Ability to refine it as per understanding,
requirements, and insight.
 Ability to explore complex query analysis on data. Figure 1: Enterprise Data Warehouse Architecture
 Availability of tools to gain deep insights from the
data. The significant difference between enterprise data
 Elimination of data silos warehouse and data lake architecture is the input data
 Democratized data access via a single, unified format, data lake takes data in native format while data
view of data across the organization when using an warehouse takes data in a pre-defined schema format [21].
effective data management platform. Now referencing the enterprise data warehouse as shown in
B. Characteristics of Data Lake Figure 1, complementing it with data lake as shown in Figure
2.
 Single shared data repository (Distributed File
System)
 Includes orchestration and job scheduling (central
operational management)
 Contains a set of applications and workflows to
consume, process or act upon data (data
preservation).

Figure 2: Data Warehouse Complementing Data Lake

X. COMPLEMENTING DATA LAKE WITH DATA WAREHOUSE
In recent times, data lake emerged as an essential In line with the development, competing with the business
development in data management [21]. It provides support environment, organization procures data lake to offload
to enterprise factors. In addition, these are imperative in costly data processing [21], to get benefit in terms of data
enterprise data management. In view of the competitive access and analysis as shown in Figure 3.
environment, research suggests a need of complementing
enterprise data warehouse (EDW) with data lake. In
consequence of the development, it will provide i) more
flexibility, ii) fast data processing, iii) capturing continuous
data streaming, and iv) frees up bandwidth for Business
Intelligence (BI) analytics.
In parallel with above emergent narrative, “managed” data
lake uses a data lake management platform to perform
operations such as i) ingestion, ii) apply metadata, and iii) Figure 3: Data Warehouse with Data Lake (offload the
enable data governance. As a consequence, a user with his processing)
competence can utilize available information.
Complementing data lake makes no change in data
warehouse [21]. Moreover, data lake is also flexible with XI. DEFINING COMPETITIVE INSIGHTS IN BUSINESS
data governance rules based on data ingestion approach. In recent times, the emergence of big data has increased the
competitiveness of a business environment. To compete
with challenges, organizations are shifting dimensions to
A. Reasons behind Complementing Data Lake with Data
sustain their current position in the market. For achieving
Warehouse
something, you have to withstand difficulties [22].
 Blue sky - to be able to explore more about Organizations competence is based on how they utilize the
enhancing data warehouse. opportunity by exploring and exploiting big data to gain
 Cut costs - reducing costs by leveraging competitive benefit over others [22]. In respect of the
commodity hardware. business environment, organizations need to keep an eye on

2792
emerging challenges and ongoing customer demands
regarding different insights [22]. For exploring new
insights, first, the data needs to be relevant and consistent.
With the adoption of changing technology infrastructure,
the benefit can be gained by integrating it with the current
infrastructure to enhance organizational capabilities.
A. Guidelines for Effective Business Insights
 Initiate steps towards emerging technology for
business growth.
 Enabling multi-structural data streaming from
variable sources
 Establish a platform for supporting analytics, by
considering it a key performance indicator in
achieving success.
 Ensuring principles for exploiting information,
data security and confidentiality.
 Invest in human resource training in defining new
insights in business and making decision making
effective.
XII. INDUSTRIAL PERSPECTIVE TOWARDS BIG DATA
WAREHOUSE
A. Data and Analytics: Organizational Prospect
Data and Analytics are an integral part of innovative
technology adopted by organizations. But having said
that, some factors are important in the successful
implementation of data and analytics such as i)
appropriate tools, ii) data champions, iii) significance of
organizational components, and iv) clear strategy.
Cultural resistance is also considered as one of the
reasons behind data and analytics failure across
organization [23]. In support of above argument, Gartner
Inc. has presented a calculated estimation of data and
analytics project delay most of the times. The reasons for
project delay are i) organizational structure, ii)
appropriate talent acquisition, and iii) non-alignment with
business strategy.
B. Information Governance Rules
Data streaming is increased with the presence of different
circles of information [24]. ‘Dark Data’, a data that we have
but don’t know what to do with it. Gartner Inc. defines
‘Dark Data’ as “information asset that organization acquires
and pre-process as per business activities, but fail to utilize
it for some purpose”. ‘Dark data is useful if it is managed
properly’, otherwise it will be problematic for the
organization. To do so, six governance rules proposed in
[24], organizations have to adapt to manage dark data
properly, as follows,
 Define and identify your dark data i.e. determine
the source of data user-generated or system
generated. It does not guarantee that processing
will pay off in the end
 Cost-benefit analysis of data usefulness (short
retention time frame)
2793
 What to keep and what to discard (business
requirements)
 Data mapping during retention and disposition to
justify deletion (addressing categorization during
retention)
 Secure Disposition Plan Execution
 Determine the benefit value for dark data
C. Query Processing for Big Data
In recent times, big data environment is expanded with the
growing technology insights [25]. But, Big Data processing
is initially based on Map-Reduce, which is a batch-oriented
processing. As a consequence, data processing is slow and
limited to data-at-rest. To overcome this issue,
organizations are shifting to SQL based Hadoop [25].
Furthermore, these are analytical application tools, which
combine SQL query style and Hadoop framework to speed
up analytical queries and data integration simpler. Its
execution consists of some steps, which are i) connectors
for translating SQL query in map reduce format, ii) ‘push
down’ that forgo map reduce and execute SQL with
Hadoop cluster, and iii) job distribution of allocating SQL
procedures among different nodes of data clusters. Premier
Inc., a healthcare information system provider, has shifted
from data warehouse infrastructure to Hadoop cluster [25].
Furthermore, because map-reduce is batch-oriented
processing which doesn’t support web-based business
intelligence dashboard. To overcome this issue, the Premier
Inc. shifted to impala and Cloudera Inc. SQL based Hadoop
cluster, which provides faster Hadoop query performance
and similar framework to previous Hadoop framework [25].
‘Using the SQL syntax optimizes our development work.’
said by a senior representative of the organization [25].
Overall on adopting SQL based Hadoop, saves a lot of cost
on higher processing platforms.
XIII. CONCLUSION AND FUTURE WORK
In this survey, Big Data, its problems, Insights in analytics,
Big Data Warehouse and its challenges have been
reviewed. Besides, it explains data warehouse and its
related literature. Finally, it sums up with the approaches of
big data augmentation with data warehouse, its challenges
in data and analytics to support effective decision making
along with industrial perspective. The motivation for this
literature is to support the research initiative, ‘how can we
integrate and transform big data with the data sitting inside
data warehouse to perform effective decision making’.
In future work, our focus will be not only be restricted to
the integration of data management platform to improve
decision making. But, also on the business models that will
have certain significance in analytics. This survey also
overviews the prospects that support the business analytical
dynamics. First is predictive analytical models, which is a
source to steer towards effective strategies for business. It
will help in forecasting and predict future results [26], [27].
With its integration with analytical applications, it can
improve the business decision making [28]. Second is Data
Quality, usually, business decision making is data-driven.
But, with an excessive amount of data incoming, it may
lose its objectivity, so more measures need to be explored
in order to get good data prepared for analysis [29], [30].
REFERENCES
[1] Mehdi Gheisari, Guojun Wang, Md Zakirul Alam Bhuiyan,
“A survey on deep learning in big Data”, presented at IEEE
International Conference on Computational Science and
Engineering (CSE) and IEEE International Conference on
Embedded and Ubiquitous Computing (EUC), 2017.
[2] Gore Sumit Sureshrao, Ambulgekar H. P., “MapReduce-
Based warehouse systems: a survey”, presented at IEEE
International Conference on Advances in Engineering &
Technology Research (ICAETR - 2014), August 2014.
[3] A. Sunny Kumar, “Performance analysis of mysql partition,
hive partition-bucketing, and apache pig”, IEEE 2016.
[4] M. Ptiček and B. Vrdoljak, “MapReduce research on
warehousing of big data”, MIPRO 2017, May 22- 26, 2017.
[5] Hai-Fei QIN, Zhi-ming QIAN, Yong-chao ZHAO, “On the
research of data warehouse in big data”, presented at
International Conference on Network and Information
Systems for Computers,2015.
[6] Alfredo Cuzzocrea, “Analytics over bigdata exploring the
convergence of data warehousing, olap, and data-intensive
cloud infrastructures”, presented at IEEE 37th Annual
Computer Software and Applications Conference, 2013.
[7] Francesco Di Tria, Ezio Lefons, and Filippo Tangorra,
“Design process for big data warehouses”, presented at Data
Science and Advanced Analytics (DSAA), International
Conference, 2014
[8] M.D. Anto Praveena, Dr. B. Bharathi, “A survey paper on big
data analytics”, presented at International Conference on
Information, Communication & Embedded Systems (ICICES
2017).
[9] Nayem Rahman,” An empirical study of data warehouse
implementation effectiveness”, presented at International
Journal of Management Science and Engineering
Management, 2017 Vol. 12, no. 1, 55–63.
[10] R. L. Hayen, C. D. Rutashobya, and D. E. Vetter, "An
investigation of the factors affecting data warehousing
success," International Association for Computer Information
Systems (IACIS), vol. 8, pp. 547-553, 2007.
[11] AlMabhouh, A., & Ahmad, A, “Identifying quality factors
within the data warehouse”. Proceedings of the Second
International Conference on Computer Research and
Development, IEEE, 65-72.
[12] Moh’d Alsqour, Mieczyzslaw L. Owoc, Abdulrahman S.
Ahmad, “Data warehouse as a source of knowledge
acquisition. an empirical study”, Proceedings of the 2014
Federated Conference on Computer Science and Information
Systems pp. 1421–1430, ACSIS, Vol. 2.
[13] Nayem Rahman, “Measuring performance for data
warehouses - a balanced scorecard approach”, Copyright ©
2013 IJCIT, ISSN 2078-5828 (Print), ISSN 2218-5224
(Online), Volume 04, Issue 01, Manuscript Code: 130701.
[14] W. DeLone and E. McLean, "The Delone and McLean Model
of Information Systems Success: A Ten-Year Update,"
Journal of Management Information Systems, Vol. 19, No. 4,
pp.9-30, 2003.
[15] Christen, P, "A Survey of Indexing Techniques for Scalable
Record Linkage and Deduplication." IEEE Transactions on
Knowledge and Data Engineering24(9): 1537-1555, 2012.
[16] Anosh Fatima, Nosheen Nazir, Muhammad Gufran Khan,
“Data cleaning in data warehouse: a survey of data pre-
2794
processing techniques and tools”, I.J. Information
Technology and Computer Science, 2017, 3, 50-61 Published
Online March 2017 in MECS (http://www.mecs-press.org/).
[17] Alfredo Cuzzocrea, “Analytics over big data exploring the
convergence of data warehousing, olap, and data-intensive
cloud infrastructures”, 2013 IEEE 37th Annual Computer
Software and Applications Conference.
[18] Lihua Sun, Mu Hu, Kaiyin Ren, Mingming Ren, “Present
situation and prospect of data warehouse architecture under
the background of big data”, 2013 International Conference
on Information Science and Cloud Computing Companion.
[19] Francesco Di Tria, Ezio Lefons, Filippo Tangorra, “Design
process for big data warehouse”, Data Science and Advanced
Analytics (DSAA), 2014 International Conference.
[20] Zaloni Inc., “Defining the data lake”, A White Paper
[Online], Available: “https://resources.zaloni.com/white-
papers/defining-the-data-lake-white-paper”.
[21] Zaloni Inc., “Why your data warehouse needs a data lake and
how to make them work together”, A White Paper [Online],
Available: https://resources.zaloni.com/white-papers/dw-
augmentation-white-paper
[22] Capgemini, “Big & fast data: the rise of insight-driven
business; insights at the point of action will redefine
competitiveness” [Online], Available:
https://www.capgemini.com/resources/big-fast-data-the-rise-
of-insight-driven-business/
[23] Carl Carande, Paul Lipinski, Traci Gusher (2017, June),
“How to integrate data and analytics into every part of your
organization”, Available: https://hbr.org/2017/06/how-to-
integrate-data-and-analytics-into-every-part-of-your-
organization.
[24] Fred A. Pulzello (2014, September), Six steps to 'dark data'
information governance. ARMA International [Online],
Available: http://searchcompliance.techtarget.com/tip/Six-
steps-to-dark-data-information-governance.
[25] Craig Stedman (2016, April), SQL engines boost Hadoop
query processing for big data users. Search Business
Analytics [Online], Available:
http://searchbusinessanalytics.techtarget.com/feature/SQL-
engines-boost-Hadoop-query-processing-for-big-data-users.
[26] A. Yu. Dorogov, “Technologies of predictive analytics for
big data”, 2015 XVIII International Conference on Soft
Computing and Measurements (SCM).
[27] Mykoa Pechenizkiy, “Predictive analytics on evolving data
streams”, 2015 International Conference on High
Performance Computing & Simulation (HPCS).
[28] Parth Wazurkar, Robin Singh Bhadoria, Dhananjai Bajpai,
“Predictive analytics in data science for business intelligence
solutions”, 2017 7th International Conference on
Communication Systems and Network Technologies
(CSNT).
[29] Pengcheng Zhang, Fang Xiong, Jerry Gao, Jimin Wang,
“Data Quality in Big Data Processing: Issues, Solutions and
Open Problems”, 2017 IEEE SmartWorld, Ubiquitous
Intelligence & Computing, Advanced & Trusted Computed,
Scalable Computing & Communications, Cloud & Big Data
Computing, Internet of People and Smart City Innovation
(SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI).
[30] Ikbal Taleb, Mohamed Adel Serhani, “Big Data Pre-
Processing: Closing the Data Quality Enforcement Loop”,
2017 IEEE International Congress on Big Data (BigData
Congress).