See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/335101436

A Framework for Data Quality in Data Warehousing

Chapter · August 2019

CITATIONS READS
0 1,464

2 authors:

Rao Nemani Ramesh Konda

College of St. Scholastica International Technological University
16 PUBLICATIONS   60 CITATIONS    7 PUBLICATIONS   86 CITATIONS   

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Rao Nemani on 10 August 2019.

The user has requested enhancement of the downloaded file.

 A Framework for Data Quality in Data Warehousing
Rao R. Nemani
University of St. Thomas, Minneapolis, MN, USA
Nema8811@stthomas.edu
Abstract
Despite the rapid growth in development and
use of Data Warehousing (DW) systems, the data
quality (DQ) aspects are not well defined and
understood. Organizations rely on the information
extracted from their DW for their day-to-day as well as
critical and strategic business operations. Creating and
maintaining a high level data quality is one of the key
success factors for DW systems. This article examines
the current practices of DQ, and proposes a research
and experienced-based framework for DQ in DW, which
describes an approach for defining, creating, and
maintaining DQ management within DW environment.
1. Introduction
. quality IS that meets user needs is becoming a critical
In today’s global and competitive business
environment, organizations rely on data and information
to make timely decisions and to offer innovative
services to meet changing needs of their customers.
Typically, organizations collect data through various
systems and try to create coherent and aggregate data
for decision-making purpose. Bryan (2002) states that
fast growth of electronic commerce in the last decade
registered a lot of business critical and confidential data
being exchanged online among companies and
customers. At the same time, rapid advancements in
storage technology made it possible for the
organizations to store vast amounts of data is collected
at a relatively low cost. However, this data failed to
provide knowledge because of its raw form. To solve
this problem, organizations adopted a transformation
process called a data warehouse, which is defined as a
“collection of subject-oriented, integrated, non-volatile
data that supports the management decision process”.
Typically, a data warehouse contains five types of data:
current detail data, older detail data, lightly summarized
data, highly summarized data, and metadata. Nowadays,
data warehousing has become an integral part of both
business and Information Technology strategies to
integrate heterogeneous information sources, and to
enable On-Line Analytic Processing (OLAP) and
Decision Support Systems (DSS).
There has been great progress and improvement
in core technology of DW; however the DQ aspects are
one of the crucial issues that were not adequately
Ramesh Konda
University of Phoenix, Phoenix, AZ, USA
Konda1991@yahoo.com
addressed. Ensuring high-level DQ is one of the most
expensive and time-consuming tasks to perform in data
warehousing projects. Many data warehouse projects
have failed halfway through due to poor DQ. This is
often because DQ problems do not become apparent
until the project is underway. Any changes to DW at the
implementation stage cost a substantial amount that may
push project budget limits. If all the considerations are
examined thoroughly at the strategy and design stage of
DW, controls can be formulated into design for DQ that
can decrease operational costs, increase customer
satisfaction, and improve effective decision-making and,
employee confidence in using the data (Andrea &
Miriam, 2005). The quality of information systems (IS)
is critically important for companies to derive a return
on their investments in IS projects, and the DW is no
different in that sense. Therefore, developing good
theme for information technology management
(Guimaraes, Staples & McKeen, 2007).
In this paper, the authors examine the current DQ
practices in DW, and propose a framework for
improving the quality of data in DW environment. In
the Section 2, the problem statement around which this
paper offers a solution is discussed. In Section 3, a brief
literature review is discussed. Following which in
Section 4, a framework for the DQ in DW is presented.
Section 5 describes the typical use case. The last section
summarizes the paper.
2. Problem Statement
Literature review related to issues with DW reveal
that DQ is one of the most prominent issues. It is not
known how and to what extent the DQ issues are
affecting the DW implementation. Poor data quality
increases operational costs, adversely affects customer
satisfaction, and has serious ramifications on the society
at large. Some examples of poor DQ can be seen in
Nord’s work (2005) that describe that billions of dollars
have lost due to the fraud in food stamps wherein some
of the recipients received the benefits even after they
were long dead. Further Nord states that due to
inaccurate and outdated information in data systems,
organizations have experienced financial losses. Data
quality problems range from a minor insignificant to
major problematic issues (Bielski, 2005).

3. Literature Review
The science of DQ is yet to be advanced to the point
where standard measurement methods can be devised
for any of these issues. Inappropriate, misunderstood, or
ignored DQ has a negative affect on business decisions,
performance and value of data warehouse systems.
English (2001) argues that managing quality of your
information is equally important as managing your
business. English (2001) listed several examples in his
paper that draw attention to the negative impact of the
DQ issues in DW. Some of them include errors in
students Basic Standards Test scores, pension
withholdings, invoicing, and food processing that led to
the loss of billions of dollars as well as loss of
reputation of those businesses.
In a survey by Friedman, Nelson, and Radcliffe
(2004), it was stated that 75 percent of survey
respondents reported significant problems stemming
from defective and fragmented data, over 50 percent
have incurred cost for data reconciliations, and 33
percent were delayed IT systems owing to data quality
problems. In another survey, Ambler (2006) reported
several metrics from the response that indicate Data
Quality has been the major issue and requires
considerable attention to solve this problem. For
example, the following chart illustrates only 2 percent of
the respondents feel good about the data quality in their
data warehousing and the remaining 98 percent indicate
some kind of data quality issues that need be addressed.
Figure 1. Current State of Data Quality (Ambler, 2006)
One of the major factors of influencing the DQ is
user perception over time. If user assumptions or
perceptions are unchecked, then it starts to become ‘the
truth’ whether or not it has an objective or factual basis,
from both business and technical perspectives (Bryan,
2002). DQ indicates how well enterprise data matches
up with the real world at any given time.
There are many sources of ' dirty data', which
includes a) poor data entry, which includes misspellings,
typographical errors and transpositions, and variations
in spelling or naming, b) data missing from database
fields, c) lack of company-wide or industry-wide data
coding standards, d) multiple databases scattered
throughout different departments or organizations, with
the data in each structured according to the rules of that
particular database and e) older systems that contain
poorly documented or obsolete data (Andrea & Miriam,
2005). Nord (2005) mentioned that the DQ has become
an increasingly critical concern and it has been rated as
a top concern to data consumers in many organizations.
Nord (2005) continued stating that the data quality is
gaining its importance in the research and among the
consumer organizations.
Above discussion indicates that DQ issues are
critical and need to be addressed in order to be
successful with DW environment.
4. A Framework for Data Quality in Data
Warehousing
Iain and Don (2005) argue that in order to tackle this
difficult issue, organizations need both a top-down
approach to DQ sponsored by the most senior levels of
management and a comprehensive bottom up analysis of
data sourcing, usage and content including an
assessment of the enterprise' s capabilities in terms of
data management, relevant tools and people skills. Xu,
Nord and Brown (2002) believe that for organizations
considering implementing of DW, it is essential that DQ
issues be thoroughly understood and the organizations
should obtain knowledge of the critical success factors
essential to ensure DQ during the implementation
process. However, the crucial question of defining data
quality is often ignored until late in the process. Jean-
Pierre (2004) believes that this could be due to the lack
of solid methodology to deal with DQ. Quality is a
relative statement and varies by individuals based upon
their perceptions. In simplistic data quality is perceived
as “true and accurate”. This makes DQ hard to define
and measure. To understand how to tackle the problem,
DQ needs to be understood thoroughly from the
organizations point of view, and then a process can be
established to deal with DQ within their organization.
The main components of data that determines the
DQ are, completeness, appropriateness, accuracy,
grouping accuracy, access, confidence, currency,
regulators and legal compliance and meta-linking. Data
interface, data replication and data migration and
movement all share common characteristics such as
volume of data, timeliness of movement and processing,
direction of flow between sources and targets (Bryan,
2002).
 DQ tools generally fall into one of three
categories: auditing, cleansing and migration. Data
auditing tools apply predefined business rules against a
source database. These tools enhance the accuracy and
correctness of the data at the source. Some of the data
cleansing tools compare the data against an independent
source e.g. US Postal Codes for verifying the data. Data
is typically moved from the source to intermediate
staging area where the data cleansing activities are
performed. Data migration is an activity where data is
extracted and transported from one source to another.
Data migration tools perform the activity of extraction,
transportation and mapping for data from one platform
to another. Poor DQ will impacts the typical enterprise
in many ways such as customer dissatisfaction,
increased cost, and lowered employee job satisfaction.
The slightest suspicion of poor DQ often hinders
managers from reaching any decision. In order to ensure
DQ assessment in DW, Hufford (1996) proposed a
model which consists of defining DQ expectations and
metrics, identifying and assessing risks, mitigating risks,
and monitoring and evaluating results on an on-going
basis.
Xu et al., (2002) agree with the notion that DQ
means accurate, timely, complete, and consistent data.
Xu et al used the terms DQ and information quality
synonymously. Data quality is influenced by a number
of factors. Several studies have shown that success of
information system and quality is dependent on TQM
and JIT (Xu et al., 2002).
Using TQM philosophy, it can described that data
quality management is a set of policies, procedures, and
follow-up actions that occur over the complete life cycle
of data starting from data generation and conversion of
data into information through the archival or discarding
of data. It consists of two major components: the data
content itself and the accompanying infrastructure
(Vikram and Sreedhar, 2006). It is important that data in
the data warehouse reflects correctly the real world, but
it is also very important that data can be easily
understood. Quality factors such as accessibility and
timeliness, believability and understandability, design
and usage flexibility play a crucial role in the success of
data warehousing.
Vikram & Sreedhar (2006) proposed a nine steps
approach to for successful deployment of a DQ program
for a DW initiative. The nine steps include identifying
data elements, defining data-quality measurement,
instituting the audit measure, defining target metrics for
each data attribute, deploying the monitoring program,
finding gaps, automating the cleansing effort,
developing procedures and establishing a continuous
monitoring program.
Nelson, Todd and Wixom (2005) had developed a
model consisting of nine fundamental determinants of
quality in an information technology context, four under
the rubric of information quality and five that describe
system quality. Their model strikes a balance between
comprehensiveness and parsimony in data warehouse
environments.
Theodoratos and Bouzeghoub (2001) had presented
a framework with a high level approach that allows
checking whether a view selection guaranteeing a data
completeness quality goal also satisfies a data currency
quality goal. So these authors have used a view to
accomplish the data quality requirements in a DW
environment.
Data warehousing depends on integrating data
quality assurance into all warehousing phases—
planning, implementation, and maintenance (Ballou &
Tayi, 1999). Practitioners in quality control
methodology recommend addressing the “root cause”
duly considering the following data quality factors:
1) Accuracy
2) Completeness
3) Timeliness
4) Integrity
5) Consistency
6) Conformity
7) Record Duplication
Based on the literature review and from our personal
experience, we propose a comprehensive version of
Data Warehouse Development Life Cycle (DWDLC)
Layers, which lists comprehensive phases and links the
DQ factors as follows.
Figure 2. Data Warehouse Development Life Cycle
(DWDLC) Layers
The major theme in each of the above presented Data
Warehouse Development Life Cycle (DWDLC) Layers
can be described as follows:
1) Planning: Apart from DQ project success, it is
evident that by defining and managing the project scope
influences the project’s overall success. Every DW
project requires a careful balance data sources,
processes, procedures, and other factors are scoped as
commensurate with the project’s size, complexity, and
importance.
2) Analysis: In this layer, one should consider analyzing
the data from various available data sources. In this
phase it is recommended to perform the data profiling of
the data, which is considered for this DW project.
3) Requirements: In this layer, DW professional will
collaborates with the business stakeholders to
understand the business problem and define and
document the required data quality factors for the DW
project.
4) Develop: In this layer, the DW professional will
develop and test the DW solution keeping in mind the
DQ factors defined in the requirement phase.
5) Implement: In this layer, the DQ solution will be
implemented after duly signed off by the quality
assurance team.
6) Measure: In this layer, a data sampling is done and a
measure to understand current process capability is
worked out on DQ factors defined in the requirements
phase. This activity will ensure to minimize the data
quality problem.
Additionally, in order to achieve the above phases, we
propose a four-prong DQ management model for
defining and ensuring data quality in a DW
environment, as described below. It is believed that
under each prong, many relevant tasks need to be
defined to achieve required DQ. Also, appropriate
metrics should be developed to measure the
effectiveness of implementation of tasks that were
defined under each prong.
Figure 3. Data Quality in Data Warehousing is a four-
pronged approach
Each of the prongs in Figure 3 defines specific
functional area. As the arrows pointed in the above
figure, quality in each of these functional areas can be
assured via on-going audits and continuous
improvement efforts. Each of the prongs is defined as
follows in a broader sense.
Basics – The very first step in any IT system is to
ensure data consistency and completeness. In this case,
one may examine individual systems and data sources to
ensure the data is complete in the sense that there is no
mission data in the fields and has the valid data values.
This can be achieved using triggers and constraints from
the database point of view. One could also check and
monitor their extract-transform-load (ETL) work flows
for monitoring time to load, stalled and suspended jobs
due to soft and hard errors, and completeness of all data
fields. Additionally, redundancies in the data can be
used as one of the metrics to minimize it.
Truth – Correctness of the data is considered as other
side of the coin that shares completeness of the data.
One of the major strengths of DW is single point of
truth. Using the data from their DW, organizations drive
day-to-day as well as long-term strategic business
activity. As a part of the quality checks and strategy, one
should consider use cases and historical sample data in
order to evaluate the correctness of the data. This should
be an on-going activity. Additionally, establishing a data
governance function that overlooks the data fields’
definitions and sample values is a key in ensuring the
data quality. The governance function would also help
in standardizing the data and the verification process
across the enterprise. Data Governance guidelines can
also be helpful in developing a process for adding or
removing new data fields or tables. More specifically,
when adding a new field, the definition of the fields
could be verified with the existing metadata to prevent
redundancy.
Coherent – Main premises of DW is merging the data
from disparate sources. In this process, it is critical to
build the coherent data using dimension keys. Ensuring
data coherency is critical for OLAP analysis as well as
building aggregates. The use cases can be critical for
verifying the data coherency. Missing values for foreign
keys and/or joining keys are a major issue falls under
this criterion, which need to be monitored.
Audit and Continuous Improvement – Plan, Do,
Check, and Act (PDCA) process can be used in this
stage. As an independent program, a frequent and
automated audit of data completeness, accuracy, and
coherency will be critical in finding the gaps. Typically,
DW would contain huge volume of data, which is very
time consuming to audit by means of manual methods.
It is strongly recommended to develop customized and
automated programs to audit and monitor as much as
possible. Raising triggers when an unusual is seen in the
data will be an important function part of the audit.
Swiftly acting on the triggers and finding the root causes
and appropriate fixes would like to continuous
improvement of DW’s ability to be a beneficial systems
for the organization.
5. Use Case
The above described Data Warehouse Development
Life Cycle (DWDLC) Layers in combination with four-
pronged Data Quality model can be used to address
most of the Data Quality issues. The following use-case
represents some of the scenarios how the above model
can be used to address data quality.
Consider this scenario: Five functional business
managers, each representing a different business
function walks into an important business strategic
planning meeting. Every one is carrying comprehensive
reports about their business functions performance.
Each manager is prepared to make some strategic
suggestions based on the reports in hand. They have all
recognized in less than an hour, their reports reflect
entirely different numbers, because the reports are not
complied from a common set of data; no one is sure
which, if any, set of numbers are accurate to consider
for the strategic planning. This has resulted in
postponing the important decision and also initiating
crucial initiatives.
The above scenario is a representation of one of the
issues faced by organizations across the globe. As it
indicates, the inconsistency of data and other quality
issues of data from which Business Intelligence (BI)
reporting are generated are some of the major concerns.
Using the proposed DWDLC model, the above scenario
can be addressed in each layer as follows:
1) Planning: As we discussed before, in this layer, we
have to scope various functions of the business unit' s.
So these five functional managers need to plan and
scope what reports they are interested to get from the
DW.
2) Analysis: In this layer, these five functional
managers should consider analyzing the data from
various available data sources. Also, their technical /
business representatives will perform the data profiling
of the data, which is considered for this DW project.
3) Requirements: In this layer, these five functional
managers will collaborates with the DW professionals to
understand the business problem and define and
document the required data quality factors for the DW
project. By doing this, the data quality issues that they
have recognized will be minimized if not completely
eliminated.
4) Develop: In this layer, the DW professionals will
develop and test the DW solution keeping in mind the
DQ factors defined by these five functional managers or
their representatives in the requirement phase.
5) Implement: In this layer, the DQ solution will be
implemented after duly signed off by the quality
assurance team.
6) Measure: In this layer, a data sampling is done and
a measure to understand current process capability is
worked out on DQ factors defined in the requirements
phase. This activity will ensure to minimize the data
quality problem.
So adhering to the above process will surely
minimize if not totally eliminate the data quality issues.
6. Conclusions
Experience suggests that one solution does not fit all;
rather the DQ assessment is an on-going effort that
requires awareness of the fundamental principles
underlying the development of subjective and objective
DQ metrics. In this article, the authors have presented
an approach that combines the subjective and objective
assessments of DQ, and demonstrated how the approach
can be defined effectively in practice.
The goal of any DW and DQ programs is to provide
decision makers with clean, consistent and relevant data.
Data Warehouses should provide a “single version of
the truth” of high quality data; this enables employees to
make informed and better decision while a low quality
data has severe effect on organization performance.
 A high quality data warehouse increases trust and
reliability of data of various applications like data
mining and its associated data-reduction techniques. In
addition, the trends as identified in the DW can be used
to ensure optimal inventory levels, high quality Website
design and to detect possible fraudulent behavior. This,
in turn, should lead to improved customer satisfaction
and an increase in market share.
With the support and commitment from the top-level
management and by employing the data quality model
and strategy proposed in this paper, the authors
confident that an effective data quality can be achieved
in a DW environment.
7. References
[11] Nord, G. D. (2005), “An Investigation of the Impact of
Organization Size on Data Quality Issues”, Journal of
Database Management, Vol. 16, No. 3, pp. 58-71.
[12] Orr, O. (1996), “Data Quality and Systems Theory”,
Communications of the ACM, Vol. 41, No. 2, pp. 66-71.
[13] Payton, F. C. & Zahay, D. (2005), “Why doesn' t
marketing use the corporate data warehouse? The role of trust
and quality in adoption of data-warehousing technology for
CRM applications”, The Journal of Business & Industrial
Marketing, Vol. 20, No. 4/5, pp. 237-244.
[14] Theodoratos, D., & Bouzeghoub, M. (2001). “Data
Currency Quality Satisfaction in the Design of a Data
Warehouse”, International Journal of Cooperative Information
Systems, 10(3), p. 299

[1] Ambler, S. W. (2006). “Data quality survey results”, [15] Vikram, R., & Sreedhar, S. (2006), “Data Quality for
www.ambysoft.com/downloads/surveys/DataQuality200609.p Enterprise Risk Management”, Business Intelligence Journal,
pt, accessed on Oct 27, 2008. Vol. 11, No. 2, pp. 18-20.

[2] Andrea, R., & Miriam, C. (2005), “Invisible data quality [16] Xu, H., Nord, J.H., Brown, N., Nord, G.D. (2002), "Data
issues in a CRM implementation”, Journal of Database quality issues in implementing an ERP", Industrial
Marketing & Customer Strategy Management, Vol. 12, No. 4, Management & Data Systems, Vol. 102, No.1, pp. 47-60.
pp. 305-314.

[3] Ballou, D., & Tayi, G. (1999, January). “Enhancing data

quality in data warehouse environments”, Communications of
the ACM, 42(1), pp. 73-78.

[4] Bielski, L. (2005), “Taking notice of data quality: as DQ

discipline goes enterprise-wide, even the "C suite" is getting
involved”, Banking Journal, Vol. 97, No. 12, pp. 41-46.

[5] Bryan, F. (2002), “Managing the quality and completeness

of customer data”, Journal of Database Management, Vol. 10,
No. 2, pp. 139–158.

[6] English, L.P. (2001), "Information quality management:

the next frontier", Annual Quality Congress Proceedings,
American Society for Quality, Milwaukee, WI, pp.529-33.

[7] Friedman, Nelson, and Radcliffe (2004), “CRM demands

data cleansing”, Gartner Research, December 2004.

[8] Guimaraes, T., Staples, D.S., & McKeen, J.D. (2007),

“Assessing the Impact from Information Systems Quality”,
Quality. Management Journal, Vol. 14, No. 1, pp. 30-44

[9] Iain, H., & Don, M. (2005), “Prioritizing and deploying

data quality improvement activity”, Journal of Database
Marketing & Customer Strategy Management, Vol. 12, No. 2,
pp. 113

[10] Nelson, R., Todd, P., & Wixom, B. (2005). “Antecedents

of Information and System Quality: An Empirical
Examination Within the Context of Data Warehousing”,
Journal of Management Information Systems, 21(4), pp. 199-
235.

View publication stats