WHITE PAPER

Why have a Data Quality Strategy?
Coursing through the electronic veins of organizations around the globe are
critical pieces of information—whether they be about customers, products,
inventories, or transactions. While the vast majority of enterprises spend
months and even years determining which computer hardware, networking,
and enterprise software solutions will help them grow their businesses,
few pay attention to the data that will support their investments in these
systems. In fact, Gartner contends, “By 2005, Fortune 1000 enterprises will
lose more money in operational ineffciency due to data quality issues than
they will spend on data warehouse and customer relationship management
(CRM) initiatives (0.9 probability).” (Gartner Inc. T. Friedman April 2004).
In its 2002 readership survey conducted by the Gantry Group LLC, DM
Review asked, “What are the three biggest challenges of implementing
a business intelligence/data warehousing (BI/DW) project within your
organization?”
Of the 688 people who responded, the number-one answer (35% of
respondents) was budget constraints. Tied with budget constraints, the
other number-one answer was data quality. In addition, an equal number
of respondents (35%) cited data quality as more important than budget
constraints.
Put simply, to realize the full benefts of their investments in enterprise
computing systems, organizations must have a detailed understanding
of the quality of their data—how to clean it, and how to keep it clean. And
those organizations that approach this issue strategically are those that will
be successful. But what goes into a data quality strategy? This paper from
Business Objects, an SAP company, explores strategy in the context of
data quality.
Data Quality Strategy:
a Step-by-Step approach
CONTENTS
1 Why Have a Data Quality Strategy?
12 Defnitions of Strategy
1 3 Building a Data Quality Strategy
24 Data Quality Goals
25 The Six Factors of Data Quality
16 Factor 1: Context
16 Factor 2: Storage
18 Factor 3: Data Flow
13 Factor 4: Workfow
15 Factor 5: Stewardship
18 Factor 6: Continuous Monitoring
24 Tying It All Together
25 Implementation and
Project Management
26 Appendix A: Data Quality
Strategy Checklist
27 About Business Objects
DefinitionS of Strategy
Many defnitions of strategy can be found in management literature. Most fall into one
of four categories centered on planning, positioning, evolution, and viewpoint. There
are even different schools of thought on how to categorize strategy; a few examples
include corporate strategies, competitive strategies, and growth strategies. Rather
than pick any one in particular, claiming it to be the right one, this paper avoids the
debate of which defnition is best, and picks the one that fts the management of
data. This is not to say other defnitions do not ft data. However, the defnition this
paper uses is, “Strategy is the implementation of a series of tactical steps.” More
specifcally, the defnition used in this paper is:
“ Strategy is a cluster of decisions centered on goals that determine
what actions to take and how to apply resources.”
Certainly a cluster of decisions—in this case concerning six specifc factors—need
to be made to effectively improve the data. Corporate goals determine how the
data is used and the level of quality needed. Actions are the processes improved
and invoked to manage the data. Resources are the people, systems, fnancing,
and data itself. We therefore apply the selected defnition in the context of data,
and arrive at the defnition of data quality strategy:
“ A cluster of decisions centered on organizational data quality goals that
determine the data processes to improve, solutions to implement, and
people to engage.”
business objects. Data Quality Strategy: A Step-by-Step Approach 2
builDing a Data Quality Strategy
This paper discusses:
• Goals that drive a data quality strategy
• Six factors that should be considered when building a strategy—context, storage,
data fow, workfow, stewardship, and continuous monitoring
• Decisions within each factor
• Actions stemming from those decisions
• Resources affected by the decisions and needed to support the actions
You will see how, when added together in different combinations, the six factors of
data quality provide the answer as to how people, process, and technology are the
integral and fundamental elements of information quality.
The paper concludes with a discussion on the transition from data quality strategy
development to implementation via data quality project management. Finally,
the appendix presents a strategy outline to help your business and IT managers
develop a data quality strategy.
business objects. Data Quality Strategy: A Step-by-Step Approach 3
Goals drive strategy. Your data quality goals must support ongoing functional
operations, data management processes, or other initiatives, such as the
implementation of a new data warehouse, CRM application, or loan processing
system. Contained within these initiatives are specifc operational goals. Examples
of operational goals include:
• Reducing the time it takes you to process quarterly customer updates
• Cleansing and combining 295 source systems into one master customer
information fle
• Complying with the U.S. Patriot Act and other governmental or regulatory
requirements to identify customers
• Determining if a vendor data fle is ft for loading into an enterprise resource
planning (ERP) system
In itself, an enterprise-level initiative is driven by strategic goals of the organization.
For example, a strategic goal to increase revenue by 5% through cross-selling and
up-selling to current customers would drive the initiative to cleanse and combine
295 source systems into one master customer information fle. The link between
the goal and the initiative is a single view of the customer versus 295 separate
views. This single view allows you to have a complete profle of the customer and
identify opportunities otherwise unseen. At frst inspection, strategic goals may
be so high-level that they seem to provide little immediate support for data quality.
Eventually, however, strategic goals are achieved by enterprise initiatives that
create demands on information in the form of data quality goals.
For example, a nonproft organization establishes the objective of supporting a
larger number of orphaned children. To do so, it needs to increase donations,
which is considered a strategic goal for the charity. The charity determines that
to increase donations it needs to identify its top donors. A look at the donor fles
causes immediate concern—there are numerous duplicates, missing frst names,
incomplete addresses, and a less-than rigorous segmentation between donor
and prospect fles, leading to overlap between the two groups. In short, the
organization cannot reliably identify its top donors. At this point, the data quality
goals become apparent: a) cleanse and standardize both donor and prospect
fles, b) fnd all duplicates in both fles and consolidate the duplicates into “best-of”
records, and c) fnd all duplicates across the donor and prospect fles, and move
prospects to the prospect fle, and donors to the donor fle.
As this example illustrates, every strategic goal of an organization is eventually
supported by data. The ability of an organization to attain its strategic goals is, in
part, determined by the level of quality of the data it collects, stores, and manages
on a daily basis.
Data Quality goalS
business objects. Data Quality Strategy: A Step-by-Step Approach 4
When creating a data quality strategy, there are six factors, or aspects, of an
organization’s operations that must be considered. The six factors are:
1. Context—the type of data being cleansed and the purposes for which it is used
2. Storage—where the data resides
3. Data fow—how the data enters and moves through the organization
4. Workfow—how work activities interact with and use the data
5. Stewardship—people responsible for managing the data
6. Continuous monitoring—processes for regularly validating the data
Figure 1 depicts the six factors centered on the goals of a data quality initiative.
Each factor requires that decisions be made, actions carried, and resources
allocated.
Figure 1: Data Quality Factors
Each data quality factor is an element of the operational data environment. It
can also be considered as a view or perspective of that environment. In this
representation (Figure 1), a factor is a collection of decisions, actions, and
resources centered on an element of the operational data environment. The arrows
extending from the core goals of the initiative depict the connection between goals
and factors, and illustrate that goals determine how each factor will be considered.
the Six factorS
of Data Quality
Decisions
Actions
Context
Continuous
Monitoring
Storage
Stewardship
Resources
Actions
Decisions
Data Flow
Work Flow
Goals
business objects. Data Quality Strategy: A Step-by-Step Approach 5
factor 1: context
Context defnes the type of data and how the data is used. Ultimately, the context
of your data determines the necessary types of cleansing algorithms and functions
needed to raise the level of quality. Examples of context and the types of data
found in each context are:
• Customer data—names, addresses, phone numbers, social security numbers,
and so on
• Financial data—dates, loan values, balances, titles, account numbers, and types
of account (revocable or joint trusts, and so on)
• Supply chain data—part numbers, descriptions, quantities, supplier codes,
and the like
• Telemetry data—for example, height, speed, direction, time, and measurement type
Context can be matched against the appropriate type of cleansing algorithms.
For example, ”title” is a subset of a customer name. In the customer name column,
embedded within the frst name or last name or by itself, are a variety of titles
—VP, President, Pres, Gnl Manager, and Shoe Shiner. It takes a specialized data-
cleansing algorithm to “know” the complete domain set of values for title, and then
be confgurable for the valid domain range that is a subset. You may need a title-
cleansing function to correct Gneral Manager to General Manager, to standardize
Pres to President, and, depending on the business rules, to either eliminate Shoe
Shiner or fag the entire record as out of domain.
factor 2: Storage
Every data quality strategy must consider where data physically resides.
Considering storage as a data quality factor ensures the physical storage medium
is included in the overall strategy. System architecture issues—such as whether
data is distributed or centralized, homogenous or heterogeneous—are important.
If the data resides in an enterprise application, the type of application (CRM,
ERP, and so on), vendor, and platform will dictate connectivity options to the data.
Connectivity options between the data and data quality function generally fall into
the following three categories:
• Data extraction
• Embedded procedures
• Integrated functionality
business objects. Data Quality Strategy: A Step-by-Step Approach 6
Data extraction
Data extraction occurs when the data is copied from the host system. It is then
cleansed, typically in a batch operation, and then reloaded back into the host.
Extraction is used for a variety of reasons, not the least of which is that native,
direct access to the host system is either impractical or impossible. For example,
an IT project manager may attempt to cleanse data in VSAM fles on an overloaded
mainframe, where the approval process to load a new application (a cleansing
application, in this case) on the mainframe takes two months, if approved at all.
Extracting the data from the VSAM fles to an intermediate location (for cleansing,
in this case) is the only viable option. Extraction is also a preferable method if
the data is being moved as part of a one-time legacy migration or a regular load
process to a data warehouse.
embedded procedures
Embedded procedures are the opposite of extractions. Here, data quality functions
are embedded, perhaps compiled, into the host system. Custom-coded, stored
procedure programming calls invoke the data quality functions, typically in a
transactional manner. Embedded procedures are used when the strategy dictates
the utmost customization, control, and tightest integration into the operational
environment. A homegrown CRM system is a likely candidate for this type of
connectivity.
integrated functionality
Integrated functionality lies between data extraction and embedded procedures.
Through the use of specialized, vendor-supplied links, data quality functions are
integrated into enterprise information systems. A link allows for a quick, standard
integration with seamless operation, and can function in either a transactional
or batch mode. Owners of CRM, ERP, or other enterprise application software
packages often choose this type of connectivity option. Links are a specifc
technology deployment option, and are discussed in additional detail below, in
the workfow factor. Deployment options are the technological solutions and
alternatives that facilitate a chosen connectivity strategy.
Data model analysis or schema design review also falls under the storage factor.
The existing data model must be assessed for its ability to support the project. Is
the model scalable and extensible? What adjustments to the model are needed?
For instance, feld overuse is one common problem encountered in a data quality
initiative that requires a model change. This can happen with personal names—for
example, where pre-names (Mr., Mrs.), titles (president, director), and certifcations
(CPA, PhD) may need to be separated from the name feld into their own felds for
better customer identifcation.
business objects. Data Quality Strategy: A Step-by-Step Approach 7
factor 3: Data floW
Each of the six strategy factors builds a different view of the operational data
environment. With context (type of data) and storage (physical location) identifed,
the next step in developing a data quality strategy is to focus on data fow—the
movement of data.
Data does not stay in one place. Even with a central data warehouse, data moves
in and out just like any other form of inventory. The migration of data can present a
moving target for a data quality strategy. Hitting that target is simplifed by mapping
the data fow. Once mapped, staging areas provide a “freeze frame” of the moving
target. A data fow will indicate where the data is manipulated, and if the usage of
the data changes context. Certainly the storage location will change, but knowing
the locations in advance makes the strategy more effective as the best location can
be chosen given the specifc goals. Work evaluating data fow will provide iterative
refnement of the results compiled in both the storage and context factors.
Data fow is important because it depicts access options to the data, and
catalogs the locations in a networked environment where the data is staged and
manipulated. Data fow answers the question: Within operational constraints, what
are the opportunities to cleanse the data? In general, such opportunities fall into
the following categories:
• Transactional updates
• Operational feeds
• Purchased data
• Legacy migration
• Regular maintenance
Figure 2 shows where these opportunities can occur in an information supply
chain. In this case, a marketing lead generation workfow is used with its
accompanying data fow. The fve cleansing opportunities are discussed in the
subsequent sections.
business objects. Data Quality Strategy: A Step-by-Step Approach 8
Figure 2: Lead Generation Workfow
transactional updates
An inherent value of the data fow factor is that it invites a proactive approach to
data cleansing. The entry points—in this case, transactions—of information into the
organization can be seen, depicting where the exposure to fawed data may occur.
When a transaction is created or captured, there is an opportunity to validate
the individual data packet before it is saved to the operational data store (ODS).
Transactional updates offer the chance to validate data as it is created or captured
in a data packet, rich with contextual information. Any defects encountered can
immediately be returned to the creator or originator for confrmation of change. This
contextual setting is lost as the data moves further in the workfow and away from
the point of entry.
The difference between a created and captured transaction is subtle, but
important. A created transaction is one where the creator (owner of the data)
directly enters the data into the electronic system as a transaction. A good
example is a new subscriber to a magazine who logs onto the magazine’s Web
site and flls out an order for a subscription. The transaction is created, validated,
and processed automatically without human intervention.
Engage
Prospects
Store
Leads
Qualify
Leads
Distribute
Leads
Trade
Show
Collect
Leads
Contact
Records
Qualified
Prospect
Records
Sales
Prospect
Lists
List of
Attendees
Raw
Leads
Purchased
Lists
Legacy
Migration
Operational
Feeds
CRM System
Obsolete,
Home-Grown
Call Center
Transactional
Updates
Operational
Feeds
Transactional
Updates
Active
Prospect
Records
Maintenance
business objects. Data Quality Strategy: A Step-by-Step Approach 9
Alternatively, a captured transaction is where the bulk of data collection takes place
offine and is later entered into the system by someone other than the owner of the
data. A good example is a new car purchase where the buyer flls out multiple paper
forms, and several downstream operators enter the information (such as registration,
insurance, loan application, and vehicle confguration data) into separate systems.
Created and captured data workfows are substantially different from each other.
The ability to correct the data with owner feedback is substantially easier and less
complex at the point of creation, than in the steps removed during capture.
operational feeds
The second opportunity to cleanse data is operational feeds. These are regular,
monthly, weekly, or nightly updates supplied from distributed sites to a central data
store. A weekly upload from a subsidiary’s CRM system to the corporation’s data
warehouse is an example. Regular operational feeds collect the data into batches
that allow implementation of scheduled batch-oriented data validation functions
in the path of the data stream. Transactional updates, instead of being cleansed
individually (which implies slower processing and wider implementation footprint),
can be batched together if immediate feedback to the transaction originator is
either not possible or necessary. Transaction-oriented cleansing in this manner
is implemented as an operational data feed. Essentially, transaction cleansing
validates data entering an ODS, such as a back-end database for a Web site,
whereas operational-feed validation cleanses data leaving an ODS, passing to
the next system—typically a data warehouse, ERP, or CRM application.
purchased Data
A third opportunity to cleanse is when the data is purchased. Purchased data is a
special situation. Many organizations erroneously consider data to be clean when
purchased. This is not necessarily the case. Data vendors suffer from the same
aging, context-mismatch, feld overuse, and other issues that all other organizations
suffer. If a purchased list is not validated upon receipt, the purchasing organization
essentially abdicates its data quality standards to those of the vendor.
business objects. Data Quality Strategy: A Step-by-Step Approach 10
Validating purchased data extends beyond verifying that each column of data is
correct. Validation must also match the purchased data against the existing data
set. The merging of two clean data sets is not the equivalent of two clean rivers
joining into one; rather, it is like pouring a gallon of red paint into blue. In the case
of a merge, 1 + 1 does not always equal 2, and may actually be 1.5, with the
remainder being lost because of duplication. To ensure continuity, the merged data
sets must be matched and consolidated as one new, entirely different set. A hidden
danger with purchased data is it enters the organization in an ad hoc event, which
implies no regular process exists to incorporate the data into the existing systems.
The lack of established cleansing and matching processes written exclusively for
the purchased data raises the possibility that cleansing will be overlooked.
legacy Migration
A fourth opportunity to cleanse data is during a legacy migration. When you export
data from an existing system to a new system, old problems from the previous
system can infect the new system unless the data is robustly checked and
validated. For example, a manufacturing company discovers during a data quality
assessment that it has three types of addresses—site location, billing address, and
corporate headquarters—but only one address record per account. To capture all
three addresses, the account staff was duplicating account records. To correct the
problem, the account record structure model of the new target system is modifed
to hold three separate addresses, before the migration occurs. Account records
that are duplicated because of different addresses can then be consolidated
during the migration operation.
A question often arises at this point: The account managers were well aware of
what they were doing, but why was the duplication of accounts not taken into
consideration during the early design of the target system? The answer lies in the
people involved in the design of the new system—what users were interviewed, and
how closely the existing workfow practices were observed. Both of these topics are
covered in the workfow and data stewardship factors discussed later in this paper.
business objects. Data Quality Strategy: A Step-by-Step Approach 11
regular Maintenance
The ffth and fnal opportunity to cleanse data is during regular maintenance.
Even if a data set is defect-free today (highly unlikely), tomorrow it will be fawed.
Data ages. For example, each year, 17% of U.S. households move, and 60% of
phone records change in some way. Moreover, every day people get married,
divorced, have children, have birthdays, get new jobs, get promoted, and change
titles. Companies start up, go bankrupt, merge, acquire, rename, and spin off. To
account for this irrevocable aging process, organizations must implement regular
data cleansing processes—be it nightly, weekly, or monthly. The longer the interval
between regular cleansing activities, the lower the overall value of the data.
Regular maintenance planning is closely tied to the sixth strategy factor—
Continuous Monitoring. Both require your organization to assess the volatility of
its data, the frequency of user access, the schedule of operations that use the
data, and the importance–and hence, the minimum required level of quality for the
data. Keeping all of this in mind, your organization can establish the periodicity
of cleansing. The storage factor will have identifed the location of the data and
preferred connectivity option.
business objects. Data Quality Strategy: A Step-by-Step Approach 12
factor 4: WorkfloW
Workfow is the sequence of physical tasks necessary to accomplish a given
operation. In an automobile factory, a workfow can be seen as a car moving along
an assembly line, each workstation responsible for a specifc set of assembly
tasks. In an IT or business environment, the workfow is no less discrete, just less
visually stimulating. When an account manager places a service call to a client,
the account manager is performing a workfow task in the same process-oriented
fashion as an engine bolted into a car.
Figure 3 shows a workfow for a lead generation function where a prospect visits
a booth at a tradeshow and supplies contact information to the booth personnel.
From there, the workfow takes over and collects, enters, qualifes, matches,
consolidates, and distributes the lead to the appropriate sales person, who then
adds new information back to the new account record.
Figure 3. Workfow Touch Points and Data Quality Deployment Options
In Figure 3 above, two different concepts are indicated. Workfow touch points,
shown in red, are the locations in the workfow where data is manipulated. You
can consider these as the locations where the workfow intersects the data fow.
Legacy
Migration
Sales contacts
lead and enters
into sales process
Sales plans
approach to lead
Sales learn
lead information
Notify sales
of lead
Qualify lead
Enter lead
(data)
Collect leads
quality center
Tradeshow
(Event)
Prospective
Customer
Lead data
Data Entry
Real-time
Point of Capture
Real-time
Information Extraction
Enterprise App Plug-in
Contract Management
Custom Application
Matching,
Consolidation,
and Data
Appending
Manual Batch
Matching,
(Leads to Territories)
Automated Batch
CRM
Lead
Information
Data Entered
Data Converted
to Information
business objects. Data Quality Strategy: A Step-by-Step Approach 13
Some of these locations, like “Point of Capture,” actually spawn a data fow.
Data quality deployment options, shown in purple, are a specifc type of software
implementation that allows connectivity or use of data quality functionality at the
point needed. In regard to workfow, data quality operations fall into the following
areas:
• Front-offce transaction—real-time cleansing
• Back-offce transaction—staged cleansing
• Back-offce batch cleansing
• Cross-offce enterprise application cleansing
• Continuous monitoring and reporting
Each area broadly encompasses work activities that are either customer-facing
or not, or both, and the type of cleansing typically needed to support them.
Specifc types of cleansing deployment options help facilitate these areas. Not to
be confused with the connectivity options discussed in the workfow factor, the
three general methods for accessing the data are connectivity options—extraction,
embedded procedures, and integrated functionality. Deployment options are forms
of cleansing technology implementations that support a particular connectivity
strategy. The deployment option list below identifes the types of options:
• Low-level application program interface (API) software libraries—high-control
custom applications
• High-level API software libraries—quick, low-control custom applications
• Web-enabled applications—real-time e-commerce operations
• Enterprise application plug-ins—ERP, CRM, and extraction, transformation, and
load (ETL) integrations
• Graphical user interface (GUI) interactive applications—data profling
• Batch applications—auto or manual start
• Web services and application service provider (ASP) connections—access to
external or outsourced functions
Each option incorporates data quality functions that measure, analyze, identify,
standardize, correct, enhance, match, and consolidate the data.
business objects. Data Quality Strategy: A Step-by-Step Approach 14
In a workfow, if a data touch point is not protected with validation functions,
defective data is captured, created, or propagated per the nature of the touch point.
An important action in the workfow factor is listing the various touch points to identify
locations where defective data can leak into your information stream. Superimposing
the list on a workfow diagram gives planners the ability to visually map cleansing
tactics, and logically cascade one data quality function to feed another.
If a “leaky” area exists in the information pipeline, the map helps to position
redundant checks around the leak to contain the contamination. When building the
list and map, concentrate on the data defned by the goals. A workfow may have
numerous data touch points, but a subset will interact with specifed data elements.
For example, a teleprospecting department needs to have all of the telephone area
codes for their contact records updated because rather than making calls, account
managers are spending an increasing amount of time researching wrong phone
numbers stemming from area code changes. The data touch points for just the
area code data are far fewer than that of an entire contact record. By focusing on
the three touch points for area codes, the project manager is able to identify two
sources of phone number data to be cleansed, and limit the project scope to just
those touch points and data sources. With the project scope narrowly defned,
operational impact and costs are reduced, and expectations of disruption are
lowered. The net result is that it is easier to obtain approval for the project.
factor 5: SteWarDShip
No strategy is complete without the evaluation of the human factor and its effect
on operations. Workfows and data fows are initiated by people. Data itself has no
value except to fulfll purposes set forth by people. The people who manage data
processes are, in the current data warehouse vernacular, called data stewards. A
plain, nonspecialized steward is defned in the dictionary as, “One who manages
another’s property, fnances, or other affairs.” Extending that defnition for our
purposes, a data steward is a person who manages information and activities that
encompass data creation, capture, maintenance, decisions, reporting, distribution,
and deletion. Therefore, a person performing any of these functions on a set of data
is a data steward.
Much can be said about each of these activities, not to mention the principles
of how to manage, provide incentives for, assign accountability, and structure
responsibilities for data stewards. A discussion on organizational structures for
data stewards could easily occupy a chapter in a book on data quality.
business objects. Data Quality Strategy: A Step-by-Step Approach 15
In the defnition of steward, there is a caption to emphasize: “One who manages
another’s property …” Many times project managers complain they can not move
their project past a certain point because the stakeholders can’t agree on who
owns the data. This is dead center a stewardship issue. No steward owns the data.
The data is owned by the organization, just as surely as the organization owns its
name, trademarks, cash, and purchased equipment. The debate on ownership is
not really about ownership, but usually centers on who has the authority to approve
a change to the data. The answer is the data stewardship team.
An action in the stewardship factor is to identify the stakeholders (stewardship
team) of the source data. Inform them of the plans, ask each one about their
specifc needs, and collect their feedback. If there are many stakeholders,
selecting a representative from each user function is highly encouraged. To do
less will surely result in one of three conditions:
• A change is made that alienates half of the users and the change is rolled back
• Half of the users are alienated and they quit using the system
• Half of the users are alienated, but are forced to use the system, and grumble and
complain at every opportunity
Most would agree that any of these three outcomes are not good for future working
relationships!
Some organizations have progressed to the point where a formal data stewardship
team is appointed. In this case, someone has already identifed the stakeholders,
and selected them as representatives on the team. This defnitely makes strategy
development a quicker process, as data stewards don’t have to be located.
business objects. Data Quality Strategy: A Step-by-Step Approach 16
When evaluating the data stewardship factor for a new project the following tasks
need to be performed:
• Answer questions, such as: Who are the stakeholders of the data? Who are the
predominant user groups, and can a representative of each be identifed? Who
is responsible for the creation, capture, maintenance, reporting, distribution, and
deletion of the data? If one of these is missed—any one of them—their actions will
fall out of sync as the project progresses, and one of those, “You never told me
you were going to do that!” moments will occur.
• Carefully organize requirements-collecting sessions with the stakeholders. Tell
these representatives any plans that can be shared, assure them that nothing
yet is fnal, and gather their input. Let these people know that they are critical
stakeholders. If strong political divisions exist between stakeholders, meet with
them separately and arbitrate the disagreements. Do not setup a situation where
feuds can erupt.
• Once a near-fnal set of requirements and a preliminary project plan are ready,
reacquaint the stakeholders with the plan. Expect changes.
• Plan to provide training and education for any new processes, data model
changes, and updated data defnitions.
• Consider the impact of new processes or changed data sets on organizational
structure. Usually a data quality project is focused on an existing system, and
current personnel reporting structures can absorb the new processes or model
changes. Occasionally, however, the existing system may need to be replaced
or migrated to a new system, and large changes in information infrastructure are
frequently accompanied by personnel shifts.
Data quality projects usually involve some changes to existing processes. The goal
of half of all data quality projects is, after all, workfow improvement. For example,
a marketing department in one organization sets a goal of reducing processing
time of new leads from two weeks to one day. The existing process consists
of manually checking each new lead for duplications against its CRM system.
The department decides to implement an automated match and consolidation
operation. The resulting workfow improvement not only saves labor time and
money, but also results in more accurate prospect data. With improvement comes
change (sometimes major, sometimes minor) in the roles and responsibilities of the
personnel involved. Know what those changes will be.
business objects. Data Quality Strategy: A Step-by-Step Approach 17
A plan to compile and advertise the benefts (return on investment) of a data
quality project deserves strategic consideration. This falls in the stewardship
factor because it is the data stewards and project managers that are tasked with
justifcation. Their managers may deliver the justifcation to senior management, but
it’s often the data stewards who are required to collect, measure, and assert the
“payoff” for the organization. Once the message is crafted, do not underestimate
the need for and value of repeatedly advertising how the improved data will
specifcally beneft the organization. Give your organization the details as a
component of an internal public or employee relations campaign. Success comes
from continually reinforcing the benefts to the organization. This builds inertia,
while hopefully managing realistic expectations. This inertia will see the project
through budget planning when the project is compared against other competing
projects.
factor 6: continuouS Monitoring
The fnal factor in a data quality strategy is continuous monitoring. Adhering
to the principals of Total Quality Management (TQM), continuous monitoring
is measuring, analyzing, and then improving a system in a continuous manner.
Continuous monitoring is crucial for the effective use of data, as data immediately
ages after capture, and future capture processes can generate errors.
Consider the volatility of data representing attributes of people. As stated earlier,
in the United States, 17% of the population moves annually, which means the
addresses of 980,000 people change each week. A supplier of phone numbers
reports that 7% of non-wireless U.S. phone numbers change each month, equating
to approximately 3.5 million phone numbers changing each week. In the United
States., 5.8 million people have a birthday each week, and an additional 77,000
are born each week. These sample statistics refect the transience of data. Each
week mergers and acquisitions change the titles, salaries, and employment status
of thousands of workers. The only way to effectively validate dynamic data for use
in daily operations is to continuously monitor and evaluate using a set of quality
measurements appropriate to the data.
business objects. Data Quality Strategy: A Step-by-Step Approach 18
A common question in this regard is, “How often should I profle my data?”
Periodicity of monitoring is determined by four considerations:
1. How often the data is used—for example, hourly, daily, weekly, or monthly.
2. The importance of the operation using the data—mission critical, life
dependent, routine operations, end of month reporting, and so on.
3. The cost of monitoring the data. After the initial expense of establishing the
monitoring system and process, the primary costs are labor and CPU cycles.
The better the monitoring technology, the lower the labor costs.
4. Operational impact of monitoring the data. There are two aspects to consider:
the impact of assessing operational (production) data during live operations,
and the impact of the process on personnel. Is the assessment process highly
manual, partially automatic, or fully automatic?
The weight of these considerations varies depending on their importance to
the operation. The greater the importance, the less meaningful the cost and
operational impact of monitoring will be. The challenge comes when an operation
is of moderate importance, and cost and operational impact are at the same
level. Fortunately, data is supported by technology. While that same technology
improves, it lowers the costs of monitoring, and lowers operational impacts.
Data stored in electronic media and even data stored in nonrelational fles can
be accessed via sophisticated data profling software. It is with this software that
fully automated and low-cost monitoring solutions can be implemented, thereby
reducing the fnal consideration of continuous monitoring to “how often” it should
be done. When purchased or built, a data profling solution could be rationalized
as “expensive,” but when the cost of the solution is amortized over the trillions of
measurements taken each year or perhaps each month, the cost per measurement
quickly nears zero. Another factor that reduces the importance of cost is the
ultimate value of continuous monitoring—fnding and preventing defects from
propagating, and therefore eliminating crisis events where the organization is
impacted from those defects.
As the previous data-churn statistics show, data cleansing cannot be a one-
time activity. If data is cleansed today, tomorrow it will have aged. A continuous
monitoring process allows an organization to measure and gauge the data
deterioration so it can tailor the periodicity of cleansing. Monitoring is also the
only way to detect spurious events such as corrupt data feeds—unexpected and
insidious in nature. A complete continuous monitoring plan should address each of
the following areas.
business objects. Data Quality Strategy: A Step-by-Step Approach 19
• identify measurements and metrics to collect. Start with project goals. The
goals determine the frst data quality strategy factor—the context. In the context
factor, it’s determined what data supports the goals. The measurements focus on
this data. Various attributes (format, range, domain, and so on) of the data elements
can be measured. The measurements can be rolled up or aggregated (each having
its own weight) into metrics that combine two or more measurements. A metric of
many measurements can be used as a single data quality score at the divisional,
business unit, or corporate level. A group of measurements and metrics can form
a data quality dashboard for a CRM system. The number of defective addresses,
invalid phone numbers, incorrectly formatted email addresses, and nonstandard
personnel titles can all be measured and rolled up into one metric that represents
quality of just the contact data. Then, if the quality score of the contact data does
not exceed a threshold defned by the organization, a decision is now possible to
postpone a planned marketing campaign until cleansing operations raise the score
above the threshold.
• identify when and where to monitor. The storage, data fow, and workfow
factors provide the information for this step. The storage factor tells what data
systems house the data that needs to be monitored. The workfow factor tells how
often the data is used in a given operation and will provide an indication as to
how often it should be monitored. The data fow factor tells how the data moves,
and how it has been manipulated just prior to the proposed point of measure. A
decision continuous monitoring will face is whether to measure the data before
or after a given operation. Is continuous monitoring testing the validity of the
operation, or testing the validity of the data to fuel the operation, or both?
One pragmatic approach is to put a monitoring process in place to evaluate a
few core tables in the data warehouse on a weekly basis. This identifes defects
inserted by processes feeding the data warehouse, and defects caused by
aging during the monitoring interval. It may not identify the source of the defects
if multiple inputs are accepted. To isolate changes from multiple events, the
monitoring operation would need to be moved further upstream or timed to occur
after each specifc update.
Organizations should be aware that although this simple approach doesn’t
optimally ft an organization’s goals, but suffces for an initial implementation.
An enhancement to the simple plan is to also monitor the data at the upstream
operational data store or staging areas. Monitoring at the ODS identifes defects
in isolation from the data warehouse, and captures them closer to the processes
that caused them. The data in the ODS is more dynamic and therefore
monitoring may need to be performed in greater frequency—for example,
nightly instead of weekly.
business objects. Data Quality Strategy: A Step-by-Step Approach 20
• implement monitoring process. This involves confguring a data profling software
solution to test specifc data elements against specifc criteria or business rules,
and save the results of the analysis to a metadata repository. Once established,
when to monitor and where to implement the process is relatively straightforward.
Most data profling packages can directly access relational data sources identifed
in the storage factor. More sophisticated solutions are available to monitor
nonrelational data sources, such as mainframe data and open systems fat fles.
Confguring the data profling software involves establishing specifc business
rules to test. For example, a part number column may have two allowed formats:
###A### and ###-###, where # is any valid numeric character, and A is any
character in the set A, B, C, and E. The user would enter the two valid formats into
the data profling software where the rules are stored in a metadata repository. The
user can then run the rules as ad hoc queries or as tasks in a regularly scheduled,
automated monitoring test set.
• run a baseline assessment. A baseline assessment is the frst set of tests
conducted to which subsequent assessments in the continuous monitoring
program will be compared. Identifying the business rules and confguring the data
profling software for the frst assessment is where the majority of work is required
in a continuous monitoring program. Building the baseline assessment serves as a
prototyping evolution for the continuous monitoring program. First iterations of tests
or recorded business rules need to be changed as they will not effectively evaluate
criteria that are meaningful to the people reviewing the reports. Other rules and the
data will change over time as more elements are added or the element attributes
evolve. The initial setup work for a baseline assessment is leveraged when the fnal
set of analysis tasks and business rules runs on a regular basis.
• post monitoring reports. A common failing of a continuous monitoring program is
poor distribution or availability of the analysis results. A key purpose of the program
is to provide both information and impetus to correct fawed data. Restricting
access to the assessment results is counterproductive. Having a data profling
solution that can post daily, weekly, or monthly reports automatically, after each run,
to a corporate Intranet is an effective communication device and productivity tool.
The reports should be carefully selected. The higher the level of manager reviewing
the reports, the more aggregated (summarized) the report data should be.

business objects. Data Quality Strategy: A Step-by-Step Approach 21
The report example below in Figure 4 offers two different measurements
superimposed on the same chart. In this case, a previous business rule for the
data stipulated there should be no NULL values. When numerous NULL values
were indeed found, another test was implemented to track how effective the
organization was at changing the NULLs to the valid values of N or P.

Figure 4: Report Example
This level of reporting is appropriate for feld-level analysts and managers who
have to cure a specifc process problem, but is too low level for a senior manager.
For a director level or higher position, a single aggregate score of all quality
measurements in a set of data is more appropriate.
• Schedule regular data steward team meetings to review monitoring trends.
Review meetings can be large or small, but they should occur regularly. Theoreti-
cally, they could occur as often as the battery of monitoring tests. If the tests are
run nightly, meeting daily as a team may be a burden. A single person could be
assigned to review the test runs, and call the team together as test results warrant.
However, a typical failing of continuous monitoring programs is follow-through.
The information gained is not acted upon. While tremendous value can be derived
from just knowing what data is defective and avoiding those defects, the greatest
value comes from fxing the defects early in the trend. This cannot be done unless
the stewardship team, either as individuals, or as a team, implements a remediation
action to both cleanse the data and cure the process that caused the defects.
140
Dev Area Visibility Codes N and P
120
C
o
u
n
t
100
80
60
40
20
0
Jan Feb Mar
2003
April May
N- QRY
P- QRY
business objects. Data Quality Strategy: A Step-by-Step Approach 22
In summary, continuous monitoring alerts managers to deterioration in data quality
early in the trend. It identifes which actions are or are not altering the data quality
conditions. It quantifes the effectiveness of data improvement actions, allowing
the actions to be tuned. Last, and most importantly, it continually reinforces the
end users’ confdence in the usability of the data.
The irony is many systems fall into disuse because of defective data, and stay
unused even after strenuous exertions by IT to cleanse and enhance the data.
The reason is perception. The system is perceived by the users, not IT, to still
be suspect. A few, well-placed and ill-timed defects can destroy overnight the
reliability of a data system. To regain the trust and confdence of users, a steady
stream of progress reports and data scores need to be published. These come
from a continuous monitoring system that shows and convinces users over time
the data is indeed improving.
business objects. Data Quality Strategy: A Step-by-Step Approach 23
In order for any strategy framework to be useful and effective, it must be scalable.
The strategy framework provided here is scalable from a simple one-feld update,
such as validating gender codes of male and female, to an enterprise-wide
initiative, where 97 ERP systems need to be cleansed and consolidated into one
system. To ensure the success of the strategy, and hence the project, each of
the six factors must be evaluated. The size (number of records/rows) and scope
(number databases, tables, and columns) determines the depth to which each
factor is evaluated.
Taken all together or in smaller groups, the six factors act as operands in data
quality strategy formulas:
• Context by itself = The type of cleansing algorithms needed
• Context + Storage + Data Flow + Workfow = The types of cleansing and
monitoring technology implementations needed
• Stewardship + Workfow = Near-term personnel impacts
• Stewardship + Workfow + Continuous Monitoring = Long-term personnel
impacts
• Data Flow + Workfow + Continuous Monitoring = Changes to processes
It is a result of using these formulas that people come to understand that
information quality truly is the integration of people, process, and technology in
the pursuit of deriving value from information assets.
tying it all together
business objects. Data Quality Strategy: A Step-by-Step Approach 24
Where the data quality strategy formulation process ends, data quality project
management takes over. In truth, much, if not all of the work resolving the six
factors, can be considered data quality project planning. Strategy formulation often
encompasses a greater scope than a single project and can support the goals of
an entire enterprise, numerous programs, and many individual projects. Sooner or
later, strategy must be implemented through a series of tactics and actions, which
fall in the realm of project management. While the purpose of this paper is not to
cover the deep subject of data quality project management, it does set the stage
for a clear transition from strategy formulation to the detailed management of the
tasks and actions that ensure its success.
Once a strategy document is created—big or small, comprehensive or narrowly
focused—it can be handed to the project manager and everything he or she needs
to know to plan the project should be in that document. This is not to say all the
work has been done. While the goals have been documented, and the data sets
established, the project manager must build the project requirements from the
goals. The project manager should adhere to the sound project management
principals and concepts that apply to any project, such as task formulation,
estimation, resource assignments, scheduling, risk analysis, mitigation, and project
monitoring against critical success factors. Few of these tactical issues are
covered in a strategy-level plan.
Another facet of a successful data quality strategy is consideration of the skills,
abilities, and culture of the organization. If the concept of data quality is new to your
organization, a simple strategy is best. Simple strategies ft pilot projects. A typical
pilot project may involve one column of data (phone numbers, for example) in one
table, impacting one or two users, and involved in one or two processes. A simple
strategy for this project, encompassing all six factors, can ft on one page of paper.
However, the more challenging the goals of a data quality strategy, the greater the
returns. An organization must accept that with greater returns come greater risks.
Data quality project risks can be mitigated by a more comprehensive strategy.
Be aware that the initial strategy is a frst iteration. Strategy plans are “living”
work products. A complex project can be subdivided into mini-projects, or pilots.
Each successful pilot builds inertia. And therein lies a strategy in itself: divide
and conquer. Successful pilots will drive future initiatives. Thus an initial strategy
planning process is part of a larger recurring cycle. True quality management is,
after all, a repeatable process.
iMpleMentation anD
proJect ManageMent
business objects. Data Quality Strategy: A Step-by-Step Approach 25
To help the practitioner employ the data quality strategy methodology, the core
practices have been extracted from the factors and listed here.
• A statement of the goals driving the project
• A list of data sets and elements that support the goal
• A list of data types and categories to be cleansed
1

• A catalog, schema, or map of where the data resides
2

• A discussion of cleansing solutions per category of data
3

• Data fow diagrams of applicable existing data fows
• Workfow diagrams of applicable existing workfows
• A plan for when and where the data is accessed for cleansing
4

• A discussion of how the data fow will change after project implementation
• A discussion of how the workfow will change after project implementation
• A list of stakeholders affected by the project
• A plan for educating stakeholders as to the benefts of the project
• A plan for training operators and users
• A list of data quality measurements and metrics to monitor
• A plan for when and where to monitor
5

• A plan for initial and then regularly scheduled cleansing
appenDix a:Data Quality
Strategy checkliSt

1
Examples of type are text, date, or time, and examples of category are street address,
part number, contact name, and so on.
2
This can include the name of the LAN, server, database, and so on.
3
This should include possible and desired deployment options for the cleansing solution.
See the section entitled Workfow for specifc deployment options.
4
This covers the when (during what steps in the data fow and workfow will the cleansing
operation be inserted) and the where (on what data systems will the cleansing operation be employed)
of the cleansing portion of the project.
5
This includes running a baseline assessment, and then selecting tests from the baseline
to run on a regular basis. Reports from the recurring monitoring will need to be posted,
and regular review of the reports scheduled for the data stewardship team.
business objects. Data Quality Strategy: A Step-by-Step Approach 26
Business Objects, an SAP company, has been a pioneer in business intelligence
(BI) since the dawn of the category. Today, as the world’s leading BI software
company, Business Objects transforms the way the world works through intelligent
information. The company helps illuminate understanding and decision-making
at more than 44,000 organizations around the globe. Through a combination of
innovative technology, global consulting and education services, and the industry’s
strongest and most diverse partner network, Business Objects enables companies
of all sizes to make transformative business decisions based on intelligent,
accurate, and timely information.
More information about Business Objects can be found at
www.businessobjects.com.
about buSineSS obJectS
business objects. Data Quality Strategy: A Step-by-Step Approach 27
© 2008 Business Objects. All rights reserved. Business Objects owns the following U.S. patents, which may cover products that are offered and licensed by Business Objects: 5,555,403; 5,857,205;
6,289,352; 6,247,008; 6,490,593; 6,578,027; 6,831,668; 6,768,986; 6,772,409; 6,882,998; 7,139,766; 7,299,419; 7,194,465; 7,222,130; 7,181,440 and 7,181,435. Business Objects and the Business
Objects logo, BusinessObjects, Business Objects Crystal Vision, Business Process On Demand, BusinessQuery, Crystal Analysis, Crystal Applications, Crystal Decisions, Crystal Enterprise, Crystal Insider,
Crystal Reports, Desktop Intelligence, Inxight, the Inxight Logo, LinguistX, Star Tree, Table Lens, ThingFinder, Timewall, Let there be light, Metify, NSite, Rapid Marts, RapidMarts, the Spectrum Design, Web
Intelligence, Workmail and Xcelsius are trademarks or registered trademarks in the United States and/or other countries of Business Objects and/or affliated companies. All other names mentioned herein
may be trademarks of their respective owners. March 2008 WP3131-A
businessobjects.com

DefinitionS of Strategy

Many definitions of strategy can be found in management literature. Most fall into one of four categories centered on planning, positioning, evolution, and viewpoint. There are even different schools of thought on how to categorize strategy; a few examples include corporate strategies, competitive strategies, and growth strategies. Rather than pick any one in particular, claiming it to be the right one, this paper avoids the debate of which definition is best, and picks the one that fits the management of data. This is not to say other definitions do not fit data. However, the definition this paper uses is, “Strategy is the implementation of a series of tactical steps.” More specifically, the definition used in this paper is: “Strategy is a cluster of decisions centered on goals that determine what actions to take and how to apply resources.” Certainly a cluster of decisions—in this case concerning six specific factors—need to be made to effectively improve the data. Corporate goals determine how the data is used and the level of quality needed. Actions are the processes improved and invoked to manage the data. Resources are the people, systems, financing, and data itself. We therefore apply the selected definition in the context of data, and arrive at the definition of data quality strategy: “A cluster of decisions centered on organizational data quality goals that determine the data processes to improve, solutions to implement, and people to engage.”

business objects. Data Quality Strategy: A Step-by-Step Approach 

builDing a Data Quality Strategy This paper discusses: • Goals that drive a data quality strategy • Six factors that should be considered when building a strategy—context. storage. Finally. and technology are the integral and fundamental elements of information quality. business objects. and continuous monitoring • Decisions within each factor • Actions stemming from those decisions • Resources affected by the decisions and needed to support the actions You will see how. stewardship. The paper concludes with a discussion on the transition from data quality strategy development to implementation via data quality project management. the appendix presents a strategy outline to help your business and IT managers develop a data quality strategy. the six factors of data quality provide the answer as to how people. process. data flow. workflow. Data Quality Strategy: A Step-by-Step Approach  . when added together in different combinations.

S.Data Quality goalS Goals drive strategy. a nonprofit organization establishes the objective of supporting a larger number of orphaned children. and manages on a daily basis. In short. The charity determines that to increase donations it needs to identify its top donors. incomplete addresses. b) find all duplicates in both files and consolidate the duplicates into “best-of” records. To do so. Contained within these initiatives are specific operational goals. and move prospects to the prospect file. an enterprise-level initiative is driven by strategic goals of the organization. A look at the donor files causes immediate concern—there are numerous duplicates. however. the organization cannot reliably identify its top donors. such as the implementation of a new data warehouse. For example. Patriot Act and other governmental or regulatory requirements to identify customers • Determining if a vendor data file is fit for loading into an enterprise resource planning (ERP) system In itself. Eventually. At this point. This single view allows you to have a complete profile of the customer and identify opportunities otherwise unseen. data management processes. leading to overlap between the two groups. The ability of an organization to attain its strategic goals is. As this example illustrates. it needs to increase donations. Examples of operational goals include: • Reducing the time it takes you to process quarterly customer updates • Cleansing and combining 295 source systems into one master customer information file • Complying with the U. and a less-than rigorous segmentation between donor and prospect files. business objects. determined by the level of quality of the data it collects. Data Quality Strategy: A Step-by-Step Approach  . missing first names. the data quality goals become apparent: a) cleanse and standardize both donor and prospect files. and donors to the donor file. stores. every strategic goal of an organization is eventually supported by data. At first inspection. For example. in part. CRM application. strategic goals may be so high-level that they seem to provide little immediate support for data quality. or other initiatives. or loan processing system. strategic goals are achieved by enterprise initiatives that create demands on information in the form of data quality goals. and c) find all duplicates across the donor and prospect files. which is considered a strategic goal for the charity. The link between the goal and the initiative is a single view of the customer versus 295 separate views. Your data quality goals must support ongoing functional operations. a strategic goal to increase revenue by 5% through cross-selling and up-selling to current customers would drive the initiative to cleanse and combine 295 source systems into one master customer information file.

Data Quality Strategy: A Step-by-Step Approach  . Context Continuous Monitoring Storage Goals Decisions Actions Stewardship Data Flow Decisions Actions Work Flow Figure 1: Data Quality Factors Resources Each data quality factor is an element of the operational data environment. Each factor requires that decisions be made. there are six factors. and illustrate that goals determine how each factor will be considered. Data flow—how the data enters and moves through the organization 4. In this representation (Figure 1). It can also be considered as a view or perspective of that environment. Storage—where the data resides 3. The arrows extending from the core goals of the initiative depict the connection between goals and factors.the Six factorS of Data Quality When creating a data quality strategy. actions carried. a factor is a collection of decisions. Workflow—how work activities interact with and use the data 5. and resources centered on an element of the operational data environment. actions. The six factors are: 1. Stewardship—people responsible for managing the data 6. of an organization’s operations that must be considered. Continuous monitoring—processes for regularly validating the data Figure 1 depicts the six factors centered on the goals of a data quality initiative. and resources allocated. Context—the type of data being cleansed and the purposes for which it is used 2. business objects. or aspects.

You may need a titlecleansing function to correct Gneral Manager to General Manager. loan values. System architecture issues—such as whether data is distributed or centralized. and platform will dictate connectivity options to the data. and Shoe Shiner. and so on). In the customer name column. supplier codes. phone numbers. descriptions. Examples of context and the types of data found in each context are: • Customer data—names. ”title” is a subset of a customer name. height. Ultimately. depending on the business rules. time. account numbers. vendor. the type of application (CRM. quantities. and so on) • Supply chain data—part numbers. If the data resides in an enterprise application. the context of your data determines the necessary types of cleansing algorithms and functions needed to raise the level of quality. For example. social security numbers. It takes a specialized datacleansing algorithm to “know” the complete domain set of values for title. and. titles.factor 1: context Context defines the type of data and how the data is used. Gnl Manager. embedded within the first name or last name or by itself. Pres. Considering storage as a data quality factor ensures the physical storage medium is included in the overall strategy. direction. President. Connectivity options between the data and data quality function generally fall into the following three categories: • Data extraction • Embedded procedures • Integrated functionality business objects. to standardize Pres to President. and measurement type Context can be matched against the appropriate type of cleansing algorithms. factor 2: Storage Every data quality strategy must consider where data physically resides. and so on • Financial data—dates. balances. speed. addresses. Data Quality Strategy: A Step-by-Step Approach  . ERP. are a variety of titles —VP. to either eliminate Shoe Shiner or flag the entire record as out of domain. and the like • Telemetry data—for example. homogenous or heterogeneous—are important. and then be configurable for the valid domain range that is a subset. and types of account (revocable or joint trusts.

). Is the model scalable and extensible? What adjustments to the model are needed? For instance. where the approval process to load a new application (a cleansing application. data quality functions are embedded. perhaps compiled. Mrs. and can function in either a transactional or batch mode. and tightest integration into the operational environment. Here. or other enterprise application software packages often choose this type of connectivity option. integrated functionality Integrated functionality lies between data extraction and embedded procedures. Owners of CRM. field overuse is one common problem encountered in a data quality initiative that requires a model change. Links are a specific technology deployment option. For example. This can happen with personal names—for example. business objects. It is then cleansed. vendor-supplied links. Custom-coded. and then reloaded back into the host. Extraction is also a preferable method if the data is being moved as part of a one-time legacy migration or a regular load process to a data warehouse.. and certifications (CPA. A link allows for a quick. Data model analysis or schema design review also falls under the storage factor. into the host system. Extraction is used for a variety of reasons. standard integration with seamless operation. in this case) on the mainframe takes two months. Deployment options are the technological solutions and alternatives that facilitate a chosen connectivity strategy. Through the use of specialized. A homegrown CRM system is a likely candidate for this type of connectivity. The existing data model must be assessed for its ability to support the project. in the workflow factor. in this case) is the only viable option. ERP. typically in a batch operation. PhD) may need to be separated from the name field into their own fields for better customer identification.Data extraction Data extraction occurs when the data is copied from the host system. direct access to the host system is either impractical or impossible. embedded procedures Embedded procedures are the opposite of extractions. Data Quality Strategy: A Step-by-Step Approach  . data quality functions are integrated into enterprise information systems. Embedded procedures are used when the strategy dictates the utmost customization. where pre-names (Mr. director). and are discussed in additional detail below. stored procedure programming calls invoke the data quality functions. an IT project manager may attempt to cleanse data in VSAM files on an overloaded mainframe. Extracting the data from the VSAM files to an intermediate location (for cleansing. if approved at all. titles (president. typically in a transactional manner. control. not the least of which is that native.

Certainly the storage location will change. the next step in developing a data quality strategy is to focus on data flow—the movement of data. The migration of data can present a moving target for a data quality strategy. such opportunities fall into the following categories: • Transactional updates • Operational feeds • Purchased data • Legacy migration • Regular maintenance Figure 2 shows where these opportunities can occur in an information supply chain.factor 3: Data floW Each of the six strategy factors builds a different view of the operational data environment. Data flow is important because it depicts access options to the data. but knowing the locations in advance makes the strategy more effective as the best location can be chosen given the specific goals. Hitting that target is simplified by mapping the data flow. data moves in and out just like any other form of inventory. staging areas provide a “freeze frame” of the moving target. Work evaluating data flow will provide iterative refinement of the results compiled in both the storage and context factors. and catalogs the locations in a networked environment where the data is staged and manipulated. business objects. Data flow answers the question: Within operational constraints. With context (type of data) and storage (physical location) identified. Once mapped. Even with a central data warehouse. a marketing lead generation workflow is used with its accompanying data flow. Data Quality Strategy: A Step-by-Step Approach  . what are the opportunities to cleanse the data? In general. In this case. and if the usage of the data changes context. The five cleansing opportunities are discussed in the subsequent sections. A data flow will indicate where the data is manipulated. Data does not stay in one place.

When a transaction is created or captured. but important. A created transaction is one where the creator (owner of the data) directly enters the data into the electronic system as a transaction. This contextual setting is lost as the data moves further in the workflow and away from the point of entry. Any defects encountered can immediately be returned to the creator or originator for confirmation of change. and processed automatically without human intervention. validated. The entry points—in this case. there is an opportunity to validate the individual data packet before it is saved to the operational data store (ODS).Maintenance Obsolete. business objects. The transaction is created. rich with contextual information. depicting where the exposure to flawed data may occur. Data Quality Strategy: A Step-by-Step Approach  . transactions—of information into the organization can be seen. The difference between a created and captured transaction is subtle. Transactional updates offer the chance to validate data as it is created or captured in a data packet. A good example is a new subscriber to a magazine who logs onto the magazine’s Web site and fills out an order for a subscription. Home-Grown Call Center Legacy Migration CRM System Purchased Lists Operational Feeds Transactional Updates Operational Feeds Transactional Updates List of Attendees Raw Leads Contact Records Qualified Prospect Records Sales Prospect Lists Active Prospect Records Trade Show Collect Leads Store Leads Qualify Leads Distribute Leads Engage Prospects Figure 2: Lead Generation Workflow transactional updates An inherent value of the data flow factor is that it invites a proactive approach to data cleansing.

This is not necessarily the case. Essentially. or CRM application. such as a back-end database for a Web site. Created and captured data workflows are substantially different from each other. can be batched together if immediate feedback to the transaction originator is either not possible or necessary. transaction cleansing validates data entering an ODS. than in the steps removed during capture. ERP. monthly. business objects. A weekly upload from a subsidiary’s CRM system to the corporation’s data warehouse is an example. Regular operational feeds collect the data into batches that allow implementation of scheduled batch-oriented data validation functions in the path of the data stream. and other issues that all other organizations suffer. Transaction-oriented cleansing in this manner is implemented as an operational data feed. purchased Data A third opportunity to cleanse is when the data is purchased. and several downstream operators enter the information (such as registration. The ability to correct the data with owner feedback is substantially easier and less complex at the point of creation. or nightly updates supplied from distributed sites to a central data store. insurance. and vehicle configuration data) into separate systems. loan application. Transactional updates. These are regular. instead of being cleansed individually (which implies slower processing and wider implementation footprint).Alternatively. the purchasing organization essentially abdicates its data quality standards to those of the vendor. operational feeds The second opportunity to cleanse data is operational feeds. weekly. context-mismatch. whereas operational-feed validation cleanses data leaving an ODS. field overuse. If a purchased list is not validated upon receipt. a captured transaction is where the bulk of data collection takes place offline and is later entered into the system by someone other than the owner of the data. passing to the next system—typically a data warehouse. Data Quality Strategy: A Step-by-Step Approach 10 . Many organizations erroneously consider data to be clean when purchased. Data vendors suffer from the same aging. Purchased data is a special situation. A good example is a new car purchase where the buyer fills out multiple paper forms.

1 + 1 does not always equal 2. it is like pouring a gallon of red paint into blue. rather. but why was the duplication of accounts not taken into consideration during the early design of the target system? The answer lies in the people involved in the design of the new system—what users were interviewed. old problems from the previous system can infect the new system unless the data is robustly checked and validated. To ensure continuity. and how closely the existing workflow practices were observed. The lack of established cleansing and matching processes written exclusively for the purchased data raises the possibility that cleansing will be overlooked. and may actually be 1. the merged data sets must be matched and consolidated as one new. with the remainder being lost because of duplication. which implies no regular process exists to incorporate the data into the existing systems.Validating purchased data extends beyond verifying that each column of data is correct. entirely different set. legacy Migration A fourth opportunity to cleanse data is during a legacy migration.5. For example. When you export data from an existing system to a new system. Both of these topics are covered in the workflow and data stewardship factors discussed later in this paper. The merging of two clean data sets is not the equivalent of two clean rivers joining into one. business objects. the account staff was duplicating account records. To correct the problem. before the migration occurs. billing address. In the case of a merge. a manufacturing company discovers during a data quality assessment that it has three types of addresses—site location. the account record structure model of the new target system is modified to hold three separate addresses. Data Quality Strategy: A Step-by-Step Approach 11 . Validation must also match the purchased data against the existing data set. A question often arises at this point: The account managers were well aware of what they were doing. A hidden danger with purchased data is it enters the organization in an ad hoc event. and corporate headquarters—but only one address record per account. Account records that are duplicated because of different addresses can then be consolidated during the migration operation. To capture all three addresses.

The storage factor will have identified the location of the data and preferred connectivity option. tomorrow it will be flawed. weekly. To account for this irrevocable aging process. have birthdays. and spin off. For example. households move. your organization can establish the periodicity of cleansing. get new jobs.S. the schedule of operations that use the data. Regular maintenance planning is closely tied to the sixth strategy factor— Continuous Monitoring. have children. organizations must implement regular data cleansing processes—be it nightly. each year. get promoted. Both require your organization to assess the volatility of its data. rename. merge.regular Maintenance The fifth and final opportunity to cleanse data is during regular maintenance. Keeping all of this in mind. business objects. or monthly. the frequency of user access. divorced. go bankrupt. the minimum required level of quality for the data. Data Quality Strategy: A Step-by-Step Approach 1 . every day people get married. and change titles. 17% of U. and the importance–and hence. Data ages. acquire. Even if a data set is defect-free today (highly unlikely). Moreover. and 60% of phone records change in some way. The longer the interval between regular cleansing activities. the lower the overall value of the data. Companies start up.

Figure 3 shows a workflow for a lead generation function where a prospect visits a booth at a tradeshow and supplies contact information to the booth personnel. who then adds new information back to the new account record. Matching. just less visually stimulating. the workflow takes over and collects. You can consider these as the locations where the workflow intersects the data flow. In an IT or business environment. the workflow is no less discrete.factor 4: WorkfloW Workflow is the sequence of physical tasks necessary to accomplish a given operation. a workflow can be seen as a car moving along an assembly line. business objects. qualifies. Workflow Touch Points and Data Quality Deployment Options In Figure 3 above. When an account manager places a service call to a client. In an automobile factory. consolidates. each workstation responsible for a specific set of assembly tasks. (Leads to Territories) Automated Batch Data Entry Real-time Tradeshow (Event) Collect leads quality center Enter lead (data) Data Entered Qualify lead Data Converted to Information Notify sales of lead Prospective Customer Lead data Legacy Migration Point of Capture Real-time CRM Lead Information Sales learn lead information Information Extraction Enterprise App Plug-in Contract Management Custom Application Sales plans approach to lead Sales contacts lead and enters into sales process Figure 3. the account manager is performing a workflow task in the same process-oriented fashion as an engine bolted into a car. Workflow touch points. From there. Consolidation. Data Quality Strategy: A Step-by-Step Approach 1 . two different concepts are indicated. are the locations in the workflow where data is manipulated. and Data Appending Manual Batch Matching. shown in red. matches. and distributes the lead to the appropriate sales person. enters.

correct. In regard to workflow. or both. Specific types of cleansing deployment options help facilitate these areas. analyze. shown in purple. embedded procedures.Some of these locations. enhance. and consolidate the data. transformation. and extraction.” actually spawn a data flow. Data quality deployment options. and integrated functionality. identify. Data Quality Strategy: A Step-by-Step Approach 1 . data quality operations fall into the following areas: • Front-office transaction—real-time cleansing • Back-office transaction—staged cleansing • Back-office batch cleansing • Cross-office enterprise application cleansing • Continuous monitoring and reporting Each area broadly encompasses work activities that are either customer-facing or not. and load (ETL) integrations • Graphical user interface (GUI) interactive applications—data profiling • Batch applications—auto or manual start • Web services and application service provider (ASP) connections—access to external or outsourced functions Each option incorporates data quality functions that measure. low-control custom applications • Web-enabled applications—real-time e-commerce operations • Enterprise application plug-ins—ERP. are a specific type of software implementation that allows connectivity or use of data quality functionality at the point needed. Not to be confused with the connectivity options discussed in the workflow factor. match. the three general methods for accessing the data are connectivity options—extraction. Deployment options are forms of cleansing technology implementations that support a particular connectivity strategy. like “Point of Capture. business objects. CRM. The deployment option list below identifies the types of options: • Low-level application program interface (API) software libraries—high-control custom applications • High-level API software libraries—quick. standardize. and the type of cleansing typically needed to support them.

a person performing any of these functions on a set of data is a data steward. If a “leaky” area exists in the information pipeline. The people who manage data processes are. called data stewards. maintenance. a data steward is a person who manages information and activities that encompass data creation. The net result is that it is easier to obtain approval for the project. and logically cascade one data quality function to feed another. Workflows and data flows are initiated by people. Data itself has no value except to fulfill purposes set forth by people. Therefore. distribution. With the project scope narrowly defined. or propagated per the nature of the touch point. By focusing on the three touch points for area codes. finances. but a subset will interact with specified data elements. the map helps to position redundant checks around the leak to contain the contamination. the project manager is able to identify two sources of phone number data to be cleansed. factor 5: SteWarDShip No strategy is complete without the evaluation of the human factor and its effect on operations. and limit the project scope to just those touch points and data sources. and deletion. operational impact and costs are reduced. A workflow may have numerous data touch points. capture.In a workflow. Data Quality Strategy: A Step-by-Step Approach 1 . A discussion on organizational structures for data stewards could easily occupy a chapter in a book on data quality. and structure responsibilities for data stewards. created. Superimposing the list on a workflow diagram gives planners the ability to visually map cleansing tactics. concentrate on the data defined by the goals. or other affairs. defective data is captured. reporting. The data touch points for just the area code data are far fewer than that of an entire contact record. “One who manages another’s property. A plain. if a data touch point is not protected with validation functions. provide incentives for. nonspecialized steward is defined in the dictionary as. When building the list and map. a teleprospecting department needs to have all of the telephone area codes for their contact records updated because rather than making calls.” Extending that definition for our purposes. in the current data warehouse vernacular. business objects. assign accountability. Much can be said about each of these activities. For example. An important action in the workflow factor is listing the various touch points to identify locations where defective data can leak into your information stream. not to mention the principles of how to manage. and expectations of disruption are lowered. decisions. account managers are spending an increasing amount of time researching wrong phone numbers stemming from area code changes.

The data is owned by the organization. No steward owns the data. If there are many stakeholders. The debate on ownership is not really about ownership. This definitely makes strategy development a quicker process. To do less will surely result in one of three conditions: • A change is made that alienates half of the users and the change is rolled back • Half of the users are alienated and they quit using the system • Half of the users are alienated. just as surely as the organization owns its name. Data Quality Strategy: A Step-by-Step Approach 1 . An action in the stewardship factor is to identify the stakeholders (stewardship team) of the source data. there is a caption to emphasize: “One who manages another’s property …” Many times project managers complain they can not move their project past a certain point because the stakeholders can’t agree on who owns the data. cash. but usually centers on who has the authority to approve a change to the data. In this case. someone has already identified the stakeholders. selecting a representative from each user function is highly encouraged. trademarks. and collect their feedback. The answer is the data stewardship team. and purchased equipment. as data stewards don’t have to be located. but are forced to use the system. business objects. ask each one about their specific needs.In the definition of steward. and selected them as representatives on the team. This is dead center a stewardship issue. Inform them of the plans. and grumble and complain at every opportunity Most would agree that any of these three outcomes are not good for future working relationships! Some organizations have progressed to the point where a formal data stewardship team is appointed.

Know what those changes will be. and gather their input. • Once a near-final set of requirements and a preliminary project plan are ready. distribution. a marketing department in one organization sets a goal of reducing processing time of new leads from two weeks to one day. If strong political divisions exist between stakeholders. after all. Let these people know that they are critical stakeholders. and can a representative of each be identified? Who is responsible for the creation. business objects. Data quality projects usually involve some changes to existing processes.When evaluating the data stewardship factor for a new project the following tasks need to be performed: • Answer questions. Occasionally. the existing system may need to be replaced or migrated to a new system. Do not setup a situation where feuds can erupt. meet with them separately and arbitrate the disagreements. Data Quality Strategy: A Step-by-Step Approach 1 . assure them that nothing yet is final. and large changes in information infrastructure are frequently accompanied by personnel shifts. • Carefully organize requirements-collecting sessions with the stakeholders. such as: Who are the stakeholders of the data? Who are the predominant user groups. The department decides to implement an automated match and consolidation operation. data model changes. “You never told me you were going to do that!” moments will occur. The resulting workflow improvement not only saves labor time and money. For example. and one of those. capture. Tell these representatives any plans that can be shared. With improvement comes change (sometimes major. • Consider the impact of new processes or changed data sets on organizational structure. Expect changes. workflow improvement. reacquaint the stakeholders with the plan. The existing process consists of manually checking each new lead for duplications against its CRM system. maintenance. however. and deletion of the data? If one of these is missed—any one of them—their actions will fall out of sync as the project progresses. Usually a data quality project is focused on an existing system. reporting. and current personnel reporting structures can absorb the new processes or model changes. and updated data definitions. The goal of half of all data quality projects is. but also results in more accurate prospect data. • Plan to provide training and education for any new processes. sometimes minor) in the roles and responsibilities of the personnel involved.

and employment status of thousands of workers. do not underestimate the need for and value of repeatedly advertising how the improved data will specifically benefit the organization. factor 6: continuouS Monitoring The final factor in a data quality strategy is continuous monitoring. As stated earlier. This inertia will see the project through budget planning when the project is compared against other competing projects. salaries. In the United States. phone numbers change each month. The only way to effectively validate dynamic data for use in daily operations is to continuously monitor and evaluate using a set of quality measurements appropriate to the data. These sample statistics reflect the transience of data. 5. 17% of the population moves annually. Continuous monitoring is crucial for the effective use of data. Give your organization the details as a component of an internal public or employee relations campaign. measure. in the United States.S. Adhering to the principals of Total Quality Management (TQM). Once the message is crafted. and an additional 77. but it’s often the data stewards who are required to collect. Each week mergers and acquisitions change the titles. business objects. as data immediately ages after capture. Their managers may deliver the justification to senior management. Success comes from continually reinforcing the benefits to the organization. equating to approximately 3. Consider the volatility of data representing attributes of people. and assert the “payoff” for the organization. while hopefully managing realistic expectations.5 million phone numbers changing each week.8 million people have a birthday each week. analyzing. and then improving a system in a continuous manner. This builds inertia. which means the addresses of 980. Data Quality Strategy: A Step-by-Step Approach 1 . and future capture processes can generate errors. This falls in the stewardship factor because it is the data stewards and project managers that are tasked with justification.000 people change each week.A plan to compile and advertise the benefits (return on investment) of a data quality project deserves strategic consideration. continuous monitoring is measuring.000 are born each week. A supplier of phone numbers reports that 7% of non-wireless U..

Data Quality Strategy: A Step-by-Step Approach 1 . and the impact of the process on personnel. A complete continuous monitoring plan should address each of the following areas. life dependent. A continuous monitoring process allows an organization to measure and gauge the data deterioration so it can tailor the periodicity of cleansing. routine operations. As the previous data-churn statistics show. When purchased or built. The importance of the operation using the data—mission critical. It is with this software that fully automated and low-cost monitoring solutions can be implemented. 3. tomorrow it will have aged. 2. data is supported by technology. and cost and operational impact are at the same level. the lower the labor costs. There are two aspects to consider: the impact of assessing operational (production) data during live operations. thereby reducing the final consideration of continuous monitoring to “how often” it should be done. or monthly. Operational impact of monitoring the data. The cost of monitoring the data. After the initial expense of establishing the monitoring system and process. daily. a data profiling solution could be rationalized as “expensive. “How often should I profile my data?” Periodicity of monitoring is determined by four considerations: 1. How often the data is used—for example. partially automatic.A common question in this regard is. weekly. Is the assessment process highly manual. hourly. While that same technology improves. the less meaningful the cost and operational impact of monitoring will be. Monitoring is also the only way to detect spurious events such as corrupt data feeds—unexpected and insidious in nature. The better the monitoring technology. The greater the importance. The challenge comes when an operation is of moderate importance. or fully automatic? The weight of these considerations varies depending on their importance to the operation. business objects. Fortunately. the cost per measurement quickly nears zero. If data is cleansed today. and lowers operational impacts. Data stored in electronic media and even data stored in nonrelational files can be accessed via sophisticated data profiling software. 4. data cleansing cannot be a onetime activity.” but when the cost of the solution is amortized over the trillions of measurements taken each year or perhaps each month. Another factor that reduces the importance of cost is the ultimate value of continuous monitoring—finding and preventing defects from propagating. and so on. end of month reporting. the primary costs are labor and CPU cycles. it lowers the costs of monitoring. and therefore eliminating crisis events where the organization is impacted from those defects.

and captures them closer to the processes that caused them. nightly instead of weekly. The storage factor tells what data systems house the data that needs to be monitored. and how it has been manipulated just prior to the proposed point of measure. business objects. or corporate level. data flow. if the quality score of the contact data does not exceed a threshold defined by the organization. The number of defective addresses. it’s determined what data supports the goals. It may not identify the source of the defects if multiple inputs are accepted. and defects caused by aging during the monitoring interval. a decision is now possible to postpone a planned marketing campaign until cleansing operations raise the score above the threshold. and workflow factors provide the information for this step. The workflow factor tells how often the data is used in a given operation and will provide an indication as to how often it should be monitored. This identifies defects inserted by processes feeding the data warehouse. The storage. and so on) of the data elements can be measured. Organizations should be aware that although this simple approach doesn’t optimally fit an organization’s goals. incorrectly formatted email addresses. Data Quality Strategy: A Step-by-Step Approach 0 . Is continuous monitoring testing the validity of the operation. or both? One pragmatic approach is to put a monitoring process in place to evaluate a few core tables in the data warehouse on a weekly basis. Various attributes (format. An enhancement to the simple plan is to also monitor the data at the upstream operational data store or staging areas. A decision continuous monitoring will face is whether to measure the data before or after a given operation. but suffices for an initial implementation. invalid phone numbers. the monitoring operation would need to be moved further upstream or timed to occur after each specific update. Start with project goals. In the context factor. or testing the validity of the data to fuel the operation. • identify when and where to monitor. domain. A metric of many measurements can be used as a single data quality score at the divisional. The data in the ODS is more dynamic and therefore monitoring may need to be performed in greater frequency—for example. range. The measurements focus on this data. Then. Monitoring at the ODS identifies defects in isolation from the data warehouse. business unit.• identify measurements and metrics to collect. The data flow factor tells how the data moves. A group of measurements and metrics can form a data quality dashboard for a CRM system. The measurements can be rolled up or aggregated (each having its own weight) into metrics that combine two or more measurements. To isolate changes from multiple events. and nonstandard personnel titles can all be measured and rolled up into one metric that represents quality of just the contact data. The goals determine the first data quality strategy factor—the context.

• post monitoring reports. a part number column may have two allowed formats: ###A### and ###-###. First iterations of tests or recorded business rules need to be changed as they will not effectively evaluate criteria that are meaningful to the people reviewing the reports. weekly. the more aggregated (summarized) the report data should be. Identifying the business rules and configuring the data profiling software for the first assessment is where the majority of work is required in a continuous monitoring program. when to monitor and where to implement the process is relatively straightforward. This involves configuring a data profiling software solution to test specific data elements against specific criteria or business rules. Building the baseline assessment serves as a prototyping evolution for the continuous monitoring program. and save the results of the analysis to a metadata repository. Restricting access to the assessment results is counterproductive. after each run. A baseline assessment is the first set of tests conducted to which subsequent assessments in the continuous monitoring program will be compared. The user would enter the two valid formats into the data profiling software where the rules are stored in a metadata repository. business objects. The user can then run the rules as ad hoc queries or as tasks in a regularly scheduled. Having a data profiling solution that can post daily. C. B. and E. More sophisticated solutions are available to monitor nonrelational data sources. A key purpose of the program is to provide both information and impetus to correct flawed data. The higher the level of manager reviewing the reports. to a corporate Intranet is an effective communication device and productivity tool. • run a baseline assessment. where # is any valid numeric character. automated monitoring test set. Most data profiling packages can directly access relational data sources identified in the storage factor. such as mainframe data and open systems flat files. The reports should be carefully selected. For example. Other rules and the data will change over time as more elements are added or the element attributes evolve. A common failing of a continuous monitoring program is poor distribution or availability of the analysis results. or monthly reports automatically.• implement monitoring process. Data Quality Strategy: A Step-by-Step Approach 1 . Configuring the data profiling software involves establishing specific business rules to test. and A is any character in the set A. The initial setup work for a baseline assessment is leveraged when the final set of analysis tasks and business rules runs on a regular basis. Once established.

If the tests are run nightly. a typical failing of continuous monitoring programs is follow-through.QRY Figure 4: Report Example This level of reporting is appropriate for field-level analysts and managers who have to cure a specific process problem. a previous business rule for the data stipulated there should be no NULL values. In this case. For a director level or higher position. business objects. but they should occur regularly. or as a team. When numerous NULL values were indeed found. Theoretically. While tremendous value can be derived from just knowing what data is defective and avoiding those defects. The information gained is not acted upon.QRY P. a single aggregate score of all quality measurements in a set of data is more appropriate. • Schedule regular data steward team meetings to review monitoring trends. Data Quality Strategy: A Step-by-Step Approach  . implements a remediation action to both cleanse the data and cure the process that caused the defects. and call the team together as test results warrant. However. Review meetings can be large or small. This cannot be done unless the stewardship team. meeting daily as a team may be a burden. they could occur as often as the battery of monitoring tests. another test was implemented to track how effective the organization was at changing the NULLs to the valid values of N or P.The report example below in Figure 4 offers two different measurements superimposed on the same chart. but is too low level for a senior manager. A single person could be assigned to review the test runs. the greatest value comes from fixing the defects early in the trend. either as individuals. Dev Area Visibility Codes N and P 140 120 100 Count 80 60 40 20 0 Jan Feb Mar 2003 April May N.

These come from a continuous monitoring system that shows and convinces users over time the data is indeed improving. The irony is many systems fall into disuse because of defective data. not IT. Last. a steady stream of progress reports and data scores need to be published. and most importantly. business objects. to still be suspect. A few. continuous monitoring alerts managers to deterioration in data quality early in the trend. To regain the trust and confidence of users. It quantifies the effectiveness of data improvement actions.In summary. It identifies which actions are or are not altering the data quality conditions. The system is perceived by the users. allowing the actions to be tuned. Data Quality Strategy: A Step-by-Step Approach  . well-placed and ill-timed defects can destroy overnight the reliability of a data system. The reason is perception. it continually reinforces the end users’ confidence in the usability of the data. and stay unused even after strenuous exertions by IT to cleanse and enhance the data.

such as validating gender codes of male and female. it must be scalable. Taken all together or in smaller groups. and columns) determines the depth to which each factor is evaluated. and hence the project.tying it all together In order for any strategy framework to be useful and effective. tables. To ensure the success of the strategy. process. where 97 ERP systems need to be cleansed and consolidated into one system. each of the six factors must be evaluated. The strategy framework provided here is scalable from a simple one-field update. to an enterprise-wide initiative. the six factors act as operands in data quality strategy formulas: • Context by itself = The type of cleansing algorithms needed • Context + Storage + Data Flow + Workflow = The types of cleansing and monitoring technology implementations needed • Stewardship + Workflow = Near-term personnel impacts • Stewardship + Workflow + Continuous Monitoring = Long-term personnel impacts • Data Flow + Workflow + Continuous Monitoring = Changes to processes It is a result of using these formulas that people come to understand that information quality truly is the integration of people. Data Quality Strategy: A Step-by-Step Approach  . business objects. and technology in the pursuit of deriving value from information assets. The size (number of records/rows) and scope (number databases.

True quality management is. impacting one or two users. and many individual projects. The project manager should adhere to the sound project management principals and concepts that apply to any project. scheduling. and the data sets established. Strategy formulation often encompasses a greater scope than a single project and can support the goals of an entire enterprise. While the goals have been documented. the greater the returns. Few of these tactical issues are covered in a strategy-level plan. can fit on one page of paper. can be considered data quality project planning. it does set the stage for a clear transition from strategy formulation to the detailed management of the tasks and actions that ensure its success. Strategy plans are “living” work products. and involved in one or two processes. and project monitoring against critical success factors. a repeatable process. Thus an initial strategy planning process is part of a larger recurring cycle. estimation. strategy must be implemented through a series of tactics and actions. for example) in one table. Data Quality Strategy: A Step-by-Step Approach  . However. business objects. such as task formulation. A simple strategy for this project. risk analysis. Data quality project risks can be mitigated by a more comprehensive strategy. numerous programs. comprehensive or narrowly focused—it can be handed to the project manager and everything he or she needs to know to plan the project should be in that document. Sooner or later. A typical pilot project may involve one column of data (phone numbers. Each successful pilot builds inertia. Once a strategy document is created—big or small. resource assignments. abilities. if not all of the work resolving the six factors. encompassing all six factors. the more challenging the goals of a data quality strategy. much. Another facet of a successful data quality strategy is consideration of the skills. the project manager must build the project requirements from the goals. or pilots. If the concept of data quality is new to your organization. which fall in the realm of project management. A complex project can be subdivided into mini-projects. Simple strategies fit pilot projects. In truth. Be aware that the initial strategy is a first iteration. While the purpose of this paper is not to cover the deep subject of data quality project management. a simple strategy is best. after all.iMpleMentation anD proJect ManageMent Where the data quality strategy formulation process ends. Successful pilots will drive future initiatives. An organization must accept that with greater returns come greater risks. mitigation. data quality project management takes over. This is not to say all the work has been done. And therein lies a strategy in itself: divide and conquer. and culture of the organization.

or time. and examples of category are street address. and regular review of the reports scheduled for the data stewardship team. or map of where the data resides2 • A discussion of cleansing solutions per category of data3 • Data flow diagrams of applicable existing data flows • Workflow diagrams of applicable existing workflows • A plan for when and where the data is accessed for cleansing4 • A discussion of how the data flow will change after project implementation • A discussion of how the workflow will change after project implementation • A list of stakeholders affected by the project • A plan for educating stakeholders as to the benefits of the project • A plan for training operators and users • A list of data quality measurements and metrics to monitor • A plan for when and where to monitor5 • A plan for initial and then regularly scheduled cleansing Examples of type are text. contact name. 3 This should include possible and desired deployment options for the cleansing solution. and so on. 1 business objects. the core practices have been extracted from the factors and listed here. date. See the section entitled Workflow for specific deployment options. schema. • A statement of the goals driving the project • A list of data sets and elements that support the goal • A list of data types and categories to be cleansed1 • • A catalog. part number. Data Quality Strategy: A Step-by-Step Approach  . 5 This includes running a baseline assessment. database. 4 This covers the when (during what steps in the data flow and workflow will the cleansing operation be inserted) and the where (on what data systems will the cleansing operation be employed) of the cleansing portion of the project. and so on. Reports from the recurring monitoring will need to be posted. and then selecting tests from the baseline to run on a regular basis. 2 This can include the name of the LAN. server.appenDix a:Data Quality Strategy checkliSt To help the practitioner employ the data quality strategy methodology.

More information about Business Objects can be found at www. Data Quality Strategy: A Step-by-Step Approach  .com. The company helps illuminate understanding and decision-making at more than 44.000 organizations around the globe. business objects.about buSineSS obJectS Business Objects. global consulting and education services. accurate. an SAP company. and timely information. as the world’s leading BI software company. and the industry’s strongest and most diverse partner network. Business Objects transforms the way the world works through intelligent information. Today. has been a pioneer in business intelligence (BI) since the dawn of the category.businessobjects. Through a combination of innovative technology. Business Objects enables companies of all sizes to make transformative business decisions based on intelligent.

409. the Spectrum Design. Workmail and Xcelsius are trademarks or registered trademarks in the United States and/or other countries of Business Objects and/or affiliated companies.831. Star Tree. Web Intelligence. Timewall.986. 6.139. 5.419. 6.businessobjects. BusinessQuery. 6. 7.130.440 and 7. Table Lens.768. RapidMarts.857. LinguistX.194. 7.998.435. Crystal Analysis. BusinessObjects.247. Business Objects owns the following U. March 2008 WP3131-A . NSite. 6.766. 7. 7.181.299.593.222. Crystal Enterprise. 7.882. Inxight.668.181.772. which may cover products that are offered and licensed by Business Objects: 5. patents. Crystal Insider. Desktop Intelligence. 6.com © 2008 Business Objects.490.352. All rights reserved.205.555.289. Crystal Decisions. 6. Crystal Applications. Business Objects Crystal Vision.008. the Inxight Logo.465. Crystal Reports.027. Business Process On Demand. Rapid Marts. Let there be light.403. 6. ThingFinder. Metify.S. All other names mentioned herein may be trademarks of their respective owners. Business Objects and the Business Objects logo. 6.578.

Sign up to vote on this title
UsefulNot useful