E-Guide

BI INTEGRATION
CHALLENGES:
DATA ERRORS,
BIG DATA AND
REAL-TIME NEEDS

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

ser demands for faster access to business

intelligence data and the rise of big data

analytics are complicating data integration for BI systems. This e-guide examines both the drivers of BI data integration and key roadblocks,
including data quality and data loading issues. Find out how new
data integration techniques and tools can help organizations move
forward with integration projects.

PA G E 2 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

BI DATA INTEGRATION PROCESS CHALLENGED BY

REAL-TIME NEEDS, 'BIG DATA'
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

Alan Earls
Business intelligence (BI) systems and their supporting data warehouses are
only as good as the data that goes into them. And if you arent properly handling
the BI data integration process, your end users -- and ultimately, your organization -- may be in for trouble.
With BI tools becoming more and more pervasive in organizations, and
more critical to the success of business operations, making sure that you have
a well-designed and well-executed process for integrating BI data is of paramount importance, according to data management analysts such as Ted Friedman of Gartner Inc.
Friedman said Gartner sees data integration challenges related to BI as a
drag on the success of BI and analytics initiatives -- and a big reason for outright
project failures.
As the data that organizations are trying to harness gets more and more
complex, with more kinds and sources of data and now big data thrown into
PA G E 3 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

the mix, a significant amount of time and effort is involved in matching, cleaning and preparing data for BI applications, he said. Its a darned hard problem,
particularly when you add in older, legacy systems where you sometimes need
to do archaeology first in order to interpret the data.
Another complicating factor is that things are changing in the world of data
integration technology as business users demand faster access to BI data.
ETL STILL BEST BET FOR BI DATA INTEGRATION?

The traditional workhorse technology for managing BI data integration is extract, transform and load (ETL) software that pulls data from source systems
in bulk batch processing jobs. Friedman said newer data integration techniques
offer lower latency than ETL tools do. For example, change data capture software and other real-time data integration tools let you push new or modified
information to data warehouse and BI systems in real or near real time, which
can be particularly useful for tasks like fraud detection. It is streaming [data]
in granular form rather than big chunks in batch, which is what ETL is using,
he said.
Another option: federated and virtualized approaches to data integration
and delivery that dont move the data out of source systems at all but instead
PA G E 4 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

create consolidated views of data from multiple sources for BI uses. With data
virtualization tools, the integrated data doesnt persist anywhere, Friedman
said. Youre grabbing it in real time and joining it together and making it seem
as if it is one database somewhere to the applications using it.
Despite the emergence of this new wave of data integration and delivery
tools, though, Friedman thinks it would be a mistake to view ETL software as
obsolete or no longer valuable. ETL is still relevant, he said. We think there
will always be a role for ETL-style processing because not all data can or should
be delivered in real time.
Indeed, Friedman warned that data integration vendors are pushing sexy
real-time options for BI data integration when many organizations can still get
what they need from a batch approach. Real-time [integration] costs money
and it requires a change from what organizations have been doing, so there
needs to be a strong business case for it, he said.
ETL still has a role -- it is the heavy lifter of data integration, agreed Claudia Imhoff, president of Intelligent Solutions Inc., a consultancy in Boulder,
Colo. Still, she noted that its newer competitors can be more flexible and faster
to deploy and are better suited to delivering timely data to business users for
operational BI applications.
PA G E 5 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

REAL TIME NOT ALWAYS RIGHT BUT MORE OF A REALITY

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

Although he acknowledges that real-time data integration for BI is frequently

neither necessary nor desirable, Barry Devlin, founder of 9sight Consulting
in Cape Town, South Africa, points out that BI and analytics applications are
increasingly moving in that direction. I think it is a really interesting time in
terms of how this will pan out, he said.
As an example of an experimental use case, Devlin cited the U.S. insurance
industry, where real-time data from cars -- braking and speed data, time spent
driving and other information -- is being transmitted to business users at insurance companies through mobile phone networks, enabling the insurers to
modify premiums or even provide rebates on the fly.
As Friedman noted, the increasing focus on capturing and analyzing big
data, including Web server logs, social media data and other forms of unstructured information, adds another layer of complexity to the BI data integration
process within many organizations.
James Kobielus, until recently an analyst at Forrester Research Inc., said
while he was still working there that unstructured data can be as critical as
structured data to what youre doing in BI and analytics. Even companies that
are still planning or just beginning to implement big data analytics programs
PA G E 6 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

should look ahead and make sure theyre prepared for the data integration challenges ahead, added Kobielus, who has since taken a job at IBM. You need to
be ready, he said, for things like massive data inputs from social media and
start to budget and staff up.

DATA ERRORS, OTHER MISSTEPS CAN WAYLAY BI DATA

INTEGRATION STRATEGY
Data integration is a critical component of business intelligence (BI) processes,
which are predicated on pulling together data from multiple source systems
and consolidating it in data warehouses for analysis. But data management
analysts warned that a BI data integration strategy can quickly go awry if it isnt
well designed and properly executed.
For example, Claudia Imhoff, president of consultancy Intelligent Solutions Inc. in Boulder, Colo., said bad timing can spoil everything: Data needs to
be loaded into a data warehouse in time to be used for the planned BI purposes.
Fully understanding an organizations BI requirements, particularly when
PA G E 7 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

users need real- or near-real-time access to data, is a must for BI and data integration teams, Imhoff said.
Good data quality is just as important, according to Imhoff, who said that
correcting and cleaning up bad data shouldnt be a function solely of the BI data
integration process. Errors are happening everywhere else along the way, so
you need to figure out where they are coming from, she said -- and then work
to prevent data mistakes from finding their way into source systems in the first
place. In effect, Imhoff added, data integration and BI professionals are given
the job of consolidating faulty data and then get the blame when it isnt perfect.
We need to get people to understand that they shouldnt just shoot the messenger, she said.
Ted Friedman, an analyst at Gartner Inc. in Stamford, Conn., thinks that
not paying enough attention to data quality is the biggest BI data integration
danger companies face. Ive been following data integration for more than
10 years, he said. And I still spend days talking to organizations that are not
getting the usage and trust and acceptance and value out of their BI efforts
because the quality of the data is not good enough, and they havent done the
right things to fix that.
Data quality problems clearly affect more than BI data in wayward
PA G E 8 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

organizations, Friedman said, but he sees poor data quality as one of the primary barriers to successful BI programs. The shortcomings, he added, typically
result from not focusing [on data quality] early and often enough, and simply
not doing enough to mitigate quality issues as information is moved into data
warehouses.
James Kobielus, who was an analyst at Forrester Research Inc. in Cambridge, Mass., before taking a job with a technology vendor earlier this year,
also pointed to missteps on data quality as a common source of trouble for BI
data integration efforts.
Organizations think they can simply load data from their various back-end
applications into a data warehouse and it will be usable without cleansing it or
doing match-and-merge or transform [processes], Kobielus said while he was
still at Forrester. But doing so sets up companies for some nasty surprises, he
added. For example, they end up with six records on the same person and dont
know which one is the right one, Kobielus said.
BI DATA INTEGRATIONS DRAMATIC EFFECT

Another big source of inconsistent data, and drama, stems from internal debates over what constitutes a system of record, said Jill Dyche, co-founder of
PA G E 9 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

Baseline Consulting in Sherman Oaks, Calif. For example, she noted, there can
be arguments about which transaction system should be used as the source of
customer addresses. Such conversations often then turn to the definition of
address: Is it a customers billing address or shipping address, or its headquarters location if that differs from the other two?
Thats when the arguments ensue and business people become disaffected
with the BI teams ability to understand and deliver the right data, Dyche said.
So then someone just decides to forklift everything into a single database,
which the business people then refuse to use.
Barry Devlin, founder of 9sight Consulting in Cape Town, South Africa,
thinks the most problematic mistake is not including the right people in the
process of crafting a BI data integration strategy and plan. The people who
really understand data and what it means are a particular subset of the business community who have been playing with data over the years -- they are
the gurus and the power users, Devlin said. As a result, he added, theyre best
equipped to define what data needs to be integrated in order to create effective
BI applications.
But in many cases, its left to the IT department to develop the data integration plan in addition to doing the implementation work, Devlin said. While IT
PA G E 1 0 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

pros may have a reasonable understanding of an organizations data, to Devlin

they arent the real experts. Bringing the two groups together to work on BI data
integration can be a challenge, but its a must, he said.
Imhoff said there also is a strong tendency, especially in organizations that
are new to BI, to inadequately scope the data integration requirements of a
BI project and then put together an unrealistic schedule for delivering whats
needed. Integrating data and loading it into a data warehouse can take up as
much as 60% to 80% of the overall BI development effort, Imhoff said. And, she
cautioned, a project team that tries to do too much of that at one time can end
up falling flat on its face. You cant eat the whole elephant at once, she said.

PA G E 1 1 O F 1 2

SPONSORED BY

BI INTEGRATION CHALLENGES: DATA ERRORS, BIG DATA AND REAL-TIME NEEDS

FREE RESOURCES FOR TECHNOLOGY PROFESSIONALS

Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy

TechTarget publishes targeted technology media that address

your need for information and resources for researching products, developing strategy and making cost-effective purchase
decisions. Our network of technology-specific Web sites gives
you access to industry experts, independent content and analysis and the Webs largest library of vendor-provided white papers, webcasts, podcasts, videos, virtual trade shows, research
reports and more drawing on the rich R&D resources of technology providers to address
market trends, challenges and solutions. Our live events and virtual seminars give you access to vendor neutral, expert commentary and advice on the issues and challenges you
face daily. Our social community IT Knowledge Exchange allows you to share real world
information in real time with peers and experts.

WHAT MAKES TECHTARGET UNIQUE?

TechTarget is squarely focused on the enterprise IT space. Our team of editors and network of industry experts provide the richest, most relevant content to IT professionals and
management. We leverage the immediacy of the Web, the networking and face-to-face opportunities of events and virtual events, and the ability to interact with peersall to create
compelling and actionable information for enterprise IT professionals across all industries
and markets.

PA G E 1 2 O F 1 2

SPONSORED BY