EII – dead on arrival

Author: Andy Hayler

The computer industry loves its buzzwords, and one that has cropped up in recent years in
“enterprise information integration” (EII). The idea of this is that everyone knows that companies
have their data locked up in multiple, incompatible IT systems: ERP, CRM, supply chain, etc. At
present, the only way to make sense of it is to extract data from these systems, try and resolve
inconsistencies and data quality issues, and then load the result into a data warehouse, from
which you can report on the data in a common form. Unfortunately this approach is hard: you
discover that the data quality in even the shiniest new ERP systems is not what it might be, you
have to unravel the differences between the way that various business units classify products,
channels and customers, then you have to design and build a data warehouse, and the subset
“data marts” from which you can report using one of the many well-established reporting tools
around (such as Business Objects).

EII vendor’s technology has genuine application in trying to answer questions like “give me a view
of the all the data we have on customer x”, which involve access to current data, what some term
“lightweight BI”. However they have recently been peddling their products for more general
business intelligence applications. After all, why go to all the trouble of building a data warehouse
when someone can come along with a technological magic wand? Vendors with “EII” solutions
have whitepapers that scorn today’s approach to business intelligence, promising that their
technology can merely look at all those inconsistent source systems and somehow run queries
that will give the answers without having to go through all that dull work of building and
populating a data warehouse.

Well – that’s it then: what were we all thinking? The data was there all the time in the source
systems, and for over a decade people have unaccountably been copying it somewhere else in
order to report on it; what a bunch of dopes they were! How much simpler just to access the data
directly in real-time from the sources: how very “real time enterprise”.

Some people who should know better have swallowed this EII mirage hook, line and sinker, and a
number of start-up companies have been funded flaunting “EII for business intelligence”
messages. The only problem with this new futurist approach is that it is absolutely and utterly
flawed.

Let’s consider the problem again. You have data in dozens of incompatibly structured source
systems. Your new EII software is somehow going to build a presumably fairly complex set of
distributed queries that will zip off to the source transaction systems, interrogate them and bring
back a result set that will somehow produce a consistent answer.
The first problem is: how exactly does the EII software know what the linkages are between the
differently coded source system structures? Somewhere it is going to have to have a catalogue
which will translate the differences, rather like a dictionary to translate words from one language
to another. This sounds suspiciously like a metadata dictionary of the type that data warehouses
have to construct, but let’s leave that aside for the moment.

What exactly happens when those distributed queries make their way through to the source
systems? For a start, the unpredictable nature of queries will upset the careful load balancing
done by operations departments to optimize on-line throughput. Or rather, it won’t, because no
systems managers are going to allow this technology anywhere near their delicately balanced
systems, at least not after the first time it brings the ordering system to a grinding halt.

The next problem with the EII approach is that there is no history. For transaction systems, you
want to archive data quickly in order to maintain high performance (there is no need to worry
about what your account balance was last year, just what it is now; last year’s balance can be
archived). However for an inquiry like “show me the trend in account withdrawals over the last ye

Next, do these vendors really think that all the analysis hierarchies needed are embedded within
the ERP systems? To take the example of marketing, there are normally complex segmentation
hierarchies for analysis purposes that are usually held in entirely separate places from the core
transaction systems, and are not stored along with each order or invoice.

Just as importantly, the EII tools entirely ignore the tedious problem of data quality. It may be
news to vendors who have more experience producing Powerpoint slides than production code,
but the quality of data lurking in the transaction systems is not what it might be. This is why there
is an industry of products to assist with improving data quality, and why a significant chunk of any
data warehouse project budget is associated with data quality. Oh that’s right; you don’t need a
data warehouse anymore, so I guess you may as well ignore that pesky data quality problem as
well.

Finally, what happens if there is actually a change in the structure of the transaction system e.g. a
change to the general ledger structure, or the way in which the back accounts are grouped,
perhaps following a reorganization? Disappointment using the EII approach, since no history of the
hierarchies in place at the time is kept. To be fair this problem can also challenge conventional
data warehouse approaches, but at least it can be tackled, albeit with difficulty.

So, with EII for business intelligence, you can’t deal with business change at all, data quality is
AWOL, you can’t look at trends, you are likely to dim the lights in the computer room and cause
the key operational systems of the company to come to a grinding halt. Other than that it is a
great idea.

Next time someone tries to sell you some software that appears to be a bit too close to sleight of
hand, check very carefully the customer references of people actually using the software in this
way. According to a leading industry analyst, only two EII vendors can give any decent customer
references at all. The software industry has years of practice of writing convincingly argued
whitepapers that spin a compelling case, yet only when customers hand over hard cash do they
seriously invest in the development to make it work. Always remember the caution used in the
wonderful film the Princess Bride: “Life is pain, and anyone who tells you different is trying to sell
you something.”

About the Author

Andy is an established enterprise software industry expert and commentator, named a Red
Herring Top 10 Innovator in 2002. Andy founded Kalido as an independent software company
after originally setting up the software venture within the Shell Group. He became an
independent consultant in August 2006.

Prior to leading Kalido's spin off from Shell in June 2003, Andy was CEO of Kalido Ltd in January
2001. In previous roles at Shell, Andy led a 290-person global consultancy practice of Shell
Services International, and was Technology Planning Manager of Shell UK Oil. Prior to Shell,
Andy worked in a number of senior technology positions within Exxon.

A 20-year veteran of data warehousing and integration projects, Andy is a regular speaker at
international conferences such as ETRE, Tornado Insider, Red Herring, Gartner and Enterprise
Outlook. See his award winning blog www.andyonenterprisesoftware.com for his insights on the
industry.

Andy has a BSc (Hons) Mathematics degree from Nottingham University.