This action might not be possible to undo. Are you sure you want to continue?
Return On Investment In Business Intelligence
Data Management & Warehousing ETIS ‒ Istanbul ‒ Oct 09
A performance measure used to evaluate the eﬃciency of an investment or to compare the eﬃciency of a number of diﬀerent investments. To calculate ROI, the beneﬁt (return) of an investment is divided by the cost of the investment; the result is expressed as a percentage or a ratio. The return on investment formula: ROI = (Gain from Investment – Cost of Investment) Cost of Investment Return on investment is a very popular metric because of its versatility and simplicity. That is, if an investment does not have a positive ROI, or if there are other opportunities with a higher ROI, then the investment should not be undertaken.
Clearly valuing BI is not an exact science and often comes down to mindset. Comparing BI to a college education: It may be expensive and time-‐consuming, but there are many less tangible beneﬁts, like increased earning power and overall improved quality of life, which come years later. It's not easy to persuade someone to go to college based on a purely ﬁnancial or numbers game, and the same thing goes for BI. You just have to believe that BI is absolutely essential for you as an organization to invest, that this is a fundamental core competency that you have to have.
Bill Hostmann, Vice President & Analyst, Gartner, March 2008
Good BI is the fusion of the right information, the right time, the right format, and the right human and/or system resources. If we wish to improve BI, we ask these questions:
Do business users have the information needed, when they need it, to make decisions? Do those people have the expertise, training and mindset to use that information in the best way for the good of the organization? Are they doing their job better because of the information being delivered? How much diﬀerence does that information make to them? Dorothy Miller, BI Metrics, Feb 2009
The best approach to evaluating a BI solution is a technical review combined with a business user perception survey
No organisation has ever delivered the ROI originally promised:
The business has moved on in the time it takes to develop the (original) solution meaning that the value has diminished The biggest gains and losses have come from Black Swans and not planned events The biggest costs have come from White Elephants and the failure to recognise where the costs are hidden
Even if the development costs are on budget the on-‐going OPEX costs are hidden and far out-‐weigh the the beneﬁt
High-‐impact, hard-‐to-‐predict, and rare events beyond the realm of normal expectations The term Black Swan comes from the 17th century European assumption that 'All swans are white'. In that context, a black swan was a symbol of something that was impossible or could not exist. In the 18th Century, the discovery of black swans in Western Australia metamorphosed the term to connote that a perceived impossibility may actually come to pass.
Data Warehouse requirements delivered Data Warehouse analysis identiﬁes no management of product master data and no product master list Project risk raised and escalated to executive level BI Project postponed and staﬀ re-‐tasked to deﬁne and deliver Corporate Product MDM Implements ERP based MDM solution and reduces product catalogue by 90% Directly aﬀects corporate bottom line Restarts BI Project six months later with massively improved data quality and data integration
Telco in DotCom boom consistently overstates earnings and subscriber growth to boost share price Data Warehouse project identiﬁes issue and produces a set of correct numbers Business refuses to adjust statutory reporting as they have been using the ‘Excel’ numbers, it would be embarrassing to restate the ﬁgures and the new ones are just too diﬀerent Business collapses and is taken over, national newspapers report that ‘internal systems’ had the correct ﬁgures The failure to use real data was highlighted by auditors as a signiﬁcant factor in the poor decisions made by management
Supplier of confectionary to a major supermarket chain First Release of Data Warehouse CEO was un-‐happy with the numbers from the new system
He knew that the supermarket was their biggest customer
Every last loading step and transformation had been tested & checked
A Salesman’s commission doubled for the ﬁrst quarter on new accounts before dropping to standard rate in subsequent quarters The Salesman responsible for the supermarket chain created a new customer each quarter Corporate revenues aﬀected enough for an exchange ﬁling
Sudden market collapse Many customers suddenly change proﬁle
Looking for value – lower spend
An opportunity to demonstrate the true value of BI (It might save your job too!)
A white elephant is a valuable possession of which its owner cannot dispose and whose cost (particularly cost of upkeep) is out of proportion to its usefulness. The term derives from the sacred white elephants kept by Southeast Asian rulers. To possess a white elephant was (and still is) regarded as a sign that the monarch was ruling with justice and power, and that the kingdom was blessed with peace and prosperity. The animals were considered sacred and laws protected them from labour, receiving a gift of a white elephant from a monarch was both a blessing and a curse: a blessing because the animal was sacred and a sign of the monarch's favour, and a curse because the animal had to be kept and could not be put to practical use to oﬀset the cost of maintaining it.
Bank with existing data warehouse Extension to support Balanced Scorecard Wanted to use a diﬀerent data modelling technique in the same data warehouse for the new elements Justiﬁed as a management decision because “we can’t aﬀord to re-‐ develop the old model but we want the best technique for the new parts” Resultant model could simply not be used ‘On Rails’
Values “DRY -‐ Don’t repeat yourself” & “Convention over conﬁguration” Use each component in a consistent way
Aids Understanding and Reduces Maintenance
Use each component for the purpose it was intended
Second Generation Data Warehouse Build
Existing platform major RDBMS Recommended solution: Componentised architecture with high performance database engine and simple ETL architecture developed by small in house team Chosen solution: Existing RDBMS vendor with tightly coupled ETL tool from a diﬀerent vendor and major SI doing development on site – “because that’s the way we do it here and management won’t consider anything else” Current issues:
Considered too costly and is being delivered late Already having to review technology choices because of performance issues before roll-‐out complete SI has left taking all the knowledge with them
Existing data warehouse in personal ﬁnance company with a ETL load built from SQL scripts and a shell script and tight control via SVN in production for 3 years New IT Director commissions review by vendor that recommends the vendors ETL tool for US$400K Six Months, 4 Consultants @ US$1.5K per day (a total spend >US$600K) later …
All ETL converted Runs 20% slower Other BI developments delayed whilst changes made
Eventually reverted back 12 months later when vendor quoted for upgrade to system
Interactive Media corporation moves from traditional RDBMS to commodity appliance technology Data Volumes doubling every six months Internal Review
ETL tools can’t handle load ETL experts too expensive for long term engagements SQL scripts can be developed by more resources SQL scripts allow more work to be done in the appliance SQL scripts allow more agile approach SQL scripts allow tighter change management
New architecture componentised, commodity based with simple script engine reduces costs by 90% and increased productivity by 100%
BI Projects command headline CAPEX budget ﬁgures BI Projects are widely publicised during development Most of the money is spent in OPEX
It comes from User Support, Training, Changes and Operational Support It is normally 2x-‐3x and often 5x more than the cost of the build over the lifetime of the system It is hidden – spread over multiple budgets in such a way that it is hard to evaluate and often ignored
Any action (especially tactical actions) in the build/test stage that will result in increased OPEX is deadly to ROI
Data Warehouse Second Generation Build
New project – new interface concept Current system users were ‘un-‐happy’ with the existing tool
The reality was largely issues with the data model and data quality of the current system
Replaced with an equivalent reporting tool
Retraining of over 2000 staﬀ Failure to de-‐commission old reporting tool 50% too many licences bought for the new tool User dissatisfaction with the (new) reporting tool
Business intelligence vendors like to talk up a 20/80 split: only 20 percent of users are actually consuming BI technologies; the remaining 80 percent are disenfranchised. Reasons Security limitations Slow query performance Internal politics and (more precisely) internal power struggles Nigel Pendse of BI Survey shows just over 8 percent of employees are actually using BI tools. Even in industries that have aggressively adopted BI tools (e.g., wholesale, banking, and retail), usage barely exceeds 11 percent.
Hardware cost Data availability Software cost Software was too hard to use User scalability
Any delivery has stages
Requirements / Analysis / Design / Build / Test/ Deploy etc.
Requirements: Mind Experiment Method Analysis: Cross-‐checking with other sources Design: Algorithmic Checks Build: Unit Tests / Boundary Checking Testing: System Tests / Integration Test
Fixing something 1 stage later = 2x more expensive Fixing something 2 stages later = 4x more expensive, etc.
Have to handle short term slippages If you haven’t got time to test then you are planning to spend OPEX
Massive user perception impact as information is right ﬁrst time
‘On two occasions I have been asked "Pray, Mr. Babbage, if you put into the machine wrong ﬁgures, will the right answers come out?" … I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.’ – Charles Babbage 1791-‐1871
Data Quality is more critical than ever A failure to address Data Quality at every stage will always lead to additional costs Plan for your BI project to spin oﬀ dozens of data quality projects and continue to do so throughout its life Data Quality issues drive users away which directly increases cost of ownership and reduces ROI
Data Warehouse Component Delivered
De-‐commission the previous reporting system NO -‐ REALLY -‐ TURN IT OFF !
It costs money to run (and it is all invisible OPEX) Users will compare results and distrust the new system even if it can be proved to be more correct than the old system Users will continue to use the old one because it is familiar and no-‐one likes change Users that do not migrate but have been trained on the new system will have to be re-‐trained when they start to use it
The proposition suggests that doing BI has not delivered the expected ROI for most organisations. Does this mean that organisations should stop developing BI solutions? Categorically NO – BI should be hugely worthwhile But remember you can only succeed if:
Your organisation fully embraces Business Intelligence You expect and embrace Black Swans You avoid and mitigate White Elephants You make your own luck
Lucky people frequently happen upon chance opportunities
There are positive black swans in every organization, it is just a matter of identifying them as they occur
Members of your team will have insights beyond their remit based on their experience, identify these people and exploit their insights
Every project will face set-‐backs -‐ plan for them
Every project will face set-‐backs -‐ embrace them as an opportunity and change your project/remit and reset your goals
Based on work by Prof. Richard Wiseman, University of Hertfordshire, 2003
Black swans (handled positively) massively increase Gain from investment White elephants (eliminated) signiﬁcantly reduce the Cost of the Investment Whatever the CAPEX investment of the project
the OPEX will be signiﬁcantly more – design for this
Return On Investment In Business Intelligence