Black Swans & White Elephants

Return On Investment In Business Intelligence

David M Walker
Data Management & Warehousing ETIS ‒ Istanbul ‒ Oct 09

Return  on  Investment  
  A  performance  measure  used  to  evaluate  the  efficiency  of  an   investment  or  to  compare  the  efficiency  of  a  number  of   different  investments.  To  calculate  ROI,  the  benefit  (return)  of   an  investment  is  divided  by  the  cost  of  the  investment;  the   result  is  expressed  as  a  percentage  or  a  ratio.       The  return  on  investment  formula:   ROI  =  (Gain  from  Investment  –  Cost  of  Investment)   Cost  of  Investment     Return  on  investment  is  a  very  popular  metric  because  of  its   versatility  and  simplicity.  That  is,  if  an  investment  does  not  have   a  positive  ROI,  or  if  there  are  other  opportunities  with  a  higher   ROI,  then  the  investment  should  not  be  undertaken.  

So  where  are  the  ROI  Metrics?  
  Clearly  valuing  BI  is  not  an  exact  science  and  often  comes  down   to  mindset.  Comparing  BI  to  a  college  education:  It  may  be   expensive  and  time-­‐consuming,  but  there  are  many  less   tangible  benefits,  like  increased  earning  power  and  overall   improved  quality  of  life,  which  come  years  later.     It's  not  easy  to  persuade  someone  to  go  to  college  based  on  a   purely  financial  or  numbers  game,  and  the  same  thing  goes  for   BI.    You  just  have  to  believe  that  BI  is  absolutely  essential  for  you   as  an  organization  to  invest,  that  this  is  a  fundamental  core   competency  that  you  have  to  have.      
Bill  Hostmann,  Vice  President  &  Analyst,  Gartner,  March  2008  

What  is  Good  BI?  
  Good  BI  is  the  fusion  of  the  right  information,  the  right  time,   the  right  format,  and  the  right  human  and/or  system  resources.   If  we  wish  to  improve  BI,  we  ask  these  questions:  
  Do  business  users  have  the  information  needed,  when  they  need  it,   to  make  decisions?     Do  those  people  have  the  expertise,  training  and  mindset  to  use   that  information  in  the  best  way  for  the  good  of  the  organization?     Are  they  doing  their  job  better  because  of  the  information  being   delivered?     How  much  difference  does  that  information  make  to  them?     Dorothy  Miller,  BI  Metrics,    Feb  2009  

  The  best  approach  to  evaluating  a  BI  solution  is  a  technical   review  combined  with  a  business  user  perception  survey  

  No  organisation  has  ever  delivered  the  ROI  originally   promised:  
  The  business  has  moved  on  in  the  time  it  takes  to  develop   the  (original)  solution  meaning  that  the  value  has  diminished      The  biggest  gains  and  losses  have  come  from  Black  Swans   and  not  planned  events     The  biggest  costs  have  come  from  White  Elephants  and  the   failure  to  recognise  where  the  costs  are  hidden  

  Some  (mainly  those  highly  committed  to  BI)  have  far   exceeded  the  promised  ROI  but  most  fail  to  achieve  it    
  Even  if  the  development  costs  are  on  budget  the  on-­‐going   OPEX  costs  are  hidden  and  far  out-­‐weigh  the  the  benefit  

Black  Swans  
  High-­‐impact,  hard-­‐to-­‐predict,  and  rare  events  beyond  the  realm   of  normal  expectations     The  term  Black  Swan  comes  from  the  17th  century  European   assumption  that  'All  swans  are  white'.  In  that  context,  a  black   swan  was  a  symbol  of  something  that  was  impossible  or  could   not  exist.  In  the  18th  Century,  the  discovery  of  black  swans  in   Western  Australia  metamorphosed  the  term  to  connote  that  a   perceived  impossibility  may  actually  come  to  pass.  

Black  Swans:   Product  Master  Data  
  Data  Warehouse  requirements  delivered     Data  Warehouse  analysis  identifies  no  management  of  product   master  data  and  no  product  master  list     Project  risk  raised  and  escalated  to  executive  level     BI  Project  postponed  and  staff  re-­‐tasked  to  define  and  deliver   Corporate  Product  MDM     Implements  ERP  based  MDM  solution  and  reduces  product   catalogue  by  90%     Directly  affects  corporate  bottom  line     Restarts  BI  Project  six  months  later  with  massively  improved   data  quality  and  data  integration  

Black  Swans:   Financial  Reporting  
  Telco  in  DotCom  boom  consistently  overstates  earnings  and   subscriber  growth  to  boost  share  price     Data  Warehouse  project  identifies  issue  and  produces  a  set  of   correct  numbers     Business  refuses  to  adjust  statutory  reporting  as  they  have  been   using  the  ‘Excel’  numbers,  it  would  be  embarrassing  to  restate   the  figures  and  the  new  ones  are  just  too  different     Business  collapses  and  is  taken  over,  national  newspapers   report  that  ‘internal  systems’  had  the  correct  figures     The  failure  to  use  real  data  was  highlighted  by  auditors  as  a   significant  factor  in  the  poor  decisions  made  by  management  

Black  Swans:   The  Salesman  Fraud  
  Supplier  of  confectionary  to  a  major  supermarket  chain     First  Release  of  Data  Warehouse     CEO  was  un-­‐happy  with  the  numbers  from  the  new  system  
  He  knew  that  the  supermarket  was  their  biggest  customer  

  IT  defended  the  Data  Warehouse  
  Every  last  loading  step  and  transformation  had  been  tested  &  checked  

  A  Salesman’s  commission  doubled  for  the  first  quarter  on  new   accounts  before  dropping  to  standard  rate  in  subsequent  quarters     The  Salesman  responsible  for  the  supermarket  chain  created  a  new   customer  each  quarter       Corporate  revenues  affected  enough  for  an  exchange  filing  

Black  Swans:   External  Collapse  
  Sudden  market  collapse     Many  customers  suddenly  change  profile  
  Looking  for  value  –  lower  spend  

  Be  Pro-­‐active  
  An  opportunity  to  demonstrate  the  true  value  of  BI     (It  might  save  your  job  too!)  

White  Elephants  
  A  white  elephant  is  a  valuable  possession  of  which  its  owner   cannot  dispose  and  whose  cost  (particularly  cost  of  upkeep)  is   out  of  proportion  to  its  usefulness.     The  term  derives  from  the  sacred  white  elephants  kept  by   Southeast  Asian  rulers.  To  possess  a  white  elephant  was  (and   still  is)    regarded  as  a  sign  that  the  monarch  was  ruling  with   justice  and  power,  and  that  the  kingdom  was  blessed  with   peace  and  prosperity.  The  animals  were  considered  sacred  and   laws  protected  them  from  labour,  receiving  a  gift  of  a  white   elephant  from  a  monarch  was  both  a  blessing  and  a  curse:  a   blessing  because  the  animal  was  sacred  and  a  sign  of  the   monarch's  favour,  and  a  curse  because  the  animal  had  to  be   kept  and  could  not  be  put  to  practical  use  to  offset  the  cost  of   maintaining  it.  

White  Elephants   Convention  over  Configuration  
  Bank  with  existing  data  warehouse     Extension  to  support  Balanced  Scorecard     Wanted  to  use  a  different  data  modelling  technique  in  the  same  data   warehouse  for  the  new  elements       Justified  as  a  management  decision  because  “we  can’t  afford  to  re-­‐ develop  the    old  model  but  we  want  the  best  technique  for  the  new  parts”     Resultant  model  could  simply  not  be  used     ‘On  Rails’  
  Values  “DRY  -­‐  Don’t  repeat  yourself”  &  “Convention  over  configuration”     Use  each  component  in  a  consistent  way  
  Aids  Understanding  and  Reduces  Maintenance  

  Use  each  component  for  the  purpose  it  was  intended  

White  Elephants:   Not  evaluating  technology  
  Second  Generation  Data  Warehouse  Build  
  Existing  platform  major  RDBMS     Recommended  solution:  Componentised  architecture  with  high   performance  database  engine  and  simple  ETL  architecture   developed  by  small  in  house  team     Chosen  solution:  Existing  RDBMS  vendor  with  tightly  coupled  ETL   tool  from  a  different  vendor  and  major  SI  doing  development  on   site  –  “because  that’s  the  way  we  do  it  here  and  management   won’t  consider  anything  else”     Current  issues:  
  Considered  too  costly  and  is  being  delivered  late     Already  having  to  review  technology  choices  because  of   performance  issues  before  roll-­‐out  complete     SI  has  left  taking  all  the  knowledge  with  them  

White  Elephants:   Making  an  ETL  White  Elephant  
  Existing  data  warehouse  in  personal  finance  company  with  a   ETL  load  built  from  SQL  scripts  and  a  shell  script  and  tight   control  via  SVN  in  production  for  3  years     New  IT  Director  commissions  review  by  vendor  that   recommends  the  vendors  ETL  tool  for  US$400K     Six  Months,  4    Consultants  @  US$1.5K  per  day     (a  total  spend  >US$600K)    later  …  
  All  ETL  converted     Runs  20%  slower     Other  BI  developments  delayed  whilst  changes  made  

  Deemed  TOO  EXPENSIVE  to  revert  back    
  Eventually    reverted  back  12  months  later  when  vendor  quoted  for   upgrade  to  system  

White  Elephants:   Breaking  an  ETL  White  Elephant  
  Interactive  Media  corporation  moves  from  traditional  RDBMS  to   commodity  appliance  technology       Data  Volumes  doubling  every  six  months     Internal  Review  
            ETL  tools  can’t  handle  load   ETL  experts  too  expensive  for  long  term  engagements   SQL  scripts  can  be  developed  by  more  resources   SQL  scripts  allow  more  work  to  be  done  in  the  appliance   SQL  scripts  allow  more  agile  approach   SQL  scripts  allow  tighter  change  management  

  New  architecture  componentised,  commodity  based  with  simple   script  engine  reduces  costs  by  90%  and  increased  productivity  by   100%  

White  Elephants:   The  OPEX  trap  
  BI  Projects  command  headline  CAPEX  budget  figures     BI  Projects  are  widely  publicised  during  development     Most  of  the  money  is  spent  in  OPEX  
  It  comes  from  User  Support,  Training,  Changes  and   Operational  Support     It  is  normally  2x-­‐3x  and  often  5x  more  than  the  cost  of  the   build  over    the  lifetime  of  the  system     It  is  hidden  –  spread  over  multiple  budgets  in  such  a  way  that   it  is  hard  to  evaluate  and  often  ignored  

  Any  action  (especially  tactical  actions)  in  the  build/test   stage  that  will  result  in  increased  OPEX  is  deadly  to  ROI  

White  Elephants   The  New  Reporting  Tool  
  Data  Warehouse  Second  Generation  Build  
  New  project  –  new  interface  concept     Current  system  users  were  ‘un-­‐happy’  with  the  existing  tool  
  The  reality  was  largely  issues  with  the  data  model  and  data   quality    of  the  current  system  

  Replaced  with  an  equivalent  reporting  tool  

  The  cost  
        Retraining  of  over  2000  staff   Failure  to  de-­‐commission  old  reporting  tool   50%  too  many  licences  bought  for  the  new  tool   User  dissatisfaction  with  the  (new)  reporting  tool  

White  Elephants:   Reporting  Tool  Penetration  
  Business  intelligence  vendors   like  to  talk  up  a  20/80  split:   only  20  percent  of  users  are   actually  consuming  BI   technologies;  the  remaining   80  percent  are   disenfranchised.       Reasons     Security  limitations     Slow  query  performance       Internal  politics  and   (more  precisely)  internal   power  struggles     Nigel  Pendse  of  BI  Survey   shows  just  over  8  percent  of   employees  are  actually  using   BI  tools.       Even  in  industries  that  have   aggressively  adopted  BI  tools   (e.g.,  wholesale,  banking,  and   retail),  usage  barely  exceeds   11  percent.  
Hardware  cost     Data  availability   Software  cost     Software  was  too  hard  to   use     User  scalability          

White  Elephants:   Not  testing  the  delivery  
  Any  delivery  has  stages  
  Requirements  /  Analysis  /  Design  /  Build  /  Test/  Deploy  etc.  


Every  stage  can  have  tests  associated  with  it  
          Requirements:  Mind  Experiment  Method   Analysis:  Cross-­‐checking  with  other  sources   Design:  Algorithmic  Checks   Build:  Unit  Tests  /  Boundary  Checking   Testing:  System  Tests  /  Integration  Test  


Test  early  and  often  
    Fixing  something  1  stage  later  =  2x  more  expensive   Fixing  something  2  stages  later  =  4x  more  expensive,  etc.  


Many  more  small  tests  performed  early  will  save  huge  amounts  of  time  and  money  
    Have  to  handle  short  term  slippages   If  you  haven’t  got  time  to  test  then  you  are  planning  to  spend  OPEX  


Massive  user  perception  impact  as  information  is  right  first  time  

White  Elephants   Garbage  In  ::  Garbage  Out  
‘On  two  occasions  I  have  been  asked  "Pray,  Mr.  Babbage,  if  you  put  into  the  machine  wrong   figures,  will  the  right  answers  come  out?"  …  I  am  not  able  rightly  to  apprehend  the  kind  of   confusion  of  ideas  that  could  provoke  such  a  question.’  –  Charles  Babbage  1791-­‐1871  

  Data  Quality  is  more  critical  than  ever     A  failure  to  address  Data  Quality  at  every  stage  will  always   lead  to  additional  costs     Plan  for  your  BI  project  to  spin  off    dozens  of  data  quality   projects  and  continue  to  do  so  throughout  its  life     Data  Quality  issues  drive  users  away  which  directly   increases  cost  of  ownership  and  reduces  ROI    

White  Elephants:   De-­‐commission!  
  Data  Warehouse  Component  Delivered  
  De-­‐commission  the  previous  reporting  system     NO  -­‐  REALLY  -­‐    TURN  IT  OFF  !  

  If  it  is  left  on:  
  It  costs  money  to  run  (and  it  is  all  invisible  OPEX)     Users  will  compare  results  and  distrust  the  new  system  even   if  it  can  be  proved  to  be  more  correct  than  the  old  system     Users  will  continue  to  use  the  old  one  because  it  is  familiar   and  no-­‐one  likes  change     Users  that  do  not  migrate  but  have  been  trained  on  the  new   system  will  have  to  be  re-­‐trained  when  they  start  to  use  it    

The  Solution  
  The  proposition  suggests  that  doing  BI  has  not  delivered   the  expected  ROI  for  most  organisations.       Does  this  mean  that    organisations  should  stop  developing   BI  solutions?     Categorically  NO  –  BI  should  be  hugely  worthwhile     But    remember  you  can  only  succeed  if:  
        Your  organisation  fully  embraces  Business  Intelligence   You  expect  and  embrace  Black  Swans   You  avoid  and  mitigate  White  Elephants   You  make  your  own  luck  

How  to  be  lucky  
  Lucky  people  frequently  happen  upon  chance  opportunities  
  There  are  positive  black  swans  in  every  organization,  it  is  just  a  matter  of  identifying   them  as  they  occur  

  Lucky  people  listen  to  their  hunches  
  Members  of  your  team  will  have  insights  beyond  their  remit  based  on  their   experience,  identify  these  people  and  exploit  their  insights  

  Lucky  people  persevere  in  the  face  of  failure  
  Every  project  will  face  set-­‐backs  -­‐  plan  for  them  

  Lucky  people  have  the  ability  to  turn  bad  luck  into  good  fortune  
  Every  project  will  face  set-­‐backs  -­‐  embrace  them  as  an  opportunity  and  change  your   project/remit  and  reset  your  goals  

 Based  on  work  by  Prof.  Richard  Wiseman,  University  of  Hertfordshire,  2003  

Improving  ROI  
  Black  swans  (handled  positively)  massively  increase     Gain  from  investment     White  elephants  (eliminated)  significantly  reduce  the     Cost  of  the  Investment     Whatever  the  CAPEX  investment  of  the  project  
the  OPEX  will  be  significantly  more  –  design  for  this  

Black Swans & White Elephants
Return On Investment In Business Intelligence