Using Periodic Audits to Prevent Catastrophic Project Failure Dr. Paul Dorsey Dulcian, Inc.

May 22, 2008

1 of 30

The Vasa 

17th century (1628)  ³The greatest ship of all time´
years to build  2 gun decks with 64 cannons 

Gustavus Adolphus of Sweden set the measurements. 
1000 trees

were required  Triple laminated oak walls (18in/46cm thick)  Mast = 190 ft/57m  Length = 201 ft/61m 

= 5% of Sweden¶s GNP
2 of 30

Maiden Voyage 

sail on a beautiful summer day 

August 10,

1628  Passed the royal palace of Stockholm  Fired proud salutes from cannons  Sailed 1400 yards  Gust of wind came  Ship sank in less than 1 minute

3 of 30

Why is the Vasa relevant? sank the Vasa sinks FMIS projects.  We are still building Vasas.  We can¶t stop bad decisions. 

CAN stop ignoring them. 


you spend $1M and fail, it is bad.
you spend $100M and fail, it impacts the whole organization  


If you spend $1B and fail, it impacts the country. 


you are going to fail - fail cheap.
4 of 30

5 of 30

The Essence of Project Management 

kids on a hike  From time to time, climb a tree  Check for obstacles  Adjust direction  Call everyone together  "LET'S GO!"

6 of 30

Periodic Audits 

success factor for failure prevention  Must be external 
Developers cannot self-assess. 


substantial effort
is worse than useless
Provides illusion of safety 

Weak audit

7 of 30

Audit Costs 

project  Expensive 
5%-10% of

project cost 

Intrusive  Annoying  Politically


8 of 30

Team response to audit 

trust me?" time" 

"Why don't you  "Waste of 


would rather be coding." 


may feel threatened«

the team feels threatened, they probably have reason to be. of professional maturity
9 of 30 


the team welcomes the audit«. 


Audit Benefits 

detection of failure

a $300 million project fails after $20 million, that is a huge savings. 


of system architecture  Second set of eyes  Give team time to take stock  Mid-project course correction

10 of 30

Auditor Independence 

Auditor must be told that there will be no chance for followon work.  

During the audit: 

Limit contact with development team 

Otherwise the audit is suspect No economic attachment to the results No incentive to skew the results No relationship to the development team 

To enforce independence: 

"Stockholm Syndrome" After the audit, auditors can help with the project plan.  

Auditor is the agent of the person who hired him/her (no one else)   

Only reports to the contract owner are objective. No professional standard or certification for IT auditors exists. Contract creates objectivity.
11 of 30

Audits are never 100% objective bring their own biases.  There are IT "religions." 


vs. Java  "Thick database" vs. middle tier logic  Service Oriented Architecture (SOA)  Repository-based development  Business rules  Agile 

professional with a different perspective can still detect a "Vasa."
12 of 30

Justifying Audit Cost 

extensive audit requires approximately 10% of the total project cost.  Industry project failure rate is ~ 60%.  Example: Audit at the halfway point of a $1 million project 

Cost: (50% x 1,000,000) x 10% = $50,000 Benefit: (50%) x 1,000,00) x 60% = $300,000 

Given the

high failure rate, audits are very cheap

insurance.  With other benefits, it is obvious that audits are a good deal.
13 of 30

Finding the right auditor 

internal  Not from the same company as the developers  Not dedicated auditors 

be real developers, DBAs, architects  Must have built systems of similar scope and subject area

14 of 30

The Audit Team 

the subject area with projects of similar 

Experience in


Administrator (DBA)
same platform and similar database 

Experience with



in the same area (Java, .Net, Oracle, etc.) 


or health care experience
15 of 30 

Military, financial system

Structure of the Audit 
Knowledge transfer 

Deep understanding As if auditor were taking over the project Understand the system before evaluating Evaluate the existing infrastructure to support system Every area needs to be assessed. 

Infrastructure audit 

Ability to 

meet current and future user requirements

Auditor must understand the requirements 

Financial review

16 of 30

Detailed Structure of Audit 

A. Knowledge Transfer  

B. Infrastructure Audit 

Allows auditors to understand the entire system architecture as if they were taking over system development. The following areas should be reviewed for the knowledge transfer portion of the audit: 

Examine from technical soundness perspective. Compare to current industry best practices; document any discrepancies. 

System Overview Data Model Walkthrough Review/Identify Transaction Use Cases Review User Interface Code Architecture Review Reporting Requirements/Architecture Review System Architecture Review System Installation/Upgrade Mechanism. Internal Control review User Access review Handling system bugs/enhancements Multi-Lingual capabilities Basic System Requirements Process Flows Custom Framework Performance Standards Training

System and User Documentation Data Model Audit Database Review User Interface (UI) Architecture Review Distributed System/ETL Audit Security Audit Hardware/Software/Networking Review Backup/Recovery Procedures Appropriateness of system upgrade mechanism 

C. Ability to Meet Current/Future Requirements 

Examine the current system requirements, identify use cases, and review for suitability, specifically: 

Compare documented requirements to the existing use cases and how they are handled. Assess user satisfaction with the existing system. Are existing backup/recovery procedures sufficient to meet maximum downtime requirements? Assess system flexibility to meet new requirements. 

D. Financial review 

Review how resources have been spent over the lifetime of the project including budgeted vs. actual expenditures

17 of 30

Knowledge Transfer
to understand." Stephen Covey  Learn enough to take over the process: 

"Seek first

Architecture Data Model Use Cases Report Audit Configuration Management Internal Controls Documentation Training 

System may 

be bad enough that this is not possible.

Do it anyway. Preventing the Vasa from sailing is hard work.
18 of 30

Infrastructure Audit  

Compare what was learned in the knowledge transfer with best practices Each area needs to be assessed: 

Identify weaknesses in

each area: 

Documentation Data model Database design User interface architecture Security Backup & Recovery Configuration Management Internal Controls

Corrective actions Exposure Controls needed 


mistake can sink the Vasa.
System won¶t scale Security hole Inflexible design

19 of 30

Ability to Meet Requirements
use case documentation  Assess user satisfaction 

Requires careful

Sit with users Survey Request queue is not a good measure. If system is poor, users give up. 

Assess each use 


Functional requirements Performance Downtime 



Flexibility Deployment
20 of 30

Financial Review 
Stewardship  Financial History



If requirements change, price can balloon. Does this make sense? Sunk costs are "somewhat" relevant Measure decision quality

Date Budget

$ Spent

MileNotes stones/ Achieved

21 of 30

COTS Project Audits 

very different from custom projects  COTS projects fail just as often.  Review COTS architecture  Be careful about how well COTS satisfies requirements: 
COTS customizations can

be VERY expensive.  Customized COTS cannot be upgraded.

22 of 30

Audit Report 

page Executive Summary Report
we OK? 



page CIO Report
we OK? Why? 


page detailed report 

What we

did  What we found  What needs to be fixed  Next steps
23 of 30

Acting on the Audit Report 

report concludes "Vasa«." 


a second opinion  Let the development team respond 

costs are sunk costs.
of money budgeted for the project is 

The amount


is a way to change direction without admitting failure.

24 of 30

Case Study: Ethiopia FMIS University Project  Small part of major financial reform 
$38 M 


USD over 12 years  $3 M USD over 3 years for IT (very frugal) 

was ending the project and wanted to assess quality of system.  Custom development project  Dulcian was brought in to do the audit.

25 of 30

Audit Scope 


described in this presentation 


knowledge transfer and team back-up 

Support for

any ³what if?´ scenarios  Total system back-up

26 of 30

Audit Recommendations 

current system  Upgrade system internals 
Business rules  Oracle DBMS  Ultra-light Web

approach architecture

27 of 30

Audit Results 

and Harvard accepted recommendations. 
Maintain/evolve current system $1.5M  Redesign architecture $1.5M




nature of the audit (audit + handoff) made existing team very uncomfortable. 
Top three

IT people resigned.

28 of 30

Audits don't prevent failure;

they just catch failures

earlier in the process.  Audits don't replace good design. 

The audit may only help fail more small projects rather than one big project. 

Audits are 

resource intensive, expensive, annoying, politically charged.
But they are cheaper than not doing them at all in the long run. 

Auditors must 

be kept independent.

No follow-on work 


COTS and custom projects need audits.
29 of 30

Result of Audit 
Quite a 

good design

Excellent documentation Very good developer coding Supported current requirements 


architecture portions needed modification.

Database design issues
No Foreign Keys Odd design (inherited from previous team) 


Poor performance for parts of system Scalability questions Would not work on Ethiopian internet due to slow/unreliable connections
30 of 30

Contact Information
Paul Dorsey ±  Dulcian website - 

Design Using UML Object Modeling

Developer Advanced Forms & Reports

Designer Handbook

Latest book: Oracle PL/SQL for Dummies
31 of 30