Using Periodic Audits to Prevent Catastrophic Project Failure Dr. Paul Dorsey Dulcian, Inc.

May 22, 2008

1 of 30

The Vasa 
Early 
3

17th century (1628)  ³The greatest ship of all time´
years to build  2 gun decks with 64 cannons 
King

Gustavus Adolphus of Sweden set the measurements. 
1000 trees

were required  Triple laminated oak walls (18in/46cm thick)  Mast = 190 ft/57m  Length = 201 ft/61m 
Cost

= 5% of Sweden¶s GNP
2 of 30

Maiden Voyage 
Set

sail on a beautiful summer day 

August 10,

1628  Passed the royal palace of Stockholm  Fired proud salutes from cannons  Sailed 1400 yards  Gust of wind came  Ship sank in less than 1 minute

3 of 30

Why is the Vasa relevant? sank the Vasa sinks FMIS projects.  We are still building Vasas.  We can¶t stop bad decisions. 
What 
We

CAN stop ignoring them. 

If

you spend $1M and fail, it is bad.
you spend $100M and fail, it impacts the whole organization  

If

If you spend $1B and fail, it impacts the country. 

If

you are going to fail - fail cheap.
4 of 30

5 of 30

The Essence of Project Management 
Leading

kids on a hike  From time to time, climb a tree  Check for obstacles  Adjust direction  Call everyone together  "LET'S GO!"

6 of 30

Periodic Audits 
Critical

success factor for failure prevention  Must be external 
Developers cannot self-assess. 

Big, 

substantial effort
is worse than useless
Provides illusion of safety 

Weak audit

7 of 30

Audit Costs 
Delay

project  Expensive 
5%-10% of

project cost 

Intrusive  Annoying  Politically

costly

8 of 30

Team response to audit 
Project

Manager
trust me?" time" 

"Why don't you  "Waste of 

Developers 
"We

would rather be coding." 

Team 
If

may feel threatened«

the team feels threatened, they probably have reason to be. of professional maturity
9 of 30 

If

the team welcomes the audit«. 

Sign

Audit Benefits 
Early 
If

detection of failure

a $300 million project fails after $20 million, that is a huge savings. 

Validation

of system architecture  Second set of eyes  Give team time to take stock  Mid-project course correction

10 of 30

Auditor Independence 

Auditor must be told that there will be no chance for followon work.  

During the audit: 

Limit contact with development team 


Otherwise the audit is suspect No economic attachment to the results No incentive to skew the results No relationship to the development team 

To enforce independence: 

"Stockholm Syndrome" After the audit, auditors can help with the project plan.  



Auditor is the agent of the person who hired him/her (no one else)   

Only reports to the contract owner are objective. No professional standard or certification for IT auditors exists. Contract creates objectivity.
11 of 30

Audits are never 100% objective bring their own biases.  There are IT "religions." 
.Net 

Auditors

vs. Java  "Thick database" vs. middle tier logic  Service Oriented Architecture (SOA)  Repository-based development  Business rules  Agile 
A

professional with a different perspective can still detect a "Vasa."
12 of 30

Justifying Audit Cost 
An

extensive audit requires approximately 10% of the total project cost.  Industry project failure rate is ~ 60%.  Example: Audit at the halfway point of a $1 million project 


Cost: (50% x 1,000,000) x 10% = $50,000 Benefit: (50%) x 1,000,00) x 60% = $300,000 

Given the

high failure rate, audits are very cheap

insurance.  With other benefits, it is obvious that audits are a good deal.
13 of 30

Finding the right auditor 
Not

internal  Not from the same company as the developers  Not dedicated auditors 
Must

be real developers, DBAs, architects  Must have built systems of similar scope and subject area

14 of 30

The Audit Team 
Project

Manager
the subject area with projects of similar 

Experience in

scope 
Database

Administrator (DBA)
same platform and similar database 

Experience with

size 
Application 
Skill

Architect

in the same area (Java, .Net, Oracle, etc.) 

Security

Specialist
or health care experience
15 of 30 

Military, financial system

Structure of the Audit 
Knowledge transfer 
 

Deep understanding As if auditor were taking over the project Understand the system before evaluating Evaluate the existing infrastructure to support system Every area needs to be assessed. 

Infrastructure audit 
 

Ability to 

meet current and future user requirements

Auditor must understand the requirements 

Financial review

16 of 30

Detailed Structure of Audit 

A. Knowledge Transfer  

B. Infrastructure Audit 
 

Allows auditors to understand the entire system architecture as if they were taking over system development. The following areas should be reviewed for the knowledge transfer portion of the audit: 
               

Examine from technical soundness perspective. Compare to current industry best practices; document any discrepancies. 
       

System Overview Data Model Walkthrough Review/Identify Transaction Use Cases Review User Interface Code Architecture Review Reporting Requirements/Architecture Review System Architecture Review System Installation/Upgrade Mechanism. Internal Control review User Access review Handling system bugs/enhancements Multi-Lingual capabilities Basic System Requirements Process Flows Custom Framework Performance Standards Training

System and User Documentation Data Model Audit Database Review User Interface (UI) Architecture Review Distributed System/ETL Audit Security Audit Hardware/Software/Networking Review Backup/Recovery Procedures Appropriateness of system upgrade mechanism 

C. Ability to Meet Current/Future Requirements 

Examine the current system requirements, identify use cases, and review for suitability, specifically: 
  

Compare documented requirements to the existing use cases and how they are handled. Assess user satisfaction with the existing system. Are existing backup/recovery procedures sufficient to meet maximum downtime requirements? Assess system flexibility to meet new requirements. 

D. Financial review 

Review how resources have been spent over the lifetime of the project including budgeted vs. actual expenditures

17 of 30

Knowledge Transfer
to understand." Stephen Covey  Learn enough to take over the process: 
       

"Seek first

Architecture Data Model Use Cases Report Audit Configuration Management Internal Controls Documentation Training 

System may 


be bad enough that this is not possible.

Do it anyway. Preventing the Vasa from sailing is hard work.
18 of 30

Infrastructure Audit  

Compare what was learned in the knowledge transfer with best practices Each area needs to be assessed: 
       

Identify weaknesses in

each area: 
 

Documentation Data model Database design User interface architecture Security Backup & Recovery Configuration Management Internal Controls

Corrective actions Exposure Controls needed 

One 
 

mistake can sink the Vasa.
System won¶t scale Security hole Inflexible design

19 of 30

Ability to Meet Requirements
use case documentation  Assess user satisfaction 
  

Requires careful

Sit with users Survey Request queue is not a good measure. If system is poor, users give up. 

Assess each use 
 

case

Functional requirements Performance Downtime 

Future 


requirements

Flexibility Deployment
20 of 30

Financial Review 
Stewardship  Financial History

Audit  

 

If requirements change, price can balloon. Does this make sense? Sunk costs are "somewhat" relevant Measure decision quality

Date Budget

$ Spent

MileNotes stones/ Achieved

21 of 30

COTS Project Audits 
Not

very different from custom projects  COTS projects fail just as often.  Review COTS architecture  Be careful about how well COTS satisfies requirements: 
COTS customizations can

be VERY expensive.  Customized COTS cannot be upgraded.

22 of 30

Audit Report 
2-5

page Executive Summary Report
we OK? 

Are 

10-15 
Are

page CIO Report
we OK? Why? 

100

page detailed report 

What we

did  What we found  What needs to be fixed  Next steps
23 of 30

Acting on the Audit Report 
If

report concludes "Vasa«." 

Get

a second opinion  Let the development team respond 
Sunk

costs are sunk costs.
of money budgeted for the project is 

The amount

irrelevant. 
"Upgrade"

is a way to change direction without admitting failure.

24 of 30

Case Study: Ethiopia FMIS University Project  Small part of major financial reform 
$38 M 

Harvard

USD over 12 years  $3 M USD over 3 years for IT (very frugal) 
Harvard

was ending the project and wanted to assess quality of system.  Custom development project  Dulcian was brought in to do the audit.

25 of 30

Audit Scope 
Standard 
As

audit

described in this presentation 

Total

knowledge transfer and team back-up 

Support for

any ³what if?´ scenarios  Total system back-up

26 of 30

Audit Recommendations 
Maintain

current system  Upgrade system internals 
Business rules  Oracle DBMS  Ultra-light Web

approach architecture

27 of 30

Audit Results 
Government

and Harvard accepted recommendations. 
Maintain/evolve current system $1.5M  Redesign architecture $1.5M

USD

USD 

Dual

nature of the audit (audit + handoff) made existing team very uncomfortable. 
Top three

IT people resigned.

28 of 30

Conclusions 
Audits don't prevent failure;

they just catch failures

earlier in the process.  Audits don't replace good design. 

The audit may only help fail more small projects rather than one big project. 

Audits are 

resource intensive, expensive, annoying, politically charged.
But they are cheaper than not doing them at all in the long run. 

Auditors must 

be kept independent.

No follow-on work 

Both

COTS and custom projects need audits.
29 of 30

Result of Audit 
Quite a 
 

good design

Excellent documentation Very good developer coding Supported current requirements 

Some  



architecture portions needed modification.

Database design issues
No Foreign Keys Odd design (inherited from previous team) 

 

Poor performance for parts of system Scalability questions Would not work on Ethiopian internet due to slow/unreliable connections
30 of 30

Contact Information
Paul Dorsey ± paul_dorsey@dulcian.com  Dulcian website - www.dulcian.com 
Dr.

Design Using UML Object Modeling

Developer Advanced Forms & Reports

Designer Handbook

Latest book: Oracle PL/SQL for Dummies
31 of 30