The 7 +/- 2 Uses For An Estimation Tool

Andrew J. Nolan and Andrew C. Pickard Rolls-Royce plc. Andy.Nolan@Rolls-Royce.com; Andrew.C.Pickard@Rolls-Royce.com

Copyright 2012 by Rolls-Royce. Published and used by INCOSE with permission

Abstract. Rolls-Royce adopted COCOMO II in 2004 in order to improve its estimation capability. Since then COCOMOs importance has grown within the business and has had far reaching effects well beyond its initial remit. COCOMO has been applied to software development, software supplier oversight, improvement identification, improvement validation, risk identification and even business decision making. The tool has become a common language that unites projects, engineering, business, risk and improvements. The tool was also read across into hardware Engineering. Through the adoption of COCOMO, we have seen an 11% reduction (on average) of project costs as well as greater program stability and predictability. This paper is intended to describe our journey and the next steps for Rolls-Royce This paper is written in two parts. The first part describes the lessons learnt during a six sigma Black Belt to improve our estimation capability. Part two of the paper then describes the benefits we got after we had embedded the estimation capability into the business.

Introduction Rolls-Royce
Rolls-Royce provides power systems and services for use on land, at sea and in the air, and operates in four global markets - civil aerospace, defence aerospace, marine and energy. In all the business sectors in which Rolls-Royce operates, there are demands for improved capability and effectiveness of the power systems, more economic and faster product development, better transition to operation (minimum post-delivery changes) and better in-service cost and availability, with commensurate reduction in cost of purchase and/or cost of ownership. The Control Systems department of Rolls-Royces Aerospace business is responsible for the Engine Electronic Controllers (EECs) for a range of small and large gas turbine engines, for the aerospace industry. The EEC contains a significant amount of software that is designed to control the engine, as directed by the pilot, in a way that is safe for the engine, safe for the aircraft, fuel-efficient, component life efficient and environmentally efficient.

Figure 1.

Rolls Royce Trent 1000 engine used to power the Boeing 787

In the beginning
In the old days (which represents pre 2000), estimation of software projects was not particularly complex. A project was assumed to cost about as much as the last project plus a bit extra for new functionality minus any assumed benefit from improvements. In general, this approach worked because each engine project was very similar to the last similar team, similar process, similar everything. In such a stable and predictable world, you do not need estimation tools at least not complex ones. In 2002, we launched a new project, with a new team, a new hardware (electronics) supplier, new process and new tools. We also had a new software manager, a new software architecture and new programming language. Not only did we not factor for the risks with all this newness, but we actually took budget out of the project on the grounds that all these new things were improvements over the things used in the old projects. What happened is probably obvious the project cost significantly more than had been estimated. It was during the investigation that we stumbled across COCOMO II, found the factors compelling and decided to re-perform the original estimate. Instead of taking cost out (as the business had assumed), COCOMO II predicted a cost increase (low Precedence, poor architect/risk resolution, new inexperienced team and so on). We realised that COCOMO II would have successfully predicted the outcome that occurred, and that we could have taken precautions to reduce the risk associated with the changes relative to

previous projects. In the same study, we also showed that COCOMO II could express many of the problems we had seen over the years across many projects. It was this discovery that led us to launch a Six Sigma project to understand and improve our estimation capability. This paper summarises the findings along the way and shares the story of where estimation took the business.

Why estimate?
There are very few of us that would invite a builder into our house without an estimate both cost and time. Our customers expect the same of us. We estimate because most of us want a good understanding of what is going to happen in the future. An estimate suggests a range of possible futures that are most probable. If we dont like the future, then we can either change our expectations or change what we want or do. We have to estimate because in almost all cases, expectations differ from reality (sometimes we expect things to take longer than it should but more often we underestimate and expect things to be quicker and less expensive than they really will be). We estimate to align expectations with reality then change reality to get as close as possible to our expectations. Here are a number of reasons we choose to estimate We estimate to bring stability. Instability creates chaos and chaos increases costs and risk. In the same way that a single weak leg on a table will undermine the whole table, a single bad project can undermine a business. We estimate to increase the probability of success. A business that has good estimation capability can more consistently deliver on its commitments. We estimate to reveal our assumptions. Most estimates fail because the initial assumptions were wrong (or the assumptions were not maintained). The estimation process is a method for eliciting and making explicit the business and project assumptions. We estimate because we want to use the estimate as a negotiation tool with our stakeholders. If well constructed, the estimate provides stakeholders with an understanding of the project and where they might be able to help reduce costs. It is equivalent to a builder giving you a cost breakdown and alternative options. We estimate so we can save money. Once you know the costs and what is driving the costs, it is possible to begin the process of optimisation by trading out scope (low value requirements), bringing in key improvements, mitigating key risks and so on. We estimate to make tradeoffs: Cost is often a key distinguishing factor when making business or technical decisions.

We estimate because subjectivity is not enough. Relying on experts and subjectivity cannot work except in the simplest or most stable environments. Evidence from industry shows that subjectivity is more often than not wrong!

The relationship between systems and estimation

In a system engineering context, there is a relationship between systems engineering success and the budget allocated. We all recognise that if budgets are lower than required, the team is under pressure, key activities are omitted and in all probability the project will fail. A low budget is a risk to project success. A low budget will influence the outcome. Similarly, the budget cannot be allocated in independent of the systems engineering effort. The systems engineer defines the product, its architecture, its functionality. Without these a project cannot make an estimate. Estimation is part of a Systems Engineering Model based tool kit tools like COCOMO model projects behaviour and other tools are used to model products behaviour

Figure 2. The COCOMO Concept

The COCOMO concept (figure 2) can be simplified to the equation below Cost (& Schedule) = C * Size * Environment Where Size (the product): The magnitude (or quantity) of a task. Size (e.g. lines of code, function points. Requirements, models.....), Complexity, Scrap & Rework, Reuse and Risk/Uncertainty Environment: The environment in which you build the product e.g. Processes & tools, People, Management, Organisation Etc C represents a calibration constant for your domain The systems engineer may determine the size of the project but the project engineer will often determine the environment. Both are required for success and tools like COCOMO II can help express the needs of both systems and project into a single unified estimate

Why do estimates fail?

The first step of the improvement project was to baseline our ability to generate estimates. Figure 3 is a frequency distribution chart showing the actual project costs divided by the predicted. It shows two interesting things (1) it has a long tail of projects that overspend relative to estimate and (2) it is not centred. It is off centre by approximately 25%. Estimation accuracy is now a standard monitor. It is interesting to note that today, a 10% overspend by a project is considered a failure and will be investigated. This is a far cry from this chart representing pre-2000 data. An estimate can fail either through under estimation or over estimation. An overestimate can cause the loss of business or the cancellation of another project. However, on the whole, the data from across various industries suggest that in the majority of cases, we suffer Figure 3. Actual/Predicted Project Costs from under-estimation and this paper is written mainly on this assumption as pressures towards underestimation can be considerable. All project managers know that a low cost estimate is more likely to gain senior-level approval and it is a belief that if a low initial budget is set, a lower overall project cost will result. All too often it is concluded by senior management that the estimate itself must be wrong. They reason that the estimators have put in excessive contingencies and have made pessimistic assumptions, and that the contingencies and pessimism are not required this time because the problems that occurred on the last project will not recur. As most of the estimates represent the judgement of the individuals they can usually be "persuaded" by zealous superiors to modify that judgement to something more acceptable. After downward revisions, contingencies are usually the next thing to go; they may be seen as a luxury provision for something that may never happen. Finally, arbitrary cuts may be applied to bring the budget down to an "acceptable" level. We performed root cause analysis to understand the factors behind the variation we were seeing. The analysis was performed by assessing each project overspend and attributing it to a number of pre-determined root causes. The results are shown below. Number 1 represents the top root cause for estimation failure: 1 Scope creep: Often the result of not having a clearly defined scope and/or not tracking and declaring changes to scope.

2 Poor input to estimate: This refers to poor quality requirements or objectives, leaving the estimate open to risk and scope variation. The estimate is built on sand but the estimator does not factor for this or clearly express the uncertainty as a 3-point estimate. 3 Failure to clearly define the initial scope: Items may be missed out or there may be a failure to declare assumptions. This will make the project vulnerable to scope creep or conflict later in the project as all parties argue over what is in scope and what is out of scope. 4 Unrealistic expectations and assumptions: Optimistic assumptions that it will go right this time. Claiming benefit from improvements before they have been demonstrated. Overruling the estimate. 5 Failure to declare, track and reduce risk and uncertainties. The project may then be vulnerable to unnecessary risk. 6 Lack of internal peer review: Failing to benefit from peers experience. A review is a simple and cost effective way to improve estimate Quality 7 Lack of estimation experience: Estimates generated by a person who may not have experience in estimating or in the subject being estimated. 8 Failure to consider environmental factors: Undeclared and untracked elements of the environment can change having unexpected and dramatic effects for examples, loss of key individuals, change of supplier, a more complex organisation, etc. 9 Failure in the estimation tool/process: The tool or process may be unable to deliver the desired accuracy. It is essential to measure and declare the estimate tolerance of any tools used. 10 Lack of estimation process/technique: Failing to apply a systematic estimation tool, method or processes. Without this, it is hard to defend the estimate. We expected the business to show more interest in estimation tools rather than the relatively uninteresting aspects of estimate management. Figure 4 groups the root causes for failure into three 3 factors: 1 Tool failure: The tool can be shown to generate inaccurate estimates Figure 4. Contributions to Estimation Failure

2 Process Failure: The estimate failed during generation, review or management of the estimate through For example, poor estimate management: Failing to maintain the estimate through life i.e. as assumptions changed the project failed to update the estimate 3 Behaviour: the tool and process existed but human behaviour overrode it. For example having insufficient time to estimate: Assuming an estimate can be generated in only a few hours or even minutes! Rushing the estimate, taking short cuts, leaps of faith, and omitting key individuals

The Estimation Life-cycle

As a result of the root cause analysis, we embedded the following governance process. Figure 5 shows the estimation lifecycle. The first step of the improvement project to look at estimation capability was to understand how successful this loop was. Despite good and bad practices across the business, the most notable issue was the fact Figure 5. The Estimation Lifecycle that the estimate was seen as a deliverable rather than a live document. Few projects would use the estimate to identify the critical project assumptions, and then track these assumptions during project execution. 50% of all estimates failed not during step 1 above but during step 2. As critical assumptions changed, projects were not looping back to step 1 to re-baseline their estimates and re-secure commitments. It may be tempting for readers to agree with scope creep as being the number 1 root cause we have all experienced it. But actually this is an indicator that the estimate was not managed through life. The number 1 root cause for estimation failure was failing to manage the estimate. This led to the introduction of two governance processes. The formal review of estimate leaving step 1 and the on-going review of estimate during step 2. In both cases the reviews are conducted by independent peers (or managers) which relevant domain experience. In both cases checklists are provided to guide the review. It is interesting to show that even with the most accurate estimation tool in the world (which is where most tool vendors focus), we would resolve 19% of the variation. Even with world class tool and processes, there are still risks from human behaviour. The implementation needed to focus on all three elements:

1 Tools: We acquired COCOMO II as the cost estimation tool. Several versions of the tool have been developed covering software development, supplier software development, hardware engineering and improvements validation & elicitation. 2 Process: We developed an estimation process and governance body to oversee the projects roles in the business were created to buy-off, review and support the estimate. It became necessary for all estimates to be managed regularly and projects had to report changes in critical assumptions. 3 Behaviour: We developed training for the business and trained over 300 people including stakeholders and managers. Culture takes time to change and was facilitated by senior commitment to the new estimation approached.

Figure 6. Estimation Accuracy

The results
The estimation process and tools have now been in place for 6 years. The accuracy of the estimates to date has been assessed and is shown in figure 6. Although there is variation in the individual estimate the overall risk to the business is low.

Estimation tools
Having established a stable and capable process and estimation tool set, we were able to build on this foundation and develop a range a capabilities to enable the business to manage more quantitatively. This section describes the benefits we achieved from the use of COCOMO.

The adoption of COCOMO II

COCOMO is an algorithmic software cost estimation model developed by Boehm et al (1). The model uses a basic regression formula, with parameters that are derived from historical project data. COCOMO was first published in 1981 as a model for estimating effort, cost, and schedule for software projects. COCOMO II (2) is the latest extension to the original COCOMO and was developed by the combined efforts of the University of Southern California Center for Systems and Software Engineering, the Institute for Software Research at the University of California - Irvine, and the

COCOMO II Project Affiliate Organizations. The revised cost estimation model reflects the changes in professional software development practice that have come about since the 1970s. Within Rolls-Royce, the process of evaluation and tool development took around 1 month of effort. The objective was to find a simple, accurate and believable estimation tool that would allow managers to express and defend the critical project assumptions in a way that the business could understand. Believable and dependable estimates were key requirements as well as having a tool that anyone could use. It was necessary to find accurate data for historic projects and then to estimate their cost as if they were future projects. A blind estimate was generated and then validated against the actual project results. There was a common cause discrepancy which was attributed to tool calibration. In other cases, there were special cause exceptions which had to be investigated. At the end of the analysis, we had a calibrated estimation tool as well as an understanding of the COCOMO factors and how to drive them. This knowledge became part of the user guide and training program. The COCOMO II model is a very simple equation relating factors to final cost and schedule. We added front end tools to help derive estimates for size (Lines of Code) as well as back end tools to help transform the results into resource profiles, phases, plans and even error predictions. We built many versions of the tool to meet the needs of different domains including hardware development. In each case, we identified the questions we needed to ask about the project/business, selected the factors from COCOMO II that would address these questions, and then built this into a tool. In those cases where COCOMO II could not provide the factor to address a question, we would develop our own factors, gather data from the business and perform a regression analysis to understand the sensitivity and range for the new factors. Examples of new factors included requirements volatility (from our customers) and Scrap & Rework generated from our evolutionary approach to engine, hardware and electronic development.

What an Estimation Tool teaches you

An important breakthrough occurred when we realized that estimation tools, like COCOMO II, were not only useful for estimating costs, but were actually teaching us what was important about a project. Estimation tools are actually models of a project and like all models they are there to help you make good decisions. An estimation tool defines a formal and objective representation of a project or business and is by definition a simplification of reality. Estimation tools need only be as precise and accurate as required to make meaningful and accurate decisions. They are there to tell you something that you would not (or could not) know without them. They are central to good project management and control, estimation, improvement and risk management. The estimation tools are also there to reduce the subjectivity from the decision making process. It is tempting for a manager, fuelled by ego and heroics, eager to prove themselves, to exaggerate the

truth or guess at key decisions. You need a good estimation tool to help with reasoning, collaboration and negotiation in order to persuade the business to invest in the right things. We have found that a well-constructed estimate makes persuasion a whole lot easier then relying on good intentions and opinions.

The business benefits

Through the development and deployment of estimation tools across the business, we have seen an improvement in stability and productivity on average around 11% cost saving per project, as shown in figure 7. This is primarily because estimation tools, like COCOMO II, are informing you of what is important. This information has shaped what we measure, how we identify and validate improvements, how we identify and mitigate risks and how we manage and estimate projects. They help us optimize and refine the business around objective reasoning rather the subjective guesswork i.e., we make better decisions. For example, if a project is taking on novel features, or there are concerns over the aircraft maturity, we would expect a high Figure 7. COCOMO Benefits level of requirements volatility and scrap & rework. The estimation tool would quantify the impact (increased cost and a longer program) and this would then be used to drive for changes in the development approach, risk mitigations, negotiations with the customer and so on. If this was a critical factor for success, then this attribute of the project would be carefully monitored and reported. Similarly, if in order to achieve a low cost project, you assume a high performance team, then this aspect of the project will need to be carefully monitored and reported. The output from an estimate is a measurement plan of critical factors that need to be monitored, controlled and where possible, improved. In terms of return on investment, the introduction of the estimation capability represents a return on investment of over 1000:1.

The 7 +/- 2 uses for an estimation tool

Summarising all the messages from this paper, estimation tools, like COCOMO, can sit at the heart of the business and can be used in the following ways 1 Estimation! Obviously the tool can be used to estimate but they have also been used to estimate the costs of our supply chain. Being a smart customer has given us a strong position to challenge supplier costs and bring about overall cost savings.

Improvement identification: Once you complete the COCOMO estimate, the project can then ask questions about how they could improve the settings. For example, if the organisation is scattered across several continents, which would add to the cost of the project, how could the team be centralised? Figure 8 shows the full range of sensitivity Figure 8. Improvement Opportunities for each COCOMO II factors. The top of the green bar represents the best setting, the bottom of the red represents the worst and where the two meet represents an average industry project. The triangle shows an example project (or business) and how it lies on the sensitivity range. This allows us to identify where we can make the greatest improvements. Improvement validation. If COCOMO II represents a model of the project and can be trusted to estimate costs, then it is implicit that it represents a complete model of a project. Any improvement project that does not influence a COCOMO factor is unlikely to have an impact on the project. At Rolls-Royce it is now necessary to provide traceability between each improvement and the COCOMO II factors. Risk Identification: In addition to eliciting and validating improvements, the tool can also be used to elicit and validate risks. For example, if the project has an above average team, what is the risk if the team is moved onto another high priority project and the project needs to recruit a new team? Improved Monitoring: When we are asked to help a project define its measurement program, we tell them to start with their critical estimate assumptions. COCOMO II contains some 21 factors of which, for an average project, 10-15 will be sensitive and uncertain. When a project generates an estimate, it is declaring its assumptions about each of these factors. As such, they should be monitored. Diagnostics: In the same way that COCOMO II can be used to elicit improvements, we have used the tools to quickly find the root cause for project problems. By generating an estimate for a struggling project it is possible to identify its problems and greatest opportunities for improvement Benchmark. The tool contains a great deal of industry data and it is therefore possible to benchmark your business against an industry average project. Also, if you business has several domains of projects, it may seem impossible to compare cost e.g. a safety critical

project cannot be compared with a non-safety critical project. However, if the project data is corrected for domain differences, it is then possible to compare data across the business. This opens a business up to sharing a lot more information. 8 The 7 + 1 use of the tool is in decision making and trade analysis. It is not necessary to generate a complete estimate in order to make use of COCOMO II. It is possible to use the factors in isolation. For example, one factor describes the benefit of a co-located team. The business can generate a very quick business case for renting a large building by using this factor in isolation. More complex business models have been generated to help business leaders plan IT, tool acquisition, process standardisation, location of teams and suppliers and so on. The 7 + 2 use of the tool is its presence alone is enough to change culture. In demonstrating the tool around the business, there is implicit training and orientation. Few managers can now adopt a team with immaturity or lack of experience, or take on low Technology Readiness projects without at least remembering COCOMO and its warnings.

Summary
COCOMO is a unifying language that sits at the heart of the business. It can link together engineering, projects, risk management and improvement activities into a unified tool. It is also the bridge between different areas of the business that work in different domains. It can also be a bridge into industry to make direct comparisons with peer projects from other companies. COCOMO II was based on several hundred projects. For an organisation like Rolls-Royce, to acquire this level of knowledge would take several hundred years. The adoption of a tool can accelerate a businesss collection of data points and other tools in industry claim to have thousands to many 10s of thousands of data points. Each factor represents a quantified and validated view on the world and can help a business improve. Our suggestion is that every business should buy at least one of each estimation tool so as to gain a better and more refined understanding of costs. The introduction of estimation tools and processes has brought stability (project predictability in terms of time, cost and resource requirements) and given corresponding gains in productivity. Also, the spin off benefits brought improvements in the way we make decisions, identify and validate improvements and even in how we communicate within the business.

References
1. Boehm, B., Abts, C., Brown, W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D. and Steece, B., "Software Cost Estimation with COCOMO II", Prentice-Hall, ISBN 0-13-026692-2

Biography
Andrew Nolan joined Rolls-Royce in 1989 after completing a degree at Sheffield University. He is the Chief of Software Improvement for Rolls-Royce based in the UK. He is a Fellow of the British Computer Society and a chartered Engineer in software engineering. Andrew has spent over a decade managing large scale software projects as well as a decade improving the way Rolls-Royce manages projects. Andrew Pickard joined Rolls-Royce in 1977 after completing a Ph.D. at Cambridge University in Fatigue and Fracture of Metals and Alloys. He is a Rolls-Royce Associate Fellow in System Engineering, a Fellow of the Institute of Materials, Minerals and Mining, a Chartered Engineer and a member of the American Institute of Aeronautics and Astronautics, of SAE International and of INCOSE. He is Vice-Chair of the SAE Aerospace Council and represents Rolls-Royce on the INCOSE Corporate Advisory Board.