This action might not be possible to undo. Are you sure you want to continue?
Andrew J. Nolan and Andrew C. Pickard Rolls-Royce plc. Andy.Nolan@Rolls-Royce.com; Andrew.C.Pickard@Rolls-Royce.com
Copyright © 2012 by Rolls-Royce. Published and used by INCOSE with permission
Abstract. Rolls-Royce adopted COCOMO II in 2004 in order to improve its estimation capability. Since then COCOMO‟s importance has grown within the business and has had far reaching effects well beyond its initial remit. COCOMO has been applied to software development, software supplier oversight, improvement identification, improvement validation, risk identification and even business decision making. The tool has become a common language that unites projects, engineering, business, risk and improvements. The tool was also read across into hardware Engineering. Through the adoption of COCOMO, we have seen an 11% reduction (on average) of project costs as well as greater program stability and predictability. This paper is intended to describe our journey and the next steps for Rolls-Royce This paper is written in two parts. The first part describes the lessons learnt during a six sigma Black Belt to improve our estimation capability. Part two of the paper then describes the benefits we got after we had embedded the estimation capability into the business.
Rolls-Royce provides power systems and services for use on land, at sea and in the air, and operates in four global markets - civil aerospace, defence aerospace, marine and energy. In all the business sectors in which Rolls-Royce operates, there are demands for improved capability and effectiveness of the power systems, more economic and faster product development, better transition to operation (minimum post-delivery changes) and better in-service cost and availability, with commensurate reduction in cost of purchase and/or cost of ownership. The Control Systems department of Rolls-Royce‟s Aerospace business is responsible for the Engine Electronic Controllers (EECs) for a range of small and large gas turbine engines, for the aerospace industry. The EEC contains a significant amount of software that is designed to „control‟ the engine, as directed by the pilot, in a way that is safe for the engine, safe for the aircraft, fuel-efficient, component life efficient and environmentally efficient.
you do not need estimation tools – at least not complex ones. Rolls Royce Trent 1000 engine used to power the Boeing 787 In the beginning In the “old days” (which represents pre 2000). What happened is probably obvious – the project cost significantly more than had been estimated. found the factors compelling and decided to re-perform the original estimate. In 2002. In general. poor architect/risk resolution. A project was assumed to cost about as much as the last project – plus a bit extra for new functionality minus any assumed benefit from improvements. new process and new tools. We also had a new software manager. We realised that COCOMO II would have successfully predicted the outcome that occurred. we launched a new project. this approach worked because each engine project was very similar to the last – similar team. estimation of software projects was not particularly complex. new inexperienced team and so on). a new software architecture and new programming language. similar everything. and that we could have taken precautions to reduce the risk associated with the changes relative to .Figure 1. with a new team. COCOMO II predicted a cost increase (low Precedence. Not only did we not factor for the risks with all this newness. similar process. It was during the investigation that we stumbled across COCOMO II. In such a stable and predictable world. Instead of taking cost out (as the business had assumed). but we actually took budget out of the project on the grounds that all these “new” things were improvements over the things used in the “old” projects. a new hardware (electronics) supplier.
Most estimates fail because the initial assumptions were wrong (or the assumptions were not maintained). it is possible to begin the process of optimisation by trading out scope (low value requirements). Here are a number of reasons we choose to estimate We estimate to bring stability. It was this discovery that led us to launch a Six Sigma project to understand and improve our estimation capability. This paper summarises the findings along the way and shares the story of where estimation took the business. A business that has good estimation capability can more consistently deliver on its commitments. We estimate to reveal our assumptions. a single bad project can undermine a business. The estimation process is a method for eliciting and making explicit the business and project assumptions. then we can either change our expectations or change what we want or do. If we don‟t like the future. mitigating key risks and so on. Instability creates chaos and chaos increases costs and risk. It is equivalent to a builder giving you a cost breakdown and alternative options. expectations differ from reality (sometimes we expect things to take longer than it should but more often we underestimate and expect things to be quicker and less expensive than they really will be). We estimate to make tradeoffs: Cost is often a key distinguishing factor when making business or technical decisions. Why estimate? There are very few of us that would invite a builder into our house without an estimate – both cost and time. We estimate so we can save money. An estimate suggests a range of possible futures that are most probable. Our customers expect the same of us. In the same way that a single weak leg on a table will undermine the whole table. bringing in key improvements. We estimate to align expectations with reality – then change reality to get as close as possible to our expectations. we also showed that COCOMO II could express many of the problems we had seen over the years across many projects. If well constructed. We estimate because we want to use the estimate as a negotiation tool with our stakeholders.previous projects. We estimate because most of us want a good understanding of what is going to happen in the future. the estimate provides stakeholders with an understanding of the project and where they might be able to help reduce costs. We estimate to increase the probability of success. Once you know the costs and what is driving the costs. We have to estimate because in almost all cases. . In the same study.
The COCOMO Concept The COCOMO concept (figure 2) can be simplified to the equation below Cost (& Schedule) = C * Size * Environment Where Size (the product): The magnitude (or quantity) of a task. models.g. Complexity. Relying on experts and subjectivity cannot work except in the simplest or most stable environments. Size (e. Scrap & Rework.g. Evidence from industry shows that subjectivity is more often than not – wrong! The relationship between systems and estimation In a system engineering context. its functionality.. A low budget will influence the outcome. A low budget is a risk to project success. We all recognise that if budgets are lower than required.We estimate because subjectivity is not enough. function points. the team is under pressure. Processes & tools. Organisation Etc “C” represents a calibration constant for your domain The systems engineer may determine the size of the project but the project engineer will often determine the environment. Management. its architecture. Reuse and Risk/Uncertainty Environment: The environment in which you build the product e. Similarly. Requirements. key activities are omitted and in all probability the project will fail. People. The systems engineer defines the product. Both are required for success and tools like COCOMO II can help express the needs of both systems and project into a single unified estimate .. lines of code.. Estimation is part of a Systems Engineering Model based tool kit – tools like COCOMO model projects behaviour and other tools are used to model products behaviour Figure 2. there is a relationship between systems engineering success and the budget allocated. Without these a project cannot make an estimate. the budget cannot be allocated in independent of the systems engineering effort..).
The analysis was performed by assessing each project overspend and attributing it to a number of pre-determined root causes. Number 1 represents the top root cause for estimation failure: 1 Scope creep: Often the result of not having a clearly defined scope and/or not tracking and declaring changes to scope. on the whole. the data from across various industries suggest that in the majority of cases. Actual/Predicted Project Costs from under-estimation and this paper is written mainly on this assumption as pressures towards underestimation can be considerable. they may be seen as a luxury provision for something that may never happen. Estimation accuracy is now a standard monitor. a lower overall project cost will result. We performed root cause analysis to understand the factors behind the variation we were seeing. we suffer Figure 3. As most of the estimates represent the judgement of the individuals they can usually be "persuaded" by zealous superiors to modify that judgement to something more acceptable. An estimate can fail either through under estimation or over estimation. After downward revisions. a 10% overspend by a project is considered a failure and will be investigated. However. The results are shown below. It is interesting to note that today. They reason that the estimators have put in excessive contingencies and have made pessimistic assumptions. Figure 3 is a frequency distribution chart showing the actual project costs divided by the predicted. All project managers know that a low cost estimate is more likely to gain senior-level approval and it is a belief that if a low initial budget is set.Why do estimates fail? The first step of the improvement project was to baseline our ability to generate estimates. All too often it is concluded by senior management that the estimate itself must be wrong. Finally. It is off centre by approximately 25%. It shows two interesting things (1) it has a long tail of projects that overspend relative to estimate and (2) it is not centred. An overestimate can cause the loss of business or the cancellation of another project. . arbitrary cuts may be applied to bring the budget down to an "acceptable" level. and that the contingencies and pessimism are not required this time because the problems that occurred on the last project will not recur. This is a far cry from this chart representing pre-2000 data. contingencies are usually the next thing to go.
Contributions to Estimation Failure . a more complex organisation. Without this. etc. 6 Lack of internal peer review: Failing to benefit from peers‟ experience. A review is a simple and cost effective way to improve estimate “Quality” 7 Lack of estimation experience: Estimates generated by a person who may not have experience in estimating or in the subject being estimated. This will make the project vulnerable to scope creep or conflict later in the project as all parties argue over what is in scope and what is out of scope. it is hard to defend the estimate. change of supplier. Claiming benefit from improvements before they have been demonstrated. Figure 4 groups the root causes for failure into three 3 factors: 1 Tool failure: The tool can be shown to generate inaccurate estimates Figure 4.2 Poor input to estimate: This refers to poor quality requirements or objectives. method or processes. It is essential to measure and declare the estimate tolerance of any tools used. 4 Unrealistic expectations and assumptions: Optimistic assumptions that it will go right this time. leaving the estimate open to risk and scope variation. loss of key individuals. track and reduce risk and uncertainties. 9 Failure in the estimation tool/process: The tool or process may be unable to deliver the desired accuracy. The project may then be vulnerable to unnecessary risk. 8 Failure to consider environmental factors: Undeclared and untracked elements of the environment can change – having unexpected and “dramatic” effects for examples. The estimate is “built on sand” but the estimator does not factor for this or clearly express the uncertainty as a 3-point estimate. 5 Failure to declare. We expected the business to show more interest in estimation tools rather than the relatively uninteresting aspects of estimate management. 3 Failure to clearly define the initial scope: Items may be missed out or there may be a failure to declare assumptions. Overruling the estimate. 10 Lack of estimation process/technique: Failing to apply a systematic estimation tool.
It may be tempting for readers to agree with scope creep as being the number 1 root cause – we have all experienced it. review or management of the estimate through For example. The first step of the improvement project to look at estimation capability was to understand how successful this loop was. poor estimate management: Failing to maintain the estimate through life i. But actually this is an indicator that the estimate was not managed through life. The implementation needed to focus on all three elements: . In both cases checklists are provided to guide the review. Despite good and bad practices across the business. The Estimation Lifecycle that the estimate was seen as a deliverable rather than a live document. The formal review of estimate leaving step 1 and the on-going review of estimate during step 2. This led to the introduction of two governance processes. we embedded the following governance process. the most notable issue was the fact Figure 5. For example having insufficient time to estimate: Assuming an estimate can be generated in only a few hours – or even minutes! Rushing the estimate. we would resolve 19% of the variation. there are still risks from human behaviour. taking short cuts. It is interesting to show that even with the most accurate estimation tool in the world (which is where most tool vendors focus). and omitting key individuals The Estimation Life-cycle As a result of the root cause analysis. Few projects would use the estimate to identify the critical project assumptions. as assumptions changed the project failed to update the estimate 3 Behaviour: the tool and process existed but human behaviour overrode it. Even with world class tool and processes. and then track these assumptions during project execution. Figure 5 shows the estimation lifecycle. The number 1 root cause for estimation failure was failing to manage the estimate. As critical assumptions changed. 50% of all estimates failed not during step 1 above but during step 2. leaps of faith.e. projects were not looping back to step 1 to re-baseline their estimates and re-secure commitments. In both cases the reviews are conducted by independent peers (or managers) which relevant domain experience.2 Process Failure: The estimate failed during generation.
COCOMO was first published in 1981 as a model for estimating effort.1 Tools: We acquired COCOMO II as the cost estimation tool. It became necessary for all estimates to be managed regularly and projects had to report changes in critical assumptions. Culture takes time to change and was facilitated by senior commitment to the new estimation approached. review and support the estimate. The adoption of COCOMO II COCOMO is an algorithmic software cost estimation model developed by Boehm et al (1). 2 Process: We developed an estimation process and governance body to oversee the projects roles in the business were created to buy-off. The accuracy of the estimates to date has been assessed and is shown in figure 6. hardware engineering and improvements validation & elicitation. and schedule for software projects. COCOMO II (2) is the latest extension to the original COCOMO and was developed by the combined efforts of the University of Southern California Center for Systems and Software Engineering. the Institute for Software Research at the University of California . with parameters that are derived from historical project data. Several versions of the tool have been developed covering software development. and the . Estimation tools Having established a stable and capable process and estimation tool set. 3 Behaviour: We developed training for the business and trained over 300 people including stakeholders and managers. Estimation Accuracy The results The estimation process and tools have now been in place for 6 years. The model uses a basic regression formula. Although there is variation in the individual estimate the overall risk to the business is low. This section describes the benefits we achieved from the use of COCOMO. we were able to build on this foundation and develop a range a capabilities to enable the business to manage more quantitatively.Irvine. Figure 6. cost. supplier software development.
selected the factors from COCOMO II that would address these questions. They are there to tell you something that you would not (or could not) know without them. A “blind” estimate was generated and then validated against the actual project results. In those cases where COCOMO II could not provide the factor to address a question. The objective was to find a simple. we identified the “questions” we needed to ask about the project/business. like COCOMO II. Estimation tools need only be as precise and accurate as required to make meaningful and accurate decisions. but were actually teaching us what was important about a project. What an Estimation Tool teaches you An important breakthrough occurred when we realized that estimation tools. They are central to good project management and control. It is tempting for a manager. plans and even error predictions. At the end of the analysis. We added “front end” tools to help derive estimates for size (Lines of Code) as well as “back end” tools to help transform the results into resource profiles. hardware and electronic development. the process of evaluation and tool development took around 1 month of effort. The COCOMO II model is a very simple equation relating factors to final cost and schedule. In each case. were not only useful for estimating costs. Examples of new factors included requirements volatility (from our customers) and Scrap & Rework generated from our evolutionary approach to engine. An estimation tool defines a formal and objective representation of a project or business and is by definition a simplification of reality. gather data from the business and perform a regression analysis to understand the sensitivity and range for the new factors. Believable and dependable estimates were key requirements as well as having a tool that anyone could use. Within Rolls-Royce. improvement and risk management. there were special cause exceptions which had to be investigated. The estimation tools are also there to reduce the subjectivity from the decision making process. eager to prove themselves. and then built this into a tool. The revised cost estimation model reflects the changes in professional software development practice that have come about since the 1970s.COCOMO II Project Affiliate Organizations. fuelled by ego and heroics. Estimation tools are actually models of a project and like all models they are there to help you make good decisions. There was a “common cause” discrepancy which was attributed to tool calibration. we would develop our own factors. phases. This knowledge became part of the user guide and training program. accurate and believable estimation tool that would allow managers to express and defend the critical project assumptions in a way that the business could understand. In other cases. It was necessary to find accurate data for historic projects and then to estimate their cost as if they were future projects. estimation. we had a calibrated estimation tool as well as an understanding of the COCOMO factors and how to drive them. We built many versions of the tool to meet the needs of different domains – including hardware development. to exaggerate the .
how we identify and validate improvements. collaboration and negotiation in order to persuade the business to invest in the right things. The estimation tool would quantify the impact (increased cost and a longer program) and this would then be used to drive for changes in the development approach. COCOMO Benefits level of requirements volatility and scrap & rework. we make better decisions. We have found that a well-constructed estimate makes persuasion a whole lot easier then relying on good intentions and opinions.e.truth or guess at key decisions. Similarly. like COCOMO. how we identify and mitigate risks and how we manage and estimate projects. This is primarily because estimation tools. estimation tools. controlled and where possible. if in order to achieve a low cost project.. negotiations with the customer and so on. like COCOMO II. In terms of return on investment. The 7 +/. . are informing you of what is important. The business benefits Through the development and deployment of estimation tools across the business. improved. For example. Being a “smart” customer has given us a strong position to challenge supplier costs and bring about overall cost savings. then this attribute of the project would be carefully monitored and reported. risk mitigations. If this was a critical factor for success. the introduction of the estimation capability represents a return on investment of over 1000:1. we would expect a high Figure 7. if a project is taking on novel features. The output from an estimate is a measurement plan of critical factors that need to be monitored.2 uses for an estimation tool Summarising all the messages from this paper. you assume a high performance team. You need a good estimation tool to help with reasoning. They help us optimize and refine the business around objective reasoning rather the subjective guesswork i. or there are concerns over the aircraft maturity. we have seen an improvement in stability and productivity – on average around 11% cost saving per project. This information has shaped what we measure. as shown in figure 7. then this aspect of the project will need to be carefully monitored and reported. can sit at the heart of the business and can be used in the following ways 1 Estimation! Obviously the tool can be used to estimate but they have also been used to estimate the costs of our supply chain.
The triangle shows an example project (or business) and how it lies on the sensitivity range. it is declaring its assumptions about each of these factors. the bottom of the red represents the worst and where the two meet represents an average industry project. For example. Improvement Opportunities for each COCOMO II factors. The tool contains a great deal of industry data and it is therefore possible to benchmark your business against an industry average project. which would add to the cost of the project. what is the risk if the team is moved onto another high priority project and the project needs to recruit a new team? Improved Monitoring: When we are asked to help a project define its measurement program. The top of the green bar represents the best setting. Any improvement project that does not influence a COCOMO factor is unlikely to have an impact on the project. By generating an estimate for a “struggling” project it is possible to identify its problems and greatest opportunities for improvement Benchmark. This allows us to identify where we can make the greatest improvements. we tell them to start with their critical estimate assumptions.2 Improvement identification: Once you complete the COCOMO estimate. it may seem impossible to compare cost e. how could the team be centralised? Figure 8 shows the full range of sensitivity Figure 8.g. As such. Diagnostics: In the same way that COCOMO II can be used to elicit improvements. for an average project. Also. At Rolls-Royce it is now necessary to provide traceability between each improvement and the COCOMO II factors. Risk Identification: In addition to eliciting and validating improvements. we have used the tools to quickly find the root cause for project problems. When a project generates an estimate. If COCOMO II represents a model of the project and can be trusted to estimate costs. if the organisation is scattered across several continents. if you business has several domains of projects. they should be monitored. if the project has an above average team. then it is implicit that it represents a complete model of a project. 10-15 will be sensitive and uncertain. For example. Improvement validation. the project can then ask questions about how they could improve the settings. the tool can also be used to elicit and validate risks. COCOMO II contains some 21 factors of which. a safety critical 3 4 5 6 7 .
For example. cost and resource requirements) and given corresponding gains in productivity. it is then possible to compare data across the business. ISBN 0-13-026692-2 . The introduction of estimation tools and processes has brought stability (project predictability in terms of time. location of teams and suppliers and so on. Reifer. B. References 1.. and Steece. More complex business models have been generated to help business leaders plan IT.. Also. "Software Cost Estimation with COCOMO II". This opens a business up to sharing a lot more information. one factor describes the benefit of a co-located team. It is not necessary to generate a complete estimate in order to make use of COCOMO II. Our suggestion is that every business should buy at least one of each estimation tool so as to gain a better and more refined understanding of costs.. E. Brown. 8 The 7 + 1 use of the tool is in decision making and trade analysis. Each factor represents a quantified and validated view on the world and can help a business improve.. Abts.. process standardisation. C. there is implicit training and orientation. Few managers can now adopt a team with immaturity or lack of experience.. Clark. the spin off benefits brought improvements in the way we make decisions.. It is possible to use the factors in isolation. identify and validate improvements and even in how we communicate within the business.project cannot be compared with a non-safety critical project. if the project data is “corrected” for domain differences. to acquire this level of knowledge would take several hundred years. Horowitz. S. It can link together engineering. The adoption of a tool can accelerate a business‟s collection of data points and other tools in industry claim to have thousands to many 10‟s of thousands of data points. B. Boehm. It is also the bridge between different areas of the business that work in different domains. The business can generate a very quick business case for renting a large building by using this factor in isolation. COCOMO II was based on several hundred projects. risk management and improvement activities into a unified tool. Prentice-Hall. It can also be a bridge into industry to make direct comparisons with peer projects from other companies. The 7 + 2 use of the tool is its presence alone is enough to change culture. B. 9 Summary COCOMO is a unifying language that sits at the heart of the business. R. However. or take on low Technology Readiness projects without at least remembering COCOMO and its warnings. Chulani. tool acquisition. In demonstrating the tool around the business. W. Madachy.. For an organisation like Rolls-Royce. D. projects.
He is Vice-Chair of the SAE Aerospace Council and represents Rolls-Royce on the INCOSE Corporate Advisory Board. a Fellow of the Institute of Materials. Andrew has spent over a decade managing large scale software projects as well as a decade improving the way Rolls-Royce manages projects. Andrew Pickard joined Rolls-Royce in 1977 after completing a Ph. at Cambridge University in Fatigue and Fracture of Metals and Alloys. He is a Fellow of the British Computer Society and a chartered Engineer in software engineering.Biography Andrew Nolan joined Rolls-Royce in 1989 after completing a degree at Sheffield University. Minerals and Mining. of SAE International and of INCOSE. He is a Rolls-Royce Associate Fellow in System Engineering. .D. He is the Chief of Software Improvement for Rolls-Royce based in the UK. a Chartered Engineer and a member of the American Institute of Aeronautics and Astronautics.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.