This action might not be possible to undo. Are you sure you want to continue?
Karen Richey Jennifer Echard Madhav Panwar
• • • • • • • • • • • Introduction to Cost Estimating Life Cycle Costs Data Collection Data Analysis Cost Estimating Methodologies Software Cost Modeling Cross-checks and Validation Risk and Sensitivity Analysis Documentation Requirements Cost Estimating Challenges Cost Estimating Auditor Checklists
Introduction to Cost Estimating
• The National Estimating Society has defined Cost Estimating1as:
– The art of approximating the probable cost of something based on information available at the time.
• Cost estimating cannot:
– Be applied with cookbook precision, but must be tailored to a particular system,
– Substitute for sound judgment, management, or control,
– Produce results that are better than input data, or – Make the final decisions.
• Despite these limitations, cost estimating is a powerful tool because it:
– Leads to a better understanding of the problem,
– Improves management insight into resource allocation problems, and – Provides an objective baseline to measure progress.
The NES Dictionary, National Estimating Society, July 1982
Introduction to Cost Estimating
• The reliability of cost estimates varies over time.
– The closer you get to the actual completion of a project, the estimate becomes more accurate.
• Four types of cost estimates represent various levels of reliability .
– Conceptual Estimate: Rough order of magnitude or back of the envelope.
• Often inaccurate because there are too many unknowns.
– Preliminary Estimate: Used to develop initial budget, more precise. – Detailed Estimate: Serves as a basis for daily project control. – Definitive Estimate: Accuracy should be within 10% of final cost.
• Important to repeat estimating process (i.e., re-estimate) on a regular basis as more information becomes available
– This will keep estimate current as well as increase the accuracy
Introduction to Cost Estimating (cont‟d)
• All cost estimates are constructed by the following tasks:
– Identifying the purpose and scope of the new system.
• New software development, software reuse, COTS integration, etc.
– Choosing an estimate type.
• Conceptual, preliminary, detailed, or definitive type estimate
– – – – –
Identifying system performance and/or technical goals. Laying out a program schedule. Selecting a cost element structure (CES). Collecting, evaluating, and verifying data. Choosing, applying, cross-checking estimating methods to develop the cost estimate. – Performing risk and sensitivity analysis – Time-phasing the cost estimate by fiscal year for cash flow purposes.
• Example: 4 years to develop and 10 years operations and support beginning in FY2003
– Providing full documentation.
Life Cycle Costs
• Most cost estimates in the federal government represent total Life Cycle Costs (LCC).
– LCC estimates include all costs to develop, produce, operate, support, and dispose of a new system. – Important to look beyond the immediate cost of developing and producing a system and consider all costs of a system‟s life cycle.
• What may appear to be an expensive alternative among competing systems may be the least expensive to operate and support
• Life Cycle Cost Estimates can be used to:
– Compare various alternatives before committing funds to a project, – Support “Estimate-to-Budget” transition after time-phasing to account for when funds will be spent.
Life Cycle Costs (cont‟d) • A LCC estimate is summarized using a detailed cost element structure (CES) that – Identifies the activities required to complete project development and the effort. and duration of each task. – Provides a framework against which the cost estimate is organized. loading. 7 . • Enhances cost data collection and estimate reporting. • Facilitates comparing estimates when a standard CES is used.
8 . when more detailed data is available. • The CES is hierarchical in nature to accommodate early development (when relatively little data is available) through deployment. • The CES should be reviewed to ensure that there is no „double counting‟ of costs that could be allocated to more than one element. – Helps to identify costs that may be initially overlooked. – Should be tailored for each project. logistics support costs could be included in the investment or operations and support phase. – For example.Life Cycle Costs Cost Element Structure (CES) • A CES provides a standard vocabulary for the identification and classification of cost elements to be used in cost estimating.
1 System Design & Specification 1.4.1 System Management 2.7 Upgrade (Pre-Planned Product Improvement (P3I)) 1.3 Hardware Maintenance 2. Implementation & Fielding 1.4.2 Deployment Software 1.7 Site Operations (personnel.0 Investment Phase 1.1 Program Management 1.4Training 1.5 Outsource Provider Investment 1.6 Data Maintenance 2.4.) 1.Life Cycle Costs Example of CES Elements (DOD) 1.3.3 System Development 1.6 Facilities 2.5 Outsource Provider Support 188.8.131.52.5 Initial Spares 1. training.1 Deployment Hardware 1.8 Disposal Costs 9 .0 System Operations & Support Phase 2.3.2 Development Prototype and Test Site Investment 1.6 Warranties 1.3 Software Development 1.4 System Procurement 1.4Logistics Support Equipment 184.108.40.206 Software Maintenance 2.2 Concept Exploration 1.3 Initial Documentation 1.etc.4.6 System Initiation.2 Annual Operations (supplies/spares) 2.
etc. – Important to pick data that is a predictor of future cost • Important to have technical and schedule data because they act as cost drivers 10 . function points. SLOC.Data Collection Data Required • All CES elements will need data to support the estimate • Historical cost and non-cost data need to be collected to support various estimating techniques • Technical non-cost data describes the physical. # of integrated circuit boards. and engineering characteristics of a system – Examples: weight. # of design drawings. performance. square footage.
material. insurance. and fringe benefits. rent. quality assurance. • Also included are direct non-wage costs such as training. and travel. etc. • Example: sick/annual leave. health insurance. These expenses are typically charged to a company as a whole. utilities.Data Collection Data Required(cont‟d) • Identify both direct and indirect costs – Direct costs are called “touch labor” and include direct manufacturing. supplies. network charges. – Indirect costs are considered “overhead” and include such things as general & administrative support. etc. retirement pay. engineering. 11 . costs which have a direct bearing on the production of a good.
worse yet. underestimating – Important to ask when collecting data whether costs are burdened with indirect costs 12 .Data Collection Data Required (cont‟d) • Some direct costs may be burdened with indirect costs and some may not – Need to know to avoid double-counting or.
if available for earlier versions of the current system • Can save valuable research time for statistical analysis – Published cost studies • Data collection is a critical and time consuming step in the cost estimating process! 13 .Data Collection Data Required (cont‟d) • Data can be collected in a variety of ways – Contractor site visits – Data requests for all relevant CES elements – Documented cost estimates.
actual program data. proposal data. contractor reports.Data Collection Typical data sources • Two types of data sources: – Primary & Secondary – Primary data is found at the original source (e. etc.. • Second choice to primary data due to data gaps • May be best alternative when time constraints or data availability limit primary data collection 14 .g. etc. cost studies/research.) • Preferred source of data – Secondary data is derived from primary data of another similar system such as documented cost estimates.
. Microsoft. Dell. etc.Data Collection Typical data sources • Current Program Milestone schedule (for phasing program costs over time) • Current Program System Description – Important to have a technical understanding of how the system will work • Approved program funding (Compare to proposal estimate) • Contractor actual cost reports (from internal Management Information System) • Current program estimate documentation (if available) • Contractor proposals (Compare to program funding profile) • Commercial Off-the-Shelf (COTS) catalogs • Manufacturer websites (e.g.) 15 .
Data Collection Typical Data Sources (cont‟d) • Forward Pricing Rate Agreements (FPRA) • Similar program historical actual costs and estimate documentation • Engineering drawings/specifications. • Interviews with technical and program management personnel • Surveys • Professional journals and publications • Industry guides and standards • Technical Manuals 16 .
Data Analysis Data Validity/Integrity • Important to ensure that the cost data collected is applicable to the estimate. appropriate adjustments need to be made to account for differences in the new system being estimated. – Identifying limitations in the historical data is imperative for capturing uncertainty – When using historical cost data for a similar system. – Data may need to be “mapped” from the contractor‟s accounting system to the Cost Element Structure (CES) 17 .
– Participate in a fact-finding trip to discuss contractor proposal estimates and gather supporting data/evidence.Data Analysis Data Validity/Integrity (cont‟d) • Proposal data should be validated to ensure that contractor motivations to “buy-in” or low-bid their estimates are not occurring. – Compare previous contractor proposal bids and actual costs for similar programs. • Look for trends in underbidding. 18 .
Anomalies such as strikes. Production rate changes or breaks in production. major test failures. 19 .Data Analysis Normalization • Involves making adjustments to the data so that it can account for differences in – – – – – Inflation rates. Recurring and non-recurring costs. or natural disasters causing data to fluctuate. and – Learning curve (cost improvement) effects due to efficiencies gained from continually repeating a process • Analysis of the data may indicate the need for more suitable data to add credibility to the estimate. Direct/indirect costs.
• Cost estimators use inflation indices to convert costs to a constant year dollar basis to eliminate distortion that would otherwise be caused by price-level changes. • Constant dollar estimates represent the cost of the resources required to meet each year‟s workload using resource prices from one reference year (e. • Wrong to compare costs from 1980‟s to today without accounting for inflation over the past 20 years.Data Analysis Normalization (cont‟d) • Accounting for Economic Changes (e. constant 2003 dollars).. inflation) – Lack of cost data uniformity due to upward movement in prices and services over time..g.g. – Index numbers are used to deflate or inflate prices to facilitate comparison analysis. – Constant dollars reflect the reference year prices for all time periods allowing analysts to determine the true cost of changes for an item 20 .
Data Analysis Normalization (cont‟d) • For the United States. the Office of Management and Budget (OMB) is responsible for developing inflation guidance by appropriation for government estimates.htm#data) for services • Important to pick the appropriate “market basket” of goods index that most closely matches the costs to be estimated.gov/ppi/) for goods – Employment and Earnings Index (http://www. – Producers Price Index (http://www. • Most common indices used by United States Cost Estimators are published by the U.S. 21 .bls. Bureau of Labor Statistics. Department of Labor.bls.gov/ces/home.
data expressed in constant dollars should be inflated to represent current year (or “Then-year”) dollars. • Current year estimates reflect inflation – Necessary to express estimates in current year dollars when requesting funding to avoid budget shortfalls.Data Analysis Normalization (cont‟d) • For budgetary purposes. 22 . – Current year estimates calculate the cost of the resources using the estimated prices for the year in which the resources will be purchased. however.
500 * 1.285 Current year + 2 $4.2155 multiplier = $ 7.144.700 * 1.167 23 .391 Current year + 3 $8.092.394.1576 multiplier = $ 9.944 Current year + 5 $1.1025 multiplier = $ 4.400 * 1.100 * 1.0000 multiplier = $ 273.750 * 1.484.193.Data Analysis Normalization (cont‟d) Example: Assume 5% escalation rate compounded annually Base-year cost Then-year cost Current year $273.096.230 Total Estimated Labor cost (budget quality) $25.144.516.2763 multiplier = $ 1.468.0500 multiplier = $ 2.007.911.000 * 1.100 Current year + 1 $1.217 Current year + 4 $6.
Engineering. 24 . there are five methodologies available for estimating costs: – – – – – Expert Opinion. and Actual. Parametric.Cost Estimating Methodologies • Once data has been collected and normalized to constant dollars. Analogy.
Support – Available data • Historical data points from earlier system versions or similar system • Technical parameters of system 25 . Production. hardware. etc – Phase of program • Development.Estimating Methodology Considerations • Choice of methodology is dependent upon – Type of system • Software.
• Useful in assessing differences between past projects and new ones for which no historical precedent exists. Software Engineering Economics. proposed by Dr. Barry Boehm in his book.Methodology Expert Opinion • Often called Delphi method. 26 .
– Experts may introduce bias. • Cons: – Very subjective.Methodology Expert Opinion (cont‟d) • Pros: – Little or no historical data needed. – Suitable for new or unique projects. 27 . • Larger number of experts will help to reduce bias – Qualification of experts may be questioned.
• Each member of the expert group then does an independent of the resources needed. and • The expert group once again submits anonymous. • If there exists significant divergence among the estimates.Methodologies Expert Opinion . • The expert group then discusses the estimates and the divergence and works to resolve differences. 28 . • Estimates are gathered anonymously and compared. independent estimates which continues until a stable estimate results. the estimates will be returned to the expert group. • Describe overall program in enough detail so experts can provide an estimate.Steps • Gather a group of experts together.
Methodology Expert Opinion . – You hold interviews which each explaining the requirements. – Each member of the group submits their opinion of the final cost and the estimate converges to $50M. sizing level. 29 .Example • Three software engineers are renown in the ERP software development. and development process for your new system.
Methodologies Analogy • Estimates costs by comparing proposed programs with similar. – Actual costs of similar existing system are adjusted for complexity. 30 . previously completed programs for which historical data is available. or physical differences to derive new cost estimates – Analogies are used early in a program cycle when there is insufficient actual cost data to use as a detailed approach – Compares similarities and differences • Focus is on main cost drivers. technical.
Methodologies Analogy (cont‟d) • Often use single historical data point. – May have to add or delete functionality from historical costs to match new program • Consult technical experts for advice on complexity factors. • May have several analogy estimates of sub elements to make up the total cost. – (not to be confused with expert opinion method) • Impact of differences on cost is subjective. 31 . performance or physical differences. technical. • Comparison process is key to success.
32 . • Make sure actual cost data is available – A system where technology/programmatic assumptions have advanced beyond any existing cost estimating relationships (CER).Methodologies Analogy (cont‟d) • Good choice for: – A new system that is derived from an existing subsystem. – Secondary methodology/cross check • Provides link between technical assumptions and cost.
Methodologies Analogy (cont‟d) • Pros: – Inexpensive – Easily changed – Based on actual experience (of the analogous system) • Cons: – – – – Very Subjective Large amount of uncertainty Truly similar projects must exist and can be hard to find Must have detailed technical knowledge of program and analogous system 33 .
Develop factors based on prior system.Steps • • • • • • • • Determine estimate needs/ground rules.Methodologies Analogy . Assess data availability of historical similar systems. – Obtain complexity (and other translation) factors – Example: Historical database is for 4 million records and new database will need to house 24 million records => complexity factor of 6 (4*6 = 24) times the historical database cost – Apply factors to historical costs to obtain new system costs 34 .. constant FY2003$). Process/normalize data into constant year dollars (e. Collect historical system component technical and cost data.g. Define new system. – Example: Program Management is 10% of total development cost Develop new system component costs. Breakout new system into subcomponents for analogy estimating.
000 people and containing 100.000 lines of C++ code.000 lines of code system for $20M for only 2.Example Your company is developing a new IT payroll system serving 5.000.25 * $20M = $25M Plus 5. or 2. Estimated cost for new system = 1.Methodologies Analogy . Other system development cost was $20M.000 people.5 * $25M = $62.000/2. Your software engineers tell you that the new system is 25% more complicated than the other system. Another company developed a similar 100.5M total cost 35 .
• Most cases will require in-house development of CER.Methodologies Parametric • Utilizes statistical techniques called Cost Estimating Relationships (CER). • Can be used prior to development. Database size. 36 . – Relates a dependent variable (cost) to one or more independent variables – Based on specific factors that have a high correlation to total cost • • • • Number of software lines of code (SLOC) or function points. • Typically employed at a higher CES level as details are not known. Number of floors in a high rise building for cabling estimates. Square feet for office floor space. etc.
Methodologies Parametric (cont‟d) • Pros: – – – – – Can be excellent predictors when implemented correctly Once created. lower level may be not visible – Need to be properly validated and relevant to system 37 . CERs are fast and simple to use Easily changed Useful early on in a program Objective • Cons: – Often lack of data on software intensive systems for statistically significant CER – Does not provide access to subtle changes – Top level.
You have estimated 2.000 to 3.000.000 SLOC for new system Cost = 2. Cost = SLOC * 25 $/SLOC The CER is based on systems ranging from 1.Methodologies Parametric Example You have a previously developed CER to estimate a new IT system based on SLOC.000 SLOC.600.600.000 * $25 = $65M 38 .000.
5 times Quality Assurance costs – Cost to Non-Cost: $/pound. 39 . – Cost to Cost: e.g. – Various Software cost estimating models will be discussed next • Learning curves – specialized type of CER. • CERs can be cost to cost or cost to non-cost. Manufacturing costs are 1.Methodologies Parametric (cont‟d) • Cost estimators can develop their own CERs or they can use existing commercial cost models. or engineering hours/# of engineering drawings yields hours/drawing metric that can be applied to new program • Factors and ratios are also examples of parametric estimating..
40 . • CERs use quantitative techniques to develop a mathematical relationship between an independent variable and a specific cost.Methodologies Parametric CERs • CERs are defined as a technique used to estimate a particular cost by using an established relationship with an independent variable. • Can be as simple as a one variable ratio to a multi-variable equation.
or the ability to predict one variable on the basis of the knowledge of another variable) Multiple regression (a change in the dependent variable can be explained by more than one independent variable) Curvilinear regression (relationship between dependent and independent variable is not liner. but based on a curve) Learning curve (describe how costs decrease as the quantity of an item increases) 41 Must be logically sound as well as statistically sound. Useful at any stage in a program. – – – – – – In later stages of a program. Typically CERs are the main cost estimating methodology in early stages of a program. normalized data is most important for CER development.75 or higher) for goodness of fit test Least squares best fit (regression analysis. CERs serve as a cross check to other methods High correlation (r2 = 0. . Different statistical techniques may be used to judge the quality of the CER. Must determine range of data for which the CER is valid.Methodologies Parametric CERs (cont‟d) • • • • • • Reliable.
as a whole. CV: • SE divided by mean of the Y-data. relative measure of estimating error – t-stat • Tests whether the individual X-variable(s) is/are valid – F-stat • Tests whether the entire equation. Best to use a statistical software package like SAS or SPSS to quickly evaluate alternative CERs.Methodologies Parametric CERs (cont‟d) • Statistics may be used to evaluate how well the CER will produce a reliable estimate. is valid • • No single statistic may validate or invalidate a CER. How close the points are to the line) – Standard Error. quality of the input data is just as important. 42 . SE: • Average estimating error when using the equation as the estimating rule – Coefficient of Variation. R2: • Percent of the variation in the Y-data explained by the X-data. – Coefficient of determination. (ie.
.Methodologies Parametric CERs . – – Variables should be quantitatively measurable and available If there is a choice between developing a CER based on performance or physical characteristics. comparable. – – – Use statistical analysis to judge strength of relationship and validity of equation A high correlation often indicates that the independent variable will be a good predictive tool Identify independent variables tested. hours. and free of unusual costs • • • Explore the relationship between the dependent and independent variables. etc. and Document your findings.) and what the CER will estimate. performance characteristics are generally the better choice because they are known early on • Collect data concerning the relationship between the dependent and independent variables.g. 43 . data gathered along with sources. – Most difficult and time consuming step. time period (normalization for inflation effects). and any adjustments made to the data Select the relationship that best predicts the dependent variable. but essential that all data is checked to ensure that all observations are relevant.Steps • • Define the dependent variable (e. Select independent variables to be tested for developing estimates of the dependent variable. cost dollars.
the cost per unit decreases by a fixed percentage • The concept of learning curves is not new.P. – As the number of units produced doubles. 44 . Wright in the Journal of Aeronautical Sciences. similar tasks – This improvement is usually quantified at a rate Y = AXb • In simplest terms. originated in the mid-1930‟s with T.Methodologies Parametric Learning Curves • Basic premise: – Repetitive tasks should result in productivity for subsequent. the cost of manufacturing or installing a unit should decrease as the number of units involved increases.
Methodologies Parametric Learning Curve .Example Say that the first 100 tasks of an installation took 10 hours per task and the next 100 averaged 8 hours per task. the learning curve would be calculated as follows: Learning curve = 8 hours per task/10 hours per task = 0. Thus.8 Implies an 80% learning curve meaning an improvement of 20% occurred between the first 100 tasks and next 200 tasks 45 .
Methodologies Rate Impact • Basic premise: The number of units produced in a single lot effects the overall cost of producing that lot. – Costco theory that buying in bulk makes the unit cost less • Mathematical equation showing rate impact along with learning curve – Y = AXbQr • Both rate and learning curves can be impacted by the following: – Operator turnover rate (new employees do not meet expected productivity standards immediately) – Production reworks – Material handling and downtime (learning curves assume material is ready when needed) – Engineering rework – Rework of vendor supplied parts 46 .
Uses detailed cost element structure (CES). • • • • Estimate is built up from the lowest level of system costs. Must include all components and functions. – Underlying assumption is that future costs for a system can be predicted with a great deal of accuracy from historical costs of that system. 47 . – Involves examining separate work segments in detail and then synthesizing these detailed estimates along with any integration costs into a total program estimate. Can be used during development and production.Methodologies Engineering • Also referred to as bottoms up or detailed method.
etc.Methodologies Engineering (cont‟d) • Most useful for systems in production. engineering. 48 .g. manufacturing. quality control. – Design configuration has stabilized – Test results are available – Development cost actuals are available • Typically broken into functional labor categories e.
Methodologies Engineering • Pros: – Objective – Reduced uncertainty • Cons: – – – – Expensive Time Consuming Not useful early on May leave out software integration efforts 49 .
Methodologies Engineering Steps • • • • • Understand program requirements. and – – – – – Use other program history Compile estimate data Develop rates and factors Incorporate supplier/subcontractor prices Include integration costs to “glue” the separate components into an integrated system (may need to use a CER for this estimate) 50 • Summarize estimate. Prepare program baseline definition. . Develop detailed cost element structure. Develop functional estimates. Define ground rules and assumptions.
Methodologies Engineering Example New IT system can be broken down into Labor and Material costs.000 hrs * $200/hr = $50M Material: From vendor quotes Server = $10M COTS = $3M Desktops (50 @ $100k) = $5M Cost = $50M+$10M+$3M+$5M = $68M 51 . Labor: Estimated number of hours 250.
Methodologies Actual • Bases future costs on recent historical costs of same system. • Used later in development or production. – Costs are calibrated to actual development or production productivity for your organization 52 . • Preferred method.
Methodologies Actual • Pros: – Most accurate – Most objective of the five methodologies • Cons: – Data not available early on – Time consuming – Labor intensive to collect all the data necessary 53 .
According to earned value cost performance reports (CPRs) the estimate at complete for the remainder of the labor is another $20M. The labor cost to date is $30M.Example The development process is nearing completion. The material has all been procured at a cost of $20M. Cost = $20M + $30M + $20M = $70M 54 .Methodologies Actual .
$50M Analogy .$70M 55 .Methodologies Summary of the Five Cost Estimate Results • • • • • Expert .$62M Parametric – $65M Engineering – $68M Actual .
• Improper assessment of staffing levels and skills. • Accurately capturing software costs can be difficult and cost overruns often occur as a result of software requirements being difficult to estimate and track. and • Lack of well-defined requirements for the specific software activity being estimated 56 . • Inability to accurately specify a software development and support environment. – Software estimating problems occur most often because of the: • Inability to accurately size a software project.Software Cost Models • Software costs as a percentage of total system costs continues to increase while associated hardware costs decrease.
– Programming languages also have a significant effect on overall cost. • Software lines of code (SLOC) are often used to estimate software size which is the most significant driver in determining software costs. 57 . – In the early stages of software development.Software Cost Models (cont‟d) • The requirement to develop cost estimates for systems at a time when few details are known has led to the development of cost models. few details other than performance requirements and some physical constraints are available. – The need to generate estimates quickly and for many alternatives can make the analogy and engineering methods impractical due to lack of detailed requirements.
and labor rate/overhead applications. 58 . & inflation calculations to convert base year to then-year dollars. – Models approximate the real world through a combination of • Statistical analysis of historical data. • Informal/Intuitive analysis of rules of thumb based on experience.Software Cost Models (cont‟d) • Software cost models are based on statistically derived cost estimating relationships (CERs) and various estimating methodologies used to predict the cost of a system. inflation computation. – Models may contain the following features: • Support costs calculations. • Standard procedures such as learning curves. time-phasing to spread development costs by fiscal year.
3 Software Development: • • • • • • • • • • • • • Requirements Definition Analysis. Assembly. – The combination of software development and support is known as the software life cycle. Design. Test and Checkout System Operational Test and Evaluation System Independent Software Verification and Validation Site Acceptance Testing Independent Operational Test and Evaluation Program Management Installation and Checkout Corrective Maintenance 59 .Software Cost Models (cont‟d) – Regardless of how software is programmed. – The Cost Element Structure (CES) discussed earlier accounts for the software life cycle using the following sub-elements for CES 1. it must proceed through certain steps or phases and must be supported after development. and Integration Quality Assurance Program Configuration Management Software Design and Development HW/SW Integration.
Software Cost Models (cont‟d) Examples of Commercially Available Models • • • • • COCOMO COSTXPERT SLIM SEER Costar. REVIC. etc. 60 .
• COCOMO was originally published in Software Engineering Economics by Dr. 61 .Software Cost Models (cont‟d) COCOMO • Constructive Cost Model (COCOMO) is one of the earliest cost models widely used by the cost estimating community. • COCOMO is a regression-based model that considers various historical programs software size and multipliers. Barry Boehm in 1981.
62 . – Example: # of person months * loaded labor rate = Estimated Cost • Most of the other estimates (requirements. maintenance.Software Cost Models (cont‟d) COCOMO • COCOMO‟s most fundamental calculation is the use of the Effort Equation to estimate the number of Person-Months required to develop a project. etc) are derived from this quantity.
COCOMO has „scale factors‟ which allow you to tailor conditions to your project. – Scale factors are used to calibrate or adjust the model that is based on data which is not necessarily representative of the system being estimated • Intent of calibrating is to allow for general application of a model developed from a specific database • Based on the assumption that any differences between the predicted cost and the historical cost not explained by the model are due to: – Differences in the type of systems and acquisition environment represented in the two data sets 63 . • In addition to SLOC.Software Cost Models (cont‟d) COCOMO • COCOMO requires as input the project‟s estimated size in Source Lines of Code (SLOC).
and process maturity among others. • Additionally COCOMO uses „Cost Drivers‟ grouped in broad categories such as personnel. product. team cohesion. and tool experience 64 . project etc.Software Cost Models (cont‟d) COCOMO • Scale factors include: – Development flexibility. architecture. platform. risk. platform. – Product Cost drivers include: • required reliability and product complexity. – Personnel cost drivers include: • language.
65 . proprietary database of data submitted by consortium members. – Allows COCOMO to be compatible with current design methods and evolving higher order languages. – Cost estimators must calibrate the model to the program being estimated to their specific development environment and productivity level of staff. • COCOMO uses analogy and parametric methods based pm a historical.Software Cost Models (cont‟d) COCOMO • COCOMO defaults to a nominal value for all factors when model is first used.
94 * EAF * (KSLOC)**E • Where EAF is the Effort Adjustment Factor derived from the Cost Drivers and E is an exponent derived from the five Scale Drivers. 66 . KSLOC)): Effort = 2.Software Cost Models (cont‟d) COCOMO • The model makes its estimates of required effort (measured in Person-Months) based primarily on the project‟s estimate of the software size (as measured in thousands of SLOC.
• Assuming that the project consists of 15.0997 = 57. EAF would have a value of 1. COCOMO estimates that 57. – Person-months multiplied by a loaded labor rate (includes direct and overhead costs) would yield the cost estimate: • 57.77 person-months * $15K per person-month = $866.0997.000 source lines of code.77 Person-Months of effort is required to complete it. E. • Effort = 2.77 Person-Months.94 * (1.000 67 . of 1.00 and exponent.Software Cost Models (cont‟d) COCOMO • For example assuming a project with all nominal cost and scale drivers.0) * (15)**1.
Software Cost Models (cont‟d) COCOMO Model output example 68 .
Software Cost Models (cont‟d) COCOMO Example of how model defaults to nominal scale factors. 69 . unique development effort. These should be adjusted for every estimate to approximate the new.
Software Cost Models (cont‟d) COCOMO outputs by phase 70 .
Software Cost Models (cont‟d) COCOMO outputs by phase 71 .
Software Cost Models (cont‟d) COCOMO overall estimate by phase 72 .
maintenance cost and schedule estimates. schedule distribution by phase. • A free updated version of COCOMO is available at: http://sunset. and increment.Software Cost Models (cont‟d) COCOMO Summary • COCOMO provides the following information: – Software development.usc. activity.edu/research/COCOMOII/ 73 . – Cost.
etc. 74 .Software Cost Models (cont‟d) • • • • • • COCOMO COSTXPERT SLIM SEER REVIC Costar. Price S.
CostXpert maps it to the closest projects in its database and gives you results tailored to your project. function points. • When you enter project specific information. • CostXpert can accommodate a variety of software sizing elements such as SLOC. 75 . GUI metrics. – CostXpert uses both parametric and analogy estimating methods. object metrics. military. and feature points.).Software Cost Models (cont‟d) CostXpert • CostXpert builds on COCOMO by having a database of actual projects that is constantly being updated for various types of systems (commercial. etc.
com/products/downloads. CostXpert creates a detailed cost element structure (CES). Labor loading by phase and functional categories. – In addition. risk analysis.Software Cost Models (cont‟d) CostXpert • CostXpert also allows for customization specifically to your work environment. conducts „what if‟ scenarios.html 76 . and provides a wide variety of reports to include: • • • • • Cost allocations by task list. Maintenance and life-cycle costs. Software defects estimates.costxpert. and Documentation deliverables • GAO uses CostXpert during various cost analysis assessments to check the reasonableness of both the software cost estimate and accompanying schedule for delivery. • Evaluation copies can be downloaded at http://www.
Software Cost Models (cont‟d) CostXpert example of input screen 77 .
etc.Software Cost Models (cont‟d) • • • • • • COCOMO COSTXPERT SLIM SEER REVIC Costar. Price S. 78 .
Windows.Software Cost Models (cont‟d) SLIM • The Software Life Cycle Model (SLIM) is marketed by Quantitative Software (QSM). SLIM-Metrics and SLIM-Master Plan. • Set of tools include SLIM-Estimator. SLIM-Control. Function Points. – Input metrics include SLOC. Objects. Screens. 79 . SLIM-Data Manager. or Diagrams – Historical productivity and complexity are also inputs to the model • Quick start wizard enable rapid generation of estimates and details which can then be tailored for the specific project.
Software Cost Models (cont‟d) SLIM • Results are presented at the 50% probability and can be adjusted by using „sliders‟ for various parameters. 80 . • Estimates can readily be compared to historical data and model allows for analysis of „what if‟ scenarios. • A Productivity Index (PI) is calculated for your project and updates as new data is available.
the PI is a macro measure of the total development environment. • Demo copies of SLIM can be downloaded at http://www. including: – Management influence. and capable management and well-understood. – High values are associated with mature processes. process. – Low values generally are associated with poor environments. skill and experience. staffing and skill breakout. reliability estimates. and documentation sections. estimation analysis views.qsm. It incorporates many factors in software development. 81 . schedule. development methods. straightforward projects.1 to 40.com/ . • Values for PI range from . techniques. effort and cost. risk analysis. • Outputs of the model include: project description. tool.Software Cost Models (cont‟d) SLIM • As described in the user‟s manual. tools and complex systems.
82 .Software Cost Models (cont‟d) • • • • • • COCOMO COSTXPERT SLIM SEER REVIC Costar. Price S. etc.
Planning. Planning and Project Control • Hardware Project Control Hardware Estimation. Process and Assembly 83 . Including Operations and Support • Design for Manufacturability Cost Designer for Parts.Software Cost Models (cont‟d) SEER • This set of tools provides both software and hardware estimation capabilities. Project Control and LifeCycle Cost Analysis. • Software Project Control Software Estimation.
effort.galorath.shtm 84 . – SEER-IC for estimating foundry costs. delivered defects. – SEER-H for estimating hardware and hardware Operations and Support costs. and – Staffing by month.com/tools_overvw.Software Cost Models (cont‟d) SEER • The toolset includes: – SEER-SEM for evaluating software development. • Demo copies of SER-SEM and other tools can be downloaded at http://www. – SEER-DFM for manufacturability analysis. • Outputs of the model include – Labor estimates by function. – SEER-SSM for software sizing. and schedule. – A “quick estimate” providing a snapshot of size. person-months by activity. cost by activity. and SEI maturity rating.
etc.Software Cost Models (cont‟d) • • • • • • COCOMO COSTXPERT SLIM SEER REVIC Costar. Price S. 85 .
REVIC changed the coefficients based on a database of recently completed DOD projects. 86 . The main difference between REVIC and COCOMO is the coefficients used in the effort equations. REVIC's schedule estimates are often considered lengthy because it assumes that a project's documentation and reviews comply with the full requirements of DOD-STD-2167A/498. Raymond L. Kile formerly of Hughes Aerospace.Software Cost Models (cont‟d) REVIC The Revised Enhanced Version of Intermediate COCOMO (REVIC) model was developed by Mr.
Software Cost Models (cont‟d) • • • • • • COCOMO COSTXPERT SLIM SEER REVIC Costar. etc. Price S. 87 .
etc. Almost all need to be calibrated with project specific data. Most are based on COCOMO. – Crosschecking estimates leads to better cost estimate validation • The International Council on Systems Engineering (INCOSE) has a list of software tools that may be of use. Using multiple models and comparing results provides better confidence.org/tools/tooltax/costest_tools.html 88 . Price S. • • • • There are many other models available.Software Cost Models (cont‟d) Costar. – http://www.incose.
Cross-checks and Validation • After an estimate has been created. – Can be used to cross-check the reasonableness of costs forecasted using either the engineering or actual cost methods. – If both estimates are close. the target estimate has some validity. cross-checking other cost elements can further validate the entire estimate. – If time is available. – Cross-checking means using a different approach to create the estimate. • Software cost models play an important role during later stages of development phase. • This may lead to another estimating method to increase cost estimate confidence. the next step involves validating the estimate by cross-checking. • It is a good practice to cross-check major cost drivers. • This increases the level of uncertainty which must be reflected in a risk analysis. 89 . • Factors tend to be the most common approach for top level checks. – If both estimates are very different.
and The CER or cost model is a good predictor of costs. • The model‟s cost output would then be compared to the test point‟s “known value” to provide an accuracy benchmark. • Validation also includes a demonstration that: – – – – The data relationships are logical. Strong data relationships exist. • This can be done using independent test data (not included in model‟s calibration).Cross-checks and Validation (cont‟d) • Validation is the process of demonstrating that either a CER or a cost model accurately predicts past history. The data used are credible. 90 .
Cross-checks and Validation (cont‟d) • Validation also includes a demonstration that: – – – – Model users have sufficient experience and training. Formal estimating policies and procedures are established. and When applicable. Calibration processes are thoroughly documented. information system controls are maintained to ensure the integrity of the models being used. 91 .
and cost estimating risk drivers for use in risk management. 92 .Risk and Sensitivity Analysis Risk Analysis Definition • Risk analysis is a process that uses qualitative and quantitative techniques for analyzing. quantifying and reducing uncertainty associated with cost goals. – As the project matures these uncertainties decrease due to greater design definition. and less opportunity for change. all cost estimates have some uncertainty. – Earlier in development that uncertainty is higher. schedule. • By nature. – Provide credibility to the target estimate. and – Identify technical. actual experience. • Cost risk analysis aims to achieve the following objectives: – Identify program level confidence for development schedule. • Errors can also occur from historical data inconsistencies.
– Example would be will it rain two weeks from today? • Cost estimating falls more into the range of uncertainty than risk.Risk and Sensitivity Analysis Risk Analysis Definition (cont‟d) • Risk is defined as a situation in which the outcome is subject to an uncontrollable random event stemming from a known probability distribution. but most managers use the term risk analysis. – Roll of two dice is an example since the roll can result in one of 11 possible outcomes. • Uncertainty is defined as a situation in which the outcome is subject to an uncontrollable random event stemming from an unknown probability distribution. 93 .
• Sensitivity analysis highlights the factors that have the strongest impact on the overall cost estimate. – Points to management which factors deserve the most attention. • Sensitivity examines the effect of primary cost estimating outputs to changes on individual CES inputs.Risk and Sensitivity Analysis Sensitivity Analysis Definition (cont‟d) • Sensitivity analysis answers the question: – What happens if the assumptions change? • Changes to some assumptions can have profound impact. 94 . – Narrows down the number of lower level cost elements that should be examined using risk analysis techniques. while huge changes to other assumptions have little effect on results.
Risk and Sensitivity Analysis Sources of Cost Risks • Schedule and Technical risks – – – – – – – – – – Unexpected design changes Project team experience Number of business units impacted Requirements changes Integration considerations Technical difficulties or maturity issues Revised project or acquisition plans Quantity changes New labor rates Higher inflation • Cost estimating risks – Imprecision associated with the estimating techniques used. or oversights 95 . errors.
Risk and Sensitivity Analysis Risk Analysis Techniques • Risk factor approach – Adds cost to the overall point estimate (most likely estimate) to cover risks by way of a factor. • Randomly generates values for uncertain variables over and over to simulate a model. • This factor is usually a percentage derived from past data and experience. • Monte Carlo Simulation – Automatically analyzes the effect of varying inputs on outputs of a cost model using spreadsheet risk analysis. • Often applied to the estimate as a whole versus lower level cost elements. 96 .
000 – 10. • Simulation calculates multiple scenarios of the cost model by repeatedly sampling values from the probability distributions assigned to the various cost elements. – These cost ranges are usually obtained through expert judgment (engineer or technical specialist interviews) or using a Delphi technique – After all cost elements have been identified by a distribution. • Cost distribution is usually triangular in the form of optimistic. most likely. 97 . the forecasts stabilize towards a smooth frequency distribution called a cumulative frequency distribution or CDF • After thousands of trials. – While the simulation runs. statistics of the results and the certainty of any outcome can be obtained from the CDF.Risk and Sensitivity Analysis Risk Analysis Techniques (cont‟d) • Monte Carlo Simulation Steps – Analyst obtains cost distribution for each element identified as a major cost driver either through experience or sensitivity analysis. the simulation is run many times (1. and pessimistic cost estimates.000 times).
a 50% chance of it costing $ 70 million. 98 . a simulation can show there is a 10% chance of the project finishing for $50 million.Risk and Sensitivity Analysis Risk Analysis Techniques (cont‟d) • Monte Carlo Simulation Results – Reveal not only the result values for each forecast. • For example. • Decision makers can use these results to decide which projects will continue funding based on quantifiable risk parameters. but also the probability of any value occurring. – This is helpful to management because it can show the level of certainty of achieving a cost objective. and a 90% chance of developing the project for $100 million or less.
Risk and Sensitivity Analysis Types of Risk Simulation Tools • US GAO uses Crystal Ball to perform Monte Carlo Simulations • Crystal Ball is an simulation program that helps you analyze the risks and uncertainties associated with Microsoft Excel spreadsheet based models.palisade.com/crystal_ball/ • Other tools are available.html) – RiskEase (http://www.com/html/risk.pertmaster. – http://www.decisioneering.com/) 99 .riskease. such as – @Risk (http://www.com/) – Pertmaster (http://www.
e. and most important one of all. involves documenting the entire estimate process. • Important to document the estimate as it is being developed (i. document as you go). – Documenting as you prepare the estimate leads to a better quality estimate and requires minimum effort at the end. the level of detail and attention will pay big dividends when it comes time to reestimate – May also help data collection of actual analogous costs.Documentation • After cross-checking the estimate major cost drivers.. the next step. 100 . – Hard to remember rationale and judgments for adjusting data months later when it needs to be documented. • Although this is a difficult and time-consuming procedure.
Documentation (cont‟d) • Best to provide more vs. 101 . less information – Assume reader knows nothing about the system being estimated. • Most users of the documentation will either be updating the estimate at a later date or will relay on data for estimating an analogous system. – Include step-by-step instructions for how the estimate was developed. • Aim to provide enough information for the estimate to be recreated by a cost analyst independent of the team.
POC information.. and area of responsibility Program background and system description Program schedule (necessary for phasing costs over time) Scope of estimate (including what was omitted and why) Ground rules and assumptions (all technical and program specific assumptions that bound the estimate – e. constant 2003 dollars using xx inflation rates) – Summary of estimate by year to show phasing by CES – Overview of main body that describes how each CES element was estimated 102 .Documentation (cont‟d) • Documentation should include at a minimum: – – – – – – – Introduction Purpose of estimate Estimate team members.g.
Summary of estimate by year showing phasing by CES element (In constant 2002 millions of dollars)
Investment Costs Systems Eng/Program Management Development/Installation/Training Initial Biometric Hardware Costs Initial Biometric Software Costs Network Infrastructure Consulate Facility Renovation Hardware Infrastructure Upgrades Total Investment Costs FY2002 FY2003 FY2004 FY2005 FY2006 FY2007 FY2008 FY2009 FY20010-2012 Total
$ $ $ $ $ $ $ $
540 7,900 6,045 4,600 100 570 19,755
$ $ $ $ $ $ $ $
540 7,900 6,045 4,600 100 570 25,229 44,983
Operations & Support Program Management Biometric Hardware Maintenance Software/System Maintenance Network Infrastructure Maint. Visa Operating Personnel INS Operating Personnel Communications Costs Recurring Training Consulate Facility Maintenance Total Operations & Support Costs Total Life Cycle Costs
FY2002 $ $ $ $ $ $ $ $ $ $ $
FY2003 540 926 6,400 100 50,715 10,540 1,000 152 70,373 72,896
$ $ $ $ $ $ $ $ $ $ $
FY2004 540 926 6,400 100 50,715 10,540 1,000 152 70,373 72,896
$ $ $ $ $ $ $ $ $ $ $
FY2005 540 926 6,400 100 50,715 10,540 1,000 152 70,373 72,896
$ $ $ $ $ $ $ $ $ $ $
FY2006 540 926 6,400 100 50,715 10,540 1,000 152 70,373 72,896
$ $ $ $ $ $ $ $ $ $ $
FY2007 540 926 6,400 100 50,715 10,540 1,000 152 70,373 72,896
$ $ $ $ $ $ $ $ $ $ $
FY2008 540 926 6,400 100 50,715 10,540 1,000 152 70,373 72,896
$ $ $ $ $ $ $ $ $ $ $
FY2009 540 926 6,400 100 50,715 10,540 1,000 152 70,373 72,896
$ $ $ $ $ $ $ $ $ $
1,619.44 2,779 19,200 300 152,145 31,619 3,000 456 211,118 218,687
$ $ $ $ $ $ $ $ $ $ $
Total 5,398 9,263 64,000 1,000 540,225 105,396 10,000 1,520 736,802 781,785
• Main Body documentation should include:
– Enough detail for each CES element that would allow for another analyst to replicate the cost estimate.
• Each CES element should provide a description of the methods (with rationale for choosing it), techniques, and data used to generate the estimate as well as any cross-checks done to add rigor to the estimate. • Include information such as data sources, direct/indirect labor rates, labor hours, material/subcontracts, learning curve data and methodology, factors, cost estimating relationships with corresponding statistics and data ranges, cost models used, inflation indices, time phasing (e.g., development lasts 3 years and 10 years operations and support), estimator judgments, cross-checks, risk and confidence levels.
– Documentation flow should follow the same CES structure shown in the estimate summary by year. – If the estimate is an update to a prior estimate, then provide an explanation for any differences.
• Be sure to spell out all acronyms when first introduced. • Cross-reference the reader to where data is used multiple times. • Show data at the lowest levels possible.
– Include copies of vendor quotes, studies used, statistical analysis printouts, cost model input and output reports, etc.
• Include disclaimers to show the level of risk and uncertainty surrounding estimate.
– Try to quantify the uncertainty by using a simulation model that will express the summary estimate in terms of level of confidence:
• Estimate is presented at the 90% confidence level, or • Point estimate of $1 million is bounded by a range of $750,000 to $1,250,000
and usually not funded Development costs for IT systems can quickly become outdated by new programming languages Maintenance costs are even more difficult to capture because they are seen as on-going support or overhead and not as metrics System architecture change effects on cost estimates can be hard to determine • • Validity and uncertainty of data – – – Garbage in = Garbage out Can result in rough-order magnitude costs being used as budget quality estimates Cause important steps like validation and Monte Carlo simulation to be omitted Limited time to develop estimates • Resources – Lack of trained people is a problem 106 . time consuming.Cost Estimating Challenges • Access to historical data – Need to invest in database capture of historical costs and technical data for proper CER development • • • • Costly.
Spring 1999) • http://www. Auditing and Evaluating a CER & Auditing Parametric Estimates • Appendix G.004. if present.edu/publications/documents/95. support the credibility of the estimate.gov/bu2/PCEHHTML/pceh.cmu." • This checklist is designed to help managers access the credibility of software cost and schedule estimates. A few of these are shown below: – The Software Engineering Institute (SEI) has created "A Manager's Checklist for Validating Software Cost and Schedule Estimates.jsc.html – Joint Industry/Government Parametric Estimating Handbook (Second Edition.nasa.Cost Estimating Auditor Checklists • Various auditor checklists can be found in the many source documents used to create this briefing.sr. • It identifies seven issues to address and questions to ask when determining your willingness to accept and use a software estimate. Parametric Estimating System Checklist 107 . http://www.sei.reports/95.htm • Chapter 6. • Each question is associated with elements of evidence that.
ceac.army. The AFSC Cost Estimating Handbook. • Chapter 8.Cost Estimating Auditor Checklists (cont‟d) • Various auditor checklists can be found in the many source documents used to create this briefing. Staff Review Cost Estimate Checklist pages D-1 through D-8 108 . Army Cost and Economic Analysis Center.army. May 2002) • http://www.asp • Link to a zipped MS Word file for downloading or printing • Checklist on page 42. Army Cost and Economic Analysis Center.S.S. Reading. prepared for the U. Feb 2001) • http://www.S.). A few of these are shown below: – Department of the Army Economic Analysis Manual (U. Economic Analysis Checklist – Department of the Army Cost Analysis Manual (U.asafm. 1986.mil/pubs/cdfs/manual/economic. Cost Models pages 8-6 through 8-12 • Appendix D. MA.mil/pubs/default.pdf • Appendix M. Figure 3-4 Validation Considerations – TASC (The Analytic Sciences Corp. AirForce Systems Command.
Software Cost Estimation Questionnaire 109 .Cost Estimating Auditor Checklists (cont‟d) • Various auditor checklists can be found in the many source documents used to create this briefing.nps. pages 59-67.ca/English/abstracts/NRC37116.iit. Guide for Reviewers – National Research Council Canada (NRC-CNRC) Software Cost Estimation and Control (1994) • http://seg.nrc.html • Appendix A.navy.mil/drmi/chapter6.htm • Chapter 6. A few of these are shown below: – Navy Post Graduate School Economic Analysis Handbook • http://www.
Contact Information • For further information please contact us: – Karen Richey (202) 512 – 4784 or email@example.com – Madhav Panwar (202) 512.gov 110 .6228 or firstname.lastname@example.org – Jennifer Echard (202) 512 – 3875 or echardj@gao.
This action might not be possible to undo. Are you sure you want to continue?