This action might not be possible to undo. Are you sure you want to continue?
Auburn University 3101 Shelby Center Auburn, AL 36849-5347 1-(334) 844-4330
Auburn University 3101D Shelby Center Auburn, AL 36849-5347 1-(334) 844-6360
Increasingly many software development organizations must pursue at least a level 3 CMMI certification to compete in the marketplace. Not all software organizations are under this constraint, but the CMMI model offers benefit beyond just a marketing advantage. Agile methods have taken hold in many of the same organizations that stand to gain benefit from CMMI. CMMI adds some stress to the Agile approach by nature of more detailed planning and tracking. Microsoft and other software development organizations have endeavored to automate the more tedious aspects of CMMI, thus making it easier to support agile approaches within more rigid frameworks. The Microsoft Solutions Framework (MSF) initially founded on Microsoft software best practices observed internally and externally has expanded to integrate automated software configuration for team projects. The Team Foundation Server (TFS) product leverages MSF and supports both CMMI and Agile as process templates and provides tools around these templates to automate the use of either approach. In this paper, we explore how Microsoft has integrated these approaches into TFS, the rationale and implications for selecting one approach over another and how agile practices can be integrated into a CMMI framework.
Hamilton@auburn.edu 1. INTRODUCTION
Software development firms face new challenges competing in the global market place. On the one hand, there is pressure to achieve quick return on investment (ROI) through rapidly deployed software solutions. On the other hand, there is pressure to conform to increasingly complex functional requirements while retaining acceptable quality. Increasingly, Capability Maturity Model Integration (CMMI) is becoming an important model for organizations to utilize in order to compete in the market place. Not all firms need to achieve a CMMI compliance level to compete effectively. Agile development methods have proven especially suited for situations where a rapid solution is needed while CMMI is typically utilized for longer/more complex projects that require more rigors and can afford the extra overhead of CMMI. The CMMI/Agile tradeoff begs several questions – “At what point is a project too big or too critical for Agile or too small for CMMI?” What are the key deciding points, particularly as it pertains to the decision facing Information Technology (IT) professionals using the Microsoft Team Foundation Server Product (TFS), which requires the selection of one of these templates in order to start a new project? Does the selection of one approach preclude the use of the other? An important aspect in studying each approach lies with understanding that much of the value of the processes depends on the effectiveness of tools to automate more cumbersome aspects. The Microsoft Solutions Framework (MSF) prescribes templates to help with both approaches. TFS automates the use of these templates in the context of the development process. The purpose of this paper is not to extol Microsoft’s toolset but to review MSF and TFS in sufficient depth to understand how it assists in both processes. From the TFS view, Agile and CMMI are defined as process templates. For our discussion, we consider CMMI and Agile selection scenarios, implications for selecting one process template over another and integration approaches.
Categories and Subject Descriptors
D.2.9 [Management]: Software configuration management, Software process models.
Management, Measurement, Documentation,
Team development, Microsoft Solutions Framework, Agile, Capability Maturity Model Integration, Agile, CMMI, Process Templates, Development Methodologies.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ACM-SE ‘08, March 28–29, 2008, Auburn, AL, USA. Copyright 2008 ACM ISBN 978-1-60558-105-7/08/03…$5.00.
2. LITERATURE REVIEW 2.1 Agile
Agile represents a paradigm shift away from formal processdriven software development . One of the formative drivers that compelled the manifesto founders were studies that demonstrated high failure rates associated with large disciplined software development efforts. It emphasizes rapid return on investment and builds trust in the development process. The Agile Manifesto web site  outlines the following four
However. Agile insists that software development processes must be flexible or else the processes will break resulting in dissatisfied customers. The KPAs are intended to help gauge where an organization is at in the maturation process as well as provide guidance for how to achieve the desired level. These methodologies differ in technique. and subcontract management. In the case of CMMI. and an organizational process focus and definition. straightforward. Level 4: Managed . It can be used to guide process improvement across a project. CMMI’s approach is based on maturity models.principles that highlight its stand against traditional software development discipline: 1. Agile recognizes the reality that requirements for software solutions. CMMI is concerned with defining metrics and practices to ensure continuous improvement . Universal versus situational solutions: More work on how to adopt agile methods in different development situations is needed. CMMI evokes fear in some because of the processes involved in certification. The below summarize the current maturity levels associated with CMMI: Level 0: No Process Level 1: Initial – ad-hoc processes – frequently exceeding budget and schedule and unable to repeat past successes. software product engineering. Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan essential elements of effective processes. Abstract versus concrete guidance: Emphasis is needed to provide guidance to utilize suggestions rather than abstract principles only. Abrahamsson  identifies shortcomings from the following perspectives: Software development life cycle: Life cycle coverage needs to be explained and interfaces with phases not covered need to be clarified. training programs. This level includes integrated software management. Level 2: Repeatable – some project management to track schedule/cost with some repeatability but with high risk for exceeding cost and time estimates. 4. CMMI does not prescribe specific software development approaches or specific organizational structures. CMMI helps integrate traditionally separate organizational functions. These include Iterative software development. provide guidance for quality processes. and procedures are in place including peer reviews. inter-group coordination. results need to be published. collaborative arrangements with customers and developers working closely together. Because of CMMI’s emphasis on disciplined planning and organizational structure and because of the sheer size and scope of CMMI. it recognizes the bolded values on the left greater. software quality assurance. continuous verification. Scrum . 3. and adaptive – able to make last moment changes . CMMI at its core is not a methodology but rather a set of principles. particularly those that involve custom-developed solutions are not easy to envision. CMMI associates skill in different process areas with higher maturity levels.Quantitative process management and software quality management are in place. However. Level 5: Optimizing . The hallmarks of an Agile approach include incremental development consisting of small software releases with rapid cycles. many misunderstand CMMI as a traditional waterfall software approach that requires large hierarchical bureaucratic organizational structure. set process improvement goals and priorities. but to enable increasing improvement in an organization’s processes.Standards. This is a different emphasis than Agile. To be CMMI compliant at any level requires rigorous inspection. This level includes software configuration management.Defect prevention. the set of principles focuses on maturation of a software development process. as is also the case with Agile.2 CMMI CMMI is summarized on the Carnegie Mellon website as follows: “Capability Maturity Model® Integration (CMMI) is a process improvement approach that provides organizations with the 187 . Project management: Harmonization is needed to support project management activities. Achieving CMMI compliance is greatly facilitated when there are tools in place to assist in the processes. and Adaptive Software Development . CMMI is compatible with a variety of While the manifesto recognizes value in the non-bold items on the right. a division. technology change management. This involves the use of the Standard CMMI Appraisal Method for Process Improvement (SCAMPI) and often involves over 150 hours for typical organizations. process description. Level 3: Defined . but encapsulate many common recommendations that contribute to higher software quality . and provide a point of reference for appraising current processes” . The goal of CMMI is not just to support a minimum set of standards to achieve to a particular level. 2. 2. CMMI is more focused on metrics than Agile. Agile stands in contrast to CMMI and other more disciplined software processes in that it rejects many traditional viewpoints regarding detailed planning and requirements definition. architecture driven design and focus on teams Agile critics lament the lack of project management organization and point out how the process can break down. Feature Driven Development . particularly on larger projects and during the maintenance phases. customer-oriented requirements. The use of tools in and of itself is one of the requirements for achieving higher levels of CMMI. It supports both a staged approach and a continuous model for improvement. Empirical support: More empirical. situation-specific experimental work is needed. and process change management are in place. including Extreme Programming (XP) . or an entire organization. This approach is motivated by the reality that requirements change and customers do not always know what they need until they see it being produced. It provides several key process areas (KPA) at different levels. Agile more so represents a set of principles for software development than a specific methodology. easy to learn methods. Agile principles are evident in many different methodologies.
The CMMI process template is thus a super-set rather than a different set of features as illustrated in figure 1. Much of the basis for CMMI stems from Deming’s work. but embodies other roles including architects. However. Instead focus on quality assurance throughout the lifecycle. CMMI process template (Source: Randy Miller. the latest version of MSF V4. 2.0 adds additional levels of traceability and feedback for further improvement. It is based on Microsoft’s experience both inside the corporation and from the field through consulting work. It also provides an infrastructure for supporting CMMI goals by providing templates for deliverables. The CMMI template is thus an addition to the Agile framework inherent in MSF. a scalable enterprise TFS implementation requires different server roles including web as well as database and application servers. TFS leverages several different Microsoft products including SQL Server 2005 database.software development approaches in any type of organization. but rather as a framework that can support different methodologies. do the benefits of CMMI outweigh the costs. Choose the MSF for CMMI Process Improvement process for projects with longer life cycles and that require a record of decisions made…” Reading this. MSF is a deliberate and disciplined approach to technology projects based on a defined set of principles. how should the project 188 . which contains tools focused mainly on the application developer.0 mainly focused on the stages involved in software development and emphasized a spiral waterfall that was intended to be agile and ensured customer trust through rapid delivery and verification with agility.3 MSF Microsoft Solutions Framework (MSF) encompasses Microsoft’s software development principles and experience. At what point. there are advantages of CMMI not included in Agile. In this way. 6) Training on the job 7) Leadership 8) Drive out fear. and database administrators. concepts. as well as automated metric collection and reporting features available from the TFS portal. if your organization is undertaking a broad quality assurance and process improvement initiative or your team needs the assistance of explicit process guidance rather than relying on tacit knowledge and experience. this is not the intent for the two different models. The TFS product provides some guidance in the help: “Choose the MSF for Agile Software Development process for projects with short lifecycles and delivery-oriented teams who can work with lots of intermediate documentation.Agile vs. Microsoft Tech Ed 2006 Presentation DEV439) Anderson’s explanation relies heavily on the Deming feedback model where organizational improvement is driven by continuous statistical feedback . MSF historically has been very closely aligned with agile practices. centralized project management. It integrates software development process with quality assurance through rules-based configuration management. rather than a replacement. and proven practices . Of Deming’s 14 points for management. guidance documentation. The fundamental construct behind MSF 4. MSF for Agile Software Development is an iterative. Given the background on CMMI and Agile. quality control work flow and serves as a central repository and collaboration portal for a software development team.1 Differences between CMMI and Agile TFS Process Templates One of the first decisions the project leader must make when creating a new TFS project is what process template to use. TFS provides the core infrastructure for software configuration management. specifically in the context of a TFS project. manager select a template if the organization is not mandated to follow CMMI? Even if an organization formally does not need to embrace the CMMI. disciplines. is the support of agile methods while providing a framework for CMMI. The TFS infrastructure is pivotal in Microsoft Team development approach and is heavily leveraged by an increasing number of customers and partners as well as internally at Microsoft. and reporting services as well as Microsoft SharePoint Services for collaboration. utilized by both process templates. These and other myths of CMMI are espoused upon at the Carnegie Mellon website . MSF now supports many of the concepts embedded in CMMI. It provides testing automation services. Functional/Exploratory Testing Iteration Planning Test Driven Development Context-Driven Test Shadow Architecture MSF for Agile Software Development Change Requests Process Improvement Activities Formal Reviews Approvals MSF for CMMI Process Improvement Requirements Analysis Figure 1. scenario-driven development process… “ “Choose MSF for CMMI Process Improvement over MSF for Agile Software Development. there are 7 points that particularly highlight the benefits of an agile approach within a continuous improvement framework: 3) Cease dependence on quality control to achieve quality.0. guidelines. Anderson explains this in detail in his account of the how the CMMI Process Improvement process template was developed at Microsoft . one might think that choosing the CMMI model forfeits the capability to utilize agile methods. MSF 3. 2.4 Team Foundation Server Team Foundation Server (TFS) is part of the Microsoft Visual Studio 2005 Team System . Although it is possible to implement TFS on a single server. 4) Build trust and loyalty with suppliers. data analysis. However. TFS provides configuration management of both code and documentation artifacts. and collaborative and disciplined development for Microsoft team projects. testers. 3. models. MSF defines itself not as a methodology. It has evolved through various stages. The Team Suite development includes the Visual Studio Team Suite Interactive Development Environment (IDE). DISCUSSION 3.
The decision becomes one mainly of determining if the extra items associated with the CMMI template are beneficial for the organization or the software project. fixing. measurement. organizational training.” Project Monitoring and Control (PMC) 1. organizational process focus. Given that agile methods can be incorporated into a CMMI approach. this simplifies the selection basis. the number of bugs per work item and iteration are tracked through the automated test feature. management on quality rather than production numbers. risk management. but includes aspects from other levels. interface requirements. Bug Rates: Finding. Microsoft chose to target a CMMI level 3 for the 2005 version. An example report is illustrated in figure 2. 12) Remove barriers to pride of workmanship. quantitative project management and causal analysis and resolution from levels 4 and 5 are partially addressed. tasks. TFS supports extending the process template to add additional items as well as supplementing sites built from the template with additional document or site templates.1 sub-practice 2 asks to “Identify the work packages in sufficient detail as to specify estimates of project. The state of the project is a reflection of the states of the work items. product component descriptions. Through reporting features. 189 . The metrics gathered by TFS and available for both ad-hoc and standard reporting regardless of template selection include the following . requirements definition. 17 of the 21 Level 3 areas are targeted including project planning. However.” Technical Solution (TS) 2. regression testing verification. the use of CMMI would not be recommended if the organization already has a process template in place that has proven successful. Utilizing CMMI within the MSF TFS context is mainly a “stretch-to-fit” choice.1 asks to “identify task dependencies” and expects a “Critical Path Method (CPM) or Program Evaluation and Review Technique (PERT)” chart as a result. allocated requirements. conditions of use”. Not all of the CMMI levels are automated within the CMMI process template. product characteristics. Focus through the issues or are more issues materializing as we develop”? Issues List: What are the issues and how much time have they used and how much time are the expected to continue to consume? Utilizing the CMMI process in TFS pre-creates numerous documents and work item types associated with several KPAs. document review. some level 4 and level 5 items including supplier agreement management. An adapter framework is used to gather the data for the reports as well as store in a data warehouse for historical reporting. monitoring.9) Break down barriers between departments. it is important to recognize that some aspects of the CMMI requirements are not agile. at least in the context of the TFS scenario. responsibilities and schedule” and goes on with “The amount of detail in the WBS at this more detailed level helps in developing realistic schedules. configuration management.1 asks to compare “actual completion of activities and milestones against the schedule documented in the project plan. In addition. a term coined by Alistair Cockburn . and bug resolution are couched in the linkage of work items to related tasks. Although an agile mindset is accommodated within the TFS CMMI process template. PP 1. The work items are centrally manageable and become part of the data warehouse used to report project status. and closing bugs as well as bug trends – answers questions such as “Is the number of new defects increasing slower than the rate at which the defects are being corrected”? “How long is it taking to resolve defects”? Builds: Is code coverage being achieved? Are the validation tests passing? Is “code churn”  occurring wherein modules are continually being modified without any new tests being defined? Project Velocity: The rate at which the team is processing and closing work items – answers the question “Are we progressing Figure 2. control. An important measure derived from monitoring work item state is project velocity. Understanding the costs of using a CMMI versus an agile approach in a TFS scenario requires understanding TFS product capabilities. identifying significant deviations from the schedule estimates in the project plan. organizational environment for integration. and integrated supplier management are not provided in the 2005 version. Capabilities to enforce automated system builds. thereby minimizing the need for management reserve. and requirements validation. The main areas of the CMMI that require more rigidity are : Project Planning (PP) 2.2 asks to “establish a technical data package” and suggest artifacts like “product architecture description. Much of the overhead associated with a CMMI process is automated through the product. For example. The Microsoft team development model revolves around work items.Sample TFS report The TFS infrastructure provides work item tracking. the rate at which the team is processing and closing work items. Guckenheimer and Perez  describe this metric as a key indicator for estimating the completion date for the next iteration.
the literature should improve in quantity and quality. The focus is on delivering a solution that meets key customer needs and building upon success rather than trying to produce intermediate deliverables aimed at a long-term specification that may or may not prove accurate given the rate of technology and business changes which frequently occur even with short life cycles. Many organizations that moved to an agile approach now find themselves trying to conform to a CMMI approach. we have defined a flowchart to assist in formulating the decision to utilize the CMMI template (figure 4). Taking into consideration the automation provided by TFS and that the CMMI process template incorporates agile benefits.Process template decision-making flowchart 190 . whether choosing an agile or CMMI focused approach provides numerous benefits and improves the efficiency of development.2 TFS Decision Methodology Utilizing a tools-based approach such as TFS for automating a software development process. In a value-up paradigm.by the MSF for CMMI process template. In the CMMI process template. DTE Energy describes their viewpoint that CMMI is a framework of “what” while their agile methods provide an empowering philosophy of “how”. The theory is that prescriptive metrics become inaccurate when an organization’s places an undue emphasis on metrics. In the TFS environment. Many of these metrics are required for not only the CMMI level 3 targeted Yes Use Agile Process Template as Base Estimate more than X weeks Customize Agile Framework to match existing approach Use CMMI Process Template No More than Y Function Points No More than Z Team Members No Use Agile Process Template Figure 4 . There are 9 metric charts with Agile but 12 with CMMI. but also for higher CMMI levels. One of the underlying assumptions with MSF is that metrics should be descriptive rather than prescriptive. One organization that made the transition and published their results and lessons learned is DTE Energy . there are 59 work product artifacts while Agile has 25. we provide a matrix (table 1) that outlines the key issues involved in the decision along with a prioritization column to auto-calculate the best decision path. This is based on Robert Austin’s work “Measuring and Managing Performance in Organizations” . Our decision-making process emphasizes the role of the business environment over technology issues. metrics are gathered that describe what is occurring at the code development and testing levels. Agile decision Factor Traceability Quick ROI Auditing Resource shortage Time shortage Requirements known in advance CMMI + + + Agile + + + Weight Figure 3 . In addition.Weighting matrix for CMMI vs. No CMMI Mandated Yes Existing Agile Approach? No Estimated Effort above W $ No Yes Detailed auditing/ traceability No 3. As the number of organizations performing this transition increases.TFS Work Item State Transitions (Source: Brian Harry. the value of the product is realized through iterations and improvement rather than being dictated by a strict specification up-front. choosing to use the CMMI model will put more stress on the project team due to the sheer number of deliverables that are required. Although the TFS tooling helps in many areas. Table 1 . Microsoft Tech Ed 2006 Presentation DEV327) Guckenheimer and Perez also discuss the benefit of a value-up paradigm shift. through the data warehouse capabilities and the ongoing capture of information during the check-in and build processes. Anderson  estimates that 85 % of the overhead is resolved by automation and that the overall amount of effort required for a CMMI template is only 150% beyond that of Agile.
29. CMMI is a maturation model.. D.edu/cmmi/general/general. 6.us/index. In Proceedings of the 2004 ACM Workshop on interdisciplinary Software Engineering Research (Newport Beach.com/technet/solutionaccelerators/msf/d efault. This approach can be used to help automate continuous improvement regardless of the CMMI level targeted. Using a tool such as TFS that provides configuration management. Upper Saddle River. J. CONCLUSION A comparison of Agile and CMMI is not truly an “Apples-toapples” comparison.org/10. and metrics gathering facilitates a CMMI approach that still supports agility.. 2006.  Conboy. DOI= http://dx. 2006. May 15 . DOI= http://doi.pdf on December 26.. 2006.. Customization is achieved by modifying the variables and weighting used in the trade-off decisions. 2004). Auburn University for insights on improving software process efficiency and encouragement with publication of this paper and Dr. 1999. Washington. New York: Dorset House  Holland.the story of creating MSF for CMMI Process Improvement at Microsoft Corporation.com/enus/teamsystem/aa718825. 193-201. 2006. 1982.  Schwaber. Washington. 284292. 2001. MA.  Carnegie Mellon Software Engineering Institute. ICSE '05. A tools-based approach ensures objective metric collection and reduces cost by focusing resources on actual development and project activities. 2006. ADC. NY. record-keeping and other tasks better done by automated tools.  Nagappan. Adaptive Software Development: A Collaborative Approach to Managing Complex Systems. Auburn University for elaborations of CMMI and software quality.  Highsmith.aspx on December 26. 2005.php/Crystal_methodologies_ main_foyer on December 26. Visual Studio 2005 Team Foundation Server. J. 2006. and Felsing. May 17 . S. May 03 .1062514  Palmer. Salo.4. A. November 05 .2 Overview. 2006.htm on December 26. IEEE Computer Society.acm.doi. Retrieved from http://www.vtt. New York. The MIT Press. Microsoft Visual Studio 2005 Team Foundation Server.mspx on December 26. and Fitzgerald. 2002. New York.sei. while Agile is a development philosophy. 3-WoSQ. Ronkainen. J. DOI= http://doi.microsoft. J. Toward a conceptual framework of agile methods: a study of agility in different disciplines. 2000. B. Agile Software Development Methods: Review and Analysis.acm.syscon.sei. Measuring and Managing Performance in Organizations. Ptentice-Hall. Proceedings.  Austin. 1996. T. ACM Press. 2005. Oregon. WISER '04. 2006. OOPSLA’95 Workshop on Business Object Design and Implementation. 2006. 2000. T. A Practical Guide to Feature Driven Development.html on December 26.1145/1029997. DOI= http://doi. 2006.  Abrahamsson. In Proceedings of the 25th international Conference on Software Engineering (Portland.  Baker.42  Beck.acm. Agile Conference. Retrieved from http://www. 1995. Upper Saddle River. Scrum Development Process. Crystal Main Foyer. 37-44. 2006. NY. 2005).org/10. New directions on agile methods: a comparative analysis. In Proceedings of the Third Workshop on Software Quality (St. What is CMMI. 2005. Springer-Verlag.  Guckenheimer. P. 2005). DC. Software quality development and assurance in RUP. VTT Publications 478. New York. N. IEEE Computer Society.cmu. 2006. D. Retrieved from http://www.inf. Perez. and Ball. Therefore. R.com/read/231944.. 2005.org/10. 2006.org/ on December 26. Dorset House Publishing. Software Engineering with Microsoft Visual Studio Team System. USA. 244-254.fi/pdf/publications/2002/P478. Manifesto for Agile Software Development retrieved from http://agilemanifesto.21.microsoft. 2002.10.. 2003.  Microsoft. We have provided a flowchart and matrix customizable for an organization to assist in the template selection. 2004. and Grechenig. NJ. Louis. MO.1145/1083292. Retrieved from http://alistair. Siponen.  Carnegie Mellon Software Engineering Institute. Heil. Formalizing agility: an agile organization's journey toward CMMI accreditation. The distinction is subtle but significant because it means that the approaches can potentially complement each other rather than being mired in incompatibility. W. T. AddisonWesley. Capability Maturity Model Integration (CMMI) Version 1. Stretching Agile to fit CMMI Level 3 . Out of Crisis. automated testing. Retrieved from http://www. M. J. 2006. S. 2005.  Anderson.1109/ADC.edu/cmmi/adoption/pdf/cmmioverview06. the choice of one or another as a process template need not be a sacrificial proposition. S. REFERENCES  Abrahamsson. Cambridge.  Microsoft. DC.cockburn. ACM Press. Retrieved from http://dotnet. Warsta. In Proceedings of the 27th international Conference on Software Engineering (St. NJ.17. Retrieved from http://msdn2. Kai Chang.  The Agile Alliance. 1-6. Wastra. W.  Cockburn . TFS provides both Agile and CMMI process templates that integrate with Microsoft Team Suite core capabilities. Use of relative code churn measures to predict system defect density. Addison-Wesley.2005. S. 191 .1145/1062455. AKNOWLEDGEMENTS Thanks to Dr. USA.05. It is possible to leverage agile practices in a CMMI context.cmu. rather than on enforcement.1083300 5. K.  Zuser. In Proceedings of the Agile Development Conference (July 24 . 2003).pdf on December 26. 2006. MSF and XP: a comparative study. Microsoft Solutions Framework. CA. 2005). International Conference on Software Engineering.org/10. David Umphress. and Ronkainen. Extreme Programming Explained: Embrace Change.1030005  Deming. Missouri. Louis. Q/A checking.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.