Agile Versus CMMI - Process Template Selection and Integration with Microsoft Team Foundation Server

Robert Leithiser
Auburn University 3101 Shelby Center Auburn, AL 36849-5347 1-(334) 844-4330

Drew Hamilton
Auburn University 3101D Shelby Center Auburn, AL 36849-5347 1-(334) 844-6360

leithrm@auburn.edu ABSTRACT
Increasingly many software development organizations must pursue at least a level 3 CMMI certification to compete in the marketplace. Not all software organizations are under this constraint, but the CMMI model offers benefit beyond just a marketing advantage. Agile methods have taken hold in many of the same organizations that stand to gain benefit from CMMI. CMMI adds some stress to the Agile approach by nature of more detailed planning and tracking. Microsoft and other software development organizations have endeavored to automate the more tedious aspects of CMMI, thus making it easier to support agile approaches within more rigid frameworks. The Microsoft Solutions Framework (MSF) initially founded on Microsoft software best practices observed internally and externally has expanded to integrate automated software configuration for team projects. The Team Foundation Server (TFS) product leverages MSF and supports both CMMI and Agile as process templates and provides tools around these templates to automate the use of either approach. In this paper, we explore how Microsoft has integrated these approaches into TFS, the rationale and implications for selecting one approach over another and how agile practices can be integrated into a CMMI framework.

Hamilton@auburn.edu 1. INTRODUCTION
Software development firms face new challenges competing in the global market place. On the one hand, there is pressure to achieve quick return on investment (ROI) through rapidly deployed software solutions. On the other hand, there is pressure to conform to increasingly complex functional requirements while retaining acceptable quality. Increasingly, Capability Maturity Model Integration (CMMI) is becoming an important model for organizations to utilize in order to compete in the market place. Not all firms need to achieve a CMMI compliance level to compete effectively. Agile development methods have proven especially suited for situations where a rapid solution is needed while CMMI is typically utilized for longer/more complex projects that require more rigors and can afford the extra overhead of CMMI. The CMMI/Agile tradeoff begs several questions – “At what point is a project too big or too critical for Agile or too small for CMMI?” What are the key deciding points, particularly as it pertains to the decision facing Information Technology (IT) professionals using the Microsoft Team Foundation Server Product (TFS), which requires the selection of one of these templates in order to start a new project? Does the selection of one approach preclude the use of the other? An important aspect in studying each approach lies with understanding that much of the value of the processes depends on the effectiveness of tools to automate more cumbersome aspects. The Microsoft Solutions Framework (MSF) prescribes templates to help with both approaches. TFS automates the use of these templates in the context of the development process. The purpose of this paper is not to extol Microsoft’s toolset but to review MSF and TFS in sufficient depth to understand how it assists in both processes. From the TFS view, Agile and CMMI are defined as process templates. For our discussion, we consider CMMI and Agile selection scenarios, implications for selecting one process template over another and integration approaches.

Categories and Subject Descriptors
D.2.9 [Management]: Software configuration management, Software process models.

General Terms
Management, Measurement, Documentation,

Keywords
Team development, Microsoft Solutions Framework, Agile, Capability Maturity Model Integration, Agile, CMMI, Process Templates, Development Methodologies.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ACM-SE ‘08, March 28–29, 2008, Auburn, AL, USA. Copyright 2008 ACM ISBN 978-1-60558-105-7/08/03…$5.00.

2. LITERATURE REVIEW 2.1 Agile
Agile represents a paradigm shift away from formal processdriven software development [11]. One of the formative drivers that compelled the manifesto founders were studies that demonstrated high failure rates associated with large disciplined software development efforts. It emphasizes rapid return on investment and builds trust in the development process. The Agile Manifesto web site [3] outlines the following four

186

Quantitative process management and software quality management are in place. Achieving CMMI compliance is greatly facilitated when there are tools in place to assist in the processes. These methodologies differ in technique. Abrahamsson [2] identifies shortcomings from the following perspectives: Software development life cycle: Life cycle coverage needs to be explained and interfaces with phases not covered need to be clarified. particularly on larger projects and during the maintenance phases. particularly those that involve custom-developed solutions are not easy to envision.principles that highlight its stand against traditional software development discipline: 1. CMMI does not prescribe specific software development approaches or specific organizational structures. the set of principles focuses on maturation of a software development process. inter-group coordination. In the case of CMMI. process description. 4. Agile insists that software development processes must be flexible or else the processes will break resulting in dissatisfied customers. provide guidance for quality processes. a division. Agile recognizes the reality that requirements for software solutions. Scrum [20]. The KPAs are intended to help gauge where an organization is at in the maturation process as well as provide guidance for how to achieve the desired level. 3. This level includes integrated software management. Empirical support: More empirical. but to enable increasing improvement in an organization’s processes. Agile principles are evident in many different methodologies. and an organizational process focus and definition. Abstract versus concrete guidance: Emphasis is needed to provide guidance to utilize suggestions rather than abstract principles only. easy to learn methods. It can be used to guide process improvement across a project. The hallmarks of an Agile approach include incremental development consisting of small software releases with rapid cycles. This involves the use of the Standard CMMI Appraisal Method for Process Improvement (SCAMPI) and often involves over 150 hours for typical organizations. Level 2: Repeatable – some project management to track schedule/cost with some repeatability but with high risk for exceeding cost and time estimates. These include Iterative software development. Project management: Harmonization is needed to support project management activities. CMMI helps integrate traditionally separate organizational functions. set process improvement goals and priorities. The below summarize the current maturity levels associated with CMMI: Level 0: No Process Level 1: Initial – ad-hoc processes – frequently exceeding budget and schedule and unable to repeat past successes. results need to be published. continuous verification. Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan essential elements of effective processes. collaborative arrangements with customers and developers working closely together. and adaptive – able to make last moment changes [1]. CMMI at its core is not a methodology but rather a set of principles. To be CMMI compliant at any level requires rigorous inspection. This approach is motivated by the reality that requirements change and customers do not always know what they need until they see it being produced. CMMI evokes fear in some because of the processes involved in certification.2 CMMI CMMI is summarized on the Carnegie Mellon website as follows: “Capability Maturity Model® Integration (CMMI) is a process improvement approach that provides organizations with the 187 . However. CMMI associates skill in different process areas with higher maturity levels. or an entire organization. Agile more so represents a set of principles for software development than a specific methodology. CMMI’s approach is based on maturity models. Level 4: Managed . and Adaptive Software Development [14]. Level 3: Defined . CMMI is compatible with a variety of While the manifesto recognizes value in the non-bold items on the right. architecture driven design and focus on teams Agile critics lament the lack of project management organization and point out how the process can break down. It supports both a staged approach and a continuous model for improvement. Universal versus situational solutions: More work on how to adopt agile methods in different development situations is needed. and procedures are in place including peer reviews. including Extreme Programming (XP) [6]. training programs.Standards. This is a different emphasis than Agile. and process change management are in place. Because of CMMI’s emphasis on disciplined planning and organizational structure and because of the sheer size and scope of CMMI. The use of tools in and of itself is one of the requirements for achieving higher levels of CMMI. It provides several key process areas (KPA) at different levels. Level 5: Optimizing . This level includes software configuration management. The goal of CMMI is not just to support a minimum set of standards to achieve to a particular level. and subcontract management. and provide a point of reference for appraising current processes” [8].Defect prevention. but encapsulate many common recommendations that contribute to higher software quality [21]. 2. customer-oriented requirements. software product engineering. However. Feature Driven Development [19]. software quality assurance. it recognizes the bolded values on the left greater. CMMI is concerned with defining metrics and practices to ensure continuous improvement [9]. 2. as is also the case with Agile. situation-specific experimental work is needed. many misunderstand CMMI as a traditional waterfall software approach that requires large hierarchical bureaucratic organizational structure. technology change management. Agile stands in contrast to CMMI and other more disciplined software processes in that it rejects many traditional viewpoints regarding detailed planning and requirements definition. CMMI is more focused on metrics than Agile. straightforward.

TFS provides the core infrastructure for software configuration management. do the benefits of CMMI outweigh the costs. the latest version of MSF V4. disciplines. and database administrators. MSF for Agile Software Development is an iterative. These and other myths of CMMI are espoused upon at the Carnegie Mellon website [8]. this is not the intent for the two different models. 4) Build trust and loyalty with suppliers. At what point. one might think that choosing the CMMI model forfeits the capability to utilize agile methods. Although it is possible to implement TFS on a single server. a scalable enterprise TFS implementation requires different server roles including web as well as database and application servers. quality control work flow and serves as a central repository and collaboration portal for a software development team. Given the background on CMMI and Agile. MSF is a deliberate and disciplined approach to technology projects based on a defined set of principles.0 adds additional levels of traceability and feedback for further improvement. CMMI process template (Source: Randy Miller. and proven practices [16]. if your organization is undertaking a broad quality assurance and process improvement initiative or your team needs the assistance of explicit process guidance rather than relying on tacit knowledge and experience. guidance documentation. which contains tools focused mainly on the application developer. It also provides an infrastructure for supporting CMMI goals by providing templates for deliverables. 2. models. and collaborative and disciplined development for Microsoft team projects. Functional/Exploratory Testing Iteration Planning Test Driven Development Context-Driven Test Shadow Architecture MSF for Agile Software Development Change Requests Process Improvement Activities Formal Reviews Approvals MSF for CMMI Process Improvement Requirements Analysis Figure 1. scenario-driven development process… “ “Choose MSF for CMMI Process Improvement over MSF for Agile Software Development. DISCUSSION 3. MSF historically has been very closely aligned with agile practices. The TFS product provides some guidance in the help: “Choose the MSF for Agile Software Development process for projects with short lifecycles and delivery-oriented teams who can work with lots of intermediate documentation. there are 7 points that particularly highlight the benefits of an agile approach within a continuous improvement framework: 3) Cease dependence on quality control to achieve quality. Instead focus on quality assurance throughout the lifecycle. However. 3.0 mainly focused on the stages involved in software development and emphasized a spiral waterfall that was intended to be agile and ensured customer trust through rapid delivery and verification with agility. It integrates software development process with quality assurance through rules-based configuration management. The CMMI template is thus an addition to the Agile framework inherent in MSF. and reporting services as well as Microsoft SharePoint Services for collaboration. is the support of agile methods while providing a framework for CMMI. The CMMI process template is thus a super-set rather than a different set of features as illustrated in figure 1. The TFS infrastructure is pivotal in Microsoft Team development approach and is heavily leveraged by an increasing number of customers and partners as well as internally at Microsoft. there are advantages of CMMI not included in Agile. 2. It is based on Microsoft’s experience both inside the corporation and from the field through consulting work. but embodies other roles including architects. Microsoft Tech Ed 2006 Presentation DEV439) Anderson’s explanation relies heavily on the Deming feedback model where organizational improvement is driven by continuous statistical feedback [12]. testers. but rather as a framework that can support different methodologies. as well as automated metric collection and reporting features available from the TFS portal.software development approaches in any type of organization. guidelines. However. Much of the basis for CMMI stems from Deming’s work. MSF now supports many of the concepts embedded in CMMI. Anderson explains this in detail in his account of the how the CMMI Process Improvement process template was developed at Microsoft [5]. TFS leverages several different Microsoft products including SQL Server 2005 database. utilized by both process templates. MSF defines itself not as a methodology. TFS provides configuration management of both code and documentation artifacts. 6) Training on the job 7) Leadership 8) Drive out fear. rather than a replacement.Agile vs.1 Differences between CMMI and Agile TFS Process Templates One of the first decisions the project leader must make when creating a new TFS project is what process template to use.3 MSF Microsoft Solutions Framework (MSF) encompasses Microsoft’s software development principles and experience. specifically in the context of a TFS project. how should the project 188 .0. Choose the MSF for CMMI Process Improvement process for projects with longer life cycles and that require a record of decisions made…” Reading this. manager select a template if the organization is not mandated to follow CMMI? Even if an organization formally does not need to embrace the CMMI. It has evolved through various stages. MSF 3. Of Deming’s 14 points for management. centralized project management. data analysis. The Team Suite development includes the Visual Studio Team Suite Interactive Development Environment (IDE). It provides testing automation services. In this way. concepts. The fundamental construct behind MSF 4.4 Team Foundation Server Team Foundation Server (TFS) is part of the Microsoft Visual Studio 2005 Team System [17].

9) Break down barriers between departments. the rate at which the team is processing and closing work items. interface requirements. tasks. Although an agile mindset is accommodated within the TFS CMMI process template. quantitative project management and causal analysis and resolution from levels 4 and 5 are partially addressed. An important measure derived from monitoring work item state is project velocity. measurement. the number of bugs per work item and iteration are tracked through the automated test feature.1 asks to compare “actual completion of activities and milestones against the schedule documented in the project plan. 12) Remove barriers to pride of workmanship. risk management.” Technical Solution (TS) 2. control. The work items are centrally manageable and become part of the data warehouse used to report project status. An adapter framework is used to gather the data for the reports as well as store in a data warehouse for historical reporting. 189 . and requirements validation.2 asks to “establish a technical data package” and suggest artifacts like “product architecture description. identifying significant deviations from the schedule estimates in the project plan. a term coined by Alistair Cockburn [10]. organizational process focus. Microsoft chose to target a CMMI level 3 for the 2005 version. Understanding the costs of using a CMMI versus an agile approach in a TFS scenario requires understanding TFS product capabilities. some level 4 and level 5 items including supplier agreement management. The state of the project is a reflection of the states of the work items. Not all of the CMMI levels are automated within the CMMI process template. the use of CMMI would not be recommended if the organization already has a process template in place that has proven successful. TFS supports extending the process template to add additional items as well as supplementing sites built from the template with additional document or site templates.1 sub-practice 2 asks to “Identify the work packages in sufficient detail as to specify estimates of project. An example report is illustrated in figure 2. The main areas of the CMMI that require more rigidity are [14]: Project Planning (PP) 2. configuration management. regression testing verification. thereby minimizing the need for management reserve. and bug resolution are couched in the linkage of work items to related tasks. 17 of the 21 Level 3 areas are targeted including project planning. Utilizing CMMI within the MSF TFS context is mainly a “stretch-to-fit” choice. but includes aspects from other levels. The decision becomes one mainly of determining if the extra items associated with the CMMI template are beneficial for the organization or the software project. Through reporting features. responsibilities and schedule” and goes on with “The amount of detail in the WBS at this more detailed level helps in developing realistic schedules. conditions of use”. Focus through the issues or are more issues materializing as we develop”? Issues List: What are the issues and how much time have they used and how much time are the expected to continue to consume? Utilizing the CMMI process in TFS pre-creates numerous documents and work item types associated with several KPAs. allocated requirements. management on quality rather than production numbers.1 asks to “identify task dependencies” and expects a “Critical Path Method (CPM) or Program Evaluation and Review Technique (PERT)” chart as a result. PP 1. In addition. fixing. product characteristics. organizational training. organizational environment for integration. The metrics gathered by TFS and available for both ad-hoc and standard reporting regardless of template selection include the following [15]. Guckenheimer and Perez [13] describe this metric as a key indicator for estimating the completion date for the next iteration. and closing bugs as well as bug trends – answers questions such as “Is the number of new defects increasing slower than the rate at which the defects are being corrected”? “How long is it taking to resolve defects”? Builds: Is code coverage being achieved? Are the validation tests passing? Is “code churn” [18] occurring wherein modules are continually being modified without any new tests being defined? Project Velocity: The rate at which the team is processing and closing work items – answers the question “Are we progressing Figure 2. monitoring. product component descriptions. it is important to recognize that some aspects of the CMMI requirements are not agile. Much of the overhead associated with a CMMI process is automated through the product. requirements definition. Bug Rates: Finding. Given that agile methods can be incorporated into a CMMI approach. and integrated supplier management are not provided in the 2005 version.Sample TFS report The TFS infrastructure provides work item tracking. Capabilities to enforce automated system builds. document review. However. this simplifies the selection basis. For example. The Microsoft team development model revolves around work items.” Project Monitoring and Control (PMC) 1. at least in the context of the TFS scenario.

No CMMI Mandated Yes Existing Agile Approach? No Estimated Effort above W $ No Yes Detailed auditing/ traceability No 3. Although the TFS tooling helps in many areas. Our decision-making process emphasizes the role of the business environment over technology issues. One organization that made the transition and published their results and lessons learned is DTE Energy [7]. As the number of organizations performing this transition increases. In the TFS environment. through the data warehouse capabilities and the ongoing capture of information during the check-in and build processes. metrics are gathered that describe what is occurring at the code development and testing levels. The theory is that prescriptive metrics become inaccurate when an organization’s places an undue emphasis on metrics. the literature should improve in quantity and quality. but also for higher CMMI levels. The focus is on delivering a solution that meets key customer needs and building upon success rather than trying to produce intermediate deliverables aimed at a long-term specification that may or may not prove accurate given the rate of technology and business changes which frequently occur even with short life cycles. Anderson [5] estimates that 85 % of the overhead is resolved by automation and that the overall amount of effort required for a CMMI template is only 150% beyond that of Agile. Table 1 .2 TFS Decision Methodology Utilizing a tools-based approach such as TFS for automating a software development process. whether choosing an agile or CMMI focused approach provides numerous benefits and improves the efficiency of development.Weighting matrix for CMMI vs. choosing to use the CMMI model will put more stress on the project team due to the sheer number of deliverables that are required. Microsoft Tech Ed 2006 Presentation DEV327) Guckenheimer and Perez also discuss the benefit of a value-up paradigm shift. DTE Energy describes their viewpoint that CMMI is a framework of “what” while their agile methods provide an empowering philosophy of “how”.Process template decision-making flowchart 190 . Agile decision Factor Traceability Quick ROI Auditing Resource shortage Time shortage Requirements known in advance CMMI + + + Agile + + + Weight Figure 3 . One of the underlying assumptions with MSF is that metrics should be descriptive rather than prescriptive. Many organizations that moved to an agile approach now find themselves trying to conform to a CMMI approach. This is based on Robert Austin’s work “Measuring and Managing Performance in Organizations” [20]. there are 59 work product artifacts while Agile has 25. Many of these metrics are required for not only the CMMI level 3 targeted Yes Use Agile Process Template as Base Estimate more than X weeks Customize Agile Framework to match existing approach Use CMMI Process Template No More than Y Function Points No More than Z Team Members No Use Agile Process Template Figure 4 . Taking into consideration the automation provided by TFS and that the CMMI process template incorporates agile benefits. In the CMMI process template.by the MSF for CMMI process template. In addition.TFS Work Item State Transitions (Source: Brian Harry. we have defined a flowchart to assist in formulating the decision to utilize the CMMI template (figure 4). There are 9 metric charts with Agile but 12 with CMMI. the value of the product is realized through iterations and improvement rather than being dictated by a strict specification up-front. In a value-up paradigm. we provide a matrix (table 1) that outlines the key issues involved in the decision along with a prioritization column to auto-calculate the best decision path.

1996. 2006. and Grechenig. ACM Press. CA. Washington. NJ. J. and Felsing. New York. 2001. AddisonWesley. [21] Zuser.sei.. [8] Carnegie Mellon Software Engineering Institute. T. 2006. [3] The Agile Alliance. [17] Microsoft.htm on December 26. Retrieved from http://www. J. 2003). May 03 . [7] Baker. Warsta. IEEE Computer Society. Adaptive Software Development: A Collaborative Approach to Managing Complex Systems. 2006. 284292. DC. 2006. 191 . 2006. 2006. 2005. Kai Chang. 2006.2005.the story of creating MSF for CMMI Process Improvement at Microsoft Corporation. DOI= http://doi. In Proceedings of the Third Workshop on Software Quality (St.29. S. NJ.fi/pdf/publications/2002/P478. 2006. CONCLUSION A comparison of Agile and CMMI is not truly an “Apples-toapples” comparison. Q/A checking. T.acm. WISER '04. Manifesto for Agile Software Development retrieved from http://agilemanifesto. 2005. Agile Conference. This approach can be used to help automate continuous improvement regardless of the CMMI level targeted. K.. OOPSLA’95 Workshop on Business Object Design and Implementation. Toward a conceptual framework of agile methods: a study of agility in different disciplines.10. Springer-Verlag. [10] Cockburn . Microsoft Solutions Framework. Retrieved from http://www. MSF and XP: a comparative study.org/10. Heil. DC. 6. 1-6. Auburn University for insights on improving software process efficiency and encouragement with publication of this paper and Dr. Ptentice-Hall. Microsoft Visual Studio 2005 Team Foundation Server.4.edu/cmmi/general/general. and Fitzgerald. [9] Carnegie Mellon Software Engineering Institute. 2005). New York: Dorset House [15] Holland. Capability Maturity Model Integration (CMMI) Version 1. Perez.mspx on December 26. Retrieved from http://msdn2. 2006. Agile Software Development Methods: Review and Analysis. It is possible to leverage agile practices in a CMMI context. W. 193-201.acm. Proceedings. May 17 . Therefore. ADC.com/read/231944. May 15 . DOI= http://doi. New York. [11] Conboy. [13] Guckenheimer.. New York. Software Engineering with Microsoft Visual Studio Team System.doi. VTT Publications 478.pdf on December 26. Addison-Wesley. D.sei. Software quality development and assurance in RUP. T.org/10. In Proceedings of the Agile Development Conference (July 24 . 2006. the choice of one or another as a process template need not be a sacrificial proposition.syscon..1145/1083292. 3-WoSQ.acm. 2000. USA. Washington. Measuring and Managing Performance in Organizations.php/Crystal_methodologies_ main_foyer on December 26.17. 37-44. [5] Anderson. The MIT Press. [4] Austin. Cambridge. MA.microsoft. NY. In Proceedings of the 2004 ACM Workshop on interdisciplinary Software Engineering Research (Newport Beach. 2000. Ronkainen. 2002. We have provided a flowchart and matrix customizable for an organization to assist in the template selection. Retrieved from http://www. and Ball. 2005. Retrieved from http://alistair.. 2004). J.1083300 5.cmu. Customization is achieved by modifying the variables and weighting used in the trade-off decisions. International Conference on Software Engineering. [2] Abrahamsson.1145/1029997. N. 2006. W. 2005.1062514 [19] Palmer. while Agile is a development philosophy. P. DOI= http://dx. 2005). ACM Press. Dorset House Publishing. J. AKNOWLEDGEMENTS Thanks to Dr. record-keeping and other tasks better done by automated tools. Auburn University for elaborations of CMMI and software quality.42 [6] Beck. 244-254. 2004. automated testing. and metrics gathering facilitates a CMMI approach that still supports agility.05. The distinction is subtle but significant because it means that the approaches can potentially complement each other rather than being mired in incompatibility. In Proceedings of the 25th international Conference on Software Engineering (Portland. [16] Microsoft. What is CMMI.html on December 26. D. A. DOI= http://doi.org/ on December 26. Upper Saddle River.2 Overview. S.edu/cmmi/adoption/pdf/cmmioverview06. 2006..org/10. Oregon. Upper Saddle River. David Umphress. MO. B. Use of relative code churn measures to predict system defect density. In Proceedings of the 27th international Conference on Software Engineering (St. J. Stretching Agile to fit CMMI Level 3 .pdf on December 26. USA. Crystal Main Foyer. 2002. Extreme Programming Explained: Embrace Change. Visual Studio 2005 Team Foundation Server. 2006. Siponen. IEEE Computer Society. [20] Schwaber. New directions on agile methods: a comparative analysis. ICSE '05. rather than on enforcement. R.1145/1062455. Salo. and Ronkainen.21.1109/ADC. [14] Highsmith. Retrieved from http://www. Scrum Development Process. 2005). Missouri. 2005.com/technet/solutionaccelerators/msf/d efault. 1999. A Practical Guide to Feature Driven Development. 1982. S. 2003.microsoft. TFS provides both Agile and CMMI process templates that integrate with Microsoft Team Suite core capabilities. Out of Crisis. November 05 . REFERENCES [1] Abrahamsson. NY.1030005 [12] Deming.cockburn. Louis.inf. [18] Nagappan. Louis.com/enus/teamsystem/aa718825. S. 2006. M. CMMI is a maturation model. Wastra.cmu. A tools-based approach ensures objective metric collection and reduces cost by focusing resources on actual development and project activities. Retrieved from http://dotnet. Using a tool such as TFS that provides configuration management. 1995. 2006.vtt.us/index.aspx on December 26. Formalizing agility: an agile organization's journey toward CMMI accreditation.org/10.

Sign up to vote on this title
UsefulNot useful