Agile Project Management With SCRUM: Theory and Practice

Jeff Sutherland, Ph.D.
Chief Technology Officer
jeff.sutherland@computer.org http://jeffsutherland.com

The Zen of SCRUM 
       

So simple, anyone can implement it! So easy, all can benefit! So subtle, few achieve transcendent performance « How is it that a project manager does nothing, and achieves everything? Interlocking principles emerge product like jigsaw puzzle. Novice removes one piece -- engine never fires on all cylinders « Who can know why? ScrumMaster must understand deeply and practice rigorously. Only then will team members say, ³This experience changed my life!´

Complex Adaptive Systems (cas) 
Interacting agents respond to stimuli.  Stimulus-response behavior is defined in terms of rules.  Agents adapt by changing rules as experience accumulates.  Agents aggregate into meta-agents whose behavior is emergent.  How can a collection of dumb things emerge smart system behavior?
Web services? 1998 Agents 1995 Components 1993 Business Objects 1980 Classes 1970 Procedures

Chaos Fragmentation cas Self Organization Frozen

Maamar, Zakaria and Sutherland, Jeff (2000) Toward Intelligent Business Objects: Focusing on Techniques to Enhance Business Objects that Exhibit Goal-Oriented Behaviors. Communications of the ACM 40:10:99-101.

Enterprise Systems are cas
systems.  Modification time is on the order of months or years, roughly time required to change software.  Automating business processes renders parts of the business in software.  Business systems have severely constrained rule sets, making ideal test bed for cas concepts.
Sutherland, Jeff and van den Heuvel, Willem-Jan (2002) Developing and integrating enterprise components and services: Enterprise application integration and complex adaptive systems. Communications of the ACM 45:10:59-64. 

Business entities are examples of complex adaptive

 Differential "fitness": the number of copies of an element that are created in a given time varies. Simon and Schuster. Darwin's Dangerous Idea: Evolution and the Meanings of Life.Objects Meet Requirements for Evolution elements (class libraries). 343. 1995. p.  Heredity or replication: the elements have the capacity to create copies or replicas of themselves (inheritance).  Variation: there is a continuing abundance of different . depending on interactions between the features of that element and the features of the environment in which it persists (reuse). Daniel Dennett.

Do Programmers Meet Evolutionary Requirements?  Algorithms create movement through design space ± Simple minded repetitive procedure ± Incremental change to adjacent designs  "Cranes" accelerate movement through design space ± Sex and genetic engineers ± Evolutionary prototyping and smart people ± Emergent architecture Brooks. Vol. IEEE Computer. . pp. 20. No. April 1987. Fred. 10-19. 4. No Silver Bullet: Essence and Accidents of Software Engineering.

. pointers)  Unpredictability of the waterfall model of software Too many projects fail development Tony Wasserman. Nov 1996. icons. menus. IEEE Software 13:6:23-31.Change is Imperative: Wasserman's 7 Factors Driving Change  Criticality of time to market  Shift in computing economics  Powerful desktop computers  Extensive networks and the Web  Growing availability of object technology  WIMP (windows. Toward a Discipline of Software Engineering.

"Why Are Systems Late. Framing Software Reuse: Lessons from the Real World. Paul G. Yourdon Press Computing Series. . 1997. Over Budget. Wrong?" "The Waterfall Methodology!" (Paul Bassett) Analysis Paralysis ± static modeling overused ± specs are stale baked Design-from-Scratch ± no generic models ± no standard architectures Large Project Teams User Intermediaries No Early Warning Signals Bassett.

The problem-solving process ends when resources are depleted. 4. 1973. 1999. The 5th Annual JAWS S3 Proceedings. Dilemmas in a General Theory of Planning. but good-or-bad « getting all stakeholders to agree that a resolution is "good enough" can be a challenge. Various stakeholders have differing views of acceptable solutions. 1990 . 75% of [DOD] projects failed or were never used. Vol.           Wicked problems have no definitive formulation. The planner (designer) has no right to be wrong. The causes of a wicked problem can be explained in numerous ways. H and Webber M. Each wicked problem can be considered a symptom of another problem. Each wicked problem is essentially unique. Wicked Problems. Jarzombek. stakeholders lose interest or political realities change. Policy Sciences. Elsevier. There is no immediate or ultimate test of a solution to a wicked problem. Each attempt at creating a solution changes your understanding of the problem. embedded in a dynamic social context. Degrace and Hulet's book. Part of the art of dealing with wicked problems is the art of not knowing too early what type of solution to apply. and only 2% were used without extensive modification.Wicked Problems: Righteous Solutions Out of a total cost of $37B for the sample set. Rittel. Every implemented solution to a wicked problem has consequences. Righteous Solutions. Solutions to wicked problems are not true-or-false. A wicked problem is a set of interlocking issues and constraints that change over time. Wicked problems have no stopping rule. Wicked problems don't have a well-described set of potential solutions. Prentice Hall.

.  Wegner's Lemma .for a new software system.it is not possible to completely specify an interactive system [Wegner. 1995]. 1996].Software Development is an Empirical Process  Ziv's Uncertainty Principle in Software Engineering - uncertainty is inherent and inevitable in software development processes and products [Ziv. the requirements will not be completely known until after the users have used it.  Humphrey's Requirements Uncertainty Principle .

Productivity: All at Once Models  Single Super-Programmer  Handcuffing two programmers      together Brook¶s Surgical Team Borland Quattro project Goldratt¶s ³The Goal´ Senge¶s systems thinking Holland¶s complex adaptive systems .

000-95. IT Metrics Strategis IV:8:12-16. Ware. August 1998. Lawrence H.000 SLOC (source lines of code) Putnam. . 491 medium sized projects with 35. Familiar Metrics Management: Small is Beautiful-Once Again. Cutter Information Corp. and Myers.Team Size: Development Effort in Months   The smaller the better..

Cutter Information Corp.Team Size: Development Time in Months  Sweet spot is 5-7 people  491 medium sized projects with 35. . August 1998.000-95. Familiar Metrics Management: Small is Beautiful-Once Again.. IT Metrics Strategis IV:8:12-16. Lawrence H.000 SLOC (source lines of code) Putnam. and Myers. Ware.

Second Edition. 1997. McGraw Hill.000 lines of C++ code Time in months Staff Function points per staff month BWP 31 8 77 Industry standard >50 >100 2 Jones.Bell Labs Report on most productive project ever: Borland Quattro for Windows 1.000. Capers.  BQW . Applied Software Measurement.

all making technical contributions Higher communication saturation than 89% of projects More even distribution of workload ³Anti-schismogenetic´ ± no cliques Highly iterative development Strong architectural interaction with implementation More time spent in project team meetings than anything else ± several hours a day Gerry Weinberg notes that CMM Level 1 and 2 teams need strong managerial direction. QA integral to team.          One of most remarkable organizations. and Productivity. Borland team was clearly in this category. . product management. processes. although not by commonly accepted criteria.James Coplien. Quality. 1994. and development cultures seen in AT&T Bell Laboratories Pasteur process research project Project management. Proceedings of the 5th Annual Borland International Conference. Level 3 paradigm shift is self-directing team. Orlando. Borland Software Craftsmanship: A New Look at Process.

and it saves time and money. or how big it will grow.´  ³There are no rules for this kind of thing²it¶s never been done before. Oct 1987.´  ³Software is like a plant that grows.Team comments on Quattro project  ³We are satisfied by doing real work. . You can¶t predict its exact shape.´ ³Evolutionary development is best technically.´ Report of the Defense Science Board Task Force on Military Software.

IBM Federal Systems Devision ± Gerry Weinberg  1960 ± Weinberg teaching IID at IBM Systems Research Institute  1969 . Project Mercury. Craig and Basili. IEEE Computer.Earliest published reference to IID: ± Robert Glass. Vic. A History of Iterative and Incremental Development. Mar 1969 Larman. ACM Computing Surveys. June 2003 (in press)  1956 ± Benington¶s stagewise model ± USAF SAGE . Elementary Level Discussion of Compiler/Interpreter Writing.History of Iterative and Incremental Development (IID) System  1957 ± IBM Service Bureau Corp.

A History of Iterative and Incremental Development. Development. Dec 1975. Harlan. Design. In Debugging Techniques in Large Systems. IEEE Transactions on Software Engineering. Sept 1984. Integration: Space Shuttle Flight Software. 1971   1972 ± TRW uses IID on $100M Army Site Defense software 1975 ± First original paper devoted to IID ± Gasili. Vic. Craig and Basili. Communications of the ACM. Top-down programming in Large Systems. IEEE Computer. Prentice Hall.  1977-1980 ± IBM FSD builds NASA Space Shuttle software in 17 iterations over 31 months. Vic and Turner. Iterative Enhancement: A Practical Technique for Software Development. averaging 8 weeks per iteration ± Madden and Rone. June 2003 (in press) .History of Iterative and Incremental Development (IID)  1971 ± IBM Federal Systems Division ± Mills. Albert. Larman.

A Rational Development Process. June 2003 (in press) . Dynamics of Software Development. Microsoft Press. Origins of RUP   1995 ± Microsoft IID published ± ± 1996 ± Kruchten. April 1987 ±   1994 ± First SCRUM at Easel Corporation 1994 ± DOD must manage programs using iterative development ± Report of the Defense Science Board Task Force on Acquiring Defense Software Commercially. Crosstalk. June 1994. Fred. IEEE Computer. No Silver Bullet. Vic.  1986 ± Brooks. McCarthy. Proceedings of an International Workshop on Software Process and Software Environments. July. Barry. Larman.History of Iterative and Incremental Development (IID)  1985 ± Barry Boehm¶s Spiral Model ± Boehm. A Spiral Model of Software Development and Enhancement. A History of Iterative and Incremental Development. IEEE Computer. Jim. March. Craig and Basili. or its effectiveness [as incremental development]. 1995. 1985 Nothing « has so radically changed my own practice.

and only 2% were used without extensive modification. ³Research also indicates that smaller time frames. with delivery of software components early and often. Out of a total cost of $37B for the sample set. Vic. With the hindsight of iterative experience. June 2003 (in press) . will increase success rate. The 5th Annual JAWS S3 Proceedings. He had not learned of incremental development at the time and based his advice on textbooks and consultants advocating the waterfall method. A History of Iterative and Incremental Development. Craig and Basili. Jarzombek. Origin of Feature-Driven Development Top reason for massive project failures was waterfall methods.History of Iterative and Incremental Development (IID)   1996 ± Kent Beck Chrysler Project ± ± Origin of XP David Maibor expressed regret for the creation of the waterfall-based standard. he would recommend IID. IEEE Computer. 1999. 75% of projects failed or were never used. 1996 ± Larman meets with principal author of DD-STD-2167   1997 ± Coad and DeLuca rescue Singapore project ± ± 1998 ± Standish Group CHAOS Project  1999 ± Publication of extensive DOD failures ± Larman.

Craig and Basili. A History of Iterative and Incremental Development. Vic. Larman.History of Iterative and Incremental Development (IID)  2001 ± 17 process expert ³anarchists´ meet at Snow Bird ± Agile Manifesto initiated 100s of books and papers on agile development  2001 ± MacCormack¶s study of key success factors ± MacCormack. IEEE Computer. 2001. Alan. MIT Sloan Management Review 42:2. June 2003 (in press) . Product-Develoment Practices that Work.

we value the items on the left more. . Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is. while there is value in the items on the right.Manifesto for Agile Software Development We are uncovering better ways of developing software by doing it and helping others do it.

Alan.MacCormack Process Evolution  Waterfall model ± sequential process maintains a     document trail Rapid-Prototyping Model ± disposable prototype helps establish customer preference Spiral Model ± series of prototypes identifies major risks Incremental or Staged Delivery Model ± system is delivered to customer in chunks Evolutionary Delivery Model ± iterative approach in which customers test an actual version of the software MacCormack. MIT Sloan Management Review 42:2:75-84. Winter 2001. . Product-Development Practices That Work: How Internet Companies Build Software.

Product-Development Practices That Work: How Internet Companies Build Software. Winter 2001.MacCormack Success Factors  Early release of evolving product design to customers. MIT Sloan Management Review 42:2:75-84.  Daily incorporation of new software code and rapid feedback on design changes  A team with broad-based experience in shipping multiple projects  Major investment in design of product architecture MacCormack. Alan. .

reprint no.Rugby Takeuchi. NEC. Harvard Business Review 64:1:137-146 (Jan/Feb).SCRUM Origins: Takeuchi and Nonaka Lessons from Fuji-Xerox. Honda. Hirotaka and Nonaka. HP  Old model ± Relay Race ± Speed and flexibility not adequate in today¶s market  New model . Xerox. 3M. Canon. 1986. Brother. . The new new product development game. Epson. 86116. Ikujiro.

Moving the SCRUM downfield .

does not bring about speed and flexibility. But taken as a whole.Takeuchi and Nonaka Success Factors  Built-in instability  Self-organizing project teams  Overlapping development phases  ³Multilearning´  Subtle control  Organizational transfer of learning ³These characteristics are like pieces of a jigsaw puzzle.´ . the characteristics can product a powerful new set of dynamics that will make a difference. Each element. by itself.

and telling them to jump or else. removing the ladder. I believe creativity is born by pushing people against the wall and pressuring them almost to the extreme.´ .Factor 1: Built-in instability  Top management kicks off development process by signaling broad goal. ± Example: Fuji-Xerox gave FX-3500 project team two years to come up with a copier that cut costs in half  Top management creates an element of tension in the project team through challenging requirements with wide freedom to achieve strategic objective.  Project team is offered extremely challenging goals with wide measure of freedom. ± Honda Executive: ³It¶s like putting the team members on the second floor.

ScrumDown at the Radcliffe Rugby Club . the process begins to create its own dynamic order.  Left to stew.  ± Autonomy ± Self-transcendence ± Cross-fertilization  At some point.  A group possesses a self-organizing capability when it exhibits three conditions.Factor 2: Self-organizing project teams A project team takes on a self-organizing character as it is driven to a state of ³zero information´ ± where prior knowledge does not apply. the team begins to create its own concept.  The project team begins to operate like a start-up company.

 On a day to day basis. top management acts as a venture capitalist ± ³We open our purse and keep our mouth closed.´  Example: IBM development of personal computer  Example: Honda City project team.Autonomy  Headquarters involvement is limited to providing guidance.´ . average age 27. ³Develop the kind of car that the youth segment would like to drive. management seldom intervenes and the team is free to set its own direction. and moral support at the outset.  In a way. money.

they devise ways to override the status quo and make the big discovery.  Example: Canon AE-1 team Flyhalf Sarah Schooler of Radcliffe fending off Boston College .´  They elevate their goals through the development process.Self-transcendence  The project teams appear to be absorbed in the never- ending quest for ³the limit.  By pursuing what appear to be contradictory goals.

and behavior patterns carries out new product development.Cross-fertilization  Team with wide variety of specializations.  ³When all team members are in one room. others information becomes yours without even trying.  Working in one large room is best (Fuji-Xerox). thought processes.´ Radcliffe Rugby Football Club .

Factor 3: Overlapping Development Phases  Self-organizing character of the team produces unique dynamic or rhythm  Sashimi system ± Fuji Xerox Rugby system ± Honda  Hard merits (demerits) ± Speed and flexibility (watch out for muck and mall)  Soft merits ± Share responsibility and cooperation ± Stimulates involvement and commitment ± Sharpens a problem-solving focus ± Develops initiative and diversified skills ± Heightens sensitivity to market conditions .

Factor 4: Multilearning  Learning by doing in two dimensions ± Across organization ± Across specialty  Enhanced learning opportunities ± 15% of time devoted to ³dreams´ ± 3M ± Peer pressure to study ± Send team to Europe to look around ± Honda ± Bring in top academics and consultants ± HP  Everyone learns multiple skills .

control by peer pressure. ambiguity.Factor 5: Subtle Control  Management establishes checkpoints ± Prevents instability. control by love = ³subtle control´  Management responsible for: ± ± ± ± ± ± Selecting team members for balanced team Creating an open working environment Encouraging engineers to go out in the field Establishing rewards based on group performance Tolerating and anticipating mistakes Encouraging suppliers to become self-organizing . and tension from turning into chaos  Emphasis on self-control.

tools .Factor 6: Organizational Transfer of Learning  Transfer knowledge outside group ± Scatter successful team to new projects ± Institutionalize practice (monthly demos at IDX)  Consciously pursue unlearning ± Next generation must be 40% better ± Cut product cycle by 80% ± Scrap old parts. processes.

± But in times of desperation. Cuckoo Principle: Successful SCRUMs become company models (or they can get crushed because they are different). . Anti-Waterfall Principle: Operational decisions are made incrementally. SCRUMs are easily created. Push/Pull Principle: Differentiation in concept phase. integration dominates in implementation phase Spread the Wealth Principle: Non-experts take on new tasks. Strategic decisions delayed to last moment. Control Anti-Pattern: Seniority based companies have difficult time.Challenges and Opportunities  Winding the Rubber Band Principle: Broad mandate and      demanding goals create tension.

A Spiral Model of Software Development and Enhancement. California. Boehm. should be sent back to prior phases. Barry. Boehm. Proceedings of an International Workshop on Software Process and Software Environments. A Spiral Model of Software Development and Enhancement. This is the most commonly used variant of the Waterfall today. B. Trabuco Canyon.W. the phases and phase processes are still linear. May 1988. 1985. Boehm.Spiral Methodology   Barry Boehm introduced the Spiral Methodology to "fix" problems with the Waterfall Methodology.21. However. . A Spiral Model of Software Development and Enhancement. IEEE Computer. or should be ended. The Spiral methodology "peels the onion". A prototype lets users determine if the project is on track. vol. Coto de Caza. Barry. August 1986. March 27-29. ACM SIGSOFT Software Engineering Notes. pp 61-72. #5. progressing through "layers" of the development process.

 . but each iteration only addresses one subsystem.Iterative Methology The Iterative methodology improves on the Spiral methodology. Further iterations can add resources to the project while ramping up the speed of delivery.  Improves cost control. reduces risk.  Still assumes that the underlying development processes are defined and linear.  Each iteration consists of all of the standard Waterfall phases. ensures delivery of (sub)systems. and improves overall flexibility.

The deliverable is determined during the project based on the environment. The project is open to the environment until the Closure phase. Sprints are used to evolve the final product. Sprints are nonlinear and flexible. The deliverable can be changed at any time. It is treated as a black box that requires external controls. The Sprint phase is an empirical process. .SCRUM Methodology      The first and last phases (Planning and Closure) consist of defined processes.

Methodology Comparison .

 Current approaches rests on the fallacy that the development processes are defined. predictable processes. . Business Object Design and Implementation (Eds. London: Springer-Verlag.Risk with Current Methodologies Any methodology is better than nothing.  They lack flexibility needed to cope with the unpredictable results and respond to a complex environment. 1997. SCRUM Development Process. Ken.  Schwaber.). Jeff Sutherland et al.

 SCRUM can accelerate closure by inducing the phenomenon known as "punctuated equilibrium" seen in the evolution of biological species.SCRUM Lowers Risk  Development teams need to operate adaptively within a complex environment using imprecise processes. Vintage Books. Complexity: Life at the Edge of Chaos. Collier Books. . Artificial Life: A Report from the Frontier Where Computers Meet Biology. Steven. Roger. 1993. 1994. Lewin. Levy.

sutherland@computer.org http://jeffsutherland. Ph.Agile Project Management With SCRUM: Theory and Practice Jeff Sutherland. Chief Technology Officer jeff.com .D.