Down with Efficiency

David J. Anderson Lean & Agile Sweden 2008

Agile was a rebellion against the undesirable industry trends of the 1990s

Everything traditional was bundled together and demonized

And the demon was nicknamed “waterfall”

Almost all previous software engineering literature was labeled as non-agile

Despite a lot of work on iterative and incremental development

And a focus on people from as early as 1969 with Jerry Weinberg’s Psychology of Computer Programming

We can sum up the first two statements of the Agile Manifesto with…

“Perfect is the enemy of good enough”

In the early 1990s it was assumed we had to pursue perfection in order to improve software project success

The Capability Maturity Model was published in 1991

It told us that project success came from process maturity

First we should seek to be repeatable and predictable (level 2)

And mature to a defined process (level 3)

At which point we should be able to deliver projects successfully

The predominant paradigm of the day assumed that in order to be more repeatable and predictable we needed more accurate estimates

That we needed to seek perfection upfront

That we needed to eliminate ambiguity and uncertainty

And so we created more and more methods of analysis

Each additional approach (and its diagram) designed to fill some of the gap to perfect information

These diagrams meant that in pursuit of perfection we created more and more documents prior to writing code

And in those days they primarily were just documents, often created with tools like Visio

Tools that turn diagrams in to code, or map different levels of detail together (domain specific languages) did not become widely available until this decade

Although there were early pioneers like Together (that kept UML class diagrams in sync with written code) available as early as 1997

As an industry we had far exceeded the law of diminishing returns on analysis before the agile revolution started

So in essence the pursuit of repeatability and the assumption that to achieve it required the pursuit of perfection is what stimulated the Agile Manifesto value Working software over comprehensive documentation

The next level of maturity requires a defined process

And again, the assumption was that for the process to be repeatable, the definition must be precise (i.e. perfect)

This resulted in an industry forming around the definition of very large incredibly detailed prescriptive processes

These processes were so large and unwieldy that they were beyond the comprehension of most teams, and hence…

It was necessary to hire a large consulting firm to deliver the process and educate the team on its use

Some proprietary processes ran to thousands of pages of documentation and cost upwards of $100,000

The industry was being led by the Software Engineering Institute and in an attempt to understand their vision, managers were driven to ever more elaborate processes and tools

Delivered by rapidly growing bloated global systems integration firms offering processes and tools that delivered on a vision of perfection (early in the lifecycle)

Except…

We all know now that it didn’t work!

This realization really provoked the Agile Manifesto value… Individuals and interactions over processes and tools

So, summarizing, the first two statements of Agile Manifesto
Individuals and interactions over processes and tools Working software over comprehensive documentation

rebel against the pursuit of perfection and ask us to

embrace uncertainty

As the Agile community emerged around the turn of the century, the new agile paradigm seemed separated from the traditional software engineering paradigm like oil from water

On the one hand, we had a large body of literature developed mainly from government space and defense contracts

And on the other hand, we had an emerging set of thought leaders working with startup companies…

Whose experience was mainly rooted in the liberal culture of America’s west coast

For example, a significant set of the early agile community thought leaders came from the patterns movement that had its roots in Portland, Oregon

What we’ve now come to realize is that indeed these two communities were like oil and water, the cultures were incredibly different

The traditional community was rooted in government work, on large scale projects with a very high cost of failure

Sending landers to Mars

Or developing new fighter aircraft

The other community was building web sites in the dot.com bubble

The former was rooted in a low trust culture full of conservative risk averse professionals, trying to avoid disasters with tax payers money

While the latter, was innovating daily, empowering people to release new functionality to unwitting Internet consumers with an assumed negligible (zero) cost of failure

The agile movement recognized that in order to innovate at Internet pace, there simply wasn’t time to write requirements and negotiate contracts and commitments around them

It was better simply to trust people, let them work, and adopt a failure tolerant attitude

Refactoring wasn’t rework (in the traditional sense), it was embracing uncertainty

The agile movement latched on to this highly collaborative approach and without realizing it, built a whole paradigm on the assumption of a high trust, liberal culture as a pre-requisite for adoption

And it was this that led to the 3rd value of the Agile Manifesto… Customer collaboration over contract negotiation

Knowledge work is perishable

Even when requirements are written in a persistent format they are still perishable

People fail to articulate precisely

Others fail to interpret precisely

Written communication is fallible

There is always a layer of tacit knowledge

And yet folks leave jobs, or get sick or take vacations

Human memories fade – why did we decide to do it that way?

And markets move

Go-to-market strategies change

Strategic positioning changes

Competitors enter markets or release new products and services

Economic conditions change

Value is released by bringing the differentiating functionality to market fastest

Lead time – from ideation to delivery is vital to business competitiveness

And it was the recognition of the perishable nature of requirements and the business value associated with short cycle times and fast delivery…

That led to the 4th value of the Agile Manifesto … Responding to change over following a plan

So the agile movement blew up a number of myths and ended the paradigms built around them

Perfect was the enemy of good enough and we stopped trying to be perfect, instead we preferred to refactor (it’s much cheaper, faster and better)

Self-organization (or empowerment, delegation and failure tolerance) through adoption of a high trust culture was better than contracts, commitments, audit and arbitration in a low trust culture

System requirements were not assets, they were liabilities So we spent less time writing things down, relied a lot more on tacit knowledge and focused on reducing cycle times

But there was one insidious behavior of Western corporate culture that the Agile Manifesto didn’t address…

The relentless pursuit of efficiency

The pursuit of efficiency is rooted in the Scientific Management of Frederick W. Taylor

Scientific Management was based on time and motion studies that sought to optimize the utilization of workers in mass production factories

It was further institutionalized by the development of cost and management accounting (primarily) at General Motors in the 1930s

Cost accounting seeks to normalize all efficiency calculations in to a single unit of currency

What is efficiency?

What’s the most efficient way to paint a fence?

Focusing on efficiency produces better cost accounting results for large batch sizes
Queue Queue Setup Queue Q Task
Time

Task W Task Wait Cleanup

Task

Wait Wait

Because the transaction costs (in this case, the setup and cleanup) can be amortized over a greater number of value items (in this case, painted sections of fence)

But there are also coordination costs in knowledge work problems
Communication Queue Queue Setup Queue Q Task
Time

Task W Task Wait Cleanup

Task

Wait Wait

And in knowledge work problems, coordination costs grow non-linearly with batch size
Communication Queue Queue Setup Queue Q Task
Time

Task W Task Wait Cleanup

Task

Wait Wait

The prevailing paradigm in Western management was that overheads (transaction and coordination costs) were inevitable and efficiency was achieved through economies of scale (i.e. large batch sizes)

However, the Japanese took a different approach to efficiency

If the transaction and coordination costs were too high to allow a batch of work to be completed economically…

Then they simply had to find a way to reduce the overheads in order to make small batches efficient

Why?

What was so important about small batch sizes?

Quite simply, there is a direct relationship between batch size and cycle time
Device Management Ike II Cumulative Flow
240 220 200 180 160 140 120 100 80 60 40 20 0
eb eb eb 17 -F 10 -F 24 -F

Features

Lead Time

WIP
2M ar 9M ar 16 -M ar 30 -M ar 23 -M ar

Time
Inventory Started Designed Coded Complete

Smaller batches allow the right product to be brought to market at the right time

Releasing more value for the business

Batch size problems from an economies of scale paradigm are what cause US auto manufacturers to deeply discount when they over produce the wrong model or specification

Small batches avoid waste and make a business more responsive to the market

By focusing on cost and efficiency throughout the 20th Century

Western businesses lost sight of value and overall effectiveness

The paradigm that blows away this myth of efficiency through economies of scale is

Lean

And it is Lean that completes the reforms that the Agile movement started

Which might lead us to extend the agile values to include…

We value… The elimination of waste over efficiency and cost reduction

But wait…

That’s not the end of the story

Lean (Kaizen) is in large part based on the teachings of W. Edwards Deming

And his System of Profound Knowledge

But, (and this is not widely understood) Watts Humphrey’s original vision for the Capability Maturity Model…

Was to find a way of enabling Deming’s System of Profound Knowledge in the software engineering profession

Despite the fact that he based the maturity model on Philip Crosby’s Manufacturing Maturity Model

Lean (Kaizen) are very compatible with CMMI and the aspiration to achieve higher levels of maturity – quantitative management and optimization
(continuous improvement)

Lean allows the agile community to speak a language that is understood at the SEI

Lean also allows us to reinterpret much of the literature from the last 40 years and extract the positive value

It isn’t all bath water – there were some babies in there worthy of preserving

The CMMI is paradigm agnostic

The CMMI does not require the pursuit of perfection

Nor does it require a low trust conservative culture for implementation

Nor does it refute the idea that knowledge work is perishable and insist on written documentation

Nor does it require a focus on efficiency

Nor does it require a waterfall approach

People have made their own reactions to capability and maturity based on the paradigms that were familiar to them

The CMMI is not the demon

Nor is much of the traditional software engineering literature and guidance

But waterfall has been encouraged by the bad paradigms

The pursuit of perfection encouraged Waterfall

Waterfall denied the notion that knowledge work is perishable

A focus on efficiency and economies of scale encouraged Waterfall with its big batch transfers

Until 2002 GAAP accounting rules encouraged Waterfall with big batch transfers

Because it was acceptable to capitalize requirements and analysis phases on the balance sheet rather than sink them as costs on the P&L

Corporate governance rules encouraged Waterfall through the use of stage-gate approvals

So I am filled with hope

Lean will unite the software development community

Lean will enable us to cast agile ideas in a form that traditionalists will understand

And will allow the agile community to recognize value in the CMMI and other work from the SEI

Over the last 18 months, I have built an enterprise scale agile/Lean software engineering team at Corbis

During that time, I also saw them claw their way up the maturity ladder

First achieving repeatability

Then introducing defined processes

And most recently achieving a degree of quantitative management

In doing so, I saw many of the process areas required for a CMMI appraisal, emerge naturally as the team matured through a focus on kaizen style continuous improvement

Hence, I have come to see the value in CMMI

Though, I’ve never lost sight of the value in many traditional software engineering concepts like coupling, cohesion, structured methods, objectorientation, aspect-orientation or code inspections

And my trust in agile methods is constantly reinforced with superior results

So I’m even more excited about the next 7 years than I have been about the past 7 (since the Agile Manifesto)

I believe our profession has been through a period of learning

Making corrections from failed paths

And is set to deliver significant sustainable productivity and economic gains

Lean is the missing piece

The future of software engineering is Lean

About…

David Anderson is a thought leader in managing effective software team. He is a board member and founder of the APLN and signatory of the project management Declaration of Interdependence. He has 25 years experience in the software development business starting with computer games in the early 1980’s. As a pioneer in the agile software movement David has managed teams at Corbis, Sprint and Motorola delivering superior productivity and quality. More recently at Microsoft he developed the MSF for CMMI Process Improvement methodology.

David’s book, Agile Management for Software Engineering – Applying the Theory of Constraints for Business Results, introduced many ideas from Lean and Theory of Constraints in to software engineering. As Senior Director of Software Engineering at Corbis, David led a team that introduced many Lean ideas including use of kanban, oobeya, and visual control techniques. The results were astounding. The team demonstrated high levels of productivity, improved lead times and quality. Email: dja@agilemanagement.net

Thank you

Sign up to vote on this title
UsefulNot useful