8 CHAPTER 1 What Is Agile Data Warehousing?

At this point, the team is ready to start another cycle. These iterations progress
as long as there are user stories on the project’s backlog and the sponsors continue
funding the project. During an iteration’s development phase, the team’s embedded
business partner may well have reshuffled the order of the stories in the backlog,
added some new one, and even discarded others. Such “requirements churn” does
not bother the developers because they are always working within the near-planning
horizon defined by the iteration’s time box. Because Scrum has the developers con-
stantly focused on only the top of the backlog, the business can steer the team in a
completely new direction every few weeks, heading to wherever the project needs to
go next. Such flexibility often makes business partners very fond of Scrum because
it allows the developers from the information technology (IT) department to become
very flexible and responsive.

The “disappointment cycle” of many traditional projects

In contrast to Scrum’s iterative approach to delivering systems, traditional soft-
ware engineering operates on a single-pass model. The most widely cited defini-
tion of this approach can be found in the first half of a 1970 white paper entitled
“Managing the Development of Large Software Systems” by a TRW researcher
named Dr. Winston Royce. [Royce 1970] This paper has been commonly inter-
preted to suggest that, in order to avoid conceptual errors and extensive reprogram-
ming of an application, all requirements should be gathered before design begins,
all design should be completed before programmers begin coding, and the bulk of
coding should be completed before serious testing can get underway. In this pro-
cess, each work phase should “fill up” with specifications before that information
spills over into the next phase—a notion that led many to call this approach a cas-
cade or a “waterfall” method.
Many people describe this waterfall process as the “big design up-front” strategy
because it requires enormous design specifications to be drafted and approved before
Copyright © 2012. Elsevier Science & Technology. All rights reserved.

programming work can begin. [Ambler 2011] It has also been called a “plan-driven”
or “command and control” approach because the big design results in enormous pro-
ject plans, with possibly thousands of separate tasks, that project managers use to
drive the daily activities of the development teams they command. A further name for
this style of organizing development is the “big bang” approach because all the value
is theoretically dropped upon end users at the conclusion of the project.
Waterfall-style project organization can seem to be a safe approach for large
applications, especially those with multiple, intersecting data layers found in data
warehouses, because the engineers supposedly think out all aspects of the project
thoroughly ahead of time. Scrum, however, simply takes a few requirements off the
top of a list and converts them into code before the next set of features is even con-
sidered. In contrast to a waterfall method, Scrum can seem very tactical and ad hoc.
For these reasons, when software professionals first learn of agile, they often
decide that a waterfall method must be far more robust. Such conclusions are ironic

Hughes, R. (2012). Agile data warehousing project management : Business intelligence systems using scrum. Elsevier Science & Technology.
Created from bibliouniminuto-ebooks on 2023-09-11 21:39:19.
 The “disappointment cycle” of many traditional projects 9

because waterfall methods have had over 40 years to prove themselves, but statis-
tics show that they struggle to deliver applications reliably. The Standish Group’s
seminal “Chaos” reports detailed the software industry’s track record in delivering
large systems using traditional methods. [Standish Group 1999] After surveying
the results of 8380 projects conducted by 365 major America companies, results
revealed that even small projects below $750,000 were unable to deliver applica-
tions on time, on budget, and with all the promised features more than 55% of the
time. As the size of the applications grew, the success rate fell steadily to 25% for
efforts over $3M and down to zero for projects over $10M (1999 dollars).
Data warehousing projects fall easily in the middle to upper reaches of the range
documented in the Standish study. Not surprisingly, they, too, have demonstrated
trouble reliably delivering value under traditional project management methods.
A survey performed in 1994—before agile methods were common—by the data
management industry’s primary trade magazine revealed that its readers’ data ware-
house projects averaged above $12M in cost and failed 65% of the time. [Cited in
Ericson 2006] Such statistics do not indicate that every waterfall-based data ware-
housing project is destined to fail. However, if the approach was as robust as people
often assume, plan-driven, big-bang project methods should have achieved a much
higher success rate in the 40 years since Royce’s paper first defined the approach.
Unfortunately, there is a good reason to believe that waterfalls will remain a
very risky manner in building large systems: the specifications flowing into any
one of its phases will always contain flaws. Being only human, the individuals pre-
paring these enormous artifacts will have less than perfect foresight, especially for
companies situated in a world market that is constantly changing. Moreover, in this
age of increasing global competition, the engineers producing these specifications
are also frequently overworked and given too little time to thoroughly research a
project’s problem domain. In this reality, developers working from these specifica-
tions will steadily encounter the unexpected. Lamentably, waterfall methods con-
tain no back step that allows the developers to substantially revisit requirements
if they encounter a major oversight while coding. Testing, compressed to fit into
Copyright © 2012. Elsevier Science & Technology. All rights reserved.

the very last phase of development, cannot call for a major reconsideration of a
system’s design. With plan-driven project management, the team is locked into a
schedule. They must hurriedly span the gaps between their specifications and the
actual situation because, according to plan, work must move steadily downstream
so that the project meets its promised delivery date.
Rather than mitigating project risk, the waterfall approach actually inflates it.
Aiming for a big design up front, waterfall processes are heavy with documentation
and encumbered with numerous reviews in an effort to avoid oversights. However,
large specification documents acquire their own inertia by virtue of the long proce-
dures needed to update and reapprove them. With this inertia, errors committed dur-
ing requirements or design phases get “baked into” the application because it is either
too expensive or too late to substantively rework specifications when flaws are dis-
covered in them. The phases cannot validate nor contribute to one another’s success,
making each phase a single point of failure. The entire project will either succeed or

Hughes, R. (2012). Agile data warehousing project management : Business intelligence systems using scrum. Elsevier Science & Technology.
Created from bibliouniminuto-ebooks on 2023-09-11 21:39:19.
 10 CHAPTER 1 What Is Agile Data Warehousing?

fail based on the quality of the requirements package taken in isolation, then the qual-
ity of the design taken alone, and then finally the quality of the coding as a whole.
Because little flaws cannot be simply corrected when they are discovered, their con-
sequences get amplified to where any one of them can threaten the entire undertaking.
In this light, an iterative approach mitigates risk in an important way. Scrum vali-
dates the engineering of each small slice of the project by pushing it through require-
ments, design, coding, and testing before work on the next set of stories from the
backlog begins. Errors detected in any of these engineering steps can provide important
guidance on how the team should pursue the remainder of the project. Scrum practition-
ers note that the method pushes errors to the surface early in the process while there is
still time to correct assumptions and the foundations of the project’s coding. (See, for
example, [Vingrys 2011].) In contrast to the tendency of a waterfall to allow errors to
undermine a whole project, the agile approach works to contain failures to just one pro-
ject slice at a time.
Being large undertakings by nature, data warehousing initiatives pursued with
a single-delivery paradigm demonstrate waterfall’s whole project failure pattern all
too often. A quick sampling of horror stories shared with the author by prospective
customers seeking a better approach to building business intelligence systems illus-
trates the point well:
l A major waste management company complained “We hired one of the world’s
top systems integrators to build our data warehouse. It cost us $2 million and
3 years of time. All we got was one report. Seems like a very expensive report.”
l A Fortune 500 telecommunications company related with alarm “We purchased
a ‘ready-to-use’ enterprise data model from our data warehouse appliance ven-
dor. Once it was in place, their consultants told us they would need 2 years to
customize it for our company and 3 years to populate it with data before we
could build our first dashboard.
l A Fortune 50 pharmaceuticals company lamented “Our electronic data manage-
ment project required 150 people for over 3 years and got so expensive it was
starting to hurt our share price.”
Copyright © 2012. Elsevier Science & Technology. All rights reserved.

When large, plan-driven projects begin to fail, the managers start taking extraor-
dinary measures to rebalance budgets, deadlines, and intended features. As depicted
in Figure 1.2, such a project starts out with a firm forecast of cost, time, and scope.
As it becomes stressed by nasty surprises, time and cost estimates start to climb,
forcing sponsors to steadily accept less scope in order to keep the project afford-
able. After several such crises and delays, the company is eventually grateful to
push even a small subset of functions into production and declare some level of
victory, although the functionality delivered is far less than promised and the cost
often double or more than planned for.
Data warehousing/business intelligence departments turn this all-too-common
pattern of increasing costs and shrinking deliverables into a cycle of disappointment
because, still believing that a big design up front is the best strategy, they repeat this
basic approach for the next project. They believe the last project failed because they

Hughes, R. (2012). Agile data warehousing project management : Business intelligence systems using scrum. Elsevier Science & Technology.
Created from bibliouniminuto-ebooks on 2023-09-11 21:39:19.
 The “disappointment cycle” of many traditional projects 11

Fewer Features Promised

as project Progresses
Est. After
Crisis 2

Value of Features Delivered

Original
Cost Estimate
Est. After
Crisis 1

“Realistic”
Cost Estimate

Time

FIGURE 1.2
All too often, waterfall methods yield to the project “disappointment cycle.”

did not manage it carefully enough, and start the next project with even more exten-
sive planning and detailed design.
We can rule out unique occurrences of unusual circumstances as the primary cause
for failed projects such as the ones sketched earlier. Unlike Tolstoy’s unhappy fami-
lies, failed waterfall projects in data warehousing all seem to be miserable for the same
basic reasons. On an average engagement, the project manager specifies work pack-
ages in excruciating detail with 5000 lines or more in the project plan. He directs and
cajoles the developers relentlessly. Everyone works nights and weekends, and still the
Copyright © 2012. Elsevier Science & Technology. All rights reserved.

project runs months behind and millions over budget, delivering only a fraction of
what was promised. When the after-project, lessons-learned session is convened, par-
ticipants will voice the same rueful observations heard on so many previous efforts:
l We should have done better at gathering requirements.
l The business didn’t know what they wanted because they had never seen a
warehouse before.
l Design took too long, leaving no time for careful coding.
l Conflicting definitions for data elements made designs worthless.
l Coding went too slow so testing got started far too late to catch some crucial
defects.
l Programmers didn’t build what was specified.
l Dirty data created nightmares that undermined our schedule.
l By the time we finally delivered, the business had changed and the customer
wanted something else besides what we built.

Hughes, R. (2012). Agile data warehousing project management : Business intelligence systems using scrum. Elsevier Science & Technology.
Created from bibliouniminuto-ebooks on 2023-09-11 21:39:19.