What makes software so hard?

Software projects in general have an unimpressive track record when compared to traditional
engineering projects. 1 There are a vast number of reports documenting software projects that were
seriously over budget, over schedule, or even cancelled. Even those projects that eventually manage to
get a product out of the door will in most cases have to spend considerable time doing post release fixes
and corrections.

This paper examines some of the key characteristics that make software development difficult and error
prone. I will explore this topic primarily by contrasting software construction with construction of
traditional concrete, physical engineering systems, such as buildings, bridges, ships, and aircraft. I
would like to add that the difficulties I describe here are those specifically addressed by the IBM®
Rational® approach to iterative development, which you can read about elsewhere in this issue and in
the archives of The Rational Edge ezine. But even if you have enjoyed success through the application of
iterative development techniques in your software projects, I believe it is useful to explore the various
reasons those techniques are so critical.

The track record

The failure of a software project becomes more likely as the size of a project increases. The larger the
project, the more likely is it to become late, over budget, or even get cancelled. Fundamentally, the
initial cost and schedule estimates are frequently way off. This has led some people to believe that the
root cause for failures in the software industry is inadequate management practices for large-scale
software development projects. However, even though many of today's software projects are quite large
in terms of budget, duration, number of people involved, etc. most of them are in fact relatively small
when compared to traditional engineering projects, such as bridge building or dam construction.

Given the far-better track record in terms of staying on plan and on budget for traditional engineering
projects -- even those that are orders of magnitude larger than the largest software projects -- it is
tempting to conclude that the problems of the software industry could be mitigated by applying identical
project management techniques and processes as those used in traditional engineering. The problem is,
the similarities that seem apparent on the surface might only go skin deep. There are key differences
that make software projects much more difficult to plan, estimate, and complete according to plan than
traditional engineering projects.

Major problems in software projects

Commercial software projects 2 are frequently over budget and late, often in the extreme: Budget and
schedule overruns by a factor of two are not unheard of. Many projects seriously in trouble are also
cancelled at a fairly late stage, thus having incurred significant economic loss as well as lost opportunity
costs. Once they are made generally available (GA), software systems often fail to perform according to
specification. Typical shortcomings include missing functionality, poor runtime performance, and bugs.
For most software systems or applications, it will take quite some time, and lots of frustration and effort,
from initial GA until the application eventually stabilizes, hopefully by then behaving approximately as
specified. The typical variance of software projects, in terms of plans versus actuals, is thus very large.

Contrast this situation with that of traditional engineering projects -- even the very largest of traditional
engineering projects, with time spans over many years, multi-billion dollar budgets, and involving
thousands of people are commonly completed very close to the initial plan, both in terms of schedule
and budget. Thus, the variance for these projects is limited to a fairly small range.
The obvious question becomes: Why are software engineers so incapable of accurate estimations?
Design versus production
A key difference between software projects and traditional engineering projects is the proportion of time
and cost spent on the "creative" phase, that is, architecture and design, versus on the production or
manufacturing phase. In traditional engineering, the bulk of time as well as of the cost is spent in
production, while the architecture and design phase only contributes a minor part of time and cost. For
instance, for a housing construction project, or for development of a new model of a car, the "creative"
part of the work might involve a limited number of architects and designers, taking only a couple of
months, while the actual production might involve thousands of people, or even automations, such as
robots, and goes on for many years.

On the other hand, for software projects, the relationship between architecture and design versus
production is not only inverted, but in fact approaches infinity: since the production or manufacturing
costs for software are almost nil (see below), almost all cost for a software project comes from the
workflows dealing with the creative parts -- that is, the invention and design of software.

One can thus define traditional engineering projects as being primarily "production projects," while
software projects are fundamentally "design projects."

Characteristics of production projects

Typical production projects are characterized by being quite standardized in terms of the tasks,
processes, and methods involved. In addition, production projects typically operate on standardized
building blocks, such as the prefabricated construction elements of a building. Production projects are
well suited for mechanization and automation, and they can be predictably managed, traditionally by
Tayloristic approaches of Scientific Management, where each job, task, and step can be described in
detail and accurately measured and optimized for time and cost. These types of projects, with well-
defined mechanistic processes and tasks, are also easy to optimize for cost -- e.g. by relocating or
outsourcing to a low-cost provider.

Characteristics of design projects

Design projects are completely different. The jobs and tasks involved in design projects are far from
mechanical, and do not easily lend themselves to Tayloristic or other types of a priori, fully predictive
approaches. Instead, design projects demand long periods of discovery and invention, involving a very
skilled staff capable of performing qualified tasks. Developers must possess deep and broad knowledge
of both the problem and solution domains, they must be capable of intellectual work, and they must be
able to perform well in team communication and collaboration. Design projects also demand a different
approach to planning and management.

The "factory approach" to software development is bound to fail

Many large software development organizations fail to see or acknowledge the difference between
design and production projects, and attempt to use business and management approaches that might be
well suited for production and manufacturing projects, but which will not work well for software projects,
which are non-mechanical, far from trivial, and fully dependent on the quality of the people involved.

Compared to production projects, design projects demand a fundamentally different approach. Costs can
ultimately not be kept down by hiring low-pay "human automations," capable only of well defined
mechanical tasks. Time-to-market cannot be improved by adding more of these "automations" to the
project, or by asking them to work "faster." Most of the tasks involved are simply too complex for
lending themselves to mechanization. For design projects to be successful, all staff involved must be
very knowledgeable, skilled, and experienced in a broad range of software engineering topics, ranging
from understanding the problem domain and customer needs, all the way down to the specifics of the
implementation platform. Furthermore, they also need to be motivated.

The common attempts to use business and management practices from traditional production projects
on software projects -- focusing on cost reductions and standard business optimizations of workflows
which are far from mechanical, quantifiable, or trivial -- without understanding the complexities involved
in a typical design project, has led many software development organizations to invest way too little on
the key analytical and creative workflows of their projects. For instance, methodical use of prototyping,
modeling, or simulation is rare. This often results in architectures and designs being less than adequate,
making implementation troublesome, and more expensive than planned, because a large number of
"fixes" are needed at a later stage. This typical pattern of "quick and dirty" analysis and design, followed
by "jump to implementation," accompanied by "trial and error" based quality control would be totally
unthinkable in traditional engineering projects, where the cost of not getting the architecture and design
correct in the first place is totally prohibitive. Few organizations involved in traditional engineering can
cope with vast numbers of their products being recalled due to malfunctions. Unfortunately, this pattern
is very common in software development: "We'll fix the problem in a later release."

Too little emphasis has been placed on the quality of staff responsible for the truly innovative, creative,
and hard aspects of software development, which unfortunately comprise most of the workflows of a
typical software development project. Even though some software projects, particularly those in
maintenance phase, can be successfully done by a team of "average Joe's," any new development
efforts are likely to be dependent on very skilled staff for their success.

The planning of design projects needs to be adaptive in order to cope with the many unknowns and
changes. It is impossible, particularly in the early phases, to accurately estimate the effort involved in
developing any non-trivial piece of software.

"Supporting" versus "Doing" disciplines

For the rest of this discussion, it will be useful to distinguish between two categories of activities
involved in any development or construction project: "supporting" and "doing." The former deals with
traditional project management type of activities, such as planning, scheduling, staffing, estimation,
monitoring and reporting, while the latter includes all those activities that deal with the actual
construction of the system, that is, the various "crafts" involved in designing and building something,
such as analysis, design, construction, and implementation.

Supporting disciplines
On the surface, it might appear that a software project shares the same supporting activities and
workflows as a traditional engineering project. Activities such as requirements gathering, project
organization and planning, staffing, estimation, monitoring, and reporting all occur in both types of
projects. There are numerous methods and tools available for supporting these activities, and traditional
engineering projects have a proven track record in completing their work more or less as planned.

The software industry has also focused on the supporting disciplines for quite some time. There is a
plethora of processes, methods, and tools specifically targeted towards the software industry, and most
organizations have made significant investments in shaping up their supporting disciplines.

Still, few software projects perform as planned; initial as well as later estimates tend to be way off,
demanding frequent re-planning, which unfortunately also tends to be overly optimistic.

What is it then that makes planning of software projects so hard?

Doing disciplines
Abstract versus concrete things

Obviously, it is among the doing disciplines that we find the most obvious differences between software
and traditional engineering. Even though there are some similarities -- particularly in the architecture
workflow, where architects in software as well as in traditional engineering typically use models -- the
remaining activities are very different: While engineers in traditional disciplines often operate on
standardized physical entities, software engineers operate exclusively on intangible, abstract things.
Instead of welding pieces of metal, software engineers join stuff by writing code. Instead of building
physical models to scale, or using mathematical models, software designers typically "model" by writing
high level code. Instead of placing building blocks under measured forces to test their structural
integrity, software engineers write code for their testing. Regardless of workflow, software engineers are
all the time working with non-tangible, abstract, and flexible things -- pure mindware.

Method of construction

Software systems differ from traditional systems in their methods of construction or implementation.
Physical systems are to a large extent constructed by a process of assembly of standardized
components. Software systems, on the other hand, are fundamentally built by a process of discovery,
invention, and creation of new components. Even though the software industry has been pursuing reuse,
standardization of interfaces, and componentization, the success has so far been limited.

Physical systems
A physical system -- a building, a bridge, a car, an aircraft, a ship, or a computer -- exists in the
physical world. Besides occupying 3-dimensional space, their "atomic structures" -- that is, the individual
components or parts of these types of systems -- are constructed from other physical entities.
Regardless of whether we consider the physical systems themselves or their individual constituent parts,
they share some common physical properties, such as weight, length, width, height, etc. They also have
in common their tangibility: We can see, touch, feel, and move them around, and we can also easily
play and experiment with them, using whatever means the physical world and its laws makes available
for us.

We can also create scaled models of these physical systems, and subject these models to experiments
and tests governed by the same physical laws that the real, full-scale systems will be governed by,
thereby gaining valuable information aiding us in building the real thing. This is partly because the laws
governing physical systems, as well as the systems themselves, also primarily exhibit continuous
characteristics. This makes the behavior of these systems reasonably predictable, quantifiable, and
measurable to a much higher degree than for software systems, which are fundamentally discrete,
where a small change to an input can result in a hugely disproportionate change to the outputs.

Humans are quite familiar with physical systems -- there's no need to be an engineer in order to
understand the practicalities of physical systems and laws. From the day we're born, we have personal
experience in coping with the physical world. We learn first-hand about the principles of gravity the first
time we fall down, and we experience many other physical laws early on in our lives.

Software systems
Software, on the other hand, is not a physical, tangible thing. Instead, for all practical purposes,
software is a purely abstract, conceptual thing, existing only in the minds of humans. Sure, we can place
the atomic structure of software -- its individual bits -- onto media such as a disc drive or CD, thereby
creating a physical representation of it, but a software system itself is purely conceptual and intangible.
As such, it's not subject to any of the stringent laws governing physical systems. This lack of laws or
first principles makes construction of software systems as much an art as a science. For software
development there are few if any useful analytical methods available. Instead, most of software
development is fundamentally based on synthetic approaches, where experience, skill and use of "best
practices" in chosen methods and processes become the only available determinants for success or
failure.

In addition, being intangible greatly limits our possibilities to "play and experiment" with our software
system -- we cannot touch it, feel it, nor move it around -- all we can do is to think about it.

Lack of first principles (laws)

In software, there are no first principles or laws governing the creation and assembly of software. In
contrast, traditional engineering, for instance building construction, is constrained by well defined and
well known physical laws. This lack of laws makes software the ultimate in flexibility, that is, there are
very few constraints on what you can do with software, which is a good thing. Unfortunately, the almost
infinite degrees of freedom resulting from the lack of laws, and the inherent flexibility also means that
software professionals have very little assistance in terms of a priori constraints on how to do their work.
A construction engineer is fully aware of the laws that the physical world places on his designs, but the
situation for the software engineer is very different: he or she operates in a world which is essentially
free of any a priori limitations -- the only constraints for the designs in our world are those imposed by
ourselves and the methodologies, processes, and tools we use.

In a sense, creating software is far more closely related to writing a novel than to any traditional
engineering effort -- an author is free to construct whatever type of story, limited only by his
imagination and whatever facts may be necessary to the narrative. Furthermore, the author must
construct the architecture of the story in his head, and to make sure the design and implementation of
the story is consistent with the overall architecture. The quality of the author's product is to a high
degree based on whether he succeeds in keeping the details of the story consistent with the
architecture.

The abstract system that an author constructs exists only in his mind, and the physical manifestation --
the book -- is only a convenient means of storage and distribution. This is very much the same situation
as for software.

Marginal cost of systems

Traditional engineering systems have a significant marginal cost. For instance, having just successfully
built a bridge, building an identical one will cost almost as much as the first. This is very different for
software, where the marginal cost for creating a copy of an existing system is almost nil. The fact that
marginal cost for software is almost nonexistent establishes two other key differences between
traditional engineering and software:

• Many software systems solve unique problems never tackled before.

• There is a strong tendency to extend existing software systems beyond their capabilities.

Actually, the bullets above are two sides of the same coin: because the marginal cost of software is nil,
it's very tempting for organizations developing software to solve a new problem using a legacy system
as a foundation -- that is, extending and modifying an already existing system to cope with the new
problems. Within reason, this is a sound and effective practice, but when taken too far, this approach
can eventually lead to a practice of patching and hacking, quickly destroying the intended architecture,
making further extension and maintenance impossible.

Secondly, because there is a strong temptation to extend legacy code to deal with the creation of similar
but extended systems, the remaining software systems actually created from scratch are necessarily
going to be first-ever systems dealing with previously untouched problems, demanding the organization
to stray into the unknown, by stretching existing technology, as well as methods and tools, to their
limits. These first-ever projects have many similarities with pure research projects, where it is very
difficult to estimate and predict the outcome. This tendency for software projects to constantly work on
the edges of technology is caused by the seemingly never ending steady increase of the capabilities of
modern digital technology, which tends to push the envelope for what organizations will attempt to do
with software. Furthermore, another consequence of this law of Moore's is that our foundation, that is,
the platforms and fundamental technologies we use on top of the platform, are never stable -- they are
constantly changing, demanding the same rate of change in our software development projects.
Therefore, software development teams must learn to live in, and cope with a very dynamic world where
everything constantly changes 5 .

Flexibility -- a blessing or a curse?

The inherent flexibility of software in combination with the ever increasing power of digital technology
allows us to continue solving unprecedented problems each day using software. Because of its flexibility,
we can mold software into any form to do almost anything. Flexibility thus expands the "what" we can
use software for almost indefinitely. Unfortunately, flexibility also is one of the chief culprits of the
problems for software projects, because it applies equally well to the "how" of software engineering.
That is, software can not only be molded into almost any form, but we can also perform the molding
process itself in almost any way we like. Because there are so few constraints governing software, the
number of possible ways to do something is almost unlimited. Therefore, software development
organizations have almost infinite possibilities for choosing methods, tools and processes to use, and
often there exist many different ways of working within the same organization.

As an analogy, contrast building something using Lego, versus building something using Play-Doh. Even
though we can combine the individual bricks of Lego in almost infinite number of ways producing a large
variety of "systems," we are still constrained under the very strict "laws" provided by the Lego
framework and its inherent principles on how Lego blocks can be put together. The same situation exists
for physical systems. Play-Doh, on the other hand, is very similar to software -- not only can it be used
to construct almost any structure, but it also allows a constructor to do it in almost any way he pleases,
there are very few constraints on the "how's" of Play-Doh.

Requirements for prediction and estimation

In order to accurately predict and estimate a project in terms of duration, effort, etc., all the various
tasks and steps involved must not only be identified and scheduled, they also need to be accurately
quantifiable in several dimensions, such as duration, difficulty, resource consumption, and dependencies.
Basically, accurate prediction and estimation requires the existence of an algorithm, or a step-by-step
problem solving procedure with a finite number of well defined steps. For traditional engineering, where
there typically exists vast pools of experience building similar or identical systems, under the familiar
physical laws, the typical tasks and steps are well known, and estimation can be fairly accurately done.
However, for software projects, particularly the first-ever projects, such data simply does not exist. Nor
is there a vast pool of applicable analytical or quantitative methods we could use.
Software also lacks a well defined atomic unit of work -- even though we can apply various measures,
such as lines-of-code, number of classes, number of function points, number of bugs, etc. The problem
is, these measures are neither atomic nor comparable in terms of effort. That is, a sequence A with 10
lines of code might have a very different degree of complexity and difficulty compared to sequence B of
10 lines of code written in the same language. Similarly with bugs -- some bugs are orders of magnitude
more tricky to identify and correct than others, but we really have no way to consistently identify ahead
of time which ones will be the tricky ones.

Estimation and prediction of software projects becomes guesswork, where only conservative assessment
and previous experience can reduce the variables in play. This is very similar to pure research projects,
where many different paths towards a solution are tried and abandoned before a successful path can be
found.
It has also been proven that the productivity of individual software developers has very large variance.
Productivity between individuals in the same development team can differ by orders of magnitude.
Furthermore, because of software's inherent flexibility, we can build a system in almost any way we like,
using whatever means we like, further exacerbating the problems of accurate estimation.
All these factors combined contribute to making estimation and prediction less than accurate. In
essence, fully accurate prediction and estimation of projects or tasks demand a Tayloristic approach of
Scientific Management, which only is applicable to trivial, mechanical tasks. Typical software projects
are, for reasons discussed above, far from being trivial or mechanical.

Summary and conclusion

Software is hard for a number of fundamental reasons:

• Software projects are design projects, not production projects.

• Because software is pure mindware, software development is more of an art or craft than a
science or engineering discipline, making the "doing" disciplines of software development
difficult to perform, monitor, and measure. Productivity differences between individuals are
huge.
• Software lacks laws and first principles.
• Software lacks a measurable standard unit of work.
• Given software's flexibility, the "supporting" disciplines of software projects -- planning,
estimation, prediction and observation -- are quite difficult.
• Moore's law makes the foundations of software ever changing -- there's no "permanent"
platform to build on.

Even though it might be tempting to consider software engineering as "just another engineering
discipline," and thus jump to the conclusion that software can be developed using the same business,
management, and engineering principles as traditional engineering, there are some unique
characteristics that make software and software development not only special, but also more difficult to
estimate and monitor than traditional engineering projects.

Software development, with its inherent characteristics, existing in a very dynamic world governed only
by Moore's law, will remain more art than a science for a long time to come. We can only hope to tackle
the problems in software development by acknowledging that software is different, and understanding
the consequences of these differences.

Any attempts to regard software development as just another form of traditional engineering will
generally not deliver the desired results. Only by having a good understanding of what makes software
different, can we start addressing the unique issues associated with software development. Iterative
development, modeling, requirements management, and early-and-often stakeholder involvement are
among the techniques available to software teams who recognize this critical difference and are looking
for a new way forward.

Notes
1
Examples of successful engineering projects, at least with respect to schedule, are all the construction
projects for Olympic Games -- as far as I know, all Olympic Games have started on time, in
fundamentally new premises. Another successful example, the Sears Tower: built 72-74, 160M USD, 60
subcontractors, 2400 workers at peak. More info, see http://skyscraperpage.com/cities/?buildingID=5
2
Interestingly, there's some evidence indicating that non-commercial software projects, even very large
scale ones, such as the Eclipse Open Source project, or the GNU/Linux projects, have a much better
track records than fully commercial ones.
3
A car factory assembly line is a good example. Most of the tasks are now performed not by humans but
by robots.
4
http://en.wikipedia.org/wiki/Scientific_management
5
Panta Rei

Resources

• Participate in the discussion forum.

• A new forum has been created specifically for Rational Edge articles, so now you can share your
thoughts about this or other articles in the current issue or our archives. Read what your
colleagues the world over have to say, generate your own discussion, or join discussions in
progress. Begin by clicking HERE.

• Global Rational User Group Community

About the author

Tommy Lennhamn spent his formative years as a systems programmer with Philips
Elektronikindustrier, then the leading Swedish Aerospace and Defense contractor. He joined
Rational Scandinavia AB in 1991, as a Technical Consultant, and has exclusively focused on
embedded, real-time, and event-driven systems, ranging from small, microcontroller-based
gizmos to huge, multi-million line distributed systems with several hundred developers and
hundreds of processing units. He is currently a member of IBM Rational's Ericsson
Corporate Account Strategy Team, working very closely with Ericsson R and D and working
as a liaison officer between Ericsson Software Research and IBM Research, looking into
future directions of systems and software development.