SOFTWARE ENGINEERING TECHNIQUES
LESSON 16:
Topics Covered
Reliability concept, Reliability and failure intensity, Uses of
reliability studies, Reliability models, Macro models, Criteria for
a good model
Objectives
Upon completion of this Lesson, you should be able to:
• Know what is reliability concept
• Know what are the uses of reliability studies
• Know what is a reliability model – Macro model and the
criteria for a good model.
Reliability Concept
The definition that we will present here for software reliability is
one that is widely accepted throughout the field. “It is the
probability of failure-free operation of computer program for a
specified time in a specified environment”. For example, a
time-sharing system may have a reliability of 0.95 for 10 hr,
when employed by the average user. This system, when
executed for 10hr, would operate without failure for 95 of these
periods out of 100. As a result of the general way in which we
defined failure, not that the concept of software reliability
incorporates the notion of performance being satisfactory. For
example, excessive response time at a given load level may be
considered unsatisfactory, so that a routine must be recorded in
more efficient form.
Failure intensity is an alternative way of expressing reliability.
We just gave the example of the reliability of a particular system
being 0.95 for 10 hr of time. An equivalent statement is that
the failure intensity of 0.05 failure/hr. Each specification has its
advantages. The failure intensity statement is more economical,
as you only have to give one number. However, the reliability
statement is better suited to the combination of reliabilities of
components to get system reliability. If the risk of failures at
any point in time is of paramount concern, failure intensity may
be the more appropriate measure. Such would be the case for a
nuclear power plant. When proper operation of system to
accomplish some function with time duration is required
reliability specification is often best. An example would be a
space flight to the moon. Fig. 6.5 shows how failure intensity
and reliability typically vary during a test period, as failure are
removed; Note that we define failure intensity, just like we do
reliability, with respect to a specified environment.
Uses of Reliability Studies
Pressures have been increasing for achieving a more finely tuned
balance among product and process characteristics, including
reliability. Trade offs among product components with respect
to reliability are also becoming increasingly important.
Reliability and Failure Intensity
© Copy Right: Rai University
3E.253
1.0
Reliability
Failure
Intensity
Reliability
Time [hr]
Thus an important use of software reliability measurement in is
in system engineering. However, there are at lease four other
ways in which software reliability measures can be of great value
to the software engineer, manager, or user.
First, you can use software reliability measures to evaluate
software engineering technology quantitatively. New techniques
are continually being proposed for improving the process of
developing software, but unfortunately they have been exposed
to little quantitative evaluation. Because of the inability to
distinguish between good and bad, new technology has often
led to a general resistance to change that is counter productive.
Software reliability measures offer the promise of establishing
at least one criterion for evaluating the new technology. For
example, you might run experiments to determine the decrease
in failure intensity (failures per unit time) at the start of system
test resulting from design reviews. A quantitative evaluation
such as this makes the benefits of good software engineering
technology highly visible.
Second, software reliability measures offer you the possibility of
evaluating development status during the test phases of a
project. Methods such as intuition of designers of test team,
percent of tests completed and successful execution of critical
functional tests have been used to evaluate testing progress.
None of these have been really satisfactory and some have been
quite unsatisfactory. An objective reliability measure (such as
failure intensity) established from test data provides a sound
means of determining status. Reliability generally increases with
the amount of testing. Thus, reliability can be closely linked
with project schedules. Furthermore, the cost of testing is
highly correlated with failure intensity improvement. Since two
of the key process attributes that a manager must control are
schedule and cost, reliability can be intimately tied in with
project management.
Third, one can use software reliability measures to monitor the
operational performance of software and to control new
59
 features added and design changes made to the software. The
SOFTWARE ENGINEERING TECHNIQUES
reliability of software usually decreases as a result of such
changes. A reliability objective can be used to determine when,
and perhaps how large, a change will be allowed. The objective
would be based on user and other requirements. For example,
a freeze on all changes not related to debugging can be imposed
when the failure intensity rises above the performance objective.
Finally, a quantitative understanding of software quality and the
various factors influencing it and affected by it enriches into the
software product and the software development process. One
is then much more capable of making informed decisions.
Reliability Models
To model software reliability one must first consider the
principal factors that affect it: fault introduction, fault removal,
and the environment. Fault introduction depends primarily on
the characteristics of the developed code (code created or
modified for the application) and development process
characteristics, which include software engineering technologies
and tools used and level of experience of personnel. Note that
code can be developed to add features or remove
faults. Fault removal depends upon time, operational profile,
and the quality or repair activity. The environment directly
depends on the operational profile. Since some of the forego-
ing are probabilistic in nature and operate over time, software
reliability models are generally formulated in terms of the
random processes. The models are distinguished from each
other in general terms by the nature of the variation of the
random process with time.
A software reliability model specifies the general form of the
dependence of the failure process on the factors mentioned.
We have assumed that it is, by definition, time based (this is
not to say that non-time-based models may not provide useful
insights). The possibilities for different mathematical forms to
describe the failure process are almost limitless.
As with any emerging discipline, software reliability has
produced its share of models. Software reliability models are
propounded to assess reliability of software from either
specified parameters, which are assumed to be known, or from
software-error generated data.
The past few years have seen the introduction of a number of
different software reliability models. These models have been
developed in response to the urgent need of software engi-
neers, system engineers, and managers to quantity the concept
of software reliability.
After some thought about how to construct a software
reliability model, two viewpoints emerge – the macro approach
and the micro approach. In the macro approach we ignore the
differences between types of statements, details of the control
structure, etc. We step back and examine the number of
instructions, the number of errors removed, and the overall
details of the control structure and base our model on these
features. Constants for the model can be evaluated from data
on past systems as well as analysis of test data on the software
being developed. In the case of micro approach, a detailed
analysis of the statements and the control structure is per-
formed, leading to a detailed model structure. Most of the
© Copy Right: Rai University
60
models developed to date have been macro models, and the
next sub-section is devoted to a discussion of such models.
Macro-Models
A large number of macro models have been proposed. From a
data-analytic point of view, the objective of the research into
modeling is to find a model that explains failure data well
enough to forecast the future. For credibility, the model should
have natural, meaningful parameters. Each model mathemati-
cally summarizes a set of assumptions the researcher has made
about the phenomenon of software failure and debugging.
The models may give fairly accurate results even though all of
the assumptions are not satisfied. The selection of a model
needs to be justified by the realism of the model’s assumptions
and the model’s “predictive validity”.
The practitioner has several choices:
1. One model can be chosen a priori, and then that model is
only one used.
2. One model can be chosen a priori, but a recalibration
technique is later applied to adapt the model to the project.
3. Several models can be employed. The results can be
combined into a weighted average. The weighted average
with the best goodness-of-fit is used.
It is important not to use too many models in approaches 3
and 4, because two models might fit the failure data well but
disagree on how to extrapolate to the future. There is a Yiddish
proverb, “The man who owns one watch always knows what
time it is: the man who owes two watches is never quite sure”.
It is recommended that software reliability models be compared
by criteria discussed below. It is expected that comparisons will
cause some models to be rejected because they meet few of the
criteria discussed here. On the other hand, there may or may
not be a clear choice between the more acceptable models. The
relative weight to be placed on the different criteria may depend
on the context in which the model is being applied. When
comparing two models, we should consider all criteria simulta-
neously. We should not eliminate models by one criterion
before considering other criteria, except if predictive validity is
grossly unsatisfactory. It is not expected that a model must
satisfy all criteria to be useful.
The proposed criteria include predictive validity, capability,
validity, capability, and quality of assumptions, applicability, and
simplicity. We will discuss each of the criteria in more details in
the following sections.
Predictive Validity
Predictive validity is the capability of the model to predict future
failure behaviour from present and past failure behaviour (that
is, data). This capability is significant only when failure
behaviour is changing. Hence, it is usually considered for a test
phase, but is cant be applied to the operational phase when
repairs are being regularly made.
There are at least two general ways of viewing predictive validity.
These are based on the two equivalent approaches to characteriz-
ing failure random process, namely:
1. The number of failures approach and
2. The failure time approach
3E.253
We may apply various detailed methods, some representing
approximations for predictive validity. It has not been deter-
mined if one is superior at the present time.
The number of failure approach may yield a method that is
more practical to use than the failure time approach. In the
former approach, we describe the failure random process [M (t),
t> 0], representing failures experienced by time t. Such a
counting process is characterized by specifying the distribution
of M(t), including the mean value function h(t).
Assume that we have observed q failures by the end of the time
tq. We use the failure data upto time te (< te) to estimate the
parameters of h(t). Substitution the estimates of the param-
eters in the mean value function yield the estimate of the
number of failures by the time tq. The estimate is compared
with the actually observed number q. this procedure is repeated
for various values of te.
We can visually check the predictive validity by plotting the
relative error against the normalized test time. The error will
approach 0 as te approaches tq. If the points are positive
(negative), the model tends to overestimate (underestimate).
Numbers closer to 0 imply more accurate prediction and hence
better model.
Capability
Capability refers to the ability of the model to estimate with
satisfactory accuracy quantities needed by software managers,
engineers, and users in planning and managing software
development projects or running operational software systems.
We must gauge the degree of capability by looking at the relative
importance of the quantities as well as their number. The
quantities, in approximate order of importance are:
1. Present reliability, mean time to failure (MTTF), or failure
intensity,
2. Expected date of reaching a specified reliability, MTTF, or
failure intensity objective, and
3. Human and computer resource and cost requirements
related to the achievement of the objective.
Any capability of model for prediction of software reliability in
system design and early development phases is extremely
valuable because of the resultant value for system engineering
and planning purposes. We must make these predictions
through measurable characteristics of the software (size,
complexity, structure, etc.), the software development environ-
ment, and the operational environment.
Quality of Assumptions
The following considerations of quality should be applied to
each assumption in turn. If it is possible to test an assump-
tion, the degree to which it is supported by data is an important
consideration. This is especially true of assumptions that may
be common to an entire group of models. If it is not possible
to test the assumption, we should evaluate its plausibility from
the viewpoint of logical consistency and software engineering
experience. For example, does it relate rationally to other
information about software and software development? Finally,
we should judge the clarity and explicitness of an assumption.
These characteristics are often necessary to determine whether a
© Copy Right: Rai University
3E.253
model can be applied to particular software system or project
SOFTWARE ENGINEERING TECHNIQUES
circumstances.
Applicability
An another important characteristic of a model is its applicabil-
ity. We should judge a model on its degree of applicability
across software products that vary in size, structure, and
function. It is also desirable that it be usable in different
development environments, different operational environ-
ments, and different life cycle phase. However, if a particular
model gives outstanding results for just a narrow range of
products or development environments, we should not
necessarily eliminate the model.
There are at least four special situations that are encountered
commonly in practice. A model should either be capable of
dealing with them directly or should be compatible with
procedures that can deal with them. These are:
1. Program evolution,
2. Classification of severity of failures into different
categories,
3. Ability to handle incomplete failure data or data with
measurement uncertainties (although not without loss of
predictive validity).
4. Operation of the same program on computers of different
performance.
Finally, it is desirable that a model be robust with respect to
departures from its assumptions, errors in the data or param-
eters it employs, and unusual conditions.
Simplicity
A model should be simple in three aspects. The most impor-
tant consideration is that is must be simple and inexpensive to
collect the data required to particularize the model. If this is
not the case, we will not use the model. Second, the model
should be simple in concept. Software engineers without
extensive mathematical background should be able to under-
stand the model and its assumptions. They can then determine
when it is applicable and the extent to which the model may
diverge from reality in an application. Parameters should have
readily understood interpretations. This property makes it
more feasible for software engineers to estimate the values of
the parameters when data are not available. The number of
parameters in the model is also an important consideration for
simplicity. It should be pointed out that we need to compare
the number or parameters on a common basis (for example,
don’t include calendar time component parameters for one
model and not another).
Finally, a model must be readily implementable as a program
that is a practical management and engineering too. This means
that the program must run rapidly and inexpensively with no
manual intervention required (does not rule our possibility of
intervention) other than the initial input.
On the basis of the above characteristics of a good software
reliability model we select two models for discussion. Two
models were chosen because each has certain advantages not
possessed by the other. However, the effort required to learn
the application of a model makes presenting more than two a
61
 question of sharply diminishing returns. The models are the
SOFTWARE ENGINEERING TECHNIQUES

basic execution time model and the logarithmic Poisson

execution time model. Both the models have two compo-
nents, named the execution time component and the calendar
time component. Each component will be described with
respect to both models.
We have restricted ourselves to considering well-developed
models that have been applied fairly broadly with real data and
have given reasonable results. The specific forms can be
determined from the general form by establishing the values of
the parameters of the model through either:
1. Estimation – statistical inference procedures are applied to
failure data taken for the program, or
2. Prediction – determination from properties of the software
product and the development process (this can be done
before any execution of the program).
Notes

© Copy Right: Rai University

62 3E.253