124 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 2, NO.

2, APRIL-JUNE 2005

A Comprehensive Model
for Software Rejuvenation
Kalyanaraman Vaidyanathan, Member, IEEE, and Kishor S. Trivedi, Fellow, IEEE

Abstract—Recently, the phenomenon of software aging, one in which the state of the software system degrades with time, has been
reported. This phenomenon, which may eventually lead to system performance degradation and/or crash/hang failure, is the result of
exhaustion of operating system resources, data corruption, and numerical error accumulation. To counteract software aging, a
technique called software rejuvenation has been proposed, which essentially involves occasionally terminating an application or a
system, cleaning its internal state and/or its environment, and restarting it. Since rejuvenation incurs an overhead, an important
research issue is to determine optimal times to initiate this action. In this paper, we first describe how to include faults attributed to
software aging in the framework of Gray’s software fault classification (deterministic and transient), and study the treatment and
recovery strategies for each of the fault classes. We then construct a semi-Markov reward model based on workload and resource
usage data collected from the UNIX operating system. We identify different workload states using statistical cluster analysis, estimate
transition probabilities, and sojourn time distributions from the data. Corresponding to each resource, a reward function is then defined
for the model based on the rate of resource depletion in each state. The model is then solved to obtain estimated times to exhaustion
for each resource. The result from the semi-Markov reward model are then fed into a higher-level availability model that accounts for
failure followed by reactive recovery, as well as proactive recovery. This comprehensive model is then used to derive optimal
rejuvenation schedules that maximize availability or minimize downtime cost.

Index Terms—Availability, measurement-based dependability evaluation, semi-Markov reward models, software aging, software
rejuvenation, workload characterization.

1 INTRODUCTION

T HEphenomenon of “software aging,” one in which the

state of the software or its environment degrades with
time, has been reported and investigated by several studies
[17], [25]. The primary causes of this degradation are the
exhaustion of operating system resources, data corruption,
and numerical error accumulation. Eventually, this may
lead to performance degradation of the software or crash/
hang failure or both. Some common causes of “software
aging” are memory bloating and leaking, unreleased file-
locks, data corruption, storage space fragmentation, and
accumulation of round-off errors [17]. Since aging leads to
transient failures in software systems, environment diver-
sity [28], a software fault tolerance technique that is based
on changing the operating environment, can be employed
proactively to prevent degradation or crashes. This involves
occasionally stopping the running software, “cleaning” its
internal state or its environment, and restarting it. Such a
technique known as “software rejuvenation” was proposed
by Huang et al. [25]. This counteracts the aging phenom-
enon in a proactive manner by removing the accumulated
error conditions and freeing up operating system resources.
Garbage collection, flushing operating system kernel tables
. K. Vaidyanathan is with the Scalable Systems Group, Sun Microsystems,
San Diego, CA 92121. E-mail: kalyan.vaidyanathan@sun.com.
. K.S. Trivedi is with the Department of Electrical and Computer
Engineering, Duke University, Durham, NC 27708-0294.
E-mail: kst@ee.duke.edu.
Manuscript received 27 Jan. 2004; revised 17 Nov. 2004; accepted 31 Mar.
2005; published online 3 June 2005.
For information on obtaining reprints of this article, please send e-mail to:
tdsc@computer.org, and reference IEEECS Log Number TDSC-0022-0104.
1545-5971/05/$20.00 ß 2005 IEEE
and reinitializing internal data structures are some actions
by which the internal state or the environment of the
software can be cleaned.
Although we use the by-now-established phrase “soft-
ware aging,” it should be clear that no deterioration of the
software system per se is implied but rather, the software
appears to age due to the degradation of the operating
environment (for example, gradual depletion of resources)
[5]. Likewise, “software rejuvenation” actually refers to
rejuvenation of the environment in which the software is
executing.
Software rejuvenation has been implemented in several
different systems. The AT&T billing applications [25] and
telecommunications switching software [2] both have some
form of rejuvenation implemented. Proactive fault manage-
ment of this type is not only used in high-availability
software, but also in safety-critical systems. Preventive
maintenance, to maximize the probability of successful
mission completion, has been proposed for spacecraft
systems [40]. Proactive fault management was also recom-
mended for the Patriot missiles’ software system [33],
where the recommendation was for the computer system to
be switched off and on every eight hours.
More recently, two kinds of rejuvenation policies have
been implemented in cluster systems to improve perfor-
mance and availability, by taking advantage of the failover
feature [9], [27], [45]. In the periodic policy, rejuvenation of
the cluster nodes is done in a rolling fashion after every
deterministic interval. In the prediction-based policy, the
time to rejuvenate is estimated based on the collection and
statistical analysis of system data. Microsoft’s IIS 5.0 Web
server features a software rejuvenation policy known as
Published by the IEEE Computer Society
VAIDYANATHAN AND TRIVEDI: A COMPREHENSIVE MODEL FOR SOFTWARE REJUVENATION
Fig. 1. Our approach for model construction and solution.
process recycling. The popular Web server software,
Apache, implements a form of rejuvenation by killing and
recreating processes after a certain numbers of requests
have been served [31]. Software rejuvenation is implemen-
ted in specialized transaction processing servers [8].
Rejuvenation has also been proposed for cable and DSL
modem gateways [13] in Motorola’s Cable Modem Termi-
nation System [32] and in middleware applications [7] for
failure detection and prevention. Automated rejuvenation
strategies have been proposed in the context of self-healing
and autonomic computing systems [23].
In this paper, we first describe how to include faults
attributed to software aging in the framework of Gray’s
software fault classification (deterministic and transient)
and study the treatment and recovery strategies for each of
the fault classes. This helps us understand the nature of
software faults and their impact on system availability and
performance and aid in choosing the best possible recovery
strategy when a fault is triggered. We then construct a semi-
Markov reward model based on workload and resource
usage data collected from the UNIX operating system
(shown in Fig. 1). For this model, we first identify different
workload states using statistical cluster analysis, then we
estimate the transition probabilities and the state sojourn
time distributions from the measured data. The measured
sojourn time distributions fit either hypoexponential or
hyperexponential theoretical distributions leading to a
semi-Markov model rather than a Markov model. Corre-
sponding to each resource, a reward function is then
defined for the model based on the rate of resource
depletion in different states. The workload-based model
results are then compared with the results obtained by
using the purely time-based approach [17]. The model is
then solved to obtain estimated times to exhaustion for the
125
resources. The result from the semi-Markov reward model
are then fed into a higher-level availability model that
accounts for failure followed by reactive recovery, as well as
proactive recovery. This comprehensive model is then used
to derive optimal rejuvenation schedules that maximize
availability or minimize downtime cost.
The main contributions of this paper are 1) a measure-
ment-based model for capturing the effect of system
workload on operating system resources, 2) investigating
the effect of workload on system resources, particularly
with respect to exhaustion and 3) development of a
comprehensive model used to compute optimal rejuvena-
tion schedules that maximize availability or minimize
downtime cost.
The rest of this paper is organized as follows: In Section 2,
we show how to include faults attributed to software aging
into the framework of Gray’s classification. Section 3 gives
an overview of related work and brings out some
differences between this work and earlier work. The
experimental setup and data collection are briefly explained
in Section 4. Section 5 discusses the clustering approach and
state transition model for system workload characterization.
Modeling of resource depletion is explained in Section 6.
The workload model behavior and results are discussed in
Section 7. Section 8 combines the measurement-based
workload model with an availability model to obtain an
optimal rejuvenation schedule. Section 9 summarizes the
contributions of the paper.
2 CLASSIFICATION AND TREATMENT OF
SOFTWARE FAULTS
In this section, we describe how we can include software
faults attributed to software aging into Jim Gray’s fault
classification [20] and discuss the various fault tolerance
techniques to deal with these faults in the operational phase
of the software. Particular attention is given to environment
diversity, explaining its need, various approaches, and
methods in practice.
2.1 Classification of Software Faults
Faults, in both hardware and software, can be classified
according to their phase of creation or occurrence, system
boundaries (internal or external), domain (hardware or
software), phenomenological cause, intent, and persistence
[4]. In this section, we restrict ourselves to the classification
of software faults based on their phase of creation.
Some studies have suggested that since software is not a
physical entity and, hence, not subject to transient physical
phenomena (as opposed to hardware), software faults are
permanent in nature [24]. Some other studies classify
software faults as both permanent and transient. Gray [20]
classifies software faults into Bohrbugs and Heisenbugs.
Bohrbugs are essentially permanent design faults and,
hence, almost deterministic in nature. They can be
identified easily and weeded out during the testing and
debugging phase (or early deployment phase) of the
software life cycle. A software system with Bohrbugs is
analogous to a faulty deterministic finite state machine.
Heisenbugs, on the other hand, are design faults that
behave in a way similar to hardware transient or
126 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING,
Fig. 2. Venn diagram of software fault types.
intermittent faults. Their conditions of activation occur
rarely or are not easily reproducible. These faults are
extremely dependent on the operating environment (other
programs, OS, and hardware resources). Hence, these
faults result in transient failures, i.e., failures that may
not recur if the software is restarted. Some typical
situations in which Heisenbugs might surface are bound-
aries between various software components, improper, or
insufficient exception handling and interdependent timing
of various events. It is for this reason that Heisenbugs are
extremely difficult to identify through testing. In fact, any
attempt to detect such a bug may alter the operating
environment enough to change the symptoms. A software
system with Heisenbugs is analogous to a faulty non-
deterministic finite state machine. A mature piece of
software in the operational phase, released after its
development and testing stage, is more likely to experience
failures caused by Heisenbugs than due to Bohrbugs. Most
recent studies on failure data have reported that a large
proportion of software failures are transient in nature [20],
[21], caused by phenomena such as overloads or timing
and exception errors [11], [39]. The study of failure data
from Tandem’s fault tolerant computer system indicated
that 70 percent of the failures were transient failures,
caused by race conditions and timing problems [30].
We designate faults attributed to software aging as aging-
related faults. Aging-related faults can fall under Bohrbugs
or Heisenbugs depending on whether the failure is
deterministic (repeatable) or transient. Fig. 2 illustrates this
classification. Following are examples of software faults in
each of these categories. A software fault that is environ-
ment independent and, hence, deterministic, falls under the
category of nonaging related Bohrbug (for example, a set of
inputs resulting in the same failure every time). If the
software bug, for example, is related to the arrival order of
messages to a process, it is classified as a nonaging related
Heisenbug. Reorder of messages and replay might result in
the system working correctly. A bug causing a gradual
resource exhaustion deterministically every time is classi-
fied as an aging-related Bohrbug. A bug causing an
unknown resource leak during rare instances, which are
difficult to reproduce, could be classified as an aging-
related Heisenbug.
2.2 Software Fault Tolerance Techniques
In this section, we discuss the various software fault
tolerance techniques that can be used for the specific fault
classes discussed above. Design diversity has been advo-
cated as a technique for software fault tolerance mainly to
deal with Bohrbugs [3]. It relies on the assumption of
independence between multiple variants of software.
However, as some studies have shown, this assumption
may not always be valid [29]. Design diversity has also been
used to treat Heisenbugs, as in the GUARDS [34] and
VOL. 2, NO. 2, APRIL-JUNE 2005
DELTA-4 [10], [35] systems. Since there are multiple
versions of software operating, it is not likely that all of
them will experience the same transient failure. One of the
disadvantages of design diversity is the high cost involved
in developing multiple variants of software.
Data diversity [1] can work well with Bohrbugs and is
less expensive to implement than design diversity. To some
extent, data diversity can also deal with Heisenbugs since
different input data is presented and, by definition, these
bugs are nondeterministic and nonrepeatable.
Environment diversity is a recent approach to software
fault tolerance. The components that affect the behavior of a
software are the volatile state (program stack and data
segments), the persistent state (user files), and the operating
system environment (all resources like swap space, file
systems, communications channels, etc.) [46]. Transient
faults typically occur in computer systems due to design
faults in software which result in unacceptable and
erroneous states in the operating system environment.
Environment diversity therefore advocates reexecuting the
software in a different operating environment [24], [28].
Although this technique has been used for long in an ad
hoc manner, only recently has it gained recognition and
importance. Usually, restart or reexecution is done at the
instance of a failure in the software. Examples of environ-
ment diversity techniques include retry operation, restart
application and rebooting the node. The retry and restart
operations can be done on the same node or on another
spare (cold/warm/hot) node. Tandem’s fault tolerant
computer system [30] is based on the process pair approach.
It was noted that many application failures did not recur
once the application was restarted on the second processor.
This was due to the fact that the second processor provided
a different environment which did not trigger the same
error conditions which led to the failure of the application
on the first processor. Hence, in this case (as well as in
Avaya’s SwiFT [18]), hardware redundancy coupled with
software replication1 was used to tolerate most of the
software faults.
For aging-related bugs, environment diversity can be
particularly effective if utilized proactively in the form of
software rejuvenation. Rejuvenation can be triggered either
by time (deterministic intervals) or by measurement and
analysis of data of the system condition.
3 RELATED WORK
The study of software aging and rejuvenation can be
broadly classified into two approaches—measurement-
based approach and analytic modeling approach. The first
approach applies statistical analysis to data collected from
systems and applies trend analysis or other techniques to
determine a window of time over which to perform
rejuvenation in order to prevent unplanned outages. The
analytic modeling approach assumes failure and repair time
distributions of a system and obtains optimal rejuvenation
schedule to maximize availability, or minimize loss prob-
ability or downtime cost.
1. Identical copies.
VAIDYANATHAN AND TRIVEDI: A COMPREHENSIVE MODEL FOR SOFTWARE REJUVENATION
Garg et al. [17] present a measurement-based approach
for detecting and estimating trends and times to exhaustion
of operating system resources due to software aging. The
data collection technique used in present paper, including
the workload and resource usage variables monitored, are
the same as in their work. While the work by Garg et al. [17]
considers only time based trend detection and estimation of
resource exhaustion, Vaidyanathan and Trivedi [44] take
the system workload into account for building a model to
estimate resource exhaustion times. In this paper, we
discuss a comprehensive model, which first extends the
workload-based approach by performing transient analysis
and formulating the estimated time to resource exhaustion
as the mean time to accumulated reward in a semi-Markov
reward model. We then develop an upper-level availability
model that accounts for failure and rejuvenation, and helps
us derive optimal rejuvenation schedules. Cassidy et al. [8]
have developed an approach to rejuvenation for large
online transaction processing servers. Using pattern recog-
nition methods, they find that 13 out of several monitored
parameters deviate from “normal” behavior just prior to a
crash, providing sufficient warning to initiate rejuvenation.
Li et al. [31] present an approach based on time series
analysis to predict resource usage trends in a Web server
while subjecting it to an artificial workload.
Several papers have dealt with determining the optimal
times to perform preventive maintenance of operational
software systems, through analytical models. The accuracy
of this approach is determined by the assumptions made in
the model for capturing aging. In many papers [14], [15],
[25], [40], only the failures causing unavailability of the
software are considered, while Pfening et al. [36] assume a
gradually decreasing service rate of a software that serves
transactions. Garg et al. [16], however, consider both these
effects of aging together in a single model. Models proposed
in some of the papers [14], [25] are restricted to hypo-
exponentially distributed time to failure, while models
proposed in others [15], [36], [40] can accommodate general
distributions, but only for the specific aging effect they
capture. Generally distributed time to failure, as well as the
service rate being an arbitrary function of time are allowed
in the work by Garg et al. [16]. Only their model captures
the effect of load on aging. In the work by Dohi et al. [12],
software rejuvenation models are formulated using semi-
Markov processes and optimal rejuvenation schedules that
maximize availability and minimize cost are derived
analytically. Nonparametric statistical algorithms to esti-
mate the optimal schedules are developed given a sample
data of failure times. The use of rejuvenation has been
extended to cluster systems [9], [45], and analytical models
of the implementation show that employing software
rejuvenation in cluster systems results in a significant
increase in system availability and decrease in downtime
cost. In [47], models are presented for software rejuvenation
in cluster systems under varying workloads. Bobbio et al.
[6] present fine grained degradation models where one can
identify the current degradation level based on the
observation of a system parameter. Optimal rejuvenation
policies based on a risk criterion and an alert threshold are
then presented.
127
In this work, we bridge the gap between the measure-
ment-based and the analytic modeling approaches by first
building a measurement-based semi-Markov workload
model. Both the structure (states and state transitions) and
the parameters (transition probabilities and sojourn time
distributions) are determined using statistical estimation on
measured data. Measured data on the rate of resource
depletion in each state is used as reward rate leading to the
computation of the estimated time to exhaustion of each
resource. These results are then used to build a higher level
semi-Markov availability model. This comprehensive mod-
el takes into account both reactive recovery following a
failure due to resource exhaustion and rejuvenation, and is
used to derive optimal rejuvenation schedules.
Other related work in measurement-based dependability
evaluation is based on either measurements made at failure
times [11] or at error observation times [42]. In our case, we
monitor the system performance variables continuously
since we are interested in trend estimation and, hence, in
predicting the time to next failure and not in observing
interfailure times or identifying error patterns. Hsueh et al.
use [26] a semi-Markov reward model to estimate the cost of
different types of errors. The clustering method they use is
very similar to ours but, in their work, reward rates are
assigned based on error rates while, in our case, reward
rates are attached to states reflecting the rate of resource
depletion in that state.
4 EXPERIMENTAL SETUP AND DATA COLLECTION
4.1 SNMP-Based Distributed Resource
Monitoring Tool
The SNMP (Simple Network Management Protocol)-based
distributed resource monitoring tool developed by Garg
et al. [17] was used for our data collection. SNMP is an
application protocol which offers network management
services in the Internet protocol suite. The manager, the
agent, and the MIB (Management Information Base) form
the main constituents of an SNMP-based management tool.
The SNMP protocol defines a client-server relationship
between a manager and an agent, and the MIB describes the
information that can be obtained and/or modified through
interactions between the manager and the agent.
4.2 Data Collection
The resource monitoring tool referred to previously was
used to collect operating system resource usage data
(physical/virtual memory usage, file/process table usage,
etc.) and system activity data (paging activity, CPU
utilization, etc.) from nine heterogeneous UNIX work-
stations, which were connected by an Ethernet LAN at the
Duke Department of Electrical and Computer Engineering.
In our setup, shown in Fig. 3, a central monitoring station
runs the manager program which sends get requests
periodically to each of the agent programs running on the
monitored workstations. The agent programs in turn obtain
data for the manager from their respective machines by
executing various standard UNIX utility programs like
pstat, iostat, and vmstat. Also shown in the figure along with
the monitored workstations are their respective operating
systems and primary functions in the department.
128 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING,
Fig. 3. Experimental setup for data collection.
The objects or parameters monitored on the work-
stations include those that describe the state of the
operating system resources, state of the processes run-
ning, information on the /tmp file system, availability
and usage of network related resources, and information
on terminal and disk I/O activity. More than 100 such
parameters were monitored at regular intervals (10 min)
for more than three months.
In this paper, we only discuss the results for the data
collected from the machine Rossby. The two resources
selected for study are usedSwapSpace and realMemoryFree
since they are considered to be leading indicators of
aging [17], [25]. Other resources like fileTablesize and
processTablesize can also be considered, but are not
addressed in this paper. Also, only the longest stretch of
sample points in which no reboots or failures occurred
were used for building the model, which is described in
the following section.
5 WORKLOAD CHARACTERIZATION AND MODELING
We have used the two-stage method [37] for constructing
our semi-Markov workload model, where we define a
matrix P and a vector H(t). The transitions of the semi-
Markov process can be thought of taking place in two
stages. Consider a transition from state i to state j. In the
first stage, the chain stays in state i for some amount of time
described by the sojourn time distribution Hi ðtÞ.
In the second stage, the chain moves to state j
determined by the probability pij . The sojourn time
distributions Hi ðtÞ are allowed to be general by the semi-
Markov assumption whereas a (homogeneous continuous-
time) Markov chain would have allowed only exponential
distributions.
The system workload was characterized by obtaining a
number of variables pertaining to CPU activity and file
system I/O. We then identify a small number of represen-
tative workload states through statistical clustering techni-
ques. Once the states are identified, the transition
probabilities from one state to another and the distribution
of sojourn time in each state are estimated. Thus, we obtain
VOL. 2, NO. 2, APRIL-JUNE 2005
a complete state-transition model to describe the system
workload dynamics. The details of the model construction
are explained in the rest of this section.
In this study, we use the following variables to
characterize the workload:
. cpuContextSwitch: The number of process context
switches performed during the measurement inter-
val (10 min in our case).
. sysCall: The number of system calls made during the
interval.
. pageIn: The number of page-in operations (pages
fetched in from file system or swap device) during
the interval.
. pageOut: The number of page-out operations (pages
pushed out to file system or swap device) during the
interval.
Thus, a point in a four-dimensional space, (cpuContextS-
witch, sysCall, pageIn, pageOut), represents the measured
workload for a given interval of time. These variables are
thus used to define and characterize the system workload.
We next partition the data points into clusters that contain
similar points based on some predefined criteria. A
statistical clustering algorithm is used for achieving this.
5.1 Cluster Analysis
The goal of cluster analysis is to determine a partition of a
given set of points into groups or clusters such that the
points in a single cluster are similar, according to a certain
criterion, to each other than to points in a different cluster.
In our case, we use an iterative nonhierarchical clustering
algorithm called the Hartigan’s k-means clustering algorithm
[22]. The objective of this algorithm is to divide a given set
of points into k clusters so that the within-cluster sum of
squares is minimized.
If the variables for clustering are not expressed in
homogeneous units, a scale change or normalization must
be performed. In our analysis, we used the normalization
method based on the following transformation:
xi
mini fxi g
x0 i ¼ ;
maxi fxi g
mini fxi g
where x0i is the normalized value of xi . This transformation
restricts the values of all the variables to the range 0 to 1. We
need to eliminate the outliers in the data before applying
the scaling technique since they tend to distort the
transformation. Outliers in the data were identified and
eliminated by an analysis of the cumulative distribution of
each parameter, although they were assigned to clusters in
the final stage. The statistics of the measured workload
variables are shown in Table 1.
The k-means clustering algorithm was applied to the
workload data and this resulted in eight clusters. The
number of final clusters is highly data dependent. We
compute the ratio
¼ WWiþ1i  ðn
i
1Þ, where Wi is the
within-cluster sum of squares for i clusters and n is the total
number of data points. The clustering algorithm starts with
one cluster and keeps adding one at a time until the desired
value of
is reached.
VAIDYANATHAN AND TRIVEDI: A COMPREHENSIVE MODEL FOR SOFTWARE REJUVENATION 129

TABLE 1
Statistics for the Workload Variables Measured

TABLE 2
Statistics for the Workload Clusters

The statistics for the eight workload clusters are shown 5.3 Sojourn Time Distribution
in Table 2. Also shown in the table are the percentage of the To completely specify the semi-Markov process, we need to
sample data points in each cluster. We observe that more determine the distribution of sojourn time in each state. This
than 75 percent of the points belong to clusters 4, 5, and 7 distribution for all the workload states was fitted to either
that are relatively light workload states. Clusters 1 and 8 are 2-stage hyperexponential or 2-stage hypoexponential dis-
heavy workload states and contain significantly less data tribution functions. The sojourn times in workload states
W1 , W5 , and W7 were fitted to hypoexponential distribu-
points.
tions, while the sojourn times in all other states were fitted
5.2 The State-Transition Model to hyperexponential distributions [43]. The fitted distribu-
The next step, after the clusters and centroids are identified, tions were tested using the Kolmogorov-Smirnov test [43] at
is to build a state-transition model for the system workload. a significance level of 0.01.
Fig. 4 shows the Kolmogorov-Smirnov test for sojourn
This is done by determining the transition probabilities
time distribution in workload state 4 and the fitted
from one state to another. The transition probability pij from
distributions for all the workload states are listed in Table 3.
a state i to a state j can be estimated from the sample data Even though our model was constructed from real
using the following formula [26], [43]: measurements, the clustering method results only in an
observed no: of transitions from state i to state j approximate model for the workload. We made a quick
pij ¼ : check of the model accuracy by comparing the steady state
total observed no: of transitions from state i
probability of occupying a particular workload state
State transition probabilities were estimated from the computed from the model to the estimated probability
observed data for the eight workload states and the from the observed data, i.e., the fraction of the length of
resulting (8  8) transition probability matrix, P is shown time the system was in that workload state to the total
below: length of the period of observation. These values, shown in
Table 4, match closely.
P ¼
2 3
0:000 0:155 0:224 0:129 0:259 0:034 0:165 0:034 6 MODELING RESOURCE DEPLETION
6 0:071 0:000 0:136 0:140 0:316 0:026 0:307 0:004 7
6 7 The previous section discussed the development of a semi-
6 7
6 0:122 0:226 0:000 0:096 0:426 0:000 0:113 0:017 7 Markov process to describe the system workload. Since our
6 7
6 0:147 0:363 0:059 0:000 0:098 0:216 0:088 0:029 7 original aim was to estimate the exhaustion of system
6 7
6 7: resources as a function of workload, we need to incorporate
6 0:033 0:068 0:037 0:011 0:000 0:004 0:847 0:000 7
6 7 the effect of workload on the resources in this model.
6 0:070 0:163 0:023 0:535 0:116 0:000 0:023 0:070 7
6 7 Therefore, a reward function corresponding to each system
6 7
4 0:022 0:049 0:003 0:003 0:920 0:003 0:000 0:000 5 resource considered for analysis is assigned to the model.
0:307 0:077 0:154 0:231 0:077 0:154 0:000 0:000 This will enable us to study the evolution of the system
130 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 2, NO. 2, APRIL-JUNE 2005

Fig. 4. Kolmogorov-Smirnov test for sojourn time distribution in W4 .

TABLE 3
Sojourn Time Distributions in the Eight Workload States

resource in the different states of the workload model. The are no reboots during this period. The time interval for
reward rate, rij , for each workload state, i, and for each which the plots are shown for the resources, corresponds to
resource, j, is assigned the estimated slope (described in the the time interval for which the workload data used for
next subsection) of the depletion of resource j in the
workload state i.
We consider two resources here for our analysis—
usedSwapSpace and realMemoryFree. The time plots of these
two resources as measured on machine Rossby between
two successive reboots/failures are shown in Fig. 5. There

TABLE 4
Comparison of State Occupancy Probabilities

Fig. 5. Time plots of resources usedSwapSpace and realMemoryFree in

machine Rossby.
VAIDYANATHAN AND TRIVEDI: A COMPREHENSIVE MODEL FOR SOFTWARE REJUVENATION 131

TABLE 5
Slope Estimates (in KB/10 min) for usedSwapSpace and realMemoryFree at Different Workload States

building the state-transition model (in the previous section) assumption that resource depletion does depend on the
was sampled from. system workload and the rates of exhaustion vary with
workload changes. We observe that the slopes for usedSwap-
6.1 Computing the Slope
Space in all the workload states are nonnegative, and the
If a linear trend is present in the data, linear regression slopes for realMemoryFree are nonpositive in all the work-
methods can be used to estimate the true slope by load states except in one. This shows that usedSwapSpace
computing the least squares estimate of the slope. The
increases whereas realMemoryFree decreases over time. This
slope obtained by this method can deviate greatly from the
validates the “software aging” phenomenon [17], [25]. It is
true slope if there are gross errors or outliers in the data
also generally observed that the higher the system activity,
[19]. Therefore, we estimate the true slope of a resource
higher is the resource depletion. The 95 percent confidence
depletion at every workload state by using a nonparametric
intervals for the slope are relatively wide for usedSwapSpace
procedure developed by Sen [38]. Sen’s method is not
in workload state W1 and for realMemoryFree in workload
greatly affected by gross data errors or outliers, and it can
be computed even with missing data. state W8 . This reflects the relatively small number of data
The procedure for estimating the slope by this method is points in these states and accounts for a high degree of
as follows: First, n0 slope estimates are calculated for all pairs variability.
of points at i and j for which i > j; as mij ¼ ðxi
xj Þ=ði
jÞ.
We thus obtain n0 ¼ nðn
1Þ=2 slope estimates, where n is 7 RESULTS FOR THE WORKLOAD MODEL
the total number of data observations. The median of these
The irreducible semi-Markov reward model (SMP) de-
n0 values is the required slope estimate. It is also possible
scribed above was converted into an irreducible Markov
to obtain a two-sided confidence interval about the true
reward model (MRM) before solving since all the sojourn
slope [19].
Table 5 shows the slope obtained by Sen’s method for times have phase-type distributions. For examples and
these two resources, for all the eight workload states. The
slopes shown in this table are the slopes of the respective
resource depletion during the longest interval for which the
system was at that workload state. For a particular work-
load state, it is possible to obtain the slope of resource
depletion during different visits to the state. We expect the
different slopes thus obtained to be relatively close. Figs. 6
and 7 show the observed values of usedSwapSpace and
realMemoryFree, respectively, at workload state 4 during two
different visits (with different sojourn times for each visit).
The estimated slopes (in KB/10 min) and 95 percent
confidence intervals are also given in parentheses beside
the slope estimates. We observe that the slope in both the
instances are within the same confidence limits. Thus, in
our model, these slopes correspond to the reward rates for
each workload state for usedSwapSpace and realMemoryFree.
The observation that slopes in a given workload state are
within the same confidence intervals (refer to Figs. 6 and 7),
together with the observation that slopes across different
workload states are different (refer to Table 5) validates our Fig. 6. usedSwapSpace in W4 during two different visits.
132 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 2, NO. 2, APRIL-JUNE 2005

Fig. 8. Transient slope estimate of usedSwapSpace.

Fig. 7. realMemoryFree in W4 during two different visits.

where ð0Þ is the initial probability vector,
Z r
details of such a Markovizing transformation, see [26], [43].
1

First, we compute the expected steady state reward rate, CðrÞ ¼ euR1 B du R
1
1 ;
0
which can be considered the rate of depletion of the
resource averaged over all workload states. This can then be B ¼ Q1
Q2 Q
1
4 Q3 , and e is a column vector of all ones.
compared with the slope obtained using the (workload- 7.1 Model Solution
independent) pure time-based approach of [17]. The
The Markov reward model describing the system workload
expected reward rate at time t is then computed in order
and resource depletion was solved partly using the
to determine how quickly the steady state is reached.
Finally, the expected time to accumulate reward r is SHARPE (Symbolic Hierarchical Automated Reliability
computed in order to estimate the mean time to exhaustion and Performance Evaluator) [37] software package devel-
of the resource. oped by researchers at Duke University.
The following procedure is used to compute the Figs. 8 and 9 show the instantaneous reward rate (or the
expected time to accumulate reward r. The states of the transient slope estimate) of usedSwapSpace and realMemory-
MRM can be partitioned into two sets based on the assigned Free respectively. The increase in the slope in the beginning
reward rates (S), which contains all states with positive can be attributed to the initial conditions and the dynamics
reward rates, and S z ¼

S, which contains all the states of the stochastic model. The values settle down to a stable
with zero reward rates. Based on this partition, the value fairly quickly, i.e., the steady state is reached within a
generator matrix Q of the MRM can be rearranged and short time.
partitioned into four submatrices such that Table 6 gives the estimates for slope and time to

 exhaustion for usedSwapSpace and realMemoryFree computed
Q1 Q2
Q¼ ;
Q3 Q4
where Q1 contains all the transitions within S, Q2 contains
the transitions from S to S z , Q3 contains the transitions
from S z to S, and Q4 contains all the transitions within S z .
Similarly, the reward rate matrix R can be partitioned
such that


R1 0
R¼ ;
0 0
where R1 ¼ diagi2S ½ri  and each ri corresponds to the
ordering of states in Q1 .
Let T ðrÞ represent the time to accumulate reward r. The
expected time to accumulate reward r, E½T ðrÞ, is given
by [41]


CðrÞ
CðrÞQ2 Q
1
4
E½T ðrÞ ¼  ð0Þ
1 e;

Q
1
1
1
4 Q3 CðrÞ Q4 þ Q4 Q3 CðrÞQ2 Q4
Fig. 9. Transient slope estimate of realMemoryFree.
VAIDYANATHAN AND TRIVEDI: A COMPREHENSIVE MODEL FOR SOFTWARE REJUVENATION 133

TABLE 6
Estimates for Slope (in KB/10 min) and Time to Exhaustion (in Days) for usedSwapSpace and realMemoryFree

using both the time based (workload independent) and the trend estimate coincides with the final peak values of
workload-based approaches. The slope for the time-based realMemoryFree.
approach is obtained through Sen’s slope estimate [17]. The
slope for the workload based estimation is obtained as the
8 COMPREHENSIVE MODEL
expected reward rate at steady state from the model. The
95 percent confidence interval for the workload-based slope In the measurement-based model described in the previous
estimations is larger since it allows for a larger degree of section, explicit failure and rejuvenation are not captured,
variability that possibly takes into account peaks in the i.e., there are no failure states or rejuvenation states. All the
data. The workload based estimation of time to resource workload states are “up” states or available states where the
exhaustion is computed as the mean time to accumulate software is doing useful work.
reward r, where r is the maximum amount of resource The macrostates of a software system employing
which can be consumed. In other words, E½T ðrmax Þ is rejuvenation can be represented by the semi-Markov
process in Fig. 10 [16]. The software is available for service
computed for each resource of interest. For a decreasing
in state A. It can either fail in this state and enter state B
resource usage (negative reward rates), the problem is
(failure state) or a rejuvenation trigger can take it to state C.
transformed into an increasing resource usage problem by
The software is considered unavailable in both states B and
changing all negative reward rates to positive rates and by
C. It is brought back to state A from B after (reactive)
assuming the resource usage increases from zero to the
recovery and from state C after the completion of
initial value (assumed to be the maximum). To estimate the
(proactive) rejuvenation.
time to resource exhaustion using the time-based approach,
The subordinated process in state A for the transaction-
we use the initial intercept c, the slope estimate m and the
based system described in [16] was a queuing system with
simple linear equation y ¼ mx þ c. The initial points for states representing the number of current transactions in the
both the estimations is taken to be the average value over system. The measurement-based workload model devel-
the first few days. The maximum value for the usedSwap- oped in Section 5 can be substituted for this subordinated
Space in machine Rossby was 312,724 Kilobytes. process as shown in Fig. 10.
We find that workload based estimations give a lower The distribution of time to failure, F ðtÞ, of the software
time to resource exhaustion than the corresponding slopes starting from the available state, A, is the distribution of
computed using time based estimations [17]. It was time to exhaustion of the resource in the workload model.
observed that machines failed due to resource exhaustion Since the computation of this distribution, in general, is
much before the times to resource exhaustion estimated by very complicated [41] (more so with some states having
the time based method. The upper confidence level for the zero reward rates), we approximate the distribution of time
workload based estimation can be used to estimate the to failure with an increasing failure-rate (IFR) distribution
peaks in resource usage. The lower confidence level
estimate gives the general overall trend without being
affected by the peaks. The reason that peaks in the data are
more or less covered by the workload-based estimation is
that workload model takes into account those workload
states that are likely to generate peaks in the data. Hence,
the slope estimated by the workload model is larger. The
time-based estimation removes these peaks and gives us
only a general overall trend.
A comparison between the estimated times to exhaustion
can help us ascertain which resources are important for us
to monitor and manage; in this case, realMemoryFree being
the more important or critical resource. We get a slope
estimated using system workload being better than that
estimated using the workload independent method. The
workload based trend estimation coincides with the final
values of realMemoryFree and the upper confidence level Fig. 10. Macrostates of the software behavior.
134 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 2, NO. 2, APRIL-JUNE 2005

which has the same mean as that of the job completion time.
We split the time to failure into two exponential stages with
each exponential stage having a mean of =2, where  is
obtained as the mean time to resource exhaustion in
Section 7. This results in the time to failure distribution
F ðtÞ being a 2-stage Erlang given by
F ðtÞ ¼ 1
e
t
te
t
with parameter  ¼ 2=.
The rejuvenation trigger period (from state A to state C),
, is assumed to be deterministic. The distributions of time
to recover (from state B to state A) and time to perform
rejuvenation (from state C to state A) are considered general
with mean times f and r , respectively.
8.1 Model Solution
As previously explained, the software can be in one of three
states at time t: available for service (state A), failure state
(state B), and rejuvenation state (state C).
Solving for the embedded discrete time Markov chain
steady-state probabilities [43], we get [16] A ¼ 1=2,
B ¼ pAB =2, and C ¼ pAC =2, where pAC is given by
pAC ¼ 1
F ðÞ ¼ e
 ð1 þ Þ
and
pAB ¼ 1
pAC :
Let HA ðtÞ be distribution of holding time in state A of the
semi-Markov process. The mean holding time in state A,
E½HA ðtÞ, is given by
Z 
1 computing A . Therefore,
E½HA ðtÞ ¼ ð1
F ðtÞÞdt ¼ ½2
e
 ð2 þ Þ:
0 
As discussed in the previous section, the distributions of
sojourn time in states B and C are considered general, with
means f and r , respectively.
where  is the optimal rejuvenation trigger period and  ¼
2= as indicated earlier (i.e.,  is mean time to resource
exhaustion from the workload model). The above equation
can be written as a fixed point equation of the form


1 f
r
 ¼ ln ¼ ga ð Þ: ð2Þ
 f
ðf
2r Þ
It is easy to see that  lies in the open interval
 
f
0; :
ðf
2r Þ
The solution of  can now be obtained by the bisection
method, where two initial values 1 and 2 are picked such
that 1 < ga ð1 Þ and 2 > ga ð2 Þ. This computation is
repeated several times by narrowing the distance between
1 and 2 each time till the desired accuracy of  is reached.
Over a time interval ½0; T , the expected uptime is given
by, U ¼ T  Avail. The maximum (optimal) expected up-
time is then given by U  ¼ T  Avail , where Avail
corresponds to the optimal availability computed using  .
8.1.2 Expected Downtime Cost
Another objective of rejuvenation could be to minimize the
total downtime cost over a given interval. The total cost per
unit time due to downtime is given by
Cost ¼ B  cfailure þ C  crejuv ;
where cfailure and crejuv are costs per unit time due to failure
and rejuvenation respectively. Steady-state probabilities B
and C can be computed in a way similar to the one for
B f  cfailure þ C r  crejuv
Cost ¼ CðÞ ¼
A E½HA ðtÞ þ B f þ C r
f  cfailure þ e
 ð1 þ Þðr  crejuv
f  cfailure Þ
¼ 1
 ð2 þ Þ þ 
e
 ð1 þ Þð
 Þ
:
 ½2
e f f r

8.1.1 Steady-State Availability ð3Þ

The steady-state availability of the software, i.e., the steady- As before, to obtain the value of optimal rejuvenation
state probability of the semi-Markov process being in state 0
trigger period,  , which minimizes downtime cost, CðÞ, we
A, can be obtained as a function of  from the formula [43] differentiate CðÞ with respect to  and set this to 0.
A E½HA ðtÞ Therefore,
Avail ¼ A ðÞ ¼
A E½HA ðtÞ þ B f þ C r dCðÞ

e
 ð2 þ Þ
1 ¼ 0:
 ½2 d
¼1

:
 ½2
e ð2 þ Þ þ f
e
 ð1 þ Þðf
r Þ Substituting for CðÞ from (3), differentiating and simplify-
ð1Þ ing the equation, we get
To obtain the value of optimal rejuvenation trigger period, 0
ðf  cfailure
r  crejuv Þe
 þ ½f  cfailure
2r  crejuv
 , which maximizes availability, A ðÞ, we differentiate 0

A ðÞ with respect to  and set this value to 0. Therefore, þ f r ðcfailure
crejuv Þ
f  cfailure ¼ 0;

dA ðÞ where 0 is the optimal rejuvenation trigger period. The

¼ 0: above equation can be written as a fixed point equation of
d
the form
Substituting for A ðÞ from (1), differentiating and simplify-
ing the equation, we get 1 h i
0 ¼ ln f cfailure
½f cfailure ¼ gc ð0 Þ: ð4Þ
f cfailure
r crejuv

2r crejuv þf r ðcfailure
crejuv Þ0


ðf
r Þe
 þ ðf
2r Þ
f ¼ 0;
Now, 0 lies in the open interval
VAIDYANATHAN AND TRIVEDI: A COMPREHENSIVE MODEL FOR SOFTWARE REJUVENATION
Fig. 11. Expected uptime versus rejuvenation trigger period.
 
f  cfailure
0;
½f  cfailure
2r  crejuv þ f r ðcfailure
crejuv Þ
0
The solution of  can now be obtained by the bisection
method as explained earlier.
Over a time interval ½0; T , the expected cost due to
downtime is given by, C ¼ T  Cost. The minimum
expected downtime cost is then given by C 0 ¼ T  Cost0 ,
where Cost0 corresponds to the optimal cost per unit of
downtime computed using 0 .
8.2 Results for the Comprehensive Model
To illustrate the applicability of the model, we assume that
the mean time to recover from a failure, f is 4 hours and
the mean time to rejuvenate the system, r is 1 hour. The
mean time to failure is 41.38 days (from Section 5). We also
assume a cost of $5,000/hour for failure and $500/hour for
rejuvenation. We compute the expected uptime and the
expected cost over an interval of 1,000 days.
From (2) and (4), we get


3 importance among different resources and estimated time
 ¼ 496:56ln
4
0:00403


0 19; 500
 ¼ 496:56ln : and the measurement-based model to obtain a comprehen-
20; 000
38:336250
Solving the above equations, we get  ¼ 36:10 days and
0 ¼ 5:60 days. Fig. 11 shows the plot of the rejuvenation
trigger period versus the expected uptime and Fig. 12
shows the plot of rejuvenation interval versus the expected
cost due to downtime. As we can see, the results of the
optimal rejuvenation trigger periods obtained graphically
agree with the results obtained by solving the equations for
 and 0 .
9 CONCLUSION
In this paper, we included aging-related faults intro Gray’s
classification of software faults and discussed various
methods to deal with these faults. We also dealt with
software rejuvenation, a specific form of environment
diversity that is gaining importance as an effective
preventive maintenance technique. The main contribution
135
Fig. 12. Expected cost versus rejuvenation trigger period.
:
of our work is a measurement-based model that incorpo-
rates the effect of system workload on operating system
resources and an approach to investigate its effect on
software aging. Since many studies have suggested strong
correlations between workload and system reliability/
availability, this model is an improvement over the purely
time-based model [17]. We were able to clearly demonstrate
the relation between the system workload and resource
exhaustion and, thus, validate our assumption that work-
load does affect the way various system resources are
depleted. Not only was the system workload dynamics
captured in the model, but also the effect of workload on
resource depletion was quantified by means of reward rates
or slopes in the model. In doing this, we have also validated
the “software aging” phenomenon with respect to resource
exhaustion. The quantification of resource exhaustion for
different workload states helps us in obtaining a better
estimation for exhaustion rates and time to exhaustion than
just time based estimation. The analysis of relative
to resource exhaustion [17] can be done here with greater
accuracy. Finally, we also integrated the analytical model
sive model. The model was solved to obtain the optimal
rejuvenation trigger period for maximizing expected up-
time and expected cost due to downtime. Although the
estimations obtained from these methodologies cannot be
taken to be estimations of actual machine failure times since
they might depend on various other factors, all these results
take us a step further toward predicting actual failure times
and may help us in proposing new or better preventive
maintenance policies like “software rejuvenation.” These
models are very generic and can be applied to any other
data set. The models described in this paper are offline
models, as opposed to online models [9], [45]. Once a model
is built, it can be used until the configuration or the
workload pattern of the machine is changed.
A possible extension of this work could be to consider
the system workload in a more fine-grained manner and to
determine the relation between each individual process or a
group of similar processes and resource consumption.
136 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING,
Future work along these lines could be to model the
distribution of time to failure more accurately and take
multiple resources into consideration while developing the
model.
ACKNOWLEDGMENTS
This work was done while the first author was at Duke
University. The project was supported in part by Telcordia
Technologies as a core project of the CACC, by the JPL REE
project under Contract 1216658, by the AFOSR under MURI
grant F49620-1-0327, by the ARO under a MURI grant
C-DAAD19 01-1-0646, “Mathematics of Failures in Complex
Systems” and by the SITAR project of the DARPA OASIS
Grant N66001-00-C.
REFERENCES
[1] P.E. Amman and J.C. Knight, “Data Diversity: An Approach to
Software Fault Tolerance,” Proc. 17th Int’l Symp. Fault Tolerant
Computing, pp. 122-126, June 1987.
[2] A. Avritzer and E.J. Weyuker, “Monitoring Smoothly Degrading
Systems for Increased Dependability,” Empirical Software Eng. J.,
vol. 2, no. 1, pp. 59-77, 1997.
[3] A. Avizienis and L. Chen, “On the Implementation of N-Version
Programming for Software Fault Tolerance During Execution,”
Proc. IEEE COMPSAC 77 Conf., pp. 149-155, Nov. 1977.
[4] A. Avizienis, J.-C. Laprie, and B. Randell, “Fundamental Concepts
of Dependability,” LAAS Technical Report No. 01-145, LAAS,
France, Apr. 2001.
[5] Y. Bao, X. Sun, and K. Trivedi, “Adaptive Software Rejuvenation:
Degradation Models and Rejuvenation Schemes,” Proc. Int’l. Conf.
Dependable Systems and Networks (DSN-2003), June 2003.
[6] A. Bobbio, A. Sereno, and C. Anglano, “Fine Grained Software
Degradation Models for Optimal Rejuvenation Policies,” Perfor-
mance Evaluation, vol. 46, pp. 45-62, 2001.
[7] T. Boyd and P. Dasgupta, “Preemptive Module Replacement
Using the Virtualizing Operating System,” Proc. Workshop Self-
Healing, Adaptive and Self-Managed Systems (SHAMAN 2002), June
2002.
[8] K. Cassidy, K. Gross, and A. Malekpour, “Advanced Pattern
Recognition for Detection of Complex Software Aging in Online
Transaction Processing Servers,” Proc. Int’l Conf. Dependable
Systems and Networks (DSN 2002), June 2002.
[9] V. Castelli, R.E. Harper, P. Heidelberger, S.W. Hunter, K.S.
Trivedi, K. Vaidyanathan, and W. Zeggert, “Proactive Manage-
ment of Software Aging,” IBM J. Research & Development, vol. 45,
no. 2, Mar. 2001.
[10] M. Chereque, D. Powell, P. Reynier, J.-L. Richier, and J. Voiron,
“Active Replication in Delta-4,” Proc. 22nd IEEE Int’l. Symp. Fault
Tolerant Computing (FTCS-22), pp. 28-37, July 1992.
[11] R. Chillarege, S. Biyani, and J. Rosenthal, “Measurement of Failure
Rate in Widely Distributed Software,” Proc. 25th IEEE Int’l Symp.
Fault Tolerant Computing, pp. 424-433, July 1995.
[12] T. Dohi, K. Goseva-Popstojanova, and K.S. Trivedi, “Statistical
Non-Parametric Algorithms to Estimate the Optimal Software
Rejuvenation Schedule,” Proc. 2000 Pacific Rim Int’l Symp.
Dependable Computing (PRDC 2000), Dec. 2000.
[13] C. Fetzer and K. Hostedt, “Rejuvenation and Failure Detection in
Partitionable Systems,” Proc. Pacific Rim Int’l Symp. Dependable
Computing (PRDC 2001), Dec. 2001.
[14] S. Garg, A. Puliafito, and K.S. Trivedi, “Analysis of Software
Rejuvenation Using Markov Regenerative Stochastic Petri Net,”
Proc. Sixth Int’l Symp. Software Reliability Eng., pp. 180-187, Oct.
1995.
[15] S. Garg, Y. Huang, and C. Kintala, K.S. Trivedi, “Minimizing
Completion Time of a Program by Checkpointing and Rejuvena-
tion,” Proc. 1996 ACM SIGMETRICS Conf., pp. 252-261, May 1996.
[16] S. Garg, A. Puliafito, M. Telek, and K.S. Trivedi, “Analysis of
Preventive Maintenance in Transactions Based Software Systems,”
IEEE Trans. Computers, pp. 96-107, vol. 47, no. 1, Jan. 1998.
VOL. 2, NO. 2, APRIL-JUNE 2005
[17] S. Garg, A. van Moorsel, K. Vaidyanathan, K. Trivedi, “A
Methodology for Detection and Estimation of Software Aging,”
Proc. Ninth Int’l Symp. Software Reliability Eng., pp. 282-292, Nov.
1998.
[18] S. Garg, Y. Huang, C.M.R. Kintala, K.S. Trivedi, and S. Yagnik,
“Performance and Reliability Evaluation of Passive Replication
Schemes in Application Level Fault Tolerance,” Proc. Fault Tolerant
Computing Symp. (FTCS 1999), pp. 322-329, June 1999.
[19] R.O. Gilbert, Statistical Methods for Environmental Pollution Mon-
itoring. New York: Van Nostrand Reinhold, 1987.
[20] J. Gray, “Why Do Computers Stop and What Can Be Done About
It?” Proc. Fifth Symp. Reliability in Distributed Software and Database
Systems, pp. 3-12, Jan. 1986.
[21] J. Gray, “A Census of Tandem System Availability between 1985
and 1990,” IEEE Trans. Reliability, vol. 39, pp. 409-418, Oct. 1990.
[22] J.A. Hartigan, Clustering Algorithms. New York: Wiley, 1975.
[23] Y. Hong, D. Chen, L. Li, and K.S. Trivedi, “Closed Loop Design for
Software Rejuvenation,” Proc. Workshop Self-Healing, Adaptive and
Self-Managed Systems (SHAMAN 2002), June 2002.
[24] Y. Huang, P. Jalote, and C. Kintala, “Two Techniques for Transient
Software Error Recovery,” Lecture Notes in Computer Science,
vol. 774, pp. 159-170, 1994.
[25] Y. Huang, C. Kintala, N. Kolettis, and N.D. Fulton, “Software
Rejuvenation: Analysis, Module and Applications,” Proc. 25th
Symp. Fault Tolerant Computing (FTCS-25), pp. 381–390, June 1995.
[26] M.C. Hsueh, R.K. Iyer, and K.S. Trivedi, “Performability Modeling
Based on Real Data: A Case Study,” IEEE Trans. Computers, vol. 37,
no. 4, pp. 478-484, Apr. 1988.
[27] “IBM Netfinity Director Software Rejuvenation,”White Paper,
IBM Corp., Research Triangle Park, N.C., Jan. 2001.
[28] P. Jalote, Y. Huang, and C. Kintala, “A Framework for Under-
standing and Handling Transient Software Failures,” Proc. Second
ISSAT Int’l Conf. Reliability and Quality in Design, 1995.
[29] J.C. Knight and N.G. Leveson, “An Experimental Evaluation of
the Assumption of Independence in Multiversion Programming,”
Software Eng. J., pp. 96-109, vol. 12, no. 1, 1986.
[30] I. Lee and R.K. Iyer, “Software Dependability in the Tandem
GUARDIAN System,” IEEE Trans. Software Eng., vol. 21, no. 5,
pp. 455-467, May 1995.
[31] L. Li, K. Vaidyanathan, and K.S. Trivedi, “An Approach to
Estimation of Software Aging in a Web Server,” Proc. Int’l Symp.
Empirical Software Eng. (ISESE 2002), Oct. 2002.
[32] Y. Liu, Y. Ma, J.J. Han, H. Levendel, and K.S. Trivedi, “Modeling
and Analysis of Software Rejuvenation in Cable Modem Termina-
tion System,” Proc. Int’l Symp. Software Reliability Eng. (ISSRE
2002), Nov. 2002.
[33] E. Marshall, “Fatal Error: How Patriot Overlooked a Scud,”
Science, p. 1347, Mar. 1992.
[34] D. Powell, J. Arlat, L. Beus-Dukic, A. Bondavalli, P. Coppola, A.
Fantechi, E. Jenn, C. Rabejac, and A. Wellings, “GUARDS: A
Generic Upgradable Architecture for Real-Time Dependable
Systems,” IEEE Trans. Parallel and Distributed Systems, vol. 10,
no. 6, June 1999.
[35] D. Powell, “Distributed Fault Tolerance: Lessons from Delta-4,”
IEEE Micro, vol. 14, no. 1, Feb. 1994.
[36] A. Pfening, S. Garg, A. Puliafito, M. Telek, and K.S. Trivedi,
“Optimal Rejuvenation for Tolerating Soft Failures,” Performance
Evaluation, vols. 27-28, pp. 491-506, Oct. 1996.
[37] R.A. Sahner, K.S. Trivedi, A. Puliafito, Performance and Reliability
Analysis of Computer Systems—An Example-Based Approach Using
the SHARPE Software Package. Norwell, Mass.: Academic Publish-
ers, 1996.
[38] P.K. Sen, “Estimates of the Regression Coefficient Based on
Kendall’s Tau,” J. the Am. Statistical Assoc., vol. 63, pp. 1379-1389,
1968.
[39] M. Sullivan and R. Chillarege, “Software Defects and Their Impact
on System Availability—A Study of Field Failures in Operating
Systems,” Proc. 21st IEEE Int’l Symp. Fault-Tolerant Computing,
pp. 2-9, 1991.
[40] A.T. Tai, S.N. Chau, L. Alkalaj, and H. Hecht, “On-Board
Preventive Maintenance: Analysis of Effectiveness and Optimal
Duty Period,” Proc. Third Int’l Workshop Object Oriented Real-Time
Dependable Systems, Feb. 1997.
[41] M. Telek, A. Pfening, and G. Fodor, “An Effective Numerical
Method to Compute the Moments of the Completion Time of
Markov Reward Models,” Computer Math. Applications, vol. 36,
no. 8, pp. 59-65, 1998.
VAIDYANATHAN AND TRIVEDI: A COMPREHENSIVE MODEL FOR SOFTWARE REJUVENATION
[42] A. Thakur and R.K. Iyer, “Analyze-NOW—An Environment for
Collection and Analysis of Failures in a Network of Work-
stations,” Proc. Int’l Symp. Software Reliability Eng., pp. 14-23, Apr.
1996.
[43] K.S. Trivedi, Probability and Statistics, with Reliability, Queuing, and
Computer Science Applications, second ed. John Wiley, 2001.
[44] K. Vaidyanathan and K.S. Trivedi, “A Measurement-Based Model
for Estimation of Resource Exhaustion in Operational Software
Systems,” Proc. 10th IEEE Int’l Symp. Software Reliability Eng.,
pp. 84-93, Nov. 1999.
[45] K. Vaidyanathan, R.E. Harper, S.W. Hunter, K.S. Trivedi,
“Analysis and Implementation of Software Rejuvenation in
Cluster Systems,” Proc. Joint Int’l Conf. Measurement and Modeling
of Computer Systems, ACM SIGMETRICS 2001/Performance 2001,
June 2001.
[46] Y.-M. Wang, Y. Huang, P.-Y. Chung, P. Vo, and C. Kintala,
“Checkpointing and Its Applications,” Proc. Symp. Fault Tolerant
Computer Systems, pp. 22-31, June 1995.
[47] W. Xie, Y. Hong, and K.S. Trivedi, “Software Rejuvenation Policies
for Cluster Systems under Varying Workload,” Proc. 10th Int’l
Pacific Rim Dependable Computing Symp. (PRDC 2004), Mar. 2004.
137
Kalyanaraman Vaidyanathan received the BE
degree in computer science from the University
of Madras, India, and the MS and PhD degrees
in electrical and computer engineering from
Duke University (2002). He was a recipient of
the IBM Graduate Fellowship Award in 2000. His
research interests include software reliability
and performance and dependability evaluation
of computer systems. He is currently a research
engineer in the Scalable Systems Group, Sun
Microsystems, San Diego, California, exploring proactive fault monitor-
ing techniques through telemetry and pattern recognition. He is a
member of the IEEE.
Kishor S. Trivedi holds the Hudson Chair in the
Department of Electrical and Computer Engi-
neering at Duke University, Durham, North
Carolina. He has been on the Duke faculty since
1975. He is the author of a well-known text
entitled Probability and Statistics with Reliability,
Queuing and Computer Science Applications
with a thoroughly revised second edition being
published by John Wiley. He has also published
two other books entitled Performance and
Reliability Analysis of Computer Systems (Kluwer Academic Publishers)
and Queueing Networks and Markov Chains (John Wiley). His research
interests are in reliability and performance assessment of computer and
communication systems. He has published more than 390 articles and
lectured extensively on these topics. He has supervised 39 PhD
dissertations. He has made seminal contributions in software rejuvena-
tion, solution techniques for Markov chains, fault trees, stochastic Petri
nets, and performability models. He has actively contributed to the
quantification of security and survivability. He is a fellow of the IEEE and
a Golden Core Member of IEEE Computer Society. He is a codesigner
of HARP, SAVE, SHARPE, and SPNP software packages that have
been well circulated.
. For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/publications/dlib.