Preparing for Tests

(Software Reliability)
(SE-308)

By Priya Singh (Assistant Professor, Dept of SE, DTU)

1. Background
• In preparing for the test, we apply the operational profile information we have developed
to plan for an efficient test.
• Preparing for tests includes preparing test cases and test procedures and planning for any
automated tools you wish to use.
• We must prepare for each system of the product that we are testing.
• However, we can often take advantage of commonalities that may exist.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Software reliability engineering helps guide feature, load, and regression test.
• Feature test-
occurs first. It consists of single executions of operations, with interactions between the
operations minimized. The focus is on whether the operation executes properly.
• Load test-
which attempts to represent the field use and environment as accurately as possible, with
operations executing simultaneously and interacting. Interactions can occur directly,
through the data, or as a result of conflicts for resources.
• Regression test-
consists of feature test that you conduct after every build involving significant change.
It is often periodic (a week is a common period, although intervals can be as short as a day
or as long as a month (depends on factors such as system size and volatility and the degree
to which a system must be ready for rapid release when market conditions change.)
The focus is to reveal faults that may have been spawned in the process of change.
• Beta test does not require much preparation, except for any work needed to record
results, since you expose your product directly to the field environment.
By Priya Singh (Assistant Professor, Dept of SE, DTU)
2. Some terminologies
• Runs- When we execute the tests, we will be executing a set of runs. A run is a specific
instance of an operation. Thus, like an operation, you can execute it in a series of
noncontiguous time segments on different machines.
• Input variable- An input variable is a variable that exists external to an operation and
influences its execution. If an operation simply receives a data item and prints it out or
stores it, the data item is not an input variable.
• Input state- You characterize a run by the operation and its input state or complete set of
input variables and their values.
• Input space- The input space is the complete set of possible input states.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Direct input variable- A direct input variable is an input variable that is external to an
operation and controls the execution of the operation in an easily traceable way. This
permits you to recognize the relationship between values of the variable and the
processing that results, and to use this information to optimally select values to test. It
results from a deliberate design decision. Some examples are an argument, a menu
selection, or entered data
• Indirect input variable- An indirect input variable is an input variable that is external to
an operation and that influences execution of the operation in a way that is not easily
traceable. This makes it impractical to recognize the relationship between values of the
variable and the processing that results, and hence to optimally select values to test. An
indirect input variable usually involves an environmental influence. Some examples are
amount of data degradation, traffic level and history, and run sequence. We "select" values
of indirect input variables indirectly by testing over time with as accurate a representation
of field conditions as possible.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Test procedure- A test procedure is the test controller for load test that sets up
environmental conditions and invokes at various times test cases that it randomly selects
from the test case set.
• Test operational profile- The test operational profile is the operational profile modified
to ensure that critical operations are tested adequately and to account for reused
operations.
• Note- In load test, we bring the full influence of the indirect input variables to bear. We
drive load test with a test procedure.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• The concept of run recognizes any difference in functionality. Two runs within the same
operation may differ in failure behavior, but this is less likely.
• A run involves the execution of a test case. A test case is a partial specification of a run,
characterized by a complete set of direct input variables with values.
• Software reliability engineering differentiates "test case" from "run" because we want to
be able to specify test cases independently of the environment in which we are executing.
Incidentally, note that a test case is a specification and is not the same thing as a test
script.
• We control the indirect input variables carefully during feature and regression test so that
they have little influence. This is necessary to limit the possible causes of failure until we
know that the operation itself is performing reliably. Otherwise, debugging would be
unnecessarily complex. For example, in the case of feature or regression test for Fone
Follower, reinitialization would give us the same data state for each run. The resource
state would be irrelevant because we would confine test to just one operation at a time and
there would be no conflicts or queues for resources.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

3. Procedure for preparing for test
• We must, for each base product and variation:
1. Specify the new test cases for new operations for the current release, based on the
operational profile
2. Specify the test procedure, based on the test operational profile and traffic level.

In addition, we must provide for:

1. Setup and invocation of test cases.
2. Recording each run executed (operation, test case, invocation time) and outputs.
3. Recording of the number of natural or time units at frequent intervals so we can establish when
failures occurred.
4. Cleanup.

Note: Recording would not be necessary if we found all failures at the time of test and none were
found by later examination of test results, but this rarely occurs in practice.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4. Preparing test cases
Way-1
• In theory, one could prepare test cases by simply recording in the field all the input
variables (user and other interface interactions and environmental variables) needed to
initiate the runs that make up the field execution of the software.
• This might result in less effort and cost than developing the operational profile and
preparing the test cases.
Reasons to avoid recording-
• Recording would necessarily be of a previous release, so it would not adequately
represent the load expected for the new system with its new features.
• Also, you may miss some rarely occurring but critical runs not experienced in the
recording period.
• Finally, one loses the possibility of improving test efficiency through careful test
selection, using equivalence classes. This option is not possible when using record and
playback.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

Way-2
To specify the new test cases for the new operations of the current release, you must:
1. Estimate the number of new test cases that will be available for the current release
2. Allocate the new test cases among the base product and variations (supersystems use the same
test cases as their base products or variations)
3. Distribute the test cases of the base product and each variation among its new operations
4. Detail the new test cases for each new operation.

It shall be followed by-

Once you have done this, you add the new test cases to the test cases from previous releases,
removing the test cases of operations that you removed from the current release.

Definition of new operation-

We consider an operation "new" if we modify it in any nontrivial way. Deciding whether an operation
in a new release is new or reused hinges on whether there has been any change that might introduce
one or more faults. If you have any doubt, be conservative and designate it as new.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4.1 Planning number of new test cases for current release
• It involves first estimating the number of new test cases that you need, and then the
number that you have the capacity to prepare.
• From these two figures you determine the number of new test cases you will prepare.
• You typically estimate the number of new test cases you will prepare late in the
requirements phase when the general outline of the requirements becomes clear.
• If the reliability of previous releases has been inadequate, you should consider both
increasing the number of new test cases for new operations and providing additional test
cases for reused operations as well.
• You will also need more time for load test of software with high-reliability requirements
so that you cover invocation of more different sequences of test cases and more different
values of indirect input variables.
• If most use of the product involves reused operations, then it will generally be better to
estimate the number of new test cases needed by more directly taking advantage of
experience from previous releases of your product.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Let T be the total cumulative test cases from all previous releases. If experience from
these releases indicates that reliability was too low, then increase this number; if too high,
decrease it. Let N be the total occurrence probability of the new operations for the current
release, and R be the total occurrence probability of the reused operations for the current
release. Compute the new test cases needed for the current release as (N / R) x T.
• Note: We should always have N + R = 1 or something is wrong.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Factors in estimating the number of new test cases-
i. time and
ii. cost.
• In order to account for both factors, compute the number of test cases you have time to
prepare and the number you can afford to prepare.
• Take the minimum of these two numbers as the number of test cases you can prepare.
• The Preparing for Test activity does not include any execution of tests.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4.2 Allocating new test cases
• The next step is to allocate the new test cases among the base product and its variations.
• In doing this, we focus on variations that are due to functionality rather than
implementation differences.
• Functionality differences result in new and separate operations or modified operations.
• These will all need new test cases. Variations due to implementation differences will not.
• We will base allocation of new test cases on the unique new use for the associated system
(base product or variation) we are currently considering.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• We designate the unique new use by the symbol U.
• Let F be the expected fraction of field use for the associated system we are considering.
This fraction is the ratio of its use in the field to total use in the field.
• Let S be the sum of the occurrence probabilities of the new operations that are
functionally different from those of previously considered associated systems.
• You can now compute unique new use U for base product and each variation in tum as
U=FxS
• Thus the fraction of field use depends on the number of sites and the amount of field use
per site.
• We now compute a new test case allocation fraction L for the base product and each
variation by dividing its unique new use U by the total of the Us for the base product and
all the variations.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• Finally, compute the new test cases C for the base product and each variation by
multiplying the total new test cases Cα (considering the available staff would permit only
600 test cases to be developed, so what we could develop was limited to that number. ) by
the allocation fraction L for the base product and each variation.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

4. 3 Distributing new test cases among new operations
• We will now distribute the new test cases allocated to each associated system (base
product or variation) among its new operations.
• For each of these associated systems, we first determine an occurrence proportion for
each new operation.
• An occurrence proportion is the proportion of occurrences of a new operation with
respect to the occurrences of all new operations for a release.
• For an initial release, the occurrence proportions are the occurrence probabilities of the
operational profile. This is because, in the first release, all operations are new operations.
• For subsequent releases, divide the occurrence probability of each new operation by the
total of the occurrence probabilities of the new operations.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• We distribute the new test cases to each new operation initially by multiplying the number
allocated to the system by the occurrence proportion, rounding to the nearest integer.
• We must distribute a minimum of one test case for each new operation so that we can test
it.
• Next we identify critical new operations (the one for which successful execution adds a
great deal of extra value and failure causes a great deal of impact with respect to human
life, cost, or system capability).
• To identify critical new operations, look for operations that need much higher much
reliabilities and hence much lower failure intensity objectives than the system failure
intensity objective.
• Then we compute the acceleration factor A, which is given by A = FIO(system) /
FIO(operation). We will use this factor to increase the occurrence proportion, which will
increase the number of new test cases distributed to the operation by the same factor.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• We compute the modified number of new test cases for critical operations by first
multiplying their occurrence proportions by their acceleration factors. Then we multiply
the new occurrence proportions by the number of new test cases allocated to the system,
rounding to the nearest integer.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

5. Preparing test procedures
• To specify the test procedure, we must specify the test operational profile and the traffic
level and reproduce any other significant environmental conditions necessary to make
load test represent field use.
• You specify the traffic level, which is indicated by the average total operation occurrence
rate, as a function of time.
• To specify the average total operation occurrence rate as a function of time, you specify a
list of times from the start of execution of the system at which the average total operation
occurrence rate changed, along with the new average total operation occurrence rates.
• You can simplify this list by selecting the times at a constant period such as one hour, but
you should use whatever frequency is appropriate and practical for your application.

By Priya Singh (Assistant Professor, Dept of SE, DTU)

• To specify a test operational profile for an associated system, we first start with the
corresponding operational profile for the release.
• We modify the occurrence probabilities of the operational profile for the critical
operations by multiplying the occurrence probability for each critical operation by its
acceleration.
• We then normalize the results (so that the sum of the modified occurrence probabilities is
1) by dividing the modified probabilities by the sum of the modified probabilities.

By Priya Singh (Assistant Professor, Dept of SE, DTU)