Testing is a process used to help identify the correctness, completeness and quality of developed

computer software. With that in mind, testing can never completely establish the correctness of
computer software.

There are many approaches to software testing, but effective testing of complex products is essentially
a process of investigation, not merely a matter of creating and following rote procedure. One
definition of testing is "the process of questioning a product in order to evaluate it", where the
"questions" are things the tester tries to do with the product, and the product answers with its
behavior in reaction to the probing of the tester. Although most of the intellectual processes of testing
are nearly identical to that of review or inspection, the word testing is connoted to mean the dynamic
analysis of the product—putting the product through its paces.

The quality of the application can and normally does vary widely from system to system but some of
the common quality attributes include reliability, stability, portability, maintainability and usability.
Refer to the ISO standard ISO 9126 for a more complete list of attributes and criteria.

Testing helps is Verifying and Validating if the Software is working as it is intended to be working.
Thins involves using Static and Dynamic methodologies to Test the application.

Because of the fallibility of its human designers and its own abstract, complex nature, software
development must be accompanied by quality assurance activities. It is not unusual for developers to
spend 40% of the total project time on testing. For life-critical software (e.g. flight control, reactor
monitoring), testing can cost 3 to 5 times as much as all other activities combined. The destructive
nature of testing requires that the developer discard preconceived notions of the correctness of
his/her developed software.

Software Testing Fundamentals

Testing objectives include

1. Testing is a process of executing a program with the intent of finding an error.

2. A good test case is one that has a high probability of finding an as yet undiscovered error.
3. A successful test is one that uncovers an as yet undiscovered error.

Testing should systematically uncover different classes of errors in a minimum amount of time and
with a minimum amount of effort. A secondary benefit of testing is that it demonstrates that the
software appears to be working as stated in the specifications. The data collected through testing can
also provide an indication of the software's reliability and quality. But, testing cannot show the
absence of defect -- it can only show that software defects are present.

One Stop Testing has been developed to give all the information on Software Testing in one place. This
site contains the information on the following:

When Testing should start:

Testing early in the life cycle reduces the errors. Test deliverables are associated with every phase of
development. The goal of Software Tester is to find bugs, find them as early as possible, and make
them sure they are fixed.

The number one cause of Software bugs is the Specification. There are several reasons specifications
are the largest bug producer.

In many instances a Spec simply isn’t written. Other reasons may be that the spec isn’t thorough
enough, its constantly changing, or it’s not communicated well to the entire team. Planning software is
vitally important. If it’s not done correctly bugs will be created.

The next largest source of bugs is the Design, That’s where the programmers lay the plan for their
Software. Compare it to an architect creating the blue print for the building, Bugs occur here for the
same reason they occur in the specification. It’s rushed, changed, or not well communicated.
Coding errors may be more familiar to you if you are a programmer. Typically these can be traced to
the Software complexity, poor documentation, schedule pressure or just plain dump mistakes. It’s
important to note that many bugs that appear on the surface to be programming errors can really be
traced to specification. It’s quite common to hear a programmer say, “ oh, so that’s what its supposed
to do. If someone had told me that I wouldn’t have written the code that way.”

The other category is the catch-all for what is left. Some bugs can blamed for false positives,
conditions that were thought to be bugs but really weren’t. There may be duplicate bugs, multiple
ones that resulted from the square root cause. Some bugs can be traced to Testing errors.

Costs: The costs re logarithmic- that is, they increase tenfold as time increases. A bug found and
fixed during the early stages when the specification is being written might cost next to nothing, or 10
cents in our example. The same bug, if not found until the software is coded and tested, might cost $1
to $10. If a customer finds it, the cost would easily top $100.

When to Stop Testing

This can be difficult to determine. Many modern software applications are so complex, and run in such
as interdependent environment, that complete testing can never be done. "When to stop testing" is
one of the most difficult questions to a test engineer. Common factors in deciding when to stop are:

• Deadlines ( release deadlines,testing deadlines.)

• Test cases completed with certain percentages passed
• Test budget depleted
• Coverage of code/functionality/requirements reaches a specified point
• The rate at which Bugs can be found is too small
• Beta or Alpha Testing period ends
• The risk in the project is under acceptable limit.

Practically, we feel that the decision of stopping testing is based on the level of the risk acceptable to
the management. As testing is a never ending process we can never assume that 100 % testing has
been done, we can only minimize the risk of shipping the product to client with X testing done. The
risk can be measured by Risk analysis but for small duration / low budget / low resources project, risk
can be deduced by simply: -

• Measuring Test Coverage.

• Number of test cycles.
• Number of high priority bugs.
Test Strategy:

How we plan to cover the product so as to develop an adequate assessment of quality.

A good test strategy is:

Specific
Practical
Justified

The purpose of a test strategy is to clarify the major tasks and challenges of the test project.

Test Approach and Test Architecture are other terms commonly used to describe what I’m calling test
strategy.

Example of a poorly stated (and probably poorly conceived) test strategy:

"We will use black box testing, cause-effect graphing, boundary testing, and white box testing to test
this product against its specification."

Test Strategy: Type of Project, Type of Software, when Testing will occur, Critical Success factors,
Tradeoffs

Test Plan - Why

• Identify Risks and Assumptions up front to reduce surprises later.

• Communicate objectives to all team members.
• Foundation for Test Spec, Test Cases, and ultimately the Bugs we find.
Failing to plan = planning to fail.

Test Plan - What

• Derived from Test Approach, Requirements, Project Plan, Functional Spec., and Design Spec.
• Details out project-specific Test Approach.
• Lists general (high level) Test Case areas.
• Include testing Risk Assessment.
• Include preliminary Test Schedule
• Lists Resource requirements.

Test Plan

The test strategy identifies multiple test levels, which are going to be performed for the project.
Activities at each level must be planned well in advance and it has to be formally documented. Based
on the individual plans only, the individual test levels are carried out.

Entry means the entry point to that phase. For example, for unit testing, the coding must be complete
and then only one can start unit testing. Task is the activity that is performed. Validation is the way in
which the progress and correctness and compliance are verified for that phase. Exit tells the
completion criteria of that phase, after the validation is done. For example, the exit criterion for unit
testing is all unit test cases must pass.

Unit Test Plan {UTP}

The unit test plan is the overall plan to carry out the unit test activities. The lead tester prepares it
and it will be distributed to the individual testers, which contains the following sections.

What is to be tested?

The unit test plan must clearly specify the scope of unit testing. In this, normally the basic
input/output of the units along with their basic functionality will be tested. In this case mostly the
input units will be tested for the format, alignment, accuracy and the totals. The UTP will clearly give
the rules of what data types are present in the system, their format and their boundary conditions.
This list may not be exhaustive; but it is better to have a complete list of these details.
Sequence of Testing

The sequences of test activities that are to be carried out in this phase are to be listed in this section.
This includes, whether to execute positive test cases first or negative test cases first, to execute test
cases based on the priority, to execute test cases based on test groups etc. Positive test cases prove
that the system performs what is supposed to do; negative test cases prove that the system does not
perform what is not supposed to do. Testing the screens, files, database etc., are to be given in proper
sequence.

Basic Functionality of Units

How the independent functionalities of the units are tested which excludes any communication
between the unit and other units. The interface part is out of scope of this test level. Apart from the
above sections, the following sections are addressed, very specific to unit testing.

• Unit Testing Tools

• Priority of Program units
• Naming convention for test cases
• Status reporting mechanism
• Regression test approach
• ETVX criteria

Integration Test Plan

The integration test plan is the overall plan for carrying out the activities in the integration test level,
which contains the following sections.

What is to be tested?

This section clearly specifies the kinds of interfaces fall under the scope of testing internal, external
interfaces, with request and response is to be explained. This need not go deep in terms of technical
details but the general approach how the interfaces are triggered is explained.

Sequence of Integration

When there are multiple modules present in an application, the sequence in which they are to be
integrated will be specified in this section. In this, the dependencies between the modules play a vital
role. If a unit B has to be executed, it may need the data that is fed by unit A and unit X. In this case,
the units A and X have to be integrated and then using that data, the unit B has to be tested. This has
to be stated to the whole set of units in the program. Given this correctly, the testing activities will
lead to the product, slowly building the product, unit by unit and then integrating them.

System Test Plan {STP}

The system test plan is the overall plan carrying out the system test level activities. In the system
test, apart from testing the functional aspects of the system, there are some special testing activities
carried out, such as stress testing etc. The following are the sections normally present in system test
plan.

What is to be tested?

This section defines the scope of system testing, very specific to the project. Normally, the system
testing is based on the requirements. All requirements are to be verified in the scope of system
testing. This covers the functionality of the product. Apart from this what special testing is performed
are also stated here.

Functional Groups and the Sequence

The requirements can be grouped in terms of the functionality. Based on this, there may be priorities
also among the functional groups. For example, in a banking application, anything related to customer
accounts can be grouped into one area, anything related to inter-branch transactions may be grouped
into one area etc. Same way for the product being tested, these areas are to be mentioned here and
the suggested sequences of testing of these areas, based on the priorities are to be described.
Acceptance Test Plan {ATP}

The client at their place performs the acceptance testing. It will be very similar to the system test
performed by the Software Development Unit. Since the client is the one who decides the format and
testing methods as part of acceptance testing, there is no specific clue on the way they will carry out
the testing. But it will not differ much from the system testing. Assume that all the rules, which are
applicable to system test, can be implemented to acceptance testing also.

Since this is just one level of testing done by the client for the overall product, it may include test
cases including the unit and integration test level details.

A sample Test Plan Outline along with their description is as shown below:

Test Plan Outline

1. BACKGROUND – This item summarizes the functions of the application system and the tests to be
performed.
2. INTRODUCTION
3. ASSUMPTIONS – Indicates any anticipated assumptions which will be made while testing the
application.
4. TEST ITEMS - List each of the items (programs) to be tested.
5. FEATURES TO BE TESTED - List each of the features (functions or requirements) which will be
tested or demonstrated by the test.
6. FEATURES NOT TO BE TESTED - Explicitly lists each feature, function, or requirement which won't
be tested and why not. 7. APPROACH - Describe the data flows and test philosophy.
Simulation or Live execution, Etc. This section also mentions all the approaches which will be followed
at the various stages of the test execution.
8. ITEM PASS/FAIL CRITERIA Blanket statement - Itemized list of expected output and tolerances
9. SUSPENSION/RESUMPTION CRITERIA - Must the test run from start to completion?
Under what circumstances it may be resumed in the middle?
Establish check-points in long tests.
10. TEST DELIVERABLES - What, besides software, will be delivered?
Test report
Test software
11. TESTING TASKS Functional tasks (e.g., equipment set up)
Administrative tasks
12. ENVIRONMENTAL NEEDS
Security clearance
Office space & equipment
Hardware/software requirements
13. RESPONSIBILITIES
Who does the tasks in Section 10?
What does the user do?
14. STAFFING & TRAINING
15. SCHEDULE
16. RESOURCES
17. RISKS & CONTINGENCIES
18. APPROVALS

The schedule details of the various test pass such as Unit tests, Integration tests, System Tests should
be clearly mentioned along with the estimated efforts.

1. Why test - what is Testing?

2. System Testing myths and legends - What are they?
3. What are the Concepts for Application Test Management?
4. What Test Principles do you Recommend?
5. Test Analysts - What is their Value Add?
6. What do Test Analysts Need?
7. Requirements Traceability - What is this about?
8. What is involved in the Application Test Lifecycle?
9. How to manage Risk Mitigation?
10. What should the Test Team do?

11. What are the Test Team Deliverables

1. Why test - what is Testing?
Testing is a process used to help identify the correctness, completeness and quality of developed
computer software.
2. System Testing myths and legends - What are they?
Myth1: There is no need to test
Myth2: If testing must be done; two weeks at the end of the project is sufficient for testing
Myth3: Re-testing is not necessary
Myth4: Any fool can test
Myth5: The last thing you want is users involved in test
Myth6: The V-model is too complicated
3. What are the Concepts for Application Test Management?
• Testing should be pro-active following the V-model
• Test execution can be a manual process
• Test execution can be an automated process
• It is possible to plan the start date for testing
• It is not possible to accurately plan the end date of testing
• Ending testing is through risk assessment
• A fool with a tool is still a fool
• Testing is not a diagnosis process
• Testing is a triage process
• Testing is expensive

• Not testing, can be more expensive

4. What Test Principles do you Recommend?
• Test involvement early in the lifecycle - Test Architect Signs off Requirements - Test Architect
Signs off Use Cases • Fail Fast - Identify failures early via core test scripts • All Test Phases have
equal value - Each Test Phase has its own value add • RACI chart everything • Testing is a pro-
active activity - Plan the Test - Test the Plan • Finding defects is good - Ignorance of faults in a
non-conformant system is no excuse
5. Test Analysts - What is their Value Add?
• Understand the system under test
• Document Assumptions
• Create and execute repeatable tests
• Value add through negative testing
• Contribute to Impact Analysis when assessing Changes

• Contribute to the risk assessment when considering to end testing

6. What do Test Analysts Need?
• Education
• Test Environment
• Test Tools

• Access
7. Requirements Traceability - What is this about?
• Tracing requirements to test cases
• Tracing test cases to requirements
• Should be a feature of the Test Asset Management tool
• Automatic on-demand process

• Pie chart reporting

8. What is involved in the Application Test Lifecycle?
• Unit testing
• Module testing
• Component testing
• Component integration testing
• Subsystem testing
• System testing
Risk Analysis:

A risk is a potential for loss or damage to an Organization from materialized threats. Risk Analysis
attempts to identify all the risks and then quantify the severity of the risks.A threat as we have seen
is a possible damaging event. If it occurs, it exploits vulnerability in the security of a computer based
system.

Risk Identification:

1. Software Risks: Knowledge of the most common risks associated with Software development, and
the platform you are working on.

2. Business Risks: Most common risks associated with the business using the Software

3. Testing Risks: Knowledge of the most common risks associated with Software Testing for the
platform you are working on, tools being used, and test methods being applied.

4. Premature Release Risk: Ability to determine the risk associated with releasing unsatisfactory or
untested Software Prodicts.

5. Risk Methods: Strategies and approaches for identifying risks or problems associated with
implementing and operating information technology, products and process; assessing their likelihood,
and initiating strategies to test those risks.

Traceability means that you would like to be able to trace back and forth how and where any
workproduct fulfills the directions of the preceeding (source-) product. The matrix deals with the
where, while the how you have to do yourself, once you know the where.

Take e.g. the Requirement of UserFriendliness (UF). Since UF is a complex concept, it is not solved by
just one design-solution and it is not solved by one line of code. Many partial design-solutions may
contribute to this Requirement and many groups of lines of code may contribute to it.

A Requirements-Design Traceability Matrix puts on one side (e.g. left) the sub-requirements that
together are supposed to solve the UF requirement, along with other (sub-)requirements. On the
other side (e.g. top) you specify all design solutions. Now you can connect on the crosspoints of the
matrix, which design solutions solve (more, or less) any requirement. If a design solution does not
solve any requirement, it should be deleted, as it is of no value.

Having this matrix, you can check whether any requirement has at least one design solution and by
checking the solution(s) you may see whether the requirement is sufficiently solved by this (or the set
of) connected design(s).

If you have to change any requirement, you can see which designs are affected. And if you change
any design, you can check which requirements may be affected and see what the impact is.

In a Design-Code Traceability Matrix you can do the same to keep trace of how and which code solves
a particular design and how changes in design or code affect each other.

Demonstrates that the implemented system meets the user requirements.

Serves as a single source for tracking purposes.

Identifies gaps in the design and testing.

Prevents delays in the project timeline, which can be brought about by having to backtrack to fill the
gaps

Static Testing: This section covers - Static Testing, Static material Testing, Static Testing Machine,
Static Materials Testing, Software Testing Static Analysis, Static Control Testing, Static Testing
Techniques, Static Pressure Testing The Verification activities fall into the category of Static Testing.
During static testing, you have a checklist to check whether the work you are doing is going as per the
set standards of the organization. These standards can be for Coding, Integrating and Deployment.
Review's, Inspection's and Walkthrough's are static testing methodologies.

Dynamic Testing: This section covers - Dynamic Tribology Oscillation Testing, Dynamic Testing,
Dynamic Testing Machine, Dynamic Material & Tear, Wall, Load Testing Dynamic Testing involves
working with the software, giving input values and checking if the output is as expected. These are
the Validation activities. Unit Tests, Integration Tests, System Tests and Acceptance Tests are few of
the Dynamic Testing methodologies. As we go further, let us understand the various Test Life Cycle's
and get to know the Testing Terminologies. To understand more of software testing, various
methodologies, tools and techniques, you can download the Software Testing Guide Book from here.

Difference Between Static and Dynamic Testing: Please refer the definition of Static Testing to
observe the differnce between the static testing and dynamic testing.

Blackbox Testing: This section covers - What is blackbox testing, difference between blackbox testing
and whitebox testing, Blackbox Testing plans, unbiased blackbox testing Introduction

Black box testing attempts to derive sets of inputs that will fully exercise all the functional
requirements of a system. It is not an alternative to white box testing. This type of testing attempts to
find errors in the following categories:

1. incorrect or missing functions,

2. interface errors,
3. errors in data structures or external database access,
4. performance errors, and 5. initialization and termination errors.

Tests are designed to answer the following questions:

1. How is the function's validity tested?

2. What classes of input will make good test cases?
3. Is the system particularly sensitive to certain input values?
4. How are the boundaries of a data class isolated?
5. What data rates and data volume can the system tolerate?
6. What effect will specific combinations of data have on system operation?

White box testing should be performed early in the testing process, while black box testing tends to
be applied during later stages. Test cases should be derived which

1. reduce the number of additional test cases that must be designed to achieve reasonable testing,
and
2. tell us something about the presence or absence of classes of errors, rather than an error
associated only with the specific test at hand.

Equivalence Partitioning

This method divides the input domain of a program into classes of data from which test cases can be
derived. Equivalence partitioning strives to define a test case that uncovers classes of errors and
thereby reduces the number of test cases needed. It is based on an evaluation of equivalence classes
for an input condition. An equivalence class represents a set of valid or invalid states for input
conditions.

Equivalence classes may be defined according to the following guidelines:

1. If an input condition specifies a range, one valid and two invalid equivalence classes are defined.
2. If an input condition requires a specific value, then one valid and two invalid equivalence classes
are defined.
3. If an input condition specifies a member of a set, then one valid and one invalid equivalence class
are defined.
4. If an input condition is boolean, then one valid and one invalid equivalence class are defined.

Boundary Value Analysis

This method leads to a selection of test cases that exercise boundary values. It complements
equivalence partitioning since it selects test cases at the edges of a class. Rather than focusing on
input conditions solely, BVA derives test cases from the output domain also. BVA guidelines include:

1. For input ranges bounded by a and b, test cases should include values a and b and just above and
just below a and b respectively.
2. If an input condition specifies a number of values, test cases should be developed to exercise the
minimum and maximum numbers and values just above and below these limits.
3. Apply guidelines 1 and 2 to the output.
4. If internal data structures have prescribed boundaries, a test case should be designed to exercise
the data structure at its boundary.

Cause-Effect Graphing Techniques

Cause-effect graphing is a technique that provides a concise representation of logical conditions and
corresponding actions. There are four steps:

1. Causes (input conditions) and effects (actions) are listed for a module and an identifier is assigned
to each.
2. A cause-effect graph is developed.
3. The graph is converted to a decision table.
4. Decision table rules are converted to test cases.
What is blackbox testing, difference between blackbox testing and whitebox testing, Blackbox Testing
plans, unbiased blackbox testing

Whitebox Testing: This section has notes on Whitebox Testing (QA) White box testing is a test case
design method that uses the control structure of the procedural design to derive test cases. Test cases
can be derived that

1. guarantee that all independent paths within a module have been exercised at least once,
2. exercise all logical decisions on their true and false sides,
3. execute all loops at their boundaries and within their operational bounds, and
4. exercise internal data structures to ensure their validity.

The Nature of Software Defects

Logic errors and incorrect assumptions are inversely proportional to the probability that a program
path will be executed. General processing tends to be well understood while special case processing
tends to be prone to errors.

We often believe that a logical path is not likely to be executed when it may be executed on a regular
basis. Our unconscious assumptions about control flow and data lead to design errors that can only be
detected by path testing.

Typographical errors are random.

Basis Path Testing

This method enables the designer to derive a logical complexity measure of a procedural design and
use it as a guide for defining a basis set of execution paths. Test cases that exercise the basis set are
guaranteed to execute every statement in the program at least once during testing.

Flow Graphs

Flow graphs can be used to represent control flow in a program and can help in the derivation of the
basis set. Each flow graph node represents one or more procedural statements. The edges between
nodes represent flow of control. An edge must terminate at a node, even if the node does not
represent any useful procedural statements. A region in a flow graph is an area bounded by edges and
nodes. Each node that contains a condition is called a predicate node. Cyclomatic complexity is a
metric that provides a quantitative measure of the logical complexity of a program. It defines the
number of independent paths in the basis set and thus provides an upper bound for the number of
tests that must be performed.

The Basis Set

An independent path is any path through a program that introduces at least one new set of processing
statements (must move along at least one new edge in the path). The basis set is not unique. Any
number of different basis sets can be derived for a given procedural design. Cyclomatic complexity,
V(G), for a flow graph G is equal to

1. The number of regions in the flow graph.

2. V(G) = E - N + 2 where E is the number of edges and N is the number of nodes.
3. V(G) = P + 1 where P is the number of predicate nodes.

Deriving Test Cases

1. From the design or source code, derive a flow graph.
2. Determine the cyclomatic complexity of this flow graph.
Even without a flow graph, V(G) can be determined by counting
the number of conditional statements in the code.
3. Determine a basis set of linearly independent paths.
Predicate nodes are useful for determining the necessary paths.
4. Prepare test cases that will force execution of each path in the basis set.
Each test case is executed and compared to the expected results.

Automating Basis Set Derivation

The derivation of the flow graph and the set of basis paths is amenable to automation. A software tool
to do this can be developed using a data structure called a graph matrix. A graph matrix is a square
matrix whose size is equivalent to the number of nodes in the flow graph. Each row and column
correspond to a particular node and the matrix corresponds to the connections (edges) between
nodes. By adding a link weight to each matrix entry, more information about the control flow can be
captured. In its simplest form, the link weight is 1 if an edge exists and 0 if it does not. But other
types of link weights can be represented:

� the probability that an edge will be executed,

� the processing time expended during link traversal,
� the memory required during link traversal, or
� the resources required during link traversal.

Graph theory algorithms can be applied to these graph matrices to help in the analysis necessary to
produce the basis set.

Loop Testing

This white box technique focuses exclusively on the validity of loop constructs. Four different classes
of loops can be defined:

1. simple loops,
2. nested loops,
3. concatenated loops, and
4. unstructured loops.

Simple Loops

The following tests should be applied to simple loops where n is the maximum number of allowable
passes through the loop:

1. skip the loop entirely,

2. only pass once through the loop,
3. m passes through the loop where m < n,
4. n - 1, n, n + 1 passes through the loop.

Nested Loops

The testing of nested loops cannot simply extend the technique of simple loops since this would result
in a geometrically increasing number of test cases. One approach for nested loops:

1. Start at the innermost loop. Set all other loops to minimum values.
2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimums.
Add tests for out-of-range or excluded values.
3. Work outward, conducting tests for the next loop while keeping all other outer loops at minimums
and other nested loops to typical values.
4. Continue until all loops have been tested.

Concatenated Loops

Concatenated loops can be tested as simple loops if each loop is independent of the others. If they are
not independent (e.g. the loop counter for one is the loop counter for the other), then the nested
approach can be used.

Unstructured Loops

This type of loop should be redesigned not tested!!!

Other White Box Techniques
Other white box testing techniques include:

1. Condition testing
exercises the logical conditions in a program.
2. Data flow testing
selects test paths according to the locations of definitions and uses of variables in the program.

Unit Testing: This section covers - Software Unit Testing, tools, research topics, toolkits, extreme
programming unit testing In computer programming, a unit test is a method of testing the correctness
of a particular module of source code.

The idea is to write test cases for every non-trivial function or method in the module so that each test
case is separate from the others if possible. This type of testing is mostly done by the developers.

Benefits

The goal of unit testing is to isolate each part of the program and show that the individual parts are
correct. It provides a written contract that the piece must satisfy. This isolated testing provides four
main benefits:

Encourages change

Unit testing allows the programmer to refactor code at a later date, and make sure the module still
works correctly (regression testing). This provides the benefit of encouraging programmers to make
changes to the code since it is easy for the programmer to check if the piece is still working properly.

Simplifies Integration

Unit testing helps eliminate uncertainty in the pieces themselves and can be used in a bottom-up
testing style approach. By testing the parts of a program first and then testing the sum of its parts will
make integration testing easier.

Documents the code

Unit testing provides a sort of "living document" for the class being tested. Clients looking to learn
how to use the class can look at the unit tests to determine how to use the class to fit their needs.

Separation of Interface from Implementation

Because some classes may have references to other classes, testing a class can frequently spill over
into testing another class. A common example of this is classes that depend on a database; in order to
test the class, the tester finds herself writing code that interacts with the database. This is a mistake,
because a unit test should never go outside of its own class boundary. As a result, the software
developer abstracts an interface around the database connection, and then implements that interface
with their own Mock Object. This results in loosely coupled code, thus minimizing dependencies in the
system.

Limitations

It is important to realize that unit-testing will not catch every error in the program. By definition, it
only tests the functionality of the units themselves. Therefore, it will not catch integration errors,
performance problems and any other system-wide issues. In addition, it may not be trivial to
anticipate all special cases of input the program unit under study may receive in reality. Unit testing is
only effective if it is used in conjunction with other software testing activities.

Unit Testing - Software Unit Testing, Tools, Research Topics, Toolkits, Extreme Programming Unit
Testing
Requirements Testing: This section covers - (half face) Respirator fit testing requirements,
GFCI, NCLEX testing requirements, CNA testing in Georgia Requirements Usage:

To ensure that system performs correctly

To ensure that correctness can be sustained for a considerable period of time.

System can be tested for correctness through all phases of SDLC but incase of reliability the
programs should be in place to make system operational.

Objective:

Successfully implementation of user requirements,/li>

Correctness maintained over considerable period of time Processing of the application complies
with the organization’s policies and procedures.

Secondary users needs are fulfilled:

Security officer

DBA

Internal auditors

Record retention

Comptroller

How to Use

Test conditions created

These test conditions are generalized ones, which becomes test cases as the SDLC progresses
until system is fully operational.

Test conditions are more effective when created from user’s requirements.

Test conditions if created from documents then if there are any error in the documents those will
get incorporated in Test conditions and testing would not be able to find those errors.

Test conditions if created from other sources (other than documents) error trapping is effective.

Functional Checklist created.

When to Use

Every application should be Requirement tested

Should start at Requirements phase and should progress till operations and maintenance phase.

The method used to carry requirement testing and the extent of it is important.

Example

Creating test matrix to prove that system requirements as documented are the requirements
desired by the user.

Creating checklist to verify that application complies to the organizational policies and procedures.

Regression Testing: Software Regression Testing, Network Regression Testing (Software),

Automated Regression Testing, Dod regression standard testing, Regression Testing Tools,
Web Regression Testing Usage:

All aspects of system remain functional after testing.

Change in one segment does not change the functionality of other segment.
 Objective:

Determine System documents remain current

Determine System test data and test conditions remain current

Determine Previously tested system functions properly without getting effected though changes
are made in some other segment of application system.

How to Use

Test cases, which were used previously for the already tested segment is, re-run to ensure that
the results of the segment tested currently and the results of same segment tested earlier
are same.

Test automation is needed to carry out the test transactions (test condition execution) else the
process is very time consuming and tedious.

In this case of testing cost/benefit should be carefully evaluated else the efforts spend on testing
would be more and payback would be minimum.

When to Use

When there is high risk that the new changes may effect the unchanged areas of application
system.

In development process: Regression testing should be carried out after the pre-determined
changes are incorporated in the application system.

In Maintenance phase : regression testing should be carried out if there is a high risk that loss
may occur when the changes are made to the system

Example

Re-running of previously conducted tests to ensure that the unchanged portion of system
functions properly.

Reviewing previously prepared system documents (manuals) to ensure that they do not get
effected after changes are made to the application system.

Disadvantage

Time consuming and tedious if test automation not done

Regression Testing - Software Testing - Network Regression Testing - Web & Automated Regression
Testing
Error Handling Testing
Usage:
It determines the ability of applications system to process the incorrect transactions properly
Errors encompass all unexpected conditions.
In some system approx. 50% of programming effort will be devoted to handling error
condition.
Objective:
Determine Application system recognizes all expected error conditions
Determine Accountability of processing errors has been assigned and procedures provide a
high probability that errors will be properly corrected
Determine During correction process reasonable control is maintained over errors.
How to Use
A group of knowledgeable people is required to anticipate what can go wrong in the application
system.

It is needed that all the application knowledgeable people assemble to integrate their knowledge
of user area, auditing and error tracking.

Then logical test error conditions should be created based on this assimilated information.

When to Use

Throughout SDLC.

Impact from errors should be identified and should be corrected to reduce the errors to
acceptable level.

Used to assist in error management process of system development and maintenance.

Example

Create a set of erroneous transactions and enter them into the application system then find out
whether the system is able to identify the problems..

Using iterative testing enters transactions and trap errors. Correct them. Then enter transactions
with errors, which were not present in the system earlier.

Manual Support Testing

Usage:

It involves testing of all the functions performed by the people while preparing the data and using
these data from automated system.

Objective:

Verify manual support documents and procedures are correct.

Determine Manual support responsibility is correct

Determine Manual support people are adequately trained.

Determine Manual support and automated segment are properly interfaced.

How to Use

Process evaluated in all segments of SDLC.

Execution of the can be done in conjunction with normal system testing.

Instead of preparing, execution and entering actual test transactions the clerical and supervisory
personnel can use the results of processing from application system.
To test people it requires testing the interface between the people and application system.

When to Use

Verification that manual systems function properly should be conducted throughout the SDLC.

Should not be done at later stages of SDLC.

Best done at installation stage so that the clerical people do not get used to the actual system
just before system goes to production.

Example

Provide input personnel with the type of information they would normally receive from their
customers and then have them transcribe that information and enter it in the computer.

Users can be provided a series of test conditions and then asked to respond to those conditions.
Conducted in this manner, manual support testing is like an examination in which the users
are asked to obtain the answer from the procedures and manuals available to them.

Intersystem Testing

Usage:

To ensure interconnection between application functions correctly.

Objective:

Determine Proper parameters and data are correctly passed between the applications

Documentation for involved system is correct and accurate.

Ensure Proper timing and coordination of functions exists between the application system.

How to Use

Operations of multiple systems are tested.

Multiple systems are run from one another to check that they are acceptable and processed
properly.

When to Use

When there is change in parameters in application system

The parameters, which are erroneous then risk associated to such parameters, would decide the
extent of testing and type of testing.

Intersystem parameters would be checked / verified after the change or new application is placed
in the production.

Example

Develop test transaction set in one application and passing to another system to verify the
processing.
Entering test transactions in live production environment and then using integrated test facility to
check the processing from one system to another.

Verifying new changes of the parameters in the system, which are being tested, are corrected in
the document.

Disadvantage

Time consuming and tedious if test automation not done

Cost may be expensive if system is run several times iteratively.

Control Testing

Usage:

Control is a management tool to ensure that processing is performed in accordance to what

management desire or intents of management.

Objective:

Accurate and complete data

Authorized transactions

Maintenance of adequate audit trail of information.

Efficient, effective and economical process.

Process meeting the needs of the user.

How to Use

To test controls risks must be identified.

Testers should have negative approach i.e. should determine or anticipate what can go wrong in
the application system.

Develop risk matrix, which identifies the risks, controls; segment within application system in
which control resides.

When to Use

Should be tested with other system tests.

Example

file reconciliation procedures work

Manual controls in place.

Parallel Testing

Usage:

To ensure that the processing of new application (new version) is consistent with respect to the
processing of previous application version.

Objective:
 Conducting redundant processing to ensure that the new version or application performs
correctly.

Demonstrating consistency and inconsistency between 2 versions of the application.

How to Use

• Same input data should be run through 2 versions of same application system.
• Parallel testing can be done with whole system or part of system (segment).

When to Use

• When there is uncertainty regarding correctness of processing of new application where the
new and old version are similar.
• In financial applications like banking where there are many similar applications the
processing can be verified for old and new version through parallel testing

Example

• Operating new and old version of a payroll system to determine that the paychecks from both
systems are reconcilable.
• Running old version of application to ensure that the functions of old system are working fine
with respect to the problems encountered in the new system.

Stress Testing

The purpose of stress testing is to find defects of the system capacity of handling large numbers of
transactions during peak periods. For example, a script might require users to login and proceed with
their daily activities while, at the same time, requiring that a series of workstations emulating a large
number of other systems are running recorded scripts that add, update, or delete from the database.

Volume Testing

Whichever title you choose (for us volume test) here we are talking about realistically exercising an
application in order to measure the service delivered to users at different levels of usage. We are
particularly interested in its behavior when the maximum number of users are concurrently active and
when the database contains the greatest data volume.

The creation of a volume test environment requires considerable effort. It is essential that the correct
level of complexity exists in terms of the data within the database and the range of transactions and
data used by the scripted users, if the tests are to reliably reflect the to be production environment.
Once the test environment is built it must be fully utilised. Volume tests offer much more than simple
service delivery measurement. The exercise should seek to answer the following questions:

What service level can be guaranteed. How can it be specified and monitored?

Are changes in user behaviour likely? What impact will such changes have on resource
consumption and service delivery?

Which transactions/processes are resource hungry in relation to their tasks?

What are the resource bottlenecks? Can they be addressed?

How much spare capacity is there?

The purpose of volume testing is to find weaknesses in the system with respect to its handling of
large amount of data during extended time periods

Performance Testing

System performance is generally assessed in terms of response time and throughput rates under
differing processing and configuration conditions. To attack the performance problems, there are
several questions should be asked first:
 How much application logic should be remotely executed?
How much updating should be done to the database server over the network from the
client workstation?

How much data should be sent to each in each transaction?

According to Hamilton [10], the performance problems are most often the result of the client or
server being configured inappropriately.

The best strategy for improving client-sever performance is a three-step process [11]. First,
execute controlled performance tests that collect the data about volume, stress, and loading
tests. Second, analyze the collected data. Third, examine and tune the database queries and, if
necessary, provide temporary data storage on the client while the application is executing.

Software Testing Life Cycle:

The test development life cycle contains the following components:

Requirements
Use Case Document
Test Plan
Test Case
Test Case execution
Report Analysis
Bug Analysis
Bug Reporting

Typical interaction scenario from a user's perspective for system requirements studies or testing. In
other words, "an actual or realistic example scenario". A use case describes the use of a system from
start to finish. Use cases focus attention on aspects of a system useful to people outside of the system
itself.

• Users of a program are called users or clients.

• Users of an enterprise are called customers, suppliers, etc.

Use Case:

A collection of possible scenarios between the system under discussion and external actors,
characterized by the goal the primary actor has toward the system's declared responsibilities, showing
how the primary actor's goal might be delivered or might fail.

Use cases are goals (use cases and goals are used interchangeably) that are made up of scenarios.
Scenarios consist of a sequence of steps to achieve the goal, each step in a scenario is a sub (or mini)
goal of the use case. As such each sub goal represents either another use case (subordinate use case)
or an autonomous action that is at the lowest level desired by our use case decomposition.

This hierarchical relationship is needed to properly model the requirements of a system being
developed. A complete use case analysis requires several levels. In addition the level at which the use
case is operating at it is important to understand the scope it is addressing. The level and scope are
important to assure that the language and granularity of scenario steps remain consistent within the
use case.

There are two scopes that use cases are written from: Strategic and System. There are also three
levels: Summary, User and Sub-function.

Scopes: Strategic and System

Strategic Scope:

The goal (Use Case) is a strategic goal with respect to the system. These goals are goals of value to
the organization. The use case shows how the system is used to benefit the organization.,/p> These
strategic use cases will eventually use some of the same lower level (subordinate) use cases.

System Scope:
Use cases at system scope are bounded by the system under development. The goals represent
specific functionality required of the system. The majority of the use cases are at system scope. These
use cases are often steps in strategic level use cases

Levels: Summary Goal , User Goal and Sub-function.

Sub-function Level Use Case:

A sub goal or step is below the main level of interest to the user. Examples are "logging in" and
"locate a device in a DB". Always at System Scope.

User Level Use Case:

This is the level of greatest interest. It represents a user task or elementary business process. A user
level goal addresses the question "Does your job performance depend on how many of these you do in
a day". For example "Create Site View" or "Create New Device" would be user level goals but "Log In
to System" would not. Always at System Scope.

Summary Level Use Case:

Written for either strategic or system scope. They represent collections of User Level Goals. For
example summary goal "Configure Data Base" might include as a step, user level goal "Add Device to
database". Either at System of Strategic Scope.

Test Documentation

Test documentation is a required tool for managing and maintaining the testing process. Documents
produced by testers should answer the following questions:

• What to test? Test Plan

• How to test? Test Specification
• What are the results? Test Results Analysis Report

Bug Life cycle:

In entomology(the study of real, living Bugs), the term life cycle refers to the various stages that an
insect assumes over its life. If you think back to your high school biology class, you will remember that
the life cycle stages for most insects are the egg, larvae, pupae and adult. It seems appropriate, given
that software problems are also called bugs, that a similar life cycle system is used to identify their
stages of life. Figure 18.2 shows an example of the simplest, and most optimal, software bug life
cycle.
This example shows that when a bug is found by a Software Tester, its logged and assigned to a
programmer to be fixed. This state is called open state. Once the programmer fixes the code , he
assigns it back to the tester and the bugs enters the resolved state. The tester then performs a
regression test to confirm that the bug is indeed fixed and, if it closes it out. The bug then enters its
final state, the closed state.

In some situations though, the life cycle gets a bit more complicated.

In this case the life cycle starts out the same with the Tester opening the bug and assigning to the
programmer, but the programmer doesn’t fix it. He doesn’t think its bad enough to fix and assigns it
to the project manager to decide. The Project Manager agrees with the Programmer and places the Bug
in the resolved state as a “wont-fix” bug. The tester disagrees, looks for and finds a more obvious and
general case that demonstrates the bug, reopens it, and assigns it to the Programmer to fix. The
programmer fixes the bg, resolves it as fixed, and assign it to the Tester. The tester confirms the fix
and closes the bug.
You can see that a bug might undergo numerous changes and iterations over its life, sometimes
looping back and starting the life all over again. Figure below takes the simple model above and adds
to it possible decisions, approvals, and looping that can occur in most projects. Of course every
software company and project will have its own system, but this figure is fairly generic and should
cover most any bug life cycle that you’ll encounter
 The generic life cycle has two additional states and extra connecting lines. The review state is where
Project Manager or the committee, sometimes called a change Control Board, decides whether the bug
should be fixed. In some projects all bugs go through the review state before they’re assigned to the
programmer for fixing. In other projects, this may not occur until near the end of the project, or not at
all. Notice that the review state can also go directly to the closed state. This happens if the review
decides that the bug shouldn’t be fixed – it could be too minor is really not a problem, or is a testing
error. The other is a deferred. The review may determine that the bug should be considered for fixing
at sometime in the future, but not for this release of the software.

The additional line from resolved state back to the open state covers the situation where the tester
finds that the bug hasn’t been fixed. It gets reopened and the bugs life cycle repeats.

The two dotted lines that loop from the closed and the deferred state back to the open state rarely
occur but are important enough to mention. Since a Tester never gives up, its possible that a bug was
thought to be fixed, tested and closed could reappear. Such bugs are often called Regressions. It’s
possible that a deferred bug could later be proven serious enough to fix immediately. If either of these
occurs, the bug is reopened and started through the process again. Most Project teams adopt rules for
who can change the state of a bug or assign it to someone else.For example, maybe only the Project
Manager can decide to defer a bug or only a tester is permitted to close a bug. What’s important is
that once you log a bug, you follow it through its life cycle, don’t lose track of it, and prove the
necessary information to drive it to being fixed and closed.

Bug Report - Why

• Communicate bug for reproducibility, resolution, and regression.

• Track bug status (open, resolved, closed).
• Ensure bug is not forgotten, lost or ignored.

Used to back create test case where none existed before.

Formal testing conducted to enable a user, customer or other authorised entity to determine whether to
Acceptance Testing
accept a system or component.
Actual Outcome The behaviour actually produced when the object is tested under specified conditions.
Ad hoc testing Testing carried out using no recognised test case design technique
Self contained processes that run in the background on a client or server and that perform useful functions
Agents
for a specific user/owner. Agents may monitor exceptions based on criteria or execute automated tasks.
Aggregated Data Data that is results from applying a process to combine data elements. Data that is summarized.
Algorithm A sequence of steps for solving a problem.
Simulated or actual operational testing at an in-house site not otherwise involved with the software
Alpha Testing
developers
An information system containing key attributes of applications deployed in a company. Application
Application Portfolio
portfolios are used as tools to manage the business value of an application throughout its lifecycle.
a test case design technique for a component in which test cass are designed to execute branch condition
Arc Testing
outcomes
Artificial Intelligence
The science of making machines do things which would require intelligence if they were done by humans.
(AI)
Component of a business process. Assets can include people, accommodation, computer systems, networks,
Asset
paper records, fax machines, etc.
A variable that takes on values that might be numeric, text, or logical (true/false). Attributes store the
Attribute
factual knowledge in a knowledge base
The use of software to perform or support test activities, e.g. test management, test design, test execution
and results checking.
There are many factors to consider when planning for software test automation. Automation changes the
Automation
complexion of testing and the test organisation from design through implementation and test execution.
There are tangible and intangible elements and widely held myths about benefits and capabilities of test
automation.
Ability of a component or service to perform its required function at a stated instant or over a stated period
of time.
Availability
It is usually expressed as the availability ratio, i.e. the proportion of time that the service is actually
available for use by the Customers within the agreed service hours.
BS 7925-2 Glossary of British Standard Test terms
 The process of determining the value of a goal by looking for rules that can conclude the goal. Attributes in
Backward chaining
the premise of such rules may be made subgoals for further search if necessary.
An aid to organisational performance management. It helps to focus, not only on the financial targets but also
Balanced Scorecard
on the internal processes, customers and learning and growth issues.
Documents or other outputs identified which should be tested at each stage of development
A snapshot or a position which is recorded. Although the position may be updated later, the baseline
Baseline
remains unchanged and available as a reference of the original state and as a comparison against the
current position.
Basic Block A sequence of one or more consecutive, executable statements containing no branches
Basic test set A set of test cases derived from the code logic which ensure that 100% branch coverage is achieved
The combination of input values and preconditions and the required response for a function of a system. The
Behaviour
full specification of a function would normally comprise one or more behaviours
Beta Testing Operational testing at a site not otherwise involved with the software developers
Integration testing where no incremental testing takes place prior to all the system's components being
Big-bang testing
combined to form the system
Test case selection based on an analysis of the specification of the component without reference to its
Black Box Testing
internal workings
A hierarchically organised database which allows information to flow both in and out from the knowledge
Blackboard
sources.
An approach to integration testing where the lowest level components are tested first, then used to facilitate
Bottom up testing
the testing of higher level components.
A conditional transfer of control from any statement to any other statement in a component, or an
Branch unconditional transfer of control from any statement to any other statement in the component except the
next statement, or
Branch Condition A test case design technique in which test cases are design to execute combinations of branch condition
Combination Testing outcomes
Branch Condition
The percentage of branch condition outcomes in every decision that have been exercised by a test case suite
Coverage
Branch Condition
A test case design technique in which test cases are designed to execute branch condition outcomes
Testing
Branch Coverage The percentage of branches thta have been exercised by a test case suite
Branch Testing A test case design technique for a component in which test cases are designed to execute branch outcomes
A search strategy that examines all rules that could determine the value of the current goal or subgoal
Breadth first search
before backtracking through other rules to determine the value of an unknown attribute in the current rule.
Bridge Equipment and techniques used to match circuits to each other ensuring minimum transmission impairment.
Documents describing the roles, responsibilities and actions necessary to resume business processes
Business Recovery Plans
following a business disruption.
CAST Computer Aided Software Testing
A test tool which records test input as it is sent to the software under test. The input cases stored can than
Capture / Playback Tool
be used to design test cases
Case-Based Reasoning A problem-solving system that relies on stored representations of previously solved problems and their
(CBR) solutions.
A graphical representation of inputs or stimuli (causes) with their associated outputs (effects), which can be
Cause Effect Graph
used to design test cases
Cause Effect Graphing A test case design technique in which test cases are designed by consideration of cause effect graphs
Allowing confidence levels obtained from user input and rule conclusions to be combined to increase the
Certainty processing
overall confidence in the value assigned to an attribute.
The process of confirming that a system or component complies with its specified requirements and is
Certification
acceptable for operational use
The procedure to ensure that all changes are controlled, including the submission, analysis, decision
Change Control
making, approval, implementation and post implementation of the change.
A statement of test objectives, and possibly test ideas. Test charters are amongst other used in exploratory
Charter
testing.
Classes A category that generally describes a group of more specific items or objects.
One expression in the If (premise) or Then (consequent) part of a rule. Often consists of an attribute name
Clause
followed by a relational operator and an attribute value.
An analysis method that determines which parts have not been executed and therefore may require
Code Coverage
additional attention
Designing tests based on objectives derived from implementation such as tests that execute specific control
Code based testing
flow paths or use specific data items
Compatibility testing Testing whether the system is compatible with other systems with which it should communicate
Component A minimal software item for which a separate specification is available
Component Testing The testing of individual software components
Conclusion /
The Then part of a rule, or one clause or expression in this part of the rule.
Consequent
Condition A Boolean expression containing no Boolean operators.
Condition Outcome The evaluation of a condition to TRUE or FALSE
Confidence / Certainty A measure of the confidence assigned to the value of an attribute. Often expressed as a percentage (0 to
factor 100%) or probability (0 to 1.0). 100% or 1.0 implies that the attribute's value is known with certainty.
Component of an infrastructure - or an item, such as a Request For Change, associated with an
infrastructure - that is (or is to be) under the control of Configuration Management.
Configuration Item (CI)
CIs may vary widely in complexity, size and type, from an entire system (including all hardware, software
and documentation) to a single module or a minor hardware component.
The process of identifying and defining Configuration Items in a system.
Configuration
e.g. recording and reporting the status of Configuration Items and Requests For Change, and verifying the
Management
completeness and correctness of Configuration Items.
Some method of judging whether or not the component's action on a particular specified input value
Conformance Criterion
conforms to the specification
Conformance Testing The process of testing that an implementation conforms to the specification on which it is based
Planning to address unwanted occurrences that may happen at a later time. Traditionally, the term has been
Contingency Planning
used to refer to planning for the recovery of IT systems rather than entire business processes.
Control Flow An abstract representation of all possible sequences of events in a program's execution
Control Flow Graph The diagrammatic representation of the possible alternative control flow paths through a component
Elements of a knowledge base other than the attributes and rules that control the user interface, operation
Control information of the inference engine and general strategies employed in implementing a consultation with an expert
system.
Testing of programms or procedures used to convert data from existing systems for use in replacement
Conversion Testing
systems
Correctness The degree to which software conforms to its specification
The degree, expressed as a percentage, to which a specific coverage item has been exercised by a test case
Coverage
suite
Coverage Item An entity or property used as a basis for testing
A measure of success or maturity of a project or process. It can be a state, a deliverable or a milestone. An
Critical Success
example of a CSF would be 'the production of an overall technology strategy'.
Factor(CSF)
Key areas of business activity in which favourable results are necessary for a company to reach its goals.
Data Definition An executable statement where a variable is assigned a value
Data Definition C-use
The percentage of data definition C-use pairs in a component exercised by a test case suite
coverage
Data Definition C-use
A data definition and computation data use, where the data use uses the value defined in the data definition
pair
A database about data and database structures. A catalog of all data elements, containing their names,
Data Dictionary
structures, and information about their usage.
Data Flow Testing Testing in which test cases are designed based on variable usage within the code
Extraction of useful information from data sets. Data mining serves to find information that is hidden within
Data Mining
the available data.
Data Use En executable statement where the value of a variable is accessed
Debugging The process of finding and removing the causes of failures in software
A program point at which the control flow has two or more alternative routes
Decision The choice of one from among a number of alternatives; a statement indicating a commitment to a specific
course of action.
Decision Condition A condition within a decision
Decision Coverage The percentage of decision outcomes exercised by a test case suite
Decision Outcome The result of a decision which therefore determines the control flow alternative taken
A delta, or partial, Release is one that includes only those CIs within the Release unit that have actually
changed or are new since the last full or delta Release.
Delta Release
For example, if the Release unit is the program, a Delta Release contains only those modules that have
changed, or are new, since the last Full Release of the program or the last delta Release of certain modules.
A belief maintenance system that considers evidence from a number of sources, maintains information on
Dempster-Shafer Theory
the evidence, and combines the pieces of evidence to produce a decision.
A search strategy that backtracks through all of the rules in a knowledge base that could lead to determining
Depth first search.
the value of the attribute that is the current goal or subgoal.
Descriptive Model Physical, conceptual or mathematical models that describe situations as they are or as they actually appear.
Designing tests based on objectives derived from the architectural or detail design of the software. This is
Design Based Testing
excellent for testing the worst case behaviour of algorithms
A document specifying the test conditions (coverage items) for a test item, the detailed test approach and
Design specification
identifying the associated high level test cases.
Desk Checking The testing of software by the manual simulation of its execution
Mathematical models that are constructed for a condition of assumed certainty. The models assume there is
Deterministic Model
only one possible result (which is known) for each alternative course or action.
Dirty Testing Testing which demonstrates that the system under test does not work
Domain Expert A person who has expertise in a specific problem domain.
Downtime Total period that a service or component is not operational, within an agreed service times.
Error A human action producing an incorrect result
A test case design tecnique where the experience of the tester is used to postulate which faults might occur
Error Guessing
and to design tests specifically to expose them
A document produced at the end of the test process summarizing all testing activities and results. It also
Evaluation report
contains an evaluation of the test process and lessons learned.
A statement which, when compiled, is translated into object code, which will be executed procedurally when
Executable statement
the program is running and may perform an action on program data.
Exhaustive testing The last executable statement within a component.
Exit point The last executable statement within a component.
Expected outcome The behaviour predicted by the specification of an object under specified conditions.
A domain specific knowledge base combined with an inference engine that processes knowledge encoded in
Expert system
the knowledge base to respond to a user's request for advice.
Specialized domain knowledge, skills, tricks, shortcuts and rules-of-thumb that provide an ability to rapidly
Expertise
and effectively solve problems in the problem domain.
A fault, if encountered, may cause a failure, which is a deviation of the software from its expected delivery or
Failure
service
Fault A manifestation of an error in software (also known as a defect or a bug)
A rule fires when the if part (premise) is proven to be true. If the rule incorporates an else component, the
Firing a rule
rule also fires when the if part is proven to be false.
Validation carried out to demonstrate that the delivered system can be used to carry out the tasks for which
Fit for purpose testing
it was acquired.
Forward chaining Applying a set of previously determined facts to the rules in a knowledge base to see if any of them will fire.
All components of the Release unit that are built, tested, distributed and implemented together. See also
Full Release
delta Release.
The document that describes in detail the characteristics of the product with regard to its intended
Functional specification
capability.
Fuzzy variables and Variables that take on multiple values with various levels of certainty and the techniques for reasoning with
fuzzy logic such variables.
Search procedures that use the mechanics of natural selection and natural genetics. It uses evolutionary
Genetic Algorithms
techniques, based on function optimization and artificial intelligence, to develop a solution.
Geographic Information
A support system which represents data using maps.
Systems (GIS)
Glass box testing Testing based on an analysis of the internal structure of the component or system.
A designated attribute: determining the values of one or more goal attributes is the objective of interacting
Goal with a rule based expert system.
The solution that the program is trying to reach.
The process of determining the value of a goal by looking for rules that can conclude the goal. Attributes in
Goal directed
the premise of such rules may be made subgoals for further search if necessary.
Graphical User Interface A type of display format that enables the user to choose commands, start programs, and see lists of files
(GUI) and other options by pointing to pictorial representations (icons) and lists of menu items on the screen.
Harness A test environment comprised of stubs and drivers needed to conduct a test.
The informal, judgmental knowledge of an application area that constitutes the "rules of good judgment" in
the field.
Heuristics
Heuristics also encompass the knowledge of how to solve problems efficiently and effectively, how to plan
steps in solving a complex problem, how to improve performance, etc.
IT Infrastructure Library (ITIL) is a consistent and comprehensive documentation of best practice for IT
Service Management.
ITIL
ITIL consists of a series of books giving guidance on the provision of quality IT services, and on the
accommodation and environmental facilities needed to support IT.
Inference New knowledge inferred from existing facts.
Software that provides the reasoning mechanism in an expert system. In a rule based expert system,
Inference engine
typically implements forward chaining and backward chaining strategies.
The organisational artifacts needed to perform testing, consisting of test environments, test tools, office
Infrastructure
environment and procedures.
Inheritance The ability of a class to pass on characteristics and data to its decendants.
Input A variable (whether stored within a component or outside it) which is read by the component
Input Domain The set of all possible inputs
Input Value An instance of an input
Integration testing Testing performed to expose faults in the interfaces and in the interaction between integrated components.
Intelligent Agent Software that is given a particular mission, carries out that mission, and then reports back to the user.
Interface testing Integration testing where the interfaces between system components are tested.
Component testing of individual components in isolation from surrounding components, with surrounding
Isolation testing
components being simulated by stubs.
Item The individual element to be tested. There usually is one test object and many test items.
KBS Knowledge-Based System.
Key Performance
The measurable quantities against which specific Performance Criteria can be set.
Indicator
Knowledge Knowledge refers to what one knows and understands.
Knowledge Acquisition The gathering of expertise from a human expert for entry into an expert system.
Knowledge
The notation or formalism used for coding the knowledge to be stored in a knowledge-based system.
Representation
The encoded knowledge for an expert system. In a rule-based expert system, a knowledge base typically
Knowledge base
incorporates definitions of attributes and rules along with control information.
Knowledge engineering The process of codifying an expert's knowledge in a form that can be accessed through an expert system.
Knowledge-Based A domain specific knowledge base combined with an inference engine that processes knowledge encoded in
System the knowledge base to respond to a user's request for advice.
An incident or problem for which the root cause is known and for which a temporary Work-around or a
Known Error
permanent alternative has been identified.
Lifecycle A series of states connected by allowable transitions.
Linear Programming A mathematical model for optimal solution of resource allocation problems.
Log A chronological record of relevant details about the execution of tests.
Logging The process of recording information about tests executed into a test log.
The ease with which the system/software can be modified to correct faults, modified to meet new
Maintainability
requirements, modified to make future maintenance easier, or adapted to a changed environment.
Maintainability testing Testing to determine whether the system/software meets the specified maintainability requirements.
Management The planning, estimating, monitoring and control of test activities, typically carried out by a test manager.
Metric Measurable element of a service process or function.
A method to determine test case suite thoroughness by measuring the extent to which a test case suite can
Mutation analysis
discriminate the program from slight variants (mutants) of the program.
N-Transitions A sequence of N+1 transitions
Natural Language
A computer system to analyse, understand, and generate natural human-languages.
Processing (NLP)
Negative Testing Testing which demonstrates that the system under test does not work
A system modeled after the neurons (nerve cells) in a biological nervous system.
Neural Network A neural network is designed as an interconnected system of processing elements, each with a limited
number of inputs and outputs. Rather than being programmed, these systems learn to recognise patterns.
Non Functional
Testing of those requirements that do not relate to functionality. I.e. performance or usability
Requirements Testing
The process of reducing a complex data structure into its simplest, most stable structure. In general, the
Normalisation
process entails the removal of redundant attributes, keys, and relationships from a conceptual data model.
Object A software structure which represents an identifiable item that has a well-defined role in a problem domain.
Object Oriented An adjective applied to any system or language that supports the use of objects.
Objective A reason or purpose for designing and executing a test.
Operational Testing Testing conducted to evaluate a system or component in its operational environment
A mechanism to produce the predicted outcomes to compare with the actual outcomes of the software
Oracle
under test
Outcome The actual or predicted outcome of a test
Output A variable whether stored within a component or outside it which is written to by the component
Output Domain The set of all possible outputs
Output Value An instance of an output
P-use A data use in a predicate
Projects in Controlled Environments, is a project management method covering the organisation,
PRINCE2
management and control of projects. PRINCE2 is often used in the UK for all types of projects.
A program interruption that occurs when a page that is marked ‘not in real memory’ is referred to by an
Page Fault
active page.
A software development approach whereby lines of code (production and/or test) of a component are written
Pair programming by two programmers sitting at a single computer. This implicitly means ongoing real-time code reviews are
performed.
Two testers work together to find defects. Typically, they share one computer and trade control of it while
Pair testing
testing.
A test case design technique for a component in which test cases are designed to execute representatives
Partition Testing
from equivalence classes
Pass A test is deemed to pass if its actual result matches its expected result.
Pass/fail criteria Decision rules used to determine whether a test item (function) or feature has passed or failed a test.
 A sequence of events, e.g. executable statements, of a component or system from an entry point to an exit
Path
point.
Path coverage The percentage of paths that have been exercised by a test suite.
Path sensitising Choosing a set of input values to force the execution of a given path.
Path testing A white box test design technique in which test cases are designed to execute paths.
A peer group discussion activity that focuses on achieving consensus on the technical approach to be taken.
Peer Review
A technical review is also known as a peer review.
The degree to which a system or component accomplishes its designated functions within given constraints
Performance
regarding processing time and throughput rate.
A metric, in general high level, indicating to what extent a certain target value or criterion is met. Often
Performance indicator
related to test process improvement objectives, e.g. Defect Detection Percentage (DDP).
Performance testing The process of testing to determine the performance of a software product.
A tool to support performance testing and that usually has two main facilities: load generation and test
transaction measurement. Load generation can simulate either multiple users or high volumes of input
Performance testing tool data. During execution, response time measurements are taken from selected transactions and these are
logged. Performance testing tools normally provide reports based on test logs and graphs of load against
response times
A distinct set of test activities collected into a manageable phase of a project, e.g. the execution activities
Phase
of a test level.
Plan A document describing the scope, approach, resources and schedule of intended test activities.
A high level document describing the principles, approach and major objectives of the organization
Policy
regarding testing.
Portability testing The process of testing to determine the portability of a software product.
Postcondition Environmental and state conditions that must be fulfilled after the execution of a test or test procedure.
Environmental and state conditions that must be fulfilled before the component or system can be executed
Precondition
with a particular test or test procedure.
Predicted outcome The behaviour predicted by the specification of an object under specified conditions.
Priority The level of (business) importance assigned to an item, e.g. defect.
Problem Domain A specific problem environment for which knowledge is captured in a knowledge base.
Process that minimizes the effect on Customer(s) of defects in services and within the infrastructure,
Problem Management
human errors and external events.
Process A set of interrelated activities, which transform inputs into outputs.
A black box test design technique in which test cases are designed to execute business procedures and
Process cycle test
processes.
Production rule Rules are called production rules because new information is produced when the rule fires.
A project is a unique set of coordinated and controlled activities with start and finish dates undertaken an
Project
objective conforming to specific requirements, including the constraints of time, cost and resources.
Project test plan A test plan that typically addresses multiple test levels.
A strategy in system development in which a scaled down system or portion of a system is constructed in a
Prototyping
short time, tested, and improved in several iterations.
Pseudo-random A series which appears to be random but is in fact generated according to some prearranged sequence.
The totality of functionality and features of a software product that bear on its ability to satisfy stated or
Quality
implied needs.
Quality assurance Part of quality management focused on providing confidence that quality requirements will be fulfilled.
Quality attribute A feature or characteristic that affects an item’s quality.
Coordinated activities to direct and control an organization with regard to quality.
Quality management Direction and control with regard to quality generally includes the establishment of the quality policy and
quality objectives, quality planning, quality control, quality assurance and quality improvement.
Query Generically query means question. Usually it refers to a complex SQL SELECT statement.
Queuing time is incurred when the device, which a program wishes to use, is already busy. The program
Queuing Time
therefore has to wait in a queue to obtain service from that device.
The return on investment (ROI) is usually computed as the benefits derived divided by the investments
ROI made. If we are starting a fresh project, we might compute the value of testing and divide by the cost of
the testing to compute the return.
A black box test design technique where test cases are selected, possibly using a pseudo-random generation
Random testing algorithm, to match an operational profile. This technique can be used for testing non-functional attributes
such as reliability and performance.
Testing that runs test cases that failed the last time they were run, in order to verify the success of
Re-testing
corrective actions.
The capability of the software product to re-establish a specified level of performance and recover the data
Recoverability
directly affected in case of failure.
Regression testing Testing of a previously tested program following modification to ensure that defects have not been
 introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is
performed when the software or its environment is changed.
Conditions such as is equal to or is less than that link an attribute name with an attribute value in a rule's
Relational operator
premise to form logical expressions that can be evaluated as true or false.
A document identifying test items, their configuration, current status and other delivery information
Release note
delivered by development to testing, and possibly other stakeholders, at the start of a test execution phase.
Is the probability that software will not cause the failure of a system for a specified time under specified
Reliability
conditions
Repeatability An attribute of a test indicating whether the same results are produced each time the test is executed.
The capability of the software product to be used in place of another specified software product for the
Replaceability
same purpose in the same environment.
The degree to which a requirement is stated in terms that permit establishment of test designs (and
Requirements
subsequently test cases) and execution of tests to determine whether the requirements have been met.
An approach to testing in which test cases are designed based on test objectives and test conditions
Requirements-based
derived from requirements. e.g. tests that exercise specific functions or probe non-functional attributes
testing
such as reliability or usability.
The capability of the software product to use appropriate amounts and types of resources. For example the
Resource utilization amounts of main and secondary memory used by the program and the sizes of required temporary or
overflow files, when the software performs its function under stated conditions.
The consequence/outcome of the execution of a test. It includes outputs to screens, changes to data,
Result
reports, and communication messages sent out.
Resumption criteria The testing activities that must be repeated when testing is re-started after a suspension.
A detailed check of the test basis to determine whether the test basis is at an adequate quality level to act
Review
as an input document for the test process.
The person involved in the review who shall identify and describe anomalies in the product or project under
Reviewer
review. Reviewers can be chosen to represent different viewpoints and roles in the review process.
Risk A factor that could result in future negative consequences; usually expressed as impact and likelihood.
Systematic application of procedures and practices to the tasks of identifying, analyzing, prioritizing, and
Risk management
controlling risk. See The Risk Management Guide.
The degree to which a component or system can function correctly in the presence of invalid inputs or
Robustness
stressful environmental conditions.
Roll in roll out (RIRO) Used on some systems to describe swapping.
An underlying factor that caused a non-conformance and possibly should be permanently eliminated
Root cause
through process improvement.
A statement of the form: if X then Y else Z. The if part is the rule premise, and the then part is the
Rule consequent. The else component of the consequent is optional. The rule fires when the if part is determined
to be true or false.
The encoded knowledge for an expert system. In a rule-based expert system, a knowledge base typically
Rule Base
incorporates definitions of attributes and rules along with control information.
Safety testing The process of testing to determine the safety of a software product.
Scalability The capability of the software product to be upgraded to accommodate increased loads.
A scheme for the execution of test procedures. The test procedures are included in the test execution
Schedule
schedule in their context and in the order in which they are to be executed.
The person who has to record each defect mentioned and any suggestions for improvement during a review
Scribe meeting, on a logging form. The scribe has to make ensure that the logging form is readable and
understandable.
Script Commonly used to refer to a test procedure specification, especially an automated one.
Attributes of software products that bear on its ability to prevent unauthorized access, whether accidental
Security
or deliberate, to programs and data.
Severity The degree of impact that a defect has on the development or operation of a component or system.
The representation of selected behavioral characteristics of one physical or abstract system by another
Simulation
system. Using a model to mimic a process.
A device, computer program or system used during testing, which behaves or operates like a given system
Simulator
when provided with a set of controlled inputs.
A subset of all defined/planned test cases that cover the main functionality of a component or system, to
Smoke test ascertaining that the most crucial functions of a program work, but not bothering with finer details. A daily
build and smoke test is among industry best practices.
Stability The capability of the software product to avoid unexpected effects from modifications in the software.
A diagram that depicts the states that a component or system can assume, and shows the events or
State diagram
circumstances that cause and/or result from a change from one state to another.
A grid showing the resulting transitions for each state combined with each possible event, showing both
State table
valid and invalid transitions.
A black box test design technique in which test cases are designed to execute valid and invalid state
State transition testing
transitions.
Statement An entity in a programming language, which is typically the smallest indivisible unit of execution.
Statement coverage The percentage of executable statements that have been exercised by a test suite.
Statement testing A white box test design technique in which test cases are designed to execute statements.
Analysis of software artifacts, e.g. requirements or code, carried out without execution of these software
Static analysis
artifacts.
A test design technique in which a model of the statistical distribution of the input is used to construct
Statistical testing
representative test cases.
A high-level document defining the test levels to be performed and the testing within those levels for a
Strategy
programme (one or more projects).
Stress testing Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements.
Structural coverage Coverage measures based on the internal structure of the component.
A skeletal or special-purpose implementation of a software component, used to develop or test a component
Stub
that calls or is otherwise dependent on it. It replaces a called component.
An attribute which becomes a temporary intermediate goal for the inference engine. Subgoal values need to
Subgoal
be determined because they are used in the premise of rules that can determine higher level goals.
The capability of the software product to provide an appropriate set of functions for specified tasks and user
Suitability
objectives
Suspension criteria The criteria used to (temporarily) stop all or a portion of the testing activities on the test items.
Use of symbols, rather than numbers, combined with rules-of-thumb (or heuristics), in order to process
Symbolic Processing
information and solve problems.
A black box test design technique in which test cases are designed based upon the definition of the input
Syntax testing
domain and/or output domain.
System A collection of components organized to accomplish a specific function or set of functions.
System integration Testing the integration of systems and packages; testing interfaces to external organizations (e.g.
testing Electronic Data Interchange, Internet).
A peer group discussion activity that focuses on achieving consensus on the technical approach to be taken.
Technical Review
A technical review is also known as a peer review.
Test Case A specific set of test data along with expected results for a particular test condition
Test Maturity Model A five level staged framework for test process improvement, related to the Capability Maturity Model (CMM)
(TMM) that describes the key elements of an effective test process.
Test Process A continuous framework for test process improvement that describes the key elements of an effective test
Improvement (TPI) process, especially targeted at system testing and acceptance testing.
The implementation of the test strategy for a specific project. It typically includes the decisions made that
Test approach follow based on the (test) project’s goal and the risk assessment carried out, starting points regarding the
test process and the test design techniques to be applied.
The use of software to perform or support test activities, e.g. test management, test design, test execution
and results checking. There are many factors to consider when planning for software test automation.
Test automation Automation changes the complexion of testing and the test organisation from design through
implementation and test execution. There are tangible and intangible elements and widely held myths
about benefits and capabilities of test automation.
A document specifying a set of test cases (objective, inputs, test actions, expected results, and execution
Test case specification
preconditions) for a test item.
A statement of test objectives, and possibly test ideas. Test charters are amongst other used in exploratory
Test charter
testing.
Test comparator A test tool to perform automated test comparison.
The process of identifying differences between the actual results produced by the component or system
Test comparison under test and the expected results for a test. Test comparison can be performed during test execution
(dynamic comparison) or after test execution.
An item or event of a component or system that could be verified by one or more test cases, e.g. a
Test condition
function, transaction, quality attribute, or structural element.
Data that exists (for example, in a database) before a test is executed, and that affects or is affected by the
Test data
component or system under test.
Test data preparation A type of test tool that enables data to be selected from existing databases or created, generated,
tool manipulated and edited for use in testing.
A document specifying the test conditions (coverage items) for a test item, the detailed test approach and
Test design specification
identifying the associated high level test cases.
A tool that support the test design activity by generating test inputs from a specification that may be held
Test design tool in a CASE tool repository.
e.g. requirements management tool, or from specified test conditions held in the tool itself.
An environment containing hardware, instrumentation, simulators, software tools, and other support
Test environment
elements needed to conduct a test.
A document produced at the end of the test process summarizing all testing activities and results. It also
Test evaluation report
contains an evaluation of the test process and lessons learned.
Test execution The process of running a test by the component or system under test, producing actual result(s).
 The period of time in a software development life cycle during which the components of a software product
Test execution phase are executed, and the software product is evaluated to determine whether or not requirements have been
satisfied.
A scheme for the execution of test procedures. The test procedures are included in the test execution
Test execution schedule
schedule in their context and in the order in which they are to be executed.
Test execution technique The method used to perform the actual test execution, either manually or automated.
A type of test tool that is able to execute other software using an automated test script, e.g.
Test execution tool
capture/playback.
Test harness A test environment comprised of stubs and drivers needed to conduct a test.
The organisational artifacts needed to perform testing, consisting of test environments, test tools, office
Test infrastructure
environment and procedures.
Test item The individual element to be tested. There usually is one test object and many test items.
Test level A group of test activities that are organized and managed together. A test level is linked to the
responsibilities in a project.
Examples of test levels are component test, integration test, system test and acceptance test.
Test log A chronological record of relevant details about the execution of tests.
The person responsible for testing and evaluating a test object. The individual, who directs, controls,
Test manager
administers plans and regulates the evaluation of a test object.
Test object The component or system to be tested.
Test point analysis (TPA) A formula based test estimation method based on function point analysis.
Test procedure A document specifying a sequence of actions for the execution of a test. Also known as test script or
specification manual test script.
The fundamental test process comprises planning, specification, execution, recording and checking for
Test process
completion.
Test run Execution of a test on a specific version of the test object.
A document that consists of a test design specification, test case specification and/or test procedure
Test specification
specification.
A high-level document defining the test levels to be performed and the testing within those levels for a
Test strategy
programme (one or more projects).
A set of several test cases for a component or system under test, where the post condition of one test is
Test suite
often used as the precondition for the next one.
Test type A group of test activities aimed at testing a component or system regarding one or more interrelated
quality attributes.
A test type is focused on a specific test objective, i.e. reliability test, usability test, regression test etc., and
may take place on one or more test levels or test phases.
Testability The capability of the software product to enable modified software to be tested.
Tester A technically skilled professional who is involved in the testing of a component or system.
Testing The process of exercising software to verify that it satisfies specified requirements and to detect faults
A version of component integration testing where the progressive integration of components follows the
Thread testing implementation of subsets of the requirements, as opposed to the integration of components by levels of a
hierarchy.
Top-down testing An incremental approach to integration testing where the component at the top of the component hierarchy
is tested first, with lower level components being simulated by stubs.
Tested components are then used to test lower level components.
The ability to identify related items in documentation and software, such as requirements with associated
Traceability
tests. See also horizontal traceability, vertical traceability.
The capability of the software product to enable the user to understand whether the software is suitable, and
Understandability
how it can be used for particular tasks and conditions of use.
Unit testing The testing of individual software components
Unreachable code Code that cannot be reached and therefore is impossible to execute.
This type of decision situation is complex and no standard solutions exist for resolving the situation. Some
Unstructured Decisions
or all of the structural elements of the decision situation are undefined, ill-defined or unknown.
The capability of the software to be understood, learned, used and attractive to the user when used under
Usability
specified conditions.
Use case testing A black box test design technique in which test cases are designed to execute user scenarios.
Formal testing conducted to enable a user, customer or other authorised entity to determine whether to
User acceptance testing
accept a system or component.
User test A test whereby real-life users are involved to evaluate the usability of a component or system.
An evaluative term for a System's user interface. The phrase indicates that users judge the user interface
User-Friendly
as to easy to learn, understand, and use.
V model describes how inspection and testing activities can occur in parallel with other activities
Validation Correctness: Determination of the correctness of the products of software development with respects to the
 user needs and requirements
Completeness: The process of evaluating a system or component to determine whether the products of the
Verification
given development phase satisfy the conditions imposed at the start of that phase
Version Identifier A version number; version date, or version date and time stamp.
Volume Testing Testing where the system is subjected to large volumes of data.
A step-by-step presentation by the author of a document in order to gather information and to establish a
Walkthrough
common understanding of its content.
Waterline The lowest level of detail relevant to the Customer.
The capability of "asking" the software package what the effect will be of changing some of the input data
What If Analysis
or independent variables.
Documented procedure to derive and select test cases based on an analysis of the internal structure of a
White box test design component or system. Documented procedure to derive and select test cases based on an analysis of the
technique internal structure of a component or system. Testing based on an analysis of the internal structure of the
component or system.
An expert based test estimation technique that aims at making an accurate estimation using the collective
Wide Band Delphi
wisdom of the team members.
Method of avoiding an incident or problem, either from a temporary fix or from a technique that means the
Workaround
Customer is not reliant on a particular aspect of a service that is known to have a problem.
Extensible Markup Language. XML is a set of rules for designing text formats that let you structure your
data. XML makes it easy for a computer to generate data, read data, and ensure that the data structure is
XML unambiguous.
XML avoids common pitfalls in language design: it is extensible, platform-independent, and it supports
internationalization and localisation.

SilkTest Interview Question

1. How does the Recovery System Work in SilkTest?

2. What is the purpose of user-defined base state method .?

3. What are the components of SilkTest .?

4. What are the important features of SilkTest as compare to other tools?

5. How to define new class in SilkTest?

6. What is SilkMeter and how does it works with SilkTest .?

Test Director Interview Question

1. Types of vies in Datastage Director?
2. Orchestrate Vs Datastage Parallel Extender?
3. What is an Exception ? What are types of Exception ?
4. What are Routines and where/how are they written and have you written any routines before?
5. What are the command line functions that import and export the DS jobs?
6. How many types of database triggers can be specified on a table ? What are they ?
7. What is NLS in datastage? how we use NLS in Datastage ? what advantages in that ? at the time of
ins. ..
8. What are types of Hashed File?
9. What are the datatypes a available in PL/SQL ?

General Testing Interview Question

1.What is 'Software Quality Assurance'?

Software QA involves the entire software development Process - monitoring and improving the
process, making sure that any agreed-upon standards and procedures are followed, and ensuring that
problems are found and dealt with. It is oriented to 'prevention'. (See the Books section for a list of
useful books on Software Quality Assurance.)

2.What is 'Software Testing'?

Testing involves operation of a system or application under controlled conditions and evaluating the
results (eg, 'if the user is in interface A of the application while using hardware B, and does C, then D
should happen'). The controlled conditions should include both normal and abnormal conditions.
Testing should intentionally attempt to make things go wrong to determine if things happen when they
shouldn't or things don't happen when they should. It is oriented to 'detection'.

Organizations vary considerably in how they assign responsibility for QA and testing. Sometimes
they're the combined responsibility of one group or individual. Also common are project teams that
include a mix of testers and developers who work closely together, with overall QA processes
monitored by project managers. It will depend on what best fits an organization's size and business
structure.

3. What are some recent major computer system failures caused by software bugs?

* Media reports in January of 2005 detailed severe problems with a $170 million high-profile U.S.
government IT systems project. Software testing was one of the five major problem areas according
to a report of the commission reviewing the project. Studies were under way to determine which, if
any, portions of the project could be salvaged.

* In July 2004 newspapers reported that a new government welfare management system in Canada
costing several hundred million dollars was unable to handle a simple benefits rate increase after
being put into live operation. Reportedly the original contract allowed for only 6 weeks of acceptance
testing and the system was never tested for its ability to handle a rate increase.

* Millions of bank accounts were impacted by errors due to installation of inadequately tested software
code in the transaction processing system of a major North American bank, according to mid-2004
news reports. Articles about the incident stated that it took two weeks to fix all the resulting errors,
that additional problems resulted when the incident drew a large number of e-mail phishing attacks
against the bank's customers, and that the total cost of the incident could exceed $100 million.

* A bug in site management software utilized by companies with a significant percentage of worldwide
web traffic was reported in May of 2004. The bug resulted in performance problems for many of the
sites simultaneously and required disabling of the software until the bug was fixed.

* According to news reports in April of 2004, a software bug was determined to be a major contributor
to the 2003 Northeast blackout, the worst power system failure in North American history. The failure
involved loss of electrical power to 50 million customers, forced shutdown of 100 power plants, and
economic losses estimated at $6 billion. The bug was reportedly in one utility company's vendor-
supplied power monitoring and management system, which was unable to correctly handle and report
on an unusual confluence of initially localized events. The error was found and corrected after
examining millions of lines of code.

* In early 2004, news reports revealed the intentional use of a software bug as a counter-espionage
tool. According to the report, in the early 1980's one nation surreptitiously allowed a hostile nation's
espionage service to steal a version of sophisticated industrial software that had intentionally-added
flaws. This eventually resulted in major industrial disruption in the country that used the stolen flawed
software.

* A major U.S. retailer was reportedly hit with a large government fine in October of 2003 due to web
site errors that enabled customers to view one anothers' online orders.

* News stories in the fall of 2003 stated that a manufacturing company recalled all their
transportation products in order to fix a software problem causing instability in certain circumstances.
The company found and reported the bug itself and initiated the recall procedure in which a software
upgrade fixed the problems.

* In January of 2001 newspapers reported that a major European railroad was hit by the aftereffects
of the Y2K bug. The company found that many of their newer trains would not run due to their
inability to recognize the date '31/12/2000'; the trains were started by altering the control system's
date settings.

* News reports in September of 2000 told of a software vendor settling a lawsuit with a large
mortgage lender; the vendor had reportedly delivered an online mortgage processing system that did
not meet specifications, was delivered late, and didn't work.

* In early 2000, major problems were reported with a new computer system in a large suburban U.S.
public school district with 100,000+ students; problems included 10,000 erroneous report cards and
students left stranded by failed class registration systems; the district's CIO was fired. The school
district decided to reinstate it's original 25-year old system for at least a year until the bugs were
worked out of the new system by the software vendors.

* In October of 1999 the $125 million NASA Mars Climate Orbiter spacecraft was believed to be lost in
space due to a simple data conversion error. It was determined that spacecraft software used certain
data in English units that should have been in metric units. Among other tasks, the orbiter was to
serve as a communications relay for the Mars Polar Lander mission, which failed for unknown reasons
in December 1999. Several investigating panels were convened to determine the process failures that
allowed the error to go undetected.

* Bugs in software supporting a large commercial high-speed data network affected 70,000 business
customers over a period of 8 days in August of 1999. Among those affected was the electronic trading
system of the largest U.S. futures exchange, which was shut down for most of a week as a result of
the outages.

* January 1998 news reports told of software problems at a major U.S. telecommunications company
that resulted in no charges for long distance calls for a month for 400,000 customers. The problem
went undetected until customers called up with questions about their bills.

4.Why is it often hard for management to get serious about quality assurance?

* Solving problems is a high-visibility process; preventing problems is low-visibility. This is illustrated

by an old parable: In ancient China there was a family of healers, one of whom was known throughout
the land and employed as a physician to a great lord.

5.Why does software have bugs?

* Miscommunication or no communication - as to specifics of what an application should or shouldn't

do (the application's requirements).

* Software complexity - the complexity of current software applications can be difficult to comprehend
for anyone without experience in modern-day software development. Multi-tiered applications, client-
server and distributed applications, data communications, enormous relational databases, and sheer
size of applications have all contributed to the exponential growth in software/system complexity.

* Programming errors - programmers, like anyone else, can make mistakes.

* Changing requirements (whether documented or undocumented) - the end-user may not

understand the effects of changes, or may understand and request them anyway - redesign,
rescheduling of engineers, effects on other projects, work already completed that may have to be
redone or thrown out, hardware requirements that may be affected, etc. If there are many minor
changes or any major changes, known and unknown dependencies among parts of the project are
likely to interact and cause problems, and the complexity of coordinating changes may result in
errors. Enthusiasm of engineering staff may be affected. In some fast-changing business
environments, continuously modified requirements may be a fact of life. In this case, management
must understand the resulting risks, and QA and test engineers must adapt and plan for continuous
extensive testing to keep the inevitable bugs from running out of control - see 'What can be done if
requirements are changing continuously?' in Part 2 of the FAQ. Also see information about 'agile'
approaches such as XP, also in Part 2 of the FAQ.

* Time pressures - scheduling of software projects is difficult at best, often requiring a lot of
guesswork. When deadlines loom and the crunch comes, mistakes will be made.

* egos - people prefer to say things like:

* * 'no problem'

* * 'piece of cake'

* * 'I can whip that out in a few hours'

* * 'it should be easy to update that old code'

* instead of:

* * 'that adds a lot of complexity and we could end up making a lot of mistakes'

* * 'we have no idea if we can do that; we'll wing it'

* * 'I can't estimate how long it will take, until I take a close look at it'

* * 'we can't figure out what that old spaghetti code did in the first place'

If there are too many unrealistic 'no problem's', the result is bugs.

* Poorly documented code - it's tough to maintain and modify code that is badly written or poorly
documented; the result is bugs. In many organizations management provides no incentive for
programmers to document their code or write clear, understandable, maintainable code. In fact, it's
usually the opposite: they get points mostly for quickly turning out code, and there's job security if
nobody else can understand it ('if it was hard to write, it should be hard to read').

* Software development tools - visual tools, class libraries, compilers, scripting tools, etc. often
introduce their own bugs or are poorly documented, resulting in added bugs.

6.How can new Software QA processes be introduced in an existing organization?

* A lot depends on the size of the organization and the risks involved. For large organizations with
high-risk (in terms of lives or property) projects, serious management buy-in is required and a
formalized QA process is necessary.

* Where the risk is lower, management and organizational buy-in and QA implementation may be a
slower, step-at-a-time process. QA processes should be balanced with productivity so as to keep
bureaucracy from getting out of hand.

* For small groups or projects, a more ad-hoc process may be appropriate, depending on the type of
customers and projects. A lot will depend on team leads or managers, feedback to developers, and
ensuring adequate communications among customers, managers, developers, and testers.

* The most value for effort will often be in (a) requirements management processes, with a goal of
clear, complete, testable requirement specifications embodied in requirements or design
documentation, or in 'agile'-type environments extensive continuous coordination with end-users, (b)
design inspections and code inspections, and (c) post-mortems/retrospectives.

7.What is verification? validation?

* Verification typically involves reviews and meetings to evaluate documents, plans, code,
requirements, and specifications. This can be done with checklists, issues lists, walkthroughs, and
inspection meetings. Validation typically involves actual testing and takes place after verifications are
completed. The term 'IV & V' refers to Independent Verification and Validation.

8.What is a 'walkthrough'?

* A 'walkthrough' is an informal meeting for evaluation or informational purposes. Little or no

preparation is usually required.

9.What's an 'inspection'?

* An inspection is more formalized than a 'walkthrough', typically with 3-8 people including a
moderator, reader, and a recorder to take notes. The subject of the inspection is typically a document
such as a requirements spec or a test plan, and the purpose is to find problems and see what's
missing, not to fix anything. Attendees should prepare for this type of meeting by reading thru the
document; most problems will be found during this preparation. The result of the inspection meeting
should be a written report.

10.What kinds of testing should be considered?

* Black box testing - not based on any knowledge of internal design or code. Tests are based on
requirements and functionality.

* White box testing - based on knowledge of the internal logic of an application's code. Tests are
based on coverage of code statements, branches, paths, conditions.

* Unit testing - the most 'micro' scale of testing; to test particular functions or code modules. Typically
done by the programmer and not by testers, as it requires detailed knowledge of the internal program
design and code. Not always easily done unless the application has a well-designed architecture with
tight code; may require developing test driver modules or test harnesses.

* Incremental integration testing - continuous testing of an application as new functionality is added;

requires that various aspects of an application's functionality be independent enough to work
separately before all parts of the program are completed, or that test drivers be developed as needed;
done by programmers or by testers.

* Integration testing - testing of combined parts of an application to determine if they function

together correctly. The 'parts' can be code modules, individual applications, client and server
applications on a network, etc. This type of testing is especially relevant to client/server and
distributed systems.

* Functional testing - black-box type testing geared to functional requirements of an application; this
type of testing should be done by testers. This doesn't mean that the programmers shouldn't check
that their code works before releasing it (which of course applies to any stage of testing.)

* System testing - black-box type testing that is based on overall requirements specifications; covers
all combined parts of a system.

* End-to-end testing - similar to system testing; the 'macro' end of the test scale; involves testing of
a complete application environment in a situation that mimics real-world use, such as interacting with
a database, using network communications, or interacting with other hardware, applications, or
systems if appropriate.

* Sanity testing or smoke testing - typically an initial testing effort to determine if a new software
version is performing well enough to accept it for a major testing effort. For example, if the new
software is crashing systems every 5 minutes, bogging down systems to a crawl, or corrupting
databases, the software may not be in a 'sane' enough condition to warrant further testing in its
current state.

* Regression testing - re-testing after fixes or modifications of the software or its environment. It can
be difficult to determine how much re-testing is needed, especially near the end of the development
cycle. Automated testing tools can be especially useful for this type of testing.

* Acceptance testing - final testing based on specifications of the end-user or customer, or based on
use by end-users/customers over some limited period of time.

* Load testing - testing an application under heavy loads, such as testing of a web site under a range
of loads to determine at what point the system's response time degrades or fails.

* Stress testing - term often used interchangeably with 'load' and 'performance' testing. Also used to
describe such tests as system functional testing while under unusually heavy loads, heavy repetition
of certain actions or inputs, input of large numerical values, large complex queries to a database
system, etc.

* Performance testing - term often used interchangeably with 'stress' and 'load' testing. Ideally
'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA
or Test Plans.

* Usability testing - testing for 'user-friendliness'. Clearly this is subjective, and will depend on the
targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other
techniques can be used. Programmers and testers are usually not appropriate as usability testers.

* Install/uninstall testing - testing of full, partial, or upgrade install/uninstall processes.

* Recovery testing - testing how well a system recovers from crashes, hardware failures, or other
catastrophic problems.

* Failover testing - typically used interchangeably with 'recovery testing'

* Security testing - testing how well the system protects against unauthorized internal or external
access, willful damage, etc; may require sophisticated testing techniques.

* Compatability testing - testing how well software performs in a particular

hardware/software/operating system/network/etc. environment.

* Exploratory testing - often taken to mean a creative, informal software test that is not based on
formal test plans or test cases; testers may be learning the software as they test it.

* Ad-hoc testing - similar to exploratory testing, but often taken to mean that the testers have
significant understanding of the software before testing it.

* Context-driven testing - testing driven by an understanding of the environment, culture, and

intended use of software. For example, the testing approach for life-critical medical equipment
software would be completely different than that for a low-cost computer game.

* User acceptance testing - determining if software is satisfactory to an end-user or customer.

* Comparison testing - comparing software weaknesses and strengths to competing products.

* Alpha testing - testing of an application when development is nearing completion; minor design
changes may still be made as a result of such testing. Typically done by end-users or others, not by
programmers or testers.
* Beta testing - testing when development and testing are essentially completed and final bugs and
problems need to be found before final release. Typically done by end-users or others, not by
programmers or testers.

* Mutation testing - a method for determining if a set of test data or test cases is useful, by
deliberately introducing various code changes ('bugs') and retesting with the original test data/cases
to determine if the 'bugs' are detected. Proper implementation requires large computational
resources.

11.What are 5 common problems in the software development process?

* Solid requirements - clear, complete, detailed, cohesive, attainable, testable requirements that are
agreed to by all players. Use prototypes to help nail down requirements. In 'agile'-type environments,
continuous coordination with customers/end-users is necessary.

* Realistic schedules - allow adequate time for planning, design, testing, bug fixing, re-testing,
changes, and documentation; personnel should be able to complete the project without burning out.

* Adequate testing - start testing early on, re-test after fixes or changes, plan for adequate time for
testing and bug-fixing. 'Early' testing ideally includes unit testing by developers and built-in testing and
diagnostic capabilities.

* Stick to initial requirements as much as possible - be prepared to defend against excessive changes
and additions once development has begun, and be prepared to explain consequences. If changes are
necessary, they should be adequately reflected in related schedule changes. If possible, work closely
with customers/end-users to manage expectations. This will provide them a higher comfort level with
their requirements decisions and minimize excessive changes later on.

* Communication - require walkthroughs and inspections when appropriate; make extensive use of
group communication tools - e-mail, groupware, networked bug-tracking tools and change
management tools, intranet capabilities, etc.; insure that information/documentation is available and
up-to-date - preferably electronic, not paper; promote teamwork and cooperation; use protoypes if
possible to clarify customers' expectations.

12.What is software 'quality'?

* Quality software is reasonably bug-free, delivered on time and within budget, meets requirements
and/or expectations, and is maintainable. However, quality is obviously a subjective term. It will
depend on who the 'customer' is and their overall influence in the scheme of things. A wide-angle view
of the 'customers' of a software development project might include end-users, customer acceptance
testers, customer contract officers, customer management, the development organization's.

* Management/accountants/testers/salespeople, future software maintenance engineers,

stockholders, magazine columnists, etc. Each type of 'customer' will have their own slant on 'quality' -
the accounting department might define quality in terms of profits while an end-user might define
quality as user-friendly and bug-free.

13.What is 'good code'?

* * 'Good code' is code that works, is bug free, and is readable and maintainable. Some organizations
have coding 'standards' that all developers are supposed to adhere to, but everyone has different
ideas about what's best, or what is too many or too few rules. There are also various theories and
metrics, such as McCabe Complexity metrics. It should be kept in mind that excessive use of
standards and rules can stifle productivity and creativity. 'Peer reviews', 'buddy checks' code analysis
tools, etc. can be used to check for problems and enforce standards. For C and C++ coding, here are
some typical ideas to consider in setting rules/standards; these may or may not apply to a particular
situation:

* Minimize or eliminate use of global variables.

* Use descriptive function and method names - use both upper and lower case, avoid abbreviations,
use as many characters as necessary to be adequately descriptive (use of more than 20 characters is
not out of line); be consistent in naming conventions.

* Use descriptive variable names - use both upper and lower case, avoid abbreviations, use as many
characters as necessary to be adequately descriptive (use of more than 20 characters is not out of
line); be consistent in naming conventions.

* Function and method sizes should be minimized; less than 100 lines of code is good, less than 50
lines is preferable.
* Function descriptions should be clearly spelled out in comments preceding a function's code.

* Organize code for readability.

* Use whitespace generously - vertically and horizontally.

* Each line of code should contain 70 characters max.

* One code statement per line.

* Coding style should be consistent throught a program (eg, use of brackets, indentations, naming
conventions, etc.)

* In adding comments, err on the side of too many rather than too few comments; a common rule of
thumb is that there should be at least as many lines of comments (including header blocks) as lines of
code.

* No matter how small, an application should include documentaion of the overall program function
and flow (even a few paragraphs is better than nothing); or if possible a separate flow chart and
detailed program documentation.

* Make extensive use of error handling procedures and status and error logging.

* For C++, to minimize complexity and increase maintainability, avoid too many levels of inheritance
in class heirarchies (relative to the size and complexity of the application). Minimize use of multiple
inheritance, and minimize use of operator overloading (note that the Java programming language
eliminates multiple inheritance and operator overloading.)

* For C++, keep class methods small, less than 50 lines of code per method is preferable.

* For C++, make liberal use of exception handlers.

14.What is 'good design'?

* * 'Design' could refer to many things, but often refers to 'functional design' or 'internal design'.
Good internal design is indicated by software code whose overall structure is clear, understandable,
easily modifiable, and maintainable; is robust with sufficient error-handling and status logging
capability; and works correctly when implemented. Good functional design is indicated by an
application whose functionality can be traced back to customer and end-user requirements.For
programs that have a user interface, it's often a good idea to assume that the end user will have little
computer knowledge and may not read a user manual or even the on-line help; some common rules-
of-thumb include:

* The program should act in a way that least surprises the user

* It should always be evident to the user what can be done next and how to exit

* The program shouldn't let the users do something stupid without warning them.

15.What is SEI? CMM? CMMI? ISO? IEEE? ANSI? Will it help?

* SEI = 'Software Engineering Institute' at Carnegie-Mellon University; initiated by the U.S. Defense
Department to help improve software development processes.

* CMM = 'Capability Maturity Model', now called the CMMI ('Capability Maturity Model Integration'),
developed by the SEI. It's a model of 5 levels of process 'maturity' that determine effectiveness in
delivering quality software. It is geared to large organizations such as large U.S. Defense Department
contractors. However, many of the QA processes involved are appropriate to any organization, and if
reasonably applied can be helpful. Organizations can receive CMMI ratings by undergoing assessments
by qualified auditors.

* Level 1 - characterized by chaos, periodic panics, and heroic efforts required by individuals to
successfully complete projects. Few if any processes in place; successes may not be repeatable.

* Level 2 - software project tracking, requirements management, realistic planning, and configuration
management processes are in place; successful practices can be repeated.

* Level 3 - standard software development and maintenance processes are integrated throughout an
organization; a Software Engineering Process Group is is in place to oversee software processes, and
training programs are used to ensure understanding and compliance.
* Level 4 - metrics are used to track productivity, processes, and products. Project performance is
predictable, and quality is consistently high.

* Level 5 - the focus is on continouous process improvement. The impact of new processes and
technologies can be predicted and effectively implemented when required.

* Perspective on CMM ratings: During 1997-2001, 1018 organizations were assessed. Of those, 27%
were rated at Level 1, 39% at 2, 23% at 3, 6% at 4, and 5% at 5. (For ratings during the period
1992-96, 62% were at Level 1, 23% at 2, 13% at 3, 2% at 4, and 0.4% at 5.) The median size of
organizations was 100 software engineering/maintenance personnel; 32% of organizations were U.S.
federal contractors or agencies. For those rated at Level 1, the most problematical key process area
was in Software Quality Assurance.

* ISO = 'International Organisation for Standardization' - The ISO 9001:2000 standard (which
replaces the previous standard of 1994) concerns quality systems that are assessed by outside
auditors, and it applies to many kinds of production and manufacturing organizations, not just
software. It covers documentation, design, development, production, testing, installation, servicing,
and other processes. The full set of standards consists of: (a)Q9001-2000 - Quality Management
Systems: Requirements; (b)Q9000-2000 - Quality Management Systems: Fundamentals and
Vocabulary; (c)Q9004-2000 - Quality Management Systems: Guidelines for Performance
Improvements. To be ISO 9001 certified, a third-party auditor assesses an organization, and
certification is typically good for about 3 years, after which a complete reassessment is required. Note
that ISO certification does not necessarily indicate quality products - it indicates only that documented
processes are followed. Also see http://www.iso.ch/ for the latest information. In the U.S. the
standards can be purchased via the ASQ web site at http://e-standards.asq.org/

* IEEE = 'Institute of Electrical and Electronics Engineers' - among other things, creates standards
such as 'IEEE Standard for Software Test Documentation' (IEEE/ANSI Standard 829), 'IEEE Standard
of Software Unit Testing (IEEE/ANSI Standard 1008), 'IEEE Standard for Software Quality Assurance
Plans' (IEEE/ANSI Standard 730), and others.

* ANSI = 'American National Standards Institute', the primary industrial standards body in the U.S.;
publishes some software-related standards in conjunction with the IEEE and ASQ (American Society
for Quality).

* Other software development/IT management process assessment methods besides CMMI and ISO
9000 include SPICE, Trillium, TickIT, Bootstrap, ITIL, MOF, and CobiT.

16.What is the 'software life cycle'?

* The life cycle begins when an application is first conceived and ends when it is no longer in use. It
includes aspects such as initial concept, requirements analysis, functional design, internal design,
documentation planning, test planning, coding, document preparation, integration, testing,
maintenance, updates, retesting, phase-out, and other aspects.

17.Will automated testing tools make testing easier?

* Possibly For small projects, the time needed to learn and implement them may not be worth it. For
larger projects, or on-going long-term projects they can be valuable.

* A common type of automated tool is the 'record/playback' type. For example, a tester could click
through all combinations of menu choices, dialog box choices, buttons, etc. in an application GUI and
have them 'recorded' and the results logged by a tool. The 'recording' is typically in the form of text
based on a scripting language that is interpretable by the testing tool. If new buttons are added, or
some underlying code in the application is changed, etc. the application might then be retested by just
'playing back' the 'recorded' actions, and comparing the logging results to check effects of the
changes. The problem with such tools is that if there are continual changes to the system being
tested, the 'recordings' may have to be changed so much that it becomes very time-consuming to
continuously update the scripts. Additionally, interpretation and analysis of results (screens, data,
logs, etc.) can be a difficult task. Note that there are record/playback tools for text-based interfaces
also, and for all types of platforms.

* Another common type of approach for automation of functional testing is 'data-driven' or 'keyword-
driven' automated testing, in which the test drivers are separated from the data and/or actions
utilized in testing (an 'action' would be something like 'enter a value in a text box'). Test drivers can
be in the form of automated test tools or custom-written testing software. The data and actions can
be more easily maintained - such as via a spreadsheet - since they are separate from the test drivers.
The test drivers 'read' the data/action information to perform specified tests. This approach can
enable more efficient control, development, documentation, and maintenance of automated tests/test
cases.
* Other automated tools can include:

* Code analyzers - monitor code complexity, adherence to standards, etc.

* Coverage analyzers - these tools check which parts of the code have been exercised by a test, and
may be oriented to code statement coverage, condition coverage, path coverage, etc.

* Memory analyzers - such as bounds-checkers and leak detectors.

* Load/performance test tools - for testing client/server and web applications under various load
levels.

* Web test tools - to check that links are valid, HTML code usage is correct, client-side and server-side
programs work, a web site's interactions are secure.

* Other tools - for test case management, documentation management, bug reporting, and
configuration management.

WinRunner Interview Questions

1) How you used WinRunner in your project?

Ans. Yes, I have been WinRunner for creating automates scripts for GUI, functional and regression
testing of the AUT.

2) Explain WinRunner testing process?

Ans. WinRunner testing process involves six main stages

i. Create GUI Map File so that WinRunner can recognize the GUI objects in the application being tested
ii. Create test scripts by recording, programming, or a combination of both. While recording tests,
insert checkpoints where you want to check the response of the application being tested.
iii. Debug Test: run tests in Debug mode to make sure they run smoothly
iv. Run Tests: run tests in Verify mode to test your application.
v. View Results: determines the success or failure of the tests.
vi. Report Defects: If a test run fails due to a defect in the application being tested, you can report
information about the defect directly from the Test Results window.

3) What in contained in the GUI map?

Ans. WinRunner stores information it learns about a window or object in a GUI Map. When WinRunner
runs a test, it uses the GUI map to locate objects. It reads an object’s description in the GUI map and
then looks for an object with the same properties in the application being tested. Each of these
objects in the GUI Map file will be having a logical name and a physical description.

There are 2 types of GUI Map files.

i. Global GUI Map file: a single GUI Map file for the entire application
ii. GUI Map File per Test: WinRunner automatically creates a GUI Map file for each test created.

4) How does WinRunner recognize objects on the application?

Ans. WinRunner uses the GUI Map file to recognize objects on the application. When WinRunner runs
a test, it uses the GUI map to locate objects. It reads an object’s description in the GUI map and then
looks for an object with the same properties in the application being tested.

5) Have you created test scripts and what is contained in the test scripts?

Ans. Yes I have created test scripts. It contains the statement in Mercury Interactive’s Test Script
Language (TSL). These statements appear as a test script in a test window. You can then enhance
your recorded test script, either by typing in additional TSL functions and programming elements or
by using WinRunner’s visual programming tool, the Function Generator.

6) How does WinRunner evaluates test results?

Ans. Following each test run, WinRunner displays the results in a report. The report details all the
major events that occurred during the run, such as checkpoints, error messages, system messages, or
user messages. If mismatches are detected at checkpoints during the test run, you can view the
expected results and the actual results from the Test Results window.
7) Have you performed debugging of the scripts?

Ans. Yes, I have performed debugging of scripts. We can debug the script by executing the script in
the debug mode. We can also debug script using the Step, Step Into, Step out functionalities provided
by the WinRunner.

8) How do you run your test scripts?

Ans. We run tests in Verify mode to test your application. Each time WinRunner encounters a
checkpoint in the test script, it compares the current data of the application being tested to the
expected data captured earlier. If any mismatches are found, WinRunner captures them as actual
results.

9) How do you analyze results and report the defects?

Ans. Following each test run, WinRunner displays the results in a report. The report details all the
major events that occurred during the run, such as checkpoints, error messages, system messages,
or user messages. If mismatches are detected at checkpoints during the test run, you can view the
expected results and the actual results from the Test Results window. If a test run fails due to a defect
in the application being tested, you can report information about the defect directly from the Test
Results window. This information is sent via e-mail to the quality assurance manager, who tracks the
defect until it is fixed.

10) What is the use of Test Director software?

Ans. TestDirector is Mercury Interactive’s software test management tool. It helps quality assurance
personnel plan and organize the testing process. With TestDirector you can create a database of
manual and automated tests, build test cycles, run tests, and report and track defects. You can also
create reports and graphs to help review the progress of planning tests, running tests, and tracking
defects before a software release.

11) How you integrated your automated scripts from TestDirector?

Ans When you work with WinRunner, you can choose to save your tests directly to your TestDirector
database or while creating a test case in the TestDirector we can specify whether the script in
automated or manual. And if it is automated script then TestDirector will build a skeleton for the script
that can be later modified into one which could be used to test the AUT.

12) What are the different modes of recording?

Ans. There are two type of recording in WinRunner.

i. Context Sensitive recording records the operations you perform on your application by identifying
Graphical User Interface (GUI) objects.
ii. Analog recording records keyboard input, mouse clicks, and the precise x- and y-coordinates
traveled by the mouse pointer across the screen.

13) What is the purpose of loading WinRunner Add-Ins?

Ans. Add-Ins are used in WinRunner to load functions specific to the particular add-in to the memory.
While creating a script only those functions in the add-in selected will be listed in the function
generator and while executing the script only those functions in the loaded add-in will be executed
else WinRunner will give an error message saying it does not recognize the function.

14) What are the reasons that WinRunner fails to identify an object on the GUI?

Ans. WinRunner fails to identify an object in a GUI due to various reasons.

i. The object is not a standard windows object.
ii. If the browser used is not compatible with the WinRunner version, GUI Map Editor will not be able
to learn any of the objects displayed in the browser window.

15) What do you mean by the logical name of the object.

Ans. An object’s logical name is determined by its class. In most cases, the logical name is the label
that appears on an object.
16) If the object does not have a name then what will be the logical name?

Ans. If the object does not have a name then the logical name could be the attached text.

17) What is the different between GUI map and GUI map files?

Ans. The GUI map is actually the sum of one or more GUI map files. There are two modes for
organizing GUI map files.

i. Global GUI Map file: a single GUI Map file for the entire application

ii. GUI Map File per Test: WinRunner automatically creates a GUI Map file for each test created. GUI
Map file is a file which contains the windows and the objects learned by the WinRunner with its logical
name and their physical description.

18) How do you view the contents of the GUI map?

Ans. GUI Map editor displays the content of a GUI Map. We can invoke GUI Map Editor from the Tools
Menu in WinRunner. The GUI Map Editor displays the various GUI Map files created and the windows
and objects learned in to them with their logical name and physical description.

19) When you create GUI map do you record all the objects of specific objects?

Ans. If we are learning a window then WinRunner automatically learns all the objects in the window
else we will we identifying those object, which are to be learned in a window, since we will be working
with only those objects while creating scripts.

20) What is the purpose of set_window command?

Ans. Set_Window command sets the focus to the specified window. We use this command to set the
focus to the required window before executing tests on a particular window.

Syntax: set_window(, time); The logical name is the logical name of the window and time is the time
the execution has to wait till it gets the given window into focus.

21) How do you load GUI map?

Ans. We can load a GUI Map by using the GUI_load command.

Syntax: GUI_load();

22) What is the disadvantage of loading the GUI maps through start up scripts?

Ans.1.If we are using a single GUI Map file for the entire AUT then the memory used by the GUI Map
may be much high.

2.If there is any change in the object being learned then WinRunner will not be able to recognize the
object, as it is not in the GUI Map file loaded in the memory. So we will have to learn the object again
and update the GUI File and reload it.

23) How do you unload the GUI map?

Ans. We can use GUI_close to unload a specific GUI Map file or else we call use GUI_close_all
command to unload all the GUI Map files loaded in the memory.

Syntax: GUI_close(); or GUI_close_all;

24) What actually happens when you load GUI map?

Ans. When we load a GUI Map file, the information about the windows and the objects with their
logical names and physical description are loaded into memory. So when the WinRunner executes a
script on a particular window, it can identify the objects using this information loaded in the memory.

25) What is the purpose of the temp GUI map file?

Ans. While recording a script, WinRunner learns objects and windows by itself. This is actually stored
into the temporary GUI Map file. We can specify whether we have to load this temporary GUI Map file
should be loaded each time in the General Options.

26) What is the extension of gui map file?

Ans. The extension for a GUI Map file is “.gui”.

27) How do you find an object in an GUI map.

Ans. The GUI Map Editor is been provided with a Find and Show Buttons.

i. To find a particular object in the GUI Map file in the application, select the object and click the Show
window. This blinks the selected object.

ii. To find a particular object in a GUI Map file click the Find button, which gives the option to select
the object. When the object is selected, if the object has been learned to the GUI Map file it will be
focused in the GUI Map file.

28) What different actions are performed by find and show button?

Ans. 1.To find a particular object in the GUI Map file in the application, select the object and click the
Show window. This blinks the selected object.

2.To find a particular object in a GUI Map file click the Find button, which gives the option to select the
object. When the object is selected, if the object has been learned to the GUI Map file it will be
focused in the GUI Map file.

29) How do you identify which files are loaded in the GUI map?

Ans. The GUI Map Editor has a drop down “GUI File” displaying all the GUI Map files loaded into the
memory.

30) How do you modify the logical name or the physical description of the objects in GUI map?

Ans. You can modify the logical name or the physical description of an object in a GUI map file using
the GUI Map Editor.
31) When do you feel you need to modify the logical name?

Ans. Changing the logical name of an object is useful when the assigned logical name is not
sufficiently descriptive or is too long.

32) When it is appropriate to change physical description?

Ans. Changing the physical description is necessary when the property value of an object changes.

33) How WinRunner handles varying window labels?

Ans. We can handle varying window labels using regular expressions. WinRunner uses two “hidden”
properties in order to use regular expression in an object’s physical description. These properties are
regexp_label and regexp_MSW_class.

i. The regexp_label property is used for windows only. It operates “behind the scenes” to insert a
regular expression into a window’s label description.

ii. The regexp_MSW_class property inserts a regular expression into an object’s MSW_class. It is
obligatory for all types of windows and for the object class object.

34) What is the purpose of regexp_label property and regexp_MSW_class property?

Ans. The regexp_label property is used for windows only. It operates “behind the scenes” to insert a
regular expression into a window’s label description.

The regexp_MSW_class property inserts a regular expression into an object’s MSW_class. It is

obligatory for all types of windows and for the object class object.

35) How do you suppress a regular expression?

Ans. We can suppress the regular expression of a window by replacing the regexp_label property with
label property.

36) How do you copy and move objects between different GUI map files?

Ans. We can copy and move objects between different GUI Map files using the GUI Map Editor. The
steps to be followed are:

i. Choose Tools > GUI Map Editor to open the GUI Map Editor.
ii. Choose View > GUI Files.
iii. Click Expand in the GUI Map Editor. The dialog box expands to display two GUI map files
simultaneously.
iv. View a different GUI map file on each side of the dialog box by clicking the file names in the GUI
File lists.
v. In one file, select the objects you want to copy or move. Use the Shift key and/or Control key to
select multiple objects. To select all objects in a GUI map file, choose Edit > Select All.
vi. Click Copy or Move.
vii. To restore the GUI Map Editor to its original size, click Collapse.

37) How do you select multiple objects during merging the files?

Ans. Use the Shift key and/or Control key to select multiple objects. To select all objects in a GUI map
file, choose Edit > Select All.

38) How do you clear a GUI map files?

Ans. We can clear a GUI Map file using the “Clear All” option in the GUI Map Editor.

39) How do you filter the objects in the GUI map?

Ans. GUI Map Editor has a Filter option. This provides for filtering with 3 different types of options.
i. Logical name displays only objects with the specified logical name.
ii. Physical description displays only objects matching the specified physical description. Use any
substring belonging to the physical description.
iii. Class displays only objects of the specified class, such as all the push buttons.

40) How do you configure GUI map?

a. When WinRunner learns the description of a GUI object, it does not learn all its properties. Instead,
it learns the minimum number of properties to provide a unique identification of the object.

1. What is load testing? Load testing is to test that if the application works fine with the loads
that result from large number of simultaneous users, transactions and to determine weather
it can handle peak usage periods.
 2. What is Performance testing? - Timing for both read and update transactions should be
gathered to determine whether system functions are being performed in an acceptable
timeframe. This should be done standalone and then in a multi user environment to
determine the effect of multiple transactions on the timing of a single transaction.
3. Did u use LoadRunner? What version? Yes. Version 7.2.
4. Explain the Load testing process? -
Step 1: Planning the test. Here, we develop a clearly defined test plan to ensure the test
scenarios we develop will accomplish load-testing objectives. Step 2: Creating Vusers. Here,
we create Vuser scripts that contain tasks performed by each Vuser, tasks performed by
Vusers as a whole, and tasks measured as transactions. Step 3: Creating the scenario. A
scenario describes the events that occur during a testing session. It includes a list of
machines, scripts, and Vusers that run during the scenario. We create scenarios using
LoadRunner Controller. We can create manual scenarios as well as goal-oriented scenarios. In
manual scenarios, we define the number of Vusers, the load generator machines, and
percentage of Vusers to be assigned to each script. For web tests, we may create a goal-
oriented scenario where we define the goal that our test has to achieve. LoadRunner
automatically builds a scenario for us. Step 4: Running the scenario.
We emulate load on the server by instructing multiple Vusers to perform tasks
simultaneously. Before the testing, we set the scenario configuration and scheduling. We can
run the entire scenario, Vuser groups, or individual Vusers. Step 5: Monitoring the scenario.
We monitor scenario execution using the LoadRunner online runtime, transaction, system
resource, Web resource, Web server resource, Web application server resource, database
server resource, network delay, streaming media resource, firewall server resource, ERP
server resource, and Java performance monitors. Step 6: Analyzing test results. During
scenario execution, LoadRunner records the performance of the application under different
loads. We use LoadRunner’s graphs and reports to analyze the application’s performance.
5. When do you do load and performance Testing? - We perform load testing once we are
done with interface (GUI) testing. Modern system architectures are large and complex.
Whereas single user testing primarily on functionality and user interface of a system
component, application testing focuses on performance and reliability of an entire system.
For example, a typical application-testing scenario might depict 1000 users logging in
simultaneously to a system. This gives rise to issues such as what is the response time of the
system, does it crash, will it go with different software applications and platforms, can it hold
so many hundreds and thousands of users, etc. This is when we set do load and performance
testing.
6. What are the components of LoadRunner? - The components of LoadRunner are The Virtual
User Generator, Controller, and the Agent process, LoadRunner Analysis and Monitoring,
LoadRunner Books Online.
7. What Component of LoadRunner would you use to record a Script? - The Virtual User
Generator (VuGen) component is used to record a script. It enables you to develop Vuser
scripts for a variety of application types and communication protocols.
8. What Component of LoadRunner would you use to play Back the script in multi user
mode? - The Controller component is used to playback the script in multi-user mode. This is
done during a scenario run where a vuser script is executed by a number of vusers in a
group.
9. What is a rendezvous point? - You insert rendezvous points into Vuser scripts to emulate
heavy user load on the server. Rendezvous points instruct Vusers to wait during test
execution for multiple Vusers to arrive at a certain point, in order that they may
simultaneously perform a task. For example, to emulate peak load on the bank server, you
can insert a rendezvous point instructing 100 Vusers to deposit cash into their accounts at
the same time.
10. What is a scenario? - A scenario defines the events that occur during each testing session.
For example, a scenario defines and controls the number of users to emulate, the actions to
be performed, and the machines on which the virtual users run their emulations.

b. Many applications also contain custom GUI objects. A custom object is any object not belonging to
one of the standard classes used by WinRunner. These objects are therefore assigned to the generic
“object” class. When WinRunner records an operation on a custom object, it generates obj_mouse_
statements in the test script.

c. If a custom object is similar to a standard object, you can map it to one of the standard classes.
You can also configure the properties WinRunner uses to identify a custom object during Context
Sensitive testing. The mapping and the configuration you set are valid only for the current WinRunner
session. To make the mapping and the configuration permanent, you must add configuration
statements to your startup test script.

11. Explain the recording mode for web Vuser script? - We use VuGen to develop a Vuser script
by recording a user performing typical business processes on a client application. VuGen
creates the script by recording the activity between the client and the server. For example, in
web based applications, VuGen monitors the client end of the database and traces all the
requests sent to, and received from, the database server. We use VuGen to: Monitor the
communication between the application and the server; Generate the required function calls;
and Insert the generated function calls into a Vuser script.
12. Why do you create parameters? - Parameters are like script variables. They are used to vary
input to the server and to emulate real users. Different sets of data are sent to the server
each time the script is run. Better simulate the usage model for more accurate testing from
the Controller; one script can emulate many different users on the system.
13. What is correlation? Explain the difference between automatic correlation and manual
correlation? - Correlation is used to obtain data which are unique for each run of the script
and which are generated by nested queries. Correlation provides the value to avoid errors
arising out of duplicate values and also optimizing the code (to avoid nested queries).
Automatic correlation is where we set some rules for correlation. It can be application server
specific. Here values are replaced by data which are created by these rules. In manual
correlation, the value we want to correlate is scanned and create correlation is used to
correlate.
14. How do you find out where correlation is required? Give few examples from your
projects? - Two ways: First we can scan for correlations, and see the list of values which can
be correlated. From this we can pick a value to be correlated. Secondly, we can record two
scripts and compare them. We can look up the difference file to see for the values which
needed to be correlated. In my project, there was a unique id developed for each customer,
it was nothing but Insurance Number, it was generated automatically and it was sequential
and this value was unique. I had to correlate this value, in order to avoid errors while running
my script. I did using scan for correlation.
15. Where do you set automatic correlation options? - Automatic correlation from web point of
view can be set in recording options and correlation tab. Here we can enable correlation for
the entire script and choose either issue online messages or offline actions, where we can
define rules for that correlation. Automatic correlation for database can be done using show
output window and scan for correlation and picking the correlate query tab and choose which
query value we want to correlate. If we know the specific value to be correlated, we just do
create correlation for the value and specify how the value to be created.
16. What is a function to capture dynamic values in the web Vuser script? -
Web_reg_save_param function saves dynamic data information to a parameter.
17. When do you disable log in Virtual User Generator, When do you choose standard and
extended logs? - Once we debug our script and verify that it is functional, we can enable
logging for errors only. When we add a script to a scenario, logging is automatically disabled.
Standard Log Option: When you select
Standard log, it creates a standard log of functions and messages sent during script
execution to use for debugging. Disable this option for large load testing scenarios. When you
copy a script to a scenario, logging is automatically disabled Extended Log Option: Select
extended log to create an extended log, including warnings and other messages. Disable this
option for large load testing scenarios. When you copy a script to a scenario, logging is
automatically disabled. We can specify which additional information should be added to the
extended log using the Extended log options.
18. How do you debug a LoadRunner script? - VuGen contains two options to help debug Vuser
scripts-the Run Step by Step command and breakpoints. The Debug settings in the Options
dialog box allow us to determine the extent of the trace to be performed during scenario
execution. The debug information is written to the Output window. We can manually set the
message class within your script using the lr_set_debug_message function. This is useful if
we want to receive debug information about a small section of the script only.
19. How do you write user defined functions in LR? Give me few functions you wrote in your
previous project? - Before we create the User Defined functions we need to create the
external
library (DLL) with the function. We add this library to VuGen bin directory. Once the library is
added then we assign user defined function as a parameter. The function should have the
following format: __declspec (dllexport) char* <function name>(char*, char*)Examples of
user defined functions are as follows:GetVersion, GetCurrentTime, GetPltform are some of
the user defined functions used in my earlier project.
20. What are the changes you can make in run-time settings? - The Run Time Settings that we
make are: a) Pacing - It has iteration count. b) Log - Under this we have Disable Logging
 Standard Log and c) Extended Think Time - In think time we have two options like Ignore
think time and Replay think time. d) General - Under general tab we can set the vusers as
process or as multithreading and whether each step as a transaction.
21. How do you perform functional testing under load? - Functionality under load can be tested
by running several Vusers concurrently. By increasing the amount of Vusers, we can
determine how much load the server can sustain.
22. What is Ramp up? How do you set this? - This option is used to gradually increase the
amount of Vusers/load on the server. An initial value is set and a value to wait between
intervals can be
specified. To set Ramp Up, go to ‘Scenario Scheduling Options’
23. What is the advantage of running the Vuser as thread? - VuGen provides the facility to use
multithreading. This enables more Vusers to be run per
generator. If the Vuser is run as a process, the same driver program is loaded into memory
for each Vuser, thus taking up a large amount of memory. This limits the number of Vusers
that can be run on a single
generator. If the Vuser is run as a thread, only one instance of the driver program is loaded
into memory for the given number of
Vusers (say 100). Each thread shares the memory of the parent driver program, thus
enabling more Vusers to be run per generator.
24. If you want to stop the execution of your script on error, how do you do that? - The
lr_abort function aborts the execution of a Vuser script. It instructs the Vuser to stop
executing the Actions section, execute the vuser_end section and end the execution. This
function is useful when you need to manually abort a script execution as a result of a specific
error condition. When you end a script using this function, the Vuser is assigned the status
"Stopped". For this to take effect, we have to first uncheck the “Continue on error” option in
Run-Time Settings.
25. What is the relation between Response Time and Throughput? - The Throughput graph
shows the amount of data in bytes that the Vusers received from the server in a second.
When we compare this with the transaction response time, we will notice that as throughput
decreased, the response time also decreased. Similarly, the peak throughput and highest
response time would occur approximately at the same time.
26. Explain the Configuration of your systems? - The configuration of our systems refers to that
of the client machines on which we run the Vusers. The configuration of any client machine
includes its hardware settings, memory, operating system, software applications, development
tools, etc. This system component configuration should match with the overall system
configuration that would include the network infrastructure, the web server, the database
server, and any other components that go with this larger system so as to achieve the load
testing objectives.
27. How do you identify the performance bottlenecks? - Performance Bottlenecks can be
detected by using monitors. These monitors might be application server monitors, web server
monitors, database server monitors and network monitors. They help in finding out the
troubled area in our scenario which causes increased response time. The measurements
made are usually performance response time, throughput, hits/sec, network delay graphs,
etc.
28. If web server, database and Network are all fine where could be the problem? - The
problem could be in the system itself or in the application server or in the code written for
the application.
29. How did you find web server related issues? - Using Web resource monitors we can find the
performance of web servers. Using these monitors we can analyze throughput on the web
server, number of hits per second that
occurred during scenario, the number of http responses per second, the number of
downloaded pages per second.
30. How did you find database related issues? - By running “Database” monitor and help of
“Data Resource Graph” we can find database related issues. E.g. You can specify the resource
you want to measure on before running the controller and than you can see database related
issues
31. How did you plan the Load? What are the Criteria? - Load test is planned to decide the
number of users, what kind of machines we are going to use and from where they are run. It
is based on 2 important documents, Task Distribution Diagram and Transaction profile. Task
Distribution Diagram gives us the information on number of users for a particular transaction
and the time of the load. The peak usage and off-usage are decided from this Diagram.
Transaction profile gives us the information about the transactions name and their priority
levels with regard to the scenario we are deciding.
32. What does vuser_init action contain? - Vuser_init action contains procedures to login to a
server.
33. What does vuser_end action contain? - Vuser_end section contains log off procedures.
 34. What is think time? How do you change the threshold? - Think time is the time that a real
user waits between actions. Example: When a user receives data from a server, the user may
wait several seconds to review the data before responding. This delay is known as the think
time. Changing the Threshold: Threshold level is the level below which the recorded think
time will be ignored. The default value is five (5) seconds. We can change the think time
threshold in the Recording options of the Vugen.
35. What is the difference between standard log and extended log? - The standard log sends a
subset of functions and messages sent during script execution to a log. The subset depends
on the Vuser type Extended log sends a detailed script execution messages to the output log.
This is mainly used during debugging when we want information about: Parameter
substitution. Data returned by the server. Advanced trace.
36. Explain the following functions: - lr_debug_message - The lr_debug_message function
sends a debug message to the output log when the specified message class is set.
lr_output_message - The lr_output_message function sends notifications to the Controller
Output window and the Vuser log file. lr_error_message - The lr_error_message function
sends an error message to the LoadRunner Output window. lrd_stmt - The lrd_stmt function
associates a character string (usually a SQL statement) with a cursor. This function sets a SQL
statement to be processed. lrd_fetch - The lrd_fetch function fetches the next row from the
result set.
37. Throughput - If the throughput scales upward as time progresses and the number of
Vusers increase, this indicates that the bandwidth is sufficient. If the graph were to
remain relatively flat as the number of Vusers increased, it would
be reasonable to conclude that the bandwidth is constraining the volume of
data delivered.
38. Types of Goals in Goal-Oriented Scenario - Load Runner provides you with five different
types of goals in a goal oriented scenario:
1. The number of concurrent Vusers
2. The number of hits per second
3. The number of transactions per second
4. The number of pages per minute
5. The transaction response time that you want your scenario
39. Analysis Scenario (Bottlenecks): In Running Vuser graph correlated with the response time
graph you can see that as the number of Vusers increases, the average response time of the
check itinerary transaction very gradually increases. In other words, the average response
time steadily increases as the load
increases. At 56 Vusers, there is a sudden, sharp increase in the average response
time. We say that the test broke the server. That is the mean time before failure (MTBF).
The response time clearly began to degrade when there were more than 56 Vusers running
simultaneously.
40. What is correlation? Explain the difference between automatic correlation and manual
correlation? - Correlation is used to obtain data which are unique for each run of the script
and which are generated by nested queries. Correlation provides the value to avoid errors
arising out of duplicate values and also optimizing the code (to avoid nested queries).
Automatic correlation is where we set some rules for correlation. It can be application server
specific. Here values are replaced by data which are created by these rules. In manual
correlation, the value we want to correlate is scanned and create correlation is used to
correlate.
41. Where do you set automatic correlation options? - Automatic correlation from web point of
view, can be set in recording options and correlation tab. Here we can enable correlation for
the entire script and choose either issue online messages or offline actions, where we can
define rules for that correlation. Automatic correlation for database, can be done using show
output window and scan for correlation and picking the correlate query tab and choose which
query value we want to correlate. If we know the specific value to be correlated, we just do
create correlation for the value and specify how the value to be created.
42. What is a function to capture dynamic values in the web vuser script? -
Web_reg_save_param function saves dynamic data information to a parameter.

Abstract:
Software testing is any activity aimed at evaluating an attribute or capability of a
program or system and determining that it meets its required results. [Hetzel88]
Although crucial to software quality and widely deployed by programmers and
testers, software testing still remains an art, due to limited understanding of the
principles of software. The difficulty in software testing stems from the complexity of
software: we can not completely test a program with moderate complexity. Testing is
more than just debugging. The purpose of testing can be quality assurance,
verification and validation, or reliability estimation. Testing can be used as a generic
metric as well. Correctness testing and reliability testing are two major areas of
testing. Software testing is a trade-off between budget, time and quality.

Contents:
• Introduction
• Key Concepts
• Taxonomy

• Testing automation
• When to stop testing?
• Alternatives to testing

• Available tools, techniques, and metrics

• Relationship to other topics
• Conclusions
• Annotated Reference List & Further Reading

Introduction
Software Testing is the process of executing a program or system with the intent of
finding errors. [Myers79] Or, it involves any activity aimed at evaluating an attribute or
capability of a program or system and determining that it meets its required results.
[Hetzel88] Software is not unlike other physical processes where inputs are received and
outputs are produced. Where software differs is in the manner in which it fails. Most
physical systems fail in a fixed (and reasonably small) set of ways. By contrast,
software can fail in many bizarre ways. Detecting all of the different failure modes for
software is generally infeasible. [Rstcorp]

Unlike most physical systems, most of the defects in software are design errors, not
manufacturing defects. Software does not suffer from corrosion, wear-and-tear --
generally it will not change until upgrades, or until obsolescence. So once the
software is shipped, the design defects -- or bugs -- will be buried in and remain latent
until activation.

Software bugs will almost always exist in any software module with moderate size:
not because programmers are careless or irresponsible, but because the complexity of
software is generally intractable -- and humans have only limited ability to manage
complexity. It is also true that for any complex systems, design defects can never be
completely ruled out.
Discovering the design defects in software, is equally difficult, for the same reason of
complexity. Because software and any digital systems are not continuous, testing
boundary values are not sufficient to guarantee correctness. All the possible values
need to be tested and verified, but complete testing is infeasible. Exhaustively testing
a simple program to add only two integer inputs of 32-bits (yielding 2^64 distinct test
cases) would take hundreds of years, even if tests were performed at a rate of
thousands per second. Obviously, for a realistic software module, the complexity can
be far beyond the example mentioned here. If inputs from the real world are involved,
the problem will get worse, because timing and unpredictable environmental effects
and human interactions are all possible input parameters under consideration.

A further complication has to do with the dynamic nature of programs. If a failure

occurs during preliminary testing and the code is changed, the software may now
work for a test case that it didn't work for previously. But its behavior on pre-error test
cases that it passed before can no longer be guaranteed. To account for this possibility,
testing should be restarted. The expense of doing this is often prohibitive. [Rstcorp]

An interesting analogy parallels the difficulty in software testing with the pesticide,
known as the Pesticide Paradox [Beizer90]: Every method you use to prevent or find
bugs leaves a residue of subtler bugs against which those methods are ineffectual. But
this alone will not guarantee to make the software better, because the Complexity
Barrier [Beizer90] principle states: Software complexity(and therefore that of bugs)
grows to the limits of our ability to manage that complexity. By eliminating the
(previous) easy bugs you allowed another escalation of features and complexity, but
his time you have subtler bugs to face, just to retain the reliability you had before.
Society seems to be unwilling to limit complexity because we all want that extra bell,
whistle, and feature interaction. Thus, our users always push us to the complexity
barrier and how close we can approach that barrier is largely determined by the
strength of the techniques we can wield against ever more complex and subtle bugs.
[Beizer90]

Regardless of the limitations, testing is an integral part in software development. It is

broadly deployed in every phase in the software development cycle. Typically, more
than 50% percent of the development time is spent in testing. Testing is usually
performed for the following purposes:

• To improve quality.

As computers and software are used in critical applications, the outcome of a bug can
be severe. Bugs can cause huge losses. Bugs in critical systems have caused airplane
crashes, allowed space shuttle missions to go awry, halted trading on the stock market,
and worse. Bugs can kill. Bugs can cause disasters. The so-called year 2000 (Y2K)
bug has given birth to a cottage industry of consultants and programming tools
dedicated to making sure the modern world doesn't come to a screeching halt on the
first day of the next century. [Bugs] In a computerized embedded world, the quality and
reliability of software is a matter of life and death.

Quality means the conformance to the specified design requirement. Being correct,
the minimum requirement of quality, means performing as required under specified
circumstances. Debugging, a narrow view of software testing, is performed heavily to
find out design defects by the programmer. The imperfection of human nature makes
it almost impossible to make a moderately complex program correct the first time.
Finding the problems and get them fixed [Kaner93], is the purpose of debugging in
programming phase.

• For Verification & Validation (V&V)

Just as topic Verification and Validation indicated, another important purpose of testing is
verification and validation (V&V). Testing can serve as metrics. It is heavily used as a
tool in the V&V process. Testers can make claims based on interpretations of the
testing results, which either the product works under certain situations, or it does not
work. We can also compare the quality among different products under the same
specification, based on results from the same test.

We can not test quality directly, but we can test related factors to make quality visible.
Quality has three sets of factors -- functionality, engineering, and adaptability. These
three sets of factors can be thought of as dimensions in the software quality space.
Each dimension may be broken down into its component factors and considerations at
successively lower levels of detail. Table 1 illustrates some of the most frequently
cited quality considerations.

Functionality Engineering (interior Adaptability (future

(exterior quality) quality) quality)
Correctness Efficiency Flexibility
Reliability Testability Reusability
Usability Documentation Maintainability
Integrity Structure
Table 1. Typical Software Quality Factors [Hetzel88]

Good testing provides measures for all relevant factors. The importance of any
particular factor varies from application to application. Any system where human
lives are at stake must place extreme emphasis on reliability and integrity. In the
typical business system usability and maintainability are the key factors, while for a
one-time scientific program neither may be significant. Our testing, to be fully
effective, must be geared to measuring each relevant factor and thus forcing quality to
become tangible and visible. [Hetzel88]

Tests with the purpose of validating the product works are named clean tests, or
positive tests. The drawbacks are that it can only validate that the software works for
the specified test cases. A finite number of tests can not validate that the software
works for all situations. On the contrary, only one failed test is sufficient enough to
show that the software does not work. Dirty tests, or negative tests, refers to the tests
aiming at breaking the software, or showing that it does not work. A piece of software
must have sufficient exception handling capabilities to survive a significant level of
dirty tests.
A testable design is a design that can be easily validated, falsified and maintained.
Because testing is a rigorous effort and requires significant time and cost, design for
testability is also an important design rule for software development.

• For reliability estimation [Kaner93] [Lyu95]

Software reliability has important relations with many aspects of software, including
the structure, and the amount of testing it has been subjected to. Based on an
operational profile (an estimate of the relative frequency of use of various inputs to
the program [Lyu95]), testing can serve as a statistical sampling method to gain failure
data for reliability estimation.

Software testing is not mature. It still remains an art, because we still cannot make it a
science. We are still using the same testing techniques invented 20-30 years ago, some
of which are crafted methods or heuristics rather than good engineering methods.
Software testing can be costly, but not testing software is even more expensive,
especially in places that human lives are at stake. Solving the software-testing
problem is no easier than solving the Turing halting problem. We can never be sure
that a piece of software is correct. We can never be sure that the specifications are
correct. No verification system can verify every correct program. We can never be
certain that a verification system is correct either.

Key Concepts
Taxonomy

There is a plethora of testing methods and testing techniques, serving multiple

purposes in different life cycle phases. Classified by purpose, software testing can be
divided into: correctness testing, performance testing, reliability testing and security
testing. Classified by life-cycle phase, software testing can be classified into the
following categories: requirements phase testing, design phase testing, program phase
testing, evaluating test results, installation phase testing, acceptance testing and
maintenance testing. By scope, software testing can be categorized as follows: unit
testing, component testing, integration testing, and system testing.

Correctness testing

Correctness is the minimum requirement of software, the essential purpose of testing.

Correctness testing will need some type of oracle, to tell the right behavior from the
wrong one. The tester may or may not know the inside details of the software module
under test, e.g. control flow, data flow, etc. Therefore, either a white-box point of view
or black-box point of view can be taken in testing software. We must note that the
black-box and white-box ideas are not limited in correctness testing only.

• Black-box testing

The black-box approach is a testing method in which test data are derived from the
specified functional requirements without regard to the final program structure.
 It is also termed data-driven, input/output driven [Myers79], or requirements-
[Perry90]
based [Hetzel88] testing. Because only the functionality of the software module is of
concern, black-box testing also mainly refers to functional testing -- a testing method
emphasized on executing the functions and examination of their input and output data.
[Howden87] The tester treats the software under test as a black box -- only the inputs,
outputs and specification are visible, and the functionality is determined by observing
the outputs to corresponding inputs. In testing, various inputs are exercised and the
outputs are compared against specification to validate the correctness. All test cases
are derived from the specification. No implementation details of the code are
considered.

It is obvious that the more we have covered in the input space, the more problems we
will find and therefore we will be more confident about the quality of the software.
Ideally we would be tempted to exhaustively test the input space. But as stated above,
exhaustively testing the combinations of valid inputs will be impossible for most of
the programs, let alone considering invalid inputs, timing, sequence, and resource
variables. Combinatorial explosion is the major roadblock in functional testing. To
make things worse, we can never be sure whether the specification is either correct or
complete. Due to limitations of the language used in the specifications (usually
natural language), ambiguity is often inevitable. Even if we use some type of formal
or restricted language, we may still fail to write down all the possible cases in the
specification. Sometimes, the specification itself becomes an intractable problem: it is
not possible to specify precisely every situation that can be encountered using limited
words. And people can seldom specify clearly what they want -- they usually can tell
whether a prototype is, or is not, what they want after they have been finished.
Specification problems contributes approximately 30 percent of all bugs in software.
[Beizer95]

The research in black-box testing mainly focuses on how to maximize the

effectiveness of testing with minimum cost, usually the number of test cases. It is not
possible to exhaust the input space, but it is possible to exhaustively test a subset of
the input space. Partitioning is one of the common techniques. If we have partitioned
the input space and assume all the input values in a partition is equivalent, then we
only need to test one representative value in each partition to sufficiently cover the
whole input space. Domain testing [Beizer95] partitions the input domain into regions,
and consider the input values in each domain an equivalent class. Domains can be
exhaustively tested and covered by selecting a representative value(s) in each domain.
Boundary values are of special interest. Experience shows that test cases that explore
boundary conditions have a higher payoff than test cases that do not. Boundary value
analysis [Myers79] requires one or more boundary values selected as representative test
cases. The difficulties with domain testing are that incorrect domain definitions in the
specification can not be efficiently discovered.

Good partitioning requires knowledge of the software structure. A good testing plan
will not only contain black-box testing, but also white-box approaches, and
combinations of the two.

• White-box testing
Contrary to black-box testing, software is viewed as a white-box, or glass-box in
white-box testing, as the structure and flow of the software under test are visible to the
tester. Testing plans are made according to the details of the software implementation,
such as programming language, logic, and styles. Test cases are derived from the
program structure. White-box testing is also called glass-box testing, logic-driven
testing [Myers79] or design-based testing [Hetzel88].

There are many techniques available in white-box testing, because the problem of
intractability is eased by specific knowledge and attention on the structure of the
software under test. The intention of exhausting some aspect of the software is still
strong in white-box testing, and some degree of exhaustion can be achieved, such as
executing each line of code at least once (statement coverage), traverse every branch
statements (branch coverage), or cover all the possible combinations of true and false
condition predicates (Multiple condition coverage). [Parrington89]

Control-flow testing, loop testing, and data-flow testing, all maps the corresponding
flow structure of the software into a directed graph. Test cases are carefully selected
based on the criterion that all the nodes or paths are covered or traversed at least once.
By doing so we may discover unnecessary "dead" code -- code that is of no use, or
never get executed at all, which can not be discovered by functional testing.

In mutation testing, the original program code is perturbed and many mutated
programs are created, each contains one fault. Each faulty version of the program is
called a mutant. Test data are selected based on the effectiveness of failing the
mutants. The more mutants a test case can kill, the better the test case is considered.
The problem with mutation testing is that it is too computationally expensive to use.
The boundary between black-box approach and white-box approach is not clear-cut.
Many testing strategies mentioned above, may not be safely classified into black-box
testing or white-box testing. It is also true for transaction-flow testing, syntax testing,
finite-state testing, and many other testing strategies not discussed in this text. One
reason is that all the above techniques will need some knowledge of the specification
of the software under test. Another reason is that the idea of specification itself is
broad -- it may contain any requirement including the structure, programming
language, and programming style as part of the specification content.

We may be reluctant to consider random testing as a testing technique. The test case
selection is simple and straightforward: they are randomly chosen. Study in [Duran84]
indicates that random testing is more cost effective for many programs. Some very
subtle errors can be discovered with low cost. And it is also not inferior in coverage
than other carefully designed testing techniques. One can also obtain reliability
estimate using random testing results based on operational profiles. Effectively
combining random testing with other testing techniques may yield more powerful and
cost-effective testing strategies.

Performance testing

Not all software systems have specifications on performance explicitly. But every
system will have implicit performance requirements. The software should not take
infinite time or infinite resource to execute. "Performance bugs" sometimes are used
to refer to those design problems in software that cause the system performance to
degrade.

Performance has always been a great concern and a driving force of computer
evolution. Performance evaluation of a software system usually includes: resource
usage, throughput, stimulus-response time and queue lengths detailing the average or
maximum number of tasks waiting to be serviced by selected resources. Typical
resources that need to be considered include network bandwidth requirements, CPU
cycles, disk space, disk access operations, and memory usage [Smith90]. The goal of
performance testing can be performance bottleneck identification, performance
comparison and evaluation, etc. The typical method of doing performance testing is
using a benchmark -- a program, workload or trace designed to be representative of
the typical system usage. [Vokolos98]

Reliability testing

Software reliability refers to the probability of failure-free operation of a system. It is

related to many aspects of software, including the testing process. Directly estimating
software reliability by quantifying its related factors can be difficult. Testing is an
effective sampling method to measure software reliability. Guided by the operational
profile, software testing (usually black-box testing) can be used to obtain failure data,
and an estimation model can be further used to analyze the data to estimate the
present reliability and predict future reliability. Therefore, based on the estimation, the
developers can decide whether to release the software, and the users can decide
whether to adopt and use the software. Risk of using software can also be assessed
based on reliability information. [Hamlet94] advocates that the primary goal of testing
should be to measure the dependability of tested software.

There is agreement on the intuitive meaning of dependable software: it does not fail in
unexpected or catastrophic ways. [Hamlet94] Robustness testing and stress testing are
variances of reliability testing based on this simple criterion.

The robustness of a software component is the degree to which it can function

correctly in the presence of exceptional inputs or stressful environmental conditions.
[IEEE90] Robustness testing differs with correctness testing in the sense that the
functional correctness of the software is not of concern. It only watches for robustness
problems such as machine crashes, process hangs or abnormal termination. The oracle
is relatively simple, therefore robustness testing can be made more portable and
scalable than correctness testing. This research has drawn more and more interests
recently, most of which uses commercial operating systems as their target, such as the
work in [Koopman97] [Kropp98] [Ghosh98] [Devale99] [Koopman99].

Stress testing, or load testing, is often used to test the whole system rather than the
software alone. In such tests the software or system are exercised with or beyond the
specified limits. Typical stress includes resource exhaustion, bursts of activities, and
sustained high loads.

Security testing
Software quality, reliability and security are tightly coupled. Flaws in software can be
exploited by intruders to open security holes. With the development of the Internet,
software security problems are becoming even more severe.

Many critical software applications and services have integrated security measures
against malicious attacks. The purpose of security testing of these systems include
identifying and removing software flaws that may potentially lead to security
violations, and validating the effectiveness of security measures. Simulated security
attacks can be performed to find vulnerabilities.

Testing automation

Software testing can be very costly. Automation is a good way to cut down time and
cost. Software testing tools and techniques usually suffer from a lack of generic
applicability and scalability. The reason is straight-forward. In order to automate the
process, we have to have some ways to generate oracles from the specification, and
generate test cases to test the target software against the oracles to decide their
correctness. Today we still don't have a full-scale system that has achieved this goal.
In general, significant amount of human intervention is still needed in testing. The
degree of automation remains at the automated test script level.

The problem is lessened in reliability testing and performance testing. In robustness

testing, the simple specification and oracle: doesn't crash, doesn't hang suffices.
Similar simple metrics can also be used in stress testing.

When to stop testing?

Testing is potentially endless. We can not test till all the defects are unearthed and
removed -- it is simply impossible. At some point, we have to stop testing and ship the
software. The question is when.

Realistically, testing is a trade-off between budget, time and quality. It is driven by

profit models. The pessimistic, and unfortunately most often used approach is to stop
testing whenever some, or any of the allocated resources -- time, budget, or test cases
-- are exhausted. The optimistic stopping rule is to stop testing when either reliability
meets the requirement, or the benefit from continuing testing cannot justify the testing
cost. [Yang95] This will usually require the use of reliability models to evaluate and
predict reliability of the software under test. Each evaluation requires repeated
running of the following cycle: failure data gathering -- modeling -- prediction. This
method does not fit well for ultra-dependable systems, however, because the real field
failure data will take too long to accumulate.

Alternatives to testing

Software testing is more and more considered a problematic method toward better
quality. Using testing to locate and correct software defects can be an endless process.
Bugs cannot be completely ruled out. Just as the complexity barrier indicates: chances
are testing and fixing problems may not necessarily improve the quality and reliability
of the software. Sometimes fixing a problem may introduce much more severe
problems into the system, happened after bug fixes, such as the telephone outage in
California and eastern seaboard in 1991. The disaster happened after changing 3 lines
of code in the signaling system.

In a narrower view, many testing techniques may have flaws. Coverage testing, for
example. Is code coverage, branch coverage in testing really related to software
quality? There is no definite proof. As early as in [Myers79], the so-called "human
testing" -- including inspections, walkthroughs, reviews -- are suggested as possible
alternatives to traditional testing methods. [Hamlet94] advocates inspection as a cost-
effect alternative to unit testing. The experimental results in [Basili85] suggests that
code reading by stepwise abstraction is at least as effective as on-line functional and
structural testing in terms of number and cost of faults observed.

Using formal methods to "prove" the correctness of software is also an attracting

research direction. But this method can not surmount the complexity barrier either.
For relatively simple software, this method works well. It does not scale well to those
complex, full-fledged large software systems, which are more error-prone.

In a broader view, we may start to question the utmost purpose of testing. Why do we
need more effective testing methods anyway, since finding defects and removing them
does not necessarily lead to better quality. An analogy of the problem is like the car
manufacturing process. In the craftsmanship epoch, we make cars and hack away the
problems and defects. But such methods were washed away by the tide of pipelined
manufacturing and good quality engineering process, which makes the car defect-free
in the manufacturing phase. This indicates that engineering the design process (such
as clean-room software engineering) to make the product have less defects may be
more effective than engineering the testing process. Testing is used solely for quality
monitoring and management, or, "design for testability". This is the leap for software
from craftsmanship to engineering.

Available tools, techniques, and metrics

There are an abundance of software testing tools exist. The correctness testing tools
are often specialized to certain systems and have limited ability and generality.
Robustness and stress testing tools are more likely to be made generic.

Mothora [DeMillo91] is an automated mutation testing tool-set developed at Purdue

University. Using Mothora, the tester can create and execute test cases, measure test
case adequacy, determine input-output correctness, locate and remove faults or bugs,
and control and document the test.

NuMega's Boundschecker [NuMega99] Rational's Purify [Rational99]. They are run-time

checking and debugging aids. They can both check and protect against memory leaks
and pointer problems.

Ballista COTS Software Robustness Testing Harness [Ballista99]. The Ballista testing
harness is an full-scale automated robustness testing tool. The first version supports
testing up to 233 POSIX function calls in UNIX operating systems. The second
version also supports testing of user functions provided that the data types are
recognized by the testing server. The Ballista testing harness gives quantitative
measures of robustness comparisons across operating systems. The goal is to
automatically test and harden Commercial Off-The-Shelf (COTS) software against
robustness failures.

Relationship to other topics

Software testing is an integrated part in software development. It is directly related to
software quality. It has many subtle relations to the topics that software, software
quality, software reliability and system reliability are involved.

Related topics
• Software reliability Software testing is closely related to software reliability.
Software reliability can be augmented by testing. Also testing can be served as
a metric for software reliability.
• Fault injection Fault injection can be considered a special way of testing. Fault
injection and testing are usually combined and performed to validate the
reliability of critical fault-tolerant software and hardware.
• Verification, validation and certification The purpose of software testing is not only for
revealing bugs and eliminate them. It is also a tool for verification, validation
and certification.

Conclusions
• Software testing is an art. Most of the testing methods and practices are not
very different from 20 years ago. It is nowhere near maturity, although there
are many tools and techniques available to use. Good testing also requires a
tester's creativity, experience and intuition, together with proper techniques.
• Testing is more than just debugging. Testing is not only used to locate defects
and correct them. It is also used in validation, verification process, and
reliability measurement.
• Testing is expensive. Automation is a good way to cut down cost and time.
Testing efficiency and effectiveness is the criteria for coverage-based testing
techniques.
• Complete testing is infeasible. Complexity is the root of the problem. At some
point, software testing has to be stopped and product has to be shipped. The
stopping time can be decided by the trade-off of time and budget. Or if the
reliability estimate of the software product meets requirement.
• Testing may not be the most effective method to improve software quality.
Alternative methods, such as inspection, and clean-room engineering, may be
even better.
Annotated References
• [Ballista99] http://www.cs.cmu.edu/afs/cs/project/edrc-ballista/www/

Ballista COTS Software Robustness Testing Harness homepage.

• [Basili85] Victor R. Basili, Richard W. Selby, Jr. "Comparing the Effectiveness

of Software Testing Strategies",

Technical Report, Department of Computer Science, University of Maryland, College

Park, 1985.

• [Beizer90] Boris Beizer, Software Testing Techniques. Second edition. 1990

• A very comprehensive book on the testing techniques. Many testing

techniques are enumerated and discussed in detail. Domain testing, data-flow
testing, transactin-flow testing, syntax testing, logic-based testing, etc.

• [Beizer95] Beizer, Boris, Black-box Testing: techniques for functional testing

of software and systems. Publication info: New York : Wiley, c1995. ISBN:
0471120944 Physical description: xxv, 294 p.: ill. ; 23 cm.

This book is a comprehensive introduction to various methods of testing, using

intuitive examples. Complete coverage of all important testing techniques, and up-to-
date. The focus is black-box/functional testing. The author is an internationally known
software consultant with almost four decades of experience in the computer industry.

• [Duran84] Joe W. Duran, Simeon C. Ntafos, "An Evaluation of Random

Testing", IEEE Transactions on Software Engineering, Vol. SE-10, No. 4, July
1984, pp438-443.

This research investigates the effectiveness of random testing, in comparison of other

testing methods. It convinces us that random testing method is much more powerful
than it appears and should be deployed more.

• [Hetzel88] Hetzel, William C., The Complete Guide to Software Testing, 2nd
ed. Publication info: Wellesley, Mass. : QED Information Sciences, 1988.
ISBN: 0894352423.Physical description: ix, 280 p. : ill ; 24 cm.

This book is a good guide to software testing. But it may not be as complete a guide
as it was 10 years ago.

• [Howden87] William E. Howden. Functional program Testing and Analysis.

McGraw-Hill, 1987.

In-depth discussion about functional testing throughout all product life-cycle.

 • [IEEE90] IEEE Standard Glossary of Software Engineering Terminology
(IEEE Std 610.12-1990), IEEE Computer Soc., Dec. 10, 1990.

Terminology standard.

[Kaner93] Cem Kaner, Testing Computer Software. 1993.

Discusses the purpose and techniques for software testing.

[Lyu95] Michael R. Lyu , Handbook of Software Reliability Engineering.

McGraw-Hill publishing, 1995, ISBN 0-07-039400-8

See topic Software Reliability reference.

[Myers79] Myers, Glenford J., The art of software testing, Publication info: New
York : Wiley, c1979. ISBN: 0471043281 Physical description: xi, 177 p. : ill. ;
24 cm.

Classical book on software testing and still has its influence today. Software testing
still remains an art, since the first day. We still don't know how to make it a science.

[NuMega99] http://www.numega.com/devcenter/bc.shtml

Introduction of the tools BoundsChecker by NuMega

[Parrington89] Norman Parrington and Marc Roper, Understanding Software

Testing, Published by John Willey & Sons, 1989. ISBN:0-7458-0533-7; 0-470-
21462-7

This book surveys current software methods and techniques available. The authors
show how, when carried out effectively in the commercial sphere, software
testing can result in higher quality software products, more satisfied users and
lower costs, which in turn leads to more accurate and reliable results.

• [Rational99] http://www.rational.com/products/purify_unix/index.jtmpl

• Introduces the Purify tool.

• [Rstcorp] http://www.rstcorp.com/definitions/software_testing.html

A definition of and introduction to software testing.

[PERRY90] A standard for testing application software, William E. Perry, 1990

This book summarizes and standardizes many testing techniques.

[Musa97] Software-reliability-engineered testing practice (tutorial);John D. Musa;

Proceedings of the 1997 international conference on Software engineering ,
1997, Pages 628 - 629
This 2-page short tutorial gets us started with an introduction of the basic ideas
underlying SRE testing of software. Definitions of reliability, failure, fault, severity
class, operation, operational profile, feature testing, load testing and regression testing
are included. Four steps, developing operational profiles, preparing for testing,
executing tests and interpreting failure data are introduced. This article is a good
introduction to testing-for-reliability.

[Vokolos98] Performance testing of software systems; Filippos I.Vokolos, and

Elaine J.Weyuker; Proceedings of the first international workshop on Software
and performance , 1998, Pages 80 - 87

Performance testing papers are very rare in literature, although performance is what
most people care about. This paper gives a discussion on approaches to software
performance testing. A case study describing the experience of using these approaches
for testing the performance of a system used a s a gateway in a large industrial
client/server transaction processing application is presented. It is not a very
representative paper, though.

[Bugs] http://www.cnet.com/Content/Features/Dlife/Bugs/?dd

Interesting readings about bugs' life. "Bugs -- How they breed and the damage
they do" and "10 great bugs of history", etc

[DeMillo91] Progress toward automated software testing; Richard A. DeMillo;

Proceedings of the 13th international conference on Software engineering ,
1991, Page 180

This paper mainly discusses their toolset Mothra, integrated environment for
automated software validation. Mothra uses program mutation to drive the tests. Few
human intervention needed. Automated test generation.

• [Hamlet94] Dick Hamlet; Foundations of software testing: dependability

theory; Proceedings of the second ACM SIGSOFT symposium on
Foundations of software engineering , 1994, Pages 128 - 139

• The point "four epochs of testing" is interesting.

• [Koopman97]Philip Koopman, John Sung, Christopher Dingman, Daniel

Siewiorek, Ted Marz. Comparing Operating Systems Using Robustness
Benchmarks. 16th IEEE Symposium on Reliable Distributed Systems,
Durham, NC, October 22-24, 1997, pp.72-79.

• Precursor work of Ballista project. Tests selected POSIX functions for

robustness.

• Philip Koopman, John DeVale. Comparing the Robustness of

[Koopman99]
POSIX Operating Systems. Proceedings of FTCS'99, 15-18 June 1999,
Madison, Wisconsin.
 • Ballista paper: multi-version comparison method to find silent and hindering
failures.

• [Kropp98]Kropp, N. P.; Koopman, P. J.; Siewiorek, D. P. Automated robustness

testing of off-the-shelf software components. Twenty-eighth Annual
International Symposium on Fault-Tolerant Computing (Cat. No.98CB36224)

• Ballista paper. Full scale, automated robustness test harness. Can test up to
233 POSIX function calls. Tested 15 operating systems from 10 vendors.
Between 42% and 63% of components tested had robustness problems, with a
normalized failure rate ranging from 10% to 23% of tests conducted.
Robustness testing could be used by developers to measure and improve
robustness, or by consumers to compare the robustness of competing COTS
component libraries.

• [Ghosh98] Testing the Robustness of Windows NT Software

• A simple heuristic method to test the robustness of some NT and GNU library.

• [DeVale99] John DeVale, Philip Koopman & David Guttendorf. The Ballista
Software Robustness Testing Service. Proceedings of TCS'99, Washington
DC.

• Describes the Ballista testing web server and client architecture.

• [Koehnemann93] Harry Koehnemann, and Timothy Lindquist; Towards

target-level testing and debugging tools for embedded software; Conference
proceedings on TRI-Ada '93 , 1993, Page 288

• Argues that current debugging method is not efficient for embedded software
and propose an improved method.

• [Smith90] Smith, C. U. Performance Engineering of Software Systems.

Addison-Wesley, 1990.

• A software performance testing paper.

• [Yang95] Yang, M.C.K.; Chao, A. Reliability-estimation and stopping-rules

for software testing, based on repeated appearances of bugs; IEEE
Transactions on Reliability, vol.44, no.2, p. 315-21, 1995

When to stop testing? The choice is usually based on one of two decision criteria: (1)
when the reliability has reached a given threshold, and (2) when the gain in reliability
cannot justify the testing cost. Various stopping rules and software reliability models
are compared by their ability to deal with these two criteria. Two new stopping rules,
initiated by theoretical study of the optimal stopping rule based on cost, are more
stable than other rules for a large variety of bug structures.

Future Reading
 • Software testing slides of RST corporation by Jeff Voas:
http://www.rstcorp.com/presentations/tampa98/

• Says testing and debugging can worsen reliability. 15% chance

• [Pham95] Software Reliability and Testing, pp29

• A collection of papers on software reliability testing.

• http://www.cs.jmu.edu/users/foxcj/cs555/Unit12/Testing/index.htm slide show of software

testing.

• comprehensive annotated www testing resource list with http links

• http://www.eg3.com/softd/index.htm

• Embedded s/w net resources

Comprehensive annotated www testing resource list:

• http://www.thegrid.net/tech/softd/softtest.htm

• Software testing and software quality are major issues. Here is the best the net
has to offer on them. keywords include: testing, software testing, software
reliability, software verification. This page discusses key issues such as
testing, software testing, software reliability, software verification.. testing,
software testing, software reliability, software verification. are discussed on
this page.

• http://www.io.com/~wazmo/qa.html
• http://isse.gmu.edu/faculty/ofut/rsrch/mut.html

• Mutation testing page and list of papers

Security Links:

• http://cs-www.ncsl.nist.gov/

• Computer Security Resource Clearinghouse, National Institute of Standards

and Technology

• http://www.sni.net/~deckm/

• Clean-room software engineering, inc.

Fun stuff:
• http://www.mtsu.edu/cgi-bin/users/storm/researchers.pl

• Meet the people: Who's Who in Software Testing Research

• http://www.bugnet.com/

• WWW resource for bug fixes, patches, and news.

• http://hongkong1.cnet.com/Briefs/Guidebook/Bugs2/ss05i.html

• 10 greatest bugs in history.

So says Ivars Peterson, author of Fatal Defect: Chasing Killer Computer Bugs. As he
concludes in his book, "The fact that we can never be sure that a computer
system will function flawlessly constitutes a fatal defect. It limits what we can
hope to achieve by using computers as our servants and surrogates. As
computer-controlled systems become more complex and thoroughly entwined
in the fabric of our lives, their potential for costly, life-threatening failures
keeps growing."