Software

Testing
What &
Why
Introduction

Software Testing
Books:
1.Software Testing, By: Yogesh Singh,
Cambridge University Press
2. Software testing: A Craftsman Approach
By: Paul C. Jorgensen, CRC Press
3. Software testing in the real world
By: Edward Kit, Pearson Education
4. Software testing: Principles and practices
By:Srinivasan Desikan and Gopalaswamy Ramesh,
Pearson Education

Software Testing
Overview
In a typical testing project, someone gives you a program and
asks you to test it.
The program may (or may not) come with a specification and
documentation.
Documents vary in accuracy and completeness.
You create tests, run them, and report the progress and
results of your work.
New versions of the program come to you. You might
reuse some old tests, check whether bugs have been
fixed, and (or) try new tests.
Eventually, the product is either cancelled or put into production.
3

Software Testing
What is testing?
A technical investigation of the product under test conducted to
provide stakeholders with quality-related information.

Defining Testing
A technical
means including experimentation, logic, mathematics, models,
tools (testing-support programs), and tools (measuring instruments,
event generators, etc.)

investigation
An organized and thorough search for information.
This is an active process of inquiry. We ask hard questions
(run hard test cases) and look carefully at the results.

Software Testing
of the product under test
The product includes the data, the documentation, the
hardware, whatever the customer gets. If it doesnt all work
together, it doesnt work.

stakeholders

Someone who has a vested interest in the success of the testing

effort.
Someone who has a vested interest in the success of the product.

with quality-related information

The information of interest is often about the presence
(or absence) of bugs.
Other types of information are sometimes more vital to the
stakeholders

Information Objectives
Find important bugs, to get them fixed
Assess the quality of the product
Help managers make release decisions
Block premature product releases
Help predict and control costs of product support
Check interoperability with other products
Find safe scenarios for use of the product
Assess conformance to specifications

Information Objectives

Certify the product meets a particular standard

Ensure the testing process meets accountability standards

Minimize the risk of safety-related lawsuits
Help clients improve product quality & testability
Help clients improve their processes
Evaluate the product for a third party

The six essentials of

software
testing
Software Testing: Means by which people,

methods,
measurements, tools and equipments are integrated to test the
software product.
1. The quality of the test process determines the success of
test effort:
Quality and effectiveness of software testing are determined
primarily by the quality of test processes used.
2. Prevent defect migration by using early life cycle testing
techniques:
More than half the errors are usually introduced at the
requirement phase.
Effective test program prevents the migration of errors from
any development phase to any subsequent phase.
8

The six essentials of

testing
3. The time forsoftware
software testing tool
is now:

Testing tool for every platform:

Capture/play back tools,
Tools for client/server,
Structural coverage, Etc.
4. A real person must take responsibility for improving the test
process:
Tools must be used with test process including effective test planning
and design.
To improve the testing process someone must plan and manage the
progress.
5. Testing is a professional discipline requiring trained, skilled
people:
It is not entry level job or stepping stone to other thing.
9
It is a profession.

6.

The six essentials of

software
testing
Cultivate a positive team attitude of creative
destruction:

Testers supply the information like:

Can we make this product fail to do what we expect it to do?
It is going to break as we can always find errors. But is it
breaking at reasonable boundaries?
Is it stressing or breaking acceptably or unacceptably,
given its criticality?
Do we know that we have covered all the important
possibilities?
After addressing all the questions only we can release the
10
product

Objective of testing
To show that a product do what it shouldt
do.
To show that a product doest do what it
should

11

The State of art and the

state
of
practice
SW testing yet to become fundamental component

of

university syllabus
Training in the industry is there but not properly defined
Published papers giving state of art methods are unproven in
the field (many well proven methods are unused in industry)
and omit real world dimension like return on investment.

12

History Aspects
Early days:
Testing was debugging (fix the bugs), performed by
developers.
Few dedicated resources for testing that too were very late in
the development process.
1957:
Testing distinguished from debugging.
Testing was still an after development activity.
1970:
Software engineering term used more often.
1972:
First formal conference on testing was held at the University
of North California.
13

History Aspects
1979:
Myers defined testing as The process of executing a
program with the intent of finding errors.
BUT IN THE REAL WORLD OF INDUSTRY: Testing got
dumped when the schedules and the budget becomes tight.
Testing too late to be done properly
1980:
Quality was buzzword
Various groups formed some standards like:
IEEE (Institute of Electrical and Electronics engineers)
ANSI (American National Standards Institute)
ISO (International Standards Organization)
1990:
Testing tools came.

14

Where are we now?

1980:
SEI(S/w Engineering Institute) at Carnegie Mellon University
developed CMM for US dept. of defense. 1987: 5 level CMM was
proposed to measure level of s/w maturity.

How should testing be positioned?

Early testing is the key to success of testing effort.
Testing has its own life cycle.
Testing is still subordinate to development.
Result: Little or no action based on testers input.
Now things are improving.

15

Software Testing
Establishing a practical perspective
Ultimate goal of testing: Quality Maintenance and Satisfaction
of customers.
What and why?
Mistake: A human action that produces incorrect result.
(Mistake produces fault or bug)
Fault: An incorrect step, process or data definition in a computer
program. The outgrowth of a mistake. (Potentially leads to
a failure)
Failure: An incorrect result. The result of the fault.
Error: The amount by which the result is incorrect
16

Software Testing
Where are errors?
Errors are concentrated in earlier stages of development
process. (Fig. 1.1)

Fig. 1.1 Defect Distribution

17

Historical definitions of
The cost of errors: All are
costly.
testing

Most expensive: Undetected errors and errors detected late.

Undetected: Migrate to other stages and cause failure in the field
financial and legal consequences, and high lifetime costs for the
system.

Historical definitions of testing:

1. Establishing confidence that a program does what it is
supposed to do (Hetzel, 1973).
2. The process of executing a program or system with the intent of
finding error (myers 1979)
3. Detecting specification errors and deviations from the
specification.
4. Any activity aimed at evaluating an attribute or capability of a
18
program or system (hetzel, 1983)

Historical definitions of
testing
5. The measurement of software quality (hetzel, 1983)
6. The process of evaluating a program or system.
7. Verifying that a system satisfies its specified requirement or
identifying difference between expected and actual results.
8. Confirming that a program performs its intended function
correctly.

19

The IEEE/ANSI definitions of

testing
1. The process of operating a system or component under specified
conditions, observing or recording the result and making an
evaluation of some aspect of the system or component.
2. The process of analyzing a software item to detect the difference
between existing and required conditions and to evaluate the
features of the software items.

20

Status of Software
Engineering

Therequirements
specificationwas
definedlikethis

Thedevelopers
understooditin
thatway

Thisishowthe
problemwas
solvedbefore.

Thisishowtheprogramis
Thatistheprogramafter describedbymarketing
department
debugging

Thisishowthe
problemis
solvednow

This,infact,iswhatthe
customerwanted
21

Some Software failures

Ariane 5
It took the European Space Agency 10
years and $7 billion to produce Ariane 5,
a giant rocket capable of hurling a pair of
three-ton satellites into orbit with each
launch and intended to give Europe
overwhelming
supremacy
in
the
commercial space business.
The rocket was destroyed after 39 seconds
of its launch, at an altitude of two and a
half miles along with its payload of four
expensive and uninsured scientific
satellites.

22

Some Software failures

When the guidance systems own
computer tried to convert one
piece of data the sideways velocity
of the rocket from a 64 bit format
to a 16 bit format; the number was
too big, and an overflow error
resulted after 36.7 seconds. When
the guidance system shutdown, it
passed control to an identical,
redundant unit, which was there to
provide backup in case of just such
a failure. Unfortunately, the second
unit , which had failed in the
identical
manner
a
few
milliseconds before.

23

Some Software
failures
Y2K problem:
It was simply the ignorance about the
adequacy or otherwise of using only
last two digits of the year.
The 4 digit date format, like 1964,
was shortened to 2 digit format, like
64.

24

Some Software failures

The Patriot Missile
o First time used in Gulf war
o Used as a defence from Iraqi Scud
missiles
o Failed several times including one that
killed 28 US soldiers in Dhahran,
Saudi Arabia
Reasons:
A small timing error in the systems clock
accumulated to the point that after 14
hours, the tracking system was no longer
accurate. In the Dhahran attack, the
system had been operating for more than
100 hours.

25

Some Software failures

The Space Shuttle
Part of an abort scenario for the
Shuttle requires fuel dumps to
lighten the spacecraft. It was
during the second of these
dumps that a (software) crash
occurred.
...the fuel management module,
which had performed one
dump and successfully exited,
restarted when recalled for the
second fuel dump...
26

Experience of Windows XP
Released on October 25,2001
Same day company posted 18 megabytes of
patches on its website for bug fixes, compatibility
updates and enhancements.

27

EXAMPLE

Consider a program minimum()

that reads a set of integers and
prints the smallest integer.

28

29

EXAMPLE
.

Some test cases

Sr.
No.

Inputs
Size

Set of integers

Expected
output

Observed
output

Match ?

6,9,2,16,19

Yes

96,11,32,9,39,99,91

Yes

31,36,42,16,65,76,81

16

16

Yes

28,21,36,31,30,38

21

21

Yes

106,109,88,111,114,116

88

88

Yes

61,69,99,31,21,69

21

21

Yes

6,2,9,5

Yes

99,21,7,49

Yes
30

Situations
We again consider the minimum() program and concentrate on some
typical and critical situations :
(i)
A very short list (of inputs) with the size of 1,2 or 3
elements.
(ii)

An empty list i.e. of size 0.

(iii)

A list where the minimum element is the first or last element.

(iv)

A list where the minimum element is negative.

(v)

A list where all elements are negative.

Cont.
31

Situations
(vi)

A list where some elements are real numbers.

(vii)

A list where some elements are alphabetic characters.

(viii) A list with duplicate elements.

(ix)
A list where one element has a value greater than
maximum permissible limit of an integer.

32

Situations
In the previous table, we have selected elements in every list to
cover essentially the same situation :
A list of moderate size.
Containing all positive integers.
Minimum is somewhere in the middle.

33

Some critical / typical situations of

program
Minimum()
Case number with
Inputs
Expected
Observed o/p
Match ?
description

Size

Set of Integers

a
b
c
d
e
f

1
2
2
3
3
3

Case 3
A list where minimum
element is first or last
element

10,23,34,81,97

Case 4
A list where minimum
element is negative

a
b

Case 1
A very short list with
size 1,2 or 3

Case 2
An empty list i.e. of
size 0

90
12,10
10,12
12,14,36
36,14,12
14,12,36

o/p
90
10
10
12
12
12

2147483647
2147483647
2147483647
14
14
12

No
No
No
No
No
Yes

Error
message

2147483647

No

10

23

No

97,81,34,23,10

10

23

No

10,-2,5,23

-2

No

5,-25,20,36

-25

20

No

----

Cont.

34

Case number with

description

Inputs
Size

Set of Integers

Expected
o/p

Observed o/p

Match
?

Case 5
A list where all
elements are
negative

-23,-31,-45,-56,-78

-78

31

No

-6,-203,-56,-78,-2

-203

56

No

Case 6
A list where some
elements are real
numbers

12,34.56,6.9,62.14,19

5.4

Case 7
A list where some
elements are
characters

Case 8
A list with duplicate
elements

6.9

34 (Program does not take

values for index 3,4,5)
838993460 (Program
does not take any array
value)

No

No

2,3,5,6,9

23,2 I,26,6,9

1I

2,3,4,9,6,5,11,12,14,21,
22

2 (No value for index

3,4,5)
2147483647 (Program
does not take any index
value)

3,4,6,9,6

No

13,6,6,9,15

Yes

530,4294967297,23,
46,59

23

No

Case 9
a
Value greater than
maximum permissible
limit of an integer

No

No

35

What are the possible reasons

for so many failures ?
Possible reasons
Case 1
A very short list with
size 1,2 or 3

While handling the minimum, the base value of the index and / or end value
of the index of the usable array has not been handled properly.
See line no. 25 & 26

Case 2
An empty list i.e. of
size 0

Program proceeds without checking the size of the array.

See line 18 & 19

Case 3
A list where minimum
element is the first or
last element

Same as for case 1

Case 4
A list where minimum
element is negative

The program converts all negative integers into positive integers

See line no. 22

Case 5
A list where all
elements are negative

Same as for case 4

Cont.

36

What are the possible reasons

for so many failures ?
Possible reasons

Case 6
Some elements are
real numbers

Scanf() has unpredictable behaviour for inputs not according to specified

format.

Case 7
Some elements are
alphabetic characters

Same as for case 6

Case 8
A case with duplicate
elements

(a)
(b)

Case 9
A list with one value
greater than maximum
permissible limit of an
integer

This is hardware dependent problem. This is the case of the overflow of

maximum permissible value of the integer. Here, 32 bits integers are used.

Same as case 1
We are getting correct results because minimum value is in the
middle of the list and all values are positive.

37

Reasons for observed outputs

Observed
output

Cases

Remarks

1 (a)
1 (b)
1 (c)

2147483647
2147483647
2147483647

The program has ignored first & last values of the list. This is the maximum
value of a 32 bit integer to which variable minimum is initialized.

1 (d)
1 (e)

14
14

The program has ignored first and last value of the list. Middle value is 14.

1 (f)

12

Ignored first & last value. Fortunately, middle value is minimum & thus the
result is correct.

2 (a)

2147483647

Same as 1 (a)

3 (a)
3 (b)

23
23

Ignored first & last value. 23 is the minimum value in the remaining lsit.

4 (a)
4 (b)

2
20

Ignored first & last value. It has also converted negative integer(s) to positive
integer(s).

Cont.

38

Reasons for observed outputs

Observed
output

Cases

Remarks

5 (a)
5 (b)

31
56

Same as case 4

6 (a)
6 (b)

34
858993460

After getting . of 34.56, program was terminated & 34 was displayed.12 was
ignored due to first value:
For 6(b): Garbage value

7 (a)
7 (b)

2
2147483647

After getting I, program terminated adruptly.

Maximum value of integer

8 (a)
8 (b)

4
6

Same as case 3
Fortunately result is correct. Although, first & last value are ignored.

9 (a)

The program displays this value due to the overflow of the 32 bit signed integer
data type used in the program

39

1.

Modification in the program

minimum
()
The program has ignored the first & last values of the list.
Line no. 25 & 26
Two faults
25
26

i=1;
while (i<No-1)

i=0;
while (i<=No-1)

2. The program proceeds without checking the size of the array

Line no. 17 & 18
After line 18
If (No <=0)
{
printf (Invalid size specified);
}

40

Modification in the program

minimum
() to positive values
3. Program has converted
negative values
Line no. 22
Delete this line
Modified program in fig. 2

41

42

Results of modified program

Case number with
description

Inputs
Size

Set of Integers

Observed o/p

90
10
10
12
12
12

90
10
10
12
12
12

Error
message

Error message

Match ?

a
b
c
d
e
f

1
2
2
3
3
3

Case 3
A list where minimum
element is first or last
element

10,23,34,81,97

10

10

Yes

97,81,34,23,10

10

10

Yes

Case 4
A list where minimum
element is negative

10,-2,5,23

-2

-2

Yes

5,-25,20,36

-25

-25

Yes

Case 1
A very short list with
size 1,2 or 3

Case 2
An empty list i.e. of
size 0

90
12,10
10,12
12,14,36
36,14,12
14,12,36

Expected
o/p

----

Yes
Yes
Yes
Yes
Yes
Yes
Yes

Cont.

43

Case number with

description

Inputs
Size

Set of Integers

Expected
o/p

Observed o/p

Match
?

Case 5
A list where all
elements are
negative

-23,-31,-45,-56,-78

-78

-78

Yes

-6,-203,-56,-78,-2

-203

-203

Yes

Case 6
A list where some
elements are real
numbers

12,34.56,6.9,62.14,19

5.4

Case 7
A list where some
elements are
characters

Case 8
A list with duplicate
elements

6.9

12

No

2,3,5,6,9

838993460 (Program
does not take any array
value)

No

23,2 I,26,6,9

No

1I

2,3,4,9,6,5,11,12,14,21,
22

2 (No value for index

3,4,5)
2147483647 (Program
does not take any index
value)

3,4,6,9,6

Yes

13,6,6,9,15

Yes

530,4294967297,23,
46,59

23

No

Case 9
a
Value greater than
maximum permissible
limit of an integer

No

44

Results of modified program

Case 6 & case 7 are failed due to scanf() function parsing problem
Do not use scanf()
If use scanf(), display a warning message for the user before
using scanf()

45

46

Why should we test ?

Software testing is expensive
not testing is more expensive
and dangerous
Running a car without brakes
How much testing ?
Do we have methods to know it?
Do we have techniques to quantify it ?
47

Who should do testing ?

Team work
Depend on complexity & size of the project

48

Who should do testing ?

Developers should have a reduced role in testing
Testing persons should be curious, critical but non
judge mental and good communicators.
Testers should ask questions that developers might
find not be able to ask themselves or are awkward,
irritating, insulting or even threatening to the
developers.
49

Role of Persons
Sr.
No.
1
2

Persons

Roles

Customer

Provides funding, give requirements, approve changes

& some test results.

Project Manager

Software
developer (s)

Testing
Co-ordinator (s)

Plans & manages the project.

Design, codes & builds the software, participates in
code reviews & testing, fixes bugs, defects & short
comings.
Create test plans & test specifications based on the
requirements and functional and technical documents.

Testing persons (s) Execute the tests and documents results.

50

What should we test ?

We should test the programs responses to every possible
input.
All valid inputs

Invalid inputs

Suppose a program requires two 8 bits integers as inputs.

Total combinations = 28 x 28
Time required to execute a test case = 1second
Total time = 18 hours
51

What should we test ?

How to choose reasonable number of test
cases out of large pool of test cases?
What is the bottom line for testing?
Execute every statement at least once
Execute every possible path at least once
Execute every exit of branch statement at
least once
52

What should we test ?

Executing every statement of the program at least once
may seem to be a reasonable goal. However many portions
of the program may be missed with this type of criteria.
Consider the following piece of code:
1. If (x>0)
2.
{
3.
a=a+b;
4.
}
5. If (y>10)
6.
{
7.
c=c+d;
8.
}

53

This code can be represented graphically as:

A

Line Numbers

Symbol for
representation

2,3,4

6,7,8

end

54

What should we test ?

The possible paths are: ACE, ABCE, ACDE and
ABCDE
However if we choose x=9 and y=15, all statements are
covered. Hence only one test case is sufficient for 100%
statement coverage. But we have missed other three
paths. Therefore this coverage may not be sufficient,
even that may also be difficult to achieve in the real life
programs.
55

What should we test ?

Another dimension is to execute all possible
paths of the program.
A program path can be traced through the code
from the start of the program to program
termination. Two paths differ if the program
executes different statements in each, or executes
the same statements but in different order. A
program may have many paths.
56

Sequences
B
F

A
D
C

G
X

EXIT

H
I

< 20 times
through the
loop

Heres an example that shows that there are too many

paths to test in even a fairly simple program. This is from
57
Myers, The Art of Software Testing.

Sequences
B
F

A
D
C

G
X

EXIT

H
I

< 20 times
through the
loop

There are 5 ways to get from A to X. One of them is ABX--EXIT

58

Sequences
B
F

A
D
C

G
X

EXIT

H
I

< 20 times
through the
loop

A second path is ACDFX--EXIT

59

Sequences
B
F

D
G

X
E

EXIT

H
I

< 20 times
through the
loop

Third: ACDGX--EXIT
60

Sequences
B
F

A
D
C
E

EXIT

H
I

< 20 times
through the
loop

Fourth: ACEHX--EXIT
61

Sequences
B
A

F
D
C
E

EXIT

H
I

< 20 times
through the
loop

Fifth: ACEIX--EXIT. There are 5 ways to get from A to the

EXIT if you go through X only once.
62

Sequences
B
F

A
D
C
E

G
H
I

EXIT

< 20 times
through the
loop

But we can go through X more than once. Heres another

path. ACEHXABX--EXIT.
63

Sequences
B
F

A
D
C
E

G
H
I

EXIT

< 20 times
through the
loop

There are 5 ways to get to X the first time, 5 more to get

back to X the second time, so there are 5 x 5 = 25 cases for
reaching EXIT by passing through X twice.

64

Sequences
Analyzing Myers example
There are 51 + 52 + ... + 519 + 520 = 1014 =
100 trillion paths through the program.
It would take only a billion years to test
every path (if one could write, execute and
verify a test case every five minutes).
65

The Impossibility Of Testing

Everything
In his monumental book Testing Object-Oriented
Systems, Robert Binder provides an excellent
example of the impossibility of testing
"everything." Consider the following program:

int blech (int j)

{
j = j -1;
// should be j = j + 1
j = j / 30000;
return j;
}

66

The Impossibility Of Testing

Everything
Note that the second line is incorrect! The function blech
accepts an integer j, subtracts one from it, divides it by 30000
(integer division, whole numbers, no remainder) and returns
the value just computed.
If integers are implemented using 16 bits on this computer
executing this software, the lowest possible input value is
-32768 and the highest is 32767. Thus there are 65,536
possible inputs into this tiny program.
Will you have the time (and the stamina) to create 65,536 test
cases? Of course not. So which input values do we choose?
Consider the following input values and their ability to detect
this defect.
67

Input (j)

Expected Result

Actual Result

42

40000

-64000

-2

-2

Note that none of the test cases chosen have detected

this defect. In fact only four of the possible 65,536 input
values will find this defect. What is the chance that we
will choose all four? What is the chance we will choose
one of the four?
68

Four of the input values that will find the

defect:
Input (j)

Expected Result

Actual Result

30,000

29,999

-29,999

-1

-30,000

Four of the these possible 65,536 input values will find this
defect. What is the chance that we will choose all four? What
is the chance we will choose one of the four?
69

Limitation of Testing
Complete Testing Is Impossible

70

Complete testing?
What do we mean by "complete testing"?
Complete "coverage": Tested every line / branch / basis path?
Testers not finding new bugs?
Test plan complete?

Complete testing must mean that, at the end of

testing, you know there are no remaining
unknown bugs.

After all, if there are more bugs, you can find them if you do
more testing. So testing couldn't yet be "complete."
71

The Limitation Of Testing

There is a popular belief that we can test a
program completely:
Some junior level programming texts even claim to tell us how
to do it: Test the programs response to all possible inputs, or
test all possible paths through the program.

72

The Limitation Of Testing

Many managers also believe in the possibility of complete
testing. They order even the staff to do it, and assure each
other that is being done.
Sales brochures from some software testing companies
promise that they will fully test our code.
Test coverage analyzers are sometimes marketed with the
promise of telling us, whether we have fully tested the code,
and what further testing we must do to achieve complete
testing.
Many salespeople believe their software products are fully
tested and error free, and pass on this claim to customers.
73

The Limitation Of Testing

No matter, how hard we try, how cleverly we plan, how much
time we spend, and how many staff and computers we use, we
still can not do enough testing. We still miss bugs. Some of the
reasons are:
The domain of possible inputs is too large to test
There are too many possible paths through the program to
test
The user interface issues (and thus the design issues) are too
complex to completely test. If 3+3 should be 7 is written in
specifications, and program gives output of 7 (instead of 6);
is it a bug?

74

Measurement of the progress

of testing
What is the status of testing ?
We count things
How to interpret such
counts ?
Is experiencing more failures a good news or a
bad ?

The answer could be either.

75

Measurement of the progress

of testing

A high bug count could mean that testing was thorough and
very few bugs remain. Or, it could mean that the software
simply has lots of bugs and, even though many have been
exposed, lots of them remain. These counts may be illusive and
do not help us to measure the progress of testing.
When to release the software is a sensitive decision & should be
based on the status of testing. Although in the absence of testing
standards, economics, time to market and gut feeling
have become important issues over technical considerations for
the release of any software. Many reliability models are
available with many limitations and are not universally
acceptable.
76

Measurement of the progress

of testing
Software companies are facing serious challenges in
testing their products as software grows more complex.
Complex nature of testing, so take it seriously.
Researchers should also realize that companies are
ready to fund good research ideas.
Hard work hard is required to bridge the gap
between academic research and real industry
requirements.
Develop standards for software testing.
77

Program & Software

Some Terminologies

What is Software?
Computer programs and associated documentation

78

What is software?
Programs

Documentatio
n

Operating
Procedures

Software=Program+Documentation+Operating Procedures
Components of software
79

Documentation consists of different

types of manuals Formal
areSpecification
Analysis
/Specification

ContextDiagram
Data Flow
Diagrams

Design
Documentation
Manuals

Flow Charts
Entity-Relationship
Diagram
Source Code Listings

Implementation
Testing

Cross-Reference
Listing
Test Data
Test Results

List of documentation manuals

80

Documentation consists of different

types of manuals
are
System
Overview
User
Manuals

Beginners Guide
Tutorial
Reference Guide

Operating
Procedures

Installation Guide
Operational
Manuals
System
Administration Guide

List of operating procedure manuals.

81

Software Testing
Error, Mistake, Bug, Fault and Failure
People

make

errors.

good

synonym is mistake. This may be a

syntax error or misunderstanding of
specifications.

Sometimes,

there

are logical errors.

When developers make mistakes

while

coding,

we

call

these

mistakes bugs.
82

Software Testing
A fault is the representation of an error, where representation
is the mode of expression, such as narrative text, data flow
diagrams, ER diagrams, source code etc. Defect is a good
synonym for fault.
Fault is of two types:(a)Fault of commission
(b)Fault of omission

83

Software Testing
FAULT OF COMMISSION:- It occurs when we enter
something into a representation that is incorrect.
FAULT OF OMISSION:- It occurs when we fail to enter correct
Information
Of these two types, faults of omission are more difficult to
detect and resolve.

84

Software Testing
A failure occurs when a fault executes. A particular fault may
cause different failures, depending on how it has been
exercised.
Correspond to fault of commission
Failure
Fault of omission are found by
careful reviews

85

Software Testing
INCIDENT:- When a failure occurs, it may or may not be
readily apparent to the user . An incident is the symptom
associated with a failure that alerts the user to the occurrence
of failure.

86

Erroneous State
(Error)
Errors
Errors
Stress
Stressor
or overload
overloaderrors
errors
Capacity
Capacityor
or boundary
boundary
errors
errors
Timing
Timingerrors
errors
Throughput
Throughputor
or
performance
performanceerrors
errors
87

Algorithmic Fault

Algorithmic
Algorithmic Faults
Faults
Missing
Missinginitialization
initialization
Branching
Branchingerrors
errors(too
(too
soon,
soon, too
toolate)
late)
Missing
Missingtest
testfor
for nil
nil

88

Mechanical Fault

Mechanical
Mechanical Faults
Faults
(very
(very hard
hard to
to find)
find)
Documentation
Documentation does
does
not
actual
not match
match
actual
conditions
conditions or
or operating
operating
procedures
procedures

89

How do we deal with

Errors and Faults?

90

Modular Redundancy?

Modular redundancy:
Expensive

91

Declaring the Bug

as a Feature?

Declaring a bug to be a
feature
Bad practice

92

Patching?

Patching
Slows down performance

93

Testing?
Testing
Testing is never good enough

94

Test, Test Case and Test

Suite
Test and Test case terms are used interchangeably. In practice, both
are same and are treated as synonyms. Test case describes an input
description and an expected output description.
Test Case ID
Section-I
(Before Execution)
Purpose :
Pre condition: (If any)
Inputs:

Section-II
(After Execution)
Execution History:
Result:
If fails, any possible reason (Optional);

Expected Outputs:
Post conditions:
Written by:
Date:

Any other observation:

Any suggestion:
Run by:
Date:

Fig. 2: Test case template

The set of test cases is called a test suite. Hence any combination of
95
test cases may generate a test suite.

Software Testing
TEST CASE:-

The output portion of a test case is

frequently overlooked, which is unfortunate because this
is often the hard part.

For example: You were testing software that determined an optimal

route for an aircraft, given certain FAA air corridor
constraints and the weather data for a flight day.
The academic response is to postulate the existence of an
oracle who knows all the answers.
96

Software Testing

One Industrial response to this problem is known as

reference Testing, where the system is tested in the
presence of expert users.

These experts make judgments as to whether outputs of

an executed set of test cases inputs are acceptable.

97

Software Testing
INSIGHT FROM A VENN DIAGRAM
Testing is fundamentally concerned with behavior and
behavior is orthogonal to the structural view common to
software developers.
One of the continuing sources for difficulty for testers is
that the base documents are usually written by and for
developers, the emphasis is on structural, instead of
behavioral information.
So, we use venn diagram to clarify questions about testing.
98

Software Testing

Consider a universe of program behaviors. Given a

program and its specification, Consider circle labeled S of
specified Behaviors, and the circle labeled P of programmed
behaviors.
99

Software Testing
What if certain specified behavior are not programmed? These
are faults of omission
What if certain Programmed behavior are not Specified? These
are faults of commission.
The intersection of S and P is the correct portion, that is,
behavior that are both specified and implemented and it is the
good view of testing.

100

Software Testing

Consider the relationships among the sets S, P and T.

There may be specified behaviors that are not tested(regions 2 and 5).
101

Software Testing
Specified behaviors that are tested(regions 1 and 4)
Test cases that corresponds to unspecified behavior(regions3 and 7)
Programmed behaviors that are not tested(regions 2 and 6)
Programmed behavior that are tested(region 1and 3)
Test cases that corresponds to unprogrammed behavior(regions 4 and 7)

102

Software Testing
If certain test cases corresponds to unspecified behaviors, two
possibilities arise:
(a) Such a test case is unwanted
(b) Specification is deficient.

This is a fine reason to have good testers participate in

specification and design reviews.
We can see some possibilities for testing as a craft: What can tester
do to make the region where these sets all intersects(region 1)

103

Software Testing
FUNCTIONAL TESTING
With functional approach to test case identification, the only
information used is the specification of software. Functional test case
have two advantages:
(1) They are independent of how the software is implemented, so if the
implementation changes, the test case are still useful.
(2) Test cases development can occur in parallel with the implementation,
thereby reducing overall project development interval.

104

Software Testing

Above figure shows the result of test cases identified by two functional
methods.
105

Software Testing
Method A identifies a larger set of test cases than does Method B.

For both methods, set of test cases is completely contained within

the set of specified behavior.

The above is due to the fact that the functional methods are based
.
on the specified behavior, It is hard to imagine these methods
identifying behaviors that are not specified.

106

Software Testing
STRUCTURAL TESTING

Structural testing is another approach to test case identification and

based on how the function is actually implemented.

Because of its strong theoretical basis, structural testing lends

itself to the definition and use of test coverage metrics.

Test coverage metrics is way to explicitly state the extent to which

software item is tested.

This makes testing management more meaningful

107

Software Testing

The above figure shows the results of test cases identified

by two structural methods.
108

Software Testing
Method A identifies a larger set of test cases than Method B.
For both methods, the set of test cases is completely contained
within the set of programmed behavior.
Because structural method are based on the program, it is hard
to imagine these methods identifying behaviors that are not
programmed.
It is easy to imagine, however, that a set of structural test cases
is relatively small with respect to the full set of programmed
behaviors.
109

Software Testing
THE FUNCTIONAL VERSUS STRUCTURAL
DEBATE

Software Testing
If all specified behaviors have not been
implemented, structural test case will never be able
to recognize this.
Conversely, if the program implements behavior that
have not been specified, this will never be revealed
by functional test cases.
Thus: Both approaches are needed; a judicious
combination will provide the confidence of the
functional testing and the measurement of structural
testing.
111

Faults Classified by Severity

Fault Severity

Fault Example

Mild

Misspelled word

Moderate

Misleading or redundant information

Annoying

Truncated names, bill for Rs 0.00

Disturbing

Some transactions not processed

Serious

Lose a transaction

Very Serious

Incorrect transaction execution

Extreme

Frequent very serious errors

Intolerable

Database corruption

Catastrophic

System Shutdown

Infectious

Shutdown that spreads to other

112

ERROR AND FAULT TAXONOMIES

Input/output Faults
Type
Input

Output

Instances
Correct input not accepted
Incorrect input accepted
Description wrong or missing
Parameters wrong or missing
Wrong format
Wrong result
Correct result at wrong time(too early , too
late)
Incomplete or missing results
Spelling/grammar
Cosmetic

 Logic Faults
Missing case(s)
Duplicate case(s)
Extreme condition neglected
Misinterpretation
Missing Condition
Extraneous condition)s)
Test of wrong variable
Incorrect loop iteration
Wrong operator(e.g., <instead of<=)

 Computational Faults
Incorrect algorithm
Missing Computation
Incorrect operand
Incorrect operation
Parenthesis error
Insufficient precision(round-off, truncation)
Wrong built-in function

 Interface Faults
Incorrect interrupt handling
I/O timing
Call to wrong procedure
Call to nonexistent procedure
Parameter mismatch(type, number)
Incompatible types
Superfluous types

 Data Faults
Incorrect initialization
Incorrect storage/access
Wrong flag/index/value
Incorrect packing/unpacking
Wrong variable used
Wrong data reference
Scaling or units error
Incorrect data dimension
Incorrect subscript
Incorrect type
Incorrect data scope
Sensor data out of limits
Off by one
Inconsistent data

Software Testing
Deliverables and Milestones
Different deliverables are generated during various phases of
software development. The examples are source code, software
requirement and specification (SRS) document, software design
document (SDD), installation guide, user reference manual etc.
Milestones are the events that are used to ascertain the status of the
project.
Example, finalization of SRS is a milestone. Completion of SDD is
another milestone.
The milestones are essential for monitoring and planning the
progress of the development.

118

Software Testing
Alpha, Beta and Acceptance Testing
The term Acceptance Testing is used when the software is developed
for a specific customer. A series of tests are conducted to enable the
customer to validate all requirements. These tests are conducted by
the end user / customer and may range from adhoc tests to well
planned systematic series of tests.
The terms alpha and beta testing are used when the software is
developed as a product for anonymous customers.
Alpha Tests are conducted at the developers site by some potential
customers. These tests are conducted in a controlled environment.
Alpha testing may be started when formal testing process is near
completion.
Beta Tests are conducted by the customers / end users at their sites.
Unlike alpha testing, developer is not present here. Beta testing is
conducted in a real environment that cannot be controlled by the
119
developer.

Software Testing
Quality and Reliability
Software reliability is one of the important factor of software quality.
Software reliability is defined as:
the probability of failure free operation for the specified time in a
specified environment
Software Quality measures how well software is designed (quality of
design), and how well the software conforms to that design (quality of
conformance).
Software Reliability is one of the part of software quality.
To produce good quality product, a software tester must verify and
validate throughout the software development process.
120

Software Testing
Testing, Quality Assurance and Quality Control
The purpose of testing is to find bugs and find them as early as
possible.
The purpose of quality assurance activities is to enforce standards
and techniques to improve the development process and prevent
bugs from ever occurring.
Quality assurance group monitors and guides throughput the
software development life cycle. Examples are reviews, audits, etc.
Quality control attempts to build a software and test it thoroughly. It
concentrates on specific products rather than processes as in the
case of quality assurance. This is a defect detection and correction
activity which is usually done after the completion of the software
development,. Example is software testing at various levels.
121

Software Testing
Static and Dynamic Testing
Static Testing refers to testing activities without executing the code.
All verification activities like inspections, walkthroughs, reviews etc
come under this category of testing.
Static Testing should be started at earlier phases of software
development and gives very good results at reasonable cost.
Dynamic Testing refers to execute the code and see how it performs
with specific inputs. All validation activities comes in this category
where execution of program is essential.

122

Software Testing
Testing and Debugging
The purpose of testing is to find faults and find them as early as
possible.
The process used to determine the cause of the fault and to remove it
is known as debugging.

123

V Shaped Software Life cycle

Model
There is a shift in testing activities from validation to verification
where we want to review/inspect every activity of software
development life cycle.
These verification activities are treated as error preventive
exercises and are applied at requirements analysis and
specification phase, high level design, detailed design phase,
implementation phase.
We not only want to improve the quality of end products of all
phases but also want to design test cases and test plans during
these phases.

V shaped model is the modified form of waterfall model

with a special focus on testing activities at every phase .
124

V Shaped Software Life cycle

A
Model
Requirements Analysis and
Specification

Acceptance Testing

B
System testing

High level design

C
Development
Part

Detailed design

Unit and Integration

testing
Testing Part

A: Acceptance test cases design and

planning
B: System test cases design and
planning
C: Unit and integration test cases design
and planning

Implementation

125

V Shaped Software Life cycle

Model
The model brings the quality into the development of our products
The encouragement of writing test cases and test plans in the early
phases of software development life cycle is the real strength of
this model.
We require more resources to implement this model as compared
to waterfall model.
LIMITATIONS
This model suffers from many disadvantages of waterfall model like
non availability of working version of the product until late in the
life cycle, difficult to accommodate any change etc. this model has
also limited applications in todays iterative software processes

126

Effective testing and cost effective testing:

Basic forms of testing are as follows:
1.

Full testing: Starts no later than the requirement phase

and continues through acceptance testing.

2.

Partial testing: Begins any time after functional design

has been completed with less than optimum influence on
requirement and functional design.

3.

Endgame testing: It is highly validation oriented with no

influence on requirements or functional design.

127

4.
Audit level testing: It is a bare bones audit of plans,
procedures, and products for adequacy, correctness and
compliance to standards.
Use full testing for critical software or any s/w that will
have heavy and diverse usage by a large user
population.
Use partial testing for small, non-critical software
products with a small, captive user population.

128

Critical S/W (IEEE/ANSI)

Is a s/w whose failure could have an impact on safety, or
could cause large financial or social losses.
To enable an effective test effort, we need a s/w development
process that produces:

Requirement specifications (required for full testing)

Functional design specifications (required for full, partial

and endgame testing)

Internal design specification (required for maximum,

effectiveness of full and partial testing)
129

Effective testing is a testing that is successful at detecting

errors but may not be cost effective.
Do we know what testing is really costing us?

What %age of our development resource testing

represents
Are testing tools saving time?
Are they are exploited to their fullest?
Is our testing cost effective? (Fig. 2.3)
130

The more effective the error detection, the greater the

savings in the development and maintenance costs over
the life of the product.
131