Software Engineering

LECTURE 20 – 21: SOFTWARE TESTING

1
What is Software Testing?
Testing is intended to show that a program does what it
is intended to do
Discover program defects before it is put into use.
To achieve this, you execute the program with artificial
data
You check the results of the test run for errors,
anomalies or information about program's non-functional
attributes.

2
 Why Testing and Analysis?

Software is never correct no matter which developing

technique is used
Any software must be verified.
Software testing and analysis are:
◦ important to control the quality of the product (and of the
process)
◦ very (often too) expensive
◦ difficult and stimulating

3
Ariane 5
Development took 10 years & $7 billion
On June 4, 1996, the flight of the Ariane 5 launcher ended in
a failure
Only about 37 seconds after initiation of the flight sequence,
at an altitude of about 3,700 m, the launcher veered off its
flight path, broke up and exploded
“Fixed” Ariane 5 – currently being used

4
 Source: ARIANE 5 Flight 501 Failure, Report by the Inquiry
Board
A program segment for converting a floating point number to a
signed 16 bit integer was executed with an input data value of a
64-bit number.
The segment was written for Ariane 4 and the conversion was
left unprotected, because it was thought that Ariane 4 will never
attain such a value.
Ariane 5 – Root The subsystem containing this segment was reused for Ariane 5
Cause as it had been successfully used over years for Ariane 4
This run time error (out of range, overflow), which arose in both
the active and the backup computers at about the same time, was
detected and both computers shut themselves down.
This resulted in the total loss of altitude control. This breakup
was detected by an on-board monitor which ignited the explosive
charges to destroy the vehicle in the air.
Ironically, this program segment was not required by Ariane 5

5
 Adequate exception handling and redundancy strategies (real
function of a backup system, degraded modes?)

Ariane 5 – Clear, complete, documented specifications (e.g.,

preconditions, post-conditions)
Lessons But perhaps more importantly: usage-based testing (based on
Learned operational profiles), in this case actual Ariane 5 trajectories
Note this was not a complex, computing problem, but a
deficiency of the software engineering practices in place …

6
F-18 crash
An F-18 crashed because of a missing exception condition:
◦ An if ... then ... block without the else clause that was thought
could not possibly arise.

In simulation, an F-16 program bug caused the virtual plane

to flip over whenever it crossed the equator, as a result of a
missing minus sign to indicate south latitude.

7
Fatal Therac-
25 Radiation
In 1986, a man in Texas received
between 16,500-25,000 radiations
in less than 10 sec, over an area of
about 1 cm.
He lost his left arm, and died of
complications 5 months later.

8
 Incident: A patient passed away
Failure: The device applied higher frequency of radiations
than what was safe. Safety range: [1…10,000 Hz]
Fault: The software controller of the device did not have a
conditional block (if …. else statements) to perform range
Therac 25 checking on the frequency of the radiation to be applied.
Errors:
◦ 1. The SW developer of the device controller system had
forgotten to include a range checking conditional block on the
frequency of the radiation to be applied.
◦ 2. The device operator was NOT supposed to enter anything
outside [1…10,000 Hz] range.

9
Errors
Errors are part of our daily life
Humans make errors in their thoughts, actions, and in the
products that might result from their actions
Errors occur wherever humans are involved in taking actions
and making decisions
These fundamental facts of human existence make testing an
essential activity

10
Faults and Failures
Fault (bug):
Programming or design error whereby the delivered system
does not conform to specification (e.g., coding error, protocol
error)
Failure:
Software does not deliver the service expected by the user
(e.g., mistake in requirements, confusing user interface)

11
Software Error, Fault, Failure

If not
removed Defect

12
Terminology
Fault avoidance
Build systems with the objective of creating fault-‐free (bug-‐free)
software.
Fault detection (testing and verification)
Detect faults (bugs) before the system is put into operation or when
discovered after release.
Fault tolerance
Build systems that continue to operate when problems (bugs, overloads,
bad data, etc.) occur.

13
Test Cases
A test case, in software engineering, is a set of conditions or variables under
which a tester will determine whether an application, software system or one of
its features is working as it was originally established for it to do.

“A set of test inputs, execution conditions, and expected results

developed for a particular objective, such as to exercise a particular
program path or to verify compliance with a specific requirement.” (IEEE)

Developing and writing Test Casesis one of the major and most important
activity in the entire testing cycle of an application

14
How to write good test cases
When you begin writing test cases, there are a few steps that you need to follow to ensure that you
are writing good test cases
Identify the purpose of testing
You need to understand the requirements to be tested. Writing test cases for a module requires
that you understand the features/user requirements for that module
Define how to perform the testing
This involves developing typical scenarios of use
Identify any non-functional requirements
Understand other aspects of the software related to non-functional requirements

15
Structuring you Test Cases
1. Test Case ID ( This is the unique number which helps in identifying a specific test case)
2. Test Case Description
3. Module to be tested (Usually we provide Requirement ID to maintain traceability between test case
and requirements)
4. Test Data ( We provide variable and values based on need of the test case)
5. Test Steps ( Steps to be executed)
6. Expected results ( How application should behave after performing stated test steps)
7. Actual results ( Actual output tester will get after preforming steps)
8. A result ( Pass or fail after comparing expected and actual results)
9. Comments ( We can provide screen shot or any other relevant information to help developer debug
the code)

16
USE CASE
FOR ADDING
NUMBERS

17
Test CASE
FOR ADDING
NUMBERS

18
Testing Strategies
SW testing is one element of a broader topic referred to as V&V (many SQA activities).
 Verification (correctly implement specific function)
 Defect detection and correction
 Comparison between implementation and the corresponding
specification
 Are we building the product right?
 Validation (SW is traceable to customer requirements)
 Defect prevention
 Provision of sound basis for specific design decisions
 Are we building the right product?
Example: Checking the printing of receipts is verification and correct printing (info) is validation.

19
Types/Strategies of testing
Code Inspections
Software (Module) Testing
◦ Unit Testing
◦ Functional Testing

Integration Testing
◦ Compliance
◦ Interoperability Testing

System Testing
◦ Recovery
◦ Security
◦ Environment

Acceptance Testing and Release Testing

Regression Testing
Stress, Security, performance, Load Testing etc

20
 Software Testing
¨ A better definition: ”the process of running software with
the intent of finding an error”.

¨ The way most testing works is to input a set of values, then

compare the expected result with the out comes/computed
ones.

¨ Depending on the type of data input, we can identify three

levels of program correctness possible, probable, and
absolute correctness.

21
Three Levels of Correctness
Possible correctness
¨ Obtaining correct output for some arbitrary input (single
set).
¨ If the outcome is wrong, the program cannot possibly be
correct.
¨ For example a multiply program: for input 2 and 3, if
result is 6; is possibly correct.
Three Levels of Correctness
Probable correctness
¨ Obtaining correct output for a number of carefully selected inputs.

¨ If all potential problematic areas are checked in this way,

the program is probably correct.
¨ Try several values, including the obvious “problem” values
such as zero, largest negative, 1, -1 and so on.
23
Three Levels of Correctness
Absolute correctness
¨ can be demonstrated only by a test that involves every
possible combination of inputs.
¨ Requires huge amount of time; it is therefore not practical
{However for some programs we can prove correctness
mathematically}.
¨ Imagine the same above test for a 32 bit machine (how
many combinations are possible? (Hundreds of Millions).
A Small Testing Example
You are to develop a small program that reads three integer values from an input dialog. The
three values represent the lengths of the sides of a triangle.
The program displays a message that states whether the triangle is scalene, isosceles, or
equilateral.
Write a set of test cases—specific sets of data—to properly test the program
◦ The set of test data must be handled by the program correctly to be considered a successful program.

Remember that
◦ a scalene triangle is one where no two sides are equal,
◦ whereas an isosceles triangle has two equal sides, and
◦ an equilateral triangle has three sides of equal length.

25
Definitions

26
int main()
{
float a,b,c;
W H AT T E S T C A S E S D O Y O U T H I N K
cin>>a >> b >>c; SHOULD BE THERE?
if(a<(b+c) && b < (a+c) && c< (a+b)) For the input values to represent a
{ triangle, they must be integers
greater than 0 where the sum of any
cout<<"\nIt is a Triangle.”;
two is greater than the third.
if (a==b && b==c)
cout<<"\nIt is a Equilateral Triangle.”;
if(a==b || a==c || b==c)
cout<<"\nIt is a Isosceles Triangle.”;
else
cout<<"\nIt is a Scalene Triangle.”;
}
else
cout<<"This Triangle is not possible.”;
return 0;
}

27
 Questions to Answer ??
1. Do you have a test case that represents a valid scalene triangle
2. Do you have a test case that represents a valid equilateral triangle?
3. Do you have a test case that represents a valid isosceles triangle?
4. Do you have at least three test cases that represent valid isosceles triangles such that you have tried all three
permutations of two equal sides (such as, 3, 3, 4; 3, 4, 3; and 4, 3, 3)?
5. Do you have a test case in which one side has a zero value?
6. Do you have a test case in which one side has a negative value?
7. Do you have a test case with three integers greater than zero such that the sum of two of the numbers is equal to the
third? (That is, if the program said that 1, 2, 3 represents a scalene triangle, it would contain a bug.)

28
 Questions to Answer ?? (2)
8 Do you have at least three test cases in category 7 such that you have tried all three permutations where the length of
one side is equal to the sum of the lengths of the other two sides (for example, 1, 2, 3; 1, 3, 2; and 3, 1, 2)?
9 Do you have a test case with three integers greater than zero such that the sum of two of the numbers is less than
the third (such as 1, 2, 4 or 12,15,30)?
10 Do you have at least three test cases in category 9 such that you have tried all three permutations (for example, 1, 2,
4; 1, 4, 2; and 4, 1, 2)?
11 Do you have a test case in which all sides are zero (0, 0, 0)?
12 Do you have at least one test case specifying noninteger values (such as 2.5, 3.5, 5.5)?
13 Do you have at least one test case specifying the wrong number of values (two rather than three integers, for
example)?
14 For each test case did you specify the expected output from the program in addition to the input values?

29
Testing the triangle program – Some
thoughts!
A very simple program
To do exhaustive testing
◦ Test with all possible values of a, b, & c (the three sides)
◦ And their combinations
◦ 32-bit integer, 232 possible values for one integer
◦ Assuming only integers from 1 to 10, there are 1012 possible values for a triangle.
◦ Testing 1000 cases per second, you would need 317 years!
◦ Test with all the invalid inputs
◦ e.g., All strings ;)

30
Testing the triangle program – Some
thoughts!
If testing a trivial program is so complex, what about testing large programs of thousands of lines
of code, e.g., air traffic control software
Solution?

31
Testing the triangle program – Some
thoughts!
If testing a trivial program is so complex, what about testing large programs of thousands of lines
of code, e.g., air traffic control software
Solution?
◦ Use of sophisticated testing techniques
◦ Select the test cases that have most chances of triggering a failure
◦ e.g, boundary values

32
 "Bugs lurk in corners
and congregate at
Test case design boundaries ..."
OBJECTIVE ---- to uncover errors Boris Beizer
CRITERIA ----- In a complete manner
CONSTRAINT ------ with a minimum of effort and time

¨ A successful test -- -- that uncovers an as-yet undiscovered error.

¨ Minimum number of required tests with 100% functional coverage and 0%
redundancy.
¨ Rich variety of test case design methods
- Cause-effect graphing, Equivalence Class Partitioning, Boundary Analysis,
and vendor specific: client/server, OO test case design.
Possible approaches:
(1) Knowing the specified functions, tests can be conducted to demonstrate that
each function is fully operational (Black Box).
(2) Knowing the internal workings of a product, tests can be conducted to
ensure that “all gears mesh“ (White Box).

33
White Box
Testing vs
Black Box
Testing

34
 Black-Box Testing

requirements

output

input events

35
 • Types of errors regarding functional requirements of software:
• -- Incorrect or missing functions
• -- Interface errors
• -- Error in data structure & external data base access
Black Box • -- Performance errors

Testing •
•
-- Initialization & termination errors
No functional requirements NO Black Box Testing.
• Demonstrates that each function is fully operational.
• Uncovers different kind of errors than white box testing.
• Performed later in the testing process.

36
¨ Black box techniques derive a set of test cases
that satisfy the following criteria:
(1) Test cases reduce the number of
additional test cases that must be designed to
achieve reasonable testing
(2) Test cases that tell us something about the
presence or absence of classes of errors,
rather than errors associated only with the
specific test at hand.
¨ Black box techniques can supplement the test
cases generated by white box.
37
 Equivalence Boundary
class value
partitioning analysis
How to Design
Test Cases? Cause /
effect
graphing (for Error
combination Guessing
s of input
conditions)

38
Equivalence Partitioning
• Type of Black Box Testing.

• select the right subset-the subset with the highest probability of finding the most
errors.

• The input domain of a program is partitioned into a finite number of equivalence

classes such that we can reasonably assume that a test of a representative value of
each class is equivalent to a test of any other value.

• By identifying this as an equivalence class, we are stating that if no error is found by

a test of one element of the set, it is unlikely that an error would be found by a test of
another element of the set.

39
 Example
Travel service offers discounts to travelers based on their age.
◦ 0-4 years 100%
◦ 5-15 years 50%
◦ 16-64 years 0%
◦ 64 years and older 25%

Equivalence classes for age

◦ 0, 1, 2, 3, 4
◦ 5, 6, 7, …15
◦ 16, 17, 18, …64
◦ 65, 66, 67, …120

Similarly for destinations

◦ e.g., destination can be grouped based on regions (that have same fair)

Nothing special for name

40
Identifying Equivalence classes

 A key concept in the identification of classes is negation, i.e. If a

characteristic is identified as an equivalence class, then one should
immediately negate the characteristic in order to find examples of
classes which should cause the module to do something different such as
“generate an error message”.
 Partitioning each input condition into two or more groups.
 Two types of equivalence classes are identified:
1. Valid Equivalence Classes

2. Invalid Equivalence Classes

41
Identifying the Equivalence Classes
We have valid ECs and invalid Ecs

• Design tests such that each valid EC and each invalid EC are included
in at least one test case
• Test cases are based on the specifications not on code
• Better results are obtained in testing boundary values of ECs as well.
• Very important to test boundaries. Rich source of very common errors.
• More later on this.

42
Identifying the Equivalence Classes
• If an input condition specifies a range of values (for example, “the
item count can be from 1 to 999”), identify one valid equivalence
class (1 < item count < 999) and two invalid equivalence classes (item
count < 1 and item count > 999).

• If an input condition specifies the number of values (for example,

“one through six owners can be listed for the automobile”), identify
one valid equivalence class and two invalid equivalence classes (no
owners and more than six owners).

43
Identifying the Equivalence Classes
If an input condition specifies a set of input values and there is reason to
believe that the program handles each differently (e.g. “type of vehicle
must be BUS, TRUCK, TAXICAB, PASSENGER, or MOTORCYCLE”), identify a
valid equivalence class for each and one invalid equivalence class
(“TRAILER,” for example).

•If an input condition specifies a “must be” situation, such as “first

character of the identifier must be a letter,” identify one valid
equivalence class (it is a letter) and one invalid equivalence class (it is not
a letter)

44
Test Case and the Equivalence
Classes
•Formally, one test case per equivalence class should
be enough.

•A black box method aimed at increasing the

efficiency of testing and, at the same time, improving
coverage of potential error conditions

45
Test Case and the Equivalence
Classes
According to the equivalence class partitioning method:

• Each valid EC and each invalid EC are included in at least one test case.

• Note: equivalence CLASS!

• For an acceptable range of input integers is 1-10, we need at least one test with a value in the range 1-10.
• Boundaries: 1 and 10
• Invalid EC classes might be EC with values < 1, another class > 10, another one - negative numbers, ....
Definition of test cases is done separately for the valid and invalid ECs.

• Note: Run tests for invalid equivalence classes one at a time.

46
 Weak/Strong Equivalence Classes
For an example SUT, suppose there are three input variables from three domains: A, B, C
A = A1 ∪ A2 ∪ A3 ∪ … ∪ Am where ai∈A
B = B1 ∪ B2 ∪ B3 ∪ … ∪ Bn where bi∈B
C = C1 ∪ C2 ∪ C3 ∪ … ∪ Co where ci∈C
Weak Equivalence Class Testing:
◦ Choosing one variable value from each equivalence class (one ai, bi, and ci) such that all classes are covered.
◦ # of test cases?
◦ max (|A|, |B|, |C|)

Strong Equivalence Class Testing:

◦ based on the Cartesian product of the partition subsets (A x B x C),
◦ i.e., testing all interactions of all equivalence classes.
◦ # of test cases?
◦ |A| x |B| x |C|

47
Example of Weak Equivalence Testing
Class (WETC) *
Number of WETCs need= Max number of equivalence classes among {A, B, C}
Given: |A| = 3, |B| = 4, |C| = 2
4 WETCs are enough

48
Example of Strong Equivalence Class
Testing (SECT)
|A| = 3,
|B| = 4,
|C| = 2

# of test cases
◦ 3x4x2= 24

49
Testing for Robustness
If error conditions are a high priority, we should extend strong
equivalence class testing to include both valid (E) and invalid inputs
(U)
◦ For example: age < 0 and age > 120

Again, robustness can be applied with WECT and SECT

50
NextDate Example
NextDate is a function with three variables: month, day, year. It returns the date of the day after
the input date.
◦ Limitation: 1812-2020

Treatment Summary:
◦ if it is not the last day of the month, the next date function will simply increment the day value.
◦ At the end of a month, the next day is 1 and the month is incremented.
◦ At the end of the year, both the day and the month are reset to 1, and the year incremented.
◦ Finally, the problem of leap year makes determining the last day of a month interesting.

51
Equivalence Classes
Valid Equivalence Classes
◦ M1 = {1 <= month <= 12}
◦ D1 = {1 <= day <= 31}
◦ Y1 = {1812 <= year <= 2020}

52
Poor Equivalence Classes
Valid Equivalence Classes
◦ M1 = {1 <= month <= 12}
◦ D1 = {1 <= day <= 31}
◦ Y1 = {1812 <= year <= 2020}

 Should take into account special cases

 Leap year, February

53
Better classes
Valid Equivalence Classes
◦ M1 = {1, 3, 5, 7,8, 10, 12}
◦ M2 = {4, 6, 9, 11}
◦ M3 = {2}
◦ D1 = {1 <= day <= 28}
◦ D2 = {29}
◦ D3 = {30}
◦ D4 = {31}
◦ Y1 = {year = 1900}
◦ Y2 = {1812 <= year <= 2020 AND (year != 1900) AND (year mod 4 = 0)}
◦ Y3 = {1812 <= year <= 2020 AND year mod 4 != 0 }

54
Weak Equivalent Class - Test cases
#test cases=maximum partition size (D)=4

55
Strong Equivalent Class Testing - Test cases

56
57