Lecture 06 Testing

SWENG 580: Advanced Software Engineering
Software Testing
ENGINEERING DIVISION, PENN STATE UNIVERSITY
Objectives and principles Techniques Process Object-oriented testing Test workbenches and frameworks
Penn State University Last updated: Friday, February 17, 2012
Lecture Objectives
Understand
testing objectives and principles Testing techniques black-box and white-box Testing process unit and integration Object-oriented testing Test workbenches and frameworks
Software
Can We Exhaustively Test Software?

A
less than 8 cycles
less than 8 cycles
There are 250 billion unique paths between A and B. If each set of possible data is used, and a single run takes 1 millisecond to execute, it would take 8 years to test all paths.

Can we test all types of software bugs?
Software testing is mainly suitable for dealing with faults that consistently define themselves under well defined conditions

Such faults are called Bohrbugs (an allusion to Niels Bohrs simple and intelligible atomic model) Under seemingly exact conditions, the actions that a test case specifies can sometimes, but not always, lead to a failure Software engineers sometimes refer to faults with this property as Mandelbugs (an allusion to Benoit Mandelbrot, a leading researcher in fractal geometry)
Testers do encounter failures they cant reproduce.

Example: the software fault in the Patriot missile defense system responsible for the Scud incident in Dhahran

To project a targets trajectory, the weapons control computer required its velocity and the time as real values The system, however, kept time internally as an integer, counting tenths of seconds and storing them in a 24 bit register The necessary conversion into a real value caused imprecision in the calculated range where a detected target was expected next For a given velocity of the target, these inaccuracies were proportional to the length of time the system had been continuously running
Testing Objectives
Software testing can show the presence of bugs, but it can never show their absence. Therefore,
Testing
is the process of exercising a program with the specific intent of finding errors prior to delivery to the end user. A good test case is one that has a high probability of finding an error. A successful test is one that uncovers an error.
Testing Principles
All tests should be traceable to customer requirements Tests should be planned long before testing begins
The Pareto principle applies to software testing Testing should begin in the small and progress toward testing in the large Exhaustive testing is not possible To be most effective, testing should be conducted by an independent third party
6
Test Case Design

Testing must be planned and performed systematicallynot ad hoc or random. Testing can be performed in two ways:
1.
2.
Knowing the specified function that a product has been designed to perform black-box testing. Knowing the internal workings of the product and testing to ensure all parts are exercised adequately white-box testing.
7
Black-box Testing
An approach to testing where the program is considered as a black-box The program test cases are based on the system specification Test planning can begin early in the software process
8
Equivalence Partitioning
Divide the input domain into classes of data from which test cases can be derived. Strives to define a test that uncovers classes of errors reducing total number of test cases required.
Example
Specifications for DBMS state that product must handle any number of records between 1 and 16,383 (2 14 1) If system can handle 34 records and 14,870 records, then probably will work fine for 8,252 records, say. If system works for any one test case in range (1..16,383), then it will probably work for any other test case in range Range (1..16,383) constitutes an equivalence class

Any one member is as good a test case as any other member of the class
10
Example
Range (1..16,383) defines three different equivalence classes:

Equivalence
Class 1: Fewer than 1 record Equivalence Class 2: Between 1 and 16,383 records Equivalence Class 3: More than 16,383 records
11
Boundary Value Analysis

Technique that leads to selection of test cases that exercise bounding values. Selecting test case on or just to one side of boundary of equivalence class increases probability of detecting fault.
"Bugs lurk in corners and congregate at boundaries"
12
DBMS Example
Test case 1: 0 records
Member of equivalence class 1 (& adjacent to boundary value) Test case 2: 1 record Boundary value Test case 3: 2 records Adjacent to boundary value Test case 4: 723 records Member of equivalence class 2 Test case 5: 16,382 records Adjacent to boundary value Test case 6: 16,383 records Boundary value Test case 7: 16,384 records Member of equivalence class 3 (& adjacent to boundary value)
13
BVA of Output Specs

Example In 1994, minimum Social Security (FICA) deduction from any one paycheck was $0.00, and the maximum was $3571.20
cases must include input data which should result in deductions of exactly $0.00 and exactly $3571.20 test data that might result in deductions of less than $0.00 or more than $3571.20
14
test
White-box Testing
Test case design method that uses the control structure of the procedural design to derive test cases. Can derive tests that:
Guarantee all
independent paths have been exercised
at least once Exercise all logical decisions on their true and false sides Execute all loops at their boundaries and within operational bounds Exercise internal data structures to ensure validity
15
Basis Path Testing

Proposed by Tom McCabe. Use cyclomatic complexity measure as guide for defining a basis set of execution paths. Test cases derived to exercise the basis set are guaranteed to execute every statement at least once.
16
Independent Paths
CC = 5 So 5 independent paths
1. 2. 3. 4. 5.
a, c, f a, d, c, f a, b, e, f a, b, e, a, a, b, e, b, e,
a b e f
17
Graph Matrices
By forming a graph matrix the cyclomatic complexity and the basis path set can be found automatically.
a b c d e f a 1 1 1 b 1 c 1 1 d e 1 1 1 f
3-1 1-1 1-1 1-1 3-1 = = = = = 2 0 0 0 2
18
CC= 2+2+1=5
The Flowgraph
Before the cyclomatic complexity can be calculated, and the paths determined, the flowgraph must be created. Done by translating the source code into flowgraph notation:
sequence
if
while
until
case
19
Example
PROCEDURE average INTERFACE RETURNS average, total.input, total.valid; INTERFACE ACCEPTS value, minimum, maximum; TYPE value[1:100] IS SCALAR ARRAY; TYPE average, total.input, total.valid; minimum, maximum, sum IS SCALAR; TYPE i IS INTEGER;
i = 1; 2 total.input = total.valid = 0; sum = 0; DO WHILE value[i] <> -999 AND total.input < 100 increment total.input by 1;
3 6
IF value[i] >= minimum AND value[1] <= maximum
5 7 8
THEN increment total.valid by 1; sum = sum + value[i] ELSE skip
ENDIF increment i by 1;
9 ENDDO
IF total.valid > 0
10
12
11 THEN average = sum/total.valid;

ELSE average = -999; END average
13 ENDIF
20
Example
Flowgraph for average
2 3 10
12 13
11
4 5
Determine the: 1. Cyclomatic complexity

8
6 7
2. Independent paths
9
21
Condition Testing
Exercises logical conditions contained within a program module. Types of errors found include;
Boolean
operator error (OR, AND, NOT) Boolean variable error Boolean parenthesis error Relational operator error (>,<,=,!=,) Arithmetic expression error
22
Loop Testing
Focus exclusively on the validity of loop constructs. 4 types of loop can be defined:
Simple Nested Concatenated Unstructured
23
Loop Types
Simple Nested Concatenated
Unstructured
24
Simple Loops
Where n is the max number of passes, the following test can be applied:
loop entirely Only one pass 2 passes m passes (where m<n) n-1, n, n+1 passes
25
Skip
Nested Loops
If the approach for simple loops is extended, number of possible tests would grow geometrically impractical. Instead:
Start at innermost loop. Set all other loops to minimum values. Conduct simple loop test for innermost loop while holding outer loops at minimum loop counter values. Add other test for out-of-range or excluded values. Work outward, conducting tests for next loop, but keeping all other outer lops at minimum values and other nested loops to typical values. Continue until all loops tested.

26
Concatenated Loops
Test as simple loops provided each loop is independent. If two loops are concatenated and loop counter for loop 1 is used as initial value for loop 2, then test as nested loops.
27
Unstructured Loops
Cant test unstructured loops effectively. Reflects very bad practice and should be redesigned.
28
The Tester
Who does the testing?
a) b) c) d)
Developer Member of development team SQA All of the above
29
Independent Test Group

Strictly speaking, testing should be performed by an independent group (SQA or 3rd party) Members of the development team are inclined to be more interested in meeting the rapidlyapproaching due-date. The developer of the code is prone to test gently. Must remember that the objective is to find errors, not to complete test without finding them (because theyre always there!)
30
Successful Testing
The success of testing can be measured by applying a simple metric:
Errors DRE ! Errors Defects
So as defect removal efficiency approaches 1, process approaches perfection.

DRE 1.0 p
3
31
The Testing Process

Unit testing
of individual program components Often performed by the component developer Tests often derived from the developers experience! Increased productivity possible with xUnit framework
Testing
Integration testing
of groups of components integrated to create a system or sub-system The responsibility of an independent testing team Tests are based on a system specification
32
Testing
Testing Phases
Unit testing
Software developer
Integration testing
Development team/ SQA/ Independent Test Group
33
Integration Testing
Tests complete systems or subsystems composed of integrated components Integration testing should be black-box testing with tests derived from the specification Main difficulty is localizing errors Incremental integration testing reduces this problem
34
Incremental Integration Testing

A A
T1
T1 T2 T2 B T3 T3 C T4
T1 A T2 B T3 C T4 D Test sequence 1 Test sequence 2 Test sequence 3 B
T5
35
Approaches to Integration Testing

Top-down testing
Start
with high-level system and integrate from the top-down replacing individual components by stubs where appropriate components in levels until the complete system is created
Bottom-up testing
Integrate individual
In practice, most integration involves a combination of these strategies

36
Top-down Testing
Level 1
Testing sequence
Level 1
...
Level 2 Level 2 stubs
Level 2
Level 2
Level 2
Level 3 stubs
37
Bottom-up Testing
Test drivers
Level N
Level N
Level N
Level N
Level N
Testing sequence
Test drivers
Level N1
Level N1
Level N1
38
Which is Best?
In bottom-up testing:
Test
harnesses must be constructed and this takes
time. Integration errors are found later rather than earlier. Systems-level design flaws that could require major reconstruction are found last. There is no visible, working system until the last stage so is harder to demonstrate progress to clients.
39
Interface Testing
Takes place when modules or sub-systems are integrated to create larger systems Objectives are to detect faults due to interface errors or invalid assumptions about interfaces Particularly important for object-oriented development as objects are defined by their interfaces
40
Interface Testing
Test cases
C
41
Interfaces Types
Parameter interfaces
Data
passed from one procedure to another of memory is shared between procedures
Shared memory interfaces

Block
Procedural interfaces
Sub-system
encapsulates a set of procedures to be called by other sub-systems request services from other sub42
Message passing interfaces

Sub-systems
systems
Interface Errors
Interface misuse

A calling component calls another component and makes an error in its use of its interface e.g. parameters in the wrong order A calling component embeds assumptions about the behaviour of the called component which are incorrect The called and the calling component operate at different speeds and out-of-date information is accessed
Interface misunderstanding

Timing errors

43
Interface Testing Guidelines

Design tests so that parameters to a called procedure are at the extreme ends of their ranges Always test pointer parameters with null pointers Use stress testing in message passing systems In shared memory systems, vary the order in which components are activated Design tests which cause the component to fail
44
Stress Testing
Exercises the system beyond its maximum design load.
Stressing
the system often causes defects to come to
light
Stressing the system test failure behaviour.

Systems
should not fail catastrophically. Stress testing checks for unacceptable loss of service or data
Particularly relevant to distributed systems which can exhibit severe degradation as a network becomes overloaded
45
Object-Oriented Testing
The components to be tested are object classes that are instantiated as objects Larger grain than individual functions so approaches to white-box testing have to be extended No obvious top to the system for top-down integration and testing
46
Testing Levels
Test object classes Test clusters of cooperating objects Test the complete OO system
47
Object Class Testing

Complete test coverage of a class involves
Testing
all operations associated with an object Setting and interrogating all object attributes Exercising the object in all possible states
Inheritance makes it more difficult to design object class tests as the information to be tested is not localized
48
Object Integration
Levels of integration are less distinct in objectoriented systems Cluster testing is concerned with integrating and testing clusters of cooperating objects Identify clusters using knowledge of the operation of objects and the system features that are implemented by these clusters
49
Approaches to Cluster Testing

Use-case or scenario testing

Testing is based on a user interactions with the system Has the advantage that it tests system features as experienced by users

Thread testing
A thread consists of all the classes needed to respond to a single external input. Each class is unit tested, and then the thread set is exercised. Tests sequences of object interactions that stop when an object operation does not call on services from another object
Object interaction testing

Uses-based testing

Begins by testing classes that use few or no server classes. Next, classes that use the first group of classes are tested, followed by classes that use the second group, and so on.
50
Scenario-Based Testing
Identify scenarios from use-cases and supplement these with interaction diagrams that show the objects involved in the scenario Consider the scenario in the weather station system where a report is generated
51
Collect Weather Data

:CommsController request (report)
:WeatherStation
:WeatherData
acknowledge () report () summarise ()
send (report) reply (report) acknowledge ()
52
Weather Station Testing

Thread of methods executed

CommsController:request pWeatherStation:report p WeatherData:summarize
Inputs and outputs

Input Can Use
of report request with associated acknowledge and a final output of a report be tested by creating raw data and ensuring that it is summarized properly the same raw data to test the WeatherData object
53
OO Testing: Myths & Reality

Inheritance means never having to say your sorry Reuse means never having to say your sorry Black box testing is sufficient
54
Implications of Inheritance
Myth:
specializing
from tested superclasses means subclasses will be correct create new ways to misuse inherited features
Different test
Reality:
Subclasses
cases needed for each context Need to retest inherited methods, even if unchanged.
55
Implications of Reuse
Myth:
Reusing
a tested class means that the behavior of the server object is trustworthy new usage provides ways to misuse a server.
if many server object of a given class function correctly, nothing is to prevent a new client class from using it incorrectly we can't automatically trust a server because it performs correctly for one client
Even
Reality:
Every
56
Implication of Encapsulation
Myth:
White-box
testing violates encapsulation, surely black-box testing (of class interfaces) is sufficient.
Reality:
Studies
indicate that thorough BBT sometimes exercises only 1/3 of code. BBT exercises all specified behaviors, what about unspecified behaviors?!
Need
to examine implementation.
57
And What About Polymorphism?

Each possible binding of a polymorphic component requires separate testprobably separate test case!
58
Testing Workbenches
Testing is an expensive process phase. Testing workbenches provide a range of tools to reduce the time required and total testing costs Most testing workbenches are open systems because testing needs are organization-specific Difficult to integrate with closed design and analysis workbenches
59

A Testing Workbench
Test data generator Specification
Source code
Test manager
Test data
Oracle
Dynamic analyser
Program being tested
Test results
Test predictions
Execution report
Simulator
File comparator
Report generator
Test results report

60
Workbench Components
Test manager: manages the running of program tests Test data generator: selects test data from database or uses patterns to generate random data of correct form Oracle: Predicts expected results (may be previous version/prototype) Comparator: compare results of oracle and program, or program and previous version (regression test) Dynamic analyzer: counts number of times each statement is executed during test. Simulator: simulates environment (target platform, user interaction, etc)
61
xUnit Framework
Developed by Kent Beck Makes object-oriented unit testing more accessible. Freeware versions available for most objectoriented languages
www.xprogramming.com/software.htm
62
jUnit successful
63
jUnit unsuccessful
64
Simple Guide to Using xUnit

Subclass TestCase class for the object under test
Ensure test class has scope over object under test.
Add a test method to the test class for each method.

An xUnit test method is an ordinary method without parameters.
Code the test case in the test method

Creates objects necessary for the test (fixture) Exercises objects in the fixture Verifies the result.
(1) (2) (3)

65
Example Money class

(from Test Infected)
class Money { private int fAmount; private String fCurrency; public Money(int amount, String currency) { fAmount= amount; fCurrency= currency; } public int amount() { return fAmount; } public class MoneyTest extends TestCase { // public void testSimpleAdd() { Money m12CHF= new Money(12, "CHF"); // (1) Money m14CHF= new Money(14, "CHF"); Money expected= new Money(26, "CHF"); Money result= m12CHF.add(m14CHF); // (2) Assert.assertTrue(expected.equals(result)); // (3) } } Requires a way to test that two Money objects are equal (override equals in Object) but first must write test for equals in MoneyTest!
public String currency() { return fCurrency; } public Money add(Money m) { return new Money(amount()+m.amount(), currency()); } }
66
Key Points
Exhaustive testing is not possible Testing must be done systematically using black-box and white-box testing techniques Testing must be done at both unit and integration levels Object-oriented programming offers its own challenges for testing Testing workbenches and frameworks can help with the testing process
67
References
M. Grottke and K.S. Trivedi. Fighting Bugs: Remove, Retry, Replicate and Rejuvinate. IEEE Computer, February 2007, pp. 107 109. R. Pressman. Software Engineering: A Practitioners Approach, New York, NY: McGraw-Hill, 6th Ed, 2004. I. Sommerville. Software Engineering, 6th Ed. New York, NY: Addison-Wesley, 2000.
68

Lecture 06 Testing

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture 06 Testing

Uploaded by

Copyright:

Available Formats

SWENG 580: Advanced Software Engineering

ENGINEERING DIVISION, PENN STATE UNIVERSITY

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

Can We Exhaustively Test Software?

ENGINEERING DIVISION, PENN STATE UNIVERSITY

less than 8 cycles

SWENG 580: Advanced Software Engineering

ENGINEERING DIVISION, PENN STATE UNIVERSITY

Testers do encounter failures they cant reproduce.

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

Test Case Design

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

Range (1..16,383) defines three different equivalence classes:

SWENG 580: Advanced Software Engineering

Boundary Value Analysis

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

BVA of Output Specs

SWENG 580: Advanced Software Engineering

independent paths have been exercised

SWENG 580: Advanced Software Engineering

Basis Path Testing

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

ENGINEERING DIVISION, PENN STATE UNIVERSITY

IF value[i] >= minimum AND value[1] <= maximum

THEN increment total.valid by 1; sum = sum + value[i] ELSE skip

11 THEN average = sum/total.valid;

SWENG 580: Advanced Software Engineering

Determine the: 1. Cyclomatic complexity

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

Simple Nested Concatenated

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

Developer Member of development team SQA All of the above

SWENG 580: Advanced Software Engineering

Independent Test Group

SWENG 580: Advanced Software Engineering

Errors DRE ! Errors  Defects

So as defect removal efficiency approaches 1, process approaches perfection.

SWENG 580: Advanced Software Engineering

The Testing Process

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

SWENG 580: Advanced Software Engineering

Incremental Integration Testing

T1 A T2 B T3 C T4 D Test sequence 1 Test sequence 2 Test sequence 3 B

SWENG 580: Advanced Software Engineering

Approaches to Integration Testing

In practice, most integration involves a combination of these strategies

Errors DRE ! Errors Defects