Professional Documents
Culture Documents
2
References
1. Galin, D. Software Quality Assurance: From theory to
implementation, Pearson, 2004
3
Objectives
At the end of this lecture, students should
be able to:
1. Discuss the differences between various
software testing strategies
2. Describe the concept of black box and
white box testing
3. Define appropriate and applicable
software testing techniques that are
geared to discover program faults
4. Apply testing technique in order to
implement it in the software testing
process
5. Describe the various types of testing
level
4
:: T O P I C ::
1. Testing Strategies
1.1 Big Bang Testing
1.2 Incremental Testing
a. Top down
b. Bottom up
1.3 Stub vs Driver
2. Software Test Classification
2.1 Black Box Testing
2.2 White Box Testing
3. Testing Level
3.1 Unit Testing
3.2 Integration Testing
3.3 System Testing
3.4 Acceptance Testing
5
Testing Strategies
Two types of testing strategies:
1. “Big-bang testing”
- tests the sw as a whole, once the
completed package is available.
2. “Incremental testing”
- tests the sw piecemeal- sw modules are
tested as they completed (unit tests),
follows by group or modules composed of
tested modules integrated with newly
completed modules (integration tests).
Once the entire package is completed, it is
tested as a whole (system test).
6
Testing Strategies
- Big Bang Testing
Put all the modules together at the same
time and test
great with small systems
Disadvantages
1. requires both stubs and drivers to test the
independent components
2. when failures occurs, it is very difficult to
locate the faults. After the modification, we
have to go through the testing, locating
faults, modifying faults again
7
Big-bang testing -example
b c d
e f g
h i j k
l m
2. Bottom up
- the order of testing is reversed; the
lowest level modules are tested first, with
the main module tested last.
9
Incremental Testing
- Top down
Start with the top most module. Integrate
it with the modules it uses. Test and
Repeat.
Advantage
– can verify major control/fault or decision points
early in the testing process
Disadvantage
– stubs are required when perform the integration
testing, and generally, develop stubs is very
difficult
– Potential reusable modules may not be
adequately tested.
1
Top-down -example
a
b c d
e f g
h i j k
1
Bottom-up -example
b c d
e f g
h i j k
l m
1
Driver vs Stub
Driver
– “main program” that accepts the test case data, passes such
data to the module to be tested, and print the relevant results
– sw modules used to invoke a module(s) under test and often
provide test inputs, control and monitor execution and report
results (IEEE, 1990)
Stub
– serves to replace modules that are subordinates to the module
to be tested. A stub or “dummy subprogram” uses the
subordinate modules interface, may do minimal data
manipulation, prints the verification of entry and then returns
to the caller
– a test is based on the execution of the subprogram
– this subprogram may itself have to use other subprograms in
simulated ways, called stubs
b c d g
e f
j k
h i
l m
driver results
Test Module to
cases be tested
1
Test Environment Driver
1. to provide/ initialize the input values for
the calling of subprograms to be tested
2. To check the values obtained as outputs
of procedure calls
3. to control the information used or updated
by the stubs
eg; call the routine with fixed sets of
inputs, prompts for input from the user
and call appropriate routine with that
input, read arguments from file and call
routine
** In general, the test driver’s role can be fulfilled by a test tool
1
Test Environment Stub
the stub must simulate in a simple
manner the behavior of the subprogram
used
the most basic way of writing a stub is
would be to arrange to return a constant
values for the output data
eg; test (validate) the input data, print
message “I am here” with parameters,
get return values from interactive input,
read data from file and return
1
:: T O P I C ::
1. Testing Strategies
1.1 Big Bang Testing
1.2 Incremental Testing
a. Top down
b. Bottom up
1.3 Stub vs Driver
2. Software Test Classification
2.1 Black Box Testing
2.2 White Box Testing
3. Testing Level
3.1 Unit Testing
3.2 Integration Testing
3.3 System Testing
3.4 Acceptance Testing
1
Software Test Classification
?
?
?
?
2
Black Box Testing Objectives
to demonstrate that software functions
are operational, input is properly
accepted and output is correctly produced
without taking into account of knowledge
of the internal structure of the program
Functional testing attempts to find errors in
the following categories
1. incorrect or missing functions
2. interface errors
3. errors in data structure or external database
access
4. performance errors
5. initialization and termination errors
2
Techniques of Black Box
Testing
1. equivalence partitioning
2. self assessment test
(error guessing)
3. boundary value analysis
(limit testing)
4. robustness
2
1. Black box Testing
by equivalence partitioning
divide the input domain into classes of data for which test
cases can be generated
Goal: to reduce the set of possible test cases and avoid
redundancy.
based on equivalence classes for input conditions
an equivalence class represents a set of valid and invalid
states
an input condition is either a specific numeric value, range of
values, a set of related values or a Boolean condition
The idea is to identify test cases by using one element from
each equivalent class.
If the classes are chosen wisely,it will reduce potential
redundancy in test cases.(Jorgenson, 1995)
2
1. Black Box Testing
by equivalence partitioning
Specification Valid classes Invalid classes
Integer range 1…10 between 1 to 10 <1; >10; non-integers
Not less than 2 and not 2 to 10 lower case <2 chars; >10 chars;
more than 10 lower case chars Capital letters
letters
2
Exercise
Specification Valid classes Invalid classes
if x > 15 then
(16 until ) (< 16 until -)
do this
else do that
Testing problem:
1. There is limit of $70 for 1. Claims 70 1. Claims >70
accommodation claims
2. Any claims above $70 should
be rejected and cause an error
msg to be displayed
2. Claims >70 2. Claims 70
Error displayed
3. All expense amounts should be
greater than $0 ; if not error
msg should be displayed .
3. amounts 0 3. amounts < 0
Error displayed
2
What’s next?
So from the valid and invalid classes identify valid test
cases according to the value.
Testing problem: Valid Test case Invalid Test case
partition partition
1.There
is limit of $70 for 20 90
accommodation claims
Claims 70 Claims >70
80 (error 50
2.Any claims above $70 Claims >70 displayed
Claims 70
should be rejected and Error ?)
cause an error msg to be
displayed
displayed
3.Allexpense amounts
should be greater than
amounts >0 10 amounts<= 0 -25 (error
$0 ; if not error msg Error displayed
should be displayed . displayed ?)
2
2. Black box Testing by
self-assessment test (error guessing)
A test case design technique where the experience
of the tester is used to postulate what faults might
be present in a the system to be tested.(Watkins)
Problem:
The program read three integer values from a file.
The three values are interpreted as representing
the lengths of the sides of triangle. The program
prints a message that states whether the triangle is
scalene(3 sides unequal), isosceles(2 sides equal)
or equilateral(equal sides)
Test case 2
3,3,1 isosceles
2
3. Black Box Testing
by boundary value analysis
boundary boundary
values values
3
Exercise
Specification Partition tested Possible Test Case
Testing problem:
1. There is limit of $70 for
accommodation claims 1. Claims 70 1. 70, 71
2. Any claims above $70 should
be rejected and cause an error
msg to be displayed
2. Claims >70 2. 69 ,70, 71
3. All expense amounts should be
Error displayed
greater than $0 ; if not error
msg should be displayed .
3. amounts 0 3. -1, 0, 1
Error displayed
0 70
- + - +
boundary boundary
3 values values
4. Black Box Testing
robustness
to prove that no catastrophic event can be produced by
the introduction of an abnormal input data item
the values are taken at the outer bounds of the
equivalence classes
examples;
- hit the ESC key
- insert the different types of data
- use the values outside the limit
3
Fastest Train
A test model of the Fastech, a new
bullet train that will run at 360 kph
in 2011 on the Tohoku Shinkansen
Line, is unveiled Friday, with
retractable, cat ear-shaped, air
brakes deployed, at JR East's test
center here
3
3
1st
The first level test subset is generated to test all the normal
interlocking functions which the relevant technical standards
prescribe.
The test cases used to test the ability that all the parallel routes
can be established at the same time.
3
2nd
The second-level test subset is the test subset used to test all the
interlocking safety protection functions which the relevant technical
standards definitely prescribe.
The test cases used to test single operation functions of any switch in
any lock state (single lock, closed up lock, calling-on lock and route
lock).
The test cases used to test the function that all the conflicted routes can
be prevented to be established simultaneously under normal conditions.
The test cases used to test the safety protection functions of clearing
and closing any signal under some abnormal conditions, namely
malfunctions of signals, track sections or switches.
The test cases used to test the functions of locking and releasing any
track section under some abnormal conditions (at most an erroneous
operation or a fault in signaling equipments).
3
3rd
The third-level test subset is used to test interlocking software’s ability
to fulfill the Fail-Safe function under whatever complicated conditions. It
can be generated by an expert test set generation system based on a
testing knowledge base.
The test cases used to test single operation functions of any switch
under any complicated condition, which may lead to risky results.
The test cases used to test the function that all the conflicted routes can
be prevented to be established simultaneously under any complicated
condition.
The test cases used to test the Fail-Safe function while clearing or
closing any signal under any complicated condition, which may lead to
risky results.
The test cases used to test the Fail-Safe function while locking and
releasing any track section under any complicated condition, which may
lead to risky results.
3
4th
The fourth-level test subset is designed to test some
complex interlocking control functions which are not
clearly prescribed in the relevant technical standards.
3
Testing Results
3
White Box Testing
Objectives
to examine the internal structure of the
subprogram. Logical paths through the software are
tested by providing test cases that exercise specific
sets of condition and/or loops.
using white box methods, the software engineer
can derive the test cases that:
1. Guarantee that all independent paths within
modules have been exercised at least once.
2. Exercise all logical decisions on their true or false
sides.
3. Execute all loops at their boundaries and within
their operational bounds.
4. Exercise the internal data structures to assure their
validity
4
Techniques of White Box
Testing
4
1. Basic Control Structure
Method for obtaining complete test code coverage.
Pure white-box testing
Test cases are derived from the code logic and are
independent of the functional specification (Hetzel, 1988)
Goal: to calculate how many independent paths exist in
the graph.
The no of independent path indicates how many test
cases you need as a basis.
4
White Box Testing
1. Basic Control Structure
1 2
1 1 3
2
4
sequence if
2 3 4 5 6 7
1 1
8
2 case (switch) 2
3 3
do-while while
4
Basic Control Structure
(AND(&&) vs OR(||)
F
a a
F
T T
b b
F
T T F
y x x y
A B (a||b) A B (a&&b)
if ( a || b ) if ( a && b )
T T T T T T
x( ); x( );
else T F T else T F F
y( ); F T T y( ); F T F
4 F F F F F F
Exercise 1
1. Draw a flow graph for the following
simple code:
1. Read A, B
2. If A>0 THEN PERFORM COUNTA
3. If B>0 THEN PERFORM COUNTB
4
Exercise : Answer
1. Draw a flow graph for the following 1
simple code:
2
1. Read A, B T
2. If A>0 THEN PERFORM COUNTA COUNTA
3. If B>0 THEN PERFORM COUNTB
3
T
COUNTB
4
Flow Graph(Cyclomatic Complexity)
edges
regions
nodes
On a flow graph:
Arrows called edges represent flow of control
Circles called nodes represent one or more actions
Areas bounded by edges and nodes called regions
A predicate node is a node containing a condition
4
Basic Control Structure-flow
graph 1
Procedure Sort
1) while records remain do
read record
2) if record.field1 = 0
F 2 T
3) process record;
4 3
store record;
increment counter;
F T
4) else if record.field2 = 0;
6 5
5) reset counter;
6) else
process record; 7
7) end if 8
end if
8) end do 9
9) end
On a flow graph:
• Arrows called edges represent flow of control
• Circles called nodes represent one or more
actions
• Areas bounded by edges and nodes called
4 regions
White Box Testing
Cyclomatic Complexity
the cyclomatic complexity gives a quantitative measure of the
logical complexity
gives an indication of the minimum number of independence
path to be tested
- independence path is any path through a program that
introduces at least one new set of processing statement or a
new condition
provides upper bound for number of tests required to
guarantee coverage of all program statements.
4
White Box Testing
Independence of control path
the example has 4 independent
paths: 1
path 1 = 1, 9
path 2 = 1, 2, 3, 8, 1, 9 F 2 T
path 3 = 1, 2, 4, 5, 7, 8, 1, 9 4 3
path 4 = 1, 2, 4, 6, 7, 8, 1, 9 F T
6 5
How to count the no. of paths??
7
1. C=#Edges - #Nodes + 2= 11 –9 8
+2=4
2. C=#of regions of flow graph + 1 9
=3+1=4
• Arrows called edges represent flow of control
3. C=#Predicates nodes + 1 • Circles called nodes represent one or more
actions
=3+1=4 • Areas bounded by edges and nodes called
regions
5 complexity
• A predicate node is a node containing a
condition
Exercise 2
Determine complexity 1
calculation
1. No of region? 2
2. No. of edges?
3. No of nodes? 3
4. Basis testing method
Formula 1: 4
C=# regions +1
Formula 2 : 5
C=# edges-#nodes + 2
6
5. Possible independent path?
7
5
Exercise: Answer
Determine complexity calculation
1. No of region?3 1
2. No. of edges?9
3. No of nodes?7 2
4. Basis testing method
Formula 1:
C=# regions +1 3
C=3+1=4
Formula 2 : 4
C=# edges-#nodes + 2
C = 9-7+2=4 5
6
5. Possible independent path?
1. 1,2,4,5,6,7
2. 1,2,3,2,4,5,6,7 7
3. 1,2,4,7
5 4. 1,2,3,2,4,7
Exercise 3
What will this code do?
Draw flow graph, calculate complexity
1. right = 0;
2. DO J=1 UNTIL noOfQuestion;
3. IF key(J)= answer(J)
4. THEN right=right+1;
5. END
6. score =
(right/noOfQuestion)*100;
7. IF score 90
8. THEN grade=A;
9. ELSEIF score 80
10. THEN grade=B;
11. ELSEIF score 70
12. THEN grade=C;
13. ELSE GRADE = F;
14. END IF
5
Exercise 3 : Answer
What will this code do?
Draw flow graph, calculate complexity 1
1. right = 0;
2. DO J=1 UNTIL noOfQuestion; 2
3. IF key(J)= answer(J) 6
4. THEN right=right+1;
5. END 3 7
6. score =
(right/noOfQuestion)*100; 4
8 9
7. IF score 90
8. THEN grade=A; 10
5 11
9. ELSEIF score 80
10. THEN grade=B;
11. ELSEIF score 70 12 13
12. THEN grade=C;
13. ELSE GRADE = F; 14
14. END IF
5
Exercise 3 : Answer
What will this code do?
Draw flow graph, calculate complexity 1
Determine complexity calculation
1. No of region?5 2
6
2. No. of edges?18
3. No of nodes?14 3 7
4. Basis testing method
Formula 1: 4
C=# regions +1 8 9
C=5+1=6
5 10 11
Formula 2 :
C=# edges-#nodes + 2
C = 18-14+2=6 12 13
14
5
Steps in Deriving Test Cases
1. Using the design or code, draw the
corresponding flow graph
2. Determine the complexity of the flow
graph
3. Determine the basis set of independent
paths
4. Prepare test cases that will force execution
of each path in the basis set
5
:: T O P I C ::
1. Testing Strategies
1.1 Big Bang Testing
1.2 Incremental Testing
a. Top down
b. Bottom up
1.3 Stub vs Driver
2. Software Test Classification
2.1 Black Box Testing
2.2 White Box Testing
3. Testing Level
3.1 Unit Testing
3.2 Integration Testing
3.3 System Testing
3.4 Acceptance Testing
5
Unit Testing
test each module in isolation
done by individual programmers
must be done for new software
this is the time you examine your code to test
the software
code is the source of test cases
because a component is not a stand alone
program, driver and/or stub software must be
developed for each unit test
5
Integration Testing
Objective : To test program interfaces and confirm
that programs have been linked together correctly
(Hetzel, 1988).
Testing of groups of components (unit) integrated
to construct a system or subsystem
Major problems in integration testing:
– Integrating modules that have not been properly unit
tested.
– Failing to treat integration testing formally enough. Need
to be planned and designed properly, not just randomized
testing.
Uncovers errors associated with interfacing. Eg:
– wrong call (external interface),
– data integrity (information content),
– number, types and order arguments in a call (internal
interfaces integrity)
Refer Schach, 1998
5
:: SUB T O P I C ::
1. Unit testing
2. Integration Testing
3. System testing
3.1 Recovery
3.2 Security
3.3 Stress
3.4 Performance
4. Acceptance testing
6
System Testing
Systems test execution starts when integration testing has been
completed
Ends when we successfully determined what the system
capabilities and corrected known problems and the system is
ready for acceptance testing.
now, software is only one component of the system
main objective – to ensure that the system does what the
customers want to do
System testing steps:
6
2. System Testing Security
Security is a protection system that is needed for both
secure confidential information and for
competitiveness purposes to assure 3rd parties that
their data will be protected
During security testing, the testers plays the role of the
individual trying to penetrate the system.
eg. methods:
- attempt to acquire passwords through external clerical
means
- use custom software to attack the system
- overwhelm the system with request
- cause system errors and attempt to penetrate the
system during the recovery
- browse through insecure data
6
3. System Testing Stress
Stress testing is designed to determine if the
system can function when subject to large
volumes – larger than normally excepted
The areas that are stressed include :
– input transactions,
– disk space,
– output,
– communications,
– computer capacity and
– interaction with users
It attempts to find the limits at which the system
will fail through abnormal quantity or frequency of
inputs
6
4. System Testing Performance
To test whether the system produce the results
within acceptable time intervals
eg; real-time software is characterized by hard
time constraints; if the time constraint is not met,
information will be lost.
Eg: lecturer must be able to upload notes with
less than 1MB of size within 25 secs.
6
:: SUB T O P I C ::
1. Unit testing
2. Integration Testing
3. System testing
4. Acceptance testing
4.1 Parallel run test
4.2 Pilot test
4.3 Installation test
4.4 Documentation test
6
Acceptance Testing
The purpose of acceptance testing:
– is for the client to determine whether the product indeed satisfies
its specifications as claimed by the developer
– to confirm that a system is ready for operational use
Demonstrate that a system is ready for operational use.
Selection of tests to include in this test should be made by the
user, with expert professional help from test planners.
Focus on major functional and performance reqs.
Done by:
- client organization
- SQA group in the presence of client representatives
- SQA group hired by the client
6
Acceptance Testing
Key aspects:
– must be performed on actual data, rather than
test data
– perhaps using the actual hardware, software
and user environment that is planned for
production
When testing a replacement system, the
parallel run is a commonly used strategy
6
1. Acceptance Testing:
A Parallel Run Test
Same input
Old System
New System
(operating in
(as a test run)
production)
Output Output
?
match
NO YES
Analyze reason Install the new system
Fix deficiency into production
Rerun the test
7
3. Installation Testing
A testing technique used to validate that the system
under testing is able to run correctly on variety of
delivery platforms of various specifications
(hardware, OS and sw characteristics)(Watkins,
2001)
Final stage of testing
May not be needed if acceptance testing was
performed at the customer’s site
The system is installed in the environment in which
it will be used, and we verify that it works
7
4. Documentation Tests
Tests must be carefully documented. For each
test, you must record:
– what part of the analysis or design is being tested?
– how the test is to be run?
– What is the expected outcome ?
Including user documentation and help system
conforms to software reqs
The mechanism for reporting test results should
also be made clear
This will usually be part of the quality plan for the
project
Techniques review, cross-referencing checks
7
:: CONCLUSION ::
1. Testing Strategies
1.1 Big Bang Testing
1.2 Incremental Testing
a. Top down
b. Bottom up
1.3 Stub vs Driver
2. Software Test Classification
2.1 Black Box Testing
2.2 White Box Testing
3. Testing Level
3.1 Unit Testing
3.2 Integration Testing
3.3 System Testing
3.4 Acceptance Testing