CHAPTER 17

SOFTWARE TESTING STRATEGIES

1

TOPICS 
A

strategic approach to software testing  Unit Testing  Integration Testing  Validation Testing  System Testing  The ART of Debugging  Summary
2 SRIMCA

April 2004

STRATEGIC APPROACH TO SOFTWARE TESTING
Generic characteristics of software testing strategies: Testing

begins at module level and works outward towards the of integration entire computer based system.  Different testing techniques are required at different points in time.  Testing is conducted by the s/w developer and ITG ( Independent Test Group ) for large projects.  Testing and Debugging are different and Debugging is essential in any testing strategy.
3 April 2004 SRIMCA

Verification and Validation

-- Does the product meet its specifications?  Validation -- Does the product perform as desired?
April 2004 SRIMCA 

Verification

4

Software Testing Strategy
A Software Testing Strategy

5 April 2004 SRIMCA

Software Testing Strategy

6 April 2004 SRIMCA

Software Error Model
= cumulative remaining errors at time t  l0 = initial failure rate  p = exponential reduction as errors repaired f(t) = (1/p)ln(l0pt + 1) 
f(t)

7 April 2004 SRIMCA

STRATEGIC APPROACH 
Issues

to be addressed to develop a successful software testing strategy: 
Specify product

requirements in a quantifiable manner long before testing commences.  State testing objectives explicitly.  Understand the users of the software.  Develop testing plan that emphasizes ³rapid cycle testing.´
8 April 2004 SRIMCA

STRATEGIC APPROACH 
Issues

to be addressed to develop a successful software testing strategy: 
Build robust

software that is designed to test

itself.  Use effective formal technical reviews as a filter to testing.  Conduct formal technical reviews to assess test strategy and test cases.  Develop continuous improvement approach
9 April 2004 SRIMCA

UNIT TESTING 
Unit

testing -- focuses on the smallest element of software design viz. the module. 
Corresponds to

class testing in the OO context. 

Makes

heavy use of white-box testing.

10 April 2004 SRIMCA

UNIT TESTING
Unit Test Generation Considerations:
Review Design information - develop unit test cases. interface local data structures boundary conditions independent paths error handling paths

driver Module to be tested stub
April 2004

stub
SRIMCA

Test cases
11

Unit Test Generation 
Interface 
#

considerations

of input parameters = # arguments?  Parameter and argument attributes match?  Parameter and argument units match?  Order correct (if important)?  Number and order of arguments for built-ins?  References to parms not associated with entry point?  Attempt to modify input-only arguments?  Global variable definitions consistent?  Constraints passed as arguments?
12 April 2004 SRIMCA

Unit Test Generation 
External 
Files

I/O considerations

attributes correct?  OPEN/CLOSE correct? Format specification matches I/O statement?  Buffer size matches record size?  Files opened before use?  EOF handled correctly? I/O errors handled?  Textual errors in output?
13 April 2004 SRIMCA

Unit Test Generation 
Data

structure considerations 

Improper

or inconsistent typing?  Erroneous initialization or default values?  Incorrect variable names?  Inconsistent data types?  Underflow, overflow and addressing exceptions?

14 April 2004 SRIMCA

Unit Test Generation
cases must cover all execution paths  Common computational errors to be checked: 
incorrect arithmetic  mixed 

Test

mode operations  incorrect initialization  precision inaccuracy  incorrect symbolic representation of expression 
Other

tests needed 

incompatible data

types in comparisons  incorrect logical operators or precedence  comparison problems (e.g., == on floats)  loop problems
April 2004 SRIMCA

15

Unit Test Generation 
Error

handling tests 

Exception-handling is

incorrect?  Error description is unintelligible, insufficient or incorrect?  Error condition causes system interrupt before error handling completed?

16 April 2004 SRIMCA

INTEGRATION TESTING
systematic approach for constructing program structure while conducting tests to uncover errors associated with interfacing.  Tendency for Non-Incremental integration.. Big Bang approach «. Chaos !! ( usually ).  Incremental integration - program is constructed and tested in small segments. 
Top-Down Integration testing  Bottom-Up Integration testing
17 April 2004 SRIMCA 

A

INTEGRATION TESTING

18 April 2004 SRIMCA

INTEGRATION TESTING
Top-Down Approach  Begin construction and testing with main module. 
Stubs

are substituted for all subordinate modules. 

Subordinate

stubs are replaced one at a time by actual modules.  Tests are conducted as each module is integrated.  On completion of each set of tests, another stub is replaced with the real module.  Regression testing may be conducted to ensure that new errors have not been introduced.
19 April 2004 SRIMCA

Top Down Approach - Use Stubs

20 April 2004 SRIMCA

INTEGRATION TESTING
Top-Down Approach : 
Advantages: 
Verifies

major control or decision points early in the test process.  With the use of depth-first integration testing, a complete function of the software can be demonstrated. -- Confidence builder for developer/customer. 
Disadvantages: 
Since

stubs replace lower level modules, no significant data can flow upwards to the main module.
SRIMCA

21

April 2004

INTEGRATION TESTING
Bottom Up Approach : 
This

approach begins construction and testing with modules at the lowest levels in the program structure. 
Low-level

modules are combined into clusters.  A driver is written to coordinate test case input and output.  The cluster is tested.  Drivers are removed and clusters are combined moving upward in the program hierarchy.
22 April 2004 SRIMCA

Bottom Up Approach

23 April 2004 SRIMCA

INTEGRATION TESTING
Bottom Up Approach 
Advantages: 
Easier

test case design and lack of stubs. 

Disadvantages: 
The

program as an entity is does not exist until the last module is added.

Sandwich Testing:- combined approach 
Top

down strategy for upper levels and Bottom up strategy for subordinate levels.
24 SRIMCA

April 2004

INTEGRATION TESTING 
Regression

Testing

some subset of tests already conducted to ensure that the new changes do not have unintended side effects.  The Regression test suite should contain three different classes of test cases :  A representative sample of tests that will exercise all software functions  Additional tests that focus on functions that are likely to be affected by the change.  Tests that focus on software components that have changed. 25
April 2004 SRIMCA 

Re-execution of

INTEGRATION TESTING
Integration Test Documentation
1
Scope of testing

2
Test plan

3
Test Procedure n

4
Actual Test Results

5
Ref. & Appendix

Environment Test Schedule Unit Test / Resources phases test case and Overhead data Test builds software environment Order of Integration
April 2004 SRIMCA

Expected Results for build n

26

VALIDATION TESTING 
It

provides final assurance that software meets all functional, behavioral, and performance requirements.

-- Exclusive use of Black-box testing techniques.  After each validation test case either software conforms to specs or a deviation from specs is detected and a deficiency list needs to be worked. 

Alpha and Beta testing 
Alpha test

-- At developer¶s site by customer.  Beta test -- At customer¶s site in a ³live´ environment. 27
April 2004 SRIMCA

SYSTEM TESTING 
A

series of tests to verify that all system elements have been properly integrated.
Testing: 
Forces 

Recovery

software to fail in a variety of ways and verifies that recovery is properly performed. 

Security

Testing: 

Attempts to

verify the software¶s protection mechanisms. The software designer tries to make penetration cost more than the value of information obtained by breaking in. 28
SRIMCA

April 2004

SYSTEM TESTING 
Stress

Testing: 

Executes the

system in a manner that demands resources in abnormal quantity, frequency or volume. 

Performance 
To

Testing:

test the run time performance of a system within the context of an integrated system.

29 April 2004 SRIMCA

CLCS Test Approach
Operations Environment

User Acceptance Tests

System S/W Validation Tests
System Delivery

COTS H/W on Dock

System Test

User App S/W Validation Tests

Acceptance Test

User Eval CSCI Int Test
Integration Environment

Developers
Early Unit Integ User Eval Test

System Integration and Test Group Validation Group Application S/W IPT

Unit Design Test

Development Environment

Users

THE ART OF DEBUGGING
is a consequence of successful testing -- when a test case uncovers an error, it is the debugging process that results in the removal of the error.  Debugging is an ART. The external manifestation of the error and the cause of the error normally do not share an obvious relationships.
31 April 2004 SRIMCA 

Debugging

THE ART OF DEBUGGING
The Debugging process
Execution of test cases Test cases

Results Suspected causes Debugging

Additional tests

Regression tests Corrections
April 2004 SRIMCA

Identified causes
32

THE ART OF DEBUGGING 
Debugging

Approaches
: - Take memory dumps, invoke run 

Brute force

time traces. Least efficient and quite common. 
Backtracking :- Once an error is uncovered, trace

your way back to the cause of the error. 
Cause Elimination :  Use of

- Isolate potential causes,

devise cause hypotheses tests to isolate bug.

debugging tools

33 April 2004 SRIMCA

COMMENTS 
Should

the software developer be involved with testing ? 
Developer¶s

have a vested interest in demonstrating that their software is error-free.  Developer¶s (psychologically) feel that testing is destructive. 
When

are we done with testing ? 

³You

are never done with testing, the burden shifts from you to the customer.´

34 April 2004 SRIMCA

SUMMARY
Testing accounts for the largest percentage of technical effort in the software process.  Objective of Software testing -- To uncover errors, maintain software quality.  Steps : Unit, Integration, Validation, System.  Debugging is often an art and the most valuable resource is often the counsel of other software engineers.
35 April 2004 SRIMCA 

Software