Testing and Testing Methodologies - Basics

Training Objectives

The objective of this training is to provide an overview of Testing principles and practices

07/07/09

Software Testing Practice

2

   

Introduction to software testing Testing in software lifecycle Testing methodologies Introduction to testing practices

07/07/09

Software Testing Practice

3

Software Testing & Quality
Quality Assurance is planned and systematic activities necessary to provide adequate confidence that products and services are defect free William Perry Quality Control is a work bench check procedure where correctness is determined and action is initiated when nonconformance is detected William Perry

Testing is a ‘Safety Net’
07/07/09 Software Testing Practice

-Dr. Roger Pressman

4

Why Bug exist?? Common Reasons
Poor

understanding and incomplete requirements

  

Unrealistic schedule Fast changes in requirements Too many assumptions and complacency

07/07/09

Software Testing Practice

5

Why Testing ?? Some failures

Some of major computer system failures listed below gives ample evidence that the testing is an important activity of the software quality process. In April of 1999 a software bug caused the failure of a $1.2 billion military satellite launch, the costliest unmanned accident in the history of Cape Canaveral launches. The failure was the latest in a string of launch failures, triggering a complete military and industry review of U.S. space launch programs, including software integration and testing processes. Congressional oversight hearings were requested

07/07/09

Software Testing Practice

6

Why Testing ?? Some failures
• Software bugs caused the bank accounts of 823 customers of a major U.S. bank to be credited with $924,844,208.32 each in May of 1996, according to newspaper reports. The American Bankers Association claimed it was the largest such error in banking history. A bank spokesman said the programming errors were corrected and all funds were recovered. • In April of 1999 a software bug caused the failure of a $1.2 billion military satellite launch, the costliest unmanned accident in the history of Cape Canaveral launches. The failure was the latest in a string of launch failures, triggering a complete military and industry review of U.S. space launch programs, including software integration and testing processes. Congressional oversight hearings were requested

07/07/09

Software Testing Practice

7

Hurdles normally encounters are:  Usually late activity in the project life cycle  No “concrete” output and therefore difficult to measure the value addition  Lack of historical data  Recognition of importance is relatively less  Politically damaging as you are challenging the developer  Delivery commitments  Too much optimistic that the software always works correctly

07/07/09

Software Testing Practice

8

Introduction To Software Testing
      

Testing is the process of assuring that a product meets end-user requirements Executing a program with the intent of finding errors A good test is one that has a high probability of finding undiscovered errors All tests should be traceable to customer requirements and repeatable Tests should be planned long before testing begins Exhaustive testing is not possible To be most effective, testing should be conducted by an independent testing group ( ITG ) or Independent testing team

Testing cannot show the absence of software errors, it can only show their presence
07/07/09 Software Testing Practice 9

Who Should Test The Software?

Developer understands the system but will test ‘Gently’ and is driven by ‘Delivery’

ITG or Independent testing team will attempt to break and is driven by ‘Quality’

USER will verify acceptance criteria and is driven by ‘Cost’

07/07/09

Software Testing Practice

10

   

Introduction to software testing Testing in software lifecycle Testing methodologies Introduction to testing practices

07/07/09

Software Testing Practice

11

Software Development Life Cycle
URS UAT planning User Acceptance Testing

SRS

System test planning

System Testing

Verification
HLD

Validation
Integration Testing Delivery production deployment

Integration test planning

LLD

Unit test planning

Unit testing

Maintenance and enhancement

Coding
07/07/09 Software Testing Practice 12

Software Testing Life Cycle
Test scope Test planning Test engineering Test execution
- Test design - Formal specs - Test scenarios - Test cases - Test data - Tool development - Implement stubs - Test data feeders - Batch processes - Execute testing - Collate test data - Identify bugs

Defect analysis
- Check unexpected behavior - Identify defective application areas - Identify erroneous test data - Identify defect trends / patterns

- Baseline inventory - Acceptance criteria - Schedule - Prioritization - Test references - Signoff requirement

- Approach - Process & tools - Methodology - Delivery models - Risk plan - Project workflow - Quality objectives - Configuration plan

Test Closure
- Stop Testing - Prepare Reports - Prepare Test closure Document.

07/07/09

Software Testing Practice

13

What is Verification & Validation ?

Verification All “REVIEW” activities throughout the life cycle that ensure the product deliverables meet their specifications Validation The “TEST” phase of the life cycle, which ensures that the end product meets the specifications Verification Validation
07/07/09

- Are we building the product right ? - Are we building the right product ?
Software Testing Practice 14

   

Introduction to software testing Testing in software lifecycle Testing methodologies Introduction to testing practices

07/07/09

Software Testing Practice

15

Categories of Testing

Structural (White Unit Testing Box)  Integration  Functional (Black  System Box)  Acceptance Unit Testing - Structural Testing  Risk Based Integration - Structural and Functional  Heuristic System - Functional, Risk based and Heuristic

Acceptance - Functional, Risk based and Heuristic
07/07/09 Software Testing Practice 16

Testing Levels

Techniques

Structural Testing (White Box)
Inputs

Outputs
07/07/09 Software Testing Practice

Is a technique where Program structure/spec s used to define test case  Program viewed as a graph

17

White box Testing Methods
Statement Coverage  Branch Coverage  Loop Coverage  Data flow

07/07/09

Software Testing Practice

Statement Coverage
Definition This technique is used to ensure that every statement / decision in the program is executed at least once. Test Conditions
Statement1 Statement2 1. (A > 1) and (B = 0) 2. (A<=1) and (B NOT = 0) 3. (A<=1) and (B=0) 4. (A>1) and (B NOT= 0)   

Program Sample //statement 1 //statement 2 If((A > 1) and (B=0)) //sub-statement 1 Else //sub-statement 2 Description Statement coverage requires only that the if … else statement be executed once – not that substatements 1 and 2 be executed.

07/07/09

Minimum level of Structural Coverage achieved Helps to identify unreachable Code and its removal if required “Null else” problem: It does not ensure exercising the statements completely. Example: ..if x<5 then x= x+3; x>5 decision not enforced.Practice not covered Paths Software Testing 19

Branch Coverage
Definition A test case design technique in which test cases are designed to execute all the outcomes of every decision Graph 1
Y=Y+1
T F T

Program Sample
IF Y > 1 THEN Y = Y + 1 IF Y > 9 THEN Y = Y + 1 ELSE Y=Y+3 END Y=Y+2 ELSE Y=Y+4 END

3
Y>1
F

2
Y=Y+4 Y=Y+3

No. Of Paths = 3 Test Cases: 1 (Y > 1) and (Y > 9) 2 (Y > 1) and (Y <= 9) 3 (Y < = 1)

Y>9

Y=Y+1 Y=Y+2
07/07/09

Software Testing Practice

20

Branch Coverage Testing
Strengths and Weaknesses
 

Considered superior to Statement Testing. Solves the “null else” problem of Statement Testing by forcing all the decisions. It does not exercise compound decisions well. Example: if ((a>5) or (b<10)) One of the two conditions may never get exercised.

07/07/09

Software Testing Practice

21

Introduction to Cyclomatic Complexity Control Graphs
Definition Graph
1

CC is a quantitative measure to identify the logical complexity of program using control graph It is a graphical representation of the flow of execution.

2

3 4 5

Program Sample void abx(int a, int b, int x) { 1 if ( a > 1 ) && ( b == 0 ) 2 x = x / a; 3 if ( a == 2) || ( x > 1) 4 x = x + 1; 5 print(x); }
07/07/09

Description  The numbers represent various program segments in the unit  A node in the control graph (circle) represents a segment  An edge (arrow) indicates all other segments that can be reached from the given segment Software Testing Practice 22

Definition

Cyclomatic Complexity
C=E-N+2

Formula for Cyclomatic complexity

CC indicates the upper bound of the number of independent paths be tested in an unit. Program Sample void abx(int a, int b, int x) { 1 if ( a > 1 ) && ( b == 0 ) 2 x = x / a; 3 if ( a == 2) || ( x > 1) 4 x = x + 1; 5 print(x); } Calculation No. of Edges = 6, No. of Nodes = 5 C = 6 - 5 + 2 = 3, No. of Paths = 3 No. of test cases = 3 07/07/09

C is Cyclometic complexity E is number of edges in the graph N is number of nodes in the graph Graph
F a >1 b == 0 T x=x/a a==2 x>1 F Print x T x=x+1

Software Testing Practice

23

Condition Coverage - AND
Definition  Both parts of the predicate are tested  Program Sample shows that all 4 test conditions are tested Program Sample If((A > 1) AND (B=0) { //sub-statement 1 } Else { //sub-statement 2 }
07/07/09

Conditions Table ( 2 n )

Test Conditions 1. (A > 1) AND (B = 0) 2. (A > 1) AND (B NOT = 0) 3. (A<=1) AND (B NOT = 0) 4. (A<=1) AND (B = 0)

Software Testing Practice

24

Condition Coverage - OR
Definition  Both parts of the predicate are tested Conditions Table ( 2 n )

A>1 TRUE  Program Sample shows that all TRUE FALSE 4 test conditions are tested FALSE
Program Sample If((A > 1) OR (B=0) { //sub-statement 1 } Else { //sub-statement 2 }
07/07/09

OR OR OR OR

B = 0 RESULT TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE

Test Conditions 1. (A > 1) OR (B = 0) 2. (A<=1) OR (B NOT = 0) 3. (A<=1) OR (B=0) 4. (A>1) OR (B NOT= 0)

Software Testing Practice

25

Condition Coverage
Decision/Condition Coverage  Combination of decision coverage and condition coverage.  The problem of language support for condition coverage persists Multiple Condition Coverage  All combinations of simple conditions(true& false evaluations)  For n conditions, 2n combinations  Impractical if conditions are more, testers tend to confuse

07/07/09

Software Testing Practice

26

Loop Coverage
Loop Coverage  Simple  Nested Loops  Serial / Concatenated Loops  Unstructured Loops (Goto) Example of CC Coverage  Boundary value tests  Cyclomatic Complexity

I=1

F

I<N T

for ( I=1 ; I<n ; I++ ) printf (“Simple Loop”); E=5 , N=5 CC = E-N+2 CC = 2
Software Testing Practice 27

END

Print I ++

07/07/09

Loop Testing

Simple  loop

Nested  Loops

Concatenated        Loops Unstructured        Loops
Software Testing Practice 28

07/07/09

Loop Coverage
Concatenate Loop
for (I=1;I<n;I++) statement 1 for (k=1;k<n;k++) statement 1

Simple Loops  A minimum test is 2 iterations, to detect data initialization and use faults  Must exercise the domain boundary of the loop control variable Serial / Concatenated Loops  more loops in same control path  Define/use data relationship Exists -Treat them as Nested Loops

Nested Loop
for (I=1;I<n;I++) for (k=1;k<n;k++) statement 1

Nested Loops  Test inner loop first, outer last  Set all outer loop controls to minimum values

 Set inner and outer loop controls Not Exists - Treat them as Simple to typical values Loops
07/07/09 Software Testing Practice 29

Data Flow Testing
Data States
  

defined(initialized, but not used yet) used (value evaluated) killed

Test contents  A statement where a variable is defined (assigned a value)  A statement where that variable is used with that definition active  A statement where that variable is used with that definition killed/freed Features of Data flow Testing  Each flow path must be tested at least once  Benefit is to guard against defects where the wrong value is used  Usually results in more path tests than complete condition coverage
07/07/09 Software Testing Practice 30

Data Flow Testing Example
Program Sample begin x=2 loop x =x + 1 if (x=5) then exit else continue end if end loop x=0 end Data State - defined Data State - used

Data State - freed

07/07/09

Software Testing Practice

31

Functional Testing (Black Box)

Inputs

Outputs

07/07/09

Software Testing Practice

Tests of Business Requiremen ts based on external specificatio ns without knowledge of how the system is constructed

32

Black box Techniques
 Techniques

Inputs Outputs

Low Level Techniques
Equivalence

partitioning Boundary value analysis Input and Output domain
07/07/09 Software Testing Practice 33

Equivalence Partitioning

Divides the input domain of a program into classes of data Derives test cases based on these partitions An equivalence class is a set of valid or invalid states of input Test case design is based on equivalence classes for an input domain Output
07/07/09 Software Testing Practice 34

Invalid Inputs

Valid Inputs

SYSTEM

Equivalence Partitioning
Invalid 2 Valid Range 7 Invalid 12

Less than 4 Input Range [4,10]
 

Between 4 and 10

More than 10 Test values [2,7,12]

Useful in reducing the number of Test Cases required It is very useful when the input / output domain is amenable to partitioning

07/07/09

Software Testing Practice

35

Boundary Value Analysis
3 4 7 11 10

Less than 4

Between 4 and 10

More than 10

Input Range [4,10]
   

Test values [3,4,7,10,11]

07/07/09

A Black Box Testing Method Complements to Equivalence partition BVA leads to a selection of test cases that exercise bounding values Design test cases Test  min values of an input  max values of an input  just above and below input range
Software Testing Practice

36

Functional vs Structural
Program behavior
Do we require both?

Functional establishes confidence

Structural seeks faults

07/07/09

Software Testing Practice

37

What to verify during each build ?

Interface integrity
x

Internal and external interfaces are tested as each module (or cluster) is incorporated into the structure Tests designed to uncover functional errors are conducted Tests designed to uncover errors associated with local or global data structures are conducted Tests designed to verify performance bounds established during software design are conducted

Functional validity
x

Information content
x

Performance
x

07/07/09

Software Testing Practice

38

Other types of Testing
Usability

testing Security testing Smoke Testing Configuration testing Compatibility testing Installation testing Reliability testing Documentation testing
07/07/09 Software Testing Practice 39

   

Introduction to software testing Testing in software lifecycle Testing methodologies Introduction to testing practices

07/07/09

Software Testing Practice

40

Software Testing Phases
Product Release

Unit Test Environment

Test Lab Environment

Test Lab Environment

Simulated Production Environment

Unit Test
Development

Integration Test

System Test

UAT

Developers

Testers

Testers

Testers/Users

07/07/09

Software Testing Practice

41

Integration Testing
Definition

Integration testing is a search for component faults that cause Intercomponent failure Integration Tests emphasizes on interaction between modules and interfaces.

Purpose

Method

White Box and Black Box Testing

Testers
07/07/09 Software Testing Practice 42

Top Down Integration

Modules are integrated by moving downward Verifies major control or decision points early in the test
43

Driver Module under Test

Stub Tested Module

07/07/09

Software Testing Practice

Bottom-up Integration

Construction and testing with atomic modules Stubs are NOT needed Drivers are
44


Driver Module under Test
07/07/09

Stub Tested Module
Software Testing Practice

Integration testing within the system

An Independent System with several Modules

Application System
Module A Module B Module C

Testing of interaction between the modules A, B and C is known as integration testing within the system
07/07/09 Software Testing Practice 45

Integration testing outside the system

An Independent System Interacting with external systems

A B

C D

External System

Sub System

Testing between the Independent system, External System and Sub System is Integration testing outside the system

07/07/09

Software Testing Practice

46

System Testing
Objective

To check whether a system as a whole conforms to the agreed specification and to detect the discrepancies between the behavior of the constructed system & its specification. Complete application with all Interfaces, Performance, Usability, Installation etc

Scope

Method

Black Box

07/07/09

Software Testing Practice

47

User Acceptance Testing
Objective

To test the integrity of of the application by the users in UAT environment. Will act as a confidence building measure for the users before the product is released. Complete application with all interfaces, user requirements

Scope

Method

Black Box

07/07/09

Software Testing Practice

48

Systems Under Test Product Management

Testing Workflow

Business Analysts

Testing Team

Successful Smoke Test Scope
Review Test plans and TC’s

UAT Coordinator

Sign-Off

Development Team

Test Plan & Test Cases

Failure

Defects Log Release Management
07/07/09 Software Testing Practice

Success Schedule & Execution Test Results
49

Regression Testing
Objective

To test the integrity of critical business functions of an application after undergoing changes independent of the underlying software architecture Black box. Automation test tools are used

Testing method

?

Renovation
  

Unit test

Integration test

Regression test

07/07/09

Does not replace thorough unit testing nor integration testing Not a short cut for a proper testing cycle (Unit, integration, System & user test) To the extent that it substitutes for integration test or even worse unit testing, it will find those faults, but at a higher expense of rework to correct the faults
Software Testing Practice

50

Performance Testing
Objective Performance testing is performed at the system level to confirm that all the performance objectives are met. It’s also performed for the load handling capacity for the system. Load Test Stress Test Volume Test Method

Phases
?

Unit Test

Integration Test

System Test

Performance Test

•Normal or Above normal volumes of transactions can be processed within the expected time frame •The application system is structurally able to process large amounts of data •System capacity has enough resources to meet expected turnaround times •People can actually use the system at peak times.
07/07/09 Software Testing Practice 51

Performance Testing
Load Stress Volume Evaluation of system performance under normal conditions Maximum number of users or transactions Behavior of the system under abnormal conditions Performance level of software product at optimum load.

High number of users

Ability to handle large number of transactions Internal program or system limitations are exceeded Analyses the performance for the specific type of test data

Capability of system architecture to handle load System configuration remains constant

Low system resources

Recovery of the system when the “pushed over edge”

07/07/09

Software Testing Practice

52

Software Testing - Some Bad Practices

No formal testing techniques used

Testing is cut short in the name of economics

Easily reachable paths only are tested

Poor level of

automation

No early phase testing like Unit testing and Integration testing

No traceability between

Test cases and software units
07/07/09 Software Testing Practice

Stubs and drivers are not used

53

References
Software Engineering Software Engineering Black-Box Testing Effective Methods of Software Testing Testing Computer Software Managing the Testing Process Roger Pressman Ian Somerville Boris Beizer William Perry Cem Kaner Rex Black

07/07/09

Software Testing Practice

54

Thank You

Sign up to vote on this title
UsefulNot useful