1. INTRODUCTION 1.1. WHAT IS TESTING? 1.2. THE TESTING PROCESS 1.3. TESTING TECHNIQUES 1.4. TOP DOWN AND BOTTOM UP INTEGRATION 1.5.

UNIT TESTING 1.6. BLACK BOX TESTING 1.7. WHITE BOX TESTING 2. UNIT TESTING - IN GENERAL 2.1. GENERAL PROCEDURE FOR UNIT TESTING 2.2. GENERAL RULES FOR UNIT TESTING(RU-G) 2.3. GENERAL CHECKLIST FOR UNIT TESTING (CU-G) 3. EQUIVALENCE TESTING 3.1. DEFINITION 3.2. PROCEDURE FOR EQUIVALENCE TESTING (PU-EQ) 3.3. RULES FOR EQUIVALENCE CLASS IDENTIFICATION (RU-EQ) 4. BOUNDARY VALUE TESTING 4.1. DEFINITION 4.2. PROCEDURE FOR BOUNDARY TEST CASE IDENTIFICATION 4.3. RULES FOR BOUNDARY TEST CASE IDENTIFICATION : 5. HEURISTIC TESTING 5.1. DEFINITION 5.2. PROCEDURE FOR HEURISTIC TEST CASE IDENTIFICATION: 5.3. RULES FOR HEURISTIC TEST CASES IDENTIFICATION (RU-HR) 6. COVERAGE TESTING 6.1. DEFINITION 6.2. GENERAL PROCEDURES FOR ALL COVERAGE TEST CASES 6.3. GENERAL RULES FOR WHILE BOX TESTING (RW-G) 6.4. GENERAL CHECK LIST FOR WHITE BOX TESTING (CW-G) 7. GLOSSARY:

2 2 2 2 3 3 4 4 5 6 8 8 9 9 9 9 11 11 11 11 12 12 12 13 13 13 13 13 13 14

Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure

Doc. Class: Testing Process

Page No. :1/14 Version :TR-1 Date: 21st March 2001

TESTING PROCESS 1. INTRODUCTION

Testing is an important stage of s/w life cycle. There are two kinds of testing possible. One is validation and other one verification. Verification is doing right things. Validation is doing things right. Here the testing process is to the validate the source code. This process is only for manual testing. 1.1. What is Testing?

Testing is the process of operating a system or component under specified conditions, observing or recording the results and making an evaluation of some aspects of the system or component. (IEEE/ANSI, Std-10.12-1990) Testing is an important stage in the software development process. Testing consumes a significant human effort. Of the total software development effort, testing takes 35 to 50%. Testing is a very important element in the QA strategy of any software development organization. 1.2. The Testing Process

The Testing Process is generally described by the V model shown below. Requirement Analysis System Design Detailed Design Acceptance Testing System Testing Integration Testing Unit Testing Coding Testing is normally done with a bottom-up approach. The first step in testing is Unit Testing in which the individual functions are tested. During integration, these functions are integrated into a system. During Integration Testing proceeds incrementally. At the end of integration, the system as a whole is tested. This stage is called System Testing. The testing activities are thus done in three distinct stages. Each of these stages are important. 1.3. Testing Techniques

The Process of Testing checks the implementation using the desired specification as a reference with an intent to find defects in the implementation. The Implementation must fully satisfy the Requirements. While translating the requirements into implementation, the following are the possibilities: 1. Correct Implementation :Some requirements are implemented correctly 2. Incorrect Implementation :Some requirements are implemented wrongly 3. Omissions :Some requirements are not implemented and omitted by mistake 4. Surprises :Some implementation do not correspond to any requirement
Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure Doc. Class: Testing Process Page No. :2/14 Version :TR-1 Date: 21st March 2001

Requirements Implementation Omissions Correct Implementation Surprises

Incorrect Implementation Different testing techniques will have to be used to handle these different possibilities. Omissions, Surprises and Incorrect Implementation must be detected and the implementation must be corrected. The following matrix shows which technique is useful for what type of problem. Testing Approach Black box/Functional Testing White box/Implementation based Testing Heuristic Testing Ability to Catch Omissions Surprises yes yes yes Incorrect Implement. yes yes yes

The testing strategy must take this fact into account and use complementary testing techniques to unearth all the problems. Thus effective Testing is a combination of Black box, White box and Heuristic Testing Techniques. 1.4. Top down and Bottom Up Integration

Integration of software may be done in a top down or bottom up fashion. In Bottom up testing all lowest level units are tested first. Then the higher level units are tested. The units already tested are used during testing higher level modules. Stubs are not needed for bottom up testing. In top down testing, the top most layer of Units are tested first followed by the next lower layer of units. While testing the second layer of units, the already tested top layer units are used. There is no need for Drivers in the top down testing. 1.5. Unit Testing

An Unit is a basic functional block in software. It is a function, procedure or a subroutine in a programming language like C. It forms the basic Unit by which blocks, subsystems or systems are built. Unit Testing is the first stage of testing. It immediately follows the coding stage. As discussed before, Black box, White box and Heuristic Testing must be used to expose all the problems in the Unit under test. Every Unit occupies an unique position in a hierarchy of units. Before integrating the units together, it is very desirable that each unit is tested individually. The
Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure Doc. Class: Testing Process Page No. :3/14 Version :TR-1 Date: 21st March 2001

Unit Testing stage ensures that an Unit behaves correctly before starting to test the interactions between different units by integration testing. Some engineers follow a testing strategy which combines unit and integration testing. Whichever strategy is followed, one objective is to look for problems within the individual units and the second objective is to look for problems in the manner in which these units interact. Unit Testing meets the first objective while integration testing meets the second objective. 1.6. Black box Testing

Black box or responsibility based testing views the unit or system under test as a Black box and involves test cases which exercise the responsibilities to be met by the unit or system under test. In the case of units, the Input output specifications of an individual unit is taken and the unit is subjected to a set of cases that attempt to find omissions and defects in the implementation vis a vis the requirements. The most important Black box testing techniques include Equivalence Partitioning and Boundary Value Analysis. 1.7. White box Testing

White box testing assumes a knowledge of the software under test. It is also called structural testing sometimes. The structure of the software under test is used to generate test cases to problems of surprises and incorrect implementation. The white box testing techniques typically involve exercising the different sections of the code systematically. The structure of the software and ways in which they may be exercised vary so much that a large number of test cases are necessary to test “all possible combinations”. This is not necessary and instead an increasingly comprehensive coverage criteria are defined which can be used while identifying White box test cases. These criteria are called the coverage criteria. The well accepted coverage criteria include: • Statement Coverage • Branch Coverage • Decision Coverage • Loop Coverage • LCSAJ Coverage

Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure

Doc. Class: Testing Process

Page No. :4/14 Version :TR-1 Date: 21st March 2001

2.

UNIT TESTING - IN GENERAL

START Identify Black Box Test Cases I/O Specification 2 Generate Black Box Test cases

1

3

Document Generated Test Case Definitions

4

Write Drivers And Stubs

Test Case Definitions in TCE form Test Source for all TCs Updated TCE form with actual values and Coverage Details Report

5

Execute Test Cases

6

check if desired coverage achieved (exit criteria)

No

End

Yes

7

Identify uncovered Branches

8

Identify Additional white box test case

9

Generate White box Test cases

Identify Black Box Test Cases 1.1 Identify Equivalence Test
cases
Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure Doc. Class: Testing Process Page No. :5/14 Version :TR-1 Date: 21st March 2001

1.2 Identify Boundary Test
Cases

1.3 identify Heuristic
Test Case & classify them Boundary / Equivalence tcs etc.

Identify White Box Test cases 8.1

Identify Testcase for Uncovered statements

8.2

Identify Testcase for Uncovered Branches

8.3

Identify Testcase for LCSAJ

2.1. 1.

General Procedure for Unit Testing Identify Black Box Test Cases 1.1. Identify all the input and output data entry fields for the selected unit. There may be some indirect data fields also. (ex. Additem_to_a_linked_list, or a functions return value, setting a variable indirectly through the called function.) For each of the identified data fields, identify the possible values that these data fields can hold (both in the input and output space).

1.2.

Note : Here, you have to use the following testing techniques in the given order, to identify black box test cases. Equivalence Class Boundary values Heuristic
1.3.

Take a particular field as focus and list the possible values for that variable. Repeate the setps 2 to 5 for all the identified data fields.

2.

Generate Black Box Test Cases 2.1. Generate a test case by assigning a value from the list of possible values for the selected variable and assigning any one of the possible values for all other data entry fields in input and output. This is called a test case definition.
Doc. Class: Testing Process Page No. :6/14 Version :TR-1 Date: 21st March 2001

Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure

3.

Document the Generated Test Cases
3.1.

The test case definitions should be entered into the Test Case Entry Template (given in subsequent sections).

4.

Identify Called functions 4.1. 4.2. Identify called functions and decide whether they have to be called as actual functions or they need stubbing for each test case. If stubbing is necessary, for those function calls, select the necessary variable and decide the necessary values for those data entry fields also.

5.

Write drivers and stubs 5.1. 5.2. Write driver code using the above test case definitions. Write stubs with the above test case definitions obtained from step 2.7, and modify the unit accordingly to support the stub calls instead of actual calls, is necessary. These details also should be added to the test case definition.

6.

Execute Test cases 6.1. Execute the test case(s). 6.2. Analyze the result for expected result after executing all the test cases.

7.

Establish coverage 7.1. To establish coverage, while generating drivers and stubs itself, instrument the test source so that it prints the branch and node it has traversed. The same thing can also be done by debugging the test source. While executing the test cases, note down the path taken by seeing the printed mater on screen or by stepping through the FUT.

7.2.

Note : Instead of the above two steps, you can use FasTest to just establish the coverage by entering the identified testcases and executing them(incase of ‘c’). 8. Identify additional test case for coverage 8.1. If desired level of coverage is not achieved, identify additional paths and branches which are left not covered by the currently available set of test case definitions. Identify additional test case to cover the remaining statements/ branches /paths. This can be done by slightly modifying the test cases which has covered the adjacent path.

8.2. 8.3. 9.

Generate White box Test cases:

Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure

Doc. Class: Testing Process

Page No. :7/14 Version :TR-1 Date: 21st March 2001

Generate additional test cases by modifying existing test cases or generate new test cases to cover entirely new path. The test case should be generated in the given order Coverage techniques Statement coverage Branch coverage Path coverage. Deccanet’s standard is Branch coverage unless otherwise specified for a particular project.
9.2.

9.1.

After this goto setp PU_G5

2.2.
RU_G1 RU_G2 RU_G3 RU_G4 RU_G5

General Rules for Unit Testing(RU-G) The unit under test should be compliable The unit should have passed the normal inspection review process Identify and remove duplicate test cases(keep the no. of test case less) Test cases should have a reasonable chance of catching defects.

Possible values of each data entry fields should lie with in the acceptable range of the underlying data type. Consider all the direct and indirect data entry fields as a focus variable.

RU_G6 RU_G7

Consider both input and output variables / events as focus and generate test cases around all of them. Don't omit any test case generated for each focused variable’s possible values. Don’t forget to handle any of the called functions. Initialize all the pointers with appropriate value. Don’t generate too complex or too trivial test cases. Ensure that the standards are met from coverage point of view.

RU_G8 RU_G9 RU_G10 RU_G11 RU_G12

2.3.
CU_G1 CU_G2

General checklist for Unit testing (CU-G) Whether the unit under test is compliable or not.

Confirm that the unit under test had already been subjected to normal inspection / review process. Check whether the test case is too complex or too trivial? Check whether a redundant test case exists? Check whether the test cases have a reasonable change of catching defects?
Doc. Class: Testing Process Page No. :8/14 Version :TR-1 Date: 21st March 2001

CU_G3 CU_G4 CU_G5

Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure

CU_G6

Verify that the possible values of each data entry fields lie with in the acceptable range of the underlying data type. Check whether each of the direct and indirect data entry fields are considered as a focus variable. Check whether all the possible test cases are there for each of the focused variables.

CU_G7

CU_G8 CU_G9

Check whether all the called functions are handled properly by either stubbing it or calling it as an actual function. Check whether all the pointers are initialized properly? Verify whether the standards are met from coverage point of view.

CU_G10 CU_G11

3. 3.1.

EQUIVALENCE TESTING Definition

If you expect a no. of test cases to be testing the same behavior or the Function Under Test(FUT), then you must treat them as equivalent. A group of tests forms an equivalence class if you believe that : • they all test the same thing • if one test catches a bug, the others probably will too • if one test doesn’t catch a bug, the others probably won’t either. [ref. : 126 of Cem Kaner] 3.2. Procedure for Equivalence Testing (PU-EQ)

Follow the general procedure for unit testing. In addition to that, follow the procedure given below, to identify equivalence classes: 1. 2. Analyze the possible values that can be entered for each data fields(both in input and output space) and group those values into possible equivalence classes. To group the values into equivalence classes, decided whether the possible values of the identified data field fall under any one of the following category • Range of value • List of values • Specific value As per the rules for the corresponding category define the test cases. For all the data entry field (in the input and output space) repeat the steps 2 and 3 until all the variables are subjected to this procedure.

3.
4.

3.3.

Rules for Equivalence Class Identification (RU-EQ)

RU_EQ1

For any variable whose possible values lies in a range, then generate 3 equivalence classes.
Doc. Class: Testing Process Page No. :9/14 Version :TR-1 Date: 21st March 2001

Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure

Ex: If your possible values lies in a range(say 10 to 90) for a particular variable, then the equivalence classes are • <10 Invalid equivalence class • >90 Invalid equivalence class • 10 to 90 Valid equivalence class For any variable, if the possible range of values can be further subdivided into ‘n’ sub ranges, then define n+2 equivalence classes. If the range can be further sub divided into sub ranges(say 10-30, 30-60 60-90), then consider the following equivalence classes also
RU_EQ2

• • • • •
RU_EQ3

10 - 30 Valid Equivalence class 31-60 Valid equivalence class 61-90 Valid equivalence class <10 Invalid equivalence class >99 Invalid equivalence class

For any variable, if the possible value is a list of ‘n’ items, then ‘n+1’ equivalence class should be defined, if the behavior of the FUT is expected to be different for each. Ex: If the possible values lie in a list of n item, then n+1 equivalence class should be generated with each list item as an individual valid equivalence class and an item not present in the list should form the n+1th equivalence class, which is an invalid equivalence class. Ex: If your possible values lie in a list of items, (say a, m, k) then the equivalence classes are • Each of the list items is a valid equivalence class (a, m, k - 3 valid ECs ) • Items which are not there in the list forms one Invalid equivalence class If the possible value is a specific values, then two equivalence class should be generated one with the specific value and other class with any values other than the specific Ex: If your possible value is a specific value (say 0 always), then the equivalence classes are • specific value (i.e.) 0 Valid equivalence class • all values other than 0 Invalid equivalence class
RU_EQ4

If the input specifies the number(N) of valid values, define one valid Equivalence class and two invalid equivalence classes. EX : if the valid name of project requires at least one letter and not more that 8 characters then any name with 1 - 8 letters Valid equivalence class any other combination of characters Invalid equivalence class
RU_EQ5 RU_EQ6 RU_EQ7 RU_EQ8

Don’t forget to test invalid equivalence classes for any variable. Don’t omit sub equivalence classes within valid equivalence classes

Identify time-determined equivalence classes(ex. press the shift key just before, during, and just after the system finishes scanning for keyword)

Don’t forget to identify indirect(hidden) equivalence classes Ex. if the function deals with insertion of an item to linked list, then adding an item to the list when the list contains less than maximum no. of elements becomes an equivalence class. Adding an item to the list, when the list contains maximum no. of elements.
RU_EQ9 RU_EQ10

Don’t forget to focus on output variables. Consider all the output variables also for generating equivalence classes. Apply all the above rules on these variables also.
Doc. Class: Testing Process Page No. :10/14 Version :TR-1 Date: 21st March 2001

Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure

RU_EQ11

Some times, a group of variable may gives raise to a possible value for a particular situation. In this case also, apply all the above rule on these group of variables.

4. 4.1.

BOUNDARY VALUE TESTING Definition

Boundary value testing is a variant and refinement of equivalence partitioning, with a major difference - Rather than selecting any element in an equivalence class as being representative, elements are selected such that each edge of the equivalence class is the subject of a test. Boundaries are always a good place to look for defects.

4.2.

Procedure for Boundary Test case Identification

In addition to the general procedures, the following should also be considered for boundary test case generation 1. 2. Analyze the possible values that can be entered for each fields(both in input and output space) and group those values into possible boundaries with precise lower and upper limits. To group the values into boundary test cases, decide whether the possible values of the identified data filed falls under any one of the following category : • Range of Values • List of continuous values As per the rules for the corresponding category, define the test cases For all the data entry fields, repeat steps 2 to 3 until all the variables( in the inputs and output space) are subjected to this procedure.

3. 4.

4.3.

Rules for Boundary Test case Identification :

For any variable whose possible values lie in a range, then generate 3 equivalence classes. Ex :If your possible values lies in a range, say 10 to 90 then the boundary classes are • 9,91 Invalid boundary values • 10, 11, 89, 90 Valid boundary values
RU_BV1

For any variable, if the possible range of values can be further subdivided into ‘n’ sub ranges, then ‘n+2’ equivalence classes should be defined. Ex : If the range can be further sub divided into sub ranges(say 10-30, 30-60 60-90), then consider the following boundary test cases also • 9, 91 Invalid boundary test cases • 10, 11, 29, 30 Valid boundary test cases • 31, 32, 59,60 Valid boundary test cases • 61,62, 89,90 Valid boundary test cases
RU_BV2
Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure Doc. Class: Testing Process Page No. :11/14 Version :TR-1 Date: 21st March 2001

For any variable if the possible value lies in a continuous list of n items, then define the boundary test cases by keeping the following consideration in mind. Ex : If your possible values are a list of continuous items ( say m,n,o,p), then the boundary classes are • Each of the list items is a valid boundary class (m,n,o,p - valid boundary cases) • One item below the lower boundary and one item above the upper boundary list forms one Invalid equivalence class (l, q - Invalid boundary cases)
RU_BV3

Consider the no. of inputs to a program as an input data field and try defining test cases around the limit of the no. of inputs Ex: Some times no. of inputs to a program may gives raise to a test case. In this case, consider the range from 1 to n as a valid range, give 0, 1,2,n-1, n and n+1 inputs for the program and test it.
RU_BV4

Think of boundaries on hidden data fields, which are not explicit as a defined variable in the project, and generate test cases around the boundary of possible values of that hidden variable. Ex: If the functionality of the function is to add an item to linked_list, then add and item when the list is empty, when there is an item, when there is n-1 item where n is the maximum capacity of the list). This will form 3 valid boundary cases. Add an item when the list is full. This is an invalid boundary case.
RU_BV5

Look for boundaries around the loop variable limits( if any loop exists in the FUT). Ex: If any loop is there in the FUT, the executed the loop for 0 times, 1 item, n-1 time and n times, where n is the maximum no. of times the loop has to be executed.
RU_BV6 RU_BV7 RU_BV8 RU_BV9 RU_BV10 RU_BV11

Don’t forget to test invalid boundary cases for any variable. Don’t omit any valid boundary cases also. Don’t forget to generate test cases for sub ranges within a range. Look boundaries around each valid equivalence classes

Identify time-determined boundary classes. Ex: Assume, the first digit has to be dialed within 5 sec after getting dial tone, then press the first digit exactly at 5th sec just before and after 5 sec.

5. 5.1.

HEURISTIC TESTING Definition

Heuristic Testing is an approach, based on intuition and experience, to identify tests that are considered likely to expose errors. The basic idea is to make a list of possible errors or errorprone situations and then develop tests based on the list. With the past experience we need to give some inputs to the present program, so as to check whether the most common defects still exists or not. 5.2. Procedure for Heuristic Test case identification:

Identify test case based on your judgement and previous experience
Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure Doc. Class: Testing Process Page No. :12/14 Version :TR-1 Date: 21st March 2001

5.3.

Rules for Heuristic Test cases identification (RU-HR)

In addition to the general rules, the following rules also should be followed to generate heuristic test cases
RU_HR1

Don’t omit any variable which are directly or indirectly affecting the functionality of the Function Under Test. Test the unit with some random values for all the variables, without focusing on any particular variable. Any interesting feature, combinations, sequence which you feel may hold a surprise should be exercised. Check for rarely executed conditions and sequences. Identity test cases by recollecting situations from past experiences Try to create possible error conditions. Identify test case which seem to be “interesting” to you.

RU_HR2

RU_HR3

RU_HR4 RU_HR5 RU_HR6 RU_HR7

6.

COVERAGE TESTING

6.1. Definition These are the testing techniques that takes advantages of the structure of the program under test are grouped under white box or structural testing. Structural testing complements black box testing 6.2. General Procedures for all coverage test cases

Execute the setps explined in the section PU_G (PU_G14 to PU_G18) 6.3. General rules for While box testing (RW-G)

RW_G1.Confirm that the identified test cases are covering expected branches / statements. RW_G2. Don’t generate duplicate , trivial test cases which does not give an incremental coverage RW_G3. Ensure that the desired level coverage is achieved by meeting the organization standards RW_G4. Give more attention to the rarely accessed branches and statements RW_G5. Unreachable code, if any, should be properly commented, so that the coverage calculation are not affected 6.4. General check List for white box testing (CW-G)

CW_G1 Check for duplicate , trivial test cases
Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure Doc. Class: Testing Process Page No. :13/14 Version :TR-1 Date: 21st March 2001

CW_G2 Check whether all the branches are covered CW_G3 Check whether identified TCs are covering the expected branches /statements or not CW_G4 Verify whether all the rarely accessed branches and statements are given more attention CW_G5 Check whether if any unreachable code left uncommented Note : Here there is no need for specific procedure /rule and checklist for each coverage techniques and for all techniques the same procedures and rules are applicable. 7. GLOSSARY: FUT LCSAJ EC BV TC Organizational Standards : Function Under Test : Linear Code Segment And Jump : Equivalence Classes : Boundary Values : Test Case : 100 % Statement Coverage 100 % Branch Coverage

Doc. info. : Rules and steps for Testing Phase Custodian: Prashant Hemchandra Manapure

Doc. Class: Testing Process

Page No. :14/14 Version :TR-1 Date: 21st March 2001