You are on page 1of 47

Software Testing: White box testing

SE 110

SE 110 – Spring 2012

White-box testing
• White box testing is a way of testing the external functionality of the code by examining and testing the program code that realizes the external functionality. • Synonymous terms: Clear box testing, glass box testing, open box testing. • White box testing takes into account the program code, structure and internal design flow.
SE 110 – Spring 2012

White-box testing: Classification

SE 110 – Spring 2012

Static testing
• Involves only the source code and not the executables or binaries. • Does not involve executing the program on a machine but rather humans going through it or the usage of specialized tools. • Some of the properties tested by static testing: – Whether the code works according to the functional requirement – Whether the code has been written in accordance with the design developed earlier in the project life cycle – Whether the code follows all applicable standards – Whether the code for any functionality has been missed out – Whether the code handles errors properly
SE 110 – Spring 2012

SE 110 – Spring 2012 . inline pressure. • It can detect multiple defects at one go.Static testing by humans • Humans can find errors that computers can’t. • A human evaluation can compare the code against the specifications more thoroughly. • Multiple humans can provide multiple perspectives. • It reduces downstream.

• No structured method or formalism is required to ensure completeness.Desk checking • Author informally checks the code against the specifications and corrects defects found. • It relies only on the thoroughness of the author. • It may suffice for “obvious” programming errors but may not be effective for incomplete / misunderstood requirements. SE 110 – Spring 2012 . • No log or checklist is maintained.

not scalable or reproducible.Desk checking: Advantages and disadvantages • Advantages – The programmer knows the code and programming language well and hence is best suited to read the program. • Disadvantages – Person dependent. – Tunnel visioning of developers. – Reduces delay in defect detection and correction. SE 110 – Spring 2012 . – Less scheduling and logistics overheads. – Developers prefer writing new code and don’t like testing.

SE 110 – Spring 2012 . • Brings in multiple perspectives – A set of people look at the program and raise questions for the author. • Very similar to inspections.Code walkthrough • Group oriented (as against desk checking). – Unanswered questions during walkthrough are answered later by the author. but considered less formal. – The author explains the logic of the code and answers the questions.

Fagan Inspection • • • • Group-oriented activity Highly formal and structured Has specific roles Requires thorough preparation SE 110 – Spring 2012 .

Roles in formal inspection • Author – Author of the work product – Makes available the required material to the reviewers – Fixes defects that are reported • Moderator – Controls the meeting(s) • Inspectors (reviewers) – Prepare by reading the required documents – Take part in the meeting(s) and report defects • Scribe – Takes down notes during the meeting – Assigned in advance by turns – Can participate to review to the extent possible – Documents the minutes and circulates them to participants SE 110 – Spring 2012 .

g. coding standards) – Any necessary checklists (e. code review checklist) SE 110 – Spring 2012 .g..Fagan inspection: Documents • Typical documents circulated: – Program code – Design / program specifications – SRS (if needed) – Any applicable standards (e..

Fagan inspection: Meetings • Preliminary meeting (optional) – Author explains his / her perspective – Makes available the necessary documents – Highlights concern areas.” “major” / “minor.” “systemic” / “mis-execution” – Scribe documents all the findings and circulates them Follow-up meeting (optional) – Author fixes defects – If required. • • SE 110 – Spring 2012 . if any. a follow-up meeting is called to verify completeness of fixes. for which review comments are sought Defect Logging Meeting – All come prepared! – Moderator goes through the code sequentially – Each reviewer comes up with comments – Comments / defects categorized as “defect” / “observation.

Fagan inspection: Advantages and disadvantages • Advantages – Thorough. when prepared well – Brings in multiple perspectives – Has been found to be very effective • Disadvantages – Logistically difficult – Time consuming – May not be possible to exhaustively go through the entire code SE 110 – Spring 2012 .

Combining various methods • Prioritizing various programs – High priority / important codes subject to formal inspection. SE 110 – Spring 2012 . • Deciding between 100% coverage and partial coverage. medium ones for walkthrough and low ones for desk checking.

• Variables declared but not used.Static analysis tools Help identify / are used to: • Whether there are unreachable codes (usage of goto statements sometimes creates this situation. there could be other reasons too). • For calculation of cyclomatic complexity (to be discussed later). • Use of non-portable or architecture-dependent programming constructs. • Mismatch in definition and assignment of values to variables. • Illegal or error-prone type-casting of variables. • Memory allocated but not having corresponding statements for freeing up memory. SE 110 – Spring 2012 .

documentation standards etc. it is useful to have a code review checklist.Code review checklist • While following static analysis techniques. – A product or project-specific checklist that addresses issues specific to the product or project. SE 110 – Spring 2012 . • In a multi-product organization. • The code review checklist aids in organizational learning by indicating what to look for. the checklist may be at two levels: – An organization-wide checklist that will include issues such as organization coding standards.

White-box testing: Classification SE 110 – Spring 2012 .

Structural testing • Done by running the executable on the machine. • Designing test cases and test data to exercise certain portions of code. • Entails running the product against some predefined test cases and comparing the results against the expected results. • Types of structural testing – Unit / code functional testing – Code coverage testing – Code complexity testing SE 110 – Spring 2012 .

SE 110 – Spring 2012 . Done before more expensive checks. Running through an IDE. Building “debug versions”. Removes “obvious” errors.Unit/code functional testing • • • • • Initial quick checks by developer.

• Instrumented code can monitor what parts of code is covered.Code coverage testing • Exercises different parts of code. executed often. – Instrumentation rebuilds the product. • Maps parts of code to required functionality. • Finds out percentage of code covered by “instrumentation”. • Can help identify critical portions of code. SE 110 – Spring 2012 . linking it with libraries.

Types of coverage • • • • Statement coverage Path coverage Condition coverage Function coverage SE 110 – Spring 2012 .

Statement coverage   Statement coverage refers to writing test cases that execute each of the program statements. Two-way decision statements (if – then – else). Multi-way decision statements (switch statements). for. like while/do. Typical program constructs constituting program statements: – – – – – Sequential instructions. etc. • Statement coverage = (Total statements exercised/Total number of executable statements in program)*100. repeat/until. Loops. SE 110 – Spring 2012 . Additional constructs for OO-languages.

Statement coverage: Sequential statements Each circle represents one or more non-branching source code statements • Generate test data to make the program enter the sequential block. SE 110 – Spring 2012 . to make it go through the entire block.

Statement coverage: If-then-else • Have test cases to test “then” part • Have test cases to test “else” part SE 110 – Spring 2012 .

SE 110 – Spring 2012 . there would be multiple test cases to cover all possible cases of switch statement. • Thus.Statement coverage: Switch • Switch statement can be reduced to multiple two way if-then-else statements.

i. Test cases to achieve statement coverage for statements within a loop should be such that – They skip the loop completely. – They should exercise the loop between once and the maximum number of times. SE 110 – Spring 2012 . so that the situation of the termination condition being true before starting the loop is tested. the boundary value and values just above the boundary. around the boundary of the loop. – They should try to cover the loop.e. to check normal operations of the loop. with values just below the boundary.Statement coverage: loops while   until Typically loops fail in “boundary conditions”.

Example: GPA calculation float findGPA( Collection courses ) { int totalCreditHours = 0. Course courseName creditHours grade SE 110 – Spring 2012 } .grade == 'B' ) gradePoints = 3*theCourse. int gradePoints = 0.next( ). Iterator it=courses.grade == 'C' ) gradePoints = 2*theCourse.creditHours. return gradePointAverage .creditHours.grade == 'A' ) gradePoints = 4*theCourse.hasNext() ) { Course theCourse = (Course) it.creditHours.creditHours.iterator(). else if( theCourse.grade == 'F' ) gradePoints = 0*theCourse.creditHours. totalCreditHours += theCourse. } gradePointAverage = totalGradePoints / totalCreditHours. if( theCourse. else if( theCourse. else if( theCourse. while( it.grade == 'D' ) gradePoints = 1*theCourse. else if( theCourse. totalGradePoints += gradePoints. float gradePointAverage = 0.creditHours. int totalGradePoints = 0.

Statement coverage: Example Function: findGPA Test Case Id courseName IT-100 creditHours grade 4 A IT-101 F01-UT-01 4 B 4 A 4 D 4 F IT-102 IT-103 IT-104 Is the statement coverage 100%? SE 110 – Spring 2012 .

SE 110 – Spring 2012 .Statement coverage: Example Function: findGPA Test Case Id courseName IT-100 IT-101 F01-UT-02 creditHours grade 4 A 4 B 4 C 4 D 4 F IT-102 IT-103 IT-104 Statement coverage is 100%.

: IT-100 Loop count = 1 F01-UT-03 4 A courseName creditHours grade courseName creditHours grade Test Case Id.: F01-UT-04 Empty collection OOPS! (Divide by zero) Conclusion Loop count = 0 100% Statement coverage does not guarantee bug-free code SE 110 – Spring 2012 .Loop coverage: Example Test Case Id.: IT-100 IT-101 IT-102 IT-103 IT-104 Loop count > 1 F01-UT-02 4 A 4 B 4 A 4 D 4 F Test Case Id.

Branch coverage: Example Function: findGP Test Case Id courseName IT-100 IT-101 F01-UT-06 creditHours grade 4 A 4 B 4 C 4 D 4 F IT-102 IT-103 IT-104 How much is the branch coverage? SE 110 – Spring 2012 .

a test can start from the beginning and take any of the paths to its completion. SE 110 – Spring 2012 . • Path coverage splits the program into a number of paths. • Path coverage = (Total paths exercised/Total number of paths in program)*100.Path coverage • Statement Coverage may not indicate “true coverage”.

Path coverage: Example Flow chart for date validation routine • Some paths: – – – – – A B-D-G B-D-H B-C-E-H B-C-F-G SE 110 – Spring 2012 .

• Condition coverage = Total conditions exercised / total number of conditions in the program)*100. • Makes sure each constituent condition in a Boolean expression is covered. • Condition coverage is much stronger criteria than path coverage which in turn is a stronger criteria than statement coverage.Condition coverage • Further refinement of path coverage. • Protects against compiler optimizations. SE 110 – Spring 2012 .

Function coverage • Finds how many functions are covered by test cases. hence possible to achieve 100% coverage. • Function coverage = (Total functions exercised/Total number of functions in program)*100. SE 110 – Spring 2012 . • Leads more naturally to black box tests. • More “logical” than the other types of coverage. • Can be derived from “RTM”. • Higher level of abstraction. • Can also be a natural predecessor to performance testing. • Easy to prioritize as they are based on requirements.

SE 110 – Spring 2012 . Resource usage analysis. Dynamically generated code (e.g. Identifying memory leaks. Checking of critical sections.Code coverage techniques: Other uses • • • • • Performance analysis and optimization.. dynamic generation of URLs and security checking).

Code complexity testing • Which of the paths are independent? (If paths are not independent.) • Is there an upper bound on the number of tests to be executed to ensure that all the statements have been executed at least once? • Cyclomatic complexity is a metric that quantifies the complexity of the program and provides answers to some of these questions. we may be able to minimize the number of tests. SE 110 – Spring 2012 .

Cyclomatic complexity: Intuitive meaning • Provides an indication of how many “independent paths” are there in a program • What gives birth to new paths? • How can one count the new paths that are born? SE 110 – Spring 2012 .

” • Convert “complex predicates” to “simple predicates” (see next slide).Steps to compute cyclomatic complexity • Start with a conventional flow chart. • Identify the “decision points. • Make sure that all edges terminate in some node. • If there are loops. Nodes with two emanating edges are called predicate nodes. SE 110 – Spring 2012 . convert the loop conditions to simple predicates. • When a set of sequential statements are followed by a simple predicate. • Combine all the sequential statements into a single node in the flow graph. combine all the sequential statements and the predicate check into one node and have two edges emanating out of this node. adding a node to represent all the statements at the end of the program.

Converting complex predicates into simple predicates SE 110 – Spring 2012 .

Steps: Sample flow chart illustration Conventional flow chart Flow diagram for calculating Cyclomatic complexity SE 110 – Spring 2012 .

Cyclomatic complexity: definition • Formal definitions: – CC = # of predicate nodes + 1 – CC = E – N + 2 (E = edges. N = nodes) • Intuitive definition: – When a condition is introduced. it adds a new path – See the next two slides SE 110 – Spring 2012 .

• Cyclomatic complexity = P + 1 = 1 End SE 110 – Spring 2012 . • # of edges. • Cyclomatic complexity = E – N + 2 = 1 • # of predicate nodes. P = 0. E = 1. N = 2.Example #1 Hypothetical diagram with no decision nodes • # of independent paths = 1 • # of nodes.

• Cyclomatic complexity = P + 1 = 2 End SE 110 – Spring 2012 . E = 4.Example #2 Hypothetical diagram with one decision node • # of independent paths = 2 • # of nodes. • # of edges. P = 1. • Cyclomatic complexity = E – N + 2 = 2 • # of predicate nodes. N = 4.

• Find the basis set and write the test cases to execute all the paths in the basis set. • A set of independent paths that cover all the edges is called a basis set. SE 110 – Spring 2012 .Independent paths • An independent path is a path in the flow graph that has not been traversed before in other paths.

cost / effort to maintain is medium Very complex. cost / effort to maintain is high 10-20 20-40 >40 Not testable.Cyclomatic complexity: Interpretation Complexity 1-10 What it means Well-written code. any amount of money / effort to maintain may not be SE 110 – Spring 2012 enough . cost / effort to maintain is low Moderately complex. testability is high. testability is medium. testability is low.

White-box testing: Challenges • Developers “not finding time for testing” • They develop blind spots for their own defects • Fully covered code may not correspond to realistic scenarios SE 110 – Spring 2012 .