Validating and Improving TestTestCase Effectiveness

Yuri Chernak

Presented by Michelle Straughan

Overview
  

Introduction TestTest-Case Effectiveness Metric (TCE) Improving Test-Case Effectiveness Test

5 steps to improving TCE

Summary

Introduction


Software testing is crucial to the success of product Evaluation certifies product usage
 

Verification of test specifications Validation of test cases

 

New metric for evaluating test case Framework for test-case effectiveness test-

Improving Test-Case Effectiveness TestTestTest-Case Effectiveness Metric



Formulation of metric to measure the effectiveness of testing Metric Definition: TCE = Ntc /Ntot * Definition: 100% TCE value compared to a baseline to determine effectiveness (eg 75% baseline)

Improving Test-Case TestEffectiveness



Analyze causes and areas improvements are needed Proposed improvement framework based on test-escapes (defects test case suite did testnot find but were side effects found in the same test cycle)

Approach relies on analysis of missed defects by test cases

5 steps to Improvement
1.

2. 3.

4. 5.

Thoroughly understanding and properly documenting the test process Make Assumptions Gathering defect data and performing causal analysis Identifying main factors Implementing corrective actions

Step 1: Thoroughly understanding and properly documenting the test process

Stages in Test Process
Test Planning Test Design Test preparation and executionexecution-

Description
defines scope, objective and test approach designs test cases preparation and setup of test environment, execution of test cases, finding and reporting defects analyzing testing results

Deliverables
Test Plan Test-Case TestSpecifications Defect Report

Test evaluation and improvement

Summary Report

Step 2: Make Assumptions



After completing step 1, project team makes assumptions about factors affecting TCE Test Planning


Test objectives defined in test plan should be traceable to functional specifications Incomplete specifications lead to an incomplete test plan and leads to reduced TCE

Step 2: Make Assumptions



Test Design


 

Necessary test cases to test logic of functional features need to be defined, if not adequate coverage will be lacking Test case specifications should correspond to business rules Deficiencies could be caused by:
  

Incomplete test suite (all possible test cases are not defined) Incomplete test cases proper logic is not tested negative test cases Incorrect test-specifications test-

Test Preparation and Execution

 

Test cases not executed Test cases improperly executed

Factors affecting test-case testeffectiveness

Figure 1: Factors affecting test-case effectiveness test-

Step 3: Gathering defect data and performing causal analysis

Figure 2: Factors based on causal analysis logic

Step 3: Gathering defect data and performing causal analysis



    

Testers need to track defects and categorize defects. Test-escapes are Testfurther categorized according to (fig. 2) Understand reasons for missed defects Verify completeness of test specifications Verify coverage of test-cases testCheck for test deficiencies Verify the correctness of test cases

Step 4: Identifying main factors



Identifying factors that are causing major defects Analyze these defects and focus on these as most needing corrective measures

Step 4: Identifying main factors

Figure 3: Pareto chart

Step 5: Implementing corrective actions



Depending on project at hand, corrective measures could vary. These may include:
 

Revisiting and revising functional specifications, which results in revising test case specifications Use of traceability matrix to meet coverage needs of the project ensures that test cases meet business rule Training for testers on proper design techniques, using checklists to help better design test case specifications (involves revisiting and revising test case specifications) Training for testers on proper way to execute tests and verifying test results

Summary


New framework is to aid in better effectiveness of test-cases, however its not testapplicable to some projects as it requires large number of defects Framework would theoretically help to reduce defects present in a product when released to production, since it first checks the test process and if its not effective then provides steps for achieving effectiveness Formalized way of achieving effectiveness, other ways for improving effectiveness could include the use of other testing techniques