You are on page 1of 33

Unit and Integration Testing Strategies for C Programs Using Mutation-Based Criteria

Auri Marcelo Rizzo Vincenzi

Agenda
Introduction Mutation testing Experiment escription !esting Strategies
Score"Cost Incrementa# !esting Strategy Su$$icient Incrementa# !esting Strategy

Conc#usion and %uture &or'


Mutation 2000 Symposium San Jose CA October, 2000 2

Introduction
!est Acti)ities
*#anning !est Case esign !est Case Execution +esu#t Ana#ysis

!esting Strategies
Incrementa# strategy
Mutation 2000 Symposium San Jose CA October, 2000 (

Introduction
!est esign !ec-ni.ues
%unctiona# /b#ac' box testing0 Structura# /1-ite box testing0 Error based
Comp#ementary tec-ni.ues

Mutation 2000 Symposium San Jose CA October, 2000

Introduction
!esting *-ases
3nit testing Integration testing System testing How we should combine different testing criteria, covering unit and integration testing phases, using a low cost and effective strategy?
Mutation 2000 Symposium San Jose CA October, 2000 2

Mutation !esting
Mutation Ana#ysis /MA0
3nit testing Proteum too# $or C programs
56 mutant operators
Statement /620 Operator /,40 7ariab#e /50 Constant /(0

Mutation 2000 Symposium San Jose CA October, 2000

Mutation !esting
Inter$ace Mutation /IM0
App#y t-e same concepts o$ mutation testing at t-e interprocedura# #e)e#

Integration Errors
A
SI(B) incorrect

A
SI(B) incorreto

Incorrect output SO(B) incorrect

Incorrect output SO(B) incorrect

Incorrect output

Mutation 2000 Symposium San Jose CA October, 2000

Mutation !esting
Inter$ace Mutation
PROTEUM/IM too# $or C programs (( mutant operators
9roup"I /2,0 : Mutations inside t-e body o$ $unction B /inter$ace commands, g#oba# and inter$ace )ariab#es0 9roup"II /;0 : Mutations 1-ere A ca##s B
A Group-II

Mutation 2000 Symposium San Jose CA October, 2000

Group-I

Mutation !esting
Inter$ace Mutation
9roup"I mutants Mutants associated to a gi)en ca## s-ou#d be 'i##ed on#y i$ t-at ca## is been executed

Mutation 2000 Symposium San Jose CA October, 2000

Mutation !esting
*rob#ems 1it- Mutation !esting
<ig- number o$ mutants generated
!ime to execute against one or more test cases Ana#y=e t-e e.ui)a#ent mutants

A#ternati)e Criteria
+andom#y Mutation Se#ecti)e Mutation Constrained Mutation
Su$$icient Mutant Operators Set Mutation 2000 Symposium San Jose CA October, 2000 60

Experiment escription
Steps
*rogram Se#ection !oo# Se#ection !est Set 9eneration Strategies +esu#ts and ata Ana#ysis

Mutation 2000 Symposium San Jose CA October, 2000

66

Experiment escription
*rogram Se#ection

!oo# Se#ection
Proteum /Mutation Ana#ysis unit testing0 PROTEUM/IM /Inter$ace Mutation integration testing0 Mutation 2000 Symposium San Jose CA October, 2000 62

Experiment escription
!est Set 9eneration
!1o poo#s o$ test cases $or eac- program
3nit testing Integration testing

Eac- poo# is composed by


ad hoc test cases based on t-e program speci$ication +andom test cases ta'e $rom &ong>s experiments !est cases generated by -and
Ana#y=ing t-e #i)e mutants and determining t-e e.ui)a#ent ones

!-e entire poo# is an ade.uate test set 1?r?t? MA or IM Mutation 2000 Symposium San Jose CA October, 2000 6(

Experiment escription
!est Set 9eneration
%rom eac- poo#, e#e)en ade.uate test set 1as generated $or eac- program and eac- criteria
@o minimi=ation tec-ni.ue -as been app#ied

Mutation 2000 Symposium San Jose CA October, 2000

6,

Experiment escription
Strategies

Mutation 2000 Symposium San Jose CA October, 2000

62

Experiment escription
Intra"p-ases !esting Strategies
Score"Cost !esting Strategy
App#ied $or eac- criterion indi)idua##y
integration #e)e# unit #e)e#

!1o .uestionsA
How to select a subset of operators keeping the mutation score as high as possible without reducing the overall effectiveness? How to achieve a mutation score of 1. , with a minimal cost?

Mutation 2000 Symposium San Jose CA October, 2000

64

Experiment escription
Su$$icient !esting Strategy
App#ied $or eac- criterion indi)idua##y
unit #e)e# integration #e)e#

Inter"p-ase !esting Strategies


Combine bot- criteria, co)ering unit and integration testing p-ases
3nit"Integration !esting Strategy Su$$icient 3nit"Integration !esting Strategy Mutation 2000 Symposium San Jose CA October, 2000 65

Experiment escription
Score"Cost !esting Strategy
Estab#is- an incrementa# strategy o$ app#ication
E)a#uate t-e e$$ecti)eness o$ eac- operator 1?r?t? t-e $u## set o$ operators Ca#cu#ate t-e ratio bet1een t-e mutation score and t-e cost o$ eac- operator
ata co##ected $or one program IM

Mutation 2000 Symposium San Jose CA October, 2000

68

Experiment escription
Score"Cost !esting Strategy

A)erage data $or t-e 2 programs IM

Mutation 2000 Symposium San Jose CA October, 2000

6;

Experiment escription
Integration !esting
Score"Cost Inter$ace !esting Strategy /!"#!0
!"#! Strategy %irst Interaction

Mutation 2000 Symposium San Jose CA October, 2000

(( operators IM"ade.uate test set

20

Experiment escription
!"#! !esting Strategy %ina# Step

60 operators mutation score 0?;; /6(B o$ t-e e.ui)a#ent mutants0

22 operators IM"ade.uate test set /54B o$ t-e e.ui)a#ent mutants0

Mutation 2000 Symposium San Jose CA October, 2000

26

Experiment escription
3nit !esting
Score"Cost 3nit !esting Strategy /!"$!0

Mutation 2000 Symposium 65 operators mutation score 0?;; San Jose CA October, 2000 /2?2B o$ t-e e.ui)a#ent mutants0

(6 operators MA"ade.uate test set /85B o$ t-e e.ui)a#ent mutants0

22

Experiment escription
Su$$icient !esting Strategy
MA"Su$$icient Operator Set
SM!C, SS C, OEDA, O++@, 7!& and 7 !+
MA"Su$$icient MA

Mutation 2000 Symposium San Jose CA October, 2000

2(

Experiment escription
3nit !esting
Su$$icient 3nit !esting Strategy /!$!0

2 operators mutation score o$ 0?;; /65B o$ t-e e.ui)a#ent mutants0 20 operators MA"ade.uate test set /58B o$ t-e e.ui)a#ent mutants0

Comparing wit SCUS Strateg! MA"ade.uate testA C+ E (2B and E. E 85B Mutation Score o$ 0?;;A C+ E ;(B and E. E 2?2B

Mutation 2000 Symposium San Jose CA October, 2000

2,

Experiment escription
Su$$icient !esting Strategy
IM"Su$$icient Operator Set
ArgAri@eg, Co)A##@od, ir7arDit@eg, Ind7arDit@eg, Ind7ar+ep9#o, Ind7ar+epExt and Ind7ar+ep+e.
IM"Su$$icient IM

Mutation 2000 Symposium San Jose CA October, 2000

22

Experiment escription
Integration !esting
Su$$icient Inter$ace !esting Strategy /!#!0

4 operators mutation score o$ 0?;; /62B o$ t-e e.ui)a#ent mutants0

Comparing wit SCIS Strateg! IM"ade.uate testA C+ E 2,B and E. E 54B Mutation Score o$ 0?;;A C+ E 85B and E. E 6(B

Mutation 2000 Symposium 20 CA operators IM"ade.uate2000 test set San Jose October, /5,B o$ t-e e.ui)a#ent mutants0

24

Experiment escription
3nit"Inter$ace !esting Strategy /$%#!0
MA and IM are incomparab#e 1?r?t? subsume re#ation

Mutation 2000 Symposium San Jose CA October, 2000

25

Experiment escription
3nit"Inter$ace !esting Strategy /$%#!0

, operators mutation score o$ 0?;; /,?2B o$ t-e e.ui)a#ent mutants0

"#erall results $%#! generates 65,552 /C+ E (,B0 5,,42 unit #e)e# 60,6(0 interation #e)e# $%#! produces MA and IM"ade.uate test sets

Mutation 2000 Symposium 65CA operators IM"ade.uate2000 test set San Jose October, /5,B o$ t-e e.ui)a#ent mutants0

28

Experiment escription
Su$$icient 3nit"Inter$ace !esting Strategy /!$%#!0

"#erall results !$%#! generates 4,040 /C+ E 55B0 2,(0, unit #e)e# (,524 interation #e)e# $%#! produces test cases sets t-at detemines mutation score o)er 0?;; $or bot- MA and IM criteria

Mutation 2000 Symposium San Jose CA October, 2000

2;

Experiment escription
!esting Strategies
Summary o$ t-e +esu#ts

Mutation 2000 Symposium San Jose CA October, 2000

(0

Conc#usion
MA and IM are comp#ementary criteria
!"#!, !#!, !"$! and !$! a##o1 t-e tester using t-e mutant operators according 1it- t-e budget restrictions and 1it- t-e mutation score t-at is expected to be obtained $%#! co)ers t-e unit and integration testing p-ases and produces AM and IM"ade.uate test sets !$%#! co)ers a#so t-e unit and integration testing p-ases and a mutation score o)er 0?;; is obtained
Mutation 2000 Symposium San Jose CA October, 2000 (6

%uture &or'
%urt-er studies to in)estigate t-e sca#abi#ity o$ t-ese resu#ts to #arge programs and ot-er app#ication domains In)estigate t-e e$$ecti)eness o$ t-ese strategies Impro)e t-e 1ay to se#ect t-e operators considering t-e cost 1?r?t? t-e number o$ e.ui)a#ent mutants and t-e e$$ecti)eness o$ eac- operator Estab#is- -euristics to dea# 1it- e.ui)a#ent mutants
Mutation 2000 Symposium San Jose CA October, 2000

(2

oubts, FuestionsGGG

Mutation 2000 Symposium San Jose CA October, 2000

((