Code and Functional Coverage Tutorial: Explaining The System.... Explaining The System...

Code and Functional Coverage
Tutorial
May '99
Shmuel Ur Shmuel Ur
IBM Research Laborat ory i n Hai fa IBM Research Laborat ory i n Hai fa
Explaining the System.... Explaining the System.... Explaining the System.... Explaining the System....
1-2
Outline Outline Outline Outline
Introduction to Coverage Introduction to Coverage
Code Coverage Code Coverage
Regression Regression
Functional Coverage Functional Coverage
Class Exercise Class Exercise
Automatic Coverage Directed Generation Automatic Coverage Directed Generation
Comparing Functional Coverage to Code Coverage Comparing Functional Coverage to Code Coverage
Conclusions Conclusions

What is Coverage? What is Coverage? What is Coverage? What is Coverage?
3-4
Testing is based on samples Testing is based on samples
Cannot run all possible tests Cannot run all possible tests
Need to know that all areas of design are tested Need to know that all areas of design are tested
A test suite is not measured by its size A test suite is not measured by its size
Solution: Coverage Analysis Solution: Coverage Analysis
Why Coverage? Why Coverage? Why Coverage? Why Coverage?
Coverage Coverage is any metric of completeness with respect to is any metric of completeness with respect to
a test selection criterion. a test selection criterion.
The main ideas behind coverage: The main ideas behind coverage:
Systematically create a list of tasks (the testing Systematically create a list of tasks (the testing
requirements) requirements)
Check that each task is covered during the testing Check that each task is covered during the testing
Definition of Coverage Definition of Coverage Definition of Coverage Definition of Coverage
5-6
Coverage Task Coverage Task - a Boolean function on a trace - a Boolean function on a trace
Statement 534 was executed Statement 534 was executed
A plane landed on Friday the 13th A plane landed on Friday the 13th
Coverage Model Coverage Model - a cohesive set of coverage tasks - a cohesive set of coverage tasks
Execution of each statement in a program Execution of each statement in a program
A plane landed A plane landed every day of the year every day of the year
I IIlegal Task Ilegal Task - a coverage task that should not occur - a coverage task that should not occur
Task Coverage List (TCL) Task Coverage List (TCL) - a list of all the legal tasks - a list of all the legal tasks
Other Definitions Other Definitions Other Definitions Other Definitions
Coverage Goals Coverage Goals Coverage Goals Coverage Goals
Measure the "quality" of a set of tests Measure the "quality" of a set of tests
Supplement test specifications by pointing to Supplement test specifications by pointing to
untested areas untested areas
Help create regression suites Help create regression suites
Provide a stopping criteria for unit testing Provide a stopping criteria for unit testing
7-8
Risks of Using Coverage Risks of Using Coverage Risks of Using Coverage Risks of Using Coverage
Using coverage without commitment to using the Using coverage without commitment to using the
results results
Generating simple tests to cover specific uncovered Generating simple tests to cover specific uncovered
tasks tasks
The painted wall analogy: The painted wall analogy:

Some coverage models are ill-suited to deal with Some coverage models are ill-suited to deal with
common problems common problems
Missing code Missing code
Low coverage goals Low coverage goals
Automatic Test Generation and Coverage Automatic Test Generation and Coverage Automatic Test Generation and Coverage Automatic Test Generation and Coverage
Main goal is to improve biasing Main goal is to improve biasing
Working with feedback (closed loop) Working with feedback (closed loop)
Avoiding the risk of specific biasing Avoiding the risk of specific biasing
9-10
Provides a quantitative measurement of the testing Provides a quantitative measurement of the testing
effort effort
Can assist in directing the tester's future efforts Can assist in directing the tester's future efforts
Can demonstrate redundancy in test cases Can demonstrate redundancy in test cases
Can be used as entry/exit criteria between test phases Can be used as entry/exit criteria between test phases
Studies have demonstrated a direct correlation between Studies have demonstrated a direct correlation between
increased code coverage and defect detection increased code coverage and defect detection
What is the Value of Code Coverage? What is the Value of Code Coverage? What is the Value of Code Coverage? What is the Value of Code Coverage?
11-12
Types of Code Coverage Models Types of Code Coverage Models Types of Code Coverage Models Types of Code Coverage Models
Control flow Control flow
Check that the control flow of the program has been fully Check that the control flow of the program has been fully
exercised exercised
Data flow Data flow
Models that look at the flow of data in, and between, Models that look at the flow of data in, and between,
programs programs
Mutation Mutation
Models that check directly for common bugs Models that check directly for common bugs
Visual coverage models Visual coverage models
Program types - OO, concurrent, distributive Program types - OO, concurrent, distributive
Control Flow Models Control Flow Models Control Flow Models Control Flow Models
Routine (function entry) Routine (function entry)
Function call Function call
Function return Function return
Statement (block) Statement (block)
Branch Branch
Multi-condition Multi-condition
Loop Loop
13-14
C-Use (Computational use) C-Use (Computational use)
P-Use (Predicate variable use) P-Use (Predicate variable use)
All Uses All Uses
Data Flow Models Data Flow Models Data Flow Models Data Flow Models
Weak Mutation Weak Mutation
Strong Mutation Strong Mutation
Examples: Examples:
<= == <= ==
> >= > >=
& && & &&
Mutation (Relational Operation) Coverage Mutation (Relational Operation) Coverage Mutation (Relational Operation) Coverage Mutation (Relational Operation) Coverage
15-16
Hierarchy of Coverage Models Hierarchy of Coverage Models Hierarchy of Coverage Models Hierarchy of Coverage Models
Function Entry Function Entry
Function Return Function Return
Block Block
Function Call Function Call
Decision Decision
P-Use P-Use
All-Uses All-Uses
C-Use C-Use
Hardware Models Hardware Models Hardware Models Hardware Models
State machine coverage - states, transitions State machine coverage - states, transitions
All values for signals All values for signals
Stuck At Stuck At
17-18
What is What is What is What is x x x x Suds (from Bellcore)? Suds (from Bellcore)? Suds (from Bellcore)? Suds (from Bellcore)?
Principally a code coverage tool suite Principally a code coverage tool suite
Parses C and C++ source code Parses C and C++ source code
Runs on AIX, Solaris, Linux, WinNT Runs on AIX, Solaris, Linux, WinNT
x xATAC ATAC - a test coverage measurement, management & test - a test coverage measurement, management & test
creation tool creation tool
x xVue Vue - supports software maintenance by visualizing code - supports software maintenance by visualizing code
features features
x xSlice Slice - a dynamic program slicing debugger that graphically - a dynamic program slicing debugger that graphically
localizes bugs localizes bugs
x xProf Prof - an execution count-based profiler - an execution count-based profiler
x xDiff Diff - displays differences between two files in a more - displays differences between two files in a more
understandable way understandable way
xFind xFind - uses date seeds to locate Y2K sensitivities in code - uses date seeds to locate Y2K sensitivities in code
xRegress xRegress - assists in the selection of a minimum set of tests - assists in the selection of a minimum set of tests
to cover changed code to cover changed code
xSuds Tool Suite Components xSuds Tool Suite Components xSuds Tool Suite Components xSuds Tool Suite Components
19-20
How Does How Does How Does How Does x x x x Suds Work? Suds Work? Suds Work? Suds Work?
21-22
23-24
What is Missing from What is Missing from What is Missing from What is Missing from x x x x Suds? Suds? Suds? Suds?
Signing on coverage tasks Signing on coverage tasks
Removing covered coverage tasks (to improve Removing covered coverage tasks (to improve
performance) performance)
Coverage models Coverage models
Operating systems Operating systems
Programming languages Programming languages
Creating Your Own Code Coverage Tool Creating Your Own Code Coverage Tool Creating Your Own Code Coverage Tool Creating Your Own Code Coverage Tool
Avoid if possible: Avoid if possible:
It is a lot of work It is a lot of work
It is bug prone It is bug prone
It is slow It is slow
When creating such a tool, consider: When creating such a tool, consider:
Instrumentation Instrumentation
Traces Traces
Off-line vs. on-line task updating Off-line vs. on-line task updating
Reducing the size of traces Reducing the size of traces
Integrating information Integrating information
25-26
Test Coverage for Visual Programs Test Coverage for Visual Programs Test Coverage for Visual Programs Test Coverage for Visual Programs
Visual elements to be covered: Visual elements to be covered:
Connections Connections
Events Events
Actions Actions
Attributes Attributes
Path Coverage - First View Path Coverage - First View Path Coverage - First View Path Coverage - First View
27-28
Path Coverage - Second View Path Coverage - Second View Path Coverage - Second View Path Coverage - Second View
A TEST!!! A TEST!!! A TEST!!! A TEST!!!
From Cem Kaner "Software negligence and testing coverage" From Cem Kaner "Software negligence and testing coverage" From Cem Kaner "Software negligence and testing coverage" From Cem Kaner "Software negligence and testing coverage"
A program fails in the field and someone dies. At the A program fails in the field and someone dies. At the
trial, the QA Manager takes the stand and the trial, the QA Manager takes the stand and the
plaintiff's lawyer states three facts: plaintiff's lawyer states three facts:
Failure occurred when the program reached a specific line of code. Failure occurred when the program reached a specific line of code.
This line had never been tested, therefore the bug had not been This line had never been tested, therefore the bug had not been
found before the product was released. found before the product was released.
Had the QA used the coverage monitor and tested to complete Had the QA used the coverage monitor and tested to complete
coverage, this bug would have been found and the victim would be coverage, this bug would have been found and the victim would be
alive today. alive today.
The plaintiff's lawyer therefore proved that the The plaintiff's lawyer therefore proved that the
company was negligent and that the victim's family company was negligent and that the victim's family
would win the lawsuit. would win the lawsuit.
29-30
A A regression suite regression suite is a set of tests that are run on the is a set of tests that are run on the
application after every software or data change, and application after every software or data change, and
every night or every weekend, in order to check that no every night or every weekend, in order to check that no
new bugs have been introduced. new bugs have been introduced.
This is vital because every bug fix, on the average, This is vital because every bug fix, on the average,
introduces one fifth of a bug introduces one fifth of a bug. .
Regression Regression Regression Regression
31-32
Contradictory Requirements Contradictory Requirements Contradictory Requirements Contradictory Requirements
A regression suite must be: A regression suite must be:
Comprehensive so that it is likely to catch all the Comprehensive so that it is likely to catch all the
!@#$% bugs introduced !@#$% bugs introduced
Small so that it can economically be executed many Small so that it can economically be executed many
times times
How can we make our regression suite small and How can we make our regression suite small and
comprehensive? comprehensive?
Let S = {1 ... M} Let S = {1 ... M}
Let T = {T Let T = {T
1 1 ... T ... T
n n} be a set of subset of S } be a set of subset of S
The set cover problem: The set cover problem:
Find the smallest subset of T that covers S Find the smallest subset of T that covers S
The set cover problem is NP-C, however, there are a The set cover problem is NP-C, however, there are a
number of good algorithms for it. number of good algorithms for it.
The Set Cover Problem The Set Cover Problem The Set Cover Problem The Set Cover Problem
33-34
Find all the tests that contain unique tasks Find all the tests that contain unique tasks
Loop Loop
Remove all the tasks covered by selected tests Remove all the tasks covered by selected tests
Choose the test that covers most remaining tasks Choose the test that covers most remaining tasks
Implementation issues: Implementation issues:
Input type (sparse or dense) Input type (sparse or dense)
Frequency of use Frequency of use
Real data vs. random data Real data vs. random data
The Greedy Heuristic The Greedy Heuristic The Greedy Heuristic The Greedy Heuristic
Compaction Strategies Compaction Strategies Compaction Strategies Compaction Strategies
Simple Simple - if a test introduces a new task, choose it. - if a test introduces a new task, choose it.
Merge N Merge N - add every set of N new tests to the - add every set of N new tests to the
regression suite, and compact them into a new regression suite, and compact them into a new
regression suite. (Size may decrease!) regression suite. (Size may decrease!)
35-36
Insight on Regression Suite Compaction Insight on Regression Suite Compaction Insight on Regression Suite Compaction Insight on Regression Suite Compaction
An improvement of fifty percent is to be expected in An improvement of fifty percent is to be expected in
Merge Merge over over Simple. Simple.
Real information does not behave in the same way as Real information does not behave in the same way as
randomly generated information. randomly generated information.
The real resource limitation is space. The real resource limitation is space.
Incremental compaction is useful. Incremental compaction is useful.
The size of the regression suite does not change by The size of the regression suite does not change by
much, once the set is fully covered. much, once the set is fully covered.
37-38
Functional Coverage Functional Coverage Functional Coverage Functional Coverage

Summary of ATAC on a Number of Tests Summary of ATAC on a Number of Tests Summary of ATAC on a Number of Tests Summary of ATAC on a Number of Tests
39-40
Did every agency asked for every type of Did every agency asked for every type of
ticket? ticket?
Did the agencies ask for service at the same Did the agencies ask for service at the same
time? time?
How many times did Lion wait more than 10 How many times did Lion wait more than 10
minutes? minutes?
Is the system fair? Is the system fair?
Coverage Questions Not Answered by Coverage Questions Not Answered by Coverage Questions Not Answered by Coverage Questions Not Answered by
Code Coverage Tools Code Coverage Tools Code Coverage Tools Code Coverage Tools
Example of a Simple Functional Example of a Simple Functional Example of a Simple Functional Example of a Simple Functional
Coverage Model Coverage Model Coverage Model Coverage Model
A model is composed of the following parts: A model is composed of the following parts:
A semantic description (story) of the functional coverage A semantic description (story) of the functional coverage
model which contains attributes model which contains attributes
Has every agency {Agency name} Has every agency {Agency name}
asked for every type of show {Service type} asked for every type of show {Service type}
A tuple of attributes <Agency name, Service type> A tuple of attributes <Agency name, Service type>
Possible values for each attribute (e.g., Agency name: Possible values for each attribute (e.g., Agency name:
Disney, Fox, STAR, LION) Disney, Fox, STAR, LION)
Restrictions on the cross-product Restrictions on the cross-product
41-42
What are Attributes Made Out Of? What are Attributes Made Out Of? What are Attributes Made Out Of? What are Attributes Made Out Of?
Functional coverage attributes may come from any Functional coverage attributes may come from any
relevant source. relevant source.
Our favorite sources include: Our favorite sources include:
Inputs Inputs
Outputs Outputs
Program variables Program variables
Functions on program variables Functions on program variables
High level descriptions not mapped to variables High level descriptions not mapped to variables
"Homemade" Functional Coverage "Homemade" Functional Coverage "Homemade" Functional Coverage "Homemade" Functional Coverage
Implementation Implementation Implementation Implementation
Instrument your code with the following statement: Instrument your code with the following statement:
print("Agency name = %d Service type = %d \n",agency,service); print("Agency name = %d Service type = %d \n",agency,service);
Execute > trace Execute > trace
cat *.trace | sort -unique > output cat *.trace | sort -unique > output
What are the problems? What are the problems?
43-44
Simple Functional Coverage - Simple Functional Coverage - Simple Functional Coverage - Simple Functional Coverage -
Attributes Attributes Attributes Attributes
Cases Cases Cases Cases
45-46
/* example of call: traceFunc("showbiz",agency,service); */ /* example of call: traceFunc("showbiz",agency,service); */
#include <stdio.h> #include <stdio.h>
#include <sys/types.h> #include <sys/types.h>
#include <sys/errno.h> #include <sys/errno.h>
#include <sys/time.h> #include <sys/time.h>
FILE *trace_file_out; FILE *trace_file_out;
void traceFunc(char *modelname,char *agency,int service) { void traceFunc(char *modelname,char *agency,int service) {
struct timeval val; struct timeval val;
struct timezone zone; struct timezone zone;
char filename[100]; char filename[100];
static int called = 0; static int called = 0;
if ( called == 0 ) { if ( called == 0 ) {
gettimeofday(&val,&zone); gettimeofday(&val,&zone);
sprintf(filename,"%s_trace_%d",modelname,val.tv_sec); sprintf(filename,"%s_trace_%d",modelname,val.tv_sec);
trace_file_out = fopen(filename,"w"); trace_file_out = fopen(filename,"w");
} }
called==1; called==1;
fprintf(trace_file_out,"%s %d \n",agency,service); fprintf(trace_file_out,"%s %d \n",agency,service);
return; return;
} }
Functions Functions Functions Functions
Elementary Report Generation Screen for Elementary Report Generation Screen for Elementary Report Generation Screen for Elementary Report Generation Screen for
the Ticket Example the Ticket Example the Ticket Example the Ticket Example
47-48
A Basic Functional Coverage Report A Basic Functional Coverage Report A Basic Functional Coverage Report A Basic Functional Coverage Report
More Functional Models for the Simple More Functional Models for the Simple More Functional Models for the Simple More Functional Models for the Simple
Example Example Example Example
1. 1. <Agency name, Reply> <Agency name, Reply>
2. 2. <Agency name, Reply, Service> <Agency name, Reply, Service>
3. 3. <Agency name, Reply, Service, Time> <Agency name, Reply, Service, Time>
4. 4. <Agency1 calling, Agency2 calling, Agency3 calling, <Agency1 calling, Agency2 calling, Agency3 calling,
Agency4 calling> Agency4 calling>
5. 5. <Agency1 reply, Agency2 reply, Agency3 reply, <Agency1 reply, Agency2 reply, Agency3 reply,
Agency4 reply> Agency4 reply>
6. 6. <Service, Service> <Service, Service>
7. 7. Round Robin Bits Round Robin Bits
49-50
Arbitration Data Arbitration Data Arbitration Data Arbitration Data
Improvement Over Time Improvement Over Time Improvement Over Time Improvement Over Time
51-52
How Restrictions are Created How Restrictions are Created How Restrictions are Created How Restrictions are Created
The designer creates a list of restrictions for a model. The designer creates a list of restrictions for a model.
Coverage feedback modifies the restrictions. Coverage feedback modifies the restrictions.
Sub-models are used to economically check and Sub-models are used to economically check and
refine the restrictions. refine the restrictions.
Restrictions are Self-correcting Restrictions are Self-correcting Restrictions are Self-correcting Restrictions are Self-correcting
53-54
Restrictions are Created Hierarchically Restrictions are Created Hierarchically Restrictions are Created Hierarchically Restrictions are Created Hierarchically
Car Assembly Line Example Car Assembly Line Example Car Assembly Line Example Car Assembly Line Example
55-56
Simple Coverage Models Simple Coverage Models Simple Coverage Models Simple Coverage Models
< Car type, engine size, color> < Car type, engine size, color>
< Car type, total work time, total elapse time, day > < Car type, total work time, total elapse time, day >
< # cars in line, # of workers in line> < # cars in line, # of workers in line>
White Box Model - White Box Model - White Box Model - White Box Model -
Cars Location on the Assembly Line Cars Location on the Assembly Line Cars Location on the Assembly Line Cars Location on the Assembly Line
(CL1, CT1, CL2, CT2, CL3, CT3, CL4, CT4, CL5, CL6, (CL1, CT1, CL2, CT2, CL3, CT3, CL4, CT4, CL5, CL6,
CL7, Problem) CL7, Problem)
Problem: (0 = none, 1 = full stop, 2 = electrical shortage) Problem: (0 = none, 1 = full stop, 2 = electrical shortage)
So there are: 10 So there are: 10
. .
9 9
. .
8 8
. .
6 6
. .
5 5
. .
5 5
. .
3 3
. .
2 2
. .
2 2
. .
2 2
. .
2 2
. .
3 = 15,552,000 tasks ? 3 = 15,552,000 tasks ?! !
Car Location Car Type
1 0-9 Accord-Civic
2 0-8 Accord-Civic
3 0-7 Accord-Civic
4 0-5 Accord-Civic
5 0-4 Accord
6 0-4 Accord
7 0-2 Accord
57-58
Model Restrictions Model Restrictions Model Restrictions Model Restrictions
CL1 >= CL2 >= CL3 >= CL4 >= CL5 >= CL6 >= CL7 CL1 >= CL2 >= CL3 >= CL4 >= CL5 >= CL6 >= CL7
No car may pass another in the assembly line No car may pass another in the assembly line
If CT2 == Civic then CL2 > 4 or CL2 = 0 If CT2 == Civic then CL2 > 4 or CL2 = 0
Civic may only be at stages 5 to 9 for painting Civic may only be at stages 5 to 9 for painting
More Model Restrictions More Model Restrictions More Model Restrictions More Model Restrictions
Only one car can be at stages 3 and 7 Only one car can be at stages 3 and 7



Only one car can be at stages 1, 2 and 3 Only one car can be at stages 1, 2 and 3

Only two Accords can be at stages 4, 5 and 6 Only two Accords can be at stages 4, 5 and 6
Only 1415 legal tasks!!! Only 1415 legal tasks!!!
59-60
Restriction Implementation Restriction Implementation Restriction Implementation Restriction Implementation
Simulation Outputs Simulation Outputs Simulation Outputs Simulation Outputs
61-62
Simulation Outputs Simulation Outputs Simulation Outputs Simulation Outputs
Feedback and Bugs Found Feedback and Bugs Found Feedback and Bugs Found Feedback and Bugs Found
Not creating enough of the right kind of scenarios Not creating enough of the right kind of scenarios
Performance related bugs Performance related bugs
Illegal scenarios found Illegal scenarios found
63-64
Functional Coverage Models Functional Coverage Models Functional Coverage Models Functional Coverage Models
Archetypes Archetypes
Picture Picture
Black Box Black Box - based on interface details - based on interface details
White Box White Box - based on internal details - based on internal details
Broken Box Broken Box - combined internal and external - combined internal and external
information information
Model Classification Based on Information Model Classification Based on Information Model Classification Based on Information Model Classification Based on Information
Sources Sources Sources Sources
65-66
Model Classification Based on Temporal Model Classification Based on Temporal Model Classification Based on Temporal Model Classification Based on Temporal
Properties Properties Properties Properties
Snapshot Snapshot - - a single point in time a single point in time
Could be abstract. For example, if two routines are Could be abstract. For example, if two routines are
called sequentially, a model containing the values of called sequentially, a model containing the values of
variables from both routines is considered a snapshot variables from both routines is considered a snapshot
model even though the variables are not "alive" at the model even though the variables are not "alive" at the
same time. same time.
Temporal Temporal - - a coverage model that checks scenarios a coverage model that checks scenarios
Usually the type where event 1 happens first, then Usually the type where event 1 happens first, then
event 2 , and then event 3. Coverage is measured on event 2 , and then event 3. Coverage is measured on
the events (event 1 could have a few legal types), on the events (event 1 could have a few legal types), on
time between events, and on the way the events time between events, and on the way the events
interact. interact.
Special Functional Coverage Models Special Functional Coverage Models Special Functional Coverage Models Special Functional Coverage Models
Stress Stress - Coverage model that checks how many things - Coverage model that checks how many things
happen within a short interval of time, or at the same happen within a short interval of time, or at the same
time. Also a model that checks that resources have time. Also a model that checks that resources have
been used to their full capacity. been used to their full capacity.
Statistically oriented Statistically oriented - Coverage model used to - Coverage model used to
measure statistics such as how many times each task measure statistics such as how many times each task
has been covered. The main motivation is usually has been covered. The main motivation is usually
performance. (Do we need a stack that large?) performance. (Do we need a stack that large?)
67-68
Invent Your Own Model Invent Your Own Model Invent Your Own Model Invent Your Own Model

Ticket office Car assembly
Stress

Statistically
oriented
Snapshot Temporal
Black Box
White Box
Broken Box
Invent Your Own Model - cont. Invent Your Own Model - cont. Invent Your Own Model - cont. Invent Your Own Model - cont.
69-70
Model Hierarchy Model Hierarchy Model Hierarchy Model Hierarchy
Building a model hierarchically, out of sub-models, is Building a model hierarchically, out of sub-models, is
desirable for a number of reasons desirable for a number of reasons
Errors can be found earlier Errors can be found earlier
Implementation is cleaner and faster Implementation is cleaner and faster
Restrictions are more easily debugged Restrictions are more easily debugged
Sub-models should be made out of semantically Sub-models should be made out of semantically
related variables related variables
High level models can be reused many times High level models can be reused many times
Comet - COverage MEasurement Tool Comet - COverage MEasurement Tool Comet - COverage MEasurement Tool Comet - COverage MEasurement Tool
A generic functional coverage tool for use in a heavy A generic functional coverage tool for use in a heavy
industrial setting industrial setting
Development leaders: Amir Eliaz and Raanan Grinwald Development leaders: Amir Eliaz and Raanan Grinwald
71-72
Overview of Comet Overview of Comet Overview of Comet Overview of Comet
Measures coverage of a given set of event traces over Measures coverage of a given set of event traces over
user-defined models user-defined models
Based on a relational database as the repository for Based on a relational database as the repository for
coverage data coverage data
Outputs various types of coverage reports Outputs various types of coverage reports
Relational
Database
System Overview System Overview System Overview System Overview
COMET
Trace
Regresslon
Sultes
Vode|
Reports
73-74
Comet Tool Components Comet Tool Components Comet Tool Components Comet Tool Components
GUI components GUI components
Model Definition Model Definition
Coverage Measurement Control Coverage Measurement Control
Coverage Report Engine Coverage Report Engine
Insertion and measurement of coverage data engines Insertion and measurement of coverage data engines
Preparing the Domain Preparing the Domain Preparing the Domain Preparing the Domain
The Comet domain is a set of attributes with which The Comet domain is a set of attributes with which
coverage data is expressed coverage data is expressed
Based on the domain attributes, Comet creates Based on the domain attributes, Comet creates
database tables in the designated database database tables in the designated database
Event trace for storing traces Event trace for storing traces
Trace table for storing trace history information Trace table for storing trace history information
... ...
75-76
Populating the Database Populating the Database Populating the Database Populating the Database
The The trace analyzer trace analyzer is written, for every domain, to is written, for every domain, to
extract the attribute values out of the raw data. extract the attribute values out of the raw data.
The resulting The resulting traces traces are inserted into the database. are inserted into the database.
Every line in the trace is an event. Every line in the trace is an event.
.tst file .tst file trace file trace file
DP
Trace File Example Trace File Example Trace File Example Trace File Example
HOST ppc.halfa.lbm.com
F|LE defau|t.tst e|la
DOVA|N FP
TRACE
ATTRNAVES c|uster processor count mnemonlc opcode sslgn sfractlon
sblnfracsexponent stype sslgn sfractlon sblnfrac sexponent
stype sslgn sfractlon sblnfrac sexponent stype tslgn tfractlon
tblnfrac texponent ttype fpscrbefore roundmode fpscrafter
"fsub""""""""
""Norm""""
""""Norm""none""no
ne""none""none""""""
""Norm""""RoundTow
ardero"""
...
77-78
Functional Coverage Models in the Functional Coverage Models in the Functional Coverage Models in the Functional Coverage Models in the
Domain Paradigm Domain Paradigm Domain Paradigm Domain Paradigm
Comet models are composed of: Comet models are composed of:
A semantic description A semantic description
A query on the trace A query on the trace
A tuple of attributes A tuple of attributes
Model attributes are not necessarily domain Model attributes are not necessarily domain
attributes attributes
Restrictions Restrictions
Using SQL to mark some of the tasks as illegal Using SQL to mark some of the tasks as illegal
Model Definition Dialog Model Definition Dialog Model Definition Dialog Model Definition Dialog
79-80
Coverage Measurement Control Coverage Measurement Control Coverage Measurement Control Coverage Measurement Control
Monitoring the coverage process Monitoring the coverage process
Controlling the insertion of traces into the database Controlling the insertion of traces into the database
and which TCLs to measure them against and which TCLs to measure them against
Controlling various aspects of the measurement, Controlling various aspects of the measurement,
such as collecting illegal task information, optimizing, such as collecting illegal task information, optimizing,
etc. etc.
Coverage Measurement Dialog Coverage Measurement Dialog Coverage Measurement Dialog Coverage Measurement Dialog
81-82
Coverage Report Engine Tool Coverage Report Engine Tool Coverage Report Engine Tool Coverage Report Engine Tool
Querying the coverage data and getting specific cuts, Querying the coverage data and getting specific cuts,
views, statistics, graphs views, statistics, graphs
Examples (in the FP domain): Examples (in the FP domain):
Did we see all FP instructions? Did we see all FP instructions?
Did we see all rounding modes? Did we see all rounding modes?
Source and Target operand types Source and Target operand types
Exponent difference for addition and subtraction Exponent difference for addition and subtraction
Uncovered subspaces (holes in the coverage) Uncovered subspaces (holes in the coverage)
CRE Application - Basic Dialog CRE Application - Basic Dialog CRE Application - Basic Dialog CRE Application - Basic Dialog
83-84
Basic Report Results Basic Report Results Basic Report Results Basic Report Results
CRE Application - Progress Graph CRE Application - Progress Graph CRE Application - Progress Graph CRE Application - Progress Graph
85-86
CRE Application - 2D Scatter CRE Application - 2D Scatter CRE Application - 2D Scatter CRE Application - 2D Scatter
Scatter Report Results Scatter Report Results Scatter Report Results Scatter Report Results
87-88
CRE Application - Summaries CRE Application - Summaries CRE Application - Summaries CRE Application - Summaries
Comet Report Engine - Summary Report Comet Report Engine - Summary Report Comet Report Engine - Summary Report Comet Report Engine - Summary Report
89-90
91-92
Two types of Coverage Directed Two types of Coverage Directed Two types of Coverage Directed Two types of Coverage Directed
Generation Generation Generation Generation
Based on functional coverage Based on functional coverage
Create a functional coverage model Create a functional coverage model
Generate all the tasks Generate all the tasks
Transform the model into part of a test plan Transform the model into part of a test plan
Transform the tasks into tests Transform the tasks into tests
Based on an abstract model Based on an abstract model
Describe the application using a formal model (automata) Describe the application using a formal model (automata)
Derive coverage tasks from the formal model Derive coverage tasks from the formal model
Derive tests from the coverage tasks Derive tests from the coverage tasks
Expected results are built-in Expected results are built-in
Usually easier to apply the first method, some tools Usually easier to apply the first method, some tools
are available are available
The Problem The Problem The Problem The Problem
The test effort is a software project that requires The test effort is a software project that requires
design and implementation. design and implementation.
High level test design documents are required for High level test design documents are required for
effective reviews and maintainability. effective reviews and maintainability.
Because of a lean test budget, the complete test Because of a lean test budget, the complete test
design cycle is not implemented and quality is design cycle is not implemented and quality is
hindered. hindered.
93-94
A Solution for API Testing A Solution for API Testing A Solution for API Testing A Solution for API Testing
Functional models provide a simple language that Functional models provide a simple language that
enables enables high level high level description description of the test design. of the test design.
This level of description is easily reviewed and This level of description is easily reviewed and
maintained, thus increasing qualit maintained, thus increasing quality. y.
Low level test documents are produced automatically. Low level test documents are produced automatically.
Test implementation is straightforward, using low Test implementation is straightforward, using low
level test documents. Automatic generation of tests level test documents. Automatic generation of tests
is also possible. is also possible.
Time saved by the automation is used to define Time saved by the automation is used to define
stronger (e.g., time dependent) models. stronger (e.g., time dependent) models.
An Example An Example An Example An Example
A file system being tested has migrated to support A file system being tested has migrated to support
files that are bigger than 2 GB: files that are bigger than 2 GB:
There are two types of There are two types of file open file open: open and open64 : open and open64
The functional model is defined as: The functional model is defined as:
<file open, file size>
file open = {open, open64}
file size = {less than 2 GB, more than 2 GB}
95-96
Test Automation Test Automation Test Automation Test Automation
The attribute ' The attribute 'open file open file' is mapped to the class method: ' is mapped to the class method:
void open(StringBuffer apiName){
System.out.println(
apiName+ "(\"file1\", O_RDWR,
S_IRWXU)");
}
Test Automation Test Automation Test Automation Test Automation
open("open64") produces the call: open("open64") produces the call:
open64("file1", O_RDWR, S_IRWXU); open64("file1", O_RDWR, S_IRWXU);
The tool produces the following tuples to be tested: The tool produces the following tuples to be tested:
(open, less than 2 GB), (open64, less than 2 GB), etc. (open, less than 2 GB), (open64, less than 2 GB), etc.
When automatically producing tests from the tuples, When automatically producing tests from the tuples,
open64 is mapped to the method call open64 is mapped to the method call
open("open64") thus open("open64") thus producing the required code producing the required code
97-98
Model Based Automatic Testing - Model Based Automatic Testing - Model Based Automatic Testing - Model Based Automatic Testing -
GOTCHA Testing Methodology GOTCHA Testing Methodology GOTCHA Testing Methodology GOTCHA Testing Methodology
Input is a functional specification Input is a functional specification
Used to produce the model Used to produce the model
Generates abstract test case suite involving: Generates abstract test case suite involving:
Sequences of rules and states (assertions and post
conditions)
Runs tests Runs tests
Has the following benefits: Has the following benefits:
One hundred percent coverage of model
Early defect prevention:
Inconsistencies
Ambiguity
Incompleteness
TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing
Methodology - Test Design Methodology - Test Design Methodology - Test Design Methodology - Test Design
Functional
Specification
GOTCHA
Engine
GOTCHA
Abstract Test
Suite
GOTCHA
Behavior
Model
Early Defect
Discovery:
Inconsistency
Ambiguity
Incompleteness
TCBeans
Test
Object(s)
100%
Model
Coverage
99-100
TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing
Methodology - Test Execution Methodology - Test Execution Methodology - Test Execution Methodology - Test Execution
GOTCHA
Abstract Test
Suite
TCBeans
GOTCHA
Factory
Application
Under Test
Test
Results
TCBeans
Test
Object(s)
Start/Stop Test
Customize Test
Objects
Monitor Results
Defect
Discovery/Resolution
Model Adjustment
The GOTCHA Definition Language The GOTCHA Definition Language The GOTCHA Definition Language The GOTCHA Definition Language
Used to define a finite state machine
States
Transitions
Start/Final
Invariant
p p
p p
p p
p p
p p
p p
p p p p
p p
101-102
GOTCHA Suite Generation GOTCHA Suite Generation GOTCHA Suite Generation GOTCHA Suite Generation
I I I I nput nput
GOTCHA Definition Language model GOTCHA Definition Language model
Coverage model - projected state/transition Coverage model - projected state/transition
Definition of start and final states for tests Definition of start and final states for tests
Output Output
Test suite guaranteed to cover all coverable tasks Test suite guaranteed to cover all coverable tasks
List of reachable, but uncoverable, tasks List of reachable, but uncoverable, tasks
Test Test
Execution path from a start state to a final state Execution path from a start state to a final state
By-product By-product
Violated invariance encountered Violated invariance encountered
TC Beans GOTCHA Test Case Factory TC Beans GOTCHA Test Case Factory TC Beans GOTCHA Test Case Factory TC Beans GOTCHA Test Case Factory
JavaBeans Test Case Component JavaBeans Test Case Component
Input: GOTCHA abstract test suite (text) Input: GOTCHA abstract test suite (text)
Output: concrete test case execution and resu Output: concrete test case execution and resul lts ts
No addit No additional ional source files source files
No explicit compile and link phase No explicit compile and link phase
One hundred percent coverage of model One hundred percent coverage of model
Target and Expected object (JavaBeans) Target and Expected object (JavaBeans)
Rules = target method invocation Rules = target method invocation
States = expected property setting States = expected property setting
Process: Process:
Parse test suite Parse test suite
Execute Execute
Compare target and expected properties Compare target and expected properties
Interactive customization with BDK Interactive customization with BDK
103-104
A Use Case Scenario A Use Case Scenario A Use Case Scenario A Use Case Scenario
1. 1. Tester reviews functional specification. Tester reviews functional specification.
2. 2. Tester codes a behavior model in GOTCHA. Tester codes a behavior model in GOTCHA.
3. 3. Tester uncovers defects in specification. Tester uncovers defects in specification.
4. 4. Tester runs GOTCHA to create an abstract test suite. Tester runs GOTCHA to create an abstract test suite.
5. 5. Tester codes a Java class (JavaBeans): Tester codes a Java class (JavaBeans):
Wrapper methods to application methods Wrapper methods to application methods
Wrapper properties to application state variables Wrapper properties to application state variables
Static properties for interactive customization Static properties for interactive customization
6. 6. Tester runs TCBeans GOTCHA Test Case Factory. Tester runs TCBeans GOTCHA Test Case Factory.
7. 7. Tester customizes target properties. Tester customizes target properties.
8. 8. Tester runs test, uncovers lots of bugs, celebrates. Tester runs test, uncovers lots of bugs, celebrates.
9. 9. Tester analyzes structural coverage, adjusts GOTCHA Tester analyzes structural coverage, adjusts GOTCHA
model. model.
GOTCHA Hardware Test Study GOTCHA Hardware Test Study GOTCHA Hardware Test Study GOTCHA Hardware Test Study
Tests
0
50
100
150
200
250
300
350
T
r
a
n
s
i
t
i
o
n
s
ST Hand Tests
GOTCHA Transition Tests
Reachable Transitions
Cumulative Transition Coverage
105-106

Availability Availability Availability Availability
Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage
Code coverage Code coverage
Many tools are available for most operating Many tools are available for most operating
systems/languages systems/languages
Tools are hard to find or are very expensive for custom Tools are hard to find or are very expensive for custom
made/embedded environments made/embedded environments
Tools are hard to implement Tools are hard to implement
Functional coverage Functional coverage
Limited functionality tool available (Comet) Limited functionality tool available (Comet)
Works in any environment Works in any environment
Preparing a simple tool or script is easy Preparing a simple tool or script is easy
107-108
Users can benefit from the tools almost immediately Users can benefit from the tools almost immediately
The tools are easy to use; usually plug and play The tools are easy to use; usually plug and play
Need some time to learn how to define and implement models Need some time to learn how to define and implement models
Tools are not as easy to use Tools are not as easy to use
Need to have at least one person in the organization with a Need to have at least one person in the organization with a
working knowledge of functional coverage methodology working knowledge of functional coverage methodology
Takes about 3 days to acquire knowledge Takes about 3 days to acquire knowledge
Learning Curve Learning Curve Learning Curve Learning Curve
Cost of tools is usually low Cost of tools is usually low
Almost no education cost Almost no education cost
Slowdown of 10 - 100 percent Slowdown of 10 - 100 percent
Higher cost to develop tools and models Higher cost to develop tools and models
Higher education costs Higher education costs
Slowdown of a few percent (variable in # of models) Slowdown of a few percent (variable in # of models)
Cost Cost Cost Cost
109-110
Used by many projects and in many standards Used by many projects and in many standards
Papers on correlation between good coverage at unit level Papers on correlation between good coverage at unit level
and the number of integration bugs and the number of integration bugs
Refinements in tools as a result of experience Refinements in tools as a result of experience
New field, not a lot of experience New field, not a lot of experience
Harder to transfer experience from one project to another Harder to transfer experience from one project to another
Experience Experience Experience Experience
So Why Use Functional Coverage? So Why Use Functional Coverage? So Why Use Functional Coverage? So Why Use Functional Coverage?
F
C
111-112
Focus Focus Focus Focus
Uniformly spread over the entire program Uniformly spread over the entire program
Hard-coded coverage model hierarchy Hard-coded coverage model hierarchy
All models are local All models are local
Risk-driven Risk-driven
Models can be local or global Models can be local or global
Adapting to Testing Resources Adapting to Testing Resources Adapting to Testing Resources Adapting to Testing Resources
Hard to adapt; hard-coded models Hard to adapt; hard-coded models
Hard to adapt; very primitive focusing options Hard to adapt; very primitive focusing options
Easy to adapt Easy to adapt
Can use the right number of models with the desired Can use the right number of models with the desired
granularity granularity
Built-in model hierarchy Built-in model hierarchy
113-114
Impact on Project Impact on Project Impact on Project Impact on Project
Impacts testing process and testing requirements Impacts testing process and testing requirements
Impacts the design itself from early design stages Impacts the design itself from early design stages
Enables better understanding of programs and Enables better understanding of programs and
environments environments
Helps with test specifications and requirements Helps with test specifications and requirements
115-116
Which Coverage Method to Use? Which Coverage Method to Use? Which Coverage Method to Use? Which Coverage Method to Use?
High risk areas High risk areas
Complex, error prone areas Complex, error prone areas
Changes to existing code (maintenance) Changes to existing code (maintenance)
Used when cheaper than manual test inspection Used when cheaper than manual test inspection
Used for global coverage models Used for global coverage models
On all code that is written, given sufficient resources On all code that is written, given sufficient resources
As a criterion for finishing unit testing As a criterion for finishing unit testing
Conclusions Conclusions Conclusions Conclusions
Code coverage is easier to use, cheaper, and more Code coverage is easier to use, cheaper, and more
available than functional coverage. available than functional coverage.
Given the resources, code coverage should be used Given the resources, code coverage should be used
on the entire code base. on the entire code base.
Functional coverage is an excellent way to improve Functional coverage is an excellent way to improve
testing of risky, error prone areas. testing of risky, error prone areas.
The quality of functional coverage depends directly The quality of functional coverage depends directly
on the quality of the created functional coverage on the quality of the created functional coverage
models models
117-118
Conclusions - cont. Conclusions - cont. Conclusions - cont. Conclusions - cont.
Functional coverage can help at all design stages. Functional coverage can help at all design stages.
Functional coverage models should be created at the Functional coverage models should be created at the
high level design stage. high level design stage.
Combining code based coverage and functional Combining code based coverage and functional
coverage can lead to high quality testing. coverage can lead to high quality testing.
Any Questions? Any Questions? Any Questions? Any Questions?
119-120

Code and Functional Coverage Tutorial: Explaining The System.... Explaining The System...

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Code and Functional Coverage Tutorial: Explaining The System.... Explaining The System...

Uploaded by

Copyright:

Available Formats

Code and Functional Coverage

You might also like