Research and Technology
TACOS‘04, Barcelona, March 2004
Test Case Design by means of
CTM and CTE
Joachim Wegener
DaimlerChrysler AG, Research and Technology
Joachim.Wegener@DaimlerChrysler.com
Introduction and Motivation
State of the Art
Classification-Tree Method
Classification-Tree Editor CTE
Conclusion and Future Work
Wegener, Software Technology Research Dept., Slide 1
Introduction and Motivation
Research and Technology
Software Development Requirements at DC
Software as differentiating
Increasing amount of software factor for competitive
in products in almost all areas advantage
Cost-effective Increasing
development and complexity of
time-to-market systems
crucial for
competitiveness
Increasing significance
of software in safety
High consequential relevant areas
costs due to faults
up to 60 communicating embedded controllers
up to High requirements
1,000,000 for software
lines of software code
based systems
various buses, hundredsdueoftomessages
norms,
standards and legal regulations
> 2 km wire
Wegener, Software Technology Research Dept., Slide 2
Introduction and Motivation
Research and Technology
Applications at DC
Wegener, Software Technology Research Dept., Slide 3
Introduction and Motivation
Research and Technology
Applications at DC
Wegener, Software Technology Research Dept., Slide 4
Introduction and Motivation
Research and Technology
System Development at DC
DaimlerChrysler Acceptance Test
SPICE Vehicle Tests
Pre-Development Assessments
Vehicle Integration
Prototyping
Code Quality
Specification System Test Assessments
Integration Test
Component Development
Integration
Implementation Unit Test
Suppliers
Wegener, Software Technology Research Dept., Slide 5
Introduction and Motivation
Research and Technology
Testing in Practice
• testing is an immense cost factor in the course of system development
average distribution of costs for embedded systems
system and
acceptance 20
testing
50 other development activities
30
module and
integration
testing
• high cost
Test automation
• testing is too unsystematic
• low error detection rate Test methods
Wegener, Software Technology Research Dept., Slide 6
Introduction and Motivation
Research and Technology
Objectives of Testing
Through executing a system with selected test data the test is aiming at
z detecting errors in the system under test and
z gaining confidence in the correct functioning of the test object.
Strong Features
z taking into consideration the real environment (e.g. target computer, compiler) and
z testing the dynamic system behavior (e.g. run time behavior, memory space requirement)
Weak Features
z an exhaustive test is usually impossible
test data selection by certain criteria
Wegener, Software Technology Research Dept., Slide 7
State of the Art
Research and Technology
Test Activities
Test Planning
Specification Program
Test Documentation
Test Organization
Test Case Design
Test Execution Monitoring
Test Evaluation
• most important for test quality
• methods and tools needed
Wegener, Software Technology Research Dept., Slide 8
State of the Art
Research and Technology
Testing Methods
Functional test:
• test case design performed on the basis of the specification
• most important test approach
• common methods show insufficient systematics when partitioning
the input domain, difficult procedure, or no efficient tool support
• test size is hard to quantify (coverage of requirements, function points, ...)
• difficult to automate
• examples: equivalence partitioning, boundary value test, category-partition method
Structural test:
• test case design performed on the basis of the program structure
• common test approach (included in many standards)
• not possible to check whether all requirements have been implemented
• difficult to automate (limits of symbolic execution)
• examples: statement test, branch test, path test
Wegener, Software Technology Research Dept., Slide 9
State of the Art
Research and Technology
Testing Methods
Statistical Test:
• test data generation in accordance with operational profile
• operational profile hard to determine, especially for new systems
• extensive test evaluation, if no test oracle available
Random Test:
• generation of random test data
• specific test-relevant value combinations are produced only with little probability
• extensive test evaluation, if no test oracle available
Evolutionary Test:
• generation of test data by means of search techniques
• test objectives have to be transformed into an optimization problem
Mutation Test:
• test cases are build in order to kill mutants of original program
• no procedure how to define the test cases
Wegener, Software Technology Research Dept., Slide 10
Classification-Tree Method
Research and Technology
Functional Testing
• lack of functional testing methods which are
• efficient
• easy to use in practice
• suited for automation
• lack of testing tools for black-box test case design
Classification-Tree Method
efficient functional test method CTM:
• systematic stepwise procedure which is easy to
understand
• graphical notation with compact representation
of the over-all test
• good test documentation
• tool support with the CTE XL
• proven in practical use
Wegener, Software Technology Research Dept., Slide 11
Classification-Tree Method CTM
Research and Technology
Classification-Tree Method
Test object: computer vision system recognizing the size of different
building blocks
large
Aspects
Size Color Shape Size Color Shape
small large red green blue circle triangle square
Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Input Domain
Wegener, Software Technology Research Dept., Slide 12
Classification-Tree Method CTM
Research and Technology
Classification-Tree Method
Size Color Shape
small large red green blue circle triangle square
Shape of
triangle
equi-
lateral isosceles scalene
Wegener, Software Technology Research Dept., Slide 13
Classification-Tree Method CTM
Research and Technology
Classification-Tree Method
Size Color Shape
small large red green blue circle triangle square
Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Wegener, Software Technology Research Dept., Slide 14
Classification-Tree Method CTM
Research and Technology
Classification-Tree Method
Size Color Shape
small large red green blue circle triangle square
Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Wegener, Software Technology Research Dept., Slide 15
Classification-Tree Method CTM
Research and Technology
Classification-Tree Method
Test object: computer vision system recognizing the size of different
building blocks
classification
large
class
Aspects
Size Color Shape Size Color Shape
small large red green blue circle triangle square
Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Input Domain
Wegener, Software Technology Research Dept., Slide 16
Classification-Tree Method CTM
Research and Technology
Classification-Tree Method
Test object: computer vision system recognizing the size of different
building blocks
Minimality criterion: 5 testcases
large five disjoint classes for
shape
Aspects
Size Color Shape Size Color Shape
small large red green blue circle triangle square
Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Input Domain
Wegener, Software Technology Research Dept., Slide 17
Classification-Tree Method CTM
Research and Technology
Classification-Tree Method
Test object: computer vision system recognizing the size of different
building blocks
Maximality criterion: 30 testcases
large 2*3*5
Aspects
Size Color Shape Size Color Shape
small large red green blue circle triangle square
Shape of
triangle
equi-
lateral isosceles scalene
1
2
3
Input Domain
Wegener, Software Technology Research Dept., Slide 18
Classification-Tree Editor CTE
Research and Technology
Classification-Tree Editor CTE
• syntax directed
• context sensitive
• graphical
editor to support the systematic test case design by means of the classification-tree method
Supporting
• hierarchies to structure large classification trees and
• hierarchies to structure sets of test cases
• description of logical dependencies between classes of the classification-tree
• combination rules for the generation of test cases according to logical dependencies and
combination rules
• sorting of test cases
• comprehensive export functionalities
• open API and architecture supporting the implementation of tool integrations such as with
• DOORS
• Test Director
• Tessy
• Time Partition Testing
• Win Runner
Wegener, Software Technology Research Dept., Slide 19
Hierarchies for large trees
Research and Technology
2. Test cases can be hierachically grouped:
1. Subtrees can be folded/unfolded at will:
folded
subtree
groups of
test cases
Wegener, Software Technology Research Dept., Slide 20
Implicit (Syntactical) Rules
Research and Technology
1. Classes belonging to the same classification 2. If a class is marked all parent tree nodes
are disjoint (i.e. exclusive). are marked, too.
e.g., red, green, and blue e.g., equilateral implies triangle
‘triangle’
is implicitly
marked
disjoint
classes
Wegener, Software Technology Research Dept., Slide 21
Usage of Logical Dependencies
Research and Technology
1. Specification of impossible or required
2. Verification of dependencies
class combinations e.g., circle => NOT red
e.g., circle => NOT red
verification
of logical
dependencies
specification
of logical
dependencies
red circle
Wegener, Software Technology Research Dept., Slide 22
Combination Rules
Research and Technology
• express the desired combinatorics between classes and classifications
• usage of combinatorial expressions
specification
of combination
rules
Wegener, Software Technology Research Dept., Slide 23
Automated Test Case Generation
Research and Technology
• based on combination rules and logical dependencies
• algorithm based on heuristic combinatorial search
Wegener, Software Technology Research Dept., Slide 24
Classification-Tree Editor CTE
Research and Technology
Real-World Example: Positioning
Wegener, Software Technology Research Dept., Slide 25
API Supporting Tool Integrations
Research and Technology
Examples Tool Integration
Acceptance Test
Vehicle Tests
Vehicle Integration
dSPACE
Specification DOORS TPT MSC TTCN3
Auto-
mation
Win
System Test Runner
Desk
System Test
Integration Test
Integration
MTest
Unit Test Tessy
Wegener, Software Technology Research Dept., Slide 26
Conclusion and Future Work
Research and Technology
Summary
• Classification-Tree Method enables systematic test case design for functional testing
• successfully applied in practice to nearly all application domains
• Efficient tool support by means of the CTE XL
• available on www.systematic-testing.com
• you can start using it
• Many tool integrations available. Further integrations under development.
Future Work
• Extensions for test management and product line approaches
• Description of expected behavior
• Generation rules for sub-trees
• Metrics to assess test case design quality
Wegener, Software Technology Research Dept., Slide 27
