37 Testing-Part1

Testing and Inspecting to Ensure High
Quality
Part 1:
Basic Definitions
Effective and Efficient Testing
Lots of Personal Notes
Overview
Testing and Inspections – two very important concepts / activities in
software development.
We often have ‘roles’ of testers;

individual developers; for different ‘kinds’ of testing;
independent test groups; quality assurance groups, etc.
Quality Control; Editors; Various sizes and shapes….
Very unique to the environment within which the software is developed.
We have many different types of tests too.

Exist at many different levels for many different stakeholders.
Some are automated; many are hands-on…
Some tests test specific parts of system; others whole;
Some at different stages of development; ….
Many Many names!!!!
Heuristic: a successful test is one that identifies a flaw.
We will present a lot of very important material in these lectures.
© Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality 2
Tests – just a few types of tests….
White box testing; Black Box Testing
Verification and validation (V&V)
Coverage Testing:
Branch testing, Path testing, Statement testing
Alpha testing, Beta testing
Acceptance Testing
Unit test, Subsystem test, System test, Integrated tests
Environmental System Testing; Operational Field Testing

Running in parallel; Running live; Implement in pieces.
Organizational Considerations
Deficiency Report (DIREP)
Burroughs Hardware / UNISYS
Field Assistance Branch; Customer Service
Tracked, prioritized, resources allocated...
Reported to senior levels of management
Incident Report –
Honeywell
Problem Ticket –
Fix Tickets (LPS)
Office of Primary Responsibility (OPR);

Office of Collateral Responsibility… (OCR)
Product Manager; Account Rep…
Basic Definitions
• A failure is an unacceptable behavior exhibited by a system
—The frequency of failures measures the reliability
—Important design objective: achieve a
- very low failure rate and hence
- a high reliability.
Basic definitions: Failure Causes:
• A failure can result from a violation of an explicit or

implicit requirement
• Specific
—functional requirement;
—non-functional requirements…
Basic definitions: MTBF MTTR Metrics
• We clearly want to increase the Mean Time Between

Failures (MTBF) and reduce the Mean Time to Repair
(MTTR):
• and damage caused by failures.
• MTBF and MTTR (old engineering metrics): are

considered measures of reliability.
Basic Definitions - continued
• A defect is a flaw in any aspect of the system that

contributes, or may potentially contribute, to the occurrence
of one or more failures
—Might take several defects to cause a particular failure
—Defects can occur anywhere – requirements, design,
implementation, testing, etc.
—They can also occur at any time!
• True ‘War’ Story: “Mr. Smith said this couldn’t happen!”
• An error is a slip-up or inappropriate decision by a

software developer that leads to the introduction of a defect
To test effectively, you must use a strategy that
uncovers as many defects as possible.
Testing really attempts to find defects!

Pure & simple!
• To test efficiently, find the

• largest possible number of defects, using the
• fewest possible tests.
• Testing costs time, money and other resources!

• Testing is not cheap! It is quite expensive!
• This is where the problems come in. What is the
expected return on investment?
• There are many kinds of tests designed to uncover
different kinds of defects.
Effective and Efficient Testing – more…
Testing must be efficient. So what do we test??
We cannot test everything.
(Exhaustive Testing is not possible – proven mathematically)
Rather, we design different kinds of tests for ‘sufficient coverage.’
All kinds of tests: Many Categories

Some are lumped into Coverage Tests.
statement coverage
branch coverage
path coverage…..
Others tests focus on creating appropriate test values (Equivalence

Testing)
Black-Box Testing
Testers provide the system with inputs; observe outputs
• They can see none of:
— Source code
— Internal data (no algorithms or data structures, …)
— Design documentation describing the system’s internals
• They simply do inputting based on requirements and observe
outputs.
•Testing Charge: Often depends on the kind and level of

testing and what the ‘charge’ of the testing group (organization)
really is.
Black-Box Testing - more
• Kind of testing end-users most frequently undertake – for obvious reasons.
• Sometimes called
• validation testing /
• acceptance testing - IF done when system is declared ‘finished.’
• Black box testing may sometimes be called other things (alpha testing;
beta testing…) but these tests (alpha and beta) usually have other
connotations associated with them…
• Black Box Testing is designed to show functionality

and often (sometimes) satisfaction of non-functional requirements.
(such as load considerations; simultaneously users, distribution, more)
Verification and Validation (V&V)
• Developers usually do their own testing first (verification)
—Usually unit testing; testing within development team.
—May involve subsystem testing, integrated testing. …
—But, in general, the developers are testing their products.
—Generally their testing is white box testing, but can be
black box testing.
• End users test system against requirements (validation)

—This is essentially black box testing.
Glass-Box Testing – White-Box Testing
Also called ‘structural’ testing
Developers and Testers have access to the system design
• They can
—Examine the Design and these Documents
—View the code for Standardization / conformance!
—Observe at run time the steps taken by algorithms and
their internal data
• Individual programmers often informally employ glass-box

testing to verify their own code and functionality
• Developers (using tools) can tell what parts of their
programs are executed the most (most time spent in them)
and then these may be candidates for optimization….
Glass-Box Testing - White Box Testing Unit Testing
Developers are always responsible for their own ‘modules’.
modules (units) = programs, classes, subsystems, etc.
Developers ‘verify’ their products.

Developer programs implement the design
Programs satisfy functional requirements, etc…
A lot of work to undertake ‘unit testing.’
Hazardous though, since they may be required to

simulate inputs, outputs, files passed,
parameter values passed, etc….
Many kinds of unit testing…
Glass-Box - White Box Testing - More
Path Testing – exhaustive; impossible; there is an infinite number of
paths in a non-trivial program.
Branch testing – Edge testing – the most feasible

Design a test so that all of the edges of a node are executed.
Often shown via flowgraphs. (ahead)
What do we mean by ‘edge?’ (all ‘else’ branches taken; all ‘then’

branches executed…)
Consider a flowgraph of an algorithm (two slides up) ….

Emphasis will be on control and coverage.
White Box: Branch Testing - Flow Graphs
Want minimum number of tests assuring high degree of reliability.
Know we cannot test all paths in a non-trivial program.

Want a ‘representative set.’
Given a flow graph (next slides), there are a couple of formulas that
will provide the minimum number of tests . They are:
• (number of nodes – number of edges + 1)
• Number of Regions + 1 ….(cyclomatic complexity)
Number of Regions is considered to be a measure of complexity.
http://www.scism.sbu.ac.uk/law/Section5/chap3/s5c3p22.html
Consider: The number of possible paths through a subprogram is equal to the number
of regions in the subprogram's flowgraph.
The flowgraph below is for a SquareRoot function and has regions numbered, as shown.
Can see there are four regions, as numbered, plus the outer region, or 5 in total.
http://www.scism.sbu.ac.uk/law/Section5/chap3/s5c3p22.html
This indicates that there are five possible paths through the flowgraph which can be described
by listing the sequence in which the nodes must are visited.
The five paths in the SquareRoot flowgraph are as follows.
etc…
White Box: Equivalence Class Testing
• This is a Bigee!
• It is inappropriate to test by brute force, using every possible
input value
—Discuss; Takes a huge amount of time
—Is impractical and is pointless!
• Far better: divide the possible inputs into groups which you
believe will be treated similarly by all algorithms.
—Such groups are called equivalence classes.
—Remember Computational Structures?? (COT 3100??)
Equivalence Class Testing
• Recall ramdomizing in File Structures (hashing…).
• If we used an ‘Equivalence mod 4’, we essentially divided the set of all non-
negative integers by four and only take their ‘remainders’ 0, 1, 2, and 3.
• If a numeric key was used as a primary or relative key, this approach was often
used to arrive at a disk address or an address in a hash table.
• Thus ‘Equivalence mod 4’ effectively partitions the set of all integers into one
of five disjoint equivalent classes.
• ALL divisions result in one of these remainders – and those having the same
remainder (members of the same equivalence class) need only require one of
its members to be tested to represent ALL values that fall into this
equivalence class..
• We applied this equivalence relation to the set of integers and obtained a set of
equivalence classes the Union of which constitutes the original set of Integers.
—EC1 U EC2 U … ECn = set of I+ (non-negative)
Example of Equivalence Classes
—A tester need only to test one member of an
equivalence class rather than ALL numbers of that
class.
—The tester has to
- understand the required input,
- appreciate how the software may have been designed
• Example:
—Valid integer input is a month number (1-12)
—Equivalence classes are: [-∞..0], [1..12], [13.. ∞]
• These are the three equivalence classes.
• In Equivalence Class Testing, we select an input from
each of the equivalence classes as inputs to testing.
More on Equivalence Class Testing
• So, to test for valid integers ranging from 1 through 12,
we have three equivalence classes as indicated:
—Integers less than 1;

—Integers between 1 and 12 inclusive
—Integers greater than 12.
• Is this enough?
• Equivalence Class Testing cannot handle all kinds of

testing:
• What if there is more than a single value we are testing?
Combinations of Equivalence Classes
• Combinatorial explosion means that you may not be
able to realistically test every possible system-wide
equivalence class.
—Let’s say there are 4 inputs each of which have 5
possible values. This means that are 54 (i.e.625)
possible system-wide equivalence classes.
• While you should first make sure that at least one test is
run with every equivalence class of every individual
input, you should also test all combinations where one
input is likely to affect the interpretation of another.
Compatibility Tests; Associativity Tests
• Compatibility tests; Associativity tests.
—Example of Associativity Test: Example from a Manpower
Project in the USAF.
- Field/attribute: Rated Performance Indicator (rpi);
- Rule: If the rpi input value for an individual had a value of non-
zero (individual had to be on an aircrew), then the ‘rank’ attribute
for this individual (s/he must be an officer), must be in the range
between O1-O10 (and could NOT be in the range E1-E9).
—Example of Compatibility Test:

- Range tests: If rank = ‘O4’,
salary must fall between $nnnn and $nnnnnn.
Many others and many different types of tests.
Testing at Boundaries of Equivalence Classes
Boundary Value Testing
• More errors in software occur at the boundaries of

equivalence classes than at any other place!! AMEN!!!
• The idea of equivalence class testing should be expanded to
specifically test values at the extremes of each equivalence
class
—E.g. The number 0 often causes problems
• E.g.: If the valid input is a month number (1-12)

—Test equivalence classes as before
- Test -24; 6, +24, (representative values)
—Test boundaries: 0, 1, 12 and 13 as well as very large
positive and negative values
—Are only integers allowed? (perhaps another test)
Coming
Defects in ordinary algorithms….

37 Testing-Part1

Uploaded by

Document Information

Original Title

Copyright

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

37 Testing-Part1

Uploaded by

Copyright:

Testing and Inspecting to Ensure High

We often have ‘roles’ of testers;

We have many different types of tests too.

Alpha testing, Beta testing

Unit test, Subsystem test, System test, Integrated tests

Environmental System Testing; Operational Field Testing

Office of Primary Responsibility (OPR);

—Important design objective: achieve a

- very low failure rate and hence

• A failure can result from a violation of an explicit or

• We clearly want to increase the Mean Time Between

• MTBF and MTTR (old engineering metrics): are

• A defect is a flaw in any aspect of the system that

• True ‘War’ Story: “Mr. Smith said this couldn’t happen!”

• An error is a slip-up or inappropriate decision by a

Testing really attempts to find defects!

• To test efficiently, find the

• Testing costs time, money and other resources!

All kinds of tests: Many Categories

Others tests focus on creating appropriate test values (Equivalence

•Testing Charge: Often depends on the kind and level of

• Black Box Testing is designed to show functionality

• End users test system against requirements (validation)

• Individual programmers often informally employ glass-box

Developers ‘verify’ their products.

Hazardous though, since they may be required to

Many kinds of unit testing…

Branch testing – Edge testing – the most feasible

What do we mean by ‘edge?’ (all ‘else’ branches taken; all ‘then’

Consider a flowgraph of an algorithm (two slides up) ….

Know we cannot test all paths in a non-trivial program.

Number of Regions is considered to be a measure of complexity.

—EC1 U EC2 U … ECn = set of I+ (non-negative)

—Integers less than 1;

• Equivalence Class Testing cannot handle all kinds of

—Example of Compatibility Test:

Many others and many different types of tests.

• More errors in software occur at the boundaries of

• E.g.: If the valid input is a month number (1-12)

Defects in ordinary algorithms….

You might also like