Software Testing Presentation

SIM Group Ltd.
, SQS Group AG, 2002

ISEB Foundation Certificate in Software Testing
Testing Terminology
Testing Terminology
SIM Group Ltd., SQS Group AG, 2002
Testing Terminology

IT is an industry riddled with mysterious words, terms and
IT is an industry riddled with mysterious words, terms and
TLA
TLA
s
s

Software testing is no exception
Software testing is no exception

Historically there has not been a generally accepted set of
Historically there has not been a generally accepted set of
testing definitions
testing definitions
Testing Terminology

To resolve this issue, the BCS SIGIST created a Standard
To resolve this issue, the BCS SIGIST created a Standard
Glossary of Testing Terms
Glossary of Testing Terms
British Computer Society British Computer Society
Special Interest Group In Software Testing Special Interest Group In Software Testing

This is now a British Standard
This is now a British Standard
BS7925 BS7925- -1 1
Testing Terminology

You will need to read the document and start to learn the
You will need to read the document and start to learn the
terminology used in software testing
terminology used in software testing

You will be exposed to the terminology and customers will
You will be exposed to the terminology and customers will
assume you know what they mean
assume you know what they mean
Why Testing is Necessary

In this session we will

Understand what testing is and why it is necessary
Understand what testing is and why it is necessary

Define some of the words used in software testing
Define some of the words used in software testing

Look at the cause and cost of errors
Look at the cause and cost of errors

Why Test?
Why Test?
To find bugs
To find bugs
What is a bug?
What is a bug?
What is Testing?
What is Testing?
the process of executing a program with the intent to certify it the process of executing a program with the intent to certify it s s
Quality Quality
Mills Mills
the process of executing a program with the intent of finding the process of executing a program with the intent of finding
failures / faults failures / faults
Myers Myers
the process of exercising software to detect bugs & to verify th the process of exercising software to detect bugs & to verify that at
it satisfies specified functional & non it satisfies specified functional & non- -functional requirements functional requirements

Why is Testing becoming more important?
Why is Testing becoming more important?
The Y2K problem The Y2K problem
EMU EMU
e Commerce e Commerce
Increased user base Increased user base
Increased complexity Increased complexity
Speed to Market Speed to Market
Definitions
Definitions
ERROR
ERROR
A human action that produces an incorrect result

A human action that produces an incorrect result
FAULT
FAULT
A manifestation of an ERROR in software

A manifestation of an ERROR in software

Definitions
FAULT
FAULT
A Fault, if encountered, may cause a failure
A Fault, if encountered, may cause a failure
FAILURE
FAILURE
A deviation of the software from its expected delivery or

A deviation of the software from its expected delivery or
service
service

Definitions
DEFECT
DEFECT
The departure of a quality characteristic from its specified val

The departure of a quality characteristic from its specified val
ue
ue
that results in a product or service not satisfying its normal
that results in a product or service not satisfying its normal
usage requirements
usage requirements

Definitions
RELIABILITY
RELIABILITY
The probability that software will not cause the failure of a

The probability that software will not cause the failure of a
system for a specified period of time under specified
system for a specified period of time under specified
conditions
conditions

Definitions
QUAL
ITY
ITY
The totality of the characteristics of an entity that bear on it

The totality of the characteristics of an entity that bear on it
s
s
ability to satisfy stated or implied needs
ability to satisfy stated or implied needs

Testing & Quality

Does testing improve quality?
Does testing improve quality?

Testing does not build Quality into the software
Testing does not build Quality into the software

Testing is a means of determining the Quality of the Software
Testing is a means of determining the Quality of the Software
under Test
under Test
Definitions
QUALITY ASSURANCE
QUALITY ASSURANCE
All those planned actions used to fulfil the requirements for

All those planned actions used to fulfil the requirements for
quality
quality

Why do errors occur?

How Errors Occur
How Errors Occur

No one is perfect!
No one is perfect!
We all make mistakes or omissions We all make mistakes or omissions

The more pressure we are under the more likely we are to
The more pressure we are under the more likely we are to
make mistakes
make mistakes

In IT development we have time and budgetary deadlines to
In IT development we have time and budgetary deadlines to
meet
meet

Poor Training
Poor Training

How Errors Occur
How Errors Occur

Poor Communication
Poor Communication

Requirements not clearly defined
Requirements not clearly defined

Requirements change & are not properly documented
Requirements change & are not properly documented

Data specifications not complete
Data specifications not complete

ASSUMPTIONS!
ASSUMPTIONS!

The Cost Of Errors
The Cost Of Errors

A single failure may incur little cost
A single failure may incur little cost
-
-
or millions
or millions

Report layouts may be wrong
Report layouts may be wrong
-
-
little true cost
little true cost

Or a significant error may cost millions
Or a significant error may cost millions
Ariane Ariane V, Venus Probe, Mars Explorer and Polar Lander V, Venus Probe, Mars Explorer and Polar Lander

The Cost Of Errors
The Cost Of Errors

In extreme cases a software or systems error may cost
In extreme cases a software or systems error may cost
LIVES
LIVES

Usually safety critical systems are tested exhaustively
Usually safety critical systems are tested exhaustively
Aeroplane, railway, nuclear power systems etc Aeroplane, railway, nuclear power systems etc

Unfortunately there are exceptions to this
Unfortunately there are exceptions to this
London Ambulance Service London Ambulance Service

The Cost Of Errors
The Cost Of Errors

The cost of defects increases proportionately (tenfold?) with
The cost of defects increases proportionately (tenfold?) with
the passing of each successive stage in the system
the passing of each successive stage in the system
development process before they are detected
development process before they are detected

To correct a problem at requirements stage may cost
To correct a problem at requirements stage may cost
1
1

To correct the problem post
To correct the problem post
-
-
implementation may cost
implementation may cost
000
000
s
s

Why Test?
Why Test?

Identifies faults
Identifies faults
Reduces live defects Reduces live defects
Improves the quality of the users application Improves the quality of the users application
Increases reliability Increases reliability
To help ensure live failures don To help ensure live failures don t impact costs & profitability t impact costs & profitability
To help ensure requirements are satisfied To help ensure requirements are satisfied
To ensure legal requirements are met To ensure legal requirements are met
To help maintain the organisation To help maintain the organisation s reputation s reputation

Provides a measure of quality
Provides a measure of quality

Quality Measurements
Quality Measurements

Quality can be measured by testing for
Quality can be measured by testing for
correctness correctness
reliability reliability
usability usability
maintainability maintainability
testability testability
reusability reusability

How much is enough?
How much is enough?

How do we know when to stop?
How do we know when to stop?
There are many factors to consider There are many factors to consider

Exhaustive Testing
Exhaustive Testing

Find all faults by testing everything?
Find all faults by testing everything?

To test everything is rarely possible
To test everything is rarely possible

The time required makes it impractical
The time required makes it impractical

The resource commitment required makes it impractical
The resource commitment required makes it impractical

If we can
If we can
t test everything, what can we do?

t test everything, what can we do?

Managing and reducing Risk
Managing and reducing Risk

Carry out a Risk Analysis of the application
Carry out a Risk Analysis of the application

Prioritise tests to focus on the main areas of risk
Prioritise tests to focus on the main areas of risk

Apportion time relative to the degree of risk involved
Apportion time relative to the degree of risk involved

Understand the risk to the business of the software not
Understand the risk to the business of the software not
functioning correctly
functioning correctly

Should you stop testing when the product goes live?
Should you stop testing when the product goes live?

Continuing testing beyond the implementation date should be
Continuing testing beyond the implementation date should be
considered
considered

Better that you find the errors than the users
Better that you find the errors than the users

Summary
Summary

The purpose of testing is to find faults
Faults can be fixed, making better software Faults can be fixed, making better software
Better software is more reliable, less prone to failures Better software is more reliable, less prone to failures
Establish the relationship between the software and its Establish the relationship between the software and its
specification specification

Testing enables us to measure the quality of the software
Testing enables us to measure the quality of the software

This enables us to understand & manage the risk to the
This enables us to understand & manage the risk to the
business
business
The Test Process
The Test Process
The Test Process


Look at the steps involved in the test process
Look at the steps involved in the test process

Look in detail at each step
Look in detail at each step
The Test Process

What is the objective of a test?
What is the objective of a test?

A
A
successful
successful
test is one that

test is one that
does
does
detect a fault
detect a fault
The Test Process
C
O
M
P
L
E
T
I
O
N
E
X
E
C
U
T
I
O
N
TEST MANAGEMENT
TEST ENVIRONMENT MANAGEMENT
TEST ASSET MANAGEMENT
R
E
C
O
R
D
I
N
G
P
L
A
N
N
I
N
G
S
P
E
C
I
F
I
C
A
T
I
O
N
The Test Process

Five steps in the Test Process are
Five steps in the Test Process are

Test Planning
Test Planning

Test Analysis and Specification
Test Analysis and Specification

Test Execution
Test Execution

Test Recording (Verification)
Test Recording (Verification)

Checking for Completion
Checking for Completion
Test Planning
The Test Plan describes how The Test Plan describes how
the Test Strategy is the Test Strategy is
implemented implemented
Test Planning
Contents of a Test Plan include Contents of a Test Plan include
Background Background
Reference documents Reference documents
Approach Approach
Method Method
Timetable Timetable
Resources Resources
Dependencies Dependencies
Reporting Reporting
Test Asset Identification Test Asset Identification
Exit Criteria Exit Criteria
Test Planning
The most critical stage of the The most critical stage of the
process process
Effort spent now will be Effort spent now will be
rewarded later rewarded later
The foundation on which The foundation on which
testing is built testing is built
Test Specification
Three step process Three step process
Preparation & analysis Preparation & analysis
Building test cases Building test cases
Define expected results Define expected results
Test Specification
Test preparation Test preparation
Analyse the Application Analyse the Application
Identify good test conditions Identify good test conditions
Identify test cases Identify test cases
Document thoroughly Document thoroughly
Cross Cross- -referencing referencing
Test Specification
Build Test Cases Build Test Cases
Test cases comprise: Test cases comprise:
Standing Data Standing Data
Transaction Data Transaction Data
Actions Actions
Expected Results Expected Results
Test Specification
Expected Results Expected Results
The outcome of each action The outcome of each action
The state of the application The state of the application
during & after during & after
The state of the data during The state of the data during
& after & after
Test Specification
Cross Cross- -Referencing & Referencing &
Classification Classification
Enables maintainability of Enables maintainability of
Test Assets Test Assets
Allows testing to be Allows testing to be
performed in a focussed performed in a focussed
manner directed at specific manner directed at specific
areas areas
Test Execution

Test Execution checklist
Test Execution checklist

Test execution schedule / log
Test execution schedule / log

Identify which tests are to be run
Identify which tests are to be run

Test environment primed & ready
Test environment primed & ready

Resources ready, willing & able
Resources ready, willing & able

Back
Back
-
-
up & recovery procedures in place
up & recovery procedures in place

Batch runs planned & scheduled
Batch runs planned & scheduled

Then we are ready to run the tests
Then we are ready to run the tests
Test Recording

Test verification
Test verification

If planning and preparation is sufficiently detailed this is the
If planning and preparation is sufficiently detailed this is the
easy part of testing
easy part of testing

The test is run to verify the application under test
The test is run to verify the application under test

The test itself either passes or fails!
The test itself either passes or fails!
Test Recording

The test log should record
The test log should record

Software and test versions
Software and test versions

Specifications used as test base
Specifications used as test base

Test timings
Test timings

Test results
Test results
Actual results Actual results
Expected results Expected results

Defect details for erroneous tests
Defect details for erroneous tests
Test Completion
Test Exit Criteria Test Exit Criteria
Used to determine when to Used to determine when to
implement the software implement the software
Key Functionality tested Key Functionality tested
Test coverage Test coverage
Budget used? Budget used?
Defect detection rate Defect detection rate
Performance satisfactory Performance satisfactory
The Test Process

Summary
Summary

There are five steps in the Test Process
There are five steps in the Test Process
Test Planning Test Planning
Test Analysis and Specification Test Analysis and Specification
Test Execution Test Execution
Test Recording (Verification) Test Recording (Verification)
Checking for Completion Checking for Completion
The Psychology Of Testing


Understand what qualities make good testers
Understand what qualities make good testers

Look at a testers
Look at a testers
relationship with developers

relationship with developers

Look at a testers
Look at a testers
relationship with management

relationship with management

Understand the issues with testing independence
Understand the issues with testing independence
What makes a Tester?

Testing is primarily to find faults
Testing is primarily to find faults

Can be regarded as
Can be regarded as
destructive
destructive

Development is
Development is
constructive
constructive

Testing asks questions
Testing asks questions

Testers need to ask questions
Testers need to ask questions

A tester needs many qualities...
A tester needs many qualities...

Intellectual qualities
Intellectual qualities

Can absorb incomplete facts
Can absorb incomplete facts

Can work with incomplete facts
Can work with incomplete facts

Can learn quickly on many levels
Can learn quickly on many levels

Good verbal communication
Good verbal communication

Good written communication
Good written communication

Ability to prioritise
Ability to prioritise

Self
Self
-
-
organisation
organisation

Knowledge
Knowledge

How projects work
How projects work

How computer systems and business needs interact
How computer systems and business needs interact

What makes IT tick
What makes IT tick
-
-
technology
technology

What makes IT tick
What makes IT tick
-
-
commercial aspects
commercial aspects

Testing techniques
Testing techniques

Testing best practice
Testing best practice

To be able to think inside and outside of a system
To be able to think inside and outside of a system
specification
specification

More skills to acquire
More skills to acquire

How to find bugs
How to find bugs
-
-
planning, preparation & execution
planning, preparation & execution

How to understand systems
How to understand systems

How to read specifications
How to read specifications

How to extract testable functionality
How to extract testable functionality

How to work efficiently
How to work efficiently

How to focus on essentials
How to focus on essentials
Reporting Defects

Defects need to be reported to
Defects need to be reported to

Developers to enable them to fix them
Developers to enable them to fix them

Management so they can track progress
Management so they can track progress

Communication to both groups is vital
Communication to both groups is vital
Communication with Developers

A good relationship is vital
A good relationship is vital

Developers need to keep testers up to date with changes to
Developers need to keep testers up to date with changes to
the application
the application

Testers need to inform developers of defects to allow fixes to
Testers need to inform developers of defects to allow fixes to
be applied
be applied
Communication with Management

Managers need progress reports
Managers need progress reports

The best way is through metrics
The best way is through metrics

Number of tests planned & prepared
Number of tests planned & prepared

Number of tests executed to date
Number of tests executed to date

Number of defects raised & fixed
Number of defects raised & fixed

How long planning, preparation and execution stages take
How long planning, preparation and execution stages take
The Psychology of Testing
Testing Independence

It is important that testing is separate from
It is important that testing is separate from
development
development

The developer is likely to confirm adherence not deviation
The developer is likely to confirm adherence not deviation

The developer will make assumptions
The developer will make assumptions
-
-
the same when
the same when
testing as developing
testing as developing

Levels of Independence
Levels of Independence

Low
Low
-
-
Developers write their own tests
Developers write their own tests

Medium
Medium
-
-
Tests are written by another developer
Tests are written by another developer

High
High
-
-
Tests written by an independent body
Tests written by an independent body
Tests written by another section Tests written by another section
Tests written by another organisation Tests written by another organisation

Utopia
Utopia
-
-
Tests generated automatically
Tests generated automatically

Summary
Summary

Testers require a particular set of skills
Testers require a particular set of skills
The desire to break things The desire to break things
The desire to explore and experiment The desire to explore and experiment
Communication Communication
Questioning Questioning

Testing requires a different mentality to development
Testing requires a different mentality to development
Destroying Destroying things rather than creating them things rather than creating them
Testing should be separate from development Testing should be separate from development
Re
Re
-
-
Testing & Regression Testing
Testing & Regression Testing
Re-Testing & Regression Testing


Understand how fault fixing and re
Understand how fault fixing and re
-
-
testing are key to the
testing are key to the
overall testing process
overall testing process

Look at how test repeatability helps
Look at how test repeatability helps

Understand what regression testing is and where it fits in
Understand what regression testing is and where it fits in

Understand how to select test cases for regression testing
Understand how to select test cases for regression testing
Re-Testing

Testing is directed at finding faults
Testing is directed at finding faults

These faults will need to be corrected and a new version of
These faults will need to be corrected and a new version of
the software released
the software released

The tests will need to be run again to prove that the fault is
The tests will need to be run again to prove that the fault is
fixed
fixed

This is known as Re
This is known as Re
-
-
Testing
Testing
Re-Testing

The need for re
The need for re
-
-
testing needs to be planned & designed
testing needs to be planned & designed
for
for

Schedules need to allow for re
Schedules need to allow for re
-
-
testing
testing

Tests need to be easily re
Tests need to be easily re
-
-
run
run

The environment needs to be easily restored
The environment needs to be easily restored
Regression Testing
Re
Re
-
-
testing of a previously tested program following
testing of a previously tested program following
modification to ensure that FAULTS have not been
modification to ensure that FAULTS have not been
introduced or uncovered as a result of the changes
introduced or uncovered as a result of the changes
made
made

Regression Testing
Module 14 v1.4.2
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Declare String x =Date();
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Module 14 v1.4.2
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Declare String x =Date();
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Module 41 v1.8.9
~~~~~~~~~~~~~~~~
Declare Integer x = 0;
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
For x = 1 to 5
~~~~~~~~~~~~~~~~
Module 41 v1.8.9
~~~~~~~~~~~~~~~~
Declare Integer x = 0;
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
For x = 1 to 5
~~~~~~~~~~~~~~~~
Module 93 v1.3.5
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Print (x);
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Module 93 v1.3.5
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Print (x);
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Module 14 v1.4.3
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Global String x = Date();
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Module 14 v1.4.3
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
Global String x = Date();
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~
How can you fix this
but break that?
How can you fix this
but break that?


Regression Testing

Tests will also need to be re
Tests will also need to be re
-
-
run when checking
run when checking
software upgrades
software upgrades

Regression tests should be run whenever there is a change
Regression tests should be run whenever there is a change
to the software or the environment
to the software or the environment

Regression tests are executed to prove aspects of a system
Regression tests are executed to prove aspects of a system
have not changed
have not changed

Regression Testing is a vital testing technique
Regression Testing is a vital testing technique
Regression Testing

Selecting cases for regression:
Selecting cases for regression:

Tests that cover safety or business critical functions
Tests that cover safety or business critical functions

Tests that are repetitive
Tests that are repetitive

Tests that need accurate data
Tests that need accurate data

Tests for areas that change regularly
Tests for areas that change regularly

Tests of functions that have a high level of defects
Tests of functions that have a high level of defects
Regression Testing

Regression Testing is the ideal foundation for
Regression Testing is the ideal foundation for
Automation
Automation

Selecting test cases is vital, and requires a degree of
Selecting test cases is vital, and requires a degree of
knowledge of the applications and their expected evolution
knowledge of the applications and their expected evolution
Re-Testing & Regression Testing

Summary
Summary

Once a fault has been fixed the software MUST be re
Once a fault has been fixed the software MUST be re
-
-
tested
tested
New faults may have been introduced by the fix New faults may have been introduced by the fix
Existing faults may have been uncovered by the fix Existing faults may have been uncovered by the fix
Tests need to be written to enable their re Tests need to be written to enable their re- -use use

Re
Re
-
-
testing is the rerunning of failed tests once a fix has been
testing is the rerunning of failed tests once a fix has been
implemented to check that the fix has worked
implemented to check that the fix has worked

Regression testing is running of a wider test suite to check for
Regression testing is running of a wider test suite to check for
unexpected errors in unchanged code
unexpected errors in unchanged code
Expected Results
Expected Results
Expected Results


Understand the need to define expected results
Understand the need to define expected results

Understand where expected results can be found
Understand where expected results can be found
Expected Results
Expected results = expected Expected results = expected
outcomes outcomes
Identifying required behaviour Identifying required behaviour
If the expected outcome of a If the expected outcome of a
test is not defined the actual test is not defined the actual
output may be misinterpreted output may be misinterpreted
Expected Results
Running a test with only a Running a test with only a
general concept of the general concept of the
outcome is fatal outcome is fatal
Expected Results
It is vital the Expected Results It is vital the Expected Results
are defined with the tests, are defined with the tests,
before they are used before they are used
You cannot decide whether a You cannot decide whether a
test has passed just because test has passed just because
it looks right it looks right
Expected Results
It may be possible to identify It may be possible to identify
the expected result of a test the expected result of a test
based on knowledge or based on knowledge or
experience of the experience of the
application application but not the code but not the code
Expected Results
Consult the Oracle! Consult the Oracle!
Expected Results

Summary
Summary

Without defining expected results how do you know if a test
Without defining expected results how do you know if a test
has passed or failed?
has passed or failed?

Expected results can be found in the system specification and
Expected results can be found in the system specification and
by asking experienced business users
by asking experienced business users
Prioritisation of Tests


Understand why we need to prioritise tests
Understand why we need to prioritise tests

Understand how we decide the priority of individual tests
Understand how we decide the priority of individual tests
Prioritisation of tests
It is not possible to test It is not possible to test
everything everything
Therefore faults will get Therefore faults will get
through to the live system through to the live system
We must do the best testing We must do the best testing
possible in the time available possible in the time available
This means we must This means we must
prioritise and focus testing prioritise and focus testing
on the priorities on the priorities
Aspects to Consider
Severity Severity
Probability Probability
Visibility Visibility
Priority of Requirements Priority of Requirements
Customer Requirements Customer Requirements
Frequency of Change Frequency of Change
Vulnerability to error Vulnerability to error
Technical criticality Technical criticality
Complexity Complexity
Resource availability Resource availability
Business Criticality
What elements of the What elements of the
application are essential to application are essential to
the success of the the success of the
organisation? organisation?
Customer factors
How visible would a failure How visible would a failure
be? be?
What does the customer What does the customer
want? want?
Technical Factors
How complex is it? How complex is it?
How likely is an error? How likely is an error?
How often does this change? How often does this change?

Summary
Summary

Due to the last minute nature of testing, it is necessary to
Due to the last minute nature of testing, it is necessary to
prioritise tests
prioritise tests

There will never be enough time to complete all tests
There will never be enough time to complete all tests

Therefore those tests that cover those areas deemed most
Therefore those tests that cover those areas deemed most
important (to the business, highest risk etc) must be tested
important (to the business, highest risk etc) must be tested
first
first
Models for Testing
Models for Testing
Models for Testing


Look at two of the most commonly accepted testing models
Look at two of the most commonly accepted testing models
Testing Models

There are many approaches to testing
There are many approaches to testing

Of these 2 are widely used & referred to:
Of these 2 are widely used & referred to:
V,V&T V,V&T
The V The V- -Model Model

There are various other models
There are various other models
They are not part of this course They are not part of this course
V,V&T

Verification
Verification
The process of evaluating a system or component to determine The process of evaluating a system or component to determine
whether the products of the given development phase satisfies whether the products of the given development phase satisfies
the conditions imposed at the start of that phase the conditions imposed at the start of that phase

Validation
Validation
Determination of the correctness of the products of software Determination of the correctness of the products of software
development with respect to the user needs and requirements development with respect to the user needs and requirements

Testing
Testing
process of exercising software to verify that it satisfies speci process of exercising software to verify that it satisfies specified fied
requirements and to detect faults requirements and to detect faults
V-Model

The V
The V
-
-
Model is the most commonly used model in
Model is the most commonly used model in
testing
testing

It represents the Software Development Life Cycle
It represents the Software Development Life Cycle

Shows the various stages in development and testing
Shows the various stages in development and testing

Shows the relationships between the various stages
Shows the relationships between the various stages
A simple V-Model
high level
low level
time
requirements
design
code unit
test
system
test
UAT
T
T
T
T
T
T
A typical development life-cycle V-Model
level
time
Service Level
Testing
Operational
Requirements
Design
Specifications
Functional
Specification
Component
Testing
System
Testing
Integration
Testing (large)
User Acceptance
Testing
Program
Specification
Business
Requirements
Models for Testing
level
time
Requirements
Requirements
Design
Design
Code
Code
Unit Test
Unit Test
System Test
System Test
UAT
UAT
T
T
T
T
T
T
V-Model - where do we start testing?
Models for Testing
V-Model - Start Testing Early
Reqs
level
time
T
T
T
T
T
T
T
T
T
T
R
R
R
R
R
R
R
R
R
R
P
P
P
P
R
R
P
P
T
T
Spec
Code
UAT
System Test
Unit Test
Models for Testing

Summary
Summary

The V
The V
-
-
Model is the most common approaches model for
Model is the most common approaches model for
testing
testing

There are other models used for testing, but these are less
There are other models used for testing, but these are less
widely used
widely used
Inc. V,V & T Inc. V,V & T
Other models have been created Other models have been created
Economics of Testing


Look at the cost of testing (or the cost of not testing)
Look at the cost of testing (or the cost of not testing)

Appreciate the value of good, timely testing
Appreciate the value of good, timely testing

The earlier a fault is found, the cheaper it is to remedy
The earlier a fault is found, the cheaper it is to remedy

Faults in the requirements can lead to major re
Faults in the requirements can lead to major re
-
-
engineering of
engineering of
the entire system
the entire system

Many faults can be found using reviews
Many faults can be found using reviews
Reviews of documentation and / or code Reviews of documentation and / or code

Allows testing to find more
Allows testing to find more
substantial
substantial
faults
faults

The average cost of fixing a defect increases tenfold
The average cost of fixing a defect increases tenfold
with every step of the development process
with every step of the development process

In code reviews you will find and fix defects in an average of
In code reviews you will find and fix defects in an average of
1
1
-
-
2 minutes
2 minutes

In initial testing, defect fix times will average between
In initial testing, defect fix times will average between
10 to 20 minutes
10 to 20 minutes

In integration testing each defect can cost an hour or
In integration testing each defect can cost an hour or
more
more

In system test each defect can cost 10 to 40 or more
In system test each defect can cost 10 to 40 or more
engineer hours
engineer hours
Watts Humphrey Watts Humphrey

Early test design can prevent fault multiplication
Early test design can prevent fault multiplication

Analysis of specification during test preparation often brings
Analysis of specification during test preparation often brings
faults in the specification to light
faults in the specification to light

If faults in documentation are not found then the system may
If faults in documentation are not found then the system may
be developed incorrectly
be developed incorrectly
Economics of testing
An example
An example

Finding the error in the requirements
Finding the error in the requirements

Change requirements document

Finding the error in the specifications
Finding the error in the specifications


Change specifications

Finding the error in the code
Finding the error in the code



Change the code
Change the code

Finding the error in system testing
Finding the error in system testing



Change the code
Change the code

Change the tests, re
-
-
test & regression test

Finding the error in UAT
Finding the error in UAT



Change the code
Change the code

-
-

Re
Re
-
-
do UAT
do UAT

Finding the error in Live



Change the code
Change the code

-
-

Re
Re
-
-
do UAT
do UAT
Plus...
Plus...


User to report and log fault
User to report and log fault

Users must be advised of fix, work
Users must be advised of fix, work
-
-
around etc
around etc

The live database may need to be fixed
The live database may need to be fixed

Deploying the new build may take hours
Deploying the new build may take hours

Loss of business
Loss of business

Additionally
Additionally

Faults found in coding or later require changes to the code
Faults found in coding or later require changes to the code
library
library

Documentation and help files will have to be changed
Documentation and help files will have to be changed

Every fault requires investigating and logging
Every fault requires investigating and logging

Administration costs
Administration costs

The cost of not Testing
The cost of not Testing

What is the cost of testing?
What is the cost of testing?

Which is cheaper?
Which is cheaper?
Testing and finding faults before release Testing and finding faults before release
Not testing and finding faults once the system is live Not testing and finding faults once the system is live

Companies do not typically have figures for either
Companies do not typically have figures for either

Summary
Summary


The earlier a fault is found the cheaper (and easier) it is to f
The earlier a fault is found the cheaper (and easier) it is to f
ix
ix

Starting testing early will help find faults quicker
Starting testing early will help find faults quicker
High Level Test Planning

Look at how a test plan is put together Look at how a test plan is put together
Understand how it should be used and maintained Understand how it should be used and maintained
Understand why they are so important to a testing project Understand why they are so important to a testing project

What is a test plan?
What is a test plan?

A project plan for testing
A project plan for testing

Covering all aspects of testing
Covering all aspects of testing

Before you plan
Before you plan

A test strategy must be in place
A test strategy must be in place
The Different Sections of a Test Plan
Test plan identifier Test plan identifier
Introduction Introduction
Test items Test items
Features to be tested Features to be tested
Features not to be tested Features not to be tested
Approach Approach
Item pass / fail criteria Item pass / fail criteria
Suspension criteria & Suspension criteria &
resumption criteria resumption criteria
Test deliverables Test deliverables
Testing tasks Testing tasks
Environment Environment
Responsibilities Responsibilities
Staffing and training needs Staffing and training needs
Schedules Schedules
Risks and contingencies Risks and contingencies
Approvals Approvals
Based upon IEEE 829 Based upon IEEE 829- -1998 standard for 1998 standard for
software test documentation software test documentation
Test Plan Identifier

A unique identifier for the test plan
A unique identifier for the test plan
Introduction

The introduction should
The introduction should

Give an overview of the plan
Give an overview of the plan
A summary of the requirements A summary of the requirements
Discuss what needs to be achieved Discuss what needs to be achieved
Detail why testing is needed Detail why testing is needed

Reference to other documents
Reference to other documents
Quality assurance and configuration management Quality assurance and configuration management
Test Items

The various items to be used in the tests
The various items to be used in the tests

The software items
The software items

Their versions numbers / identifiers
Their versions numbers / identifiers

How they will be handed over to testing
How they will be handed over to testing

References to relevant documents
References to relevant documents
Features to Be Tested

List all features of the SUT that will be tested under this
List all features of the SUT that will be tested under this
plan
plan
Features Not to Be Tested

List all features of the SUT that will not be tested under
List all features of the SUT that will not be tested under
this plan
this plan
Between the test items, features to be tested and features not t Between the test items, features to be tested and features not to be o be
tested we have scope of the project tested we have scope of the project
Approach

Describes the approach to testing the SUT
Describes the approach to testing the SUT

This should be high level, but sufficient to estimate the time
This should be high level, but sufficient to estimate the time
and resources required
and resources required
What this approach will achieve What this approach will achieve
Specify major activities Specify major activities
Testing techniques Testing techniques
Testing tools / aids Testing tools / aids
Constraints to testing Constraints to testing
Support required Support required - - environment & staffing environment & staffing
Item Pass / Fail Criteria

How to judge whether a test item has passed
How to judge whether a test item has passed

Expected vs. Actual results
Expected vs. Actual results

Certain % of tests pass
Certain % of tests pass

Number of faults remaining (known and estimated)
Number of faults remaining (known and estimated)

Should be defined for each test item
Should be defined for each test item
Suspension criteria & resumption requirements

Reasons that would cause testing to be suspended
Reasons that would cause testing to be suspended

Steps necessary to resume testing
Steps necessary to resume testing
Test Deliverables

Everything that goes to make up the tests
Everything that goes to make up the tests

All documentation
All documentation
e.g. Specification, test plans, procedures, reports e.g. Specification, test plans, procedures, reports

Code releases
Code releases

Testing tools
Testing tools
Test management tools, automation tools, excel, word etc Test management tools, automation tools, excel, word etc

Test systems
Test systems
Manual and automated test cases Manual and automated test cases
Testing Tasks

Preparation to perform testing
Preparation to perform testing
Test case identification Test case identification
Test case design Test case design
Test data storage Test data storage
Baseline application Baseline application

Special skills needed
Special skills needed
Spreadsheet skills, test analysis, automation etc Spreadsheet skills, test analysis, automation etc

Inter
Inter
-
-
dependencies
dependencies
Environment

Requirements for test environment
Requirements for test environment

Hardware & software
Hardware & software
PCs, servers, routers etc PCs, servers, routers etc
SUT, interfaced applications, databases SUT, interfaced applications, databases

Configuration
Configuration
Maybe operating systems or middleware to test against Maybe operating systems or middleware to test against

Facilities
Facilities
Office space, desks, internet access Office space, desks, internet access
Responsibilities

Who is responsible?
Who is responsible?

For which activities
For which activities

For which deliverables
For which deliverables

For the environment
For the environment
Staffing and Training Needs

Staff required
Staff required
Test managers, team leaders, testers, test analysts Test managers, team leaders, testers, test analysts

Skill levels required
Skill levels required
Automation experience Automation experience
Spreadsheet skills, etc Spreadsheet skills, etc

Training requirements
Training requirements
Tools specific training Tools specific training
Refresher courses, etc. Refresher courses, etc.
Overall project resource requirements Overall project resource requirements
Schedule

Timescales, dates and milestones
Timescales, dates and milestones

Resources required to meet milestones
Resources required to meet milestones

Availability of software and environment
Availability of software and environment

Deliverables
Deliverables
Risks and Contingencies

What might go wrong?
What might go wrong?

Actions for minimising impact on testing should things go
Actions for minimising impact on testing should things go
wrong
wrong
Approvals

Who has approved the test plan
Who has approved the test plan

Names and dates of approval
Names and dates of approval

Why is it so important
Why is it so important
Evidence that the document has been viewed Evidence that the document has been viewed
Shows that the approach has been agreed and has the backing Shows that the approach has been agreed and has the backing
of those who matter of those who matter
You have commitment, now make them stick to it! You have commitment, now make them stick to it!

Summary
Summary

Test plans are created to ensure that the requirements are
Test plans are created to ensure that the requirements are
understood
understood

To ensure that maximum test coverage is achieved
To ensure that maximum test coverage is achieved

To identify all items that will form the test
To identify all items that will form the test
Acceptance Testing
Acceptance Testing
Acceptance Testing


Understand what acceptance testing is
Understand what acceptance testing is
Why you would want to do it Why you would want to do it
How you would plan it and prepare for it How you would plan it and prepare for it
What you need to actually do it What you need to actually do it

Understand the different types of acceptance testing
Understand the different types of acceptance testing
Introduction

What is Acceptance Testing ?
What is Acceptance Testing ?
"Formal testing conducted to enable a user, customer or other
"Formal testing conducted to enable a user, customer or other
authorised entity to determine whether to accept a system or
authorised entity to determine whether to accept a system or
component."
component."
User Acceptance Testing

What is User Acceptance Testing (UAT)?
What is User Acceptance Testing (UAT)?

Exactly what it says it is!!
Exactly what it says it is!!

Users of end product conducting the tests
Users of end product conducting the tests

The set
The set
-
-
up will represent a working Environment
up will represent a working Environment

Covers all areas of a project, not just the system
Covers all areas of a project, not just the system

A.K.A Business Acceptance Testing or Business Process
A.K.A Business Acceptance Testing or Business Process
Testing
Testing
Systems
Test
UAT
Alpha /
Beta
Unit
Test
Where does it fit in?
level
time
Service Level
Testing
Operational
Requirements
Design
Specifications
Functional
Specification
Component
Testing
System
Testing
Integration
Testing (large)
User Acceptance
Testing
Program
Specification
Business
Requirements
A typical development life-cycle V-Model
Planning UAT

Why plan?
Why plan?

Things to consider
Things to consider

Preparing Tests
Preparing Tests

Manual
Manual

Automated
Automated
Why Plan?

If you don
If you don
t then how do you know you have achieved

t then how do you know you have achieved
what you set out to do!
what you set out to do!

Avoids repetition
Avoids repetition

Test according to code releases
Test according to code releases

Makes efficient and effective use of time and resources
Makes efficient and effective use of time and resources
Things to consider

Timescales
Timescales

Resources
Resources

Availability
Availability
Preparing your tests

Take a logical approach
Take a logical approach

Identify the business processes
Identify the business processes

Separate into everyday business scenarios
Separate into everyday business scenarios
How the business fits together...
The Business
Process Process Process
Task Task Task Task Task Task Task Task Task
Data Requirements

Copied Environments
Copied Environments

Created Environments
Created Environments
Running the Tests

Order of Tests
Order of Tests

Confidence Checks
Confidence Checks

Automated and Manual test runs
Automated and Manual test runs
Contract Acceptance

A demonstration of the acceptance criteria

A demonstration of the acceptance criteria

Acceptance criteria will have been defined in the contract
Acceptance criteria will have been defined in the contract

Before the software is accepted it is necessary to show that
Before the software is accepted it is necessary to show that
its matches its
its matches its
specification as defined in the criteria

specification as defined in the criteria
Alpha Testing

Letting your customers do your testing
Letting your customers do your testing

Requires a stable version of the software
Requires a stable version of the software

People who represent your target market use the product in
People who represent your target market use the product in
the same way(s) as if they had bought the finished version
the same way(s) as if they had bought the finished version

Alpha testing is conducted in
Alpha testing is conducted in
-
-
house (at the developers site)
house (at the developers site)
Beta Testing

As Alpha testing but users perform tests at their site
As Alpha testing but users perform tests at their site
Acceptance Testing

Summary
Summary

You can UAT anything!
You can UAT anything!

Every Deliverable should pass through UAT
Every Deliverable should pass through UAT

User representation is VITAL
User representation is VITAL

If the product does not pass UAT then a decision about
If the product does not pass UAT then a decision about
implementation needs to be made
implementation needs to be made
Integration Testing in the Large


Understand what Integration Testing is
Understand what Integration Testing is

Understand what Integration Testing in the Large is
Understand what Integration Testing in the Large is

Look at how & why we test system integration
Look at how & why we test system integration

Integration testing
Integration testing
Testing performed to expose faults in the interfaces and in the Testing performed to expose faults in the interfaces and in the
interaction between integrated components interaction between integrated components

Integration testing in the large
Integration testing in the large
Integration Testing in the Large is testing performed to expose Integration Testing in the Large is testing performed to expose
faults in the interfaces and in the interaction between systems faults in the interfaces and in the interaction between systems

Why do we need to do integration testing in the large
Why do we need to do integration testing in the large
J ust because the SUT works in the test lab doesn J ust because the SUT works in the test lab doesn t mean it will t mean it will
work in the real world work in the real world

All components of a computer system work on only
All components of a computer system work on only
one thing
one thing
DATA
DATA

Data is manipulated within the SUT
Data is manipulated within the SUT

It may then passed to another system
It may then passed to another system

This is done to satisfy a business process requirement
This is done to satisfy a business process requirement
Env
Env
Env
Env
Env
Env
SUT
SUT
SUT
SUT
SUT
SUT
A typical system
Data Model
Data Model
Databases
Databases
Data
Data
Ext apps.
Ext apps.

Integration Testing on a Large Scale
Integration Testing on a Large Scale

Test the way the SUT interfaces with external systems
Test the way the SUT interfaces with external systems

The elements may run on the same and / or different
The elements may run on the same and / or different
platforms
platforms

They may be located in separate locations and use
They may be located in separate locations and use
communication protocols to pass data back & forth
communication protocols to pass data back & forth

Typical approach
Typical approach

Test the system
Test the system
In a test environment In a test environment
Use stubs & drivers where external systems aren Use stubs & drivers where external systems aren t available t available

Test the system
Test the system
in situ
in situ
In a replica of the production / live environment In a replica of the production / live environment
Use stubs & drivers where external systems aren Use stubs & drivers where external systems aren t available t available

Test the system and its
Test the system and its
interfaces with other systems

interfaces with other systems
In a replica of the production / live environment In a replica of the production / live environment
Access test versions of the external systems Access test versions of the external systems

Fundamentally we are still looking at
Fundamentally we are still looking at
DATA
DATA

& how that data is obtained, processed, manipulated, stored
& how that data is obtained, processed, manipulated, stored
and made available or transported for use elsewhere within or
and made available or transported for use elsewhere within or
outside the
outside the
organisation
organisation

Additional challenges posed by large scale:
Additional challenges posed by large scale:

Multiple Platforms
Multiple Platforms

Communications between platforms
Communications between platforms

Management of the environments
Management of the environments

Coordination of changes
Coordination of changes

Different technical skills required
Different technical skills required
The approach used
Integration Testing

Planning testing
Planning testing

We need to determine when and at what levels Integration
Testing will be performed

Identify the boundaries

A similar process will need to be followed for all elements
Once these elements have been tested individually they are Once these elements have been tested individually they are
further integrated to support the business process further integrated to support the business process
Integration Testing

Approaches to integration testing

Incremental
Incremental
Top down Top down
Bottom Bottom- -up up
Functional Functional

Non
Non
-
-
incremental
incremental
Big bang Big bang

Summary
Summary

Integration testing is needed to test how components interact
with each other and how the system under test works with
other systems
other systems

Large scale integration testing is needed to test how the
Large scale integration testing is needed to test how the
system under test works with other systems
system under test works with other systems

J ust because a system works in a test environment doesn
J ust because a system works in a test environment doesn
t
t
mean it will work in a live environment
mean it will work in a live environment
Non
Non
-
-
Functional System Testing
Non-Functional System Testing


Understand what Non
Understand what Non
-
-
Functional System Testing is
Functional System Testing is

Understand the need for non
Understand the need for non
-
-
functional system testing

Look at the various types of non
Look at the various types of non
-
-

Non
Non
-
-
Functional System Testing is defined as:
Functional System Testing is defined as:
Testing Of Those requirements that do not relate to the

Testing Of Those requirements that do not relate to the
Functionality e.g. Performance, Load, usability, security etc
Functionality e.g. Performance, Load, usability, security etc

Users, generally, focus on the Functionality of the system
Users, generally, focus on the Functionality of the system

Functional testing may show that the system performs as per
Functional testing may show that the system performs as per
the requirements
the requirements

Will the system work if now deployed?
Will the system work if now deployed?

There are other factors critical to the successful use of a
There are other factors critical to the successful use of a
system
system
Security Testing
Testing whether the system meets specified security

Testing whether the system meets specified security
requirements
requirements

How easy is it for an unauthorised user to access the
How easy is it for an unauthorised user to access the
system?
system?

How easy is it for an unauthorised person to access the data?
How easy is it for an unauthorised person to access the data?
Usability Testing
Testing the ease with which users can learn and use the
Testing the ease with which users can learn and use the
product
product

Employ people to use the application as a user and study
Employ people to use the application as a user and study
how they use it
how they use it

A beta
A beta
-
-
test may be a cheap way of doing Usability testing
test may be a cheap way of doing Usability testing

There is no simple way to examine HOW people use the
There is no simple way to examine HOW people use the
product
product
Storage Testing
Testing whether the system meets its specified storage

Testing whether the system meets its specified storage
objectives
objectives

Study how memory and storage is used by the product
Study how memory and storage is used by the product

Predict when extra capacity may be needed
Predict when extra capacity may be needed
Installability Testing
Testing concerned with the installation procedures for the

Testing concerned with the installation procedures for the
system
system

Does the installation work?
Does the installation work?

Is it easy to carry out?
Is it easy to carry out?

Does it uninstall correctly?
Does it uninstall correctly?
Documentation Testing
Testing concerned with the accuracy of the documentation

Testing concerned with the accuracy of the documentation

Recovery Testing
Testing aimed at verifying the system

Testing aimed at verifying the system
s ability to recover from

s ability to recover from
varying degrees of failure
varying degrees of failure

Should include recovery from system back
Should include recovery from system back
-
-
ups
ups
Load Testing
Testing geared to assessing the applications ability to deal wit

Testing geared to assessing the applications ability to deal wit
h
h
the expected throughput of data & users
the expected throughput of data & users

Stress Testing
Assesses individual components by exercising them to and

Assesses individual components by exercising them to and
beyond the limits of expected use
beyond the limits of expected use

Performance Testing
Tests the efficiency of individual components of an application

Tests the efficiency of individual components of an application

Volume Testing
Testing where the system is subjected to large volumes of

Testing where the system is subjected to large volumes of
data
data


Summary
Summary

J ust because a system
J ust because a system
s functions have been tested doesn

s functions have been tested doesn
t
t
mean that testing is complete
mean that testing is complete

There are a range of non
There are a range of non
-
-
functional tests that need to be
functional tests that need to be
performed upon a system
performed upon a system


Understand what functional system testing is
Understand what functional system testing is

Understand the benefits of functional system testing
Understand the benefits of functional system testing

What is System Testing?
What is System Testing?

Testing of the complete system
Testing of the complete system
May be the last step in integration testing in the small May be the last step in integration testing in the small
May be the first time that enough of the system has been put May be the first time that enough of the system has been put
together to make a working system together to make a working system

Ideally done by an independent test team
Ideally done by an independent test team

Two types
Two types
-
-
functional and non
functional and non
-
-
functional
functional

A Functional Requirement is
A Functional Requirement is
A requirement that specifies a function that a system or system

A requirement that specifies a function that a system or system
component must perform
component must perform

Functional System testing is geared to checking the function
Functional System testing is geared to checking the function
of the system against specification
of the system against specification

May be requirements based or business process based
May be requirements based or business process based

Testing Based on Requirements
Testing Based on Requirements

Requirements specification used to derive test cases
Requirements specification used to derive test cases

System is tested to ensure the requirements can be met
System is tested to ensure the requirements can be met

Testing carried out against Business processes
Testing carried out against Business processes

Based on expected use of the system
Based on expected use of the system

Builds use cases
Builds use cases
-
-
test cases that reflect actual or expected
test cases that reflect actual or expected
use of the system
use of the system

Summary
Summary

Functional system testing allows us to test the system against
Functional system testing allows us to test the system against
the specifications, user requirements and business processes
the specifications, user requirements and business processes
Integration Testing in the Small


Understand what
Understand what

is
is

Look at how & why we do
Look at how & why we do



Integration testing
Integration testing
interaction between integrated components interaction between integrated components

Integration testing in the small
Integration testing in the small
interaction between components interaction between components
A sample system
Databases
Databases
External
systems
External
systems
The SUT
- made up of 000s of units
The SUT
- made up of 000s of units

All components of a computer application work on only
All components of a computer application work on only
one thing
one thing
DATA
DATA

Data is passed from one component to another
Data is passed from one component to another

Data is passed from one system to another
Data is passed from one system to another

Software components will be required to add, amend, &
Software components will be required to add, amend, &
delete information, validate the information and report on the
delete information, validate the information and report on the
information
information

Integration testing
Integration testing

Testing should start with the smallest, definable component
Testing should start with the smallest, definable component

Building tested components up to the next level of definition
Building tested components up to the next level of definition

Eventually the complete business processes is tested
Eventually the complete business processes is tested
End to
End to
End
End


Testing components
Testing components

Each component needs to be tested individually to ensure it
Each component needs to be tested individually to ensure it
processes the data as required
processes the data as required

This is Component Testing
This is Component Testing

All possible options & paths should be tested
All possible options & paths should be tested

Once components have been tested we can link them
Once components have been tested we can link them
together to see if they can work together
together to see if they can work together

This is Integration Testing in the Small
This is Integration Testing in the Small
Linking components together to ensure that they communicate Linking components together to ensure that they communicate
correctly correctly
Gradually linking more and more components together Gradually linking more and more components together
Prove that the data is passed between components as required Prove that the data is passed between components as required
Steadily increase the number of components and create & test Steadily increase the number of components and create & test
sub sub- -systems and then the complete system systems and then the complete system
The approach used
Integration Testing

Planning testing
Planning testing



Once these elements have been tested individually they are Once these elements have been tested individually they are
further integrated to support the business process further integrated to support the business process
Integration Testing

Stubs and drivers
Stubs and drivers

Used to emulate the
Used to emulate the
missing bits
missing bits
External systems External systems

Sub Sub- -systems systems
Missing components Missing components

May need to be written specifically for the SUT
May need to be written specifically for the SUT
Integration Testing


Incremental
Incremental
Top down Top down
Bottom Bottom- -up up
Functional Functional

Non
Non
-
-
incremental
incremental
Big bang Big bang

Summary
Summary

other systems
other systems

Integration testing in the small is concerned with the
Integration testing in the small is concerned with the
interaction between the various components of a system
interaction between the various components of a system
Component Testing
Component Testing
Component Testing


Understand what Component Testing is
Understand what Component Testing is

Look at the Component Testing Process
Look at the Component Testing Process

Look at the myriad types of Component Testing
Look at the myriad types of Component Testing
Component Testing

What is component testing?
What is component testing?

The testing of an individual software component

The testing of an individual software component

What is a component?
What is a component?

A minimal software item for which a separate specification is

A minimal software item for which a separate specification is
available
available

Component Testing

Component testing is the lowest form of testing
Component testing is the lowest form of testing

At the bottom of the V
At the bottom of the V
-
-
model
model

Each component is tested in isolation
Each component is tested in isolation
Prior to being integrated into the system Prior to being integrated into the system

Involves testing the code itself
Involves testing the code itself
Test the code in the greatest detail Test the code in the greatest detail
Usually done by the components developer Usually done by the components developer
Component Testing

Component Test Process
Component Test Process

Component Test Planning
Component Test Planning

Component Test Specification
Component Test Specification

Component Test Execution
Component Test Execution

Component Test Recording
Component Test Recording

Checking for Component Test Completion
Checking for Component Test Completion
Component Testing
Testing techniques Testing techniques
Equivalence Partitioning Equivalence Partitioning
Boundary Value Analysis Boundary Value Analysis
State Transition Testing State Transition Testing
Cause Cause- -Effect Graphing Effect Graphing
Syntax Testing Syntax Testing
Statement Testing Statement Testing
Branch/Decision Testing Branch/Decision Testing
Data Flow Testing Data Flow Testing
Branch Condition Testing Branch Condition Testing
Branch Condition Branch Condition
Combination Testing Combination Testing
Modified Condition Decision Modified Condition Decision
Testing Testing
LCSAJ Testing LCSAJ Testing
Random Testing Random Testing
Other Testing Techniques Other Testing Techniques
For definitions see BS7925 For definitions see BS7925- -2 2
Component Testing

What about
What about
Playtime
Playtime
?
?

Often a short unscripted session with a component can be of
Often a short unscripted session with a component can be of
benefit
benefit

More common at later stages (once formal test techniques
More common at later stages (once formal test techniques
have been completed)
have been completed)
Component Testing

Not done too well?
Not done too well?

Some basics would quickly improve it:
Some basics would quickly improve it:

Checklists
Checklists

Standards
Standards

Procedures
Procedures

Code & version control
Code & version control

Sign off
Sign off
Component Testing

Summary
Summary

Component testing is intended to test a software component
Component testing is intended to test a software component
before it is knitted into the system
before it is knitted into the system

There are lots of different approaches
There are lots of different approaches
Maintenance Testing
Maintenance Testing
Maintenance Testing


Look at the challenges that face testers when an application
Look at the challenges that face testers when an application
changes post implementation
changes post implementation

See how to ensure that maintenance applied to the system
See how to ensure that maintenance applied to the system
does not cause failures
does not cause failures
Maintenance Testing

What is Maintenance Testing?
What is Maintenance Testing?

Testing of changes to existing, established systems
Testing of changes to existing, established systems
Where maintenance has been performed on the (old) code Where maintenance has been performed on the (old) code

Checking that the fixes have been made and that the system
Checking that the fixes have been made and that the system
has not regressed
has not regressed
Maintenance Testing

The Challenge
The Challenge

Old Code
Old Code

No Documentation
No Documentation

Out of Date Documentation (worse)
Out of Date Documentation (worse)

How much testing to do and when to stop testing
How much testing to do and when to stop testing
Mitigation of risk vs. lost revenue Mitigation of risk vs. lost revenue
Maintenance Testing

Often applications will need maintenance
Often applications will need maintenance

In doing this minor changes will be required
In doing this minor changes will be required

These need to be tested quickly & effectively to allow service
These need to be tested quickly & effectively to allow service
to be restored
to be restored

These systems may have been in place and working for
These systems may have been in place and working for
years
years
They are likely to be vital to the running of the business They are likely to be vital to the running of the business
They may be in use 24/7 They may be in use 24/7
Maintenance Testing

You need to be able to test the changes quickly and
You need to be able to test the changes quickly and
effectively
effectively

As well as being able to prove that the change introduced
As well as being able to prove that the change introduced
does not impact the other functions
does not impact the other functions

A Regression test is in order
A Regression test is in order
Maintenance Testing

A regression test is
A regression test is

Retesting of a previously tested program following

Retesting of a previously tested program following
modification to ensure that faults have not been introduced or
modification to ensure that faults have not been introduced or
uncovered as a result of the changes made
uncovered as a result of the changes made

Maintenance Testing

Other Risks
Other Risks

Reactive, high risk when changing
Reactive, high risk when changing

Impact analysis is vital but is difficult
Impact analysis is vital but is difficult

Stopping testing too soon
Stopping testing too soon
-
-
errors may be missed
errors may be missed

Stopping testing too late
Stopping testing too late
-
-
missed business through delayed
missed business through delayed
implementation
implementation
Maintenance Testing

Summary
Summary

All established systems need maintenance from time to time
All established systems need maintenance from time to time

The changed code / function will need to be tested
The changed code / function will need to be tested

A regression test will need to be executed to ensure that the
A regression test will need to be executed to ensure that the
change(s) have been made correctly and that they have not
change(s) have been made correctly and that they have not
affected some other part of the system
affected some other part of the system

Impact analysis is key, but difficult
Impact analysis is key, but difficult
Black & White Box Testing


Understand the differences between Black & White Box
Understand the differences between Black & White Box
testing and where they feature in a testing lifecycle
testing and where they feature in a testing lifecycle

Understand how a systematic approach provides confidence
Understand how a systematic approach provides confidence

Understand how tools can be used to improve and increase
Understand how tools can be used to improve and increase
productivity
productivity

Black Box testing
Black Box testing

Test Case selection that is based on an analysis of the

specification of a component without reference to its internal
workings
workings

White Box testing
White Box testing

Test Case selection based on an analysis of the internal

structure of a component

Black Box Testing

Concentrates on the Business Function
Concentrates on the Business Function

Can be used throughout testing cycle
Can be used throughout testing cycle

Dominates the later stages of testing
Dominates the later stages of testing
Although is relevant throughout the development life cycle Although is relevant throughout the development life cycle

Little / No knowledge of the underlying code needed
Little / No knowledge of the underlying code needed
Black Box Testing
We have all done it! We have all done it!
What do you do when What do you do when
changing a light bulb? changing a light bulb?
`Remove Old Bulb `Remove Old Bulb
Insert New Bulb Insert New Bulb
Switch light / lamp on Switch light / lamp on
Switch light / lamp off Switch light / lamp off
White Box Testing

Testing at a detailed level
Testing at a detailed level

A.K.A.
A.K.A.
Glass Box testing

Glass Box testing
or
or
Structural testing
Structural testing

Focuses on Lines of Code
Focuses on Lines of Code

Looks at specific conditions
Looks at specific conditions

Looks at the mechanics of the Application
Looks at the mechanics of the Application

Useful in the early stages of testing.
Useful in the early stages of testing.
White Box Testing

Requires a detailed knowledge of the code
Requires a detailed knowledge of the code

Business Function / Process not a prime consideration
Business Function / Process not a prime consideration

Aim to achieve code coverage, not just functional coverage
Aim to achieve code coverage, not just functional coverage

Plays a lesser role later in the testing cycle
Plays a lesser role later in the testing cycle
White Box Testing

Code Coverage objective
Code Coverage objective

To devise tests that exercise each significant line of program
To devise tests that exercise each significant line of program
code at least once
code at least once

This does not mean every combination of paths and data
This does not mean every combination of paths and data
-
-
which is normally unattainable
which is normally unattainable

Techniques and measurements
Techniques and measurements

Systematic techniques exist for black and white box testing
Systematic techniques exist for black and white box testing
Give us a systematic approach to testing Give us a systematic approach to testing
Test cases are developed in a logical manner Test cases are developed in a logical manner
Enable us to have a sub Enable us to have a sub- -set of tests that have a high probability set of tests that have a high probability
of finding faults of finding faults

By using defined techniques we should be able to produce
By using defined techniques we should be able to produce
corresponding measurements
corresponding measurements
Means we can see how testing is progressing Means we can see how testing is progressing

Software Testing Tools
Software Testing Tools

Are useful for both Black & White Box Testing
Are useful for both Black & White Box Testing

Can increase productivity and quality of tests
Can increase productivity and quality of tests

Particularly useful for white box testing
Particularly useful for white box testing

Summary
Summary

A systematic approach is needed for both
A systematic approach is needed for both

Tests need to be planned, prepared, executed and verified
Tests need to be planned, prepared, executed and verified
against the code
against the code

Expected results need to be defined and understood
Expected results need to be defined and understood

Tools can help increase productivity and quality
Tools can help increase productivity and quality
Black box test techniques


Understand what black box testing is
Understand what black box testing is

Look at some of the different types of black box testing
Look at some of the different types of black box testing

Black box testing is
Black box testing is


workings
workings


Why do we need black box test techniques?
Why do we need black box test techniques?

Exhaustive testing is not possible
Exhaustive testing is not possible
Due to the constraints of time, money and resources Due to the constraints of time, money and resources

Therefore we must create a sub
Therefore we must create a sub
-
-
set of tests
set of tests
These must have fewer tests, but should not reduce coverage These must have fewer tests, but should not reduce coverage

We should also focus on areas of likely risk
We should also focus on areas of likely risk
Those places where mistakes may occur Those places where mistakes may occur

Each black box test technique has
Each black box test technique has

A technique
A technique
i.e. how to do it i.e. how to do it

A test case design approach
A test case design approach
How to create test cases using the approach How to create test cases using the approach

See BS7925
See BS7925
-
-
2 for detailed information
2 for detailed information

BS7925
BS7925
-
-
2 lists all the black box test techniques
2 lists all the black box test techniques

Equivalence Partitioning

Boundary Value Analysis

State Transition
State Transition

Cause & Effect Graphing
Cause & Effect Graphing

Syntax Testing
Syntax Testing

Random Testing
Random Testing

Uses a model of the component to

Partition Input & Output values into sets such that each value
Partition Input & Output values into sets such that each value
within a set can be reasonably expected to be treated in the
within a set can be reasonably expected to be treated in the
same manner
same manner

Therefore only one example of that set needs to be input to or
Therefore only one example of that set needs to be input to or
resultant from the component
resultant from the component

Test Case Design
Test Case Design

Inputs to the component

Partitions exercised:
Partitions exercised:
Valid partitions Valid partitions
Invalid partitions Invalid partitions

Expected results
Expected results

Example
Example

Course grade
Course grade
Coursework Coursework - - out of 25 marks out of 25 marks
Examination Examination - - out of 75 marks out of 75 marks
Grades A Grades A- -D based on Coursework and Examination marks D based on Coursework and Examination marks
Identify Valid Partition and Invalid Partitions Identify Valid Partition and Invalid Partitions
Define test cases including expected results Define test cases including expected results


Identify the values close to and on partition boundaries
Identify the values close to and on partition boundaries

For input and output, valid & invalid data
For input and output, valid & invalid data

Chosen specifically to exercise the boundaries of the area
Chosen specifically to exercise the boundaries of the area
under test
under test

Test Case Design
Test Case Design


Boundaries to be exercised:
Boundaries to be exercised:
J ust below J ust below
On On
J ust above J ust above

Expected results
Expected results

Examples
Examples

Using the grading system defined earlier
Using the grading system defined earlier

The exam mark and the coursework mark both have two
The exam mark and the coursework mark both have two
boundaries
boundaries

Select values on, immediately above and immediately below
Select values on, immediately above and immediately below
each boundary
each boundary
State Transition

Uses analysis of the specification of the component to
Uses analysis of the specification of the component to
model its behaviour by state transitions
model its behaviour by state transitions

Identifies
Identifies
The various states that the software may be in The various states that the software may be in
The transitions between these points The transitions between these points
The events that cause the transactions The events that cause the transactions
Actions resulting from the transactions Actions resulting from the transactions
State Transition

Test Case Design
Test Case Design

Identify possible transitions
Identify possible transitions

Identify initial state and the data / events that will trigger t
Identify initial state and the data / events that will trigger t
he
he
transition
transition

Identify the expected results
Identify the expected results
State Transition

Example
Example

Using the course grade example
Using the course grade example

There is a grade indicator on screen
There is a grade indicator on screen
When coursework and examination marks are entered it When coursework and examination marks are entered it
displays the persons grade (A displays the persons grade (A- -D) D)

Select sets of data to test for each grade
Select sets of data to test for each grade
And extra sets of data to change from one grade to the next And extra sets of data to change from one grade to the next
Other techniques

Cause & Effect Graphing & Tabling
Cause & Effect Graphing & Tabling
Analysis of the specification to produce a graphical model of Analysis of the specification to produce a graphical model of
causes and their effects causes and their effects
Cases expressed as Boolean conditions in the form of a Cases expressed as Boolean conditions in the form of a
decision table decision table

Syntax Testing
Syntax Testing
Uses the syntax of the components Uses the syntax of the components inputs as the basis for the inputs as the basis for the
design of the test case design of the test case

Random Testing
Random Testing
Not as odd a concept as it sounds Not as odd a concept as it sounds
Not Ad Not Ad- -Hoc testing, but a means of testing functionality with a Hoc testing, but a means of testing functionality with a
randomly selected set of values randomly selected set of values
Black Box Testing Techniques

Summary
Summary

Black box testing concentrates on testing the features of the
Black box testing concentrates on testing the features of the
system
system

Techniques enable us to maximise testing
Techniques enable us to maximise testing
Create an achievable set of tests that offer maximum coverage Create an achievable set of tests that offer maximum coverage
Ensure possible areas of risk are tested Ensure possible areas of risk are tested

Black box testing is relevant throughout the testing process
Black box testing is relevant throughout the testing process
White Box Test Techniques


Understand what White Box testing is
Understand what White Box testing is

Look at some of the different types of White Box testing
Look at some of the different types of White Box testing

White Box testing
White Box testing



Also known as Glass Box testing, Clear Box testing
Also known as Glass Box testing, Clear Box testing
Why do we need White Box Why do we need White Box
techniques? techniques?
Provide formal structure to Provide formal structure to
testing code testing code
Enable us to measure how Enable us to measure how
much of a component has been much of a component has been
tested tested
Example Example
<100 lines of code <100 lines of code
100,000,000,000,000 possible 100,000,000,000,000 possible
paths paths
At 1,000 tests per second At 1,000 tests per second
would still take 3,170 years to would still take 3,170 years to
test all paths test all paths

To plan and design effective cases requires a
To plan and design effective cases requires a
knowledge of the
knowledge of the

Programming language used
Programming language used

Databases used
Databases used

Operating system(s) used
Operating system(s) used

And ideally knowledge of the code itself
And ideally knowledge of the code itself

BS7925
BS7925
-
-
2 lists all the white box test techniques
2 lists all the white box test techniques

Statement Testing
Statement Testing

Branch / Decision Testing
Branch / Decision Testing

Branch Condition Testing
Branch Condition Testing

Branch Condition Combination Testing
Branch Condition Combination Testing

Modified Condition Decision Testing
Modified Condition Decision Testing

Linear Code Sequence & J ump
Linear Code Sequence & J ump

Data Flow Testing
Data Flow Testing

Statement Testing
Statement Testing

A test case design technique for a component in which test

A test case design technique for a component in which test
cases are designed to execute statements
cases are designed to execute statements

Test cases are designed and run with the intention of
Test cases are designed and run with the intention of
executing every statement in a component
executing every statement in a component

Statement Testing example
Statement Testing example
a; a;
if (b) if (b)
{ {
c; c;
} }
d; d;
Any test case with b TRUE will achieve full statement coverage Any test case with b TRUE will achieve full statement coverage
NOTE: NOTE: Full statement coverage can be achieved without Full statement coverage can be achieved without
exercising with b FALSE exercising with b FALSE

Branch / Decision testing
Branch / Decision testing

A technique used to execute all branches the code may take
A technique used to execute all branches the code may take
based on decisions made
based on decisions made

Test Cases designed to ensure all branches & decision points
Test Cases designed to ensure all branches & decision points
are covered
are covered

Branch / Decision testing example
Branch / Decision testing example
a; a;
if (b) if (b)
{ {
c; c;
} }
d; d;
100% statement coverage requires 1 test case (b = True) 100% statement coverage requires 1 test case (b = True)
100% branch / decision coverage requires 2 test cases 100% branch / decision coverage requires 2 test cases
(b = True & b = False) (b = True & b = False)

Other techniques
Other techniques

Branch Condition testing, Branch Condition Combination
Branch Condition testing, Branch Condition Combination
testing & Modified Condition Decision testing
testing & Modified Condition Decision testing
Condition testing is based upon an analysis of the conditional Condition testing is based upon an analysis of the conditional
control flow within the component control flow within the component

LCSAJ
LCSAJ
Linear Code Sequence And J ump Linear Code Sequence And J ump

Data flow testing
Data flow testing
Tests the flow of data through a component Tests the flow of data through a component

Summary
Summary

White box testing can be done immediately after code is
White box testing can be done immediately after code is
written
written
Doesn Doesn t need the complete system t need the complete system
Does need knowledge of the code Does need knowledge of the code

A combination of all techniques are required for a successful
A combination of all techniques are required for a successful
test
test
Don Don t rely on just one technique t rely on just one technique
Error Guessing
Error Guessing
Error Guessing


Understand how you can use experience to
Understand how you can use experience to
predict
predict
where
where
errors may occur
errors may occur

Understand how this can benefit testing
Understand how this can benefit testing
Error Guessing

Used to guess where errors may be lurking
Used to guess where errors may be lurking

Based on information about how the system has been put
Based on information about how the system has been put
together and previous experience of testing it
together and previous experience of testing it

Use to complement more systematic techniques not instead
Use to complement more systematic techniques not instead
of
of

Not ad
Not ad
-
-
hoc testing, but testing that targets certain parts of the
hoc testing, but testing that targets certain parts of the
application
application
Error Guessing

Based on the user's or tester's experience of the
Based on the user's or tester's experience of the
commercial aspects of the system
commercial aspects of the system

e.g.
e.g.
Table based calculations such as calculating benefits payments Table based calculations such as calculating benefits payments
Where the user is allowed a high degree of flexibility in GUI Where the user is allowed a high degree of flexibility in GUI
navigation using multiple windows navigation using multiple windows
Error Guessing

Experience of the developers or the development cycle
Experience of the developers or the development cycle

e.g.
e.g.
Knowledge of an individual's Knowledge of an individual's style style
Knowledge of the development lifecycle and especially the Knowledge of the development lifecycle and especially the
change management process change management process
Error Guessing

Experience of the operating systems used
Experience of the operating systems used

e.g.
e.g.
It is known that Windows NT is better at storage management It is known that Windows NT is better at storage management
than Windows 95 than Windows 95
Error Guessing

Should still be planned
Should still be planned

Is part of the test process
Is part of the test process
Should be used as a supplement to systematic techniques Should be used as a supplement to systematic techniques

Ad
Ad
-
-
Hoc testing can be encouraged if
Hoc testing can be encouraged if
The tester maintains a log of those tests performed & is able to The tester maintains a log of those tests performed & is able to
reasonably recollect what actions were performed reasonably recollect what actions were performed

If you have the time to run an extra (Ad
If you have the time to run an extra (Ad
-
-
Hoc) test that passes
Hoc) test that passes
there is little loss
there is little loss
If it finds a defect then that If it finds a defect then that s a bonus s a bonus
Error Guessing

Summary
Summary

You can never do enough testing!
You can never do enough testing!
The more tests that you can run the more errors you may find The more tests that you can run the more errors you may find

Error guessing can help ensure that areas where defects are
Error guessing can help ensure that areas where defects are
likely to occur are fully tested
likely to occur are fully tested

Uses experience and gut feeling to supplement systematic
Uses experience and gut feeling to supplement systematic
testing techniques
testing techniques
Reviews and the Test Process
Static Testing

Over the next three sessions we will
Over the next three sessions we will

Discover the various approaches to Static Testing
Discover the various approaches to Static Testing

Discover the benefits of Static Testing
Discover the benefits of Static Testing

Discover the issues with Static Testing
Discover the issues with Static Testing
Static Testing

What is Static Testing?
What is Static Testing?

Testing of an object without execution on a computer


Static Testing
Session Topics
Session Topics
1. Reviews and the Test Process
2. Types of Review
2. Types of Review
3. Static Analysis
3. Static Analysis
Reviews & the Test Process

Session Objectives
Session Objectives

To identify what and when to Review
To identify what and when to Review

To cover the Costs & Benefits of Reviews
To cover the Costs & Benefits of Reviews

Testing without data
Testing without data



How is this done
How is this done

By reviewing the system deliverables
By reviewing the system deliverables

Why review
Why review

To identify faults as soon as possible in the development
To identify faults as soon as possible in the development
lifecycle
lifecycle

Reviews offer the chance to find faults in the system
Reviews offer the chance to find faults in the system
specifications
specifications

This should lead to:
This should lead to:
Development productivity improvements Development productivity improvements
Reduced development time Reduced development time- -scales scales
Lifetime cost reductions Lifetime cost reductions
Reduced fault levels Reduced fault levels

When to review?
When to review?

As soon as an object is ready, before it is used as a product
As soon as an object is ready, before it is used as a product
or the basis for the next step in development
or the basis for the next step in development

What to Review
What to Review

Anything and everything can be reviewed
Anything and everything can be reviewed

Requirements, System and Program Specifications should be
Requirements, System and Program Specifications should be
reviewed prior to publication
reviewed prior to publication

System Design deliverables should be reviewed both in terms
System Design deliverables should be reviewed both in terms
of functionality & technical robustness
of functionality & technical robustness

What should be reviewed?
What should be reviewed?

Program specifications should be reviewed before
Program specifications should be reviewed before
construction
construction

Programs should be reviewed before execution
Programs should be reviewed before execution

Test Plans should be reviewed before execution
Test Plans should be reviewed before execution

Test Results should be reviewed before implementation
Test Results should be reviewed before implementation

Reviews can be hazardous
Reviews can be hazardous

If misused they can lead to friction
If misused they can lead to friction

The errors & omissions found should be regarded as a good
The errors & omissions found should be regarded as a good
thing
thing

The author(s) should not take errors & omissions personally
The author(s) should not take errors & omissions personally

It is a positive step to identify defects and issues before the
It is a positive step to identify defects and issues before the
next stage
next stage

Building a Quality culture
Building a Quality culture

In order for them to work, Reviews must be regarded as a
In order for them to work, Reviews must be regarded as a
positive step and must be properly organised
positive step and must be properly organised

This is often a part of the company
This is often a part of the company
s Quality culture
s Quality culture
Define the review panel Define the review panel
Define the roles of its Define the roles of its members members
Define the review procedures Define the review procedures
Sell Sell the quality culture to staff the quality culture to staff

There are other dangers in a regime of regular reviews
There are other dangers in a regime of regular reviews

Lack of Preparation
Lack of Preparation

Familiarity breeds contempt

Familiarity breeds contempt

No follow up to ensure correction has been made
No follow up to ensure correction has been made

The wrong people doing them
The wrong people doing them

Systems development is very complex
Systems development is very complex

This means that it is difficult to build a system with all
This means that it is difficult to build a system with all
elements completely & accurately specified
elements completely & accurately specified

By reviewing a deliverable, its deficiencies can be identified
By reviewing a deliverable, its deficiencies can be identified
and remedied earlier
and remedied earlier
saving time and money

saving time and money
Test plans Test plans
Test cases Test cases
Test results Test results
Defects Defects
Causes of defect Causes of defect
Test metrics Test metrics
Development metrics Development metrics
Operational defects Operational defects
What else can Testers review?
We can measure how well we do

The costs of reviews
The costs of reviews

On
On
-
-
going reviews cost approximately 15% of development
going reviews cost approximately 15% of development
budget
budget

This includes activities such as the review itself, metric
This includes activities such as the review itself, metric
analysis & process improvement
analysis & process improvement

Reviews are highly cost effective
Reviews are highly cost effective
Finding and fixing faults in reviews is significantly cheaper th Finding and fixing faults in reviews is significantly cheaper than an
finding and fixing faults in later stages of testing finding and fixing faults in later stages of testing

The benefits of reviews
The benefits of reviews

Reviews save time and money
Reviews save time and money

Development productivity improvements
Development productivity improvements
People take greater care People take greater care
They have more pride in doing good work They have more pride in doing good work
They have a better understanding of what they are required to They have a better understanding of what they are required to
deliver deliver

Reduced development costs
Reduced development costs
Reduced fault levels Reduced fault levels
Reduced lifetime cost reductions Reduced lifetime cost reductions

Summary
Summary

Reviews enable us to test that the systems specification is
Reviews enable us to test that the systems specification is
correct and relates to the users
correct and relates to the users
requirements
requirements

Anything generated by a project can be reviewed
Anything generated by a project can be reviewed

In order for them to be effective, reviews must be well
In order for them to be effective, reviews must be well
managed
managed
Types of Review
Types of Review
Static Testing
Session Topics
Session Topics
2. Types of Review
2. Types of Review
3. Static Analysis
3. Static Analysis
Types of Review


Look at the types of review & the activities performed
Look at the types of review & the activities performed

Look at goals & deliverables
Look at goals & deliverables

Look at roles & responsibilities
Look at roles & responsibilities

Examine the suitability of each type of review
Examine the suitability of each type of review

Discover review pitfalls
Discover review pitfalls
Types of Review

Author
Author
s Review
s Review

Informal Review
Informal Review

Walkthroughs
Walkthroughs

Presentations
Presentations

Prototypes
Prototypes

Checklists
Checklists

Templates
Templates

Technical Review
Technical Review

Inspections
Inspections
Types of Review

Author
Author
s review
s review

Before issuing a document, the author should ALWAYS
Before issuing a document, the author should ALWAYS
check
check
Conformance to company standards Conformance to company standards
Fitness for purpose Fitness for purpose
Presentation Presentation
Spelling Spelling
Types of Review

Informal Review
Informal Review

It is very helpful to ask a colleague to read through the
It is very helpful to ask a colleague to read through the
deliverable to
deliverable to
Identify jargon to eliminate or translate Identify jargon to eliminate or translate
Make sure it all makes sense Make sure it all makes sense
Rationalise the sequence Rationalise the sequence
Identify where the author Identify where the author s familiarity with the subject assumes s familiarity with the subject assumes
the reader knows more than he or she actually does the reader knows more than he or she actually does

This is the least effective form of review as there is no
This is the least effective form of review as there is no
recognised way of measuring it
recognised way of measuring it
Types of Review

Walkthrough
Walkthrough

For some technical deliverables or complex processes it is
For some technical deliverables or complex processes it is
very useful for the author to walk through the document
very useful for the author to walk through the document

Reviewers are asked to comment and query the concepts
Reviewers are asked to comment and query the concepts
and processes that are to be delivered
and processes that are to be delivered
Types of Review

Presentations
Presentations

Used to explain to users your understanding of their
Used to explain to users your understanding of their
Requirements
Requirements

To explain how you will test those requirements i.e. your
To explain how you will test those requirements i.e. your
approach
approach

To gain agreement for the next step
To gain agreement for the next step
Types of Review

Prototypes
Prototypes

Allows the non
Allows the non
-
-
technician to see what is going to be built
technician to see what is going to be built

Enables the business to refine their business flows or adjust
Enables the business to refine their business flows or adjust
sequences on input screens
sequences on input screens

Is used to demonstrate and display features of the [proposed]
Is used to demonstrate and display features of the [proposed]
system
system
GUI screens, Database designs, Network architecture etc GUI screens, Database designs, Network architecture etc
Has been around for years in the form of mock ups and screen Has been around for years in the form of mock ups and screen
layouts layouts
Much improved with the advent of GUI Much improved with the advent of GUI s and tools to paint s and tools to paint
screens quickly and effectively screens quickly and effectively
Types of Review

Checklists
Checklists

Extremely simple and useful method of ensuring that all
Extremely simple and useful method of ensuring that all
elements of a deliverable have been considered
elements of a deliverable have been considered

Tends to ensure conformity of content, sequence and style
Tends to ensure conformity of content, sequence and style

Can be devised for almost any activity
Can be devised for almost any activity
Types of Review

Templates
Templates

Very useful to standardise the style, sequence & content of
Very useful to standardise the style, sequence & content of
code and documents
code and documents

Saves time re
Saves time re
-
-
inventing
inventing

Requires agreement by all who will use them
Requires agreement by all who will use them
Types of Review

Technical (Peer) Review
Technical (Peer) Review

Your Peers can also perform more formal technical reviews in
Your Peers can also perform more formal technical reviews in
larger groups
larger groups

Often used by workgroups as a self
Often used by workgroups as a self
-
-
checking mechanism
checking mechanism

It
It
s format is defined, agreed and practised with documented

s format is defined, agreed and practised with documented
processes and outputs
processes and outputs

Requires technical expertise
Requires technical expertise
Types of Review

Inspections
Inspections

Focus on reviewing code or documents
Focus on reviewing code or documents

Formal process based on rules and checklists
Formal process based on rules and checklists
Chaired by Moderator Chaired by Moderator
Every attendee has a defined role Every attendee has a defined role
Includes metrics and entry & exit criteria Includes metrics and entry & exit criteria
Training is required Training is required

Reviewers must be prepared prior to attendance
Reviewers must be prepared prior to attendance
Types of Review

Fagan Inspections
Fagan Inspections

Are
Are
very effective
very effective
-
-
to the point where they have replaced
to the point where they have replaced
unit testing
unit testing

Defects are reported
Defects are reported
-
-
no debate
no debate

Contentious issues and queries are resolved outside the
Contentious issues and queries are resolved outside the
inspection and the result reported back to the Moderator
inspection and the result reported back to the Moderator
Types of Review

Goals
Goals

Validation and verification against specifications and
Validation and verification against specifications and
standards
standards

Achieve consensus before moving on to next stage
Achieve consensus before moving on to next stage
Types of Review

Activities
Activities

Reviews need to be planned
Reviews need to be planned

They may require overview meetings
They may require overview meetings

Preparation
Preparation

The review meeting
The review meeting

Follow
Follow
-
-
up
up
Types of Review

Roles & responsibilities
Roles & responsibilities

For each type of review (especially the formal ones) roles and
For each type of review (especially the formal ones) roles and
responsibilities must be clearly defined
responsibilities must be clearly defined

Training may be required to understand and perform role
Training may be required to understand and perform role
Types of Review

Deliverables
Deliverables

Any project deliverable can (and should) be reviewed
Any project deliverable can (and should) be reviewed

Reviewing these items can lead to
Reviewing these items can lead to
Product changes Product changes
Source document changes Source document changes
Improvements (to both reviews and development) Improvements (to both reviews and development)
Types of Review

Pitfalls
Pitfalls

Lack of training
Lack of training
And therefore understanding of role, responsibilities and And therefore understanding of role, responsibilities and
purpose of review purpose of review

Lack of documentation
Lack of documentation

Lack of management support
Lack of management support
Types of Review

Review Management
Review Management

Formal reviews require proper management
Formal reviews require proper management
Allow people time to prepare Allow people time to prepare
Issue guidance notes (if not in Standards) Issue guidance notes (if not in Standards)
Give plenty of notice Give plenty of notice
Practice Presentations & Walkthroughs Practice Presentations & Walkthroughs
Sound out Sound out contentious issues contentious issues
ALWAYS document outcomes & Actions ALWAYS document outcomes & Actions
Types of Review
Walkthrough
What
Approach
Why / when
By Who
To Whom
Peer
Review
Technical
Reviews
Inspections
Everything
Informal
Always*
Peer
Author
Everything
Formal
Before detail
Author(s)
IT & Business
Specs & Code
Formal
Before sign off
Moderator
Reviewers
Always* : can be done at all points when a document is released.
Always* : can be done at all points when a document is released.
Req.'s & Designs
Informal
Before detail
Author(s)
IT & Business
Types of Review

Summary
Summary

There are various types of reviews
There are various types of reviews
Companies must decide which one(s) are best for them Companies must decide which one(s) are best for them

In order to gain maximum benefit from them they must be
In order to gain maximum benefit from them they must be
organised and implemented
organised and implemented
Static Analysis
Static Analysis
Static Testing
Session Topics
Session Topics
2. Types of Review
2. Types of Review
3. Static Analysis
3. Static Analysis
Static Analysis

In this session we will:
In this session we will:

Understand what static analysis is
Understand what static analysis is

Look at some of the elements of Static Analysis
Look at some of the elements of Static Analysis
Compilers Compilers
Static analysis tools Static analysis tools
Data flow analysis Data flow analysis
Control flow analysis Control flow analysis
Complexity analysis Complexity analysis
Static Analysis

What is Static Analysis?
What is Static Analysis?

Analysis of the code without dynamic execution
Analysis of the code without dynamic execution
Examine the logic and syntax of the code Examine the logic and syntax of the code

Attempting to identify faults in the code
Attempting to identify faults in the code

Provides metrics on flow through the SUT and its complexity
Provides metrics on flow through the SUT and its complexity

A form of automation
A form of automation
-
-
usually done with tools
usually done with tools
Static Analysis

What do we hope to find?
What do we hope to find?

Faults in the code
Faults in the code
Unreachable code Unreachable code
Undeclared variables Undeclared variables
Parameter type mismatches Parameter type mismatches
Uncalled functions and procedures Uncalled functions and procedures
Possible array boundary violations Possible array boundary violations
Static Analysis

What else will we get?
What else will we get?

Information on the code
Information on the code
Its complexity Its complexity
The flow of data through it The flow of data through it
The control flow through it The control flow through it
Metrics and measurements Metrics and measurements
Static Analysis

Compilers
Compilers

Convert program language code into Machine Code for faster
Convert program language code into Machine Code for faster
execution
execution

Will perform some basic Static Analysis
Will perform some basic Static Analysis
Usually produces a source code list Usually produces a source code list
Often produces a memory map Often produces a memory map
Often produces a cross Often produces a cross- -reference of labels and variables reference of labels and variables
Will do a syntax check on the code prior to compilation Will do a syntax check on the code prior to compilation
Static Analysis

Static Analysis Tools
Static Analysis Tools

A Static Analyser is used to parse the Source Code & identify
A Static Analyser is used to parse the Source Code & identify
defects, such as
defects, such as
Missing branches and links Missing branches and links
Non returnable Non returnable performs performs
Variables not initialised Variables not initialised
Standards not met Standards not met

Will also find any faults found by a Compiler
Will also find any faults found by a Compiler
Static Analysis

Data Flow Analysis
Data Flow Analysis

Technique to follow the path that various specific items of
Technique to follow the path that various specific items of
data take through the code of a program
data take through the code of a program

Looking for possible anomalies in the way the code handles
Looking for possible anomalies in the way the code handles
the data items
the data items
Variables used without being first created Variables used without being first created
Variables being Variables being used used after they have been after they have been killed killed

Consider a CRUD matrix to detail which components affect
Consider a CRUD matrix to detail which components affect
which data items
which data items
Static Analysis

Control Flow Analysis
Control Flow Analysis

Examines the flow of control through the code
Examines the flow of control through the code

Can identify
Can identify
Infinite loops Infinite loops
Cyclomatic complexity and other metrics Cyclomatic complexity and other metrics
Static Analysis
B1
B1
B2
B2
B3
B3
B8
B8
B5
B5
B9
B9
B6
B6
B7
B7
B4
B4
Control Flow Graph
Control Flow Graph
Static Analysis

Metrics
Metrics

Static Analysers generate metrics based on the code
Static Analysers generate metrics based on the code

Cyclomatic Complexity
Cyclomatic Complexity
Identifies the number of decision points in a control flow Identifies the number of decision points in a control flow
Can be calculated by counting the number of decision points in Can be calculated by counting the number of decision points in
a control flow graph and adding 1 a control flow graph and adding 1

Other Complexity Metrics
Other Complexity Metrics
Number of Lines of Code Number of Lines of Code
Number of nested statements Number of nested statements
Static Analysis

Summary
Summary

The inspection of program code and logic for errors
The inspection of program code and logic for errors

The code is analysed without being executed by the tool
The code is analysed without being executed by the tool

Errors are highlighted and metrics are produced
Errors are highlighted and metrics are produced
Test Organisation
Test Organisation
Test Organisation


Gain an appreciation of the many different testing structures
Gain an appreciation of the many different testing structures
in place
in place

Understand that it is likely that every organisation will have a
Understand that it is likely that every organisation will have a
unique testing set
unique testing set
-
-
up
up

Look at some of the more common, general set
Look at some of the more common, general set
-
-
ups and
ups and
appreciate the impact they may have on the success of the
appreciate the impact they may have on the success of the
testing carried out
testing carried out
Test Organisation

Testing may be the responsibility of the individual
Testing may be the responsibility of the individual
developers
developers

Developers are only responsible for testing their own work
Developers are only responsible for testing their own work

Goes against the ideal that testing ought to be independent
Goes against the ideal that testing ought to be independent

Tends to confirm the system works rather than detect errors
Tends to confirm the system works rather than detect errors
Test Organisation

Testing may be the responsibility of the developer
Testing may be the responsibility of the developer
team
team

A
A
Buddy system
Buddy system

Each developer is responsible for testing his / her colleagues
Each developer is responsible for testing his / her colleagues
work
work
Test Organisation

Each team has a single tester
Each team has a single tester

Is not involved in the actual development, solely concentrates
Is not involved in the actual development, solely concentrates
on testing
on testing

Able to work with the team throughout the development
Able to work with the team throughout the development
process
process

May be too close to the team to be objective
May be too close to the team to be objective
Test Organisation

Dedicated test teams may be in place
Dedicated test teams may be in place

These do no development whatsoever
These do no development whatsoever

Take a more objective view of the system under test
Take a more objective view of the system under test
Test Organisation

Internal Test Consultants
Internal Test Consultants

Provide testing advice to project teams
Provide testing advice to project teams

Involvement throughout the lifecycle
Involvement throughout the lifecycle
Test Organisation

Testing outsourced to specialist agencies
Testing outsourced to specialist agencies

Guarantees independence
Guarantees independence

Empathy with the Business of the organisation?
Empathy with the Business of the organisation?
Test Organisation
Production of Testing Production of Testing
Strategies Strategies
Creation of Test Plans Creation of Test Plans
Production of Testing Scripts Production of Testing Scripts
Proving of Testing Scripts Proving of Testing Scripts
Execution of Testing Scripts Execution of Testing Scripts
Logging Results Logging Results
Executing necessary re Executing necessary re- -tests tests
Automation expertise Automation expertise
Technical support Technical support
Database admin, Database admin,
environment admin environment admin
User Interface expertise User Interface expertise

Multi
Multi
-
-
disciplined teams needed to cover
disciplined teams needed to cover
Test Organisation

Testing teams will typically include
Testing teams will typically include

Test analysts to prepare, execute and analyse tests
Test analysts to prepare, execute and analyse tests

Test Consultants prepare strategies and plans
Test Consultants prepare strategies and plans

Test automation experts
Test automation experts

Database administrator or designer
Database administrator or designer

User interface experts
User interface experts

Test environment management
Test environment management

And others..
And others..
Test Organisation

Summary
Summary

Testing structures will vary between organisations
Testing structures will vary between organisations

Testing teams require a range of skills
Testing teams require a range of skills
Configuration Management


Gain an understanding of Configuration Management
Gain an understanding of Configuration Management

Look at the potential impact of poor Configuration
Look at the potential impact of poor Configuration
Management on testing
Management on testing

And understand how introducing Configuration Management
And understand how introducing Configuration Management
helps address those problems
helps address those problems

What is Configuration Management?
What is Configuration Management?

Ensuring that no
Ensuring that no
-
-
one is able to change a part of the system
one is able to change a part of the system
without proper procedures being in place
without proper procedures being in place

Closely linked to Version Control
Closely linked to Version Control

Version Control looks at each component
Version Control looks at each component

Holds the latest version of each component
Holds the latest version of each component

What versions of components works with others in a
What versions of components works with others in a
configuration
configuration

Without
Without
it,
it,
testing can be seriously impeded
testing can be seriously impeded

Developers may not be able to match source to object code
Developers may not be able to match source to object code

Simultaneous changes may made to the same source
Simultaneous changes may made to the same source
Although this may be allowed Although this may be allowed

Unable to identify source code changes made in a particular
Unable to identify source code changes made in a particular
version of the software
version of the software

Critical to successful testing
Critical to successful testing

It associates programs with specifications, with test plans,
It associates programs with specifications, with test plans,
with test data
with test data

When a component is successfully tested test results can be
When a component is successfully tested test results can be
included
included

It enables you to understand what versions of
It enables you to understand what versions of
components work with each other
components work with each other

It allows you to understand the relationship between test
It allows you to understand the relationship between test
cases, specifications, and components
cases, specifications, and components

If any configuration component is out of step
If any configuration component is out of step
-
-
Results
Results
are unpredictable
are unpredictable

This applies throughout the project lifecycle, and beyond
This applies throughout the project lifecycle, and beyond
is as old as the pyramids. It allows you at any point in

is as old as the pyramids. It allows you at any point in
time to examine the product of your work. (a program,
time to examine the product of your work. (a program,
requirements, tests etc.,) and know: Who, What When,
requirements, tests etc.,) and know: Who, What When,
Where, Why & How
Where, Why & How


Everything intended to last must be subject to
Everything intended to last must be subject to

Anything that isn
Anything that isn
t doesn
t doesn
t exist as part of the product

t exist as part of the product

Other Configuration Management terms

Configuration identification
Configuration identification
Requires that all Configuration Items (CI Requires that all Configuration Items (CI s) and their versions s) and their versions
are known are known

Configuration control
Configuration control
Maintenance of the Configuration Items in a library and Maintenance of the Configuration Items in a library and
maintenance of records on how Configuration Items change maintenance of records on how Configuration Items change
over time over time


Status accounting
Status accounting
The function recording and tracking problem reports, change The function recording and tracking problem reports, change
requests etc requests etc

Configuration auditing
Configuration auditing
The function to check on the contents of libraries etc for The function to check on the contents of libraries etc for
standards compliance standards compliance

Summary
Summary

Configuration Management is a necessary part of any system
Configuration Management is a necessary part of any system
development
development

All assets must be known and controlled
All assets must be known and controlled
Test Estimation, Monitoring & Control


Understand how we estimate how long we need for testing
Understand how we estimate how long we need for testing

Understand how we monitor the progress of testing once it
Understand how we monitor the progress of testing once it
has started
has started

Understand what steps we take to ensure that the effort
Understand what steps we take to ensure that the effort
progresses as smoothly as possible
progresses as smoothly as possible
Test Estimation
Test Estimation

Test Estimation
Test Estimation

The same as estimating for any project
The same as estimating for any project

We need to identify the number of tasks (tests) to be
We need to identify the number of tasks (tests) to be
performed, the length of each test, the skills and resources
performed, the length of each test, the skills and resources
required and the various dependencies
required and the various dependencies

Testing has a high degree of dependency
Testing has a high degree of dependency
You cannot test something until it has been delivered You cannot test something until it has been delivered
Faults found need to be fixed and re Faults found need to be fixed and re- -tested tested
The environment must be available whenever a test is to be run The environment must be available whenever a test is to be run

Factors to Consider
Factors to Consider

Risk of Failure
Risk of Failure

Complexity of Code / Solution
Complexity of Code / Solution

Criticality
Criticality

Coverage
Coverage

Stability Of SUT
Stability Of SUT

Testing is a tool to aid Risk Management
Testing is a tool to aid Risk Management

Consider the cost to the business of failure of a feature
Consider the cost to the business of failure of a feature

Test the most important features as soon as possible
Test the most important features as soon as possible

Be prepared to repeat these tests
Be prepared to repeat these tests

Consider the complexity of the supporting code
Consider the complexity of the supporting code

Estimate & plan tests for these ASAP
Estimate & plan tests for these ASAP

These are likely to be re
These are likely to be re
-
-
run many times, allow for this in
run many times, allow for this in
plans
plans

Data is critical to testing
Data is critical to testing

You can in certain situations load the data for testing & test i
You can in certain situations load the data for testing & test i
n
n
one pass
one pass

These tests will be run first but only once
These tests will be run first but only once
unless...
unless...

The testing environment may need to be re
The testing environment may need to be re
-
-
set before a test
set before a test
can be re
can be re
-
-
run
run

Use previous experience or best estimates to predict
Use previous experience or best estimates to predict
the reliability of components
the reliability of components

Plan to run tests for the unreliable elements sooner rather
Plan to run tests for the unreliable elements sooner rather
than later
than later

Allow time for re
Allow time for re
-
-
testing
testing

Time must be included for identifying, investigating & logging
Time must be included for identifying, investigating & logging
faults
faults

Faults that have been
Faults that have been
fixed
fixed
must be re
must be re
-
-
tested
tested

Whenever a change is made to the SUT or the environment a
Whenever a change is made to the SUT or the environment a
regression test should be run
regression test should be run

How do we build these factors into the estimation?
How do we build these factors into the estimation?
Test Monitoring
Test Monitoring

Test Preparation
Test Preparation

Estimate number of tests needed
Estimate number of tests needed

Estimate time to prepare
Estimate time to prepare

Refine these figures
Refine these figures

Track percentage of tests prepared
Track percentage of tests prepared

Test Execution
Test Execution

Assess the coverage achieved to date by testing
Assess the coverage achieved to date by testing
Coverage Coverage - - amount of tests completed, amount of code amount of tests completed, amount of code
exercised by tests, amount of features tested so far exercised by tests, amount of features tested so far

Estimate time required to run the test suite
Estimate time required to run the test suite

Whilst testing you can monitor
Whilst testing you can monitor

The number of tests run
The number of tests run

The number of tests passed & failed
The number of tests passed & failed

The number of defects raised
The number of defects raised
These can be categorised by Severity, Priority & Probability These can be categorised by Severity, Priority & Probability

The number of re
The number of re
-
-
tests
tests
-
-
that pass & that fail
that pass & that fail

The status of the project should be regularly reported
The status of the project should be regularly reported

Any deviations from the schedule raised ASAP
Any deviations from the schedule raised ASAP

Any critical faults found should be raised immediately
Any critical faults found should be raised immediately
Test Control
Test Control

Controlling measures
Controlling measures

Assign extra resource
Assign extra resource

Re
Re
-
-
allocate resource
allocate resource

Adjust the test schedule
Adjust the test schedule

Arrange for extra test environments
Arrange for extra test environments

Refine the completion criteria
Refine the completion criteria

Summary
Summary

Multiple factors must be considered when estimating the
Multiple factors must be considered when estimating the
length of time we need to perform testing
length of time we need to perform testing

Once testing has started it is necessary to monitor the
Once testing has started it is necessary to monitor the
situation as it progresses
situation as it progresses

Careful control must be kept to ensure project success
Careful control must be kept to ensure project success
Incident Management
Incident Management
Incident Management

In this session we shall
In this session we shall

Understand what an
Understand what an
incident
incident
is
is

Understand the impact they have on the testing process
Understand the impact they have on the testing process

Understand how they are logged, and tracked
Understand how they are logged, and tracked

Understand why & how they should be analysed to help
Understand why & how they should be analysed to help
ensure they don
ensure they don
t happen again
t happen again
Incident Management

An
An
incident
incident
is when testing cannot progress as

is when testing cannot progress as
planned
planned

The application does not function as anticipated
The application does not function as anticipated

Components are missing etc
Components are missing etc
Incident Management

What is an incident?
What is an incident?

When actual results differ from expected results
When actual results differ from expected results

They may be raised against
They may be raised against
Documentation Documentation
The code The code
The System Under Test The System Under Test
The Test Environment The Test Environment
Or the tests themselves Or the tests themselves
Incident Management

The incident needs to be viewed in terms of its impact
The incident needs to be viewed in terms of its impact
on testing
on testing

As testing progresses through its lifecycle priorities can
As testing progresses through its lifecycle priorities can
change
change

The progress of the testing activity can be monitored by
The progress of the testing activity can be monitored by
analysing incidents
analysing incidents
Incident Management

Top Priority must be given to any issues that prevent
Top Priority must be given to any issues that prevent
testing from progressing
testing from progressing

At the second level we need to assign those that can be
At the second level we need to assign those that can be
worked around
worked around

In the lower categories we log what are minor issues &
In the lower categories we log what are minor issues &
questions
questions
Incident Management

As project deadlines get nearer what is considered
As project deadlines get nearer what is considered
high priority may well change
high priority may well change

The emphasis will move towards the product and what will
The emphasis will move towards the product and what will
compromise its success
compromise its success
Incident Management

When logging incidents testers should include
When logging incidents testers should include

Test ID, System ID, Testers ID
Test ID, System ID, Testers ID

Expected and Actual results
Expected and Actual results

Environment in use
Environment in use

Severity or priority of the incident
Severity or priority of the incident
Incident Management

Service levels should be set up for each category of
Service levels should be set up for each category of
incident
incident

Progress monitored accordingly
Progress monitored accordingly

Involve Development, Testers and Users
Involve Development, Testers and Users

Issue regular status reports
Issue regular status reports
Incident Management

STATUS should also be tracked
STATUS should also be tracked

This ensures that we are able to swiftly establish the situation
This ensures that we are able to swiftly establish the situation
with an incident
with an incident
Incident Management

A typical status flow is
A typical status flow is

When first raised an incident is OPEN
When first raised an incident is OPEN

It then passes to development to fix
It then passes to development to fix
-
-
WITH DEVELOPMENT
WITH DEVELOPMENT

Development may question the incident
Development may question the incident
-
-
PENDING
PENDING

Or fix the problem and return it for
Or fix the problem and return it for
-
-
RETEST
RETEST

Should the retest prove successful the defect is CLOSED
Should the retest prove successful the defect is CLOSED
Incident Management

Analysing incidents
Analysing incidents

Incidents may be analysed to help monitor the test process
Incidents may be analysed to help monitor the test process

And to aid in improvement of the test process
And to aid in improvement of the test process
Incident Management

Summary
Summary

An incident is
An incident is
any significant, unplanned event that occurs

any significant, unplanned event that occurs
during testing that requires subsequent investigation and / or
during testing that requires subsequent investigation and / or
correction
correction

Incidents should be recorded and tracked
Incidents should be recorded and tracked

Analysis of incidents will enable us to see where problems
Analysis of incidents will enable us to see where problems
arose and to aid in test process improvement
arose and to aid in test process improvement
Standards for Testing


Understand the various standards that may influence testing
Understand the various standards that may influence testing

There are three types of standards that affect testing
There are three types of standards that affect testing

Quality Assurance standards

Industry specific standards

Testing Standards
Testing Standards


Only specify that testing should be done
Only specify that testing should be done
e.g. e.g. ISO 9000 ISO 9000- -3:1991 3:1991
Quality management and quality assurance standards Quality management and quality assurance standards- -Part 3: Part 3:
Guidelines for the application of ISO 9001 to the development, Guidelines for the application of ISO 9001 to the development,
supply and maintenance of software supply and maintenance of software


Specify what level of testing to perform
Specify what level of testing to perform
e.g. railway, medical, insurance e.g. railway, medical, insurance
ASTM F153 ASTM F153- -95 Standard Test Method for Determining the Yield of 95 Standard Test Method for Determining the Yield of
Wide Inked Computer Ribbons Wide Inked Computer Ribbons

Testing standards
Testing standards

Specify how to perform testing
Specify how to perform testing
e.g. e.g. BS7925 BS7925- -1 1 Software Testing vocabulary Software Testing vocabulary
BS7925 BS7925- -2 2 Software component testing Software component testing
IEEE 829 IEEE 829 Standard for Test Documentation Standard for Test Documentation

Summary
Summary

There are a range of standards that can influence testing
There are a range of standards that can influence testing

These come from different sources and affect different areas
These come from different sources and affect different areas
of testing
of testing

Additionally, various industries have legal requirements that
Additionally, various industries have legal requirements that
will influence testing
will influence testing
Types of CAST Tool
Types of CAST Tool
Types of CAST Tool


Understand that help is at hand in the form of CAST Tools
Understand that help is at hand in the form of CAST Tools
CAST CAST - - Computer Aided Software Testing Computer Aided Software Testing

Look at the range of tools available in the market today
Look at the range of tools available in the market today

Identify the areas in which tools can best help the testing
Identify the areas in which tools can best help the testing
process
process
Types of CAST Tool

Why do we need / use them?
Why do we need / use them?

To automate the most repetitious and time consuming tasks
To automate the most repetitious and time consuming tasks

Tools are able to do some things that people cannot easily do
Tools are able to do some things that people cannot easily do
Types of CAST Tool
Requirements testing Requirements testing
Static Analysis Static Analysis
Test Design Test Design
Data Preparation Data Preparation
Character Based Test Character Based Test
Running Running
GUI Test Running GUI Test Running
Test Harnesses Test Harnesses
Drivers & Simulators Drivers & Simulators
Performance Performance
Dynamic Analysis Dynamic Analysis
Debugging Debugging
Comparison Comparison
Test Management Test Management
Coverage Measurement Coverage Measurement
Types of CAST Tool

Requirements Testing tools
Requirements Testing tools

There are tools available to help verify and validate
There are tools available to help verify and validate
requirements
requirements

They use requirements models
They use requirements models

Can check consistency and animations
Can check consistency and animations
Types of CAST Tool

Static Analysis tools
Static Analysis tools

There are tools on the market that can help with analysing
There are tools on the market that can help with analysing
code
code

These tools analyse the quality and complexity of the code
These tools analyse the quality and complexity of the code
and paths through the code
and paths through the code

Many use McCabe, Halstead or McClure metrics
Many use McCabe, Halstead or McClure metrics
Types of CAST Tool

Test Design
Test Design
tools
tools

Help with designing of test cases
Help with designing of test cases

These generate test cases from a specification
These generate test cases from a specification
Normally held in a CASE tool repository Normally held in a CASE tool repository
Or from formally specified requirements held in the tool itself Or from formally specified requirements held in the tool itself

Some tools generate test cases from analysis of the code
Some tools generate test cases from analysis of the code
Types of CAST Tool

Data Preparation tools
Data Preparation tools

Help with data preparation
Help with data preparation

These enable data to be selected from existing databases or
These enable data to be selected from existing databases or
created, generated, manipulated and edited for use in tests
created, generated, manipulated and edited for use in tests

May create data in a simple database or a complex
May create data in a simple database or a complex
mainframe database
mainframe database
Types of CAST Tool

Character Based Testing tools
Character Based Testing tools

Used with character based applications
Used with character based applications
Terminals on Mainframes, Unix and mid Terminals on Mainframes, Unix and mid- -range systems (System range systems (System
36/38) 36/38)
Early PC (DOS) systems Early PC (DOS) systems

The tools simulate user entered keystrokes and capture
The tools simulate user entered keystrokes and capture
screen responses for comparison
screen responses for comparison
Normally used to automate regression and functional system Normally used to automate regression and functional system
testing testing

Many tools available to cover this
Many tools available to cover this
Types of CAST Tool

GUI Test Running tools
GUI Test Running tools

Testing for GUI based applications
Testing for GUI based applications

Testing software running in graphical operating systems i.e.
Testing software running in graphical operating systems i.e.
Windows
Windows
Most of our work in this area Most of our work in this area
Mostly for functional and regression tests Mostly for functional and regression tests

Tools must be able to identify and interact with the objects on
Tools must be able to identify and interact with the objects on
screen
screen

Tests scripted in a Test Script Language
Tests scripted in a Test Script Language
Types of CAST Tool

Web Testing tools
Web Testing tools

Testing web based applications
Testing web based applications

Increasingly tools are becoming available to help test E
Increasingly tools are becoming available to help test E
-
-
Commerce, web
Commerce, web
-
-
based applications
based applications
Existing tools that have Existing tools that have evolved evolved to include web testing to include web testing
Dedicated web testing tools Dedicated web testing tools
Types of CAST Tool Test

Harnesses & Drivers
Harnesses & Drivers

These are key to building up integration tests
These are key to building up integration tests

Used to execute the software under test
Used to execute the software under test
i.e. for systems that do not have a user interface i.e. for systems that do not have a user interface

These are small modules that drive data into the area under
These are small modules that drive data into the area under
test
test
Types of CAST Tool

Simulators
Simulators

Simulate events or systems
Simulate events or systems

There will be times when it is necessary to test a system that
There will be times when it is necessary to test a system that
interacts with an event or system that does not physically
interacts with an event or system that does not physically
exist
exist
Types of CAST Tool

Performance Testing tools
Performance Testing tools

Measure what a system operates like in normal
Measure what a system operates like in normal
circumstances
circumstances
Drive the SUT & monitor the response time for specified Drive the SUT & monitor the response time for specified
transactions transactions

Load generation
Load generation
How the system performs when accessed by many users How the system performs when accessed by many users
The tool simulates a number of users accessing the SUT The tool simulates a number of users accessing the SUT

The tools can simulate multiple sessions from one device
The tools can simulate multiple sessions from one device
LoadRunner, QALoad etc LoadRunner, QALoad etc
Types of CAST Tool

Dynamic Analysis tools
Dynamic Analysis tools

Provide run time information on the state of the executing
Provide run time information on the state of the executing
software
software

Typically used to monitor system resources
Typically used to monitor system resources

Find errors that can be difficult to find
Find errors that can be difficult to find
statically
statically

i.e. use and de
i.e. use and de
-
-
allocation of system memory and flag memory
allocation of system memory and flag memory
leaks
leaks
Types of CAST Tool

Debugging tools
Debugging tools

Used mainly by programmers to reproduce bugs and
Used mainly by programmers to reproduce bugs and
investigate the state of programs
investigate the state of programs

Debuggers enable programmers to execute programs line by
Debuggers enable programmers to execute programs line by
line, to halt the program at any statement and to set and
line, to halt the program at any statement and to set and
examine program variables
examine program variables
Types of CAST Tool

Comparison tools
Comparison tools

Comparing two elements is a key activity in testing
Comparing two elements is a key activity in testing

Eyeballing
Eyeballing
information is labour intensive and prone to error

information is labour intensive and prone to error

Tools are available on many platforms to help here
Tools are available on many platforms to help here

Comparison tools detect differences between files
Comparison tools detect differences between files
Typically actual and expected results files Typically actual and expected results files
Types of CAST Tool

Test Management tools
Test Management tools

Enable testers to manage their tests
Enable testers to manage their tests

Tests are created in the tool, linked together to form workflow,
Tests are created in the tool, linked together to form workflow,
tests executed and results collated
tests executed and results collated

Some of the better known tools
Some of the better known tools
i.e. TestDirector, SQA Manager, QADirector i.e. TestDirector, SQA Manager, QADirector
Types of CAST Tool

Coverage Measurement tools
Coverage Measurement tools

Report amount of code exercised by test executions
Report amount of code exercised by test executions

The code is instrumented before compilation
The code is instrumented before compilation

As the program is executed the instrument code records the
As the program is executed the instrument code records the
coverage data
coverage data

Reports generated can show the amount of code exercised
Reports generated can show the amount of code exercised
during tests
during tests
The most common coverage measures are statement and The most common coverage measures are statement and
branch coverage branch coverage
Types of CAST Tool

Summary
Summary

CAST Tools can help automate areas of the testing process
CAST Tools can help automate areas of the testing process

There is a wide range of functions that can be automated
There is a wide range of functions that can be automated
Tool Selection & Implementation

In the session we will
In the session we will

Examine the processes involved in identifying where tools
Examine the processes involved in identifying where tools
can help the testing process
can help the testing process

Learn how to approach the selection and implementation of
Learn how to approach the selection and implementation of
those tools
those tools

As indicated tools can help many aspects of testing
As indicated tools can help many aspects of testing

How do you select where you want to start introducing CAST
How do you select where you want to start introducing CAST
support?
support?

The selection and implementation of a tool is a major task
The selection and implementation of a tool is a major task
and should be regarded as a project in its own right
and should be regarded as a project in its own right

If testing is currently badly organised tools may not be
If testing is currently badly organised tools may not be
the immediate answer
the immediate answer

Automating chaos simply speeds the delivery of chaos
Automating chaos simply speeds the delivery of chaos

The benefits of tools usually depend on a systematic and
The benefits of tools usually depend on a systematic and
disciplined test process being in place
disciplined test process being in place

Address the test processes and introduce disciplines first
Address the test processes and introduce disciplines first

Examine the test process in detail
Examine the test process in detail

Identify the weaknesses, and the cause of those weaknesses
Identify the weaknesses, and the cause of those weaknesses

Find tools that fit in with your processes and address the
Find tools that fit in with your processes and address the
weak areas
weak areas
This may be more important than selecting the tool with the This may be more important than selecting the tool with the
most number of features most number of features

Prioritise the areas of greatest importance
Prioritise the areas of greatest importance

Pick a number of tools to trial in a certain area
Pick a number of tools to trial in a certain area

Define clear requirements & success criteria
Define clear requirements & success criteria

Understand each tools
Understand each tools
capabilities & weaknesses

capabilities & weaknesses

Run trials or small pilot exercises to evaluate the tools
Run trials or small pilot exercises to evaluate the tools

Review and select tool
Review and select tool

Examine the Test Environment(s)
Examine the Test Environment(s)

One of the most crucial areas for tool implementation
One of the most crucial areas for tool implementation

Shared environments will not accept automation unless the
Shared environments will not accept automation unless the
SUT allows the data to be entered fresh every time
SUT allows the data to be entered fresh every time

Automation may also fill an environment up and make it run
Automation may also fill an environment up and make it run
out of space as it allows a large amount of data to be entered
out of space as it allows a large amount of data to be entered
in a short space of time
in a short space of time

Test facilities may be (will be!!!) less than Production facilit
Test facilities may be (will be!!!) less than Production facilit
ies
ies
and use of the tools with the SUT can cause systems to run
and use of the tools with the SUT can cause systems to run
out of memory
out of memory

Funding must be secured
Funding must be secured

Software testing tools are not low cost items and there is
Software testing tools are not low cost items and there is
usually a large amount of man hours required as well
usually a large amount of man hours required as well

It is necessary to budget for both the tool and the resources
It is necessary to budget for both the tool and the resources
required to implement
required to implement

Evaluate the results of the trial
Evaluate the results of the trial

Select the best candidate
Select the best candidate
-
-
pilot project
pilot project

Plan the implementation & use of that tool
Plan the implementation & use of that tool

Implement & use
Implement & use

Broadcast the benefits
Broadcast the benefits

Plan the roll
Plan the roll
-
-
out of the tool
out of the tool

Gain commitment from the tools users
Gain commitment from the tools users

Ensure new projects are aware of the cost of introducing a
Ensure new projects are aware of the cost of introducing a
tool
tool

Summary
Summary

Tools can improve the organisation and efficiency of a test
Tools can improve the organisation and efficiency of a test
process
process

Before tools can be implemented a sound testing process
Before tools can be implemented a sound testing process
must be in place
must be in place

Tool selection and implementation must be carefully
Tool selection and implementation must be carefully
considered and managed
considered and managed

Software Testing Presentation

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Software Testing Presentation

Uploaded by

Copyright:

Available Formats

SIM Group Ltd.

, SQS Group AG, 2002

A human action that produces an incorrect result

A manifestation of an ERROR in software

SIM Group Ltd., SQS Group AG, 2002

A deviation of the software from its expected delivery or

SIM Group Ltd., SQS Group AG, 2002

The departure of a quality characteristic from its specified val

SIM Group Ltd., SQS Group AG, 2002

The probability that software will not cause the failure of a

SIM Group Ltd., SQS Group AG, 2002

The totality of the characteristics of an entity that bear on it

SIM Group Ltd., SQS Group AG, 2002

All those planned actions used to fulfil the requirements for

SIM Group Ltd., SQS Group AG, 2002

t test everything, what can we do?

test is one that

relationship with developers

relationship with management

SIM Group Ltd., SQS Group AG, 2002

SIM Group Ltd., SQS Group AG, 2002

t then how do you know you have achieved

A demonstration of the acceptance criteria

specification as defined in the criteria

interfaces with other systems

Testing Of Those requirements that do not relate to the

Testing whether the system meets specified security

Testing whether the system meets its specified storage

Testing concerned with the installation procedures for the

Testing concerned with the accuracy of the documentation

SIM Group Ltd., SQS Group AG, 2002

Testing aimed at verifying the system

s ability to recover from

Testing geared to assessing the applications ability to deal wit

SIM Group Ltd., SQS Group AG, 2002

Assesses individual components by exercising them to and

SIM Group Ltd., SQS Group AG, 2002

Tests the efficiency of individual components of an application

SIM Group Ltd., SQS Group AG, 2002

Testing where the system is subjected to large volumes of

SIM Group Ltd., SQS Group AG, 2002

s functions have been tested doesn

A requirement that specifies a function that a system or system

Integration Testing in the Small

Integration Testing in the Small

SIM Group Ltd., SQS Group AG, 2002

SIM Group Ltd., SQS Group AG, 2002

External systems External systems

The testing of an individual software component

A minimal software item for which a separate specification is

SIM Group Ltd., SQS Group AG, 2002

Retesting of a previously tested program following

SIM Group Ltd., SQS Group AG, 2002

Test Case selection that is based on an analysis of the

Test Case selection based on an analysis of the internal

SIM Group Ltd., SQS Group AG, 2002

Glass Box testing

Test Case selection that is based on an analysis of the

SIM Group Ltd., SQS Group AG, 2002

Test Case selection based on an analysis of the internal

A test case design technique for a component in which test

Testing of an object without execution on a computer

SIM Group Ltd., SQS Group AG, 2002

Testing of an object without execution on a computer