Basic Testing Concepts and Terminology - 2006 PDF

T-76.
5613 Software Testing and Quality Assurance
Lecture 2, 13.9.2006
Basic Testing Concepts and

Terminology
Juha Itkonen
SoberIT
HELSINKI UNIVERSITY OF TECHNOLOGY

Contents
Realities and principles of testing

Testing terminology and basic concepts
V-model, test levels and types
The concept of software quality
Juha Itkonen, 2005 2

HELSINKI UNIVERSITY OF TECHNOLOGY SoberIT/HUT
Realities and principles of testing

The problem of complete testing
Why not just test everything?

Complete testing would require testing
¾ every possible valid and invalid input of the system
¾ and every possible valid and invalid output
¾ and every possible combination of inputs
¾ in every possible sequence of inputs and outputs
¾ and user interface errors
¾ and configuration and compatibility issues
¾ and environment related failures
¾ …
There is definitely too much cases to test completely
any realistic software module, not to mention a
whole system
Testing is always prioritizing
Time is always limited Possible ranking criteria

1. test where a failure would
Use company’s be most severe
significant risks to 2. test where failures would
be most visible
focus testing effort
3. test where failures are
¾ what to test first most likely
¾ what to test most 4. ask the customer to
prioritize the
¾ how thoroughly to test
requirements
each feature 5. what is most critical to
¾ what not to test the customer’s business
6. areas changed most often
Most important tests
7. areas with most problems
first in the past
8. most complex areas, or
technically critical
Realities of testing (1/2)
It is impossible to test a program completely

Testing can show the presence of defects but cannot
show the absence of defects
¾ You can report found defects and what you have tested
and how
All bugs cannot be found
All found bugs will not be fixed
¾ It’s really not a bug
¾ It’s too risky to fix, there is not enough time
¾ There is more important tasks to do
Testing does not create quality software or remove
defects
The critique is focused on the product, not the
developer
Realities of testing (2/2)
Product specifications are never final

¾ You cannot wait until specifications are final
¾ There will always be changes to specifications
¾ The specifications are never complete
The more bugs you find, the more bugs there

are
¾ Programmers have bad days
¾ Programmers often make same mistake
¾ Some bugs are really just the tip of the iceberg
The Pesticide Paradox

¾ Programs become immune to unchanging tests
Software testing should be based on risks
Testing principles
When the test objective is to detect defects, the a good test case
is one that has a high probability of revealing a yet undetected
defect(s).
Test results should be inspected meticulously.
A test case must contain the expected results.
Test cases should be developed for both valid and invalid input
conditions.
The probability of the existence of additional defects in software
component is proportional to the number of defects already
detected in that component.
Testing should be carried out by a group that is independent of
the development team.
Tests must be repeatable and reusable.
Testing should be planned.
Testing activities should be integrated into the software life
cycle.
Testing is a creative and challenging task.
Burnstein 2003
Testing terminology and basic concepts

Validation and verification (V&V)
Validation – are we building the right product?

¾ Implementation meets customer requirements, needs
and expectations
Verification – are we building the product right?

¾ Program conforms to its specification
Note! In the chapter 1 of the course book (Burnstein, 2003, p.

6) there is different definition for V&V. The above definition,
however, is more common and makes more clear distinction
between the terms.

Black-box and white-Box testing
Black-box (functional, behavioral, data-driven)

¾ The software under test considered as a black-box and
no knowledge of the internal structure or how the
software actually works is utilized in testing
¾ Testing based on inputs and respective outputs
¾ The size of the black box can vary from one class or
component to a whole system
White-box (structural, logic-driven)

¾ Testing based on knowing the inner structure of the
system and the program logic

Functional testing
Testing conducted to evaluate the compliance of a

system with specified functional requirements.
IEEE Standard Glossary of Software Engineering Terminology
Compare to non-functional testing

¾ E.g. performance, reliability, usability, security testing

Dynamic and static quality practices
Dynamic methods – testing – executes code

¾ Software testing in the traditional sense
¾ Dynamic analysis methods
Static methods do not execute code

¾ Reviews, inspections, static analysis
¾ Some authors name these static testing

Scripted and non-scripted testing
In scripted (test case based) testing test cases are

pre-documented in detailed, step-by-step
descriptions
¾ Different levels of scripting possible
¾ Scripts can be manual or automated
Non-scripted (exploratory) testing is usually manual
testing without detailed test case descriptions
¾ Can be disciplined, planned, and well documented
exploratory testing
¾ … or ad-hoc testing

What is a ”bug”?
Error - a human action that produces an incorrect

result
Fault - a manifestation of an error in software
¾ also known as a defect, bug, issue, problem, anomaly,
incident, variance, inconsistency, feature, …
¾ if executed, a fault may cause a failure
Failure - deviation of the software from its expected
delivery or service
Failure is an event; fault is a state of

the software, caused by an error.

Error –> Fault -> Failure
A programmer
makes an
error…
…that creates a fault
in the software…
…that can cause a

failure in
operation

What is a fault?
A software error is present when the program does
not do what its end user reasonably expects it to do
(Myers, 1979)
There can never be an absolute definition for bugs,
nor an absolute determination of their existence. The
extent to which a program has bugs is measured by
the extent to which it fails to be useful. This is a
fundamentally human measure. (Beizer, 1984)

Rules for identifying a defect
The software does not do something that the

specification says it should do
The software does something that the specification
says it should not do
The software does something that the specification
does not mention
The software does not do something that the
specification does not mention but it should
The software is difficult to understand, hard to use,
slow, or will be viewed by the end user as just plain
not right
Suorce: Software Testing, Ron Patton

Test Oracles – How do we know it’s broken?
An oracle is the principle or mechanism by which you
recognize a problem
Test oracle provides the expected result for a test,
for example
¾ Specification document
¾ Formula
¾ Computer program
¾ Person
In many cases it is very hard to find an oracle
¾ Even the customer and end user might not be able to
tell which is the correct behaviour
Oracle problem is one of the fundamental issues in test automation:
How do we teach an automated test to recognize a defect or
failure when it happens? – This is not a trivial problem.
Oracle – Is this correct?
Font size test Arial 22pt
¾ Font size test Arial 20pt
¾ Font size test Arial 18pt
— Font size test Arial 16pt
— Font size test Arial 14pt
Font size test Times 22pt

¾ Font size test Times 20pt
¾ Font size test Times 18pt
— Font size test Times 16pt
— Font size test Times 14pt

Severity and priority
Severity of a software fault refers to the severity of the

consequences of a failure caused by that fault
¾ A tester or customer is probably the best person to set the
severity
Priority is the fixing order of the found faults and a result
of separate prioritisation activity
¾ A tester is probably not the best person to set the priority
¾ Prioritisation is typically a managerial decision
¾ business priorities
¾ customer priorities
The priority of a fault
¾ other faults and features
depends on other issues
¾ quality policy
¾ release strategy
¾ Prioritisation is usually based on severity

Re-testing vs. regression testing
Re-tests Regression tests

New version of software Running a set of tests that
with ”fixed fault” have been run before
Re-run the same test that After software or
revealed the defect environment changes
¾ must be exactly ¾ For confidence that
repeatable everything still works
¾ same environment, ¾ To reveal any
versions (except for the unanticipated side effects
software which has been An asset (regression test
intentionally changed!) suite/pack)
¾ same inputs and ¾ Standard set of tests -
preconditions needs designing &
If test now passes, fault maintaining
has been fixed correctly – ¾ Well worth automating
or has it?
Regression testing
Regression tests look for

unexpected side-effects
Fault fix introduces or
(but may not find all of them)
uncovers new faults
Breadth tests x
x Depth tests x
x
x9
Test finds fault
Re-test to check fix
V-model, test levels and types

V-model of testing
Acceptance
Requirements
testing
Functional System
specification testing
Architecture Integration
design testing
st
Bu
Te
ild Module Unit
design testing
Coding

V-model is easy to understand
Simple generic model

Easy to understand and explain
Describes the test levels and phases
Shows how testing activities are related to other
development activities
Matches to familiar waterfall model

V-model says…
It is good to have requirements laid out first.

It is good to have plans and specifications done before
implementation.
It is good to complete one task before proceeding to the
next that depends on the output of the first task.
It is good to test each intermediate product.
It is good to use each development specification as a basis
for test process.
It is much easier to find faults in small units than in large
entities. Therefore, test small units first alone before
putting them together.
Testing of large units can be carried out more easily when
the smaller parts are already tested.

Acceptance
Requirements
testing
Functional System
specification testing
Architecture
design
Integration
testing Test Levels
st
Bu
Te
ild Module Unit
design testing
Coding
Test level – a test level is a group of test activities that

focus into certain level of the test target
¾ Test levels can be seen as levels of detail and abstraction
¾ How big part of the system are we testing?
¾ E.g. unit test, integration test, system test, acceptance test

Acceptance
Unit testing testing
System
testing
A unit = smallest testable software Integration
component testing
¾ Objects and methods Unit

testing
¾ Procedures / functions
¾ Reusable components
Focus on programming errors
¾ Verifying that the code unit works as the
specification says and the developer designed it
¾ Most thorough look at the details
Testing units in isolation
Usually done by a programmer (in practice)
¾ A pair tester can help
Also known as component, module, or program testing

Acceptance
Integration Testing testing
System
testing
More than one (tested) unit work together Integration
Focus on interfaces testing
¾ Communication between units Unit

testing
Helps assembling incrementally a whole system
Integration strategy: big-bang vs. incremental (top-down,
bottom-up, functional, …)
Done by developers/designers or independent testers
¾ Preferably developers and testers in collaboration
Modern development methods emphasize continuous
integration, no separate integration phase
Integration testing can be done on different levels
¾ Class level
¾ Component level
¾ Subsystem level
¾ Systems integration level
Acceptance
System testing testing
System
Testing the system as a whole

testing

Integration
Focus testing
Unit
¾ Verifying that specifications are met testing
¾ Validating that the system can be used for

the intended purpose
¾ By the development organization
Functional as well as non-functional testing
¾ Non-functional requirements often poorly
specified
Independent testing group
¾ Collaborating developers and testers

Acceptance
Acceptance testing testing
System
Final stage of validation testing
¾ Customer (user) should perform or be closely Integration

testing
involved Unit
¾ Customer should be able to perform any test they testing
wish, usually based on their business processes

¾ Final customer sign-off
Approach
¾ Should be performed in real operating environment
¾ Environment related problems
¾ Can the system be used for the intended purpose in
the real operating environment
¾ Problems that the real users have with the system
Project business Product business
Contract acceptance Final checks on releases, (alpha, beta)
Customer viewpoint User viewpoint throughout the
Validates that the right system was build development; validation
Levels, types, and phases – not a same thing
Test level – a test level is a group of test activities

that focus into certain level of the test target.
¾ E.g. unit test, integration test, system test, acceptance
test
¾ Test levels can be seen as levels of detail and
abstraction
Test type – A test type is a group of test activities
aimed at evaluating a system for a number of
associated quality characteristics.
¾ E.g. functional test, performance test, stress test,
usability test
Test phase – Test phases are used to describe
temporal parts in testing process that follow
sequentially each other, with or without overlapping
Common test types
Functional testing Non-Functional testing
Function testing Performance testing
Installation testing Load testing
Smoke testing Stress testing
Configuration testing Volume testing
Compatibility testing Reliability testing
Exception testing Security testing
Interface testing Usability testing
Conversion testing Recoverability testing
… Maintainablity testing
Localization testing
Documentation testing
Data quality testing
See course book chapter Platform testing
6.13 System Test: The Different Types Concurrency testing
…

Software Quality

Which one describes high a quality software system?
Defect free Feature rich

Popular in industry Award winning product
Fast to learn Fast to use
Nice graphics Low system requirements
Easy to install and upgrade Reliable customer support
Robust Cool
Cheap Expensive
Simple Big and complicated
Small and compact Large and scalable
Customizable Standardized
Open-source Reliable vendor
Fit-for-purpose Multipurpose
Maintainable Fast time-to-market
Appeals majority Perfect for target segment
… …

Definition of software quality
Quality: (1) The degree to which a system, component or

process meets specified requirements. (2) The degree to
which a system, component, or process meets customer or
user needs or expectations.
(IEEE Standard Glossary of Software Engineering Terminology [IEEE610.12])
Quality is value to some person(s)

(Gerald M. Weinberg, 1992, “Quality Software Management”)

Users’ viewpoint differs from yours
All stakeholders have different viewpoints to quality

¾ Customer
¾ End user
¾ Programmer
¾ Project manager
¾ Tester
Whose opinion counts?
¾ A bug free product can be unacceptable to the users
¾ The goal of a software system is to solve customers
problem
¾ A minor annoyance for one user can make the product
useless for another

What does product quality mean?
Product
¾ A general purpose text editor application
What does quality mean for
¾ Developer of the application (technical view)
¾ Salesman (sales view)
¾ Product manager (product view)
¾ Tester (tester’s view)
¾ Coder -user
¾ Note taker -user
¾ Report writer –user
¾ Occasional -user

The McCall quality model
Maintainability Portability
Flexibility Product Revision Product Transition Reusability
Testability Interoperability
Product Operations
Correctness
Reliability
Efficiency
Integrity
Usability

ISO 9126 Quality characteristics
The capability to provide functions
that meet stated or implied needs.
The capability to
The capability to be maintain a
transferred from one specified level of
Functionality performance,
environment to
another. under stated
conditions for a
Portability Reliability specified period of
time.
ISO/IEC
9126
Maintainability
Usability
The capability to
The capability to
be understood,
be modified and
Efficiency learned, used and
make changes.
attractive to user.
The capability to provide appropriate

performance, relative to amount of
resources used, under stated conditions.
Good Enough Quality
Defined by James Bach

¾ IEEE Computer, 1997, vol. 30, no. 8, pp. 96-98.
To claim that any given thing is good enough is to
agree with all of the following propositions:
¾ It has sufficient benefits
¾ It has no critical problems
¾ The benefits sufficiently outweigh the problems
¾ In the present situation, and all things considered,
further improvement would be more harmful than
helpful

Software quality
In general, it is wise to strive for good quality and try to

avoid making too many mistakes during the development
project
¾ Bad quality leads to rework, bigger costs, delays, and
problems in operation
Quality
¾ is meeting the specifications
¾ is meeting the expectations of the customer or user
¾ is value to someone
¾ differs depending on who you ask
¾ is difficult to define and hard to measure
General standards for software quality
¾ E.g. ISO 9126, McCall quality model, …
¾ Gives a nice taxonomy, terminology, definitions and
conceptual framework
¾ Cannot tell what are the most important quality attributes
and how to achieve them in a particular context and situation


Basic Testing Concepts and Terminology - 2006 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Basic Testing Concepts and Terminology - 2006 PDF

Uploaded by

Copyright:

Available Formats

T-76.

5613 Software Testing and Quality Assurance

Basic Testing Concepts and

HELSINKI UNIVERSITY OF TECHNOLOGY

 Realities and principles of testing

Juha Itkonen, 2005 2

Juha Itkonen, 2005 3

 Why not just test everything?

 Time is always limited  Possible ranking criteria

 It is impossible to test a program completely

 Product specifications are never final

 The more bugs you find, the more bugs there

 The Pesticide Paradox

Juha Itkonen, 2005 9

 Validation – are we building the right product?

 Verification – are we building the product right?

 Note! In the chapter 1 of the course book (Burnstein, 2003, p.

Juha Itkonen, 2005 10

 Black-box (functional, behavioral, data-driven)

 White-box (structural, logic-driven)

Juha Itkonen, 2005 11

 Testing conducted to evaluate the compliance of a

 Compare to non-functional testing

Juha Itkonen, 2005 12

 Dynamic methods – testing – executes code

 Static methods do not execute code

Juha Itkonen, 2005 13

 In scripted (test case based) testing test cases are

Juha Itkonen, 2005 14

 Error - a human action that produces an incorrect

Failure is an event; fault is a state of

Juha Itkonen, 2005 15

…that can cause a

Juha Itkonen, 2005 16

Juha Itkonen, 2005 17

 The software does not do something that the

Suorce: Software Testing, Ron Patton

Juha Itkonen, 2005 18

— Font size test Arial 14pt

 Font size test Times 22pt

Juha Itkonen, 2005 20

 Severity of a software fault refers to the severity of the

Juha Itkonen, 2005 21

Re-tests Regression tests

Regression tests look for

Juha Itkonen, 2005 24

Juha Itkonen, 2005 25

 Simple generic model

Juha Itkonen, 2005 26

 It is good to have requirements laid out first.

Juha Itkonen, 2005 27

 Test level – a test level is a group of test activities that

Juha Itkonen, 2005 28

¾ Objects and methods Unit

Juha Itkonen, 2005 29

¾ Communication between units Unit

Testing the system as a whole

¾ Validating that the system can be used for

Juha Itkonen, 2005 31

¾ Customer (user) should perform or be closely Integration

wish, usually based on their business processes

 Test level – a test level is a group of test activities

Juha Itkonen, 2005 34

Realities and principles of testing

Why not just test everything?

Time is always limited Possible ranking criteria

It is impossible to test a program completely

Product specifications are never final

The more bugs you find, the more bugs there

The Pesticide Paradox

Validation – are we building the right product?

Verification – are we building the product right?

Note! In the chapter 1 of the course book (Burnstein, 2003, p.

Black-box (functional, behavioral, data-driven)

White-box (structural, logic-driven)

Testing conducted to evaluate the compliance of a

Compare to non-functional testing

Dynamic methods – testing – executes code

Static methods do not execute code

In scripted (test case based) testing test cases are

Error - a human action that produces an incorrect

The software does not do something that the

Font size test Times 22pt

Severity of a software fault refers to the severity of the

Simple generic model

It is good to have requirements laid out first.

Test level – a test level is a group of test activities that

Test level – a test level is a group of test activities

Defect free Feature rich

Quality: (1) The degree to which a system, component or

Quality is value to some person(s)

All stakeholders have different viewpoints to quality

Defined by James Bach

In general, it is wise to strive for good quality and try to