CTFL-Chapter 2-Fundamentals of Testing v2.2.0

Software Testing Foundations
Chapter
Chapter 2
2 Page 1
Certified Tester
Fundamentals of Testing
Software Testing Foundations Certified Tester
MSTB-GTB
2017
Version 2.2.0
© Copyright MSTB-GTB
V 2.2.0 / 2017
Disclaimer
• These slides are published under CC BY-NC-SA 4.0. This means besides
other things that they can be freely changed and used for non-commercial
purposes. By that we like to reach maximum flexibility for the CTFL
teachers at universities, colleges, and alike. However, we also have to
take into account the interests of the commercial trainings providers,
whose explicit request is not to have these slides freely available in the
Internet.
• Hence, we ask you to make use of the slides only within your lecture
series and exercises on CTFL related matter. Please also advise the
students that a free distribution of the slides would risk the GTB and/or
MSTB cooperation with universities, colleges, etc. – at least wrt. the
exchange of the slides.
• CC BY-NC-SA 4.0: https://creativecommons.org/licenses/by-nc-sa/4.0/
Chapter 2 Page 2 Software Testing Foundations Certified Tester
V 2.2.0 / 2017
3 After this lecture, you should….
• Be able to use examples to describe the way in which a software defect can cause harm to
people, to the environment, or to a company;
• Know what is meant by the terms deficiency, defect, failure, defect condition / defect and error
and explain this with examples and compare;
• Be able to distinguish between the cause of a defect and its effects;
• Derive by examples why testing is necessary;
• Be able to describe why testing is part of quality assurance and give examples of how testing
contributes to a higher quality;
• Know the quality characteristic according to ISO 9126;
• Recall the general objectives and principles of testing;
• Differentiate between testing and debugging;
• Know how the fundamental test process looks like;
• Describe how expected values are calculated during testing;
• Know and characterize the psychological problems during the testing;
• Differentiate between the different mindset of a tester and a developer;
• Know the code of ethics of a software tester.
V 2.2.0 / 2017
Learning Objectives for Fundamentals of Testing
3
3 (according to the Certified Tester Foundation Level Syllabus, Version 2011)
• 1.1 Why is Testing Necessary? (K2)

– LO-1.1.1 Describe, with examples, the way in which a defect in software can cause
harm to a person, to the environment or to a company (K2)
– LO-1.1.2 Distinguish between the root cause of a defect and its effects (K2)
– LO-1.1.3 Give reasons why testing is necessary by giving examples (K2)
– LO-1.1.4 Describe why testing is part of quality assurance and give examples of how
testing contributes to higher quality (K2)
– LO-1.1.5 Explain and compare the terms error, defect, failure and the
corresponding terms mistake and bug, using examples (K2)
• 1.2 What is Testing? (K2)

– LO-1.2.1 Recall the common objectives of testing (K1)
– LO-1.2.2 Provide examples for the objectives of testing in different phases of the
software life cycle (K2)
– LO-1.2.3 Differentiate testing from debugging (K2)
V 2.2.0 / 2017
Learning Objectives for Fundamentals of Testing
3
3 (according to the Certified Tester Foundation Level Syllabus, Version 2011)
• 1.3 Seven Testing Principles (K2)

– LO-1.3.1 Explain the seven principles in testing (K2)
• 1.4 Fundamental Test Process (K1)

– LO-1.4.1 Recall the five fundamental test activities and respective tasks from planning
to closure (K1)
• 1.5 The Psychology of Testing (K2)

– LO-1.5.1 Recall the psychological factors that influence the success of testing (K1)
– LO-1.5.2 Contrast the mindset of a tester and of a developer (K2)
V 2.2.0 / 2017
2
Chapter
Fundamentals
of Testing
} Terms and Motivation
Principles of Testingda
Fundamental Test Process Test Process
Test Cases, Expected Results and Test Oracles
Psychology of Testing
Ethics of Testing
V 2.2.0 / 2017
In General: What is a Defect or a Deficiency?
A situation can only be classified as defective if it is pre-defined, how

the expected, correct and therefore the non-defective situation should
look like.
Defect
Is the inability to fulfill a specific requirement. It is a discrepancy
between the actual behaviour (during the execution of the tests or
identified operation) and the expected behaviour (in the test
specification or the stated requirements).
Deficiency
Non-fulfilment of a requirement related to an intended or specified
use. A deficiency is for example the impairment of usability, while
meeting performance or failure to perform any reasonable
expectation.
Source: DIN EN ISO 9000:2005
Quality Management Systems –
Fundamentals and Terms (German)
Beuth Verlag, Berlin, 2005
V 2.2.0 / 2017
Connection:
Error - Defect - Failure
People make mistakes; they commit errors.
Example:
A programmer commits
an error by re-using a Error
a piece of software which is
not intended in the context
of the current project
(the Ariane 5 satellite
launching rocket)
ISTQB Glossary:
Error: Human action that produces an incorrect result [IEEE 610].
V 2.2.0 / 2017
Error – Other Definitions
Other defintions:
1. Human action (of the developer) that
produces an error condition in the software
2. Human action (of the user) that produces
an unwanted result (failure) as a
consequence (misoperation)
3. Unknowingly, inadvertently or intentionally
performed act, or omission which leads under given
circumstances (task, environment) to an impairment of the
required function of a product
V 2.2.0 / 2017
Connection:
An error made by a person
Defect
(e.g. a developer) can result
in a defect in the sofware.
Example: After the

Error
programmer has
re-used the piece of
code in the wrong
context, the software
contains a defect. in a program
ISTQB Glossary:
Defect: A flaw in a component or system that can cause the component
or system to fail to perform its required function, e.g., an incorrect statement or
data definition A(the
defect, if encountered
full definition during
follows execution,
on page 12) may cause a
failure of the component or system.
V 2.2.0 / 2017
Connection: Failure
A failure is the effect of Defect

a defect when executing
a program (test object)
which appears to the
“outside”.
Error
Example:
The Ariane 5 Boom
rocket crashes
on its first in a program
mission.
Cost ~ $7 billion
which appears
ISTQB Glossary:
Deviation of the component or
system from its expected delivery,
service or result [Fenton].
V 2.2.0 / 2017
Defect – Failure
Additional comments
• Defect (full definition from ISTQB glossary): A flaw in a
component or system that can cause the component or system to
fail to perform its required function, e.g., an incorrect statement or
data definition. A defect, if encountered during execution, may
cause a failure of the component or system.
• Defect masking: An occurrence in which one defect prevents the
detection of another [IEEE 610]
• A failure
– can also be called a “malfunction” or an “external defect”
– can also (but rarely) be caused by cosmic radiation, electro-
magnetic fields or hardware failure. Such failures can cause
slow execution, incorrect output or the termination of execution.
• Debugging is the development activity that finds, analyzes and
removes the cause of the failure.
V 2.2.0 / 2017
Testing – Definition and Goals
The process consisting of all life cycle activities, both static and
dynamic, concerned with planning, preparation and evaluation of
software products and related work products to
– determine that they satisfy specified requirements
– demonstrate that they are fit for purpose
– detect defects
Source:
ISTQB Glossary
V 2.2.0 / 2017
Validation and Verification - Definitions
Validation
Confirmation by examination and through provision of

objective evidence that the requirements for a specific
intended use or application have been fulfilled
[ISO 9000, ISTQB Glossary]
Did we implement the right system?
Verification
Confirmation by examination and through the provision

of objective evidence that specified requirements have
been fulfilled [ISO 9000, ISTQB Glossary]
Did we implement the system right?
V 2.2.0 / 2017
What is Software Quality?
• A feature or characteristic that affects an item's quality [IEEE 610].

• A set of attributes of a software product by which its quality is
described and evaluated. A software quality characteristic may be
refined into multiple levels of sub-characteristics [ISO 9126]*.
• Quality attributes relate to requirements
– Functional requirements (capabilities, interfaces,
professionalism, ...)
– Non-functional requirements (quality and implementation
requirements, project-specific requirements, ...)
*Remark:
The description of product quality attributes provided in ISO9126 is
used as a guide to describing software quality attributes. Other
standards, such as the ISO/IEC 25000 series, may also be of use.
V 2.2.0 / 2017
Software Quality according to ISO/IEC 9126 (1/2)
Software Quality
External and
Quality in Use
Internal Quality
Effectiveness Productivity Security Satisfaction
Fulfillment of tasks Fulfillment of tasks Fulfillment of tasks

within the within the within the limits of Subjective user Source:
accuracy and expenditure limits risk (life and satisfaction within ISO/IEC 9126: Rate of software products,
completeness for users (time, health, the context of use quality characteristics and guidance on usage
limits costs, ...) business...)
Note: Since 2011, ISO/IEC 9126 is replaced
by ISO/IEC 25000 Software engineering –
Software product Quality Requirements and
Evaluation (SQuaRE)
V 2.2.0 / 2017
Software Quality according to ISO/IEC 9126 (2/2)
Software Quality
External and
Quality in Use
Internal Quality
Functionality Reliability Usability Efficency Maintainability Portability
Suitability Understandability Analyzability Adaptivity

Maturity Time behavior /
Accuracy Learnability Performance Changeability Installability
Fault tolerance
Interoperability Operability Resource Stability Co-Existence
Recoverability utilization
(Data) Security Attractiveness Testability Replaceability
Compliance Compliance
Compliance Compliance Compliance Compliance
Source:
ISO/IEC 9126: Evaluation of software products,
quality characteristics and guidance on usage
V 2.2.0 / 2017
External Quality Attribute: Functionality (1/2)
Existence of functions with specified attributes. These functions meet the

specified or implied requirements.
Suitability
The capability of the software product to provide an appropriate set of
functions for specified tasks and user objectives [ISO 9126].
Accuracy
The capability of the software product to provide the right or agreed results or
effects with the needed degree of precision [ISO 9126].
V 2.2.0 / 2017
External Quality Attribute: Functionality (2/2)
Interoperability
The capability of the software to interact with one or more specified
components or systems [ISO 9126].
(Data) Security
Attributes of software that bear on its ability to prevent unauthorised access,
whether accidental or deliberate, to programs and data [ISO 9126].
Compliance
Attributes of the software which demonstrate that the software fulfills
application-specific standards or agreements or statutory directives or similar
regulations [ISO 9126].
Note: this also applies to all other quality attributes (e.g. reliability)
V 2.2.0 / 2017
External Quality Attribute: Reliability
Attributes that relate to the ability of the software to maintain a specified level of
performance under given conditions and for a specified time period.
Maturity
The capability of the software product to avoid failure as a result of defects in the
software [ISO 9126].
Fault tolerance
The capability of the software product to maintain a specified level of performance
in cases of software faults (defects) or of infringement of its specified interface
[ISO 9126].
Recoverability
The capability of the software product to re-establish a specified level of
performance and recover the data directly affected in case of failure [ISO 9126].
V 2.2.0 / 2017
External Quality Attribute Usability
Attributes that relate to the effort required to use, and on the individual
assessment of such use by a specific or implied group of users.
Understandability
The capability of the software product to enable the user to understand whether
the software is suitable, and how it can be used for particular tasks and conditions
of use [ISO 9126].
Learnability
The capability of the software product to enable the user to learn its application
[ISO 9126].
Operability
The capability of the software product to enable the user to operate and control
it [ISO 9126].
V 2.2.0 / 2017
External Quality Attribute: Efficiency
Attributes that relate to the relationship between the performance levels of the
software and the amount of equipment used under specified conditions.
Time behaviour
The degree to which a system or component accomplishes its designated
functions within given constraints regarding processing time and throughput
rate [IEEE 610].
Resource utilization
The capability of the software product to use appropriate amounts and types
of resources, for example the amounts of main and secondary memory used
by the program and the sizes of required temporary or overflow files, when
the software performs its function under stated conditions [ISO 9126].
V 2.2.0 / 2017
Internal Quality Attribute: Maintainability (1/2)
Attributes that relate to the effort that is necessary to carry out software
changes.
Analyzability
The capability of the software product to be diagnosed for deficiencies or
causes of failures in the software, or for the parts to be modified to be
identified [ISO 9126].
Changeability
The capability of the software product to enable specified modifications to be
implemented [ISO 9126].
V 2.2.0 / 2017
Internal Quality Attribute: Maintainability (2/2)
Stability
The capability of the software product to avoid unexpected effects from
modifications in the software [ISO 9126].
Testability
The capability of the software product to enable modified software to be
tested [ISO 9126].
V 2.2.0 / 2017
Internal Quality Attribute: Portability (1/2)
Attributes that relate to the ability of software to be transferred from one

environment to another.
Adaptability
The capability of the software product to be adapted for different specified
environments without applying actions or means other than those provided for
this purpose for the software considered [ISO 9126].
Installability
The capability of the software product to be installed in a specified
environment [ISO 9126].
V 2.2.0 / 2017
Internal Quality Attribute: Portability (2/2)
Co-Existence
• The capability of the software product to co-exist with other
independent software in a common environment sharing common
resources [ISO 9126].
Replaceability
• The capability of the software product to be used in place of
another specified software product for the same purpose in the
same environment [ISO 9126].
V 2.2.0 / 2017
Quality Requirements
• Quality requirements state which quality attributes the product

should have at which quality level.
– The sum of all quality attributes and the specific variants
• Not all quality attributes can be achieved equally well.

– E.g. efficiency may come at the expense of maintainability
• Set priorities
– In close consultation with clients and users.
– Quality requirements (with the exception of “functionality”) are
part of the non-functional requirements in the specifications.
V 2.2.0 / 2017
Testing and Quality
• Testing measures the quality e.g., based on the number of located

failures.
• Testing increases the quality indirectly, as defects are detected
which can be corrected prior to delivery.
• Testing increases the process quality indirectly, as defects are
documented which can then be analysed and help prevent similar
errors from taking place in the future.
• Testing increases the confidence in the quality of the system when
few or no failures and defects are found.
V 2.2.0 / 2017
Quality Assurance - QA
Quality Assurance
Analytic QA Constructive QA
Norms, standards,
project management,
Artefacts Processes software techniques,
Audits exchange of experiences,
training
Software, Documentation, ...
Software, etc.
etc.
Dynamic
Static Testing:
Testing:
Reviews,
White-Box,
Static Analysis
Black-Box
V 2.2.0 / 2017
 Software Quality according to ISO 25010
ISO 25010
Functional Performance
Reliability Usability Security Compatibility Portability Maintain-ability
suitability Efficiency
Learnability
Appropriateness
Functional Confidentiality Modularity
Maturity Tecognizability
completeness Time behavior
Integrity Installability Reusability
Availability Operability Co-Existence
Functional Resource
Non-repudiation Replaceability Analyzability
correctness Recoverability User error Interoperability utilization
protection Accountability Adaptability Modifiability
Functional Fault tolerance Capacity
appropriateness User interface Authenticity Testability
aesthetics
Accessibility
ISO 25010 Neu, vorher unter „Functionality“
V 2.2.0 / 2017
2
Chapter
Fundamentals
of Testing
Principles of Testing
Fundamental Test Process Test Process
Ethics of Testing
V 2.2.0 / 2017
Seven Principles of Testing (1/3)
In the previous 50 years, the following seven testing principles have

emerged and can serve as guidelines:
Principle 1: Testing shows presence of failures

Testing can show that failures are present, but cannot prove that
there are no defects.
“Program testing can be used to show the presence of bugs,
but never to show their absence!” Edger W. Dijkstra, 1970
Principle 2: Exhaustive testing is impossible

Complete / exhaustive testing is not feasible except for trivial
programs (Refer slides in Chapter 1 on Exhaustive Testing)
V 2.2.0 / 2017
Principle 3: Early testing

Testing is not a phase in the software development that happens at
the end; it shall be started as early as possible. Through early testing
(e.g. reviews) which take place parallel to the development activities,
defects are detected earlier and thus cost less to fix.
Principle 4: Defect clustering

Testing effort shall be focused proportionally to the expected and the
observed defect density of modules. A small number of modules often
contains most of the defects discovered during pre-release testing, or
is responsible for most of the operational failures.
V 2.2.0 / 2017
Principle 5: Repetitions have no effects - Pesticide Paradox

Just repeating tests brings no new insights. Test cases need to be
regularly reviewed, revised and modified.
Principle 6: Testing is context-dependent

Testing is depending on context. For example, safety-critical software
systems are tested differently (more intensely and using other
techniques) from entertainment applications on different mobile
devices.
Principle 7: Absence-of-errors fallacy

A system without failures does not imply that the system will meet the
users’ expectations.
V 2.2.0 / 2017
Testing Effort?
• Testing is economically useful, as long as the costs of finding and

fixing a defect in testing are lower than the costs that are
associated with the occurrence of a defect when used.
Source:
M. Pol, T. Koomen, A. Spillner:
Management and Optimization of the Test Process
(German). dpunkt-Verlag, 2. Edition, 2002
• A good test is like a liability insurance: it costs big money, but

allows the project manager and the customer to sleep peacefully. A
good sleep includes a good insurance policy that covers all
possible risks. The trust in a software product includes a good test
that fully covers the reality of production.
Source:
Siedersleben, J. (Hrsg):
Software methodology: Practical Knowledge for
Software Engineers, (German). 2. Edition, Hanser,
2003
• Successful testing (detection of failures) reduces costs
V 2.2.0 / 2017
More regarding Testing (1/1)
Testing effort in practice:

• 25% to 50% of the development effort
• Test intensity and scope depends on risks and the criticality of the
project, the system and the domain
Defects can result in huge costs:

• Estimated losses due to software defects in medium and large
companies in Germany are approximately 84.4 billion Euros per
annum
• Productivity losses due to computer failures caused by software
defects are approximately 2.6% of sales - 70 billion Euros per
annum Source:
Study by LOT Consulting, Karlsruhe
IT-Services 3/2001, P. 31 (German)
V 2.2.0 / 2017
More regarding Testing (2/2)
Testing is always subject to limited resources

(e.g. time, test personal)
Particularly important:
• Select appropriate test procedures which are compatible with
the test objectives and quality objectives
• Avoid unnecessary tests which provide no new information
• Take positive and negative test conditions into account
• It can also be useful to test for functionality that has not been
requested
V 2.2.0 / 2017
 Prioritization of Tests
Limited resources (time and personnel) require that the most important
test cases are carried out first.
Criteria for prioritization:
• Expected impact (the severity of the damage a defect would cause)
• Probability of occurrence of a failure
• Combined consideration of impact and likelihood of occurrence
( Risk = Likelihood * Impact)
• Perception of the failure
• Prioritization of requirements by the customer
• Importance of quality characteristics by the customer
• Priority of test cases from the perspective of development (serious
consequences and / or complex cases first)
• High project risk
• Where many defects are found, more defects are likely to be
discovered. A change in the priority must be possible.
V 2.2.0 / 2017
2
Chapter
Fundamentals
of Testing
Fundamental Test Process
Ethics of Testing
V 2.2.0 / 2017
SW-Development Models: Waterfall-Model - 1970
Source:
Royce, W. W.:
Managing the Development of Large Software
Systems: Concepts and Techniques
Proc. WESCON, IEEE Computer Society Press, Los
Alamitos, CA, 1970.
Reprinted at the ICSE'87, Monterey, California, USA.
March 30 - April 2, 1987
V 2.2.0 / 2017
SW-Development Models: Waterfall-Model - 1981
Source:
Boehm, B.:
Software Engineering Economics
Prentice-Hall Inc., London, 1981
V 2.2.0 / 2017
V-Model (B. Boehm, 1979)
Source:
Barry W. Boehm:
Guidelines for Verifying and Validating Software
Requirements and Design Specification.
EURO IFIP 79, P.A. Samet (eds.) North-Holland, IFIP
1979, 711-719
V 2.2.0 / 2017
General V-Model
Definition of Requirements Acceptance Testing
Functional System Testing

System Design
Technical Integration Testing

System Design
Component Component Testing

Specification
Programming
Legend:
Test cases based on corresponding
documents
V 2.2.0 / 2017
Test Process
• Is closely linked with software development

• Is, however, a separate, independent process
• A test plan is necessary for every test level (level test plan)
• Testing cannot be considerd as a single activity (e.g. test execution)
• Many individual tasks are performed within testing
• The test process represents these individual testing activities in a coherent
process
V 2.2.0 / 2017
Activities of the Fundamental Test Process
Start
• Test Planning and Control

• Test Analysis and Design Planning and
• Test Implementation and Execution

Analysis and Design
• Evaluating Exit Criteria and
Reporting
Implementation and
Control
• Test Closure Activities Execution
Evaluation and
Report
• These activities may overlap or take
place concurrently.
Closure
• The Fundamental Test Process is to
be tailored to each test level (e.g.
module test, system test) Finish
V 2.2.0 / 2017
Start
Testing Tasks: Test Planning and Control (1/2) Planning and
• Specification of Test Plan

Analysis and
Design
– Determine test objectives, amount and Implementation

and Execution Control
risks of testing.
– Specify test approach (techniques*, Evaluation and
Report
test objects, coverage, teams who

participate in testing, testware). Test Closure
• Details of Test plan Finish
– Test resources (e.g. staff, test environment, PCs).

– How intensive should particular parts of the system be tested?
– Which test techniques should be used?
– Determine exit criteria. What level coverage should be reached?
– Prioritizing of test (taking defect risks into consideration).
– Planning for tool support (use, acquisition, training).
• Test schedule and test strategy are to be recorded in the test plan
*Refer to Chapter 4 and 5
V 2.2.0 / 2017
Start
Testing Tasks: Test Planning and Control (2/2) Planning and
Analysis and
• Drawing up a rough schedule

Design
– Schedule tasks for test analysis and test

Implementation
specification. Evaluation and

Report
– Schedule test implementation, test
execution and test evaluation. Test Closure
• Test Control Finish
– Measuring and analysing results.

– Monitoring and recording of test progress,
achieved test coverage and exit criteria.
– Initiate corrective actions.
– Adapting time and resource planning.
– Making decisions.
• Refer to Chapter 6 for more details
V 2.2.0 / 2017
Start
Testing Tasks: Test Analysis and Design (1/2) Planning and
Analysis and
• General testing objectives are detailed to specific

Design
test requirements and criteria. Implementation

• Basis are all documents showing the Evaluation and

requirements for the software (test basis). Report
• Reviewing the test basis (requirements, Test Closure

architecture, design, interface, ...).
• Evaluating testability of requirements and the system. Finish
• Identifying test conditions / test requirements.

• Designing the test environment (infrastructure, tools, ...).
• Designing test cases using the chosen test design techniques.
• Distinction to be made between high level test cases and detailed
test cases.
• Creating bi-directional traceability between test basis and test
cases
V 2.2.0 / 2017
Start
Testing Tasks: Test Analysis and Design (2/2) Planning and
Analysis and
• Test cases contain more that just test data! Design
– Criteria and constraints that have to be met Implementation

before execution (e.g. condition of test

object, data, network status) Evaluation and
Report
– Before test execution, determine which result Test Closure

or behavior is expected and the post-conditions
that have to be fulfilled after the test, e.g., condition
Finish
of the test object and the database.
– Refer to the next section in this chapter for further details
• Test cases are partly linked together to create test suites (test
procedure specification) for test execution
– Each test case should only contain one or a few elementary
process steps or system calls
– Pay attention in the test design to the post-conditions of the
test cases. These are the pre-conditions of next test cases.
V 2.2.0 / 2017
Start
Testing Tasks: Test Implementation Planning and
Analysis and
Design
• Specifying and prioritizing test cases. Implementation

Control
and Execution
• Creating test data and test scenarios. Evaluation and

Report
• Creating test suites. Test Closure
• Preparing test harnesses and (possibly) writing

Finish
automated test scripts.
• Verifying that the test system and test environment has been set
up correctly.
• Before test execution: Verifying completeness of parts to be tested.
V 2.2.0 / 2017
Start
Testing Tasks: Test Execution (1/3) Planning and
Analysis and
Design
• First check main functions. If any incidents Implementation

Control
and Execution
are found here, continuing with deeper tests
may not make much sense! Evaluation and
Report
• Execute test cases either manually or Test Closure
by using test execution tools according to the

Finish
test procedure specification
• Record the test execution accurately and completely (versions of
test object, test tools and testware) in the test log.
V 2.2.0 / 2017
Start
Analysis and
Design
• Compare actual results with expected results. Implementation

• Report discrepancies as incidents and analyze Evaluation and

them. If a discrepancy of actual and expected Report
result is detected, it may not always mean Test Closure

a software defect has been detected.
• Check if any of the following aspects is true: Finish
– incorrect expected result

– test environment not set up properly
– the test case specification is incorrect
• Determine defect severity and priority for fixing
• Repeat test activities as a result of action taken to resolve each
incident/defect (Re-Testing).
V 2.2.0 / 2017
Start
Analysis and
Design
• Create a test log Implementation

• The test log must contain details on which Evaluation and

parts were tested, when, by whom, Report
how intensive and with what result. Test Closure
• On the one hand the description of test

execution in the test log has to be Finish
comprehensive for those not directly involved

(e.g. clients)
• On the other hand it has to be traceable if the planned test strategy
has really been applied
V 2.2.0 / 2017
Start
Testing Tasks: Evaluating Exit Criteria Planning and
Analysis and
• Compare results of test execution with the Design
defined objectives of the test. Implementation

• Checking test logs against the exit criteria Evaluation and

specified in test planning. Report
• Reached test end? Test Closure
• Achieved coverage? Finish

(Problem: unreachable program code)
• Deciding if more tests are needed or if the specified exit criteria
should be changed (only after consulting stakeholders!)
• Further effort justified?
• Addition practical factors for ending the test:
• No more time
• No more money
V 2.2.0 / 2017
Start
Testing Tasks: Test Reporting (1/3) Planning and
Analysis and
• Write a test summary report for

Design
stakeholders. Implementation
• The test summary report may have two Evaluation and

Report
purposes:
Test Closure
– Communicate test status
– Report on test completion Finish
• Refer to Chapter 6 for more details on test reporting
V 2.2.0 / 2017
Start
Analysis and
Design
• Example for exit criteria: Implementation

Control
and Execution
Found defects in order of failure severity
Evaluation and
Report
New / Testing Hour
3 Test Closure
2,5 Finish
2
Failure Severity
1,5 low
1 medium
high
0,5
critical
0
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 Wx: Calendar Week
V 2.2.0 / 2017
Start
Analysis and
Design
• Example for exit criteria: Implementation

Control
and Execution
Comparison of found failures and corrected defects
Evaluation and
Report
60
Test Closure
50
Finish
40
30
New failures
20
Corrected defects
10
In Process
0
Wx: Calendar Week
W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11
V 2.2.0 / 2017
Start
Testing Tasks: Test Closure Activities (1/2) Planning and
Analysis and
Design
• Collect data from completed test phases and Implementation

Control
consolidate experience, testware, facts, metrics and Execution
etc. Evaluation and

Report
• Check which planned deliverables have been Test Closure

delivered
• Close incident reports or raise change requests Finish
for any that remain open

• Document the acceptance of the system
V 2.2.0 / 2017
Start
Testing Tasks: Test Closure Activities (2/2) Planning and
Analysis and
Design
• Finalize and archive testware, the Implementation

Control
test environment and the test infrastructure and Execution
Evaluation and
• Preserve and hand over the testware to the Report
maintenance organization – everything should Test Closure

be reusable during maintenance, so it has to be
portable and updatable Finish
• Analyse and document any "lessons learned"

for later projects and to improve test maturity
– Evaluation of test process
– Assessment of test process
– Recognizing potential for improvement
• Improve the test process by applying the
findings
V 2.2.0 / 2017
2
Chapter
Fundamentals
of Testing
Test Cases, Expected Results, Test Oraclesch\
Ethics of Testing
V 2.2.0 / 2017
Criteria for Test Cases
• Test cases for verification of specified results and of test object
delivered results and reactions
– Positive test; expected input / operation (normal)).
• Test cases that verify the specified handling of exceptional
situations and defects also need to be considered
– Negative test; expected false input / operation (exceptional).
– Remark: It is often difficult to create the conditions necessary
for execution of negative test cases (e.g. the overload of a
network connection).
• Test cases for verification of reactions of the test object for invalid
and unexpected inputs or constraints, for which there was no
exception handling specified
– Negative test / robustness test; unexpected erroneous inputs /
operations (catastrophic).
V 2.2.0 / 2017
Test Specification –
High Level and Specific Test Cases (1/3)
Example:
A company has ordered a program that should calculate the
Christmas bonus of the staff in relation to the time they have been
working there. In the description of the requirements you find the
following text:
»Staff who have been with the company for more that three years will
receive 50 % of the monthly salary as Christmas bonus. Staff who
have been with the company for more than five years will receive 75
%. Staff who have been with the company for more than eight years
will receive 100 % of their monthly salary.«
How do the test cases look like?
V 2.2.0 / 2017
• From the text you can set up the following relationship between the
allowance of the bonus and the time working for the company:
years with the company <= 3 equals bonus = 0 %

3< years with the company <= 5 equals bonus = 50 %
5< years with the company <= 8 equals bonus = 75 %
years with the company >8 equals bonus = 100 %
V 2.2.0 / 2017
High Level (logical) 1 2 3 4

Test Case
Input value x x<=3 3<x<=5 5<x<=8 x>8
(years with the company)
Expected result 0 50 75 100

(bonus in %)
Specific Test Case 1 2 3 4
Input value x 2 4 7 12
(years with the company)
Expected result 0 50 75 100

(bonus in %)
Remarks:
• no pre- and post-conditions or constraints are considered
• the test cases were not derived systematically
• only positive tests with expected results
V 2.2.0 / 2017
Expected Results and Test Oracle
• After each executed test case, it must be decided whether there is

a failure or not.
• For this decision, the monitored result (actual result/behavior) has
to be compared with the expected result (expected
result/behavior).
• The expected behavior of the test object has to be determined in
advance for each test case.
• The tester must obtain this information from appropriate sources
when specifying a test case.
• In this connection, we also speak of an oracle or test oracle that
can be consulted and that predicts the expected results.
V 2.2.0 / 2017
Expected Results and Test Oracle
The expected results are to be deducted of the specification.
Following three possibilities:
• The tester derives the expected date from the input date on the basis
of the specification of the test object. This is the common practise.
• If a formal specification is available, an executable prototype can be
created with tool support. The results of the prototype can be used
as a test oracle for testing the actual program.
• Parallel designs of the software program may be created by different
development groups. Each version of the program will then be tested
against another version using the same test data. If there are
different results in the two versions there must be a failure in one of
the versions. Only those failures that have the same effect in all the
versions remain undetected. This procedure is called Back-to-back
test.
V 2.2.0 / 2017
Other Test Oracles
• Making use of user manuals

• The system itself (create operational profile)
This is the worst possibility
• (Tested!) previous versions
• Programs with similar functionality
• Exact values cannot always be predicted or calculated. Determine
range of tolerance, verify plausibility
• Experience is important
V 2.2.0 / 2017
2
Chapter
Fundamentals
of Testing
Test cases, Expected Results, Test Oracle
Ethics of Testing
V 2.2.0 / 2017
• Making mistakes is “human”

- but who likes admitting that?
• Development = constructive
Test = destructive?
Source:
“Testing is an extremely creative Myers, Glenford J.:
and intellectually challenging task” Systematic Testing of Programs
Oldenbourg, 2001 (7. Edition)
• Is it possible for developer to test

his own program?
• What to you think?
V 2.2.0 / 2017
Psychology of Testing – Developer Test
• Being blind to see own mistakes

– If the developer has implemented a basic design error, e.g.,
because he has misunderdstood the task, he is not able to
detect this through his own tests.
– He will not be able to specify appropriate test cases.
• No familiarisation
– The developer knows his test object
V 2.2.0 / 2017
Psycholoy of Testing - Independent Test Team
• An independent test team is unbiased

– It his not “ his/their ” product
– The possible assumptions and misunderstandings of the developer
are not necessarily the assumptions and misunderstandings of the
tester.
• Need for familiarization

– To create test cases the tester needs to gain knowledge about the test
object. This requires time.
• Test Know-how
– The tester should have test know-how
– A developer probably does not have this or first has to gain it (often
with not enough time available)
– Even better, the know-how has already been acquired at university ☺
V 2.2.0 / 2017
Psychology of Testing – Possible Levels
• Tests are carried out with different levels of independence

– By the developer himself
– By colleagues of the developer in the same project
– By persons of other departments
– By persons of other organizations.
• Tools can be used to support all possibilities.
• The selection depends on the product as well as the project.
• It is important to get the right mix and balance between
“independent testing“ and tests performed by the developer.
V 2.2.0 / 2017
Psychology of Testing – Communication of Defects
• Communication of identified defects or incidents to the developer

and/or the management requires
– neutral, fact-focused and constructive way of communication
– Undisturbed, open communication.
• Proving another person‘s error is not an easy task. It requires good
“ interpersonal skills ”.
• Reproduceability of defects is important!
– Recording the test environment
– Differences to the development environment
• Clear requirements, precise specification
– “ It’s not a bug, it’s a feature! ”
• Mutual understanding between testers and developers
– Collaboration rather than battles!
V 2.2.0 / 2017
2
Chapter
Fundamentals
of Testing
Test cases, Expected Results, Test Oracle
Ethics of Testing
V 2.2.0 / 2017
Ethics of Testing (1/3)
• Software testers often have access to confidential and legally privileged

information. This must not be used inappropriately.
• Recognizing the ACM and IEEE code of ethics for engineers, the ISTQB
states the following code of ethics:
1. PUBLIC
Certified software testers shall act consistently with the public interest.
2. CLIENT AND EMPLOYER

Certified software testers shall act in a manner that is in the best interests
of their client and employer, consistent with the public interest.
V 2.2.0 / 2017
3. PRODUCT
Certified software testers shall ensure that the deliverables they provide
(on the products and systems they test) meet the highest professional
standards possible.
4. JUDGMENT
Certified software testers shall maintain integrity and independence in
their professional judgment.
5. MANAGEMENT
Certified software test managers and leaders shall subscribe to and
promote an ethical approach to the management of software testing.
V 2.2.0 / 2017
6. PROFESSION
Certified software testers shall advance the integrity and reputation of
the profession consistent with the public interest.
7. COLLEAGUES
Certified software testers shall be fair to and supportive of their
colleagues, and promote cooperation with software developers.
8. SELF
Certified software testers shall participate in lifelong learning regarding
the practise of their profession and shall promote an ethical approach to
the practice of the profession.
V 2.2.0 / 2017
✓ Summary
• Failure – Not fulfilling the requirements!
– Describing this situation as failure
– cause: defect in the software
– that has been caused by an error made by a person
• Tests are important measures for securing the quality criteria
– Functionality, reliability, usability, efficiency, maintainability und
portability [ISO 9126]
• Exhaustive testing is not possible. Tests are always random
checks, therefore they can detect failures only with a certain
probability
• The intensity and amount of testing depends on the expected risk
of faulty behavior of the program
• Tests should be prioritized
V 2.2.0 / 2017
✓ Summary
• The fundamental test process consists of the main activities
– Test Planning and Control
– Test Analysis and Design
– Test Implementation and Execution
– Evaluating Exit Criteria and Reporting
– Test Closure Activities
• A test case consists of

– an input value and
– an expected result as well as
– preconditions and constraints for the execution of the test case, and
– postconditions that have to be fulfilled after the test case.
• While executing the test case, the test object shows its actual behavior. If there is a discrepancy
between expected and actual behavior there is a failure.
• A test oracle determines the expected values for each test case before test execution
• Humans make mistakes – but they do not like admitting them!
V 2.2.0 / 2017
By now, you should be able to answer the following
??? questions
• Define the terms software failure, defect, error.

• What is defect masking?
• Explain the difference between testing and debugging.
• Define the terms verification and validation.
• Explain why tests cannot be exhaustive.
• Name the main software quality attributes according to ISO 9126.
• Define the term reliability of a system.
• Explain the main activities of the fundamental test process.
• What is a test oracle?
• Why should developers not test their own programs?
V 2.2.0 / 2017
Finally
V 2.2.0 / 2017

CTFL-Chapter 2-Fundamentals of Testing v2.2.0

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CTFL-Chapter 2-Fundamentals of Testing v2.2.0

Uploaded by

Copyright:

Available Formats

Software Testing Foundations

• 1.1 Why is Testing Necessary? (K2)

– LO-1.1.3 Give reasons why testing is necessary by giving examples (K2)

• 1.2 What is Testing? (K2)

– LO-1.2.3 Differentiate testing from debugging (K2)

• 1.3 Seven Testing Principles (K2)

• 1.4 Fundamental Test Process (K1)

• 1.5 The Psychology of Testing (K2)

– LO-1.5.2 Contrast the mindset of a tester and of a developer (K2)

A situation can only be classified as defective if it is pre-defined, how

People make mistakes; they commit errors.

Example: After the

A failure is the effect of Defect

Confirmation by examination and through provision of

Confirmation by examination and through the provision

• A feature or characteristic that affects an item's quality [IEEE 610].

Effectiveness Productivity Security Satisfaction

Fulfillment of tasks Fulfillment of tasks Fulfillment of tasks

Functionality Reliability Usability Efficency Maintainability Portability

Suitability Understandability Analyzability Adaptivity

Existence of functions with specified attributes. These functions meet the

Attributes that relate to the ability of software to be transferred from one

• Quality requirements state which quality attributes the product

• Not all quality attributes can be achieved equally well.

• Testing measures the quality e.g., based on the number of located

ISO 25010 Neu, vorher unter „Functionality“

In the previous 50 years, the following seven testing principles have

Principle 1: Testing shows presence of failures

Principle 2: Exhaustive testing is impossible

Principle 3: Early testing

Principle 4: Defect clustering

Principle 5: Repetitions have no effects - Pesticide Paradox

Principle 6: Testing is context-dependent

Principle 7: Absence-of-errors fallacy

• Testing is economically useful, as long as the costs of finding and

• A good test is like a liability insurance: it costs big money, but

• Successful testing (detection of failures) reduces costs

Testing effort in practice:

Defects can result in huge costs:

Testing is always subject to limited resources

Definition of Requirements Acceptance Testing

Functional System Testing

Technical Integration Testing

Component Component Testing

• Is closely linked with software development

• Test Planning and Control

• Test Implementation and Execution

Testing Tasks: Test Planning and Control (1/2) Planning and

• Specification of Test Plan

– Determine test objectives, amount and Implementation

test objects, coverage, teams who

• Details of Test plan Finish

– Test resources (e.g. staff, test environment, PCs).

*Refer to Chapter 4 and 5

Testing Tasks: Test Planning and Control (2/2) Planning and

• Drawing up a rough schedule

– Schedule tasks for test analysis and test

specification. Evaluation and

• Test Control Finish

– Measuring and analysing results.

Testing Tasks: Test Analysis and Design (1/2) Planning and

• General testing objectives are detailed to specific

test requirements and criteria. Implementation