Software Quality & Testing

Syllabus
Unit – 1 : Basics of Software Testing
 Humans, Errors and Testing, Software Quality; Requirements,
Behavior and Correctness, Correctness Vs Reliability; Testing
and Debugging; Test Metrics; Software and Hardware Testing;
Testing and Verification; Defect Management; Execution
History; Test Generation Strategies; Static Testing; Test
Generation from Predicates.
Unit – 2 : A perspective on testing, Examples
 Basic definitions, Test cases, Insights from a Venn diagram,
Identifying test cases, Error and fault taxonomies, Level of
testing, Examples: Generalized pseudocode, The triangle
problem, the NextDate function, The commission problem, The
SATM (Simple Automation Teller Machine) problem, The
currency converter, Saturn windshield wiper.
 Syllabus
 Unit – 3 : Boundary Value testing, Equivalence
class testing, Decision-table based testing
Boundary value analysis, Robustness testing, Worst-case
testing, special value testing, Examples, Random testing,
Equivalence classes, Equivalence test cases for triangle problem,
NextDate function and commission problem, Guidelines and
observations, Decision tables, Test cases for triangle problem.
 Unit – 4 : Path testing, Dataflow testing
DD Paths, Test coverage metrics, Basis path testing,
guidelines and observations, Definition Use testing, Slice based
testing, Guidelines and observations.
Fault-based testing
Assumptions in fault-based testing, Mutation Analysis, Fault-
based Adequacy Criteria; Variations on mutation Analysis; From
Test case specification to Test Cases, Scaffolding, Generic vs
specific Scaffolding, Test Oracles, Self checks as oracles, Capture
and Replay.
Syllabus
 Unit– 5 : Planning and Monitoring the
Process, Documenting Analysis and Test
Quality and Process, Test and Analysis strategies and
plans, Risk Planning, Monitoring the Process,
Improving the process, The quality team, Organizing
documents, Test strategy document, Analysis and
test plan, Test design specifications documents, Test
and analysis reports.
Course Outcomes
 Test process and continuous quality improvement & Test
generation from requirements
 Modelling techniques : UML : FSM and State Charts,
Combinatorial design ; and others
 Test generation from models
 Test adequacy assessment
 Industrial applications
 Discussion oriented lectures by the instructor, In-class
group presentations by teams, laboratory exercises using
advanced testing tools, and invited talks by experts from
the industry will be the primary mechanisms for learning
and the dissemination of knowledge
 Unit – 1 Outcomes
 At the end of this Unit, a student will be able to :

Understand the basic concepts of software testing like

- Human, Errors and testing
- Software quality attributes
- Requirements, Behaviour and Correctness
- Correctness Vs Reliability
- Testing Vs Debugging
- Test Metrics
Unit – 1 Outcomes Contd…

 Software and Hardware testing

 Testing and Verification
 Defect Management
 Execution history
 Test Generation Strategies
 Static testing
 Test Generation from Predicates
Unit - 1
Humans, Errors and Testing
Errors
 Errors are a part of our daily life.

 Humans make errors in their thoughts, actions, and in

the products that might result from their actions.

 Errors occur wherever humans are involved in taking

actions and making decisions.
Examples of Errors
Errors, Faults and Failures
 Test Automation
 Testing Complex Systems is very effort-intensive
 Execution of many tests can be boring and error-prone
too
 Since test requirements vary from one organization to
another, they build their own in-house testing tools to
test their applications
Note :
Developer Vs Tester
- Typical roles in software development
- A person who writes the code is the developer
- A tester is a person who tests the code i.e. to test the
conformance to requirements
Software Quality
- Multidimensional Measurable Quantity

Static quality attributes: structured, maintainable,

testable code as well as the availability of correct and
complete documentation.
Dynamic quality attributes: software reliability,
correctness, completeness, consistency, usability,
and performance
Software Quality Contd…
Completeness refers to the availability of all features listed
in the requirements, or in the user manual. An incomplete
software is one that does not fully implement all features
required.

Consistency refers to adherence to a common set of

conventions and assumptions. For example, all buttons in the
user interface might follow a common color coding
convention. An example of inconsistency would be when a
database application displays the date of birth of a person in
the database.
Software Quality Contd…
Usability refers to the ease with which an application can be
used. This is an area in itself and there exist techniques for
usability testing. Psychology plays an important role in the
design of techniques for usability testing.

Performance refers to the time the application takes to

perform a requested task. It is considered as a non-functional
requirement. It is specified in terms such as ``This task must
be performed at the rate of X units of activity in one second
on a machine running at speed Y, having Z gigabytes of
memory."
Requirements, Behaviour and
Correctness
Requirements leading to two different programs:

Requirement 1: It is required to write a program that inputs two

integers and outputs the maximum of these.

Requirement 2: It is required to write a program that inputs a

sequence of integers and outputs the sorted version of this
sequence.
 Requirements : Incompleteness

Suppose that program max is developed to satisfy Requirement 1.

The expected output of max when the input integers are 13 and 19
can be easily determined to be 19.

Suppose now that the tester wants to know if the two integers are to
be input to the program on one line followed by a carriage return, or
on two separate lines with a carriage return typed in after each
number. The requirement as stated above fails to provide an answer
to this question.
Requirements : Ambiguity

 Requirement 2 is ambiguous. It is not clear whether the

input sequence is to sorted in ascending or in
descending order. The behavior of sort program, written
to satisfy this requirement, will depend on the decision
taken by the programmer while writing sort.
Input Domain (Input Space)

The set of all possible inputs to a program P is known as the input

domain or input space, of P.

Using Requirement 1 above we find the input domain of max

to be the set of all pairs of integers where each element in the pair
integers is in the range -32,768 till 32,767.

Using Requirement 2 it is not possible to find the input domain

for the sort program.
 Input Domain (Contd…)
Modified Requirement 2:
It is required to write a program that inputs a sequence of integers and
outputs the integers in this sequence sorted in either ascending or
descending order. The order of the output sequence is determined by an
input request character which should be ``A'' when an ascending sequence
is desired, and ``D'' otherwise.

While providing input to the program, the request character is input first
followed by the sequence of integers to be sorted; the sequence is
terminated with a period.

Based on the above modified requirement, the input domain for sort is a
set of pairs. The first element of the pair is a character. The second element
of the pair is a sequence of zero or more integers ending with a period.
Valid/Invalid Inputs
The modified requirement for sort mentions that the
request characters can be ``A'' and ``D'', but fails to answer
the question ``What if the user types a different character ?’’

When using sort it is certainly possible for the user to type a

character other than ``A'' and ``D''. Any character other than
``A'’ and ``D'' is considered as invalid input to sort. The
requirement for sort does not specify what action it should
take when an invalid input is encountered.
Correctness Vs Reliability
Though correctness of a program is desirable, it is almost
never the objective of testing.

To establish correctness via testing would imply testing a

program on all elements in the input domain. In most cases
that are encountered in practice, this is impossible to
accomplish.

Correctness is established via mathematical proofs of

programs.
Correctness and Testing
While correctness attempts to establish that the program is error
free, testing attempts to find if there are any errors in it.

Thus completeness of testing does not necessarily demonstrate

that a program is error free.

Testing, debugging, and the error removal processes together

increase our confidence in the correct functioning of the program
under test.
Software Reliability : Definition

Software reliability [ANSI/IEEE Std 729-1983]: is the probability

of failure free operation of software over a given time interval and
under given conditions.

Software reliability is the probability of failure free operation of

software in its intended environment.
What is an Operational Profile ?

An operational profile is a numerical description

of how a program is used.
Consider a sort program which, on any given
execution, allows any one of two types of input
sequences. Sample operational profiles for sort
follow.
Operational Profile Contd…
Operational Profile Contd…
Testing and Debugging

- Testing is the process of determining if a program has any

errors.

- When testing reveals an error, the process used to determine the

cause of this error and to remove it, is known as debugging.
Differences between testing and debugging
Testing Debugging

The purpose of debugging is to

The purpose of testing is to find
correct those bugs found during
bugs and errors.
testing.

Debugging is done by programmer

Testing is done by tester.
or developer.

It can be automated. It can’t be automated.

It can be done by outsider like It must be done only by insider

client. i.e. programmer.

Most of the testing can be done Debugging can’t be done without

without design knowledge. proper design knowledge.
A test/debug cycle
Test Plan
A test cycle is often guided by a test plan.

Example: The sort program is to be tested to meet the

requirements given earlier. Specifically, the following needs to
be done.

Execute sort on at least two input sequences, one with ``A'' and
the other with ``D'' as request characters.
Test Plan (Contd…)

• Execute the program on an empty input sequence.

• Test the program for robustness against erroneous

inputs such as ``R'' typed in as the request character.

• All failures of the test program should be recorded in a

suitable file using the Company Failure Report Form.
Test Case/Data
A test case is a pair consisting of test data to be input to the program
and the expected output. The test data is a set of values, one for each
input variable.

A test set is a collection of zero or more test cases.

Sample test case for sort:

Test data: <''A'’ 12 -29 32 >

Expected output: -29 12 32
Program Behaviour

Can be specified in several ways: plain natural language, a

State diagram, formal mathematical specification, etc.

A state diagram specifies program states and how the

program changes its state on an input sequence.
 Program Behaviour - Example

Consider a menu
driven application.
Program Behaviour – Example (Contd…)
Behaviour : Observation and Analysis
The tester finds out whether the program under the test is correct
or not. Divided into two steps as follows

In the first step one observes the behavior.

In the second step one analyzes the observed behavior to check if

it is correct or not. Both these steps could be quite complex for
large commercial programs.

The entity that performs the task of checking the correctness of the
observed behavior is known as an Oracle.
 Oracle: Example

38
Human Oracles - Disadvantages

 Possibility of making errors in analysis i.e. error-

prone
 Speed of computation
 Trivial I/O behaviours

Note :
Human Oracles are the best when compared with
automated oracles
 Oracle: Programs

Oracles can also be programs designed to check the

behavior of other programs.

For example, one might use a matrix multiplication

program to check if a matrix inversion program has
produced the correct output. In this case, the matrix
inversion program inverts a given matrix A and generates
B as the output matrix.

40
 Oracle: Construction

Construction of automated oracles, such as the one to check

a matrix multiplication program or a sort program, requires
the determination of input-output relationship.

In general, the construction of automated oracles is a

complex undertaking.

41
 Oracle construction: Example

42
Test Metrics
 Metric is a standard of measurement
 There are different types of metrics for testing
Types of Test Metrics
a) Organizational
b) Project
c) Process
d) Product
i) Static
ii) Dynamic
Test Metrics
Organizations

Establish test processes

Used in projects

To test products
 Test Metrics
 Core Areas in the design of metrics
1) Schedule
- Actual completion times of various activities
and comparing with the estimated time to
completion
2) Quality
- Measures the quality of a product or a process
3) Resources
- Measures items like cost in rupees, manpower,
and tests executed
4) Size
- Measures the source code and number of test in a test suite
 Test Metrics
1) Organizational metrics
- Metrics that are used in the overall project planning and
management
- Obtained by combining two or three metrics together across
several projects
Examples of Organizational metrics
1) Number of defects before release of the product
2) Number of defects after release of the product
3) Testing costs per LOC
4) Delivery Schedule Slippage
5) Time to complete system testing etc…
Note:
- These metrics need to be computed at regular intervals
Test Metrics

Advantages of Organizational Metrics

a) Allow the top management to monitor the overall
strength of the organization
b) Points to several areas of weakness that can be
strengthened in the future
c) Helps the management to set new goals and plan
for resources to realize these goals
Test Metrics

2. Project Metrics
- Metrics used to measure the status of a project
- Could be measured in terms of tester-man-months,
tester-man-years, tester-man-days etc…
- Usually used by the Project Manager
Examples :
- Actual-to-planned system test effort
- Ratio of the number of successful tests to the total
number of tests in the testing phase
Test Metrics
3. Process Metrics
- Evaluating the efficiency and goodness of
processes involved in a project

Examples
- How many defects were present in each testing
phase i.e. unit testing, integration testing, and
system testing etc…
- Metrics taken during the SDLC
Test Metrics
4. Product Metrics : Generic
- Metrics related to a specific software product
Examples
a) “Can this product be released on 10/8/2018”
b) “Has user satisfaction done for this product”
c) Cyclomatic complexity and Halstead Metrics
Cyclomatic Complexity
- Coined by Thomas Mc Cabe
Formula
V(G) = E – N + 2
V(G) = P + 1 where ‘P’ stands for Predicate nodes in a graph
Test Metrics
Metric Meaning Formula
n Halstead’s Volume Vocabulary n1 + n2

N Size N2 + N2

V Volume N * log2 n

D Difficulty n1/2 * N2/n2

E Effort V*D

B Errors V / 3000

Halstead’s Metrics
Test Metrics

5. Product Metrics : OO Software

a) Reliability metrics
b) Defect density
c) Defect severity
d) Test Coverage
e) Cyclomatic Complexity
f) Weighted Methods per class (WMC) etc…
 Test Metrics
6. Progress Monitoring using Metrics
- Performing at regular intervals
- Show the trends of the application over time
Eg: Measuring the defects during a certain interval of
time
- Drawing necessary graphs to deliver a visual view of
the application Eg:
 Test Metrics
7. Static and Dynamic Metrics
- Static metrics are computed from a static application
- Dynamic metrics are computed for changing requirements
8. Testability
Defn : The degree to which a system of component facilitates the
establishment of test criteria and the performance of tests to determine
whether those criteria have been met.
- Existence of static and dynamic testability metrics
Eg: Software complexity is an example of a static testability metric
- The more complex an application, the lower the testability
Eg: Code-based coverage metrics are examples of dynamic testability
metrics
- The level of testability should be identified at the requirements
stage itself.
Software and Hardware testing
 Software failures and hardware failures
 Presence of BIST (Built-In Self Test) – Applied
to hardware initially, applied now to software
also.
Fault Models
- Hardware tests are generated depending on
the fault model of devices
- Software tests are generated to check the
level of correctness of a software application
 Software and Hardware testing
- Hardware tests are performed at the following different
levels
a) Transistor level
b) Gate level
c) Circuit level
d) Function level
- Software test models also exist i.e. depends on the
tester, whether he uses it or not
- Mutation testing is done to test software
Software and Hardware testing
Software and Hardware testing
Original Program Mutant Program

If (x>y)
If(x<y)
Print "Hello"
Print "Hello"
Else
Else
Print "Hi"
Print "Hi"
Software and Hardware testing

Test Domain
 Domain differs for software and hardware
platforms
 Sequential circuits vs Combinatorial circuits
Test Coverage
- Acceptable test coverage should be taken care
- Mutation testing in software
 Testing and Verification
 Verification
- Showing that the program works for all possible inputs
that satisfy a set of conditions
- Humans make mistake while verifying certain artefacts
For eg: Incorrect assumption of input conditions,
incorrect understanding of the application delivery
etc…

 Testing
- Ensuring that a program or an application works error
free during the lifetime of its usage
Software and Hardware testing
Major differences
1) Difficulty in reproducing bugs
2) Cost of bug fixes
3) Need for test cases
4) The cost of testing
5) Testing the durability aspects
6) Hardware and firmware testing
7) Lab requirements
8) Testing environment
9) Agile development
 Defect Management
 Activities in a Defect Management Process
1) Defect Prevention – Better standards
2) Defect Discovery – Inspecting the system at regular intervals
3) Defect Recording and Reporting – Keeping a record of the
defects and informing the immediate higher-ups, Recorded
defects takes a state “Open”
4) Defect Classification – High severity Vs Low severity
5) Defect Resolution & - Finding a quick permanent solution
6) Defect Prediction – Using advanced statistical techniques to
predict defects in future releases of the software product
Note :
Presence of a number of defect tracking systems like bugzilla,
bugzero, Fogbugz that can be used to measure the complexity of
applications.
Execution History
 Execution history is otherwise called as “Execution
trace” is an organized collection of information about
various elements of a program during a given
execution
Execution Slice
- An executable sub-sequence of execution history
Representing an execution history
1) Sequence in which the functions of a program are
executed against a given input
2) Sequence in which program blocks are executed
Execution History

Note: A complete execution history recorded from

the start of a program’s or application’s execution
until its termination represents a single execution
path through the program or application
Test Generation Strategies

 Different ways tests can be generated

1) Using the source document
2) Mix of formal (SRS) and informal methods
3) Buildingformal models for test generation
using the Requirements.
 Test Generation Strategies

a) Model-based: require that a subset of the

requirements be modeled using a formal notation
(usually graphical). Models: Finite State Machines,
Timed automata, Petri net, etc.
b) Specification-based: require that a subset of the
requirements be modeled using a formal
mathematical notation. Examples: B, Z, and other
symbols.
c) Code-based: generate tests directly from the code.
Test Generation Strategies
Test Generation Strategies
Petri Nets
Static testing

Process
1) Done by people who have not written the code
2) Involvement of a team of individuals
Advantages of static testing
1) Helps in discovering faults in the application
2) Ambiguities and Errors in the requirements
3) Testing other application-related documents at a lower
cost
Static testing

Figure : Elements of Static Testing

Static Testing
Activities under static testing
a) Walkthroughs
- Informal process to review any application-related
document
- Code gets reviewed using code walkthrough (peer code
review)
- Begins with a document agreed by all the stakeholders
- In requirements walkthrough, the test team reviews
the requirements to ensure it conforms to the user
needs, free from ambiguities and inconsistencies
Static Testing
b) Inspections
- More formal process when compared with walkthroughs
- Improves code quality at a lower cost
- Improves productivity and software quality
- Done by a team
- The team works on an inspection plan that consists of
the following elements
i) Statement of purpose
ii) Work product (Source code and associated documents)
to be inspected
iii) Team formation (roles and tasks to be performed)
iv) Inspection rate
v) Data collection forms etc…
Static testing

Inspection Roles
1) Moderator
2) Reader
3) Recorder
4) Author
Static testing
c) Use of static code analysis tools in static testing
- Using tools with good GUI capabilities
- Provide control flow and data flow information
(CFG)
- Availability of several tools., Proprietary Vs Open
Source
Eg : Purify, Klockwork (Proprietary)
Lapse (open source) used to analyze java
programs
Static testing
d) Software Complexity and Static testing
- A module that has high complexity after
inspection should be given the highest priority
and vice versa
Note:
Static testing is very important for several software
applications.
Dynamic testing is also very much essential as the
requirements of applications has increased over
time
 Test Generation from Predicates
 A condition is represented by a predicate
 Concept of decision tables
 Testing to ensure that there are no errors in the
implementation of predicates is also known as predicate
testing
1. Predicates and Boolean Expressions
a) Relational Expression (Definition)
An expression of the form e1 relop e2 where e1 and e2 are
expressions that assume values from a finite or infinite set S.
b) Condition
A condition can be represented as a simple predicate or a
compound predicate
 Test Generation from Predicates
c) Simple Predicate
- A simple predicate is a Boolean variable or a relational
expression, where any one or more variables might be
negated Eg: a Λ b, a - b > c
d) Compound Predicate
- A simple predicate or an expression formed by joining
two or more simple predicates, possibly negated, with a
binary Boolean operator
Eg: (a Λ b) .and. (c Λ d)
e) Boolean expression
- An expression composed of one or more Boolean
variables joined by Boolean operators (p Λ q) V r
 Test Generation from Predicates
2. Fault Model for Predicate Testing
Three kinds of faults
a) Boolean Operator fault
– Incorrect Boolean operator
- Missing negation or negation placed incorrectly
- Incorrect parentheses
- Incorrect Boolean variable
b) Relational Operator fault
- Incorrect relational operator is used
c) Arithmetic Expression fault
- Arithmetic errors i.e. placing a “+” in the place of a “-”
Note : Missing or Extra Boolean variable faults
 Test Generation from Predicates
3. Predicate Testing Criteria
Objective:
Generating a test set T from a given predicate pr such
that
a) T is minimal
b) T guarantees the detection of any fault in the
implementation of pr
- Define three testing criteria namely BOR-, BRO- and
BRE-
where BOR- means Boolean Operator, BR0- means
Boolean and Relational Operator, and BRE- means
Boolean and Relational Expression.
 Test Generation from Predicates
 BOR-testing Criteria
A test set T that satisfies the BOR-testing criterion for a compound
predicate pr, guarantees the detection of single or multiple Boolean
operator faults in the implementation of pr . T is referred to as a BOR-
adequate test set and sometimes written as TBOR
 BRO-testing Criteria
A test set T that satisfies the BRO-testing criterion for a compound
predicate pr, guarantees the detection of single or multiple Boolean
operator and relational operator faults in the implementation of pr . T
is referred to as a BRO-adequate test set and sometimes written as
TBRO
 BRE-testing Criteria
A test set T that satisfies the BOR-testing criterion for a compound
predicate pr, guarantees the detection of single or multiple Boolean
operator, relational expression, and arithmetic expression faults in the
implementation of pr . T is referred to as a BRE-adequate test set and
sometimes written as TBRE
 Test Generation from Predicates

4. Cause-Effect Graphs and Predicate Testing

A requirement modeling and test-generation
technique
 Cause-effect relations are extracted from the
requirements
 Thecause portion can be expressed as a
predicate
 The effect portion is used to construct the oracle
 Similar to a decision-table technique
 References
 http://www.utdallas.edu/~ewong/SE6367/03-Lecture/03-Preliminary-one-slide.pdf
 https://www.thecrazyprogrammer.com/2018/01/difference-testing-debugging.html