Universita’ degli Studi di Perugia

Dipartimento di Matematica e Informatica

Corso di Laurea in Informatica

Ingegneria delSoftware
Prof. Alfredo Milani

Test e Verifica del Software

Introduzione
Materiale note, grazie al contributo di: Alfredo Milani, Fabrizio Montecchiani, Carlo Ghezzi, Alessandro Riccardi, Paolo Mengoni et al.
 Verification
Not looking for compiler errors!

Absence of defects:

• Results different from what expected

• Runtime errors, Exceptions, Fault and failures

 Why, what, where?
Zero defect software practically impossible to achieve
• Careful and continuous verification needed
• Everything must be verified
(especially documents, design documents, test data, …)

even the verification must be verified!

• Verification along all development process, not just at the

end
 Verification in engineering
• Example of bridge design
• One test assures infinite correct situations
 Software engineering
Programs do not display a “continuous” behavior

• Verifying the function in one point does not tell us

anything about other points

– Example 1
...
a = … / (x +20) ...
...
Any value of x is ok, except for x = -20!
 Difficulties
• For many qualities, it is not a binary (yes/no)
property

• Many properties are subjective

• Some are even implicitly stated

 Main Approaches
TESTING
– experimenting with behavior of the products
– sampling behaviors
• GOAL: find “counterexamples”
– is a dynamic technique

ANALYSIS
– analytic study of properties (es. Model checking, time
dependency) – it is a static and formal technique
 Testing

Program testing can be used to show the

presence of bugs, but never to show their
absence. (Dijkstra 1972)

• random testing is not effective to find flaws

• Random testing is “blind”, it does not “look for bugs”
 Testing and Debugging
• Testing should identify the presence of errors
• Debugging must locate and eliminate errors

• Then a test may be repeated to see if the error has been

eliminated
– repeatability is essential!
– For concurrent /real-time it may be difficult to
achieve, time depending behavior es.mutual exclusion
 Definitions (1) Program, Input/Output Domain, Correctness

P (program), D (input domain), R (output domain)

A program is a function mapping D into R
– P: D → R ( the function may be partial)

• correctness defined by OR ⊆ D × R
– P(d) correct if the pair <d, P(d)> ∈ OR
– P correct if for each d ∈ OR all P(d) are correct
 Definitions (2) Failure, Error, Fault
• FAILURE
– P(d) is not correct
may be undefined (error state) or may be the wrong result

• ERROR
– anything that causes a failure
Ex. typing mistake, programmer forgot to test “x = 0”

• FAULT
– incorrect intermediate state entered by program

Warning: definitions are not standardized!

 Testing & Debugging Goals

• TESTING
– Try to increase the probability that the presence
of an error results in a failure

• DEBUGGING
– Try to identify faults as early as possible
 Definitions (3) Test case, Test set
• Test case t
– an element of D
• Test set T
– a finite subset of D
• Test t is successful if P(t) is correct
• Test set T is successful if P correct for all t in T

Warning: definitions are not standardized!

 Definition (4) Ideal Test Set
• Ideal test set T
– if P is incorrect, there is an element of T such
that P(d) is incorrect

• if an ideal test set exists for any program, we could

prove program correctness by testing
 Test Criteria
• A criterion C defines finite subsets of D (test sets)
– C⊆(D)  part-of

• A test set satisfies C if is an element of C

– Example one variable x
C = {<x1, x2,..., xn> | n ≥ 3 ∧ ∃ i, j, k, ( xi<0 ∧ xj=0 ∧ xk>0)}
<-5, 0, 22> is a test set that satisfies C
<-10, 2, 8, 33, 0, -19> also does
<1, 3, 99> does not
Properties of criteria (1) consistency, completeness
• Criteria C is consistent
– for any pairs T1, T2 satisfying C, T1 is successful iff T2 is
successful
so any of them provides the “same” information

• Criteria is complete
– if P is incorrect, there is a test set T of C that is not successful

• Criteria C is complete and consistent

– identifies an ideal test set
– allows correctness to be proved!
 Properties of criteria (2)
Criteria C1 is finer than Criteria C2
– if for any program P

• for any T1 satisfying C1 there is a subset T2 of T1 which

satisfies C2
 Properties of Properties
• None is effective, i.e., no algorithms exist to state if a program, test
set, or criterion has that property
• In particular, there is no algorithm to derive a test set that would
prove program correctness

– there is no constructive criterion that is consistent and complete

See: Theoretical Undecidability: Termination Decisional Problem

 Empirical testing criteria
Empirical Testing are an attempted compromise between the
impossible and the inadequate

• Goal: try to divide D into subdomains D1, D2, …, Dn where any

element of each Di is likely to have similar behavior
– D = D1 ∪ D2 ∪… ∪ Dn

• Select one test for each subdomain

• If Dj ∩ Dk ≠ ∅ then use an element in D’=Dj ∩ Dk to reduce tests

 Empirical testing criteria
We try to achieve a “complete coverage principle”:
If D is partitioned into D1, D2, …, Dn, the test set T
contains at least one element from each Di
 Testing Modules
Two approaches:

• BLACK BOX (functional) testing

– partitioning criteria based on the module’s specification,
not knowing any internal detail

• WHITE BOX (structural) testing

– partitioning criteria based on module’s internal
Code, knowing the structure