Chapter 1

Introduction

The importance of proper software testing is obvious. Most of defects

discovered after delivery of software products could be considered as the
result of inadequate or insufficient software testing. The successful software
testing which satisfies a given testing criterion assures high quality and
reliability of software product since the practice can detect the hidden bugs or
potential threats to software and allow developers to correct the faults before
product release. This could increase user-satisfaction towards the software
products as well as decrease the long-term maintenance expense. In the
theory of testing we have an assumption by the name of oracle. It is a
mechanism against which one can determine whether the outcome of a
program on test case is correct or not. But there are some situations where the
availability of oracle is either not there or it is excessively expensive for
application. For instance assuring the quality of applications such as those in
the fields of scientific calculations, simulations, optimizations, data mining, etc.
presents a challenge because conventional software testing processes do not
always apply: in particular, it is difficult to detect subtle errors, faults, defects
or anomalies in many applications in these domains because there is no
reliable “test oracle” to indicate what the correct output should be for arbitrary
input. The general class of software systems with no reliable test oracle
available is sometimes known as “non-testable programs” so this oracle
problem give rise to a new form of testing that is metamorphic testing.
Moreover, though manual forecasting and comparing of test result is the
possibility but it certainly needs more time and chances of error are
exceedingly high.

Another crucial aspect is the utilization of the successful test cases in a

pragmatic way. The maximum numbers of test cases are successful cases and
seldom expose any failure. In traditional testing these test cases are supposed
not of much use and as a result not taken course of or just kept aside for any
regression testing at some other point of time. In such a case a worth full

Page
1
information as available in those successful test cases goes unused which
could otherwise be of great advantage. A Metamorphic Testing method has
been proposed by Chen et al. [1] to employ successful test cases and better
the oracle problem. This method has been proposed to test programs without
the involvement of an oracle. It employs properties of target function known as
metamorphic relations to generate follow up test cases and verify the outputs
automatically. Such relations are expected relations between the inputs and
outputs of the multiple target program execution.

Nevertheless the verification of Metamorphic relations for majority of the test

inputs is a compulsion regarding the correctness of a program but still not
sufficient in itself. Metamorphic testing makes the check of relations among
most of the executions instead of making a check about the correctness of
individual outputs. The construction of follow up test cases and verification of
the test results need not to have an oracle or manual estimates. As a result
metamorphic testing can be kept wholly automated. It is taken use of in
combination with other test case selection strategies such as both black and
white testing strategies. To supplement this testing, the mutation testing
comes handy. It is a technique that measures the effectiveness of the test
cases and it is common fault based testing technique.

Page
2
 Chapter 2

Objectives

My work will have following objectives:-

1. Conducting Automated Metamorphic testing within the context of the

application.

2.Design Of effective Metamorphic Relations

3. Verification of results.

Page
3
 Chapter 3

Metamorphic Testing

A Metamorphic Testing method has been proposed by Chen et al. to employ

successful test cases and better the oracle problem. Metamorphic testing is a
strategy which is property based. This method has been proposed to test
programs without the involvement of an oracle. It employs properties of target
function known as Metamorphic Relations to generate follow up test cases and
verify the outputs automatically.

It also laments that even if original test case does not provide us any failure it
has some information hidden there in it or we can say that we can generate
follow up test cases from it to verify the some important relations of the target
function in our software. These relations are called Metamorphic Relations.

3.1 Metamorphic Testing- an overview

Metamorphic testing is a technique to generate follow up test cases based on

the existing test cases that have not revealed any failure and are known as
successful test cases. Based on the successful test cases follow up test cases
can be generated by making reference to metamorphic relations, that is,
relations among multiple execution of target function. It was proposed as a
way to address oracle problem. In this technique the main idea is to generate
new test cases based on existing test cases. These existing test cases are in
majority successful test cases. These successful test cases are considered
useless in conventional testing.

Suppose we have

• Program-P

• Function implemented in program P:- f

• Domain input-D

Page
4
 • Test case selection strategy-S

• Let T={t1,t2,t3…tn} subset of D

If outputs p(t1), p(t2),…p(tn) reveal no failure, then we encounter successful

test cases. At this stage MT is employed. By referring to certain properties
metamorphic relations (MR) of target function f, follow up test cases are
generated, executed and checked to further verify program p without the need
of an oracle.

3.2 Metamorphic Testing v/s Other testing techniques

• Metamorphic properties are identity relations, that can be expressed in

other forms also such as in equalities. No such inequalities are covered in
other techniques.

• Another crucial aspect is the utilization of the successful test cases in a

pragmatic way. The maximum numbers of test cases are successful
cases and seldom expose any failure. In traditional testing these test
cases are supposed not of much use and as a result not taken course of
or just kept aside for any regression testing at some other point of time.
In such a case a worth full information as available in those successful
test cases goes unused which could otherwise be of great advantage. A
Metamorphic Testing method employs successful test cases and better
the oracle problem.

• Testers aim at providing a probabilistic oracle, in order to estimate

whether the corresponding output is likely to be correct. Whereas
Metamorphic Testing is based on property based test case selection
strategy rather than providing some form of alternative oracle.

• In traditional testing oracle matches calculated output with the desired

output with human involvement also. but in metamorphic testing many
properties of the target function are checked and that too in multiplicity
and without human involvement .

Page
5
3.3 Application areas of Metamorphic Testing

• Metamorphic Testing is a pragmatic way of testing to test the problems

of partial differential equations, statistics, graph theory,
computer graphics, compilers etc.

• Properties of machine learning applications to define transformation

function on input, such that new output is generated on original output.

• Metamorphic testing is used to test a range of bioinformatics

programs specially those for large scale simulation and data analysis.

• Online service oriented applications are also tested with the help of
metamorphic testing. It makes use of set of successful test cases
generated by offline testing to construct their corresponding follow up
test cases for online testing. These test cases are executed by
metamorphic services that encapsulates the services under test as well
as the implementation of metamorphic relations

• Integration testing of the functions of the system in context sensitive

Middle ware-Based applications can be done conveniently with
metamorphic testing by placing check points as the starting and
endpoints of the test case.

• Metamorphic testing can be effectively used in testing of decision

support systems. These systems are increasingly complex and, as a
result, they are very difficult to test because of the lack of an ideal test
oracle. Lack of testing may result in poor software quality and wrong
decisions being made and, hence, bring serious consequences.

• Metamorphic testing can be used to test Queuing Network model.

Queuing network modeling is a modeling technique for capacity planning
studies of computer and communication systems. Due to complexity of
the technique, it is very difficult to know from the computed outputs
whether the computation of the modeling software is correct. It is

Page
6
 necessary to have an effective testing technique to address this
problem.

• Metamorphic testing approach can be used for automated test data

generation on the analyses of Feature Models. A feature model is a
compact representation of all the products of a software product line.

3.4 Benefits of Metamorphic Testing

• Metamorphic testing test programs without the involvement of an oracle.

It employs properties of target function known as Metamorphic Relations
to generate follow up test cases and verify the outputs automatically.

• It can be fully automated as no human involvement is required to

generate follow-up test cases and verify the results.

• Metamorphic relations are not limited to identity relations. It comprise of

inequalities, sub assumption relations and convergence properties also.

• It helps in debugging as it can be used for program testing, proving and

debugging.

• A Metamorphic Testing method employs successful test cases and better

the oracle problem. Follow-up test cases are generated. Then the results
of successful test cases and follow up test cases are verified to check the
correctness of the program under test. By this we have effective
utilization of successful test cases

• Metamorphic relations are not limited to identity relations. It includes

inequalities, sub assumption relations and convergence properties also.

3.5 Metamorphic Relations

Some necessary conditions of target function of our software are needed to be

checked. These mandatory conditions are called metamorphic relations.

Page
7
‘Metamorphic relation is any relation among the inputs and the
outcomes of multiple executions of the target program’

Let we have cosine function (cos(x)). Suppose at x=42 we get -0.3999. A

property of cosine function is cos(x)= cos(-x). Using this property as a
Metamorphic relation, we can design new test case with x=-42. We get the
output for this input ie 0.4235. While comparing both outputs, in case of
disparity we conclude that the program is not correct.

MR’s are not confined to identity relations only. Any expected relation
containing inputs and outputs of two or more executions of the program can be
picked as MR.

Definition: - Let we have a program p in which function f is implemented. We

have n distinct inputs x1,x2,x3…xn and we get corresponding n computation
outputs f(x1),f(x2),f(x3)…f(xn). Relation R is a relation over these inputs which
must induce necessary property rf for multiple executions of target software f.
this relation must be satisfied when we replace f by p; otherwise P will not be
correct implementation of f.

Selection of useful Metamorphic Relations

All metamorphic relations are not useful to us and selection of good MR’s is a
difficult task. Useful Metamorphic relations are those which can make multiple
executions of a program as different as possible. Importance of structure of the
program under test should be main part for selection strategy to emphasize
on. It is mandatory to understand the algorithm so as to select good
metamorphic relations. The reason being the algorithms are easier to
understand then source code.

Some key points in selecting MR’s

• Equalities- metamorphic relations that represent equality are weak.

They are unable to detect the variations if the result is multiplied by a
factor or if a constant is added to result. For e.g. det(A)=det(A’)

Page
8
 • Equalities of Linear Combinations- If the metamorphic relation is an
equation with linear combinations on each side (with at least two terms
on one side), there is vulnerability against erroneous multiplications.

• Semantically rich relations- Good metamorphic relations must be

semantically rich as much as they can. For e.g.
k.gcd(a,b)=gcd(k.a,k.b),

gcd(a,b).gcd(c,d)=gcd(ac,bd)

• Close to common Implementations/Algorithms –original test cases

and follow up test cases are basically same in case where metamorphic
relations matches to the strategy used by typical implementation or
algorithms and hence there suitability is also limited.

3.6 Procedure of Metamorphic Testing

• At the very first instance testers make an identification and also make
MR’s ie MR1, MR2..MRn. It is done from the target function f. For this
purpose testers construct a function generate so that follow up test
cases may be generated from the initial test case.

• Next step involves constructing of the function ‘veri’ for each MRi, so
that it can be verified that weather multiple inputs and the
corresponding outputs satisfy MRi.

• Next step involves testers generating a set of initial test cases T in

accordance with a priority test case selection strategy.

• The final stage, the test drivers takes the course of function ‘geni’ for
every test case T so as to generate follow up test cases and thus
applying the function veri so as to test the satisfaction criteria about the
metamorphic relation MRi. In case a Metamorphic Relation MRi is not
followed by any test case ‘veri’ reports that an error is detected in the
software considered under test.

Page
9
 Chapter 4

Literature Survey

To verify the correctness of the program in those situations where the oracle is
not available, Metamorphic Testing serves as a useful technique. It takes the
channel of successful test cases and yielding the follow up test cases along
with the metamorphic relations.

Chen et al. [1] took the use of metamorphic testing to solve the partial
differential equations to different programs. This results into removing the
oracle problem while testing numerical component. From it the effectiveness of
metamorphic testing, by testing of a program that solves an elliptic partial
differential equation with boundary conditions, is well demonstrated. It is also
useful to find out the subtle errors of the kind which may not come forth by
taking special test cases.

Furthermore, Chen and colleagues [2] brought forward a semi proving method
which is capable of verifying expected necessary conditions in regard to the
correctness of the program. Symbolic evaluation and MT techniques are
integrated to envisage this method. Both structural (white box) information
when performing global symbolic evaluation and functional (black box)
information of the problem domain are taken course of when metamorphic
relations are to be identified. This combination of black and white box
information helps in better tackling of subtle errors in white box testing such as
missing path has. From the viewpoint of testing this integration helps to
mitigate the problem that software testing does not assure the absence of
faults. Target properties can be verified by using appropriate sets of inputs
which are identified through MR. From the viewpoint of program proving it also
presents the procedure to prove selected pertinent conditions of program
correctness and it also helps in identification of failure causing inputs if there
are any.

Page
10
An overview of the basic concepts of metamorphic testing, some interesting
extensions in the domains of program testing, proving and debugging is
provided by Chen et al. [3]. Even in the absence of oracle the metamorphic
relations can be tested. A many other critical issues about fault detection
effectiveness of metamorphic testing are highlighted. By giving the symbolic
inputs for the verification of the program, metamorphic testing serves as a
supportive semi proving method. Besides program verification semi proving
technique works as a support in debugging by generating limit expressions in
regard to those inputs which cause failure.

In further research a description of metamorphic testing involves

implementation of shortest and critical path algorithm. This article[4] conducts
a case study with the use of mutants of two programs in graph theory where
oracle problem does not exists.

The following are the results obtained-

(a) theoretical knowledge of the problem domain is not sufficient for making
a distinction of good MR’s, and

(b) We consider those Mr’s good those can help in the multiple undertaking of
the program as varied as possible. MR selection strategy stresses upon the
pertinence of the structure of the program under consideration. Nevertheless it
is not important for the testers to wholly understand the program code before
putting it to test. Algorithm that the program follows should be the main key
area to decide good MR’s because algorithms are easier to understand then
the source code It is important to mention here that

a. MT is a technique which helps to generate follow up test cases,

Putting simply, MT is not enough for assuring quality. Other test
case selection strategies are to be taken use of to assure quality.
b. It has been shown by the experiments that different MR’s are
capable in finding out failures in regard to defects in different
program.

Page
11
It has been further demonstrated by the Author [5] that the use of
metamorphic testing is a compliment to special value testing. It can bring out
faults where special value testing finds itself incapable to do so. Examples
presented in this paper lack test oracles so the route of special value testing is
taken. Special value testing has a short coming that by its use one cannot
determine its reliability. Metamorphic testing does away with this issue
through testing of metamorphic relationships and enhances the number of test
cases with the help of random test values in its set of test data.

A metamorphic approach to integration testing of context sensitive middleware

based applications is brought forward by Chan and colleagues.[6] In context
sensitive middle ware based software, middle ware finds out a solution and
makes use the appropriate function of the application to be tested. As the
middleware remains active and the situation continues to make itself better
still the conclusions of some test cases are difficult to reach at. This paper
purposes to take the course of check points as the starting and closing points
of a test case. As the middle ware by itself will not activate any function in the
course to a check point but it may start actions in between to such situations.
A convenient environment is offered by this concept for the conduction and
integration testing the functions of a system.

To check a class of metamorphic relations A.Gotlieb and B.Botella [7] brought

forth an automated framework. The basis of usual techniques for automatic
test data generation is the assumption that we should have a complete oracle
when testing a process. This paper brings forth a framework for automatic test
data generation even without the presence of oracle during testing process. It
is capable of checking so called metamorphic relations. The framework takes
the course of constraint logic programming techniques to find test data that is
feared to be violating metamorphic relations in question. It is also find in some
circumstances that the program are present which satisfies this relation.

An approach is presented by W.K.Chan and colleagues[9] to test online service

oriented applications. It involves formulation of metamorphic service that
contains the characteristic of an access wrapper and as an agent for

Page
12
conducting metamorphic testing. It contain services under test along side the
implementation of metamorphic relations. This approach involves making the
use of a successful test case for offline testing as if they are the original test
case for online testing.

A broad range of applications of metamorphic testing for instance numeric,

computer graphics, graph theory, interactive software and compilers are
brought forward in the continued work by Chan and colleagues [10]. It can
work in combination with any test case selection strategy for practitioners to
further make use of these successful test cases MT has proved to be useful. It
is important to have sufficient knowledge of the problem domain so that Mt
can be applied effectively. The reason being the Mr’s are identified in
accordance with the specification. Since it is automated the follow up test
cases can be tested and verified independent of human involvement.

It is demonstrated by Mayer and Guderlei [11] that how to use metamorphic

testing to automate the testing of image processing application. For doing so
different models are used to randomly generate test data metamorphic
relations necessary properties and important values are identified. These can
be of use in the evaluation of test and generation of follow up test cases. A
comparison is made of models using mutant analysis.

Furthermore, an empirical study on Metamorphic Testing is presented by

Mayer and Guderlei[12]. It is an approach to oracle problem. In order to
evaluate the usefulness of MT approach this study was undertaken with
common JAVA implementation of determinant computations. It also establishes
a general criterion that can be of use to fastly access metamorphic relations
about their suitability. MT has its basis on so called Metamorphic relations on
input output tuples which can be easily detected. It is still crucial to evaluate
these relations in accordance to their usefulness. The empirical study makes
us capable to derive general rules that may be useful to instantly access
Metamorphic Relations and make an identification of those that needs to be
taken into consideration and requires more detailed study by using the other
methods.

Page
13
Sami Beydeda[13] demonstrated an approach to improve the program with
self-testability. It will decrease the amount of information which is to be
thoroughly tested be human tester and it can also simplify the testing process.
An approach to self-testability in which test case generation and test
evaluation is there is presented in this paper. This approach proposed is an
integration of self-testing COTS components (STECC) method and metamorphic
testing approach.

Another method is suggested by Mayer and Guderlei [14]. It involves an

integration of statistical hypotheses test and metamorphic testing. It is
applicable to the randomised software. Metamorphic testing is helpful in
constructing statistical hypothesis test without knowing exact theoretical
characteristic or in the absence of reference implementation. To meet that
purpose two or more independent output sequences are produced with the
implementation under test (IUT). After this, a comparison is made of these
sequences according to the metamorphic relations using statistical hypothesis
test.

An advantage is taken by Murphy et al. [15] of metamorphic testing approach

to test the machine learning application, for which correct answers are not
available. In metamorphic testing properties of the application are
advantageously used for defining transformation functions on input, in such a
way that new output can be predicted based on original outputs. It has another
crucial characteristic that can be implemented in both supervised machine
learning and unsupervised machine learning application.

Another approach is proposed by Murphy et al. [16] for testing software in the
absence of test oracles that uses metamorphic testing, run time assertion
checking and the JML specification language in conjunction. Metamorphic
properties are satisfied by JML (a behavioural interface language). For doing so
an implementation framework comes handy to convert the specification of
metamorphic properties into test methods. It can be then executed by making
the use of assertion checking. Furthermore it can also be made use of the
testing of applications for which a test oracle is present for creating new test

Page
14
cases (based on previous test cases) for regression testing and program
evolution in the coming future.

Page
15
 Chapter 5

Proposed Framework

Metamorphic Testing serves as a useful technique of testing to eliminate the

oracle problem. A Metamorphic Testing method has been proposed by Chen et
al. [1] to employ successful test cases and better the oracle problem. This
method has been proposed to test programs without the involvement of an
oracle. By taking use of the valuable information in successful test cases, it
generates the follow up test cases to verify the results. In the very first step I
started with the study of metamorphic testing, its applications and
metamorphic relations. Now I will put emphasis on creation of metamorphic
relations. All the metamorphic relations are not good. In metamorphic testing
we need only those relations which yield maximum number of failures. In next
step I will be concentrating on the choice of effective metamorphic relations so
that more errors can be extracted from the system.

Page
16
References

[1] T.Y. Chen, J. Feng and T.H. Tse, “Metamorphic testing of programs on
partial differential equations: a case study”. In Proceedings of the 26th Annual
International Computer Software and Applications Conference (COMPSAC
2002), IEEE Computer Society Press, pp.327-333, 2002.

[2] T.Y. Chen, T.H. Tse and Z.Q. Zhou, “Semi-proving: An Integrated Method
Based on Global Symbolic Evaluation and Metamorphic Testing”. In
Proceedings of the ACM SIGSOFT International Symposium on Software Testing
and Analysis (ISSTA 2002) ACM Press, New York, pp. 191-195, 2002.

[3] T. Y. Chen, F.-C. Kuo, T. H. Tse and Z. Q. Zhou,“ Metamorphic Testing and
Beyond”. In Proceedings of the Eleventh Annual International Workshop on
Software Technology and Engineering Practice (STEP’04) , IEEE Computer
Society Press, pp. 94–100, 2004.

[4] T. Y. Chen, D.H. Huang, T. H. Tse and Z. Q. Zhou, “Case study on the
selection of useful relations in metamorphic testing”. In Proceedings of the 4th
Ibero-American Syposium on Software Engineering and Knowledge
Engineering(JIsic 2004), Polytechnic University of Madrid, Spain , pp.569-583,
2004.

[5] T. Y. Chen, F.-C. Kuo., Y. Liu, and A. Tang, “Metamorphic testing and
testing with special values”. In Proceedings of the Fifth International
Conference on Software Engineering, Artificial Intelligence, Networking, and
Parallel /Distributed Computing (SNPD 2004), International Association for
Computer and Information Science, Mt. Pleasant, Michigan, 2004.

[6] W.K. Chan, T.Y. Chen, H. Lu, T.H. Tse and S.S. Yau, “A Metamorphic
Approach to Integration Testing of Context-Sensitive Middleware-Based
Applications”. In Proceedings of the 5th Annual International Conference on

Page
17
Quality Software (QSIC 2005), IEEE Computer Society, Los Alamitos, California,
pp.470-476, 2005.

[7] A. Gotlieb and B. Botella,“Automated Metamorphic Testing”. In

Proceedings of the 27th Annual International Computer Software and
Applications Conference (COMPSAC 2003), IEEE Computer Society Press, Los
Alamitos, California, pp. 34–40, 2003.

[8] W.K. Chan, S.C. Cheung, and K. R. P. H. Leung, “Towards a Metamorphic

Testing Methodology for Service-Oriented Software Applications”, The First
International Workshop on Services Engineering (SEIW 2005), In Proceedings of
the Fifth Annual International Conference on Quality Software (QSIC 2005),
IEEE Computer Society, Los Alamitos, California, pp.470-476, 2005.

[9] W. K. Chan, S. C. Cheung, and K. R. P. H. Leung, “A Metamorphic Testing

Approach For Online Testing of Service-Oriented Software Applications”.
International Journal of Web Services Research, pp.60–80, 2007.

[10] Z.Q.Zhou, D.H.Huang, T.H.Tse, Z.Yang, H.Huang and T.Y.Chen,

“Metamorphic testing and its applications”. In Proceedings of 8th International
Symposium on Future Software Technology (ISFST’04), Software Engineers
Association, Japan (2004).

[11] Johannes Mayer, Ralph Guderlei, “On Random Testing of Image

Processing Applications”. In Proceedings of Sixth International Conference on
Quality Software(QSIC’06), IEEE Computer Society Press, pp.85-92, 2006.

[12] Johannes Mayer, Ralph Guderlei, “An Empirical Study on the Selection of
Good Metamorphic Relations”. In Proceedings of the 30 th Annual International
Computer Software and Application Conference (COMPSAC’06), IEEE Computer
Society Press, pp.475-484, 2006.

[13] S.Beydeda, “Self Metamorphic-Testing Components”. In Proceedings of

the 30th Annual International Computer Science and Applications Conference
(COMPSAC’06), IEEE Computer Press, pp. 264–272, 2006.

Page
18
[14] R.Guderlei and J.Mayer, “Statistical metamorphic testing- testing
programs with random output by means of statistical hypothesis tests and
metamorphic testing.” In Proceeding of the Seventh International Conference
on Quality Software (QSIC’07), IEEE Computer Society Press, pp. 404-409,
2007.
[15] C.Murphy, G. Kaiser, L. Hu and L. Wu., “Properties of Machine Learning
Applications for Use in Metamorphic Testing”. In Proceeding of the 20th
International Conference on Software Engineering and Knowledge Engineering
(SEKE), pp. 867–872, 2008.

[16] C.Murphy, K. Shen, and G. Kaiser, “Using JML Runtime Assertion Checking
to Automate Metamorphic Testing in Applications Without Test Oracles”. In
Proceeding of the Second IEEE International Conference on Software Testing,
Verification and Validation(ICST), pp.436-445, 2009.

Page
19