See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/228981071

A tool for automated test data generation (and execution) based on

combinatorial approach

Article  in  International Journal of Software Engineering and its Applications · August 2007

CITATIONS READS

23 195

7 authors, including:

Kamal Z Zamli Nor Ashidi Mat Isa

Universiti Malaysia Pahang Universiti Sains Malaysia
177 PUBLICATIONS   1,972 CITATIONS    266 PUBLICATIONS   3,592 CITATIONS   

SEE PROFILE SEE PROFILE

Siti Norbaya Azizan

Sunway University
33 PUBLICATIONS   216 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Masters Project (Design and Evaluation of Pocket Education System): An SMS-Based Learning for Distance Learners View project

Computational Search Techniques for Solving Optimization Problems View project

All content following this page was uploaded by Kamal Z Zamli on 04 March 2015.

The user has requested enhancement of the downloaded file.

 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

A Tool for Automated Test Data Generation (and Execution) Based

on Combinatorial Approach

Kamal Zuhairi Zamli, Nor Ashidi Mat Isa, Mohamed Fadel Jamil Klaib and
Siti Norbaya Azizan
School of Electrical and Electronic Engineering
Universiti Sains Malaysia, Engineering Campus,
14300 Nibong Tebal, Penang, Malaysia
eekamal@eng.usm.my

Abstract
Covering as much as 40 to 50 percent of the development costs and resources, software
testing is an integral part of the software development lifecycle. Despite its importance,
current software testing practice lacks automation and is still primarily based on highly
manual processes from the generation of test cases (i.e. from the specifications documents) up
to the actual execution of the test. These manually generated tests are sometimes executed
using ad hoc approach, typically requiring the construction of a test driver for the particular
application under test. The construction of a test driver is tedious, error prone, and
cumbersome process, as it puts extra burden to test engineers. In addition, test engineers are
also under pressure to test increasing lines of code in order to meet market demands and
deadlines for more software functionalities. To attain the required level of quality and
maintain acceptable test coverage, test engineers need to consider a significantly large
number of test cases. Many combinations of possible input parameters, hardware/software
environments, and system conditions need to be tested and verified against for conformance
based on the system’s specification. Often, this results into combinatorial explosion of test
cases. Addressing some of the aforementioned issues, this paper describes a new automated
and combinatorial software testing tool, called JTst, based on the use of Java technology.
The main novel features of JTst are the fact that it permits combinatorial test data generation
and automated concurrent execution, enabling higher product quality at lower testing costs.

1. Introduction
Software testing relates to the process of executing a program or system with the intent of
finding errors [17]. Although an integral part of software development lifecycle (i.e. covering
as much as 40 to 50 percent of the development costs and resources [5]), current software
testing practice is still primarily based on highly manual processes from the generation of test
cases (i.e. from the specifications documents) up to the actual execution of the test. These
manually generated tests are sometimes executed using ad hoc approach, typically requiring
the construction of a test driver for the particular application under test. The construction of a
test driver is tedious, error prone, and cumbersome process, as it puts extra burden to test
engineers.
In addition, test engineers are also under pressure to test increasing lines of code in order to
meet market demands and deadlines for more software functionalities. To attain the required
level of quality and maintain acceptable test coverage, test engineers need to consider a
significantly large number of test cases. Many combinations of possible input parameters,
hardware/software environments, and system conditions need to be tested and verified against

19
International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

for conformance based on the system’s specification. Often, this results into combinatorial
explosion of test cases.
As an illustration, consider the option dialog in Microsoft Excel software (see Figure 1 in
the next page). Even if only View tab option is considered, there are already 20 possible
configurations to be tested. With the exception of Gridlines color which takes 56 possible
values, each configuration can take two values (i.e. checked or unchecked). Here, there are
220x56 (i.e. 58,720,256) combinations of test cases to be evaluated. If the time required for
one test case is 5 minutes, then it would require nearly 559 years to complete all tests merely
for View tab option. Given the limited time and resources, the research question now is what
are the smaller optimum sets of test cases to be used for testing?

Figure 1. Option Dialog in Microsoft Excel

Addressing some of the aforementioned issues, this paper describes a new automated and
combinatorial software testing tool, called JTst, based on the use of Java technology. The
main novel features of JTst are the fact that it permits combinatorial test data generation and
automated concurrent execution, enabling higher product quality at lower testing costs.

2. Introducing JTst
Our work is developed based on our earlier tool, called SFIT [1-3][19-21]. Despite being
useful, SFIT is too rudimentary and primitive with limited automation features. Building and
complementing from these experiences, we have opted to develop JTst [22].
A key idea in JTst is the fact that tests are performed based on the values of the interface
parameters (i.e. on the data types of the parameter lists) and not on the behavioral
specifications. Thus, JTst is suitable for performing automated black box testing particularly
involving commercial-of-the-shelves-components (COTs) where no source code and design
are usually available apart from some user documentations.

20
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

Failure
E m pirical D ata
JTst
C om binatorial Test
C ase G enerations

A utom atic
Test
E xecution
R equirem ent B ase Test
S pecifications C ases

Test O utcom e
P ass Application
U nder Test
Fail X
C onform ance
A nalysis

Figure 2. JTst Philosophy

in te ra c t
C u s to m e rs & R e q u ire m e n t
S ta k e h o ld e rs E n g in e e rs
s o lic it p ro d u c e
K n o w n F a ilu re
R E Q U IR E M E N T E m p iric a l D a ta
E L IC IT A T IO N
SOFTWARE ENGINEERING PRODUCT LIFECYCLE

PHASE
S o ftw a re in te ra c t
re fe r S p e c ific a tio n
re fe r

In d e p e n d e n t
S o ftw a re
IM P L E M E N T A T IO N V& V Team s
E n g in e e rs &
P ro g ra m m e rs PHASE
S o ftw a re
D e s ig n V A L ID A T IO N
d e s ig n PHASE

d e v e lo p a n a ly z e fo r d e v e lo p
c o n fo rm a n c e

S o ftw a re Test
T e s t c a s e e x e c u tio n
Im p le m e n ta tio n C ases

S o ftw a re
R e le a s e

Figure 3. Applying JTst within the Software Engineering Product Lifecycle

Another key idea in JTst is that the test cases can be combinatorially generated based on
some base test cases, the concept borrowed from Ammann and Offutt [4]. Ideally, these base
test cases can either be collected from known combination of input variables that causes
failures to the module under test from real systems in various application domains [14] or
from the program specifications (see Figure 2 and 3 given earlier). As far as JTst is
concerned, the base test cases can be both actual test values to be used for test execution or
some symbolic data (e.g. On, Off, Left, Right etc). It should be noted that symbolic

21
International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

combinatorial data values can be used to systematically guide acceptance tests [7] based on
some user configuration.
Kuhn and Okum [15] suggest that from empirical observation, the number of input
variables involved in software failures is relatively small (i.e. in the order of 3 to 6), in some
classes of software. If t or fewer variables are known to cause fault, test cases can be
generated on some t-way combinations (i.e. resulting into a smaller set of test cases for
consideration). Empirical evidence suggests that in some software implementation, the
execution of these smaller subset of test cases can typically uncover 50% to 75% of faults in a
program [10][16][18].
Concerning its implementation, JTst consists of a number of related components consisting
of the Class Inspector, Test Editor, Test Combinator, Automated Loader and Data Logger
(see Figure 4). The working context (i.e. process flow and the main user interface) for these
components are captured in the JTst main interface.
Test
E n g in e e r

in itia te s a n a ly z e s
JTst in itia te s o b s e rv e s
o u tc o m e

B ase Test C la s s D a ta
Test C ases s e ts E d ito r In s p e c to r Logger
q u e rie s c la s s
lo a d s in te rfa c e s a n d
Test a ttrib u te s
C o m b in a to r M o d u le U n d e r
Test
p e rm u ta te s
Autoutoto mate atete d
AAA
uuto mmm aate ddd lo a d s te s t c a s e s
Loader
LLoLoaoadadederer r
lo a d s lo a d s
C o m b in a to ria l
T est C ases
m a in ta in s

Log

Figure 4. JTst Main Components

Figure 5. JTst Main Interface

22
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

The functionalities for each of these components will be discussed next.

Class Inspector
In the absence of source codes, the class inspector can optionally be used to obtain details
information of the Java class interface (see Figure 6). To do so, the class inspector exploits
Java Reflection API in order interrogate Java classes for method interfaces including public,
private, and protected ones. This information can be used to set up base test cases.

Figure 6. JTst Class Inspector

Test Editor
As its name suggests, the test editor allows the user to edit and setup the test cases (i.e.
including the base test cases) in a JTst fault file. Here, the test case definitions (up to 15,000
test cases) can be straightforwardly described using JTst predefined markup language in order
to facilitate the parsing of test case data values for automatic recombination (see Figure 7). As
noted earlier, to allow execution, test data must use actual values and not symbolic data.

@FaultFile

/////////////////////////////////////////
Common Header Definition
/////////////////////////////////////////
classname : adder
methodname : add_basictypes_integer
specifier: private
paramtypes : 2
returntype: int
parameter : partypes[0]=Integer.TYPE
parameter : partypes[1]=Integer.TYPE

23
International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

/////////////////////////////////////////
Body - Test case 0
/////////////////////////////////////////
arglist:arglist[0]=new Integer(Integer.MAX_VALUE)
arglist:arglist[1]=new Integer(Integer.MAX_VALUE)

/////////////////////////////////////////
Body - Test case 1
/////////////////////////////////////////
arglist:arglist[0]=new Integer(Integer.MIN_VALUE)
arglist : arglist[1]=new Integer(Integer.MIN_VALUE)

……………

Figure 7. Sample Fault File

Test Combinator
Test combinator manipulates the base test case in order to generate combinatorial test cases
(see Figure 8). The combinator interface can be seen in Figure 9.

BASE JTst COMBINATORIAL

TEST CASES Test Combinator TEST CASES
Test case 1:
Test case 1: on n a,b,c ...
ti io
a,b,c ... u ta u ta t
rm rm Test case 2:
pe pe n
ta ti o
Test case 2: p e rm
u a,b,f ...
d,e,f ... per mu tation Test case 3:
Test case 3: p e rm a,b,i ...
u ta ti
on
g,h,i ... pe Test case 4:
... rm
pe

u ta a,e,c ...
rm

tio
...

n
ut
a

Test case N: Test case 4:

tio
n

... a,e,f ...

Test case 4:
a,e,i ...
...
Figure 8. Test Combinator

Figure 9. JTst Combinator Interface

24
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

To illustrate the functionality of the JTst test combinator, consider the following running
example. Let a system or a method M1 has four possible inputs variables X = {A,B,C,D}. For
simplicity sake, let us assume that the base test cases for M1 have been identified in Table 1.

Table 1. Base Data Values for Method M1

The test cases data can be viewed as a matrix with specified columns and rows. Here, one
can traverse one column at a time (called sensitivity variable in JTst implementation), whilst
keeping other column fixed to permutate and generate new test cases from existing ones. For
instance, Table 2 depicts the possible combinatorial test cases with the sensitivity variable set
to A.

Table 2. Base and Combinatorial Data Values for Method M1 With Sensitivity variable = A

Apart from permitting sensitivity column to be a single column, JTst test combinator also
allows the sensitivity column to be a combination of 2 or more columns (t-way combinations).
Here, the algorithm employed by JTst test combinator is based on the modified greedy
algorithm as discussed in [6][8][18]. In the case of t-way combinations, JTst test combinator
ensures that for each t-way definition of input variables, at least one test case will be
randomly generated to cover the required t-way combinations. Although not discussed in this
paper, it should be noted here that because the generation of t-way combinations of test cases
is random, it is highly unlikely that the same sets of test cases are generated even during
multiple t-way combinations exercise using the same fault file.
To illustrate the greedy algorithm implemented as part of JTst test combinator, Table 3
depicts the possible combinatorial test cases generated with sensitivity variable set to triplets
(ACD) as compared to Table 4 with sensitivity variable set to all variables. Obviously, less
number of test cases can be seen in the case when t-way combinations are used (see Table 3
and Table 4 for comparison). In fact, the number of test cases would be even smaller given
the smaller t-way combination (e.g. AB).

Automated Loader
JTst automated loader have two main responsibilities. The first responsibility is to
iteratively parse the test cases (defined in JTst fault files), and automatically generates and

25
International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

executes the appropriate Java code driver. The second responsibility is to manage concurrent
execution of test cases. Here, the JTst automated loader is actually consists of two sub-
components: Loader and Concurrent Manager (see Figure 10).

Table 3. Base and Combinatorial Data Values for Method M1 With Sensitivity variable = t-
way (ACD)

Table 4. Base and Combinatorial Data Values for Method M1 With Sensitivity variable = All
(ABCD)

Concurrent execution is achieved in JTst through a well-known token passing algorithm.

Here, a token is always associated for each concurrent execution. Once all the tokens have
been used up, no further concurrent execution is allowed until one or more concurrent
executions have terminated (i.e. release its token). Here, the number of defined tokens in the
pool of tokens can be dynamically configured through the user interface provided should the
need arise (see Figure 11). Obviously, the more tokens are allowed, the slower the test case
executions will be. In the current version, JTst has been tested to execute up to 10 concurrent
execution of test cases, each of which can support up to 15,000 test cases per execution (see
Figure 12).

26
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

JTst Loader JTst

returns JTst Loader Automated
JTst Loader
token upon Loader
completion Loader

assigns token
checks-out token to the JTst loader
if it is available
JTst
Concurrent
Manager
Pool of
tokens regenerate tokens
upon request

Figure 10. JTst Automated Loader & Token Passing Mechanism

Figure 11. JTst Concurrent Setting for Token Generations

Figure 12. JTst Concurrent Execution of Test Cases

Data Logger/Log

27
International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

Data logger is a text browser equipped with customized search capability to perform
offline analysis of the output captured by the automated loader in the form of logs. Here, logs
are special database storing the input output behavior of the module under test (MUT). If the
specification of the MUT method exists, conformance analysis can be made using this
database.
Nevertheless, in the absence of source codes and formal specification, the trivial outcome
of “doesn’t hang and doesn’t crash” suffices to determine whether MUT passes the minimum
testing requirement. In this case, the operating system can be queried if the test program
terminates abnormally and a process monitor can be employed to detect hangs. A key issue
here is the fact that the faults can always be reproducible with the same sets of inputs

3. On JTst Case Study – Jada

In order to evaluate its features, there is obviously a need to subject JTst to a case study
problem. Here, Jada [9], a distributed shared memory implementation of Linda in Java, has
been chosen. The rationale for choosing Jada stemmed from the fact that it is a public domain
Java library freely accessible for download in the internet. Although an overview of its
methods and functionalities are given in the documentation (see reference [9]), Jada does not
come with complete source codes.
The focus of this case study is two folds. The first is to evaluate the applicability of JTst,
and the second is to perform robustness assessment of Jada.
In order to perform robustness assessment of Jada, there is a need to understand how Jada
works. Jada is actually the Java implementation of Linda. Linda is a parallel programming
model that was proposed by David Gelernter to solve the problem of programming parallel
machines [11]. Tuple space, essentially a distributed shared memory, is the Linda's
mechanism for creating and coordinating multiple execution threads. Tuple space stores
tuples, where a tuple is simply a sequence of typed fields.
The Linda model is often embedded in a computation language (such as C, Lisp, and Java)
and the result is a parallel programming language. The Linda model defines four operations
on tuple space:
• out(t); Causes tuple t to be added in the tuple space.
• in(s); Causes some tuple t that matches the template s to be withdrawn from the tuple
space. The values of the actuals of t are assigned to the formals of s and the executing process
continues. If no matching t is available when in(s) executes, the executing process suspends
until one is (i.e. blocking). If many matching t's are available, one is chosen arbitrarily.
• read(s); Its operation is the same as in(s); expect that the matching tuple is not
withdrawn from the tuple space.
• eval(t); Causes tuple t to be added to the tuple space but t is evaluated after rather
than before it enters the tuple space. A new process is created to perform the evaluation
As far as Jada is concerned, it implements most of the Linda operations including the non-
blocking version of out(t), in(s) and rd(s), although, the eval(t) has not been implemented.
Because no source code is available, it is impossible to know the exact class structure and
dependencies of Jada. Nevertheless, the Jada documentation highlights the following Jada
class hierarchy.

28
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

Figure 13. Jada Class Hierarchy

Although testing all the classes defined in Jada is equally beneficial (see Figure 13), the
Jada Object Space class that is the focus of this work. The rationale for such a focus is due to
the fact that it is the Object Space class that implements the Linda tuple space operations. It
should be noted that although given in the Jada documentation, the methods for manipulating
tuple space defined in the Object Space Class can also be discovered automatically using the
JTst Class Inspector.

29
International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

4. Running Experiment with Jada

16 experiments have been devised to evaluate the robustness of Jada primitives for
manipulating the tuple spaces. With the exception of Experiments 15 and 16 where 15,000
test cases were used (i.e. combinatorially generated), 40 base case values were identified
based on equivalent class and boundary value analysis. Using these base values,
combinatorial values were generated using sensitivity = all variables. It should be noted that
the total combinatorial test cases are not uniform in each experiment (i.e. depending on the
parameter values).

These experiments are summarized below:

• Experiment 1: public void out (Object item)
• Experiment 2: public void out (Object objects [ ], int n_objects)
• Experiment 3: public Object in (Object match)
• Experiment 4: public Object in (Object match [ ], int n_objects)
• Experiment 5: public Object in (Object match, long timeout)
• Experiment 6: public Object in (Object match [ ], int n_objects, long timeout)
• Experiment 7: public Object in_nb (Object match)
• Experiment 8: public Object in_nb (Object match [ ], int n_objects)
• Experiment 9: public Object read (Object match)
• Experiment 10: public Object read (Object match, long timeout)
• Experiment 11: public Object read (Object match [ ], int n_objects)
• Experiment 12: public Object read (Object match [ ], int n_objects, long timeout)
• Experiment 13: public Object read_nb (Object match)
• Experiment 14: public Object read_nb (Object match [ ], int n_objects)
• Experiment 15: public Object read (object match) with 15,000 test cases
• Experiment 16: public Object read (object match) with 5 concurrent test cases each
with 15,000 test cases

The complete description of these experiments is beyond the scope of this paper.
Interested readers are referred to our earlier work described in [2][21]. Nevertheless,
the summary of the experiments will be highlighted in the next section.

5. Lessons Learned
As discussed earlier, the main issues under consideration in this section relates to the
applicability JTst as well as on the robustness assessment of Jada.

Applicability of JTst
The fact that JTst can seamlessly test all the relevant Jada methods is a positive indication
of its applicability. The features offered by JTst appear sufficiently complete in order to
permit robustness testing of Jada.
Test automation is the key feature of JTst. As a comparison, Figure 14 depicts traditional
testing versus automated testing provided by JTst.

30
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

Start Testing Java Code

..the old way ..using JTst

Develop Test Cases

Must gain access to source code Gain access to either source code or class flies

JTst
Develop test cases for each method Automation
in a class file

Develop test cases in the fault file

Develop Java test drivers for each

test case in every class

Optionally use recombinator to

permutate test cases
Build the test drivers

Compile time
errors?

Run tests

Execute test cases defined in fault

Execute each test driver sequentially
file automatically in parallel

Runtime
errors?

Conformance analysis

Manually analyse results Analyse categorised log files

Java Code Tested

Figure 14. Traditional vs JTst Automated Testing

31
International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

As the figure illustrates, a number of saving in terms of test driver generations and
feedback loops can be observed. In this respect, JTst can be compared to JUnit [12]. In order
to use JUnit, programmers would still be required to write test drivers. In JTst, no
programming would be required as the test drivers are automatically generated by the JTst
automated loader. Such a feature is helpful to relieve the test engineers from the mundane
tasks inherent in the testing process.
The execution of the test cases can also be concurrent in JTst. In experiments 15 and 16,
we have managed to demonstrate this aforementioned feature. Obviously, this feature can
significantly alleviate the execution bottlenecks of test cases. Theoretically, JTst can support
unlimited number of concurrent executions of test cases. Nevertheless, our experience with
JTst indicates that allowing more than 10 concurrent executions can seriously slows down
overall execution of each concurrent process.
The test combinator is also another key feature of JTst. The test combinatior is useful to
generate t-way combinatorial test cases that can be used to locate unknown faults.
Combinatorial techniques are known to cause test data explosion [6]. Suppose that test input
variables are 10, each had 3 values say 0, 1, and 2. Then, there are 310 = 59,049 possible
parameter combinations. Now, if the test input variables are increased to 13, then there are 3
13
= 1,594,323 possible parameter combinations. This simple example illustrates that a small
change in the input parameters can cause massive increase in the parameter combinations.
Exhaustive testing for all combinations is desirable, of course, in the expense of costs. For
these reasons, partitioning into t-way combinations as implemented in JTst can offer trade-
offs between testing efforts, costs, and quality assurances. As discussed earlier, JTst ensures
that at least one test case will be generated to cover the required t-way combinations.
Comparatively, JTst can be seen as a complement to AETG [6] and IPO [18]. Similar to
AETG and IPO, t-way test cases can be combinatorially generated for analysis (i.e. both as
symbolic data and actual data values). Concerning combinatorial generation based on
symbolic data values, JTst, like AETG and IPO, can even be used to suggest systematic test
values for embedded system modules involving hardware, that is, through the t-way
combination features. To illustrate this point, consider a hardware product with 20 on/off
switches. To test all possible combination would require 220 test cases. If the time required
for one test case is 5 minutes, then it would take nearly 10 years for a complete test. As
discussed earlier, execution of a smaller set of test cases based on t-way combinations can
significantly reduces the test data space, in turn, reducing costs and efforts.
Unlike AETG and IPO, JTst also supports concurrent execution (and automation) of the
generated test cases for actual data values. As the work is still progressing, we are still
investigating ways to improve the modified greedy algorithm used in JTst. For this reason, no
performance measure is yet available.
The work context exemplified by JTst components and adopted as the JTst user interface
can also be helpful to put the testing process into perspective (see Figure 5). In this manner, a
subtle guidance can be given to the test engineers to assist them to understand how each piece
fits together into the whole picture.
The fact that JTst can interrogate a class for method interfaces (i.e. through the JTst class
inspector) is also useful when no source code for that class is available. In this way, unit
testing is also possible using JTst for classes (or even COTs) in the absence of source codes.

32
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

Robustness Assessment of Jada

Referring to all of the experiments undertaken, a number of observations can be made on
Object Space class of Jada. It seems that all the methods behave as expectation when the
classes such as String, Integer and Float are being used as the passing parameter. However,
when the Long, Double, and user defined class are used, most methods fails to respond
properly. For example, in experiment involving method public void out (Object item), the
method unexpectedly blocks when a Long, Double or a user defined class are used as the
passing parameter for the variable item. Similarly, in experiment 2 (involving the method
public out (Object [] item, int n_objects), the methods also blocks when a Long, Double or a
user defined class is used as the passing parameter for the array item. In fact, this observation
is true to other experiments as well.
Although defined as objects, the fact that only String, Integer and Float are supported as
the valid passing parameters for the object variable in all the methods of the Object Space
class raises an issue relating the usefulness of Jada. At a glance, it may appear that Jada
implementation might not be sufficiently extensive for manipulating distributed shared
memory. Nevertheless, a counter argument suggests that ad hoc approach may be adopted in
order to simulate the use Long, Double and user defined class as passing parameter, for
instance, by representing the required passing parameter (e.g. item) as String. One known
extension of Jada addressing this issue does exist, solving this problem by creating a form
tuple that can hold any object types [13]. In this manner, matching rule for that item in the
tuple space can also be simplified.
Testing Jada Object Space class with values more the allowable boundary also causes
problems in Jada. For instance, when manipulating extreme operation on Java predefined
Integer.MIN_VALUE, Integer.MAX_VALUE, Float.MAX_VALUE, and
Float.MIN_VALUE or any of their combination in the passing parameters, there is a tendency
for all the methods to hang (and never return). This can be seen, for example, in experiment 5
involving the method public object in (object item, long timeout). Here, when the variable
timeout is given a boundary value, the method unexpectedly hangs. Similar observation can
be seen in other methods, for example, involving experiment 14. Here, the method under
testing is given as public Object read_nb (Object match [], int n_objects). If the variable
n_objects uses boundary value (and out of range value) the method also unexpectedly hangs.
Another observation worth mentioning is on the Jada Object Space methods involving
array values as in experiment 8. In this case, the method is defined as public object in_nb
(Object [] match, int n_objects). From the Jada specification, the number of values defines in
the array of object match must ideally tally with the values of n_objects. However, our
observation indicates that Jada is flexible enough to accept any values of n_objects and yet
still gives the expected result provided that the values are within range (see the previous
paragraph). Again, similar observation can be seen in all other methods involving array
values.
Overall, while useful for manipulating distributed shared memory, Jada appears to be
unsuitable for highly available and safety critical applications. As seen above, Jada lacks
robustness, that is, it always fails to behave accordingly when unsupported or out of range
input values are used.

33
International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

6. Conclusion
In conclusion, this paper has described the design and implementation of an automated and
combinatorial Java unit testing tool called JTst. The prototype implementation tool has been
completed and used to perform robustness assessment of Linda implementation called Jada.
As part of our on-going work, we are currently implementing a parallel version of JTst to
support execution over heterogeneous distributed environment such as the GRID.

7. Acknowledgements
This research is partially funded by the eScience Fund – “Development of a Software Fault
Injection Tool to Ensure Dependability of Commercial-off-the-Shelf Components (COTs) for
Embedded System Applications” and the USM Short Term Grants – “Development of An
Automated Unit Testing Tool for Java Program”.

8. References

[1] M.I. Ahmad. “A Software Fault Injection Tool” , MSc ESDE Thesis, School of Electrical and Electronics,
University Sains Malaysia, May 2005.
[2] M.D. Alang Hassan. “Enhancing and Evaluating A Software Fault Injection Tool”, MSc ESDE Thesis, School
of Electrical and Electronics, University Sains Malaysia, December 2005.
[3] M.D. Alang Hassan, K.Z. Zamli, and N.A. Mat Isa. “SFIT: A Software Fault Injection Tool”. In Proc. of the
1st Malaysian Software Engineering Conference (MySec2005), Penang, December 2005. pp. 260-262.
[4] P.E. Ammann and A.J. Offutt. “Using Formal Methods to Derive Test Frames in Category-Partition Testing”.
In Proc. of the 9th Annual Conference on Computer Assurance (COMPASS’94), IEEE CS Press, June 1994, pp.
69-80.
[5] B. Beizer. Software Testing Technique. Thomson Computer Press, 2nd Edition, 1990.
[6] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton. “The AETG System: An Approach to Testing Based
on Combinatorial Design”. IEEE Transactions on Software Engineering 23(7), July 1997, pp. 437-444.
[7] M.B. Cohen, J. Snyder, and G. Rothermel. “Testing Across Configurations: Implications for Combinatorial
Testing” In Workshop on Advances in Model Based Software Testing (A-MOST), November 2006, pp. 1-9.
[8] C.J. Colbourn, M.B. Cohen, and R.C. Turban, A Deterministic Density Algorithm for Pairwise Interaction
Coverage. In Proc. of the IASTED Intl. Conference on Software Engineering, February 2004, pp. 345–352.
[9] P. Ciancarini, and D. Rossi. “Jada: A Coordination Toolkit for Java”, Technical Report UBLCS-96-15,
Department of Computer Science, University of Bologna, Italy, 1997.
[10] S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz, “Model-
Based Testing In Practice”. In Proc. of the 21st Intl. Conf. on Software Engineering, ACM Press, May 1999, pp.
285–294.
[11] D. Gelernter. “Generative Communication in Linda”. ACM Transactions on Programming Languages and
Systems, 7 (1), 1985, pp. 80-112.
[12] JUnit Website - URL http://www.junit.org
[13] D. Kuhni. “APROCO: A Programmable Coordination Medium”, Diploma Thesis, Faculty of Science,
University of Berne, Switzerland, 1998.
[14] D.R. Kuhn and M.J. Reilly. “An Investigation of the Applicability of Design of Experiments to Software
Testing”. In Proc. of the 27th NASA/IEEE Software Engineering Workshop, IEEE CS Press, December 2002, pp.
69-80.
[15] D.R. Kuhn and V. Okun. “Pseudo-Exhaustive Testing for Software”. In Proc. of the 30th NASA/IEEE
Software Engineering Workshop, IEEE CS Press, April 2006.
[16] D.R. Kuhn and D.R. Wallace. “Software Fault Interactions and Implications for Software Testing”. IEEE
Transactions on Software Engineering 30(6), June 2004, pp. 418-421.
[17] G.J. Myers. The Art of Software Testing, John Wiley and Sons, 1976.

34
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

[18] K.C. Tai and Y. Lei. “A Test Generation Strategy for Pairwise Testing”. IEEE Transactions on Software
Engineering 28(1), January 2002, pp. 109-111.
[19] K.Z. Zamli, N.A. Mat Isa, O. Sidek, and M.I. Ahmad. “Ensuring Dependability of COTs: A Work in
Progress”. In the CD-ROM Proc. of the Intl. Conference on Robotics, Vision, and Information Processing
(ROVISP05), Penang, Malaysia, July 2005.
[20] K.Z. Zamli, M.D. Alang Hassan, N.A. Mat Isa, and S.N. Azizan. “SFIT – An Automated Software Fault
Injection Tool”. In Proc. of the 2nd Malaysian Software Engineering Conference (MySec2006), Kuala Lumpur,
December 2006, pp. 99-105.
[21] K. Z. Zamli, M. D. Alang Hassan, N.A. Mat Isa, S.N. Azizan. “An Automated Software Fault Injection Tool
For Robustness Assessment of Java COTs”. In Proc. of the IEEE Intl. Conference on Computing and Informatics
(ICOCI06), Kuala Lumpur, June 2006.
[22] K.Z. Zamli, N.A. Mat Isa, M.F.J. Klaib and S.N. Azizan. “Designing a Combinatorial Java Unit Testing
Tool”. In Proc. of the IASTED Intl. Conference on Advances in Computer Science and Technology (ACST 2007),
Phuket, Thailand, April 2007.

Authors
Kamal Zuhairi Zamli

Received a B.Sc. degree in Electrical Engineering from WPI, USA, 1992, and
MSc degree (Real Time Software Engineering) from Universiti Teknologi
Malaysia, 2001, and PhD degree in Software Engineering from University of
Newcastle upon Tyne, UK, 2003. His research interests include Software
Engineering, Testing Automation, and Algorithm design.
Nor Ashidi Mat Isa

Received a B.Eng in Electrical Engineering from Universiti Sains Malaysia in

2000, and PhD in Image Processing and Neural Networks from the same
university in 2003. His research interests include Image Processing,
Neural Networks, and Algorithm design.
Mohamed Fadel Jamil Klaib

Received a B.Eng in Electronic Engineering from Al-Quds University in

Jurusalem in 2002, and MSc in Computer Engineering from Near East
University, Cyprus, 2004. Currently, a PhD candidate in the area of testing
automation and combinatorial testing.

Siti Norbaya Azizan

Received a B.Eng in Electrical Engineering from Universiti Sains Malaysia in

2005. Currently undertaking MSc by research in the area of combinatorial
testing.

35
 International Journal of Software Engineering and Its Applications
Vol. 1, No. 1, July, 2007

INTENTIONAL BLANK

36

View publication stats