You are on page 1of 7

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/281772544

Testing Patterns in Action: Designing a Test-Pattern-Based Suite

Article  in  International Review on Computers and Software (IRECOS) · May 2015


DOI: 10.15866/irecos.v10i5.5963

CITATIONS READS

2 7,924

3 authors, including:

Bouchaib Falah Mohammed Akour


Al Akhawayn University Prince Sultan University
28 PUBLICATIONS   80 CITATIONS    83 PUBLICATIONS   516 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Joint programs and framework for doctoral education in software engineering - THE «PATHWAYS TO PHD» - PWS@PHD View project

OO Design View project

All content following this page was uploaded by Bouchaib Falah on 18 September 2015.

The user has requested enhancement of the downloaded file.


International Review on Computers and Software (I.RE.CO.S.), Vol. 10, N. 5
ISSN 1828-6003 May 2015

Testing Patterns in Action: Designing a Test-Pattern-Based Suite

Bouchaib Falah1, Mohammed Akour2, Nissrine El Marchoum1

Abstract – Design patterns constitute a revolution in the field of software engineering, as they
emphasize the importance of reuse and its impact on the software process and the software
product quality.
A special type of design patterns is testing patters; these can be used in the testing phase to reduce
redundancy, save time and resources and provide an effective reuse mechanism for more coverage
and better quality of service at the same time.
Many design patterns exist to test different aspects of the implemented functionality separately.
However, in this paper, we will suggest a new concept, which consists of incorporating different
testing patterns into the same test suite to test different aspects through running one single test
exactly once. It will also allow the users to track the performance of their test suite quality
attributes using a simple representation. Copyright © 2015 Praise Worthy Prize S.r.l. - All rights
reserved.

Keywords: Software Testing, Test Patterns, Design Patterns, Quality Metrics, Testing Quality

I. Introduction It goes without saying that one of the main goals of


the testing activity in the software life cycle is to identify
When the object oriented paradigm came as a faults and errors in the system and fix them thought pre-
revolution in the software engineering field, it brought defined techniques. One key consideration that the
one of its most useful principles: reuse. This principle software testers worry about the most is the quality of the
has evolved from simply reusing simple declarations or software being tested.
code fragments into software reuse and even logic reuse, Many existing applications of testing patterns exist
which is the key motivation behind what is referred to as varying from drafts of concepts to actual
software design patterns. implementations. However, these applications involving
These patterns are being used to save the software test patterns usually focus on generating separate test
engineers, developers as the software testers the burden cases for separate quality goals.
of wasting time and resources over producing redundant The underlying idea of this paper is to combine the
modules. The idea of reusing logic to solve similar types existing knowledge in the field to come up with a sample
of problems has been working pretty well over the last test suite that will incorporate different test patterns to
decades and proved to be a smart choice when it comes improve different quality goals in one single test suite.
to dealing with recurring engineering problems. Although working towards realizing multiple quality
Design patterns in general and testing patterns goals concurrently introduces tradeoffs, we can still
especially have been a revelation in the field of software come up with combinations of test patterns that can at
engineering. These patterns are considered as a way to least improve the overall performance of the testing
capture the knowledge and experience of the software process and satisfy the quality requirements of the
engineering community and build a basis for reuse [1]. customer.
With the growing popularity of patterns and their As much as there are countless applications of design
incorporation into the software engineering process, patterns to different scenarios ranging from software
many issues arose like compatibility, efficiency and design to software deployment, our ultimate focus is to
effectiveness, along with suitability to the current emphasize the importance of using test patterns to the
scenarios. Many argued that the best way to deal with software testing process to enhance the overall testing
these challenges, many argued that using patterns needs quality. Therefore, the focus will be on test patterns. We
to be accompanied by documentation [1]. fill first give a glance on the existing literature related to
This explains the remarkable number of publications the topic in section II, then depict how test patterns are
suggesting new concepts such as pattern contracts, used to test software in section III, then we will suggest
pattern test case templates and many other our testing concept incorporating test patterns into one
documentation techniques to guarantee that the patterns single test suite in Section IV and show an example in
at hand can be used efficiently and tailored easily to the section V. Finally we will briefly mention the future
different requirements of each single system. work in Section VI then conclude.

Copyright © 2015 Praise Worthy Prize S.r.l. - All rights reserved

489
Bouchaib Falah, Mohammed Akour, Nissrine El Marchoum

II. Related Work of traditional test code is the reduction of maintenance


costs by providing test cases which are easily adaptable
Over the last decade, the effectiveness of using design to change.
patterns for software design and implementation has been
questioned by many researchers in the field. Soundarajan
and Hallstrom et al. [2] for example suggest a solution to III. Testing Patterns in Practice
test how a given pattern used to solve a specific problem
stated in the requirement, complies to the requirements of In this section, we will be outlining some applications
the used pattern itself. This technique tests whether a of test patterns for different purposes. Although the core
pattern has been used correctly as to enable its use for motivation of using these patterns is to optimize the
multiple systems and avoid redundancy. This emphasizes testing process, there are many enhancements and real-
the main motivation behind using patterns in the software world applications that show the different contexts where
engineering process, which effective reuse of existing a test pattern can be used successfully.
solutions and tailoring them to every single system Testing patterns are divided into two main categories:
accordingly. For this purpose, the authors used what is extensible test patterns and static ones.
called Pattern Test Case Templates (PTCT), which is a These latter ones use dynamic features such as
codified solution of the pattern that identifies all the dynamic binding and dynamic typing, which makes the
defects associated with its implementation in different testing process using them pretty complex and
software systems. potentially prone to errors [9].
Even after the rise of testing patterns as a class of With the increasing development speed as well as the
patterns of its own, these patterns also were used in competitive edge each firm has to bring compare to the
association with PTCTs to evaluate their performance. other existing entities in the market, shortening the time
As [1] explains, these pattern test case templates are span allocated for testing while realizing a fault coverage
not only extracted from the requirements’ contracts to as high as possible has become an increasingly important
predict the faults, but also act as a starting point for test concern in the field of software testing. As way to realize
suite generation depending on the intended outcomes. these two objectives (better quality testing and big fault
Another important advantage to the technique coverage) at once in the most efficient way Automatic
suggested by this paper is also the early detection of Test Case Generation (ATCG) tools come into play [5].
design bugs, which are usually hard to localize and get These tools generate fault models and provide a
expensive as they go undetected throughout the software mechanism to measure the fault coverage of the test
engineering process. patterns used. They also ensure an efficient and effective
Another interesting take on the topic is depicted by test case generation process. Test patters can be used
Feudjio and Schieferdecker [3] who take testing patterns separately, but also in combination with other test
to a further level of automation. With the growing patterns to achieve different quality goals. When this
customer needs and the short time to market, test happens a methodology needs to be used to coordinate
automation appears to be an appealing solution for the between them. Among these algorithms we find test
testers and an effective way to test similar modules with ordering algorithms and test data flow controlling
slight modifications. This paper outlines some design algorithms [6]. These algorithms help prioritize test cases
patterns for reactive software systems and shows how and pick which ones should go first. This way
important it is to have such patterns in an environment redundancy is avoided and tradeoffs between the
where interactivity is a key characteristic of the units different patterns are reduced. Other algorithms on the
under test. other hand, focus on reducing test data efficiently even
In the same context of automated testing using with multiple fault models [7].
patterns, Rybalov [4] also emphasizes how crucial There is also a tool used to measure the quality of test
automated testing is to ease the process and help testers patterns themselves. It uses a ration of covered
and developers to do refactoring easily. However the conditional branches by the pattern to the number of
author applies automated testing with patterns to conditional branches in total [8]. This ratio called “the
customer testing. The author draws the attention to an branch pass index” has allowed the software testers to
important fact, which is that test cases are programs, use a benchmark to evaluate the test patterns quality.
therefore they are also vulnerable to any faults and errors
application code might have. This is an aspect of testing IV. Patternsuite: The Testing- Pattern-
that has been overlooked for quite long as the testing
programs have always been considered to be simple and
Based Software Testing Suite
easy. However with the evolving complexity of units This technique uses an iterative incremental six-tep
under test, having effective test cases has also become an process. It is based on the elicitation of quality metrics
issue that takes up the time of the testers and developers. and requirements from the customer, aping them to
This is where test patterns come into play as they help generated test patterns, then tracking the metrics’
minimizing the cost of updating and refactoring test realization and creating test suites based on the findings.
cases as the application code under test evolves. So the This process illustrated in the flow chart in Fig. 1 will
main motivation behind using patterns for testing instead be thoroughly explained in this section.

Copyright © 2015 Praise Worthy Prize S.r.l. - All rights reserved International Review on Computers and Software, Vol. 10, N. 5

490
Bouchaib Falah, Mohammed Akour, Nissrine El Marchoum

Fig. 2. Scenario-based test service level agreement generation


in PATTERNSUITE

IV.2. Automatic Test Pattern Generation


Automatic test pattern generation tools have attracted
researchers for over thirty years in spite of the many
challenges that arise with their use such as scalability and
the flexibility in addition to the ability to handle different
fault models at once [10].
Automatic Test Pattern Generation (ATPG) tools do
Fig. 1. The six-step process of creating a test-patter-based test suite not only provide high quality testing to different fault
models; they also help identifying design faults early on,
IV.1. Service Agreements with the Customer which is an incredible functionality given the difficulty
of pointing out those faults early on in the software
The concept of service agreements is very popular in development stage [10].
the cloud computing field. Service-level agreements have ATPG function in two main phases: fault activation
been used over the last decade to specify the level of and fault propagation. The fault activation phase consists
service clients are interested in out of the wide variety of producing a value at the fault site. In the fault
offered by the cloud under the form of Software as a propagation phase, the effect previously produced is
Service (SaaS) [11]. propagated from the initial fault location to a primary
As for the field of software testing, a new concept output site [10]. The goal of the process is to come up
seems to follow the same logic; it is referred to as with a test sequence (test input) that allows the testers to
customer scenario focused end-to-end testing. distinguish between the correct path and that affected by
This method is essential to ensure the quality of the propagated fault. The effectiveness of the test
software products, as it is usually impossible to cover all sequence is assesses by measuring the coverage of the
customer scenarios in the designed test suites by the test fault model. The effectiveness also depends on the fault
engineers. That is why this method suggests a common model used the unit under test, the abstraction level at
technique that models scenario-based test suites that which it has been represented and other quality
converge towards providing high customer scenario considerations. As for the efficiency of the whole ATPG
coverage [12]. The way this concept is incorporated into tool, it is directly related to the decision making
our PATTERNSUITE technique is depicted in Fig. 2. throughout the process of giving adequate inputs to the
In order to realize high coverage, the customer tool [10]. This step of the process is the most innovative
scenarios are modeled in terms of the services provided idea that this concept brings to software testing. Instead
by the unit under test, using the Persona-Experience- of selecting traditional existing testing patterns, we can
Usecase hierarchy. The Personas refer to the different use automation to save time and have more accurate
user categories, the experience to the different patterns based on the program logic and data.
functionality expected by each individual user class, and Automatic Test Pattern Generation Tools (ATPGT)
the usecase classifies the functionality requirements of are in charge of generating fault models.
each use category [12]. These fault models can be used in combination with
During this step, a clear formulation of the intended the existing testing techniques to measure the fault
quality goals of the customer is done. Instead of giving a coverage and to train the testing process as to get to the
whole set of raw requirements, the testing engineers best possible coverage. They also generate test patterns
should help the customer refine the quality requirements that can be reused over time to maximize reusability.
and come up with a precise and most importantly ordered Those test patterns can then be fed a finite state
list of priorities in terms quality goals. machine (FSM) that tests the unit under test accordingly.

Copyright © 2015 Praise Worthy Prize S.r.l. - All rights reserved International Review on Computers and Software, Vol. 10, N. 5

491
Bouchaib Falah, Mohammed Akour, Nissrine El Marchoum

IV.3. Quality Metrics Establishment TABLE I


A SAMPLE SELECTION OF TESTING PATTERNS ALONG
Identifying and establishing the right metrics for WITH THEIR TARGET QUALITY GOALS
software testing in general is not an easy task. It is a Test Pattern Usage Target Quality Metric
tedious process to be followed and usually involves a Template Pattern Testing units, Customer Satisfaction
components and Customer Testing
great deal of experience from the testing engineers. That subsystems without
is why various benchmarks are usually used to set changing the original
specific values or thresholds to the quality metrics code
previously extracted in the first phase. Object Genie Using a mother class to Customer Satisfaction
do the recurring Low Coupling
Although using benchmarks and predefined values is instantiation of all the
of great help, the experience and intuition of software important classes to the
testers remains a plus. UUT
It is through dealing with recurring problems that the Domain Test Object Encapsulation of visual Fast and Effective
arts of the application GUI Testing
testers develop an experience that allows them to into objects for reuse Good and Esy
approximately tell the extent to which quality metrics are Acceptance Testing
realized –or not- by a given test suite. Transporter Creation of a central Fast Automated
The quality metrics used for evaluating the software entity which navigates Testing
through the units under Good Customer
testing process are rather quantitative, rather than giving test Testing
a broad description of” how well the test case does”, Dumb Testing Reduce knowledge about Environment-change-
ratios, percentages and frequencies are calculated in navigability to check agnostic testing
order to give an accurate measurement tool of the quality logic and user interface strategy
details Low Coupling
of the testing process. Independent Testing The state of the UUT Stability
In this step, the testing engineers establish thresholds remains unchanged no Coherence
and values to the specific quality metrics that have been matter if the test succeeds
discussed in the first step. These quality metrics are or fails
Don’t Repeat Avoid repeated code or Lower Test
usually quantitative (percentage, number, fraction): Yourself (DRY) logic to optimize testing Maintenance Cost
 Fault Coverage. code No duplication
 Conditional Branch Coverage.
Multiple Failures Continue to execute even Robustness
 Reaction to Bugs. if failures occur High Fault Coverage
 Cost to maintain test.
 Time of execution. Depending on the services required by the customer in
addition to the quality metrics set in the previous steps,
IV.4. Test Case Design and Pattern Incorporation the testers need to proceed by trial and error to make
candidate test suites that combine these test cases.
After generating patterns using an automatic test
pattern generation tool, it is up to the software testers to
design test cases that incorporate these patters. ATPG IV.5. Quality Metrics Tracking Dashboard
tools usually result in a test case which consists of test As previously discussed in an earlier section, coverage
inputs. measurement tools provide many mechanisms to
Designing test cases that incorporate the test pattern visualize the output of the different testing techniques in
would mainly aim at transforming the raw test case into terms of measured performance. Our technique can
an object-oriented entity which fulfills the objet oriented implement a user-friendly interface that allows the user
paradigm characteristics such as encapsulation, to display the quality metrics recorded either individually
separation of concerns, cohesion … etc. or to display charts and comparative views of the
The traditional way of selecting test patterns would presented data. And therefore track the testing
typically consist of finding the testing pattern that meets performance and take decisions accordingly.
the quality requirements of the customer.
Table I outlines an example of a small selection of
testing patterns and their purpose. IV.6. Decision Making: Final Test Suite Selection
If the test patterns were to be selected manually by tge According to the statistics displayed on the dashboard,
software tester, the task would be to select the test pattern the testing engineers pick the combination of generated
whose target quality metrics comply with those of the testing patterns that results in the best quality metric
customer. selected. This way the final test suite maximizes the
But since anything done annually involves a great deal quality metric based on the customer’s requirements.
of error risk, it is better to use automatic test pattern
generation tools.
In this step, based on the output of the automatic test V. Patternsuite Example
pattern generation tools’ output, the testers design test
In this section, we will provide a general example to
cases that incorporate the resulting patterns.
show how the suggested technique can be applied.

Copyright © 2015 Praise Worthy Prize S.r.l. - All rights reserved International Review on Computers and Software, Vol. 10, N. 5

492
Bouchaib Falah, Mohammed Akour, Nissrine El Marchoum

We suppose that we have a meeting with a client for in a user interface that the testers can have easy access
requirements elicitation from which we derive the quality too. Data can either be displayed in the table format or as
requirements and use an automatic test pattern generation a graph to have more visibility to the testers and help
that we will call PatternSuperTool. them in the decision making process.

V.1. Service Agreement Phase V.5. Final Test Suite Selection


The service level agreement with the client states the This is the final phase where the testers decide which
following: test suite from the candidates will be picked. In this case,
 The first quality requirement is that the test cases t4 is the best test suite because it gives the best results for
applied to the unit under test should try to cover as both quality metrics, and will therefore be selected as the
many conditional branches as possible. final one. We should mention that the choice is not
 Second test cases ran by the client’s team during always obvious, as sometimes there is no best solution.
acceptance testing should not take too long to However in situations of ties, the testers should pick
execute. the test case that gives the best value of the first quality
Other quality requirements were discussed during the metric on the list. This is where the service level
meeting but the goal of this specific service level agreements play their most imminent role: prioritization.
agreement is to focus on those two quality requirements
specifically.
VI. Future Work
V.2. Automatic Test Pattern Generation Now that the concept is described, the future
contribution would be a blueprint for a real-life
In this step, we run PatternSuperTool feeding it with application. Typical blueprint would tackle a common
some sample data provided by the customer. This data application example and produce all the deliverables
will be used to generate fault models, and then including the service level agreements, the test cases
generating four different test patterns that we will call p1 along with their originating fault models, the candidate
p2, p3 and p4. test suites, the quality metrics and the result displayed on
a dashboard in a simple user-friendly interface.
The next step should compare the requirements
V.3. Quality Metrics Establishment Phase
realization using this technique to the already existing
Based on the service-level agreement, we establish a techniques; a benchmark should be chosen against which
threshold for both quality metrics as follows: the comparison should be done. This would demonstrate
 Required ratio of branch index is ¾; the feasibility of the project, along with the advantages
 Each test case should have an average execution time that it brings compared to the traditional testing
of 4 seconds. techniques.

V.4. Test Case Design and Pattern Incorporation VII. Conclusion


and Dashboard Display Phase
Software testing patterns are indeed a great asset at the
Now that we have set up the quality metrics, we will disposition of software testers.
design test cases that use the generated test patterns in The smart use of this incredible tool can result in a
the second phase. The next step would consist of creating remarkable shift in the fixed quality metrics realization
all the possible combinations of test suites using these by any given testing team.
test cases. Let’s consider the test cases t1, t2, t3 and t4 as Furthermore, the use of patterns in testing can
being the test cases incorporating the generated patterns improve the quality of the process itself; logic reuse
p1, p2, p3, and p4 respectively. along with building an archive of reusable mechanisms
Creating the test suites and computing the metrics for can only get more done within less time.
each case will give the results in Table II. The main advantage of this technique is that it gives
the testers a mechanism to realize many quality attributes
TABLE II using one single test suite. In the traditional way, it is
QUALITY METRICS MEASURED FOR CANDIDATE TEST SUITES
practically extremely challenging to avoid the tradeoffs
Average Execution
Test Suite Test Cases Branch Index
Time that arise when trying to balance between different
1 t1,t2,t3 20% 1s customer requirements at once. Furthermore, the process
2 t1,t2,t4 60% 1s can be subject to many refinement iterations.
3 t1,t3,t4 70% 8s This process does not have to follow a waterfall
4 t2,t3,t4 80% 3s
model; especially that it takes time and effort to take a
decision regarding the final test suite. Therefore, the fact
In this example we will only focus on test suites that that this technique lends itself to an incremental or
have three test cases for simplicity. This data is displayed iterative paradigm can be a key factor in its success.

Copyright © 2015 Praise Worthy Prize S.r.l. - All rights reserved International Review on Computers and Software, Vol. 10, N. 5

493
Bouchaib Falah, Mohammed Akour, Nissrine El Marchoum

The suggested technique cannot only be refine din [10] Cheng, Krstic, “Current Directions in Automatic Test-Pattern
Generation”, Computer, Vol. 32, Issue 11, pp. 58-64.
terms of the sequence of activities and the precision of
[11] Kübert, Sewner, “High performance computing as a service with
the generated deliverables. It can also be further Service-Level Agreements”, 2012 IEEE Ninth International
enhanced by incorporating many innovative techniques. Conference on Service Computing, pp. 579- 585.
Good example would be using test pattern scenarios
instead along with test pattern contracts for a more
accurate formulation and documentation of the test Authors’ information
patterns being used. Moreover, many algorithms can be 1
Al Akhawayn University in Ifrane, Morocco
used for test pattern prioritization such as decompression 2
algorithms [6]. On the other hand, the activities involved Yarmouk University, Jordan.
in this technique can be extremely time-consuming if not Dr. Bouchaib Falah is currently an Assistant
handled in a smart way. This technique could be Professor at Al Akhawayn University, teaching
extremely efficient, but only if carried out by testing graduate and undergraduate software
engineers who have experience in the field of testing and engineering courses, School of Science and
Engineering. Beside teaching high school level
good knowledge about testing patterns. math in Morocco and college mathematics and
It is also worth mentioning that the process of computer science at Harrisburg Area
designing a test case based on a given pattern is a non Community College in Pennsylvania, Suny
trivial task. Another drawback to the suggested technique Orange Community College in New York, Pennsylvania State
University in Pennsylvania, Central Pennsylvania College in
is that it is still at the conceptual level. And it goes Pennsylvania, Concordia College in Minnesota, and North Dakota State
without saying that implementing the necessary University in North Dakota, Dr. Bouchaib Falah has an extensive
mechanisms will result in challenges, bugs and errors industrial experience with Agri-ImaGIS, Synertich, and
that cannot be detected at the conceptual phase. Commonwealth of Pennsylvania Department of Environmental
Protection.
Although the suggested technique –like any new E-mail: b.falah@aui.ma
concept- might not be very popular, it still constitutes a
starting point for the researchers in the field, as it gives Dr. Mohammed Akour is an Assistant
incentives for new research topics. It can also be of great Professor in the Department of Computer
Information System at Yarmouk University
help to testing engineers, especially in the context of big (YU). He got his Bachelor (2006) and Master
complex projects that would take long to test. Finally, the (2008) degree from Yarmouk University in
core motivation behind this technique is reuse. Computer Information System with Honor. He
Those who might argue that the process of joined YU as a Lecturer in August 2008 after
graduating with his master in Computer
incorporating patterns might be time and resource Information System. In August 2009, He left YU to pursue his PhD in
consuming, it is a methodology that allows to build a Software Engineering at North Dakota State University (NDSU). He
testing mechanism once, but that can be reused as many joined YU again in April 2012 after graduating with his PhD
times as needed. in Software Engineering from NDSU with Honor.
E-mail: Mohammed.akour@yu.edu.jo

Nissrine El Marchoum is Allianz Managed


References Operations & Services SE. Munich Area,
[1] Soundarajan, Hallstrom, Delibas, Shu, “Testing Patterns”, Germany. She got her master degree from Al
Software Engineering Workshop, 2007. SEW207, IEEE, 2007. Pp. Akhawayn University in Ifrane.
109-120. E-mail: n.elmarchoum@aui.ma
[2] Soundarajan, Hallstrom, “Patterns: from system design to
software testing”, Innovations Syst Softw Eng,2008, pp. 71-85.
[3] Freudjio, Schieferdecker, “Test Automation Design Patterns for
Reactive Software Systems”, EuroPLop Conference, workshop E,
2009.
[4] Raybolov, “Design Patterns For Customer Testing”, white paper,
2004.
[5] Benware, Schuermyer, Tamarapalli, Tsai, Ranganathan, Madge,
Rajski, Krishnamurthy, “Impact of Multiple-Detect Patterns on
Product Quality”, Test Conference 2003, vol. 1. pp. 1031-1040.
[6] Novak, “Comparison of Test Patterns Decompression
Techniques”, Design, Automation and Test in Europe Conference
and Exhibition, 2003, Czech Republic. Pp. 1182-1183.
[7] Alampally, Venktesh, Shanmugasundaram, Parekhii, Agrawal,
“An Efficient Test Data Reduction TechniqueThrough Dynamic
Pattern Mixing Across Multiple Fault Model”, VLSI Test
Symposium (VTX), 2011 IEEE 29th. pp. 285-290.
[8] T. Aoki, T. Toriyama, K. Ishikawa, and K. Fukami, “A tool for
measuring quality of test pattern for LSIs’ functional design,” in
Proceedings of Asia and South Pacific Design Automation
Conference (ASP-DAC), 1995, pp. 75–78.
[9] W. T. Tsai, Y. Tu, W. Shao, and E. Ebner, “TestingExtensible
Design Patterns in Object-Oriented Frameworks through
Hierarchical Scenario Templates”, Proc. of IEEE COMPSAC,
1999, pp. 166-171.

Copyright © 2015 Praise Worthy Prize S.r.l. - All rights reserved International Review on Computers and Software, Vol. 10, N. 5

494

View publication stats

You might also like