Process Optimization for Testing of Domain Specific Languages in
Industrial Automation
Rajesh Kumar, Vivek Kumar
kumar.rajesh@siemens.com, Vivekk.@siemens.com
Siemens Technologies and Services limited
Bangalore India
Abstract— Software testing is essential part of software
development. The goal of the testing process is not only to
enhance the quality and robustness of the software but also verify
the correctness and non functional requirements of the software
under all working conditions. Large software has their large
testing suites to verify the stability of legacy features. Testing
processes have huge challenges to maintain effectiveness and
efficiency of the legacy test cases. There are many different
processes and techniques available all technique or processes
have their advantages and limitations. A tailored testing process
has been tried to utilize all technique together to improvise the
benefits and efficiency of testing in the industrial automation
domain. This paper tries to explain a customized approach of
utilizing the available testing techniques in such a way that it
enhances the effectiveness and efficiency of regression testing,
thus improving the time to market of large product-line
Industrial automation software
Keywords-Regression Testing;Test Effectiveness;Test suite
optimization,Software Testing process,Test Automation,Industial
automation
INTRODUCTION
In SDLC (Software development life cycle), software testing
has evolved as major activity to ensure the quality of a
software. Testers develop many test cases to test the software,
which over a period of time form the part of regression suite.
Regression testing is major part of software testing process. As
per the study, Regression Testing and its associated activities
cost almost half of software maintenance. [1][2].
Figure 1 Cost of Fixing a Defect [3][4]
Software Testing starts from requirement gathering of
software, during initial phase of software development;
Testing is motivated to test few functionality of initial version.
Testing process concentrate more on new feature test. But as
software grows up with new versions; older version also
needed to be verified (Fig 2). Test cases of older version are
executed to verify old features of software hence regression
testing comes into picture after release of some versions and it
is as important as new feature test. Regression testing can be
considered as one of factors to deliver reliable software.
Success of regression testing can be checked by its ability to
find defects in early stages. As regression suite is already
available during release of new version, effective regression
testing can ensure early verification of existing features. Early
detection of defects reduces the cost of defect fixing. [3][4].
In each version of software some new features get added and
tester needs to write test case for new features also. It is not
possible to execute all the test cases in each regression cycle;
Fig 2 illustrates the situation after some versions. For our
project it may take nearly 5000 hours to run all the test cases.
Hence a huge effort and cost gets involved in handling
regression testing. After some versions, major effort of testing
process inclines towards regression testing. To ensure software
reliabilty, lower cost, it is necessary to optimize and
streamline our regression testing process.
Figure 2 : Pictorial Representation of increasing burden on
Regression Testing
RELATED WORK
There is no direct approach till now, which consider all
technique all together. Many techniques have been discussed
and widely used in isolation. Techniques are well matured
also. Automation, one of widely used technique to optimize
test cases has been discussed in different way. [5][6].There is
great advantages along with some limitation, which have been
discussed in "Benefits and limitations of automated software
testing: Systematic literature review and practitioner survey".
[7]. "Empirical Observations on Software Testing
Automation"[8] says human involvement is must for selecting
test cases. Prioritization of test cases has been also considered
as one of technique in fulfilling the goal of early defect
detection and optimizing the cost of regression
test.[9][10].Some study shows that, prioritization is very
efficient in fault detection , while some study says it
compromises the capabilities of fault detection.[10][11].
Revision of test cases which can be used for legacy test cases
have been also discussed in many recent papers.
Combinatorial testing is one of the approaches, which can be
used for such revision. [12]. Variant of combinatorial testing
has also been discussed in some studies, which enhances the
effectiveness and efficiency of test cases. [13][14].
As we have many fundamental techniques available which
need to be implemented on test suite e.g. Automation of test
suite, Prioritization of test suite etc.Generally in a testing
process we take a solution at a time and try to optimize our
process. For example, a testing process chooses Automation as
a technique to optimize the regression suite and hence effort
goes in direction to automate the feasible group of testcases.
Does automation really helped that group of testcases to be
more effective? As Automation improves the process only in
terms of execution time, it does not enhances the effectiveness
of particular testcases.So a group of testcase which is obsolete,
may be in need of revision rather than automation.
As there is no solution, which consider all techniques together
and try to apply at most suitable test cases, which avoids the
disadvantages of the technique. Test process strategies and
project stakeholder believe automation process can reduce too
much of cost and make the process simpler. As there are many
advantages are seen in automation, many stakeholders want
their tester to automate everything. After equipped with
everything, testing process starts concentrating in automation
of all feasible test cases (Feasibility check includes technical
feasibility).
978-1-4673-6636-6/15/$31.00 ©2015 IEEE
 EXPLANATION Automation (T1): Testing process has considered this
Our use cases and experience hailed from real time project. So technique to reduce execution burden.
we would try to give a brief idea of existing testing process we
Deletion (T2): To remove irrelevant test cases.
have. Our Testing process is about testing a huge engineering
system's compiler, which supports IEC Standard PLC Prioritization (T3): To enhance the early fault detection
programming languages. Here, we will talk about two target capability of test suit.
devices, for which compiler has to be tested.Device1 supports
three plc programming languages while device2 supports two. Revision (T4): To enhance the fault detection capability of old
There are different instruction (Features) sets for both, some tests cases.
are common for both some are exclusive. Generally, for one
feature, 32 different combinations of Test cases are written. Once testing process is ready with Ts, Now it is time to define
criteria to application of particular testing technique.
Considering the technique used, criteria is needed to define
according to nature of application. Our Testing process has
considered 7 criteria and corresponding testing techniques.
Where C1 - Data Sensitive test case,C2 - Redundant test
case,C3 - Test case with higher Coverage,C4 - Legacy and
Stable Feature test case,C5 - Defect Clusters Test Case,C6 -
Figure 3 Summary of all the test cases across versions Pesticide test case,C7 - Core or Safety Feature test case. This
criterion defined above decides which testing technique will
A. The USE CASE be applicable for which test group.
As seen from table, Number of Test cases are around 30000,
which is not possible to be executed during time bounded D. Grouping of test cases
regression testing process. So, usual way of optimizing Testing process test many features of system, each feature
regression suite like prioritization and all time favorite have many test cases. Grouping of test cases is done to
automation came into picture. Both can be applied on any recognize its common nature.
regression suit test cases. Not different from any general
testing process, Automation was main concentration for Suppose A, B, C features are related to safety of system, it can
optimizing the regression suite as its advantages are many. We be said as A, B and C are feature of one group. As Grouping is
were able to automate significant amount of recently added project dependent, it can be done in other convenient way also.
test cases. But, after release of software, Increase in defect E. Characterizing of groups
slippage observed. On doing RCA (Root cause analysis)[15],
Once we have group of test case, we can characterize the
we concluded two things, concentration of faults are found
group according to its nature and our criteria defined above.
mainly in recently added features or most vulnerable areas
One group may be fulfilling one or many criteria. Similarly
Further analysis indicated a major fault in our strategies to
one or many criteria can be eligible for technique application.
automate. Once automation is done for particular group of test
A table has been defined below to describe, which criteria is
cases, Testers focus deviate to other test group for exploration
applicable for which technique.
the area which was selected for automation needed more
exploratory testing or some strategies of revision.
Crite C1 C2 C3 C4 C5 C6 C7
ria
B. Lesson learnt
Techni T1 T4 T1,T3 T1,T2 T3,T4 T2,T4 T3
All eggs should not be added to the same basket. For more ques
vulnerable area or recently added feature, Automation can be Table-2-Mapping of criteria and Technique
second priority idea. Revision or exploratory testing can be a
better idea for such scenario. However, there are many F. Apply matrix
techniques available, which can be implemented in Now, it is time to pass each test case group through this
streamlining regression test suite to make it more efficient and matrix. Suppose, X is a group of test case satisfying two
effective. In so many techniques, how to select suitable
character (C1 and C2). We will mark matrix like below. 1 is
technique to make it efficient. Till now we don’t have any
universal framework, which consider all techniques together marked in the corresponding cell. For example, for C2 we
with their preconditions. When a testing process having huge mark the cell {C2:T2}{C2:T2} and {C1:T1} (Table-3).
number of test cases, where all technique can be considered,
we need to develop a small framework, which can be used to G. Result
select most suitable technique to implement, this was our Ti = (∑ 𝐶𝑗)
motivation for which solution has been tried with some logical
reasoning. Testing Technique selected = Max (Ti) where i=0 to n
C. Approach to solution
We have tried to introduce one simple matrix consisting T1 T2 T3 T4
considered testing technique (Ts) and their criterion (Cs). C1 1
There are many techniques or way, which we can choose to C2 1 1
implement on our regression suite. Consideration of technique Table3- Solution Matrix
is dependent on some factors like cost, risk, resource project
environment etc.Our testing process considered following four So for X group of test case, T1 (Automation) has been
techniques which is identified as testing technique (Ts). considered as suitable technique.
 APPLICATION ON EXISTING REGRESSION SUITE
For the first time, we had tried to apply the method described
above on our existing test cases. To implement the solution,
we have followed following mentioned steps.
Select-> Selection of subset of existing test cases on which
this solution has to be applied. Three recent version test cases
have been selected for the implementation of software. Refer
Figure 3.
Select Classify Analayze Characterize
Figure 4- Process
Classify-> After Selection of test cases, we have 130 legacy
features selected. These 130 features have been classified
according to method mention in Grouping of test cases, five
groups evolved from the technique with more or less similar
nature. Where G1-Common features (device 1 and device 2),
G2 contains Pointer Features, G3 may contain Operations
feature, G4 may refer to Mathematical featuresG5-
Comparison features
Characterize-> in this step, groups has to be characterized
according to opinion of testers. For example, G1 (Common
feature) has been considered as high test coverage test
cases(C3) as it is executed on both the device and G1 is
considered as defect cluster test cases(C5) also according to
testers feedback. On similar track, we got a table.
G1-(C3, C5), G2-(C1, C5, C7), G3-(C1, C4, C6), G4-(C1, C2,
C4), G5-(C2, C4)
Figure 5- Defect Detection improvement for a span of 10 week
period
Apply-> now, group of test cases are characterized with their
criteria, it is time to pass each group through matrix. Firstly,
G1 is passed through matrix. We will refer matrix and Table2,
we find G1 is most suitable for T3 (Prioritization). So,
common features test cases are selected for prioritization, in
similar manner other groups are passed and selected for
appropriate technique. One thing notable here is, G3 (Move
Operation feature) was selected for T1 and T3 both. In such
cases testers have to decide among T1 and T3.
Once results are out for each group, expert resource for each
suitable technique is assigned with the task. Suppose
Automation is need for G4 (Mathematical Features), Test
automation people had been be assigned with the
mathematical family, now they have to think only to automate
mathematical family nothing else.
Figure 5 illustrates the result, which we have seen after
applying the technique, early defect detection has increased
significantly, which in turn reduces the cost of fixing.
Increasing trend in total defect detection has also observed.
Significant reduction in execution time enhanced the
possibility for exploratory testing.
CONCLUSION
A systematic approach towards selecting appropriate
technique to improve regression test enhances the confidence
in software quality. With proper characterization, we have got
Apply proper reasoning behind any testing technique applied.
Consideration of all the technique together with their positive
and negative aspects on testing process is a key factor in this
approach. Early defect detection in software development life
cycle has improved more on quality front. There are other
hidden benefits, which comes because of spared bandwidth.
This approach is a small step towards the dream of universal
framework of testing. [16]. In future, it can be further studied
and weighted matrix can be applied to enhance more accuracy.
ACKNOWLEDGEMENT
We would also like to thank Mr. Gielata Piotr for suggesting
futuristic ideas for conceptualization of this paper. We would
like specially thank our manager Mr. Shailendra Kulkarni for
the encouragement support and providing us the technical
direction towards the thought process of this paper. We would
also like to thank Mr. Sanath Shenoy also for encouraging us
throughout the creation of paper.
REFERENCES
[1] Leitner, Stefan-Helmut, and Wolfgang Mahnke. "OPC UA–service-
oriented architecture for industrial applications." ABB Corporate
Research Center (2006).
[2] Leung, Hareton KN, and Lee White. "Insights into regression testing
[software testing]." Software Maintenance, 1989., Proceedings.,
Conference on. IEEE, 1989.
[3] Beizer, Boris. "Software testing techniques. 1990." New York, ISBN: 0-
442-20672-0.
[4] Bennett, Ted L., and Paul W. Wennberg. "Eliminating embedded
software defects prior to integration test." Crosstalk, Journal of Defence
Software Engineering (2005).
[5] Boehm, Barry, and Victor R. Basili. "Software defect reduction top 10
list."Foundations of empirical software engineering: the legacy of Victor
R. Basili426 (2005).
[6] Memon, Atif M., Martha E. Pollack, and Mary Lou Soffa. "Using a
goal-driven approach to generate test cases for GUIs." Software
Engineering, 1999. Proceedings of the 1999 International Conference
on. IEEE, 1999.
[7] Bouquet, Fabrice, et al. "A test generation solution to automate software
testing." Proceedings of the 3rd international workshop on Automation
of software test. ACM, 2008.
[8] Rafi, Dudekula Mohammad, et al. "Benefits and limitations of
automated software testing: Systematic literature review and practitioner
survey."Proceedings of the 7th International Workshop on Automation
of Software Test. IEEE Press, 2012.
[9] Karhu, Katja, et al. "Empirical observations on software testing
automation."Software Testing Verification and Validation, 2009.
ICST'09. International Conference on. IEEE, 2009.
[10] Harrold, M. Jean, Rajiv Gupta, and Mary Lou Soffa. "A methodology
for controlling the size of a test suite." ACM Transactions on Software
Engineering and Methodology (TOSEM) 2.3 (1993): 270-285.
[11] Rothermel, Gregg, et al. "An empirical study of the effects of
minimization on the fault detection capabilities of test suites." Software
Maintenance, 1998. Proceedings., International Conference on. IEEE,
1998.
[12] Wong, W. Eric, et al. "Test set size minimization and fault detection
effectiveness: A case study in a space application." Journal of Systems
and Software 48.2 (1999): 79-89.
[13] [12] YAN, Jun, and Jian ZHANG. "Combinatorial testing: principles
and methods."Journal of Software 20.6 (2009): 1393-1405.
[14] Kuhn, D. Richard, Dolores R. Wallace, and Jr AM Gallo. "Software
fault interactions and implications for software testing." Software
Engineering, IEEE Transactions on 30.6 (2004): 418-421.
[15] Kuhn, Rick, Yu Lei, and Raghu Kacker. "Practical combinatorial
testing: Beyond pairwise." IT Professional 10.3 (2008): 19-23.
[16] Rooney, James J., and Lee N. Vanden Heuvel. "Root cause analysis for
beginners." Quality progress 37.7 (2004): 45-56.Bertolino, Antonia.
"Software testing research: Achievements, challenges, dreams." 2007
Future of Software Engineering. IEEE Computer Society, 2007.