See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/344122939

A Comparative Study of Different Software Testing Techniques: A Review

Article · January 2018

CITATIONS READS

2 1,380

2 authors:

Seema Rani Deepali Gupta

Guru Jambheshwar University of Science & Technology Chitkara University
5 PUBLICATIONS 4 CITATIONS 59 PUBLICATIONS 977 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Software Testing View project

All content following this page was uploaded by Seema Rani on 05 September 2020.

The user has requested enhancement of the downloaded file.

 Journal of Advances in Shell Programming
ISSN: 2395-6690 (Online)
Volume 5, Issue 1
www.stmjournals.com

A Comparative Study of Different Software Testing

Techniques: A Review
Seema Rani1,*, Deepali Gupta2
1
Department of Computer Science Engineering, Ch. Devi Lal State Institute of Engineering and
Technology, Panniwala Mota, Sirsa, Haryana, India
2
Department of Computer Science Engineering, Maharishi Markandeshwar University, Sadopur,
Ambala, Haryana, India

Abstract
Software testing is the process of identifying the correctness and quality of software program.
The purpose is to check whether the software satisfies the specific requirements, needs and
expectations of the customer. Software testing is the process to expose design, requirement and
coding errors in the program. Testing is used to identify the correctness, completeness, security
and quality of software products against the requirements of the user. It exhibits all faults,
errors and flaws in the developed software. Software testing is aimed at evaluating the
capability or usability of a program. It is a significant method of accessing the quality of
software. Though a lot of advancements have been done in formal methods and verification
techniques, still we need software to be fully tested before it could be handled to the customer
side. There are a number of testing techniques available to complete a task. Software testing is
an important area of research and a lot of development has been made in this field. In this
paper, all the software testing techniques have been described and compared. Some latest
researches have been described in brief. The concept of software testing is gaining more and
more importance in the future. There are many approaches to software testing, but effective
testing is a process of investigation, not only a matter of creating and following route procedure.
While it is not possible to detect all the faults in the software; this problem in testing thus arises
a question: what would be the strategy we should adopt for testing? In our paper, we have
described and compared the three most prevalent and commonly used software testing
techniques for detecting errors and faults. They are: white box testing, black box testing and
grey box testing.

Keywords: Black box, grey box, white box, prevention, software testing, software testing
principles, research

*Author for Correspondence E-mail: seemagadai8108@gmail.com

INTRODUCTION Software testing is an activity that is performed

Software Testing is the process of identifying
the correctness and quality of software
program. The purpose is to check whether the
software satisfies the specific requirements,
needs and expectations of the customer. In other
words, creating a program consists of the five
phases [1]:
(1) Defining a problem, (2) Designing a
program, (3) Building a program, (4) Analyzing
performances of a program, and (5) Final
arranging of a product.
According to this classification, software
testing is a component of the third phase, and
means checking if a program for specified
inputs gives correct and expected results.
for evaluating software quality and also for
improving it. Thus, the goal of testing is
systematic and stepwise detection of different
classes of errors within a minimum amount of
time and also with a much less amount of effort.
Software testing is also an important
component of software quality assurance
(SQA), and a number of software organizations
are spending up to 40% of their resources on
testing.
Software testing is not a “silver bullet” that can
guarantee the production of high quality
software systems. While a “correct” correctness
proof demonstrates that a software system
(which exactly meets its specification) will

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 1
Comparative Study of Software Testing Techniques
always operate in a given manner, software
testing that is not fully exhaustive can only
suggest the presence of flaws and cannot prove
their absence. Moreover, it is impossible to
completely test an application because: (1)
there are too many possible input paths; (2) the
domain of program inputs is too large; and (3)
design and specification issues are difficult
tests. The first and second points present
obvious complications and the final point
highlights the difficulty of determining if the
specification of a problem solution and the
design of its implementation are also correct
[2].
The main objective of software testing is to
affirm the quality of software system by
systematically testing the software in carefully
controlled circumstances; another objective is
to identify the completeness and correctness of
the software, and finally it uncovers
undiscovered errors [3, 4].
This paper is related to many research papers
published in different publications like IEEE
transactions, international journals, and
international conferences. This paper is
prepared as: Next part if the paper is describing
the objectives of software testing. The part after
that provides information about the role of
software testing in software defect prediction
process, software testing principles,
information about software testing levels,
description about the software testing
techniques and comparison between three
forms of software testing techniques. The last
part provides conclusion of the paper.
OBJECTIVES
There are four main objectives of testing:
Detection
Different kinds of defects, errors and
deficiencies are detected. Quality of all
components, system capabilities and various
limitations, and the work products of the overall
system are calculated.
Prevention
It is the process to reduce the number of errors,
to explain system specifications and system
performance. Different ways to avoid risks and
to deal with problems in the future are
identified.
JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved
Rani and Gupta
Demonstration
It shows how the system can be used with
different acceptable risks. It also demonstrates
functions with special conditions and shows
how products are ready for integration or use.
Improving Quality
By applying effective testing on software,
errors can be reduced and thus quality of
software is improved.
ROLE OF SOFTWARE TESTING IN
SOFTWARE DEFECT PREDICTION
PROCESS
The common software defect prediction
process follows the machine learning approach
[5–7]. The first step in defect prediction process
is to find out the instances from software, an
instance can be code, function, class or method
etc. These instances can be generated from the
different issue tracking systems, version control
systems or e-mail archives. An instance has
different metrics which is found out from the
software. These instances can be categorized in
bugs B or number of bugs and clean C or
number of clean. After recognizing instances
with the category and metrics, the first step of
machine learning pre-processing techniques is
used on instances to create new same type of
instance. The pre-processing is applied to
extract the features, scaling the data and
removing the noise [8–10]. It is not compulsory
to apply on all types of defect prediction models
[6, 11]. After pre-processing, the instances
generate new instances to train the defect
prediction model. The prediction model
provides the result in terms of buggy instances
and clean instances. The number of bugs in an
instance is known as regression. It produces
only two results for the instances bugs or
cleans; so, it is also known as binary
classification.
SOFTWARE TESTING PRINCIPLES
Different software testing principles are as
follows:
Test a Program so as to Make it Fail
Software testing is the process of executing a
program with the aim of finding bugs and
errors. Testing becomes more effective when
failures are exposed.
Page 2
Journal of Advances in Shell Programming
Volume 5, Issue 1
ISSN: 2395-6690 (Online)
Start Testing Early
Start testing early helps in finding and fixing a
number of errors in the early stages of
development, and thus reduces the rework of
finding the errors in the later stages.
Testing is Context Dependent
Testing should be suitable and different for
different context at different points of time.
Test Plan
Test plan usually describes test strategy, test
scope, test objectives, test environment,
deliverables of the test, risks and mitigation
involved, schedule, levels of testing to be
applied, techniques, methods and tools to be
used. Test plan should accurately meet the
needs of an organization and customer as well.
Effective Test Cases
Effective test cases must be designed properly
so that test can be measured and clear test
results can be produced.
Test Valid as Well as Invalid Conditions
In addition to valid test cases, test cases for
invalid and unexpected inputs/conditions must
also be checked. This form of testing is
sometimes specified as regression testing.
Test at Different Levels
Different testing must be done at different
levels of testing, so, different people can
perform testing differently using different
testing techniques at all levels.
End of Testing
Testing has to be stopped somewhere. It is
stopped when risks are under some limit or if
there is some limitation to it.
SOFTWARE TESTING LEVELS
There are different levels during the process of
testing [12]. Levels of testing include different
methodologies that can be used while
conducting software testing. There are main
four levels of software testing:
Unit Testing
It is done at the lowest level. It tests the basic
unit of software, which can be a module or
component. Unit is the smallest module i.e.
smallest set of lines of code which can be tested.
Unit testing is just one of the levels of testing
JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved
which contribute to make the big picture of
testing a whole system. Unit testing is generally
considered as a white box test class.
Integration Testing
It is done when two or more tested units are
combined into a larger structure. This testing is
often done on the interfaces that are between the
components and the larger structure that is
being constructed, if its quality property cannot
be properly assessed from its components.
System Testing
It tends to test the end-to-end quality of the
entire system. System test is often based on the
functional requirement specifications of the
system. Non-functional quality attributes, such
as security, reliability, and maintainability, are
also checked.
Acceptance Testing
It is done when the complete system is handed
over to the customers or users from developer
side. The aim of acceptance testing is to assure
that the system is working rather than to find
error.
SOFTWARE TESTING TECHNIQUES
The three most important techniques that are
used for finding errors are [3]:
Black Box Testing Technique
Black box testing is technique of testing
software based on output requirements and
without any knowledge of the internal structure
or coding in the program (Figure 1).
Fig. 1: Black Box Testing.
In another words, a black box is any device
whose workings are not understood by or
accessible to its user. For example, in
telecommunications, it is a resistor connected to
a phone line that makes it impossible for the
telephone company’s equipment to detect when
a call has been answered. In data mining, a
Page 3
Comparative Study of Software Testing Techniques
black box is an algorithm that does not provide
an explanation of how it works.
Types of Black Box Testing Techniques
(Figure 2)
Equivalence Class Partitioning
Boundary Value Analysis
Black Box Testing
Cause-Effect Graphing Techniques
Comparison Testing
Fuzz Testing
Model-based testing
Fig. 2: Different Types of Black Box Testing
Techniques.
Equivalence Class Partitioning: This technique
divides the input domain of a program onto
equivalence classes. Equivalence classes are set
of valid or invalid states for input conditions,
and can be defined in the following way:
1. An input condition specifies a range: one
valid and two invalid equivalence classes
are defined;
2. An input condition needs a specific value:
one valid and two invalid equivalence
classes are defined;
3. An input condition specifies a member of a
set: one valid and one invalid equivalence
class are defined;
4. An input condition is Boolean: one valid
and one invalid equivalence class are
defined.
Well, using this technique, we can get test cases
which identify the classes of errors.
Boundary Value Analysis: This technique is
like the technique equivalence partitioning,
except that for creating the test cases beside
input domain use output domain. We can form
the test cases in the following way:
1. An input condition specifies a range
bounded by values a and b: test cases
should be made with values just above and
just below a and b, respectively;
2. An input condition specifies various values:
test cases should be produced to exercise
the minimum and maximum numbers;
JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved
Rani and Gupta
3. Rules 1 and 2 apply to output conditions.
If internal program data structures have
prescribed boundaries, then produce test cases
to exercise that data structure at its boundary.
Cause-Effect Graphing Techniques: We use
this technique when we want to translate a
policy or procedure specified in a natural
language into software’s language. This
technique means: Input conditions and actions
are listed for a module ⇒ an identifier is
allocated for each one of them ⇒ cause-effect
graph is created ⇒ this graph is changed into a
decision table ⇒ the rules of this table are
modified to test cases.
Comparison Testing: In situations when
reliability of software is critical, redundant
software is produced. In that case, we use this
technique. This technique means: Software
engineering teams produce independent
versions of an application→ each version can
be tested with the same test data→ so the same
output can be ensured.
Fuzz Testing: Fuzz testing is often called
robustness testing or negative testing. It was
developed by Barton Miller at the University of
Wisconsin in 1989. This technique feeds
random input to application. The main
characteristic of fuzz testing are:
 In fuzz testing, the input is random;
 If the application crashes or hangs, the test
is failed;
 Fuzz testing can be automated to a high
degree.
Model-Based Testing: Model-based testing is
automatic generation of efficient test
procedures using models of system
requirements and specified functionality. In this
method, test cases are derived in whole or in
part from a model that describes some aspects
of the system under test. These test cases are
known as the abstract test suite. This testing
forces detailed understanding of the system
behaviour and can do early bug detection.
Advantages
1. Testers need not to have knowledge on
specific programming language.
2. Testing is done from user’s point of view.
Page 4
Journal of Advances in Shell Programming
Volume 5, Issue 1
ISSN: 2395-6690 (Online)
3. It helps to expose any ambiguities or
inconsistencies in the requirement
specifications [13].
4. Programmer and tester both are
independent of each other.
Disadvantages
1. Test cases are hard to design without clear
specifications.
2. Probability of having repetition of tests that
are already done by programmer.
3. Some parts of back end are not tested at all.
White Box Testing
White box testing is highly effective in
detecting and resolving problems, because bugs
(bug or fault is an appearance of an error in a
software [14]) can often be found before they
cause trouble. We can shortly define this
method as testing software with the knowledge
of the internal structure and coding inside the
program [15]. It is necessary for a tester to have
full knowledge of source code.
Fig. 3: White Box Testing.
White box testing is also called white box
analysis, clear box testing or clear box analysis.
It is a strategy for software debugging in which
the tester has excellent knowledge of how the
program components interact (Figure 3). This
method can be used for Web services
applications. This is purely done based on
customers view point; only tester knows the set
of inputs and predictable outputs [16, 17].
JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved
Types of White Box Testing Techniques
(Figure 4)
Condition Testing
White Box Testing
Data Flow Testing
Basis Path Testing
Loop Testing
Fig. 4: Different Types of White Box Testing
Techniques.
Condition Testing: By this technique, each
logical condition in a program is tested.
A relational expression takes a form:
E1> operator relational <E2,
Where, E1 and E2 are arithmetic expressions,
and relational operator is one of the following:
<, ≤, =, ≠, >, or ≥. This technique determines
not only errors in the conditions of a program
but also errors in the whole program.
Data Flow Testing: Data flow testing is another
type of white box testing which looks at how
data moves within a program. In data flow
testing, the control flow graph is annoted with
the information about how the program
variables are defined and used (Control flow
graph is a diagrammatic representation of a
program and its execution). We can also define
data flow testing as testing techniques which is
based on the observation that values associated
with variables can effect program execution.
Basis Path Testing: If we use this technique, we
can evaluate logical complexity of procedural
design. After that, we can employ this
technique for description of basic set of
execution paths. For obtaining the basis set and
for presentation control flow in the program, we
use flow graphs. Main components of flow
graphs are: Nodes, Edges and Regions.
Cyclomatic Complexity is software metric. The
value evaluated for cyclomatic complexity
defines the number of independent paths in the
basis set of a program. Independent path is any
path through a program that introduces at least
one new set of processing statements. For the
given graph G, cyclomatic complexity V(G) is
equal to:
1. The number of regions in the flow graph;
Page 5
Comparative Study of Software Testing Techniques
2. V(G)=E–N+2, where, E is the number of
edges, and N is the number of nodes;
3. V(G)=P+1, where, P is the number of
predicate nodes.
Loop Testing: Loop testing is another type of
white box testing which exclusively focuses on
the validity of loop construct. Loops are simple
to test unless dependencies exist between the
loops or among the loop and the code it
contains. There are four classes of loops:
Simple loops, Nested loops, Concatenated
Loops, and Unstructured loops.
Advantages
1. It reveals error in hidden code by removing
extra lines of code.
2. Maximum coverage is attained during test
scenario writing [18].
3. Developer carefully gives reasons about
implementation.
Disadvantages
1. A skilled tester is needed to carry out this
testing because knowledge of internal
structure is required.
2. Many paths will remain untested as it is
very difficult to look into every nook and
corner to find out hidden errors.
Grey Box Testing Technique
Gray-box testing attempts, and generally
succeeds, to combine the benefits of both black-
box and white-box testing. Gray-box testing
takes the straight-forward approach of black-
box testing, but also employs some limited
knowledge of the inner workings of the
application (Figure 5).
White box + Black box = Grey box
Fig. 5: Grey Box Testing.
JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved
Rani and Gupta
It is a technique to test the application with
limited knowledge of the internal working of an
application and also has the knowledge of
fundamental aspects of the system [18].
Therefore, a tester can verify both the output of
the user interface and also the process that leads
to that user interface output. Gray-box testing
can be applied to most testing phases; however
it is mostly used in integration testing. Different
types of grey box testing techniques are briefly
described below in Figure 6 [19]:
Types of Grey Box Testing Techniques
Orthogonal Array Testing
Grey Box Testing
Matrix Testing
Regression Testing
Pattern Testing
Fig. 6: Different Types of Grey Box Testing
Techniques.
Orthogonal Array Testing: This testing
technique is used as subset of all possible
combinations.
Matrix Testing: In this testing technique, the
status report of the project is stated.
Regression Testing: If any kind of new changes
are made in the software, then regression testing
implies running of test cases.
Pattern Testing: Pattern testing technique
verifies the good application for its architecture
and design.
Advantages
1. It provides combined benefit of black box
and white box testing techniques.
2. In grey box testing, tester can design
excellent test scenarios.
3. Unbiased testing.
4. Create an intelligent test authoring.
Disadvantages
1. Test coverage is limited as the access to
source code is not available.
2. Many program paths remain untested.
3. The test cases can be redundant [18].
Page 6
Journal of Advances in Shell Programming
Volume 5, Issue 1
ISSN: 2395-6690 (Online)

COMPARISON BETWEEN THREE errors and flaws in the developed software. It

FORMS OF TESTING TECHNIQUES executes the software with an intention of
(Table 1) finding errors in it. Software testing can provide
Table 1: Comparison between Three Forms of an independent view of the software to allow
the business to comprehend and appreciate the
Testing Techniques.
S. Black Box White Box Grey Box
risk of software implementation. So, to carry
No. Testing Testing Testing out software testing in a more effective
1. Black box In white box In grey box behaviour, this paper provides description and
testing analyses testing there is testing there is comparison of three main software testing
fundamental full partial knowledge techniques.
aspects only i.e. knowledge of of internal
there is no internal working.
knowledge of working. REFERENCES
internal working. 1. Parezanovic Nedeljko. Racunarstvo i
2. Black box It is performed It is performed by informatika. Beograd: Zavod za udzbenike
testing is by both end users and also i nastavna sredstva; 1996.
performed by developers by tester and 2. Kapfhammer Gregory M. Software
end users and and testers. developers (user
Testing: The Computer Science and
also by tester acceptance
and developers testing). Engineering Handbook. CRC Press; May
(user acceptance 2004.
testing). 3. Mohd. Ehmer Khan. Different Forms of
3. In black box Here, Here, granularity Software Testing Techniques for Finding
testing, the granularity is is medium. Errors. IJCSI. May 2010; 7(3)(1): 11–16p.
granularity is high.
low.
4. Azhar. Software Testing Methodologies.
4. Black box This testing This testing suits Available at http://azhar-paperpresentatio
testing does not does not suit the algorithm n.blogspot.com/2010/04/software-testing-
suit the the algorithm testing (suited for methodologies.html
algorithm testing. all). 5. Bacchelli A, Ambros MD, Lanza M. Are
testing. Popular Classes More Defect Prone? In
5. Here, the testing
Here, internals Here, the test
Proceedings of the 13th International
is based on only are fully design is based on
external known. high level data Conference on Fundamental Approaches to
exceptions and flow diagram, Software Engineering, FASE’10, Berlin,
internal database Heidelberg: Springer-Verlag; 2010; 59–
behaviour of the diagrams, internal 73p.
program is states, knowledge 6. Engstrom E, Runeson P, Wikstrand G. An
completely of algorithm and
ignored. architecture.
Empirical Evaluation of Regression
6. Black box White box Grey box testing Testing Based on Fix-Cache ¨
testing is least
testing is most process is Recommendations. In Software Testing,
exhaustive andexhaustive and somewhere in Verification and Validation (ICST), 2010
time consuming time between. Third International Conference on. Apr
process. consuming 2010; 75–78p.
process.
7. This testing can Here, the Here, in testing
7. Nagappan N, Ball T. Use of Relative Code
be tested only by testing can be the data domains Churn Measures to Predict System Defect
trial and error done better by and internal Density. In Proceedings of the 27th
method. testing data boundaries can be International Conference on Software
domains and tested and over Engineering, ICSE’05. 2005; 284–292p.
internal flow. 8. Nam J, Pan SJ, Kim S. Transfer Defect
boundaries.
Learning. In Proceedings of the 2013
International Conference on Software
CONCLUSION Engineering, ICSE’13, Piscataway, NJ,
Software testing is an activity or a process of USA: IEEE Press; 2013; 382–391p.
identifying the correctness, capability and 9. Kim S, Zhang H, Wu R, et al. Dealing with
quality of software program. The purpose is to Noise in Defect Prediction. In Proceeding
check whether the software satisfies the specific of the 33rd International Conference on
requirements, needs and expectations of the Software Engineering, ICSE’11. New
customer. Software testing exhibits all faults, York, NY, USA: ACM; 2011; 481–490p.

JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved Page 7
 Comparative Study of Software Testing Techniques
10. Menzies T, Greenwald J, Frank A. Data
Mining Static Code Attributes to Learn
Defect Predictors. IEEE Trans Softw Eng.
Jan 2007; 33(1): 2–13p.
11. Jiang T, Tan L, Kim S. Personalized Defect
Prediction. In Automated Software
Engineering (ASE), 2013 IEEE/ACM 28th
International Conference on. Nov 2013;
279–289p.
12. Rasneet Kaur Chauhan, Iqbal Singh. Latest
Research and Development on Software
Testing Techniques and Tools. IJCET.
2014; 4(4): 2368–2372p.
13. Nidhra Srinivas, Jagruthi Dondeti. Black
Box and White Box Testing Techniques: A
Literature Review. International Journal of
Embedded Systems & Applications
(IJESA). 2012; 2(2): 29–50p.
14. http://www.testingstandards.co.uk/living_
glossary.htm# Testing, Feb 08, 2009.
15. http://www.pcmag.com/encyclopedia_ter
m/0,2542,t=white+box+testing&i=54432,
00.asp, Feb 08, 2009.
16. Murnane T, Reed K. On the Effectiveness
of Mutation Analysis as a Black Box
Testing Technique. In Software
JoASP (2018) 1-8 © STM Journals 2018. All Rights Reserved
View publication stats
Rani and Gupta
Engineering Conference, 2001
Proceedings, Australia. 2001; 12–20p.
17. Mitra P, Chatterjee S, Ali N. Graphical
Analysis of MC/DC Using Automated
Software Testing. In Electronics Computer
Technology (ICECT), 2011 3rd
International Conference on. 2011; 3: 145–
149p.
18. Khan, Mohd Ehmer, Farmeena Khan. A
Comparative Study of White Box, Black
Box and Grey Box Testing Techniques.
International Journal of Advanced
Computer Sciences and Applications
(IJACSA). 2012; 3(6): 12–15p.
19. Grey Box Testing from Wikipedia
available at http://en.wikipedia.org/wiki/G
ray_box_testing
Cite this Article
Seema Rani, Deepali Gupta. A Comparative
Study of Different Software Testing
Techniques: A Review. Journal of
Advances in Shell Programming. 2018;
5(1): 1–8p.
Page 8