Testing Theory

The Art of Software Testing alone gave the industry a long-lasting, foundational guide to one
of the most important computer topics: How do you ensure that all of the software you
produce does what it was designed to do, and— just as important—doesn’t do what it isn’t
supposed to do?
In a typical programming project approximately 50 percent of the elapsed time and more
than 50 percent of the total cost were expended in testing the program or system being
developed.
The alert reader will realize that the most important component in a program tester’s bag of
tricks is the knowledge of how to write effective test cases.
Software testing is a process, or a series of processes, designed to make sure computer code
does what it was designed to do and, conversely, that it does not do anything unintended.
Software should be predictable and consistent, presenting no surprises to users. In this book,
we will look at many approaches to achieving this goal.
Software testing is a technical task, yes, but it also involves some important considerations of
economics and human psychology.
In an ideal world, we would want to test every possible permutation of a program. In most
cases, however, this simply is not possible. Even a seemingly simple program can have
hundreds or thousands of possible input and output combinations. Creating test cases for all
of these possibilities is impractical. Complete testing of a complex application would take too
long and require too many human resources to be economically feasible. In addition, the
software tester needs the proper attitude (perhaps ‘‘vision’’ is a better word) to successfully
test a software application. In some cases, the tester’s attitude may be more important than
the actual process itself. Therefore, we will start our discussion of software testing with these
issues before we delve into the more technical nature of the topic.

The Psychology of Testing:

One of the primary causes of poor application testing is the fact that most programmers
begin with a false definition of the term.
When you test a program, you want to add some value to it. Adding value through testing
means raising the quality or reliability of the program. Raising the reliability of the program
means finding and removing errors.
Therefore, don’t test a program to show that it works; rather, start with the assumption that
the program contains errors (a valid assumption for almost any program) and then test the
program to find as many of the errors as possible. Thus, a more appropriate definition is this:
“Testing is the process of executing a program with the intent of
finding errors”.
Human beings tend to be highly goal-oriented, and establishing the proper goal has an
important psychological effect on them. If our goal is to demonstrate that a program has no
errors, then we will be steered subconsciously toward this goal; that is, we tend to select test
data that have a low probability of causing the program to fail. On the other hand, if our goal
is to demonstrate that a program has errors, our test data will have a higher probability of
finding errors. The latter approach will add more value to the program than the former.
A second problem with such definitions as ‘‘testing is the process of demonstrating that
errors are not present’’ is that such a goal is impossible to achieve for virtually all programs,
even trivial programs.
A third problem with the common definitions such as ‘‘testing is the process of
demonstrating that a program does what it is supposed to do’’ is that programs that do what
they are supposed to do still can contain errors. That is, an error is clearly present if a
program does not do what it is supposed to do; but errors are also present if a program does
what it is not supposed to do.
Of course, you eventually want to use program testing to establish some degree of
confidence that a program does what it is supposed to do and does not do what it is not
supposed to do, but this purpose is best achieved by a diligent exploration for errors.

The Economics of testing:

In general, it is impractical, often impossible, to find all the errors in a program. This
fundamental problem will, in turn, have implications for the economics of testing,
assumptions that the tester will have to make about the program, and the manner in which
test cases are designed.
To combat the challenges associated with testing economics, you should establish some
strategies before beginning. Two of the most prevalent strategies include black-box testing
and white-box testing.
Blackbox Testing:
One important testing strategy is black-box testing (also known as data-driven or
input/output-driven testing). To use this method, view the program as a black box. Your goal
is to be completely unconcerned about the internal behaviour and structure of the program.
Instead, concentrate on finding circumstances in which the program does not behave
according to its specifications. In this approach, test data are derived solely from the
specifications (i.e., without taking advantage of knowledge of the internal structure of the
program).
If you want to use this approach to find all errors in the program, the criterion is exhaustive
input testing, making use of every possible input condition as a test case.
To test the triangle program exhaustively, you would have to produce virtually an infinite
number of test cases, which, of course, is not possible. If this sounds difficult, exhaustive
input testing of larger programs is even more problematic.
This discussion shows that exhaustive input testing is impossible. Two important implications
of this: (1) You cannot test a program to guarantee that it is error free; and (2) a fundamental
consideration in program testing is one of economics. Thus, since exhaustive testing is out of
the question, the objective should be to maximize the yield on the testing investment by
maximizing the number of errors found by a finite number of test cases.
Whitebox Testing:
Another testing strategy, white-box (or logic-driven) testing, permits you to examine the
internal structure of the program. This strategy derives test data from an examination of the
program’s logic (and often, unfortunately, at the neglect of the specification).

Software testing principles:

It is important that you achieve optimum test results while conducting software testing without
deviating from the goal. But how you determine that you are following the right strategy for
testing? For that, you need to stick to some basic testing principles. Here are the common seven
testing principles that are widely practiced in the software industry.

To understand this, consider a scenario where you are moving a file from folder A to Folder B.

Think of all the possible ways you can test this.

Apart from the usual scenarios, you can also test the following conditions

 Trying to move the file when it is Open

 You do not have the security rights to paste the file in Folder B
 Folder B is on a shared drive and storage capacity is full.
 Folder B already has a file with the same name, in fact, the list is endless
 Or suppose you have 15 input fields to test, each having 5 possible values, the number of
combinations to be tested would be 5^15

If you were to test the entire possible combinations project EXECUTION TIME & COSTS would
rise exponentially. We need certain principles and strategies to optimize the testing effort.

Here are the 7 Principles:

1. Exhaustive Testing is not possible:
Even a seemingly simple program can have hundreds or thousands of possible input and
output combinations. Creating test cases for all of these possibilities is impractical.
Complete testing of a complex application would take too long and require too many
human resources to be economically feasible.
If you test a program, you would have to virtually produce an infinite number of test
cases, which, of course, is not possible. Thus, since exhaustive testing is out of the question,
the objective should be to maximize the yield on the testing investment by maximizing the
number of errors found by a finite number of test cases.
 2. Defect Clustering:
Defect Clustering which states that a small number of modules contain most of the
defects detected. This is the application of the Pareto Principle to software testing:
approximately 80% of the problems are found in 20% of the modules.

By experience, you can identify such risky modules. But this approach has its own problems

If the same tests are repeated over and over again, eventually the same test cases will no
longer find new bugs.

3. Pesticide Paradox:

During agriculture, repetitive use of same pesticide to eradicate pests lead to the pests
developing resistance to the pesticide Thereby ineffective of pesticides on insects. The
same applies to software testing. If the same set of repetitive tests are conducted, the
method will be useless for discovering new defects.

To overcome this, the test cases need to be regularly reviewed & revised, adding new &
different test cases to help find more defects.

Testers cannot simply depend on existing test techniques. He must look out continually
to improve the existing methods to make testing more effective. But even after all this
sweat & hard work in testing, you can never claim your product is bug-free. To drive
home this point, let’s see this video of the public launch of Windows 98

You think a company like MICROSOFT would not have tested their OS thoroughly &
would risk their reputation just to see their OS crashing during its public launch!

4. Testing show presence of Bugs:

Hence, testing principle states that – Testing talks about the presence of defects and
don’t talk about the absence of defects. i.e., Software Testing reduces the probability of
undiscovered defects remaining in the software but even if no defects are found, it is
not a proof of correctness.

But what if, you work extra hard, taking all precautions & make your software product
99% bug-free. And the software does not meet the needs & requirements of the clients.

This leads us to our next principle, which states that- Absence of Error

5. Absence of error-fallacy:

It is possible that software which is 99% bug-free is still unusable. This can be the case if
the system is tested thoroughly for the wrong requirement. Software testing is not
mere finding defects, but also to check that software addresses the business needs. The
absence of Error is a Fallacy i.e., Finding and fixing defects does not help if the system
build is unusable and does not fulfil the user’s needs & requirements.
 To solve this problem, the next principle of testing states that Early Testing.

6. Early Testing:

Early Testing – Testing should start as early as possible in the Software Development
Life Cycle. So that any defects in the requirements or design phase are captured in early
stages. It is much cheaper to fix a Defect in the early stages of testing. But how early one
should start testing? It is recommended that you start finding the bug the moment the
requirements are defined.

7. Testing is context dependent:

Testing is context dependent which basically means that the way you test an e-
commerce site will be different from the way you test a commercial off the shelf
application. All the developed software’s are not identical. You might use a different
approach, methodologies, techniques, and types of testing depending upon the
application type. For instance, testing, any POS system at a retail store will be different
than testing an ATM machine.

Myth: “Principles are just for reference. I will not use them in practice .”
This is so very untrue. Test Principles will help you create an effective Test Strategy and draft
error catching test cases.

But learning testing principles is just like learning to drive for the first time.

Initially, while you learn to drive, you pay attention to each and everything like gear shifts,
speed, clutch handling, etc. But with experience, you just focus on driving the rest comes
naturally. Such that you even hold conversations with other passengers in the car.

Same is true for testing principles. Experienced testers have internalized these principles to a
level that they apply them even without thinking. Hence the myth that the principles are not
used in practice is simply not true.

Summary This chapter discussed a form of testing that developers do not often consider:
human code testing. Most people assume that because programs are written for machine
execution, machines should test programs as well. This assumption is invalid. Human testing
techniques are very effective at revealing errors. In fact, most programming projects should
include the following human testing techniques:

Code inspections using checklists

Group walkthroughs

Desk checking

Peer reviews
Another form of human testing is user or usability testing, a black-box technique that
evaluates software from a hands-on, end-user perspective.