Software Testing Types

Static Testing
Overview

The Verification activities fall into the category of Static Testing. During static testing,
you have a checklist to check whether the work you are doing is going as per the set
standards of the organization. These standards can be for Coding, Integrating and
Deployment. Review's, Inspection's and Walkthrough's are static testing methodologies.

Contents:

 Introduction
 Static Code Analysis
 Formal Methods

Introduction
Static testing is a form of software testing where the software isn't actually used. This is
in contrast to Dynamic testing. It is generally not detailed testing, but checks mainly for
the sanity of the code, algorithm, or document. It is primarily syntax checking of the code
or and manually reading of the code or document to find errors. This type of testing can
be used by the developer who wrote the code, in isolation. Code reviews, inspections and
walkthroughs are also used.

From the black box testing point of view, static testing involves review of requirements
or specifications. This is done with an eye toward completeness or appropriateness for the
task at hand. This is the verification portion of Verification and Validation.

Bugs discovered at this stage of development are less expensive to fix than later in the
development cycle.

Static Code Analysis

Static code analysis is the analysis of computer software that is performed without
actually executing programs built from that software (analysis performed on executing
programs is known as dynamic analysis). In most cases the analysis is performed on
some version of the source code and in the other cases some form of the object code. The
term is usually applied to the analysis performed by an automated tool, with human
analysis being called program understanding or program comprehension.

The sophistication of the analysis performed by tools varies from those that only
consider the behavior of individual statements and declarations, to those that include the
complete source code of a program in their analysis. Uses of the information obtained
from the analysis vary from highlighting possible coding errors (e.g., the lint tool) to
formal methods that mathematically prove properties about a given program (e.g., its
behavior matches that of its specification).

Some people consider software metrics and reverse engineering to be forms of static
analysis.

A growing commercial use of static analysis is in the verification of properties of

software used in safety-critical computer systems and locating potentially vulnerable
code.

Formal Methods

Formal methods is the term applied to the analysis of software (and hardware) whose
results are obtained purely through the use of rigorous mathematical methods. The
mathematical techniques used include denotational semantics, axiomatic semantics,
operational semantics, and abstract interpretation.

It has been proven that, barring some hypothesis that the state space of programs is finite
and small, finding possible run-time errors, or more generally any kind of violation of a
specification on the final result of a program, is undecidable: there is no mechanical
method that can always answer truthfully whether a given program may or may not
exhibit runtime errors. This result dates from the works of Church, Gödel and Turing in
the 1930s (see the halting problem and Rice's theorem). As with most undecidable
questions, one can still attempt to give useful approximate solutions.

Some of the implementation techniques of formal static analysis include:

• Model checking considers systems that have finite state or may be reduced to
finite state by abstraction;
• Abstract interpretation models the effect that every statement has on the state of
an abstract machine (ie, it 'executes' the software based on the mathematical
properties of each statement and declaration). This abstract machine
overapproximates the behaviours of the system: the abstract system is thus made
simpler to analyze, at the expense of incompleteness (not every property true of
the original system is true of the abstract system). If properly done, though,
 abstract interpretation is sound (every property true of the abstract system can be
mapped to a true property of the original system).
• Use of assertions in program code as first suggested by Hoare logic. There is tool
support for some programming languages (e.g., the SPARK programming
language (a subset of Ada) and the Java Modeling Language — JML — using
ESC/Java and ESC/Java2).

Dynamic Testing
Overview

Dynamic Testing involves working with the software, giving input values and checking if
the output is as expected. These are the Validation activities. Unit Tests, Integration Tests,
System Tests and Acceptance Tests are few of the Dynamic Testing methodologies.

Contents:

 Introduction
 Methodologies

Introduction

Dynamic testing
(or dynamic analysis) is a term used in software engineering to describe the testing of the
dynamic behavior of code. That is, dynamic analysis refers to the examination of the
physical response from the system to variables that are not constant and change with
time.

In dynamic testing the software must actually be compiled and run; this is in
contrast to static testing. Dynamic testing is the validation portion of Verification
and Validation.

Some of dynamic testing methodologies include:

1. Unit Testing

2. Integration Testing
 3. System Testing
4. Acceptance Testing

Black Box Testing

Introduction

Black box testing takes an external perspective of the test object to derive test cases.
These tests can be functional or non-functional, though usually functional. The test
designer selects valid and invalid input and determines the correct output. There is no
knowledge of the test object's internal structure.

This method of test design is applicable to all levels of software testing: unit, integration,
functional testing, system and acceptance. The higher the level, and hence the bigger and
more complex the box, the more one is forced to use black box testing to simplify. While
this method can uncover unimplemented parts of the specification, one cannot be sure
that all existent paths are tested.

Contents:

 Testing Strategies/Techniques
 Black Box Testing Strategy
 Black Box Testing Example
 Advantages and Disadvantages
 Dimensions for Examining Testing

Testing Strategies/Techniques

• black box testing should make use of randomly generated inputs (only a test range
should be specified by the tester), to eliminate any guess work by the tester as to
the methods of the function
• data outside of the specified input range should be tested to check the robustness
of the program
• boundary cases should be tested (top and bottom of specified range) to make
sure the highest and lowest allowable inputs produce proper output
• the number zero should be tested when numerical data is to be input
• stress testing should be performed (try to overload the program with inputs to see
where it reaches its maximum capacity), especially with real time systems
• crash testing should be performed to see what it takes to bring the system down
 • test monitoring tools should be used whenever possible to track which tests have
already been performed and the outputs of these tests to avoid repetition and to
aid in the software maintenance
• other functional testing techniques include: transaction testing, syntax testing,
domain testing, logic testing, and state testing.
• finite state machine models can be used as a guide to design functional tests
• According to Beizer the following is a general order by which tests should be
designed:

1. Clean tests against requirements.

2. Additional structural tests for branch coverage, as needed.
3. Additional tests for data-flow coverage as needed.
4. Domain tests not covered by the above.
5. Special techniques as appropriate--syntax, loop, state, etc.
6. Any dirty tests not covered by the above.

Black Box Testing Strategy

Black Box Testing is not a type of testing; it instead is a testing strategy, which does not
need any knowledge of internal design or code etc. As the name "black box" suggests, no
knowledge of internal logic or code structure is required. The types of testing under this
strategy are totally based/focused on the testing for requirements and functionality of the
work product/software application. Black box testing is sometimes also called as
"Opaque Testing", "Functional/Behavioral Testing" and "Closed Box Testing".

The base of the Black box testing strategy lies in the selection of appropriate data as
per functionality and testing it against the functional specifications in order to check for
normal and abnormal behavior of the system. Now a days, it is becoming common to
route the Testing work to a third party as the developer of the system knows too much of
the internal logic and coding of the system, which makes it unfit to test the application by
the developer.

In order to implement Black Box Testing Strategy, the tester is needed to be thorough
with the requirement specifications of the system and as a user, should know, how the
system should behave in response to the particular action.

Various testing types that fall under the Black Box Testing strategy are: functional testing,
stress testing, recovery testing, volume testing, User Acceptance Testing (also known as
UAT), system testing, Sanity or Smoke testing, load testing, Usability testing,
Exploratory testing, ad-hoc testing, alpha testing, beta testing etc.

These testing types are again divided in two groups: a) Testing in which user plays a role
of tester and b) User is not required.
Testing method where user is not required:

Functional Testing:
In this type of testing, the software is tested for the functional requirements. The tests are
written in order to check if the application behaves as expected.

Stress Testing:
The application is tested against heavy load such as complex numerical values, large
number of inputs, large number of queries etc. which checks for the stress/load the
applications can withstand.

Load Testing:
The application is tested against heavy loads or inputs such as testing of web sites in
order to find out at what point the web-site/application fails or at what point its
performance degrades.

Ad-hoc Testing:
This type of testing is done without any formal Test Plan or Test Case creation. Ad-
hoc testing helps in deciding the scope and duration of the various other testing and
it also helps testers in learning the application prior starting with any other testing.

Exploratory Testing:
This testing is similar to the ad-hoc testing and is done in order to learn/explore the
application.

Usability Testing:
This testing is also called as ‘Testing for User-Friendliness’. This testing is done if
User Interface of the application stands an important consideration and needs to be
specific for the specific type of user.

Smoke Testing:
This type of testing is also called sanity testing and is done in order to check if the
application is ready for further major testing and is working properly without
failing up to least expected level.

Recovery Testing:
Recovery testing is basically done in order to check how fast and better the
application can recover against any type of crash or hardware failure etc. Type or
extent of recovery is specified in the requirement specifications.

Volume Testing:
Volume testing is done against the efficiency of the application. Huge amount of data
is processed through the application (which is being tested) in order to check the
extreme limitations of the system.
Testing where user plays a role/user is required:

User Acceptance Testing:

In this type of testing, the software is handed over to the user in order to find out if the
software meets the user expectations and works as it is expected to.

Alpha Testing:
In this type of testing, the users are invited at the development center where they use the
application and the developers note every particular input or action carried out by the
user. Any type of abnormal behavior of the system is noted and rectified by the
developers.

Beta Testing:
In this type of testing, the software is distributed as a beta version to the users and users
test the application at their sites. As the users explore the software, in case if any
exception/defect occurs that is reported to the developers.

Black Box Testing Example

In this technique, we do not use the code to determine a test suite; rather, knowing the
problem that we're trying to solve, we come up with four types of test data:

1. Easy-to-compute data
2. Typical data
3. Boundary / extreme data
4. Bogus data

For example, suppose we are testing a function that uses the quadratic formula to
determine the two roots of a second-degree polynomial ax2+bx+c. For simplicity, assume
that we are going to work only with real numbers, and print an error message if it turns
out that the two roots are complex numbers (numbers involving the square root of a
negative number).

We can come up with test data for each of the four cases, based on values of the
polynomial's discriminant (b2-4ac):

Easy data (discriminant is a perfect square):

a b c Roots
1 2 1 -1, -1
1 3 2 -1, -2
Typical data (discriminant is positive):

a b c Roots
1 4 1 -3.73205, -0.267949
2 4 1 -1.70711, -0.292893

Boundary / extreme data (discriminant is zero):

a b c Roots
2 -4 2 1, 1
2 -8 8 2, 2

Bogus data (discriminant is negative, or a is zero):

a b c Roots
1 1 1 square root of negative number
0 1 1 division by zero

As with glass-box testing, you should test your code with each set of test data. If the
answers match, then your code passes the black-box test.

Advantages and Disadvantages

Advantages of Black Box Testing

• more effective on larger units of code than glass box testing

• tester needs no knowledge of implementation, including specific programming
languages
• tester and programmer are independent of each other
• tests are done from a user's point of view
• will help to expose any ambiguities or inconsistencies in the specifications
• test cases can be designed as soon as the specifications are complete

Disadvantages of Black Box Testing

• only a small number of possible inputs can actually be tested, to test every
possible input stream would take nearly forever
 • without clear and concise specifications, test cases are hard to design
• there may be unnecessary repetition of test inputs if the tester is not informed of
test cases the programmer has already tried
• may leave many program paths untested
• cannot be directed toward specific segments of code which may be very complex
(and therefore more error prone)
• most testing related research has been directed toward glass box testing

Dimensions for Examining Testing

There are five dimensions that can be used for examining testing:

1. People (who does the testing)

2. Coverage (what gets tested)
3. Risks (why you are testing)
4. Activities (how you are testing)
5. Evaluation (how you know you’ve found a bug)

Characteristics

Let’s use this system to understand and clarify the characteristics of black box and white
box testing.

People: Who does the testing?

Some people know how software works (developers) and others just use it (users).
Accordingly, any testing by users or other nondevelopers is sometimes called “black box”
testing. Developer testing is called “white box” testing. The distinction here is based on
what the person knows or can understand.

Coverage: What is tested?

If we draw the box around the system as a whole, “black box” testing becomes another
name for system testing. And testing the units inside the box becomes white box testing.
This is one way to think about coverage.

Another is to contrast testing that aims to cover all the requirements with testing that aims
to cover all the code. These are the two most commonly used coverage criteria. Both are
supported by extensive literature and commercial tools. Requirements-based testing could
be called “black box” because it makes sure that all the customer requirements have been
verified. Code-based testing is often called “white box” because it makes sure that all the
code (the statements, paths, or decisions) is exercised.

Risks: Why are you testing?

Sometimes testing is targeted at particular risks. Boundary testing and other attack-based
techniques are targeted at common coding errors. Effective security testing also requires a
detailed understanding of the code and the system architecture. Thus, these techniques
might be classified as “white box.”

Another set of risks concerns whether the software will actually provide value to users.
Usability testing focuses on this risk, and could be termed “black box.”

Activities: How do you test?

A common distinction is made between behavioral test design, which defines tests based
on functional requirements, and structural test design, which defines tests based on the
code itself. These are two design approaches. Since behavioral testing is based on
external functional definition, it is often called “black box,” while structural testing—
based on the code internals—is called “white box.” Indeed, this is probably the most
commonly cited definition for black box and white box testing.

Another activity-based distinction contrasts dynamic test execution with formal code
inspection. In this case, the metaphor maps test execution (dynamic testing) with black
box testing, and maps code inspection (static testing) with white box testing.

We could also focus on the tools used. Some tool vendors refer to code-coverage tools as
white box tools, and tools that facilitate applying inputs and capturing inputs—most
notably GUI capture replay tools—as black box tools. Testing is then categorized based
on the types of tools used.

Evaluation: How do you know if you’ve found a bug?

There are certain kinds of software faults that don’t always lead to obvious failures. They
may be masked by fault tolerance or simply luck. Memory leaks and wild pointers are
examples. Certain test techniques seek to make these kinds of problems more visible.
Related techniques capture code history and stack information when faults occur, helping
with diagnosis. Assertions are another technique for helping to make problems more
visible. All of these techniques could be considered white box test techniques, since they
use code instrumentation to make the internal workings of the software more visible.
These contrast with black box techniques that simply look at the official outputs of a
program.

To summarize, black box testing can sometimes describe user-based testing (people);
system or requirements-based testing (coverage); usability testing (risk); or behavioral
testing or capture replay automation (activities). White box testing, on the other hand, can
sometimes describe developer-based testing (people); unit or code-coverage testing
(coverage); boundary or security testing (risks); structural testing, inspection or code-
coverage automation (activities); or testing based on probes, assertions, and logs
(evaluation).

WhiteBox Testing
Introduction

White box testing is a security testing method that can be used to validate whether code
implementation follows intended design, to validate implemented security functionality,
and to uncover exploitable vulnerabilities.

Contents:

 Overview
 What is White Box Testing?
 Black Box and Gray Box Testing
 How to perform White Box Testing
 Testing Techniques
 Classes of Tests
 Tools
 Results to Expect
 Business Case
 Benefits
 Costs
 Alternatives
 Conclusion

Overview

This section introduces white box testing for security, how to perform white box testing,
and tools and techniques relevant to white box testing. It brings together concepts from
two separate domains: traditional white box testing techniques and security testing. It
assumes the reader to be familiar with general concepts of software security. Refer to
other content areas on this portal to learn different aspects of software security.

This section will help security developers and testers understand white box testing for
security and how to effectively use the approach, tools, and techniques applicable to
white box testing.
The section is organized into separate sections dealing with what white box testing is,
how to perform white box testing, what results to expect, the business case to justify
white box testing, skills and training required to perform white box testing, and a brief
case study.

What is White Box Testing?

The purpose of any security testing method is to ensure the robustness of a system in the
face of malicious attacks or regular software failures. White box testing is performed
based on the knowledge of how the system is implemented. White box testing includes
analyzing data flow, control flow, information flow, coding practices, and exception and
error handling within the system, to test the intended and unintended software behavior.
White box testing can be performed to validate whether code implementation follows
intended design, to validate implemented security functionality, and to uncover
exploitable vulnerabilities.

White box testing requires access to the source code. Though white box testing can be
performed any time in the life cycle after the code is developed, it is a good practice to
perform white box testing during the unit testing phase.

White box testing requires knowing what makes software secure or insecure, how to
think like an attacker, and how to use different testing tools and techniques. The first step
in white box testing is to comprehend and analyze source code, so knowing what makes
software secure is a fundamental requirement. Second, to create tests that exploit
software, a tester must think like an attacker. Third, to perform testing effectively, testers
need to know the different tools and techniques available for white box testing. The three
requirements do not work in isolation, but together.

Black Box and Gray Box Testing

Black box testing is based on the software’s specifications or requirements, without
reference to its internal workings. Gray box testing combines white box techniques with
black box input testing [Hoglund 04]. This method of testing explores paths that are
directly accessible from user inputs or external interfaces to the software.

In a typical case, white box analysis is used to find vulnerable areas, and black box
testing is then used to develop working attacks against these areas. The use of gray
box techniques combines both white box and black box testing methods in a powerful
way.

How to perform White Box Testing

Process

The figure provides a graphic depiction of the security testing process. This same process
applies at all levels of testing, from unit testing to systems testing. The use of this
document does not require subscribing to a specific testing process or methodology.
Readers are urged to fit the activities described here into the process followed within their
organization.

The general outline of the white box testing process is as follows:

1. Perform risk analysis to guide the whole testing process.

2. Develop a test strategy that defines what testing activities are needed to
accomplish testing goals.
 3. Develop a detailed test plan that organizes the subsequent testing process.
4. Prepare the test environment for test execution.
5. Execute test cases and communicate results.
6. Prepare a report.

In addition to the general activities described above, the process diagram introduces
review cycles, reporting mechanisms, deliverables, and responsibilities.

Inputs

Some of the artifacts relevant to white box testing include source code, a risk analysis
report, security specification/requirements documentation, design documentation, and
quality assurance related documentation.

• Source code is the most important artifact needed to perform white box testing.
Without access to the code, white box testing cannot be performed, since it is
based on testing software knowing how the system is implemented.

• Architectural and design risk analysis should be the guiding force behind all
white box testing related activities, including test planning, test case creation, test
data selection, test technique selection, and test exit criteria selection. If a risk
analysis was not completed for the system, this should be the first activity
performed as part of white box testing. The following section discusses risk
analysis.
• Design documentation is essential to improve program understanding and to
develop effective test cases that validate design decisions and assumptions.

• Security specifications or requirements are a must, to understand and validate the

security functionality of the software under test.
• Security testers should have access to quality assurance documentation to
understand the quality of the software with respect to its intended functionality.
Quality assurance documentation should include a test strategy, test plans, and
defect reports. Load and performance tests are important in understanding the
constraints placed on the system and the behavior of the system under stress.
• Any artifact relevant to program understanding should be available to white box
testers.

Risk Analysis
Security is always relative to the information and services being protected, the skills and
resources of adversaries, and the costs of potential assurance remedies; security is an
exercise in risk management. The object of risk analysis is to determine specific
vulnerabilities and threats that exist for the software and assess their impact. White box
testing should use a risk-based approach, grounded in both the system’s implementation
and the attacker’s mindset.

White box testing should be based on architecture and design-level risk analysis. This
content area will discuss how to use the results of risk analysis for white box testing,
while the Architectural Risk Analysis content area discusses risk analysis in detail.

Risk analysis should be the guiding force behind all white box testing related activities.
The following paragraphs briefly introduce how the risk analysis results are used in white
box testing. The subsequent sections discuss the activities in detail.

The risk analysis report, in combination with a functional decomposition of the

application into major components, processes, data stores, and data communication
flows, mapped against the environments across which the software will be deployed,
allows for a desktop review of threats and potential vulnerabilities. The risk analysis
report should help identify

• the threats present in each tier (or components)

• the kind of vulnerabilities that might exist in each component
• the business impact (consequence and cost of failure of software) of risks<
• the probability (likelihood) of the risks being realized
• existing and recommended countermeasures to mitigate identified risks

Use the above information from the risk analysis report to

• develop a test strategy: Exhaustive testing is seldom cost-effective and often not
possible in finite time. Planned testing is therefore selective, and this selection
should be based on risks to the system. The priority (or ranking) of risks from the
risk analysis should be the guiding rule for the focus of testing, simply because
highly vulnerable areas should be tested thoroughly. The test strategy captures all
the decisions, priorities, activities, and focus of testing based on the consequence
of failure of software. The following section discusses test strategy in detail. For
detailed research on risk-based test planning.
• develop test cases: While a test strategy targets the overall test activities based on
risks to the system, a test case can target specific concerns or risks based on the
threats, vulnerabilities, and assumptions uncovered during the analysis. For
example, tests can be developed to validate controls (or safeguards) put in place to
mitigate a certain risk.
• determine test coverage: The higher the consequence of failure of certain areas (or
components), the higher the test coverage should be in those areas. Risk-based
testing allows for justifying the rigor of testing in a particular area. For example, a
 specific component or functionality may have high exposure to untrusted inputs,
hence warranting extra testing attention.

Test Strategy

The first step in planning white box testing is to develop a test strategy based on risk
analysis. The purpose of a test strategy is to clarify the major activities involved, key
decisions made, and challenges faced in the testing effort. This includes identifying
testing scope, testing techniques, coverage metrics, test environment, and test staff skill
requirements. The test strategy must account for the fact that time and budget constraints
prohibit testing every component of a software system and should balance test
effectiveness with test efficiency based on risks to the system. The level of effectiveness
necessary depends on the use of software and its consequence of failure. The higher the
cost of failure for software, the more sophisticated and rigorous a testing approach must
be to ensure effectiveness. Risk analysis provides the right context and information to
derive a test strategy.

Test strategy is essentially a management activity. A test manager (or similar role) is
responsible for developing and managing a test strategy. The members of the
development and testing team help the test manager develop the test strategy.

Test Plan

The test plan should manifest the test strategy. The main purpose of having a test plan is
to organize the subsequent testing process. It includes test areas covered, test technique
implementation, test case and data selection, test results validation, test cycles, and entry
and exit criteria based on coverage metrics. In general, the test plan should incorporate
both a high-level outline of which areas are to be tested and what methodologies are to be
used and a general description of test cases, including prerequisites, setup, execution, and
a description of what to look for in the test results. The high-level outline is useful for
administration, planning, and reporting, while the more detailed descriptions are meant to
make the test process go smoothly.

While not all testers like using test plans, they provide a number of benefits:

• Test plans provide a written record of what is to be done.

• Test plans allow project stakeholders to sign off on the intended testing effort.
This helps ensure that the stakeholders agree with and will continue to support the
test effort.
 • Test plans provide a way to measure progress. This allows testers to determine
whether they are on schedule, and also provides a concise way to report progress
to the stakeholders.
• Due to time and budget constraints, it is often impossible to test all components of
a software system. A test plan allows the analyst to succinctly record what the
testing priorities are.
• Test plans provide excellent documentation for testing subsequent releases—they
can be used to develop regression test suites and/or provide guidance to develop
new tests.

A test manager (or similar role) is responsible for developing and managing a test plan.
The development managers are also part of test plan development, since the schedules in
the test plan are closely tied to that of the development schedules.

Test Automation

Test automation provides automated support for the process of managing and executing
tests, especially for repeating past tests. All the tests developed for the system should be
collected into a test suite. Whenever the system changes, the suite of tests that correspond
to the changes or those that represent a set of regression tests can be run again to see if
the software behaves as expected. Test drivers or suite drivers support executing test
suites. Test drivers basically help in setup, execution, assertion, and teardown for each of
the tests. In addition to driving test execution, test automation requires some automated
mechanisms to generate test inputs and validate test results. The nature of test data
generation and test results validation largely depends on the software under test and on
the testing intentions of particular tests.

In addition to test automation development, stubs or scaffolding development is also

required. Test scaffolding provides some of the infrastructure needed in order to test
efficiently. White box testing mostly requires some software development to support
executing particular tests. This software establishes an environment around the test,
including states and values for data structures, runtime error injection, and acts as stubs
for some external components. Much of what scaffolding is for depends on the software
under test. However, as a best practice, it is preferable to separate the test data inputs
from the code that delivers them, typically by putting inputs in one or more separate data
files. This simplifies test maintenance and allows for reuse of test code. The members of
the testing team are responsible for test automation and supporting software development.
Typically, a member of the test team is dedicated to the development effort.

Test Environment
Testing requires the existence of a test environment. Establishing and managing a proper
test environment is critical to the efficiency and effectiveness of testing. For simple
application programs, the test environment generally consists of a single computer, but
for enterprise-level software systems, the test environment is much more complex, and
the software is usually closely coupled to the environment.

For security testing, it is often necessary for the tester to have more control over the
environment than in many other testing activities. This is because the tester must be able
to examine and manipulate software/environment interactions at a greater level of detail,
in search of weaknesses that could be exploited by an attacker. The tester must also be
able to control these interactions. The test environment should be isolated, especially if,
for example, a test technique produces potentially destructive results during test that
might invalidate the results of any concurrent test or other activity. Testing malware
(malicious software) can also be dangerous without strict isolation.
The test manager is responsible for coordinating test environment preparation. Depending
on the type of environment required, the members of the development, testing, and build
management teams are involved in test environment preparation.

Test Execution

Test execution involves running test cases developed for the system and reporting test
results. The first step in test execution is generally to validate the infrastructure needed
for running tests in the first place. This infrastructure primarily encompasses the test
environment and test automation, including stubs that might be needed to run individual
components, synthetic data used for testing or populating databases that the software
needs to run, and other applications that interact with the software.

The issues being sought are those that will prevent the software under test from being
executed or else cause it to fail for reasons not related to faults in the software itself.

The members of the test team are responsible for test execution and reporting.

Testing Techniques
Data-Flow Analysis

Data-flow analysis can be used to increase program understanding and to develop test
cases based on data flow within the program. The data-flow testing technique is based on
investigating the ways values are associated with variables and the ways that these
associations affect the execution of the program. Data-flow analysis focuses on
occurrences of variables, following paths from the definition (or initialization) of a
variable to its uses. The variable values may be used for computing values for defining
other variables or used as predicate variables to decide whether a predicate is true for
traversing a specific execution path. A data-flow analysis for an entire program involving
all variables and traversing all usage paths requires immense computational resources;
however, this technique can be applied for select variables.

The simplest approach is to validate the usage of select sets of variables by executing
a path that starts with definition and ends at uses of the definition. The path and the usage
of the data can help in identifying suspicious code blocks and in developing test cases to
validate the runtime behavior of the software. For example, for a chosen data definition-
to-use path, with well-crafted test data, testing can uncover time-of-check-to-time-of-use
(TOCTTOU) flaws. The ”Security Testing” section in [Howard 02] explains the data
mutation technique, which deals with perturbing environment data. The same technique
can be applied to internal data as well, with the help of data-flow analysis.

Code-Based Fault Injection

The fault injection technique perturbs program states by injecting software source code to
force changes into the state of the program as it executes. Instrumentation is the process
of non-intrusively inserting code into the software that is being analyzed and then
compiling and executing the modified (or instrumented) software. Assertions are added to
the code to raise a flag when a violation condition is encountered. This form of testing
measures how software behaves when it is forced into anomalous circumstances.
Basically this technique forces non-normative behavior of the software, and the resulting
understanding can help determine whether a program has vulnerabilities that can lead to
security violations.

This technique can be used to force error conditions to exercise the error handling
code, change execution paths, input unexpected (or abnormal) data, change return
values, etc. In [Thompson 02], runtime fault injection is explained and advocated
over code-based fault injection methods. One of the drawbacks of code based
methods listed in the book is the lack of access to source code. However, in this
content area, the assumptions are that source code is available and that the testers
have the knowledge and expertise to understand the code for security implications.

Abuse Cases

Abuse cases help security testers view the software under test in the same light as
attackers do. Abuse cases capture the non-normative behavior of the system. While in
[McGraw 04c] abuse cases are described more as a design analysis technique than as a
white box testing technique, the same technique can be used to develop innovative and
effective test cases mirroring the way attackers would view the system. With access to the
source code, a tester is in a better position to quickly see where the weak spots are
compared to an outside attacker. The abuse case can also be applied to interactions
between components within the system to capture abnormal behavior, should a
component misbehave.

The technique can also be used to validate design decisions and assumptions. The
simplest, most practical method for creating abuse cases is usually through a process of
informed brainstorming, involving security, reliability, and subject matter expertise.
Known attack patterns form a rich source for developing abuse cases.

Trust Boundaries Mapping

Defining zones of varying trust in an application helps identify vulnerable areas of

communication and possible attack paths for security violations. Certain components of a
system have trust relationships (sometimes implicit, sometime explicit) with other parts
of the system. Some of these trust relationships offer ”trust elevation” possibilities—that
is, these components can escalate trust privileges of a user when data or control flow
cross internal boundaries from a region of less trust to a region of more trust [Hoglund
04]. For systems that have n-tier architecture or that rely on several third-party
components, the potential for missing trust validation checks is high, so drawing trust
boundaries becomes critical for such systems.

Drawing clear boundaries of trust on component interactions and identifying data

validation points (or chokepoints, as described in [Howard 02]) helps in validating those
chokepoints and testing some of the design assumptions behind trust relationships.
Combining trust zone mapping with data-flow analysis helps identify data that move
from one trust zone to another and whether data checkpoints are sufficient to prevent trust
elevation possibilities. This insight can be used to create effective test cases.

Code Coverage Analysis

Code coverage is an important type of test effectiveness measurement. Code coverage is

a way of determining which code statements or paths have been exercised during testing.
With respect to testing, coverage analysis helps in identifying areas of code not exercised
by a set of test cases. Alternatively, coverage analysis can also help in identifying
redundant test cases that do not increase coverage. During ad hoc testing (testing
performed without adhering to any specific test approach or process), coverage analysis
can greatly reduce the time to determine the code paths exercised and thus improve
understanding of code behavior.

There are various measures for coverage, such as path coverage, path testing,
statement coverage, multiple condition coverage, and function coverage. When planning
to use coverage analysis, establish the coverage measure and the minimum percentage of
coverage required. Many tools are available for code coverage analysis. It is important to
note that coverage analysis should be used to measure test coverage and should not be
used to create tests. After performing coverage analysis, if certain code paths or
statements were found to be not covered by the tests, the questions to ask are whether the
code path should be covered and why the tests missed those paths. A risk-based approach
should be employed to decide whether additional tests are required. Covering all the code
paths or statements does not guarantee that the software does not have faults; however,
the missed code paths or statements should definitely be inspected. One obvious risk is
that unexercised code will include Trojan horse functionality, whereby seemingly
innocuous code can carry out an attack. Less obvious (but more pervasive) is the risk that
unexercised code has serious bugs that can be leveraged into a successful attack.

Classes of Tests

Creating security tests other than ones that directly map to security specifications is
challenging, especially tests that intend to exercise the non-normative behavior of the
system. When creating such tests, it is helpful to view the software under test from
multiple angles, including the data the system is handling, the environment the system
will be operating in, the users of the software (including software components), the
options available to configure the system, and the error handling behavior of the system.

There is an obvious interaction and overlap between the different views; however,
treating each one with specific focus provides a unique perspective that is very helpful in
developing effective tests.

Data

All input data should be untrusted until proven otherwise, and all data must be validated
as it crosses the boundary between trusted and untrusted environments [Howard 02]. Data
sensitivity/criticality plays a big role in data-based testing; however, this does not imply
that other data can be ignored—non-sensitive data could allow a hacker to control a
system. When creating tests, it is important to test and observe the validity of data at
different points in the software.

Tests based on data and data flow should explore incorrectly formed data and stressing
the size of the data. The section ”Attacking with Data Mutation” in [Howard 02]
describes different properties of data and how to mutate data based on given properties.
To understand different attack patterns relevant to program input, refer to chapter six,
”Crafting (Malicious) Input,” in [Hoglund 04]. Tests should validate data from all
channels, including web inputs, databases, networks, file systems, and environment
variables. Risk analysis should guide the selection of tests and the data set to be
exercised.

Environment

Software can only be considered secure if it behaves securely under all operating
environments. The environment includes other systems, users, hardware, resources,
networks, etc. A common cause of software field failure is miscommunication between
the software and its environment [Whittaker 02]. Understanding the environment in
which the software operates, and the interactions between the software and its
environment, helps in uncovering vulnerable areas of the system. Understanding
dependency on external resources (memory, network bandwidth, databases, etc.) helps in
exploring the behavior of the software under different stress conditions. Another common
source of input to programs is environment variables.

If the environment variables can be manipulated, then they can have security
implications. In general, analyzing entities outside the direct control of the system
provides good insights in developing tests to ensure the robustness of the software under
test, given the dependencies.

Component Interfaces

Applications usually communicate with other software systems. Within an application,

components interface with each other to provide services and exchange data. Common
causes of failure at interfaces are misunderstanding of data usage, data lengths, data
validation, assumptions, trust relationships, etc. Understanding the interfaces exposed by
components is essential in exposing security bugs hidden in the interactions between
components. The need for such understanding and testing becomes paramount when
third-party software is used or when the source code is not available for a particular
component.

Another important benefit of understanding component interfaces is validation of

principles of compartmentalization. The basic idea behind compartmentalization is to
minimize the amount of damage that can be done to a system by breaking up the system
into as few units as possible while still isolating code that has security privileges. Test
cases can be developed to validate compartmentalization and to explore failure behavior
of components in the event of security violations and how the failure affects other
components.

Configuration

In many cases, software comes with various parameters set by default, possibly with no
regard for security. Often, functional testing is performed only with the default settings,
thus leaving sections of code related to non-default settings untested. Two main concerns
with configuration parameters with respect to security are storing sensitive data in
configuration files and configuration parameters changing the flow of execution paths.
For example, user privileges, user roles, or user passwords are stored in the configuration
files, which could be manipulated to elevate privilege, change roles, or access the system
as a valid user.

Configuration settings that change the path of execution could exercise vulnerable
code sections that were not developed with security in mind. The change of flow also
applies to cases where the settings are changed from one security level to another, where
the code sections are developed with security in mind. For example, changing an
endpoint from requiring authentication to not requiring authentication means the endpoint
can be accessed by everyone. When a system has multiple configurable options, testing
all combinations of configuration can be time consuming; however, with access to source
code, a risk-based approach can help in selecting combinations that have higher
probability in exposing security violations. In addition, coverage analysis should aid in
determining gaps in test coverage of code paths.

Error handling

The most neglected code paths during the testing process are error handling routines.
Error handling in this paper includes exception handling, error recovery, and fault
tolerance routines. Functionality tests are normally geared towards validating
requirements, which generally do not describe negative (or error) scenarios. Even when
negative functional tests are created, they don’t test for non-normative behavior or
extreme error conditions, which can have security implications. For example, functional
stress testing is not performed with an objective to break the system to expose security
vulnerability. Validating the error handling behavior of the system is critical during
security testing, especially subjecting the system to unusual and unexpected error
conditions. Unusual errors are those that have a low probability of occurrence during
normal usage.

Unexpected errors are those that are not explicitly specified in the design
specification, and the developers did not think of handling the error. For example, a
system call may throw an ”unable to load library” error, which may not be explicitly
listed in the design documentation as an error to be handled. All aspects of error handling
should be verified and validated, including error propagation, error observability, and
error recovery. Error propagation is how the errors are propagated through the call chain.
Error observability is how the error is identified and what parameters are passed as error
messages. Error recovery is getting back to a state conforming to specifications. For
example, return codes for errors may not be checked, leading to uninitialized variables
and garbage data in buffers; if the memory is manipulated before causing a failure, the
uninitialized memory may contain attacker-supplied data. Another common mistake to
look for is when sensitive information is included as part of the error messages.

Tools
Source Code Analysis

Source code analysis is the process of checking source code for coding problems based
on a fixed set of patterns or rules that might indicate possible security vulnerabilities.
Static analysis tools scan the source code and automatically detect errors that typically
pass through compilers and become latent problems. The strength of static analysis
depends on the fixed set of patterns or rules used by the tool; static analysis does not find
all security issues. The output of the static analysis tool still requires human evaluation.

For white box testing, the results of the source code analysis provide a useful insight
into the security posture of the application and aid in test case development. White box
testing should verify that the potential vulnerabilities uncovered by the static tool do not
lead to security violations. Some static analysis tools provide data-flow and control-flow
analysis support, which are useful during test case development.

Program Understanding Tools

In general, white box testers should have access to the same tools, documentation, and
environment as the developers and functional testers on the project do. In addition, tools
that aid in program understanding, such as software visualization, code navigation,
debugging, and disassembly tools, greatly enhance productivity during testing.

Coverage Analysis

Code coverage tools measure how thoroughly tests exercise programs. There are many
different coverage measures, including statement coverage, branch coverage, and
multiple-condition coverage. The coverage tool would modify the code to record the
statements executed and which expressions evaluate which way (the true case or the false
case of the expression).

Modification is done either to the source code or to the executable the compiler
generates. There are several commercial and freeware coverage tools available. Coverage
tool selection should be based on the type of coverage measure selected, the language of
the source code, the size of the program, and the ease of integration into the build
process.

Profiling

Profiling allows testers to learn where the software under test is spending most of its time
and the sequence of function calls as well. This information can show which pieces of the
software are slower than expected and which functions are called more often than
expected. From a security testing perspective, the knowledge of performance bottlenecks
help uncover vulnerable areas that are not apparent during static analysis. The call graph
produced by the profiling tool is helpful in program understanding.

Certain profiling tools also detect memory leaks and memory access errors (potential
sources of security violations). In general, the functional testing team or the development
team should have access to the profiling tool, and the security testers should use the same
tool to understand the dynamic behavior of the software under test.

Results to Expect
Any security testing method aims to ensure that the software under test meets the security
goals of the system and is robust and resistant to malicious attacks. Security testing
involves taking two diverse approaches: one, testing security mechanisms to ensure that
their functionality is properly implemented; and two, performing risk-based security
testing motivated by understanding and simulating the attacker’s approach. White box
security testing follows both these approaches and uncovers programming and
implementation errors. The types of errors uncovered during white box testing are several
and are very context sensitive to the software under test. Some examples of errors
uncovered include

• data inputs compromising security

• sensitive data being exposed to unauthorized users
• improper control flows compromising security
• incorrect implementations of security functionality
• unintended software behavior that has security implications
• design flaws not apparent from the design specification
• boundary limitations not apparent at the interface level

White box testing greatly enhances overall test effectiveness and test coverage. It can
greatly improve productivity in uncovering bugs that are hard to find with black box
testing or other testing methods alone.

Business Case

The goal of security testing is to ensure the robustness of the software under test, even in
the presence of a malicious attack. The designers and the specification might outline a
secure design, the developers might be diligent and write secure code, but it’s the testing
process that determines whether the software is secure in the real world. Testing is an
essential form of assurance. Testing is laborious, time consuming, and expensive, so the
choice of testing (black box, or white box, or a combination) should be based on the risks
to the system. Risk analysis provides the right context and information to make tradeoffs
between time and effort to achieve test effectiveness.

White box testing is typically very effective in validating design decisions and
assumptions and finding programming errors and implementation errors in software. For
example, an application may use cryptography to secure data from specific threats, but an
implementation error such as a poor key management technique can still leave the
application vulnerable to security violations. White box testing can uncover such
implementation errors.
 Benefits

There are many benefits to white box testing, including the following:

• Analyzing source code and developing tests based on the implementation details
enables testers to find programming errors quickly. For example, a white box
tester looking at the implementation can quickly uncover a way, say, through an
error handling mechanism, to expose secret data processed by a component.
Finding such vulnerabilities through black box testing require comparatively more
effort than found through white box testing. This increases the productivity of
testing effort.

• Executing some (hard to set up) black box tests as white box tests reduces
complexity in test setup and execution. For example, to drive a specific input into
a component, buried inside the software, may require elaborate setup for black
box testing but may be done more directly through white box testing by isolating
the component and testing it on its own. This reduces the overall cost (in terms of
time and effort) required to perform such tests.
• Validating design decisions and assumptions quickly through white box testing
increases effectiveness. The design specification may outline a secure design, but
the implementation may not exactly capture the design intent. For example, a
design might outline the use of protected channels for data transfer between two
components, but the implementation may be using an unprotected method for
temporary storage before the data transfer. This increases the productivity of
testing effort.
• Finding ”unintended” features can be quicker during white box testing. Security
testing is not just about finding vulnerabilities in the intended functionality of the
software but also about examining unintended functionality introduced during
implementation. Having access to the source code improves understanding and
uncovering the additional unintended behavior of the software. For example, a
component may have additional functions exposed to support interactions with
some other component, but the same functions may be used to expose protected
data from a third component. Depending on the nature of the ”unintended”
functionality, it may require a lot more effort to uncover such problems through
black box testing alone.

Costs

The main cost drivers for white box testing are the following:
 • specialized skill requirements: White box testing is knowledge intensive. White
box testers should not only know how to analyze code for security issues but also
understand different tools and techniques to test the software. Security testing is
not just validating designed functionality but also proving that the defensive
mechanisms work correctly. This requires invaluable experience and expertise.
Testers who can perform such tasks are expensive and hard to get.
• support software development and tools: White box testing requires
development of support software and tools to perform testing. Both the support
software and the tools are largely based on the context of the software under test
and the type of test technique employed. The type of tools used includes program
understanding tools, coverage tools, fault injection tools, and source code
analyzers.
• analysis and testing time: White box testing is time consuming, especially when
applied to the whole system. Analyzing design and source code in detail for
security testing is time consuming, but is an essential part of white box testing.
Tools (source code analyzers, debuggers, etc.) and program understanding
techniques (flow graphs, data-flow graphs, etc.) help in speeding up analysis.
White box testing directly identifies implementation bugs, but whether the bugs
can be exploited requires further analysis work. The consequences of failure help
determine the amount of testing time and effort dedicated to certain areas.

Alternatives

There are alternatives to white box testing:

• White box testing can complement black box testing to increase overall test
effectiveness. Based on risk assessment, certain areas of the software may require
more scrutiny than others. White box testing could be performed for specific high-
risk areas, and black box testing could be performed for the whole system. By
complementing the two testing methods, more tests can be developed, focusing on
both implementation issues and usage issues.

• Gray box testing can be used to combine both white box and black box testing
methods in a powerful way. In a typical case, white box analysis is used to find
vulnerable areas, and black box testing is then used to develop working attacks
against these areas. The white box analysis increases productivity in finding
vulnerable areas, while the black box testing method of driving data inputs
decreases the cost of test setup and test execution.

All testing methods can reveal possible software risks and potential exploits. White box
testing directly identifies more bugs in the software. White box testing is time consuming
and expensive and requires specialized skills. As with any testing method, white box
testing has benefits, associated costs, and alternatives. An effective testing approach
balances efficiency and effectiveness in order to identify the greatest number of critical
defects for the least cost.

Conclusion

White box testing for security is useful and effective. It should follow a risk-based
approach to balance the testing effort with consequences of software failure. Architectural
and design-level risk analysis provide the right context to plan and perform white box
testing. White box testing can be used with black box testing to improve overall test
effectiveness. It uncovers programming and implementation errors.

This paper introduces a risk-based approach and tools and techniques applicable to white
box testing for security. Concepts and knowledge from two areas, traditional white box
testing and security-based testing, were brought together. Other content areas on this web
portal discuss different aspects of software security in detail. To gain more in-depth
understanding of white box testing, readers are urged to supplement the knowledge
gained here with other related areas available on this web site. The links to related topics
are given in the Links section.

Unit Testing
Introduction

In computer programming, unit testing is a procedure used to validate that individual

units of source code are working properly. A unit is the smallest testable part of an
application. In procedural programming a unit may be an individual program, function,
procedure, etc., while in object-oriented programming, the smallest unit is a method;
which may belong to a base/super class, abstract class or derived/child class.

Ideally, each test case is independent from the others; mock objects and test harnesses
can be used to assist testing a module in isolation. Unit testing is typically done by
developers and not by Software testers or end-users.

Contents:

 Benefits
 Separation of interface from implementation
 Limitations of unit testing
 Applications
  Six Rules of Unit Testing

Benefits

The goal of unit testing is to isolate each part of the program and show that the individual
parts are correct. A unit test provides a strict, written contract that the piece of code must
satisfy. As a result, it affords several benefits.

Facilitates change

Unit testing allows the programmer to refactor code at a later date, and make sure the
module still works correctly (i.e. regression testing). The procedure is to write test cases
for all functions and methods so that whenever a change causes a fault, it can be quickly
identified and fixed.

Simplifies Integration

Unit testing helps to eliminate uncertainty in the units themselves and can be used in a
bottom-up testing style approach. By testing the parts of a program first and then testing
the sum of its parts, integration testing becomes much easier.

A heavily debated matter exists in assessing the need to perform manual integration
testing. While an elaborate hierarchy of unit tests may seem to have achieved integration
testing, this presents a false sense of confidence since integration testing evaluates many
other objectives that can only be proven through the human factor.

Some argue that given a sufficient variety of test automation systems, integration
testing by a human test group is unnecessary. Realistically, the actual need will ultimately
depend upon the characteristics of the product being developed and its intended uses.
Additionally, the human or manual testing will greatly depend on the availability of
resources in the organization.

Readily-available unit tests make it easy for the programmer to check whether a piece
of code is still working properly. Good unit test design produces test cases that cover all
paths through the unit with attention paid to loop conditions.

In continuous unit testing environments, through the inherent practice of sustained

maintenance, unit tests will continue to accurately reflect the intended use of the
executable and code in the face of any change. Depending upon established development
practices and unit test coverage, up-to-the-second accuracy can be maintained.

Documentation
Unit testing provides a sort of living documentation of the system. Developers looking to
learn what functionality is provided by a unit and how to use it can look at the unit tests
to gain a basic understanding of the unit API.

Unit test cases embody characteristics that are critical to the success of the unit. These
characteristics can indicate appropriate/inappropriate use of a unit as well as negative
behaviors that are to be trapped by the unit. A unit test case, in and of itself, documents
these critical characteristics, although many software development environments do not
rely solely upon code to document the product in development.

On the other hand, ordinary narrative documentation is more susceptible to drifting

from the implementation of the program and will thus become outdated (e.g. design
changes, feature creep, relaxed practices to keep documents up to date).

Separation of interface from

implementation

Because some classes may have references to other classes, testing a class can frequently
spill over into testing another class. A common example of this is classes that depend on a
database: in order to test the class, the tester often writes code that interacts with the
database. This is a mistake, because a unit test should never go outside of its own class
boundary.

As a result, the software developer abstracts an interface around the database

connection, and then implements that interface with their own mock object. By
abstracting this necessary attachment from the code (temporarily reducing the net
effective coupling), the independent unit can be more thoroughly tested than may have
been previously achieved. This results in a higher quality unit that is also more
maintainable.

Limitations of unit testing

Testing, in general, cannot be expected to catch every error in the program. The same is
true for unit testing. By definition, it only tests the functionality of the units themselves.
Therefore, it may not catch integration errors, performance problems, or other system-
wide issues. Unit testing is more effective if it is used in conjunction with other software
testing activities.
Like all forms of software testing, unit tests can only show the presence of errors; it
cannot show the absence of errors.

Software testing is a combinatorial problem. For example, every boolean decision

statement requires at least two tests: one with an outcome of "true" and one with an
outcome of "false". As a result, for every line of code written, programmers often need 3
to 5 lines of test code. Therefore, it is unrealistic to test all possible input combinations
for any non-trivial piece of software without an automated characterization test
generation tool such as JUnit Factory used with Java code or many of the tools listed in
List of unit testing frameworks.

To obtain the intended benefits from unit testing, a rigorous sense of discipline is needed
throughout the software development process. It is essential to keep careful records, not
only of the tests that have been performed, but also of all changes that have been made to
the source code of this or any other unit in the software. Use of a version control system
is essential. If a later version of the unit fails a particular test that it had previously
passed, the version-control software can provide a list of the source code changes (if any)
that have been applied to the unit since that time.

Applications
Extreme Programming

Unit testing is the cornerstone of Extreme Programming (XP), which relies on an

automated unit testing framework. This automated unit testing framework can be either
third party, e.g. xUnit, or created within the development group.

Extreme Programming uses the creation of unit tests for test-driven development. The
developer writes a unit test that exposes either a software requirement or a defect. This
test will fail because either the requirement isn't implemented yet, or because it
intentionally exposes a defect in the existing code. Then, the developer writes the
simplest code to make the test, along with other tests, pass.

All classes in the system are unit tested. Developers release unit testing code to the code
repository in conjunction with the code it tests. XP's thorough unit testing allows the
benefits mentioned above, such as simpler and more confident code development and
refactoring, simplified code integration, accurate documentation, and more modular
designs. These unit tests are also constantly run as a form of regression test.

It is also essential to implement a sustainable process for ensuring that test case failures
are reviewed daily and addressed immediately. If such a process is not implemented and
ingrained into the team's workflow, the application will evolve out of sync with the unit
test suite—- increasing false positives and reducing the effectiveness of the test suite.

Techniques

Unit testing is commonly automated, but may still be performed manually. The IEEE
does not favor one over the other. A manual approach to unit testing may employ a step-
by-step instructional document. Nevertheless, the objective in unit testing is to isolate a
unit and validate its correctness. Automation is efficient for achieving this, and enables
the many benefits listed in this article. Conversely, if not planned carefully, a careless
manual unit test case may execute as an integration test case that involves many software
components, and thus preclude the achievement of most if not all of the goals established
for unit testing.

Under the automated approach, to fully realize the effect of isolation, the unit or code
body subjected to the unit test is executed within a framework outside of its natural
environment, that is, outside of the product or calling context for which it was originally
created. Testing in an isolated manner has the benefit of revealing unnecessary
dependencies between the code being tested and other units or data spaces in the product.
These dependencies can then be eliminated.

Using an automation framework, the developer codes criteria into the test to verify the
correctness of the unit. During execution of the test cases, the framework logs those that
fail any criterion. Many frameworks will also automatically flag and report in a summary
these failed test cases. Depending upon the severity of a failure, the framework may halt
subsequent testing.

As a consequence, unit testing is traditionally a motivator for programmers to create

decoupled and cohesive code bodies. This practice promotes healthy habits in software
development. Design patterns, unit testing, and refactoring often work together so that the
most ideal solution may emerge.

Unit testing frameworks

Unit testing frameworks, which help simplify the process of unit testing, have been
developed for a wide variety of languages. It is generally possible to perform unit testing
without the support of specific framework by writing client code that exercises the units
under test and uses assertion, exception, or early exit mechanisms to signal failure.

This approach is valuable in that there is a non-negligible barrier to the adoption of

unit testing. However, it is also limited in that many advanced features of a proper
framework are missing or must be hand-coded.

Six Rules of Unit Testing

Six Rules of Unit Testing

1. Write the test first

2. Never write a test that succeeds the first time
3. Start with the null case, or something that doesn't work
4. Don't be afraid of doing something trivial to make the test work
5. Loose coupling and testability go hand in hand
6. Use mock objects

Write the test first

This is the Extreme Programming maxim. First you write the test, and enough application
code that the test will compile (but no more!). Then you run the test to prove it fails (see
point two, below). Then you write just enough code that the test is successful (see point
four, below). Then you write another test.

The benefits of this approach come from the way it makes you approach the code you
are writing. Every bit of your code becomes goal-oriented. Why am I writing this line of
code? I'm writing it so that this test runs. What do I have to do to make the test run? I
have to write this line of code. You are always writing something that pushes your
program towards being fully functional. In addition, writing the test first means that you
have to decide how to make your code testable before you start coding it. Because you
can't write anything before you've got a test to cover it, you don't write any code that isn't
testable.
Never write a test that succeeds the first time

After you've written your test, run it immediately. It should fail. The essence of science is
falsifiability. Writing a test that works first time proves nothing. It is not the green bar of
success that proves your test, it is the process of the red bar turning green. Whenever I
write a test that runs correctly the first time, I am suspicious of it. No code works right
the first time.

Start with the null case, or something that doesn't work

Where to start is often a stumbling point. When you're thinking of the first test to run on a
method, pick something simple and trivial. Is there a circumstance in which the method
should return null, or an empty collection, or an empty array? Test that case first. Is your
method looking up something in a database? Then test what happens if you look for
something that isn't there.

Often, these are the simplest tests to write, and they give you a good starting-point
from which to launch into more complex interactions. They get you off the mark.
Don't be afraid of doing something trivial to make the test work

So you've followed the advice in point 3, and written the following test:
public void testFindUsersByEmailNoMatch() {
assertEquals(
"nothing returned",
0,
new
UserRegistry().findUsersByEmail("not@in.database").length);
}

The obvious, smallest amount of code required to make this test run is:

public User[] findUsersByEmail(String address) {

return new User[0];
}

The natural reaction to writing code like that just to get the test to run is "But that's
cheating!". It's not cheating, because almost always, writing code that looks for a user and
sees he isn't there will be a waste of time - it'll be a natural extension of the code you
write when you actively start looking for users.

What you're really doing is proving that the test works, by adding the simple code and
changing the test from failure to success. Later, when you write
testFindUsersByEmailOneMatch and testFindUsersByEmailMultipleMatches, the
test will keep an eye on you and make sure that you don't change your behaviour in the
trivial cases - make sure you don't suddenly start throwing an exception instead, or return
null.

Together, points 3 and 4 combine to provide you with a bedrock of tests that make sure
you don't forget the trivial cases when you start dealing with the non-trivial ones.

Loose coupling and testability go hand in hand

When you're testing a method, you want the test to only be testing that method. You don't
want things to build up, or you'll be left with a maintenance nightmare. For example, if
you have a database-backed application then you have a set of unit tests that make sure
your database-access layer works. So you move up a layer and start testing the code that
talks to the access layer. You want to be able to control what the database layer is
producing. You may want to simulate a database failure.

So it's best to write your application in self-contained, loosely coupled components, and
have your tests be able to generate dummy components (see mock objects below) in order
to tests the way each component talks to each other. This also allows you to write one
part of the application and test it thoroughly, even when other parts that the component
you are writing will depend on don't exist.

Divide your application into components. Represent each component to the rest of the
application as an interface, and limit the extent of that interface as much as possible.
When one component needs to send information to another, consider implementing it as
an EventListener-like publish/subscribe relationship. You'll find all these things make
testing easier and not-so-coincidentally lead to more maintainable code.

Use mock objects

A mock object is an object that pretends to be a particular type, but is really just a sink,
recording the methods that have been called on it.

A mock object gives you more power when testing isolated components, because it gives
you a clear view of what one component does to another when they interact. You can
clearly see that yes, the component you're testing called "removeUser" on the user
registry component, and passed in an argument of "cmiller", without ever having to use a
real user registry component.

One useful application of mock objects comes when testing session EJBs, without the
hassle of going through the EJB container to do it. Here's a test class that checks a session
EJB correctly rolls back the containing transaction when something goes wrong. Notice
also how I'm passing a factory into the EJB - this is something that happens quite often
when you want to be able to alternate implementations between test time and
deployment.

Error Handling Testing

Definition

Error handling refers to the anticipation, detection, and resolution of programming,

application, and communications errors. Specialized programs, called error handlers, are
available for some applications. The best programs of this type forestall errors if possible,
recover from them when they occur without terminating the application, or (if all else
fails) gracefully terminate an affected application and save the error information to a log
file.

In programming, a development error is one that can be prevented. Such an error can
occur in syntax or logic. Syntax errors, which are typographical mistakes or improper use
of special characters, are handled by rigorous proofreading. Logic errors, also called
bugs, occur when executed code does not produce the expected or desired result. Logic
errors are best handled by meticulous program debugging. This can be an ongoing
process that involves, in addition to the traditional debugging routine, beta testing prior to
official release and customer feedback after official release.
A run-time error takes place during the execution of a program, and usually happens
because of adverse system parameters or invalid input data. An example is the lack of
sufficient memory to run an application or a memory conflict with another program. On
the Internet, run-time errors can result from electrical noise, various forms of malware or
an exceptionally heavy demand on a server. Run-time errors can be resolved, or their
impact minimized, by the use of error handler programs, by vigilance on the part of
network and server administrators, and by reasonable security countermeasures on the
part of Internet users.

Contents:

 Usage
 Objective
 How to Use?
 When to Use?
 Example

Usage

• It determines the ability of applications system to process the incorrect

transactions properly

• Errors encompass all unexpected conditions.

• In some system approx. 50% of programming effort will be devoted to handling
error condition.

Objective

• Determine Application system recognizes all expected error conditions.

• Determine Accountability of processing errors has been assigned and procedures

provide a high probability that errors will be properly corrected.
• Determine During correction process reasonable control is maintained over errors.

How to Use?
 • A group of knowledgeable people is required to anticipate what can go wrong in
the application system.

• It is needed that all the application knowledgeable people assemble to integrate

their knowledge of user area, auditing and error tracking.
• Then logical test error conditions should be created based on this assimilated
information.

When to Use?
• Throughout SDLC.

• Impact from errors should be identified and should be corrected to reduce the
errors to acceptable level.
• Used to assist in error management process of system development and
maintenance.

Example
• Create a set of erroneous transactions and enter them into the application system
then find out whether the system is able to identify the problems.

• Using iterative testing enters transactions and trap errors. Correct them. Then
enter transactions with errors, which were not present in the system earlier.

Manual Testing
Introduction

Manual testing is the oldest and most rigorous type of software testing. Manual testing
requires a tester to perform manual test operations on the test software without the help of
Test automation. Manual testing is a laborious activity that requires the tester to possess a
certain set of qualities; to be patient, observant, speculative, creative, innovative, open-
minded, resourceful, unopinionated, and skillful.

Contents:

 Overview
 Steps for Manual Testing
 Usage
  How to Use?

Overview

As a tester, it is always advisable to use manual white box testing and black-box testing
techniques on the test software. Manual testing helps discover and record any software
bugs or discrepencies related to the functionality of the product.

Manual testing can be replaced by test automation. It is possible to record and

playback manual steps and write automated test script(s) using Test automation tools.
Although, test automation tools will only help execute test scripts written primarily for
executing a particular specification and functionality. Test automation tools lack the
ability of decision-making and recording any unscripted discrepancies during program
execution. It is recommended that one should perform manual testing of the entire
product at least a couple of times before actually deciding to automate the more mundane
activities of the product.

Manual testing helps discover defects related to the usability testing and GUI testing area.
While performing manual tests the software application can be validated whether it meets
the various standards defined for effective and efficient usage and accessibility. For
example, the standard location of the OK button on a screen is on the left and of
CANCEL button on the right. During manual testing you might discover that on some
screen, it is not. This is a new defect related to the usability of the screen. In addition,
there could be many cases where the GUI is not displayed correctly and the basic
functionality of the program is correct. Such bugs are not detectable using test automation
tools.

Repetitive manual testing can be difficult to perform on large software applications or

applications having very large dataset coverage. This drawback is compensated for by
using manual black-box testing techniques including equivalence partitioning and
boundary value analysis. Using which, the vast dataset specifications can be divided and
converted into a more manageable and achievable set of test suites.

Steps for Manual Testing

A manual tester would typically perform the following steps for manual testing:

1. Understand the functionality of program

2. Prepare a test environment
3. Execute test case(s) manually
 4. Verify the actual result

1. Record the result as Pass or Fail

2. Make a summary report of the Pass and Fail test cases
3. Publish the report
4. Record any new defects uncovered during the test case execution

Usage
Usage:

• It involves testing of all the functions performed by the people while preparing
the data and using these data from automated system.

Objective:

• Verify manual support documents and procedures are correct.

• Determine Manual support responsibility is correct.
• Determine Manual support people are adequately trained.
• Determine Manual support and automated segment are properly interfaced.

How to Use?
How to Use:

• Process evaluated in all segments of SDLC.

• Execution of the can be done in conjunction with normal system testing.
• Instead of preparing, execution and entering actual test transactions the clerical
and supervisory personnel can use the results of processing from application
system.
• To test people it requires testing the interface between the people and application
system.

When to Use:

• Verification that manual systems function properly should be conducted

throughout the SDLC.
• Should not be done at later stages of SDLC.
• Best done at installation stage so that the clerical people do not get used to the
actual system just before system goes to production.
 Example

• Provide input personnel with the type of information they would normally receive
from their customers and then have them transcribe that information and enter it
in the computer.
• Users can be provided a series of test conditions and then asked to respond to
those conditions. Conducted in this manner, manual support testing is like an
examination in which the users are asked to obtain the answer from the
procedures and manuals available to them.