Compuware Best Practices For Testing

Compuware Best Practices for Testing
Compuware
RESTRICTED RIGHTS NOTICE SHORT FORM
(January, 2000)
Use, duplication, or disclosure by the Government is subject to the restrictions as
set forth in subparagraph (c)(1)(ii) of the rights in Technical Data and Computer Software clause
at 52.227-7013.
Copyright 2000 by Compuware Corporation.
All rights reserved. No part of this document covered by the copyright hereon may be copied or
reproduced by any meansgraphic, electronic or mechanical, including photocopying,
recording, taping or in information storage and retrieval systemswithout written permission
from the publisher.
Compuware, Abend-AID, File-AID, QACenter, QADirector, QAHiperstation, QALoad, QARun,
QualityPoint and XPEDITER are trademarks or registered trademarks of Compuware
Corporation. All other company or product names are trademarks of their respective owners.
2000 Compuware Corporation.

Compuware 2000 All Rights Reserved
Do not copy or reproduce
Chapter 1 - Overview................................................................................................... 1-1

Purpose of this Guide ........................................................................................................ 1-1
Audience ............................................................................................................................ 1-1
Scope .................................................................................................................................. 1-1
Chapter 2 - Initiating Testing...................................................................................... 2-1

Developing a Successful Testing Project........................................................................... 2-1
Software Correctness ........................................................................................................ 2-1
Chapter 3 - Test Planning........................................................................................... 3-1

Establishing a Test Strategy.............................................................................................. 3-1
Other Test Strategy Considerations................................................................................ 3-10
Test Types........................................................................................................................ 3-13
Roles and Responsibilities ............................................................................................... 3-16
Open Items....................................................................................................................... 3-17
Obtaining Test Strategy Sign-Off ................................................................................... 3-17
Chapter 4 - Creating Test Plans.................................................................................. 4-1

Introduction....................................................................................................................... 4-1
Establishing a Test Plan .................................................................................................... 4-1
Other Test Plan Considerations ........................................................................................ 4-9
Chapter 5 - Test Case Development ............................................................................ 5-1

Create Decision Tree ......................................................................................................... 5-1
Prioritize the Test Cases.................................................................................................... 5-2
Document Test Procedures ............................................................................................... 5-2
Document Test Data Requirements .................................................................................. 5-2
Identify Candidates for Automation................................................................................. 5-2
Automate Designated Test Cases ...................................................................................... 5-3
Chapter 6 - Test Environment Preparation................................................................. 6-1

Document Initial Environmental Data.............................................................................. 6-1
Document the Test Environment Configuration .............................................................. 6-2
Identify Data Recovery Points........................................................................................... 6-2
Establish Backup, Restoration and Construction Procedures ......................................... 6-2
Create Initial Data State.................................................................................................... 6-3
Chapter 7 - Test Execution ......................................................................................... 7-1

Review Test Execution Schedule from Test Plan.............................................................. 7-1
Restore Test Execution Environment ............................................................................... 7-1
Execute Test Cases ............................................................................................................ 7-2
Chapter 8 - Test Results Analysis................................................................................ 8-1

Perform a Flow Analysis ................................................................................................... 8-1
Failure Discovery............................................................................................................... 8-2
Success Discovery .............................................................................................................. 8-2
Measurement Analysis ...................................................................................................... 8-3
Chapter 9 - Management Reporting............................................................................ 9-1

Executive Summary........................................................................................................... 9-1
Consolidate Measurement Reports ................................................................................... 9-2
Chapter 10 - Productivity Tools and Professional Services....................................... 10-1

Introduction..................................................................................................................... 10-1
NuMega Tools for Quality Windows Applications......................................................... 10-3
Automated Testing Products........................................................................................... 10-5
Interactive Analysis, Testing and Debugging Tools ..................................................... 10-14
File and Data Management Tools ................................................................................. 10-22
Fault Management Tools............................................................................................... 10-29
Professional Services ..................................................................................................... 10-43
Attachment A Test Strategy Template ............................................................................ 1

Attachment B Test Plan Template.................................................................................. 7
Attachment C Test Environment Requirements........................................................... 15
Attachment D Sample Test Case .................................................................................. 25
Attachment E Sample Application Readiness Report................................................... 31
Attachment F Sample Test Summary Report ............................................................... 37
Attachment G Test Planning KPA ............................................................................... 41
Attachment H Test Case Development KPA ................................................................ 43
Attachment I Test Environment Preparation KPA ...................................................... 45
Attachment J Test Execution KPA............................................................................... 47
Attachment K Test Results Analysis KPA .................................................................... 49
Attachment L Management Reporting KPA ................................................................ 51

Chapter 1 - Overview
Purpose of this Guide
Compuware Best Practices for Testing will help an organization implement a
well-structured, repeatable, highly efficient testing process.
It contains information pertaining to the activities performed in a typical testing
process, how to establish this process and references to many Compuware
products that can help you automate a particular activity, including examples. The
guides objective is to help you put a successful testing process in place.
Audience
This guide is a working reference for quality assurance managers, application
managers, testing architects and project managers.
Scope
While there are many tasks in a typical testing project, Compuware Best Practices
for Testing focuses explicitly on the issue of building a repeatable testing process.
Other tasks, such as assessing current testing processes, building awareness and
estimating size and budget, while important, are not within the scope of this
document. Those tasks may be briefly described and referenced but are not
examined in detail.
Note that while this guide recommends a set of processes, it does not advocate
this as the only way to test. Compuware advocates a standard testing process,
centered on our QualityPoint testing methodology, which is the basis for much of
the material in this guide. You can use this guide to enhance, complete or simply
benchmark your current testing process.

1-1
Chapter 2 Initiating Testing
Chapter 2 - Initiating Testing

Developing a Successful Testing Project
One key to a successful software implementation is using software testing processes
that incorporate appropriate verification and validation activities; these ensure that
the software works when it is implemented. Another key is having project
management processes and project team infrastructure that effectively identify,
manage and control testing activities, as well as provide effective communications
about the application's readiness throughout the entire testing effort.
Using the Compuware QualityPoint software testing process has a direct impact on
these two key success criteria. QualityPoint incorporates software testing processes
and establishes a framework that directly addresses key characteristics of successful
software projects. It ensures you have the correct software and an effective project
management system.
The items below describe how QualityPoint directly impacts the areas of software
correctness and software testing project management.
Software Correctness
Business Requirements
Business requirements are the actions or results necessary for a viable product and
for implementation. Business and technical requirements are supported by the testing
effort.
Business requirements describe what an application must do and how it will perform,
but do not specify how it should be implemented. Define business requirements for
the software product or implementation during the first stage of software
development. These requirements must be testable and traceable throughout product
development stages. A testing process that does not explicitly support and link to

2-1
the business requirements can lead to results that do not meet the organization's
expectations. This means that resources were expended on testing activities that
did not support the requirements.
Technical Requirements
Technical requirements describe both the environment and computer system
architecture in which the application must perform and integrate, as well as the
performance requirements of the system itself. Defining the technical requirements
for the software product or implementation is performed during the first stage of
software development. Technical requirements, like business requirements, must also
be testable and traceable throughout product development stages, and similar
implications can result if the testing process does not explicitly support and link to
the technical requirements. Implications that could arise include:
1. Results that do not meet the organization's computer system architecture and
performance expectations.
2. Resources that are expended on testing activities that are not in support of the
requirements.
3. Testing environments that are not adequate to validate the technical
requirements.
4. Inadequate production environment and infrastructure implementation planning.
There are different types of technical requirements.
Environment Rules
Environment rules are the system infrastructure required to implement the
application. These may include server and personal computer hardware configuration
(memory, video, processor storage needs), software local and network operating
systems and drivers, database storage and disk space requirements. For example:
the application must run on both the Microsoft Windows 95 and Windows
NT 4.0 operating systems
the application must run on local workstations that have a minimum of 350
MHz processors and a minimum of 64 MB memory
the application's data repository must reside within an Oracle 8.0 database.
Performance Rules
Performance rules measure specific system characteristics such as network
utilization, processing speed, response time, resource consumption, throughput and
efficiency. The following are examples of performance rules technical
requirements:
2-2

Chapter 2- Initiating Testing
during peak performance hours (8:00 a.m. through 5:00 p.m.), server CPU
activity must not exceed 80%
screen changes within the application must not exceed five seconds.
Test Requirements
Business and technical requirements must be broken down by application
functionality and tested to validate these requirements. The final product is a test
requirement.
For example, if a business requirement states that the system must allow for the
addition of new employees, then appropriate test requirements for this business
requirement may include:
add a new U.S. employee via the main menu

add a new employee from the Current Employee List screen.
Each test requirement may produce one or more test cases.

A process that does not thoroughly address the scope of the test requirements as well
as ensure that the test requirements are linked to the business and technical
requirements can lead to such problems as:
business and technical requirements unfulfilled because they were not

completely tested
resources spent on unnecessary or nonexistent test requirementsineffective

use of resources.
Furthermore, since test cases are derived from the test requirements, incomplete test
requirements can also lead to the development of an incomplete list of test cases.
A test case is a test procedure combined with a single data record from the data set.
The test case is designed to fulfill the test requirements. A comprehensive testing
process ensures that both the test case types and quantity are known based upon the
links between the requirements and the understanding of the test approach.
Furthermore, these links help ensure that the tests support business and technical
requirements. Without this information and process, you may encounter such
problems as:
inability to scope and identify the resources necessary for test execution
what has to be tested and how long it will take to complete the testing
an incomplete list of test casesnot all business and technical requirements
may be fully tested.
Software Metrics
Metrics are used in almost all disciplines to determine how well a process works.
Within software correctness, metrics objectively evaluate progress toward meeting
2-3
the business and technical requirements and goals. Without objectivity, it is difficult
to assess how much progress has been made, how much is left to do and potential
problems or risks to address.
A metric is a number showing a relationship between two variables. A testing metric
is a number showing a relationship between two testing variables. The number must
then be compared with some standard or norm.
An example of a testing metric is the number of defects found during a specific test
phase versus total number of system defects. This metric shows the percentage of
defects identified as a result of testing. It measures test effectiveness.
Measurements are the objective data used to evaluate the process or performance.
The lack of correct measurements makes it difficult to identify areas for
improvement. If data is not, or cannot be measured, it is difficult to assess if
improvement efforts have been successful.
Measurements are a combination of metrics, and when analyzed together they
answer a specific question. For example, the percentage of defects identified, as a
result of testing, is a metric measuring test effectiveness.
Software Testing Measurements

Consider all of the following software testing measurements together when making
decisions about the readiness of the application. These are only some of the
measurements you can use, so make sure that you choose measurements appropriate
to your testing situation.
Test Coverage
Application
How many requirements were tested? The answer does not indicate how many tests
passed or failed. This can be shown as a percentage or as a trend.
Metrics
number of test requirements

number of business and technical requirements
number of test cases per test requirement.
Implementation
1. Establish a requirements matrix with links from the business and technical
requirements to the test cases.
2-4

2. Establish a counting scheme for business and technical requirements, test

requirements and test cases.
3. Execute the tests.
4. Count total number of test requirements.
5. Count total number of test cases per test requirement.
6. Count total number of test cases executed per test requirement.
7. Calculate the test requirement coverage percentage by dividing the number of
test cases executed per test requirement by the total number of test cases per test
requirement. This calculation can also be performed with the business and
technical requirements.
8. To show trends, create a graph with the x-axis representing time and the y-axis
representing the test requirement coverage percentage.
9. Plot the percentage on the graph.
Interpretation
TE S T C O V E R AG E
35
30
PERCENTAGE
25
20
15
10
0
1
T IME
Test coverage is based on the number of requirements covered during the test
process. For trends, the line should slope upward over time. This means that the test
coverage approaches 100%.

2-5
Error Discovery
Application
When are defects detected and how many defects have been detected? The defects
may be sorted by severity or represented as a total.
Metrics
number of defects
severity of defects.
Implementation
1. Establish a discrepancy reporting procedure.

2. Classify defects by severity.
3. Count and log defects.
4. Create a graph with the x-axis representing time and the y-axis representing the
number of defects by severity or total.
5. Plot defects over time.
Interpretation
Error discovery is based on the discovery rate of errors throughout the test process.
ERROR DISCOVERY
35
NUMBER OF DEFECTS
30
25
20
15
10
0
1
TIME
The slope should decrease over time. This is especially true for defects classified
with the highest level of severity. You could decide to release the application if the
trend was toward lower priority defects. If there are many low or medium priority
defects found near the targeted release date, this may be enough to delay release of
the application.
2-6

Open Defects
Application
When are defects open and how many defects are open? The defects may be sorted
by severity or represented as a total.
Metrics
number of defects
severity of defects.
Implementation

4. Create a graph with the x-axis representing time and the y-axis representing the
number of defects by severity or total.
5. Plot defects over time.
Interpretation
Open defects are based on the discovery rate of errors throughout the test process.
OPEN DEFECTS
35
30
NUMBER
25
20
15
10
0
1
TIME
The slope should decrease over time. This is especially true for defects classified
with the highest level of severity. You could decide to release the application if the
trend was toward more low priority. If there are many low or medium priority
defects found near the targeted release date, this may be enough to delay release of
the application.
2-7
Testing Activity
Application
How much testing has occurred? This does not indicate how many tests have passed
or failed and can be represented as a percentage or displayed as a trend.
Metrics
number of test cases executed

number of available test cases.
Implementation
1. Establish a counting scheme for tracking test cases.

2. Count the number of test cases that have been executed.
3. Count the number of available test cases.
4. Calculate a percentage by taking the number of test cases executed and dividing
by the number of available test cases.
5. If a trend is being developed, create a graph with the x-axis representing time and
the y-axis representing the test requirement coverage percentage.
6. Plot the percentage.
Interpretation
Testing activity is based on the number of tests executed over the number of
available tests.
TEST ACTIVITY
35
30
PERCENTAGE
25
20
15
10
0
1
TIME
The trend should slope upward over time, and the percentage should approach 100.
2-8

Success Status
Application
How many tests have passed and how much of the application is working correctly?
Metrics
number of passed test cases

number of available test cases.
Implementation
1. Establish a counting scheme for tracking test cases.

2. Count the number of test cases that have passed.
3. Count the number of available test cases.
4. Calculate a percentage by taking the number of test cases passed and dividing by
the number of available test cases.
5. If a trend is being developed, create a graph with the x-axis representing time and
the y-axis representing the success status.
6. Plot the percentage.
Interpretation
Success status is based on the number of passed test cases over the number of
available tests.
SUCCESS STATUS
40
35
PERCENTAGE
30
25
20
15
10
0
1
TIME
The trend should slope upward over time, and the percentage should approach 100.
2-9
Specific Risks
Application
This measurement tells which tests have failed and which tests have importance in
regard to release of the application.
Metrics
number of open defects

severity of open defects.
Implementation
1. Establish a requirement matrix that has links from the business and technical
requirements to the test cases.
2. Establish a counting scheme for business and technical requirements, test
requirements and test cases.
6. Associate defects with the appropriate business or technical requirement.
7. Calculate a total specific risk percentage by taking the total number of open
defects and dividing it by the total number of business and technical
requirements.
8. If a trend is being developed, create a graph with an x-axis representing time and
the y-axis representing the specific risk percentage.
9. Plot the total specific risk percentage.
10. Start another graph.
11. Calculate a subtotal specific risk trend by taking the number of open defects by
severity and plotting the totals per requirement over time.
Interpretation
Total specific risks are based on the total number of open defects divided by the total
number of business and technical requirements. This total can be further broken
down into the number of open defects divided by a specific business requirement.
One more level of detail for specific risk can be obtained by dividing the number of
open defects of a specific severity by the specific business requirement. The
2-10

breakdown from the total specific risks is intended to provide trend information in
the form of gross number totals.
Total Risk by all Business Requirements
The desired result is for the trend to be sloping downward over time, and the
percentage should be approaching zero.
Severity by Business Requirement

2-11
Total Specific Risks Priority
The resulting trend of each chart should slope downward over time, and the
percentage should approach zero.
Software Testing Project Management

Cost
It is essential to understand all aspects of cost, even though the primary driver may
be labor hours. A comprehensive testing process consists of activities that identify all
potential resource costs including hardware, software, staff labor, contract labor and
any additional capital expenditure. These activities include, but are not limited to,
identifying the test approach, understanding the environmental needs, identifying the
test cases and determining which test cases are to be automated. Not incorporating
these types of activities within the testing process can lead to unexpected
expenditures or having to react and identify options when the funds are not available.
Schedule
The test schedule plans the order of activities and identifies what needs to happen
before and after testing. Using a comprehensive test approach identifies
dependencies. Then you can schedule the resources and test cases required to support
the test approach and fulfill requirements. Poor preparation during initial planning
can result in inability to properly execute and analyze test cases. Unexpected delays
result because time is needed to address the issues while resources remain idle,
waiting for direction.
Software testing projects typically use past project data to allocate requirement
resources for the testing effort. Depending on the data and the process you use, as
2-12

well as their relevancy to the current project, the schedule might be incomplete. The
primary purpose of having a schedule is to assist in tracking and communicating the
overall progress of the project, usually based upon milestones. The schedule also
provides other benefits by identifying needed resources and potential schedule risks.
Ensure that the amount of effort required does not exceed the amount of time
scheduled for the testing projects completion.
Therefore, to improve the schedule's accuracy, the test process must include all
activities required to complete testing. Test process includes test planning, test case
development, test environment preparation, test execution, test results analysis and
management reporting and also such factors as:
the software testing environment

tester skill levels
schedule commitments to the customer
corporate culture.
Without considering all aspects, it is highly probable that by the date the application
under test is to be implemented, not all testing will be completed since neither the
time nor the resources were appropriately allocated or the different types of testing
needed were not included. In this scenario, there is a high risk that the application
being tested will not meet requirements and will be defective at the time of its
implementation.
Resources
Resources may include specific hardware or machines, software or operating
systems, skilled testers, facilities such as a dedicated testing laboratory or network
connections and operational support.
Activities within the testing process that help identify both the resource types and
appropriateness are those that identify:
what types of testing are needed

what test cases must be executed
what environment is needed
what support tools are necessary to manage the project
what tools are necessary to automate testing.
The sooner you identify these resources, the sooner you can react to any gaps
(unavailability, skills) and the sooner you can either close the gaps (as much as
possible) or develop other plans. Not understanding the resources can lead to
problems such as:
the resources cannot execute the testing needed (the hardware cannot execute
the performance test requirements)

2-13
the people are not skilled in understanding test results; therefore, the results
may not be reliable and may not accurately reflect the testing progress
resources are not available when needed; alternative plans (training, capital
procurement and the like) to close any gaps cannot be executed within the
allotted time.
Specific types of resource attributes include:

Appropriateness
The "right" resources assigned to the "right" task. For example, are only 386
workstations available to test an application designed for a Pentium? The
appropriateness of skill levels is just as important; for example, are the people
assigned to write test cases trained to write them?
Hardware/Software
Hardware and software resources refer to the specific hardware configurations,
software requirements, infrastructure environmental needs and support tools needed
throughout the testing project. Areas of consideration:
will the hardware be available at test execution?

has the software been installed?
does the software require specific levels of service updates/releases?
Human
It is important to understand which people are needed, who is available and what
gaps need to be addressed. The types of attributes or characteristics to understand
include skills, experience, expertise, authority and responsibility.
Risk
Risk is the chance of an undesirable event or an item that compromises the project's
goals or requirements. Factors used to describe risk include the probability that the
event will occur and the impact of the event. According to the Software Engineering
Institute, risk management can be illustrated as a set of functions that are identified
throughout the life cycle of a project:
2-14
Identify. Locate risks to the project before they become problems.

Analyze. Evaluate the impact, probability and timeframe of the risk. Classify
the risks and then prioritize them.
Plan. Translate risk information into decisions and mitigating actions
(present and future) and implement those actions.
Track. Monitor the risks and mitigation actions.
Control. Check for and correct deviations from the mitigation plans.
Communicate. Provide information and feedback, both internal and external

to the project, on the risk activities.
Risk management should be an ongoing activity. The ability to accurately identify,

assess and manage the software testing risks relies heavily upon the completeness
and accuracy of the testing process and its supporting activities. The less you
understand about successfully planning, executing and completing any test process
component (test planning, execution, analysis and reporting), the less likely you will
identify or assess the risk properly. Without knowing the risks, it is increasingly
difficult to plan for what might go wrong, and therefore it is highly unlikely that the
application requirements will be completed within expected project cost and
schedule requirements.
Probability
Some testing activities can directly assist in assessing the likelihood that a risk will
occur. By understanding how much needs to be done, by when and by whom, you
can significantly improve probability assessment. For example, if it is known that
some of the resources needed for testing do not meet requirements, it is likely that
more time must be allotted for test case development and test execution. If resource
identification, needs and gap analysis did not exist during test planning, the
probability of having available skilled resources wasnt assessed properly.
If significant process changes are to be implemented, the probability of risks
occurring during the affected process is likely to be high. Not knowing if changes are
necessary and what they would be, however, affects a manager's ability to accurately
assess the associated risks.
Impact
Impact is another factor of riskthe assessment of what will be impacted and the
magnitude of the consequence. Again, some of the testing process can help define
what could be impacted (cost, effort, schedule, performance and correctness) and the
extent to which it is impacted (high, medium, low).
Consider the risk of skilled resources being unavailable. A comprehensive test
planning effort produces a complete test project schedule and a resource matrix,
identifying types of people and when they are needed. These tools provide the
manager with the opportunity to create different scenarios and estimate the impact of
schedule delays and the associated cost overruns among other factors. Consequently,
you can assess what might be impacted and the potential consequences.
Issue Resolution
It is essential to create and implement test processes that help identify problems.
Identify problems as early as possible so you can develop and monitor action plans
that address and minimize their impact. If problems occur late in the process, the
2-15
costs associated with addressing them may be much higher than if they were
identified sooner.
Similar to risk management, unidentified problems lead to unexpected results or
difficulties. This also leads to unfulfilled requirements or unmet expectations.
Communication
The primary objective of any communications material is to inform project personnel
of expected results and goals as well as progress towards those goals. Effective
communication must also be:
understandable by the audience

appropriate for the audience
relevant to the application under test
timely
accurate.
Goals
If both the management and the project members do not have a clear understanding
of project expectations, then the test goals and requirements may not be met. Project
personnel may not be aware of the key activities set by the management team on
which to measure progress. Attempting to align expectations late in the project's life
cycle can lead to resource waste and can put the schedule, resource effort and cost at
high risk.
Status Reporting
Understanding the test planning process, understanding progress towards goals and
interpreting results and measurements are critical in improving a manager's ability to
provide effective status reports.
Even though the goals may be understood, not providing timely, accurate and
appropriate status reporting can lead to:
2-16
Ineffective Risk Management and Issue Resolution. Unreliable,

incomplete or untimely information affects a manager's ability to identify
risks and problems, as well as creates necessary action plans that can be
executed in the time remaining.
Ineffective Resource Utilization. Inaccurate information can result in
unnecessary or inappropriate action plans being executed.

Organizational Support Issues. When not communicating, people have a

tendency to believe that information is "hidden" from them or that the
manager does not have the project under control which will potentially lead
to problems related to trust and confidence. Consequently, this can affect the
project manager's ability to provide additional resources for mitigation or
contingency planning.
Project Metrics
Metrics are as important in project management as they are in achieving correct
software. Project management measurements typically reflect progress toward
achievement of project management goals related to cost, schedule and effort.
The lack of correct measurements makes it difficult to identify areas for
improvement. If the process cannot be measured, how can improvement efforts be
considered successful and how can the magnitude of the improvements be assessed
objectively?
Trend Analysis
Trend analysis is the ability to analyze measurements over a period of time to help
assess issues, risks and progress. Data and trend analysis may only be pertinent for a
specific project, or the manager may be able to use earlier project data to assist in
trend analysis. For example, with historical rates of early defect detection, the team
may be able to forecast the defect detection rate on the current project.
Not performing any trend analysis can result in:
inability to identify risks and issues early

inability to develop appropriate action plans to address risks/issues as early as
possible
potential ineffective resource utilization since resources may have been used
for unnecessary activities if projections had been available.
Developing the Staffing, Workspace Environment, Facilities, Training

and External Support Plans
Staffing Plan
The staffing plan is a profile of the project team. This plan incorporates:
generic role definition

responsibilities
skills needed
quantity of resources required by role and skill
planned start and end dates of team personnel
planned cost per hour and actual cost per hour (optional).

2-17
You may develop an organization chart as supporting documentation. An

organization chart contains the project team structure, including support personnel,
such as application experts or technical staff. The chart should also indicate reporting
relationships.
Identifying and securing people to match the skills identified in the staffing plan is
crucial to the success of the project. An exact match is difficult to achieve because of
the unavailability of qualified resources, business economics and schedules. You
may need to adjust your project plan to accommodate replacements. If experienced
resources are not available, additional staff training may be required.
Roles and Responsibilities
The following paragraphs describe key positions in a testing team.
Application Owner
manages the end-users or is the end-user of the application under test (AUT)
serves as the customer for the AUT
referred to as Application Subject Matter Expert (SME) in some cases
Business Analyst/Lead Business Analyst
understands the business functions within the AUT

defines business processes that support the AUT
identifies business requirements
reviews design and specifications
defines the training and end-user documentation requirements
assists with test execution
assists in creating acceptance criteria
reviews and analyzes the test results to determine application correctness
represents the functional needs of the end-users to the project team
Business Consultant
2-18
provides QualityPoint project leadership, working with the project manager

to implement QualityPoint software testing processes
assists the Project Manager with the design, creation and implementation of
the QualityPoint processes
supplies project control information to the client and provides support for
Senior Management instituting the changes required as a result of
implementing QualityPoint
serves as a primary point of contact between the consultant's company and
the client

manages the consultant organization's resources assigned to the client's

QualityPoint project
Business Expert
understands the business perspective of the AUT

outlines the business processes that align with the AUT
provides the application details such as workflow, required elements on input
screens and other details
referred to as a Business Subject Matter Expert (SME) in some cases
Development Manager
controls the process for developing and implementing software

provides application-related information to the Business Consultant
oversees testing activities from an application perspective
assists in creating Business Requirements
assists in creating acceptance criteria
assists in identifying the informational flow process requirements between.
the development organization and the project team members
Project Manager
plans, integrates and executes activities to deliver products that meet the
acceptance criteria
manages resources within the project and ensures that team members
understand roles and the logical links among QualityPoint activities
maintains an effective flow of communications
develops, monitors and controls the QualityPoint project plan
provides organizational and logistical support while the Business Consultant
is at the client site
serves as a primary point of contact between the client's company and the
consultant's company
Project Sponsor
has the overall responsibility and accountability for the success of the
provides the communication link between the client's senior management

team and the project team
provides clarification of critical issues and establishes priorities
monitors project execution
defines project expectations

2-19
Quality Assurance Analyst(s), Lead Quality Assurance Analyst
provides leadership for testing effort

oversees testing activities from a technical perspective
assists in understanding the details of each software testing process and how
each process impacts the client
monitors testing activities
tracks and analyzes defects
develops and tracks project metrics and measurements
creates test reports
assists in creating acceptance criteria, test phase exit criteria and suspension
and resumption criteria
assists in test plan development
Quality Assurance Manager
provides testing process and change control process information to the

business consultant
establishes the software quality assurance goals
oversees software configuration management and release management
provides guidelines to support testing and quality practices
responsible for test execution and results sign-off
provides recommendations on how to proceed based upon the test results
analysis
approves test plan
Senior Management
appoints the project sponsor and project manager

reviews and approves the test process assessment statement of work,
management report, QualityPoint project plan and application readiness
report
identifies the initial project that will be used to implement the client's
QualityPoint software testing process
provides input on cost, resource and schedule limitations and constraints in
which the project must work
controls the environment factors that are beyond the control of the Project
Manager
establishes and communicates company direction and how the QualityPoint
project supports that direction. Provides support for any changes required for
a successful QualityPoint implementation
Team Manager, Team Leader

2-20

coordinates a defined subset of related project activities within the

Technical Analyst(s), Lead Technical Analyst
provides technical support for a specific application

works for the Technical Support manager
establishes physical setup of environment
Technical Expert
understands the infrastructure elements of the application as it is

implemented
understands the support components of the application such as network
configuration, networking protocols and others
referred to as Technical Subject Matter Expert (SME) in some cases
Technical Support Manager
controls the technical environment where the application under test will be
developed, tested and implemented
assists in identifying the test and production environment requirements and
processes
directs the resources required to build and maintain the test and production
environments
controls the environment backup and restoration procedures
Test Manager
provides testing process information to the Business Consultant

establishes the test goals
develops or approves the test plans
provides guidelines to support testing practices
responsible for test execution and results sign-off
provides recommendations on how to proceed based upon the test results
analysis
Testing Personnel
develops test plans

executes tests and evaluates execution results
logs and communicates test execution results
Workspace Environment
This section addresses many of the objects you must have in place before testing can
begin. These objects include the following:
2-21
Facilities
Where the project team will be working on a daily basis, and the facilities, such as
cubicle, telephone and security access card are available to each team member. Also
consider common facilities, such as fax machine, conference room, printer and
copier, that will be available to the team. If commonly needed facilities are not
available at the normal work site, determine how and where the team will gain access
to them when needed. Note any constraint on the size of the team that can be
accommodated at the work site. If the project team will normally be working at
multiple physical locations, identify any potential logistic problems.
Hardware
Identify the hardware, both mainframe and PC, you will need to conduct the project
and who will provide it. Include information such as model and size and the hours
and days of availability. List any known minimum requirements.
Software
Identify the software, both mainframe and PC, you will need to conduct the project
and who will provide it. Include information such as version, release and the number
of licenses and simultaneous users. This section may also include assumptions about
any known minimum requirements.
Communication
Identify the communication links you will need to conduct the project and who will
provide them. List any known requirements.
Facilities and Fixed Assets Plan
A schedule of events and list of items that the Facilities and Information Technology
departments require to prepare for the project.
The plan for facilities should include:
securing cubicles or offices

obtaining whiteboards
accessing copy machines and faxes
accessing conference rooms
obtaining phones and voice-mail
connecting to the LAN.
The plan for Information Technology should include:
2-22
project timing
software and hardware, including the configuration to support the workload
shared data areas

internal and external communications
electronic mail
access and special requirements.
Develop your facilities plan at the same time as your staffing and project plans.
Make special note of machine capacity. Your overall planning effort must ensure
enough capacity is available for planned work.
Defining Training Requirements
Your primary training objective is to effectively transfer skills and knowledge about
testing methods, concepts, procedures and tools to the people involved in testing.
You should design training so that the people using the tools daily can be productive
immediately. Tool training should occur prior to using the tool. If training occurs all
at once, students are overwhelmed and the information is lost.
Identify the Skill Sets of Testing Personnel
Why Is This Important?
Documenting the skill sets of the test personnel:
indicates whether the people have the skills, knowledge and capabilities
required for the activities assigned to them
indicates whether the people have remained current on software testing and
software quality assurance processes as they relate to current technologies,
platforms, information technology solutions and applications
indicates a company's commitment to a quality program and the continuous
.improvement of quality programs and initiatives.
Do those responsible for testing attend software testing and quality assurance
seminars/training? Are they members of a professional organization? Understanding
the types of software quality assurance and software testing training the project
personnel have attended, as well as relevant QA professional organizations that they
are involved in, indicates whether they are current on software quality assurance
processes.
A training budget indicates the company's commitment to quality and identifies
whether funds may be available for training if skill gaps exist.

2-23
How Does This Impact Quality?

If people do not have the skills, knowledge or experience necessary, their work could
make delivering correct software impossible. Examples of problems that can be
encountered include:
the requirements may not be complete or correct

inaccurate resolutions or long cycle times to complete resolution or both
the test cases may not be accurate or appropriate (test requirements are not
complete or correct, test procedures are incorrect, test data is not appropriate,
valid or correct)
the test results analysis may not be accurate (defects are incorrectly
categorized, assumptions that a passed test case equates to a valid result are
often made, data flow issues are not identified)
it will be difficult to identify appropriate metrics and measurements that
objectively and accurately reflect progress toward fulfillment of requirements
the test approach may be defined incorrectly. An incorrect test approach can
have the following implications:
high risk of planning inaccuracies (too much or too little testing in

planned)
incorrect environmental needs assessment
the types of people needed and when they are needed may not be assessed
correctly
the list of test cases may not be appropriate (they may not support the test
approach; the number could be too little or too much).
testing activities may proceed when issues and risks exist that should be
addressed
skill gaps not known can affect the ability to accurately assess the probability
associated with a risk; therefore, the appropriate effort and attention may not
be given to the monitoring, mitigation and contingency planning associated
with a particular risk. The result may be that you have to react quickly to an
earlier identified risk late in the project. The impact to schedule and
resources to meet the requirements may also be greater than what was
predicted.
Potential Recommendations
Establish a training program for test team members that includes training on the
application(s) under test, as well as the QualityPoint software testing process.
External Support Staff
A test team needs to rely on experts in the organization to assist with building the
testing process and with running and verifying test results.
2-24

Following is a partial list of key personnel you will need to assist your test team.
The Project Manager and his or her staff can provide assistance with the
sequencing of applications to be tested. The Project Manager can help with the
procurement of the necessary testing equipment, office space, work facilities,
computer resources, financing, human resources, outside services, tool acquisition
and other high level management tasks.
A Database Administrator (DBA) may be needed to assist with the generation and
population of test databases to assess each application. Your reliance on the DBA
will depend on the current availability of test databases. If new test databases are
required, the time, resources and effort to build them must be planned and scheduled.
The process of creating a subset of database data may also require the expertise of
the DBA to make the test team aware of application relationships between different
tables, databases and flat files.
Another important function of the DBA is to ensure that test databases are backed up
and refreshable. If there are database errors encountered during testing, the expertise
of the DBA can save a lot of time. A DBA will be able to quickly analyze, recognize
and resolve problems.
You may need the assistance of OLTP Technical Support (CICS and other). For
systems with on-line transaction processing (OLTP) components, the need for one or
more test versions of the on-line system is common. The effort to build a test on-line
system is often difficult and requires coordination between DBAs and the test team
to ensure that the correct data and program set are referenced during testing. Plan
and schedule sufficient time to build any new on-line test systems that are needed.
The DASD Administrator needs to be aware of your disk space requirements so
that sufficient space can be provided. For most mainframe shops, disk space is at a
premium. The space required for each application must be calculated. Due to the
large volume of DASD required for testing, someone needs to oversee or manage it.
Identify and plan interactions with your Security Administrator to expedite access
to needed data and resources. Include obtaining access to other platforms such as
midrange, intermediate systems and other servers. Delays in accessing needed
resources and files during the testing process should be eliminated.
The Technical Support person is responsible for installing and maintaining the
testing tools needed for your testing organization.
In a mainframe environment, the Technical Support department usually has systems
programmers who are responsible for all software installation. It is important to
notify Technical Support of any new tools or new releases of products you need for
your testing organization. Scheduling of technical support resources and special
installation requirements, like initial program loads (IPLs), may be difficult. Get
started early.
2-25
You must communicate with your Network Support personnel. Access to

communications facilities, local area network (LAN) servers and printers,
workstation configuration, electronic mail and other infrastructure needed to support
the test team must be available and reliable. Make sure these facilities are in place
and fully supported via a help desk or LAN expert.
You may need to get information on the contents, scheduling and frequency of
production jobs, or you may need to schedule special overnight test data builds. For
these tasks, involve the Operations department and Job Scheduling staff such as a
Scheduling Administrator, if a job scheduling package is used. Make sure they are
aware of your status as you begin to build the test process and as you start testing
each application.
2-26

Chapter 3-Test Planning
Chapter 3 - Test Planning

Effective test strategy and test plans are the foundation for successful testing. The
test strategy is created at the same time the application requirements are defined.
The test plan provides the details that, when implemented, will meet the
objectives outlined in the test strategy. Attachment B shows an example test plan.
The first purpose of test planning is to identify the items being tested, the testing
tasks to be performed, the personnel responsible for each task and the risks
associated with the plan. The second purpose is to develop the scope, approach,
resources and schedule of the testing activities. Thorough test planning is critical
to the success of the testing process.
Ideally, the test plan is developed at the time the application requirements are
defined. This provides the test team adequate time to define the tests, locate and
configure the hardware and software resources, locate and train the human
resource and schedule the tests. The test plan is a "living document" that will
change as the application functions become more clearly defined and stable.
The test team, the application development team, all user groups and management
review and approve the test strategy and test plan. Once the test strategy has been
completed, the test plan can be created.
The process for creating the test plan, along with the test plan itself, should be
reviewed and agreed to by the test team, the development team, the user/customer
team and ultimately, the management team. This step cannot be overemphasized
because it establishes commitment from each contributing team.
Establishing a Test Strategy

The following pages detail how to establish a test strategy, giving the objectives
of each area, and the reasons that it is important to define and document those
items. Attachment A shows a sample test strategy template.

3-1
Identifier
Objective
The identifier section allows an identification number to be assigned to the test
strategy document that supports your document naming conventions.
The identification number is used to cross-reference the test strategy document
with other documents in the project. A unique identifier makes the document
easier to recognize in a listing of documents.
Purpose
Objective
To identify the appropriate readership, communicate what the reader(s) should
expect to understand after reading the test strategy and provide a brief description
of the overall testing strategy. This is achieved by providing a brief description of
the document's content, the level of detail contained within and the application
that will be tested during the project's test effort.
This section provides an introduction for the test strategy and a background of the
project and the AUT. It provides an overview of the test strategy and helps set the
expectation for the amount of effort required for the project.
Goals of the Test Effort

Objective
The purpose of the goals section is to identify and document the goals that the
testing effort is to support and achieve.
Goals provide an overview of the force driving the project. These forces can be
the need to correct errors, add new functionality and increase current response
times. When the goals of the project are specified, testing efforts can be readily
tailored to fulfill and achieve those goals.
3-2

Scope of the Test Effort

Objective
The purpose of the test scope section is to document and identify the sections of
the AUT and the business and technical requirements that will and will not be
tested.
Documenting test scope aids in determining project timelines as well as
estimating effort. Once business and technical requirements have been defined,
key and related deliverables can be identified. After requirements have been
defined, the test objectives can be outlined and responsibilities can be assigned.
Efforts can then be focused on those tests that have a higher priority in reaching
the project's objectives.
Test Environment
Objective
The purpose of the test environment section is to document the environment in
which the tests are to be executed.
Documenting the physical components required for test execution helps identify
potential gaps in what is required and what actually exists; appropriate action
plans to obtain closure can then be developed. Consider such things as:
operating system version (remember to identify service packs, fix packs and
patches)
networking software and version
communications protocols
server and workstation machines.
References
Objective
The purpose of the reference section is to identify supplemental information that
might be helpful to the reader as well as a list of terminology that may be used
throughout the document and the test effort.

3-3

Documenting the references provides a way to define specific terminology and
acronyms. These terms can be used throughout all documents to provide a
common frame of reference. Using these terms helps to customize documents to
be specific to you and your needs.
The Test Effort's Organization Needs

Objective
To identify the type of personnel required to conduct the testing effort.
Why is This Important?
Identifies the required resources for the testing activities to be performed. These
activities can include test planning, test case creation, test script development, test
tool support, among others. Once the needed personnel are identified, individual
training needs can be determined and training programs can be created if the
available people do not meet the skill and knowledge levels required to meet the
expected schedule.
Test Types
Objective
To document the testing to be completed during this test effort.
Documenting the test types needed for this effort helps define the testing activity
that will be undertaken. It also provides you with some indication of the resources
and test environments required for the testing activity. Each of the test types
should be considered; however, not every type of test is typically employed. For
example, it is not necessary to do unit testing if the software is vendor-supplied.
Mapping the test types to the test goals assists in directing test activity and
validates that the testing is actually needed.
Test Methods
Objective
To document the methods of testing to be utilized during test execution.
3-4

Documenting the test methods to be used in the test executions helps define the
way testing activity will be performed. It also provides you with some indication
of whether testing tools will be required for the testing activity. Also, project
management tools can be identified while completing this section.
Testing Key Process Areas

Objective
To identify the QualityPoint testing key process areas (KPAs) that are applicable
for this testing effort. A brief description of each KPA and expected outputs are
also documented.
By reviewing brief outlines of each of the key process areas, you can determine
which of the areas best suits your needs. KPAs can be used as needed during the
project and can be used independently of each other to complete the entire project.
Acceptance Criteria
Objective
The purpose of the acceptance criteria section is to document what constitutes
acceptance of the application based on quantifiable measures.
The process helps identify criteria required to help ensure that both applications
meet any applicable contractual or regulatory requirements or both; for example,
food, pharmaceuticals and nuclear energy industries must meet Food and Drug
Administration regulations.
The criteria can be used to identify appropriate quality control activities and
measurements that should be included in the testing process.
Quality levels used to accept the software based on the test goals and test scope
allow the test team to determine and communicate a quantifiable state of the
AUT. These states may include coverage statistics and pass/fail statistics, among
others.

3-5
Associated Processes
Objective
To document the applicable processes and procedures that must be in place to
support a successful testing effort. It is acceptable to either reference process
documentation or document the process.
By outlining each of the associated processes, you can determine procedures that
are crucial to ensure a successful testing effort. Examples of associated processes
may include:
requirements management processes
configuration management processes
defect management processes.
With appropriate documentation, associated processes can be standardized so that

all personnel involved in the testing effort will have a clear understanding of how
to report, track or act upon any defects that may be found in the application under
test.
Test Effort Deliverables

Objective
To provide a list of deliverables to be generated during the testing effort and upon
its completion. Also, both your and Compuware's responsibilities in creating and
approving the deliverables are defined. For each deliverable, the acceptance
criteria and the approval process requirements are documented.
A list of deliverables provides additional detail regarding the expectations and
requirements early on in the testing effort. Understanding the deliverables can
assist in estimating the effort and supporting process activities that must be
incorporated into the schedule, especially if similar deliverables have been
required on past projects.
Knowing the criteria for approving the deliverables and knowing who is
responsible for their approval improve the likelihood that the deliverable will
meet the expectations.
Knowing the acceptance criteria provides the opportunity to plan and implement
measurements or quality control activities during the process to help improve the
likelihood of their acceptance and minimize rework efforts.
3-6

Test Effort Assumptions

Objective
To identify the major assumptions that were used in developing the test strategy.
This section should include any assumptions, which proved incorrect, may be
used as a basis for invoking changes to the test strategy.
Assumptions act as constants when creating a test strategy or other project
deliverable. They provide a basis to make decisions affecting testing efforts.
Documenting assumptions provides a framework in which test measurements can
be validated. If an assumption is changed or has been proved inaccurate, all
efforts associated with the assumption must be reviewed and adjusted since they
can no longer be used as accurate measures within the context of the test strategy.
Test Effort Constraints

Objective
To identify specific boundaries and constraints that the test effort must operate
within. Consider business and technical limitations and constraints as they relate
to such items as schedule, resources, cost, computer infrastructure, software,
security, regulations and other concerns.
If limitations and constraints are not understood, the planning process may not
result in a test strategy that will meet its objectives. Realization of limitations and
constraints may occur late in the testing effort as they are encountered.
Consequently, management activities may become reactive instead of proactive
and contingency plans may have to be identified and implemented quickly.
Identifying the limitations and constraints early in the test strategy process
provides the opportunity to challenge them, in an effort to reduce or eliminate
their potential impact. It also provides the opportunity to identify alternative
approaches instead of implementing quick fixes that may not be optimal if
constraints are identified late in the testing effort.
Compuware financial constraints are usually not included within this section
because of the sensitivity of financial information.
It is possible that there are no known significant test effort constraints. If so, a
statement to that effect should be included.

3-7
Test Effort Dependencies

Objective
To identify test effort dependencies that are outside of the test team's control.
Also, the purpose is to identify those dependencies that may be within their
control, but may have an impact on the test effort's success if they are not
managed appropriately or are not successful.
Testing activities may not be scheduled at the right time or in the right sequence if
dependencies are not understood.
Tracking and communication mechanisms can be established to help monitor the
dependencies; this information can then be used to assess the QualityPoint
schedule risks as well as to identify when risk and issues should be escalated to
people who could be able to provide assistance.
Mitigation and contingency plans can be developed for those dependencies that
are on the critical path or can have a significant impact on the test effort's success
if they encounter difficulties.
Test Schedule
Objective
To identify the testing planning details within the project planning schedule. This
requires the preliminary schedule of the test effort and needs to be updated from
information gathered during test case development.
The test schedule controls all activities during the test effort. It also provides
management with an explicit representation of all test activities.
Determining if the test effort can be executed in the time allotted tells you the
likelihood of truly meeting the expected schedule based upon the level of effort
that is required.
If the current plan does not allow all testing efforts to be completed within the
allotted time, you may still have time to identify alternative approaches to reduce
the schedule or to request additional time. Having to identify alternatives to
reduce the schedule late in the test effort could result in a solution that is not
optimal or does not meet the test goals.
Deadlines and fixed release dates will be a major factor in determining potential
test schedule risks.
3-8

Identify Risks and Issues

Objective
To be prepared for issues that could hinder or stop completion of the testing
effort.
This process determines different elements of risk to the testing efforts, including
risk identification, risk probability and risk impact. Consider these issues:
will resources be available at the time they are needed?
are new processes being implemented?
will testing start early enough?
are requirements documented?
will training be held as necessary?
are there any verification activities prior to executing a test?
are there management changes?
Identifying risks and appropriate action plans can help avoid:
people spending time addressing problems that could have been avoided
requiring resources to address problems at inopportune times when the

schedule does not allow for any slack.
Identifying risks also provides information and knowledge of consequences that

assist with decision making. Mitigation planning can minimize the impact of a
risk should it actually occur. Without contingency planning, alternatives to
resolve problems that occur may not be optimal or may not achieve the objectives.
Contingency planning minimizes the disruption when a risk is realized; the team
does not have to spend effort determining what to do, and the cycle time to
receive an authorization to proceed should be minimal.

3-9
Other Test Strategy Considerations

Testing Approach
There are four basic approaches to testing. Each approach has several variations
that provide certain advantages and cause some disadvantages. This chapter
explains the approaches.
Full Production
With this approach, production files and data are used during testing. Complete
and total files are taken/copied from production and used for testing. You must
perform the following steps:
identify needed cycles and outputs that are to be compared
capture and save needed data and copy needed files to the test environment
modify JCL and run production to save files for later comparison
substitute renovated code and rerun production using saved data
compare identified files.
saves time and effort in obtaining data and files
gets into performing tests quicker
usually provides a good test of common or frequently used functionality
usually results in shorter test phase duration.
Advantages
Disadvantages
3-10
requires using production data, which is often voluminous and possibly

sensitive
consumes large amounts of processing time
usually does not provide a complete test of the functionality
still have to rerun production in the test environment to save needed files.

Full Regression
With this approach, production data is augmented to test all, or at least a high
percentage, of the applications functionality. The steps usually follow those in
full production, but you must also do the following:
examine programs for code coverage
bring in additional production data or create the needed data
modify JCL and re-run until sufficient coverage is attained.
saves time and effort in creating data files
extremely good test of applications functionality
usually tests all date-related logic due to high code coverage.
Advantages
Disadvantages
requires using production data, which is often voluminous and possibly

sensitive
consumes large amounts of processing time.
Concentrated Regression
This approach builds a test bed of files and data for performing all tests.
Production data is not used, except as an initial source from which to build or
extract data. You must perform the following steps:
determine what is required and analyze the best method for obtaining it
create and/or capture required data
create or extract representative files
create or modify JCL
assure adequate code coverage
execute programs to create a baseline.

3-11
Advantages
most efficient and effective testing approach
saves time and resources during test performance
the more testing that is required, the greater the return on the effort to set up
provides an excellent test of an applications functionality
usually tests all date-related logic due to high code coverage.
Disadvantages
requires the most time and effort to set up
longest lead time before getting to testing
requires more involvement from application-knowledgeable personnel.
Compliance Assurance
This approach is used for testing applications that you feel are compliant. Since
the code is not being renovated, you do not need to regression test to check for
impact to the functionality. The steps are similar to full production but you must
also do the following:
examine programs for date processing
bring in additional data or create data with the needed dates
modify or age files to create pseudo-baselines for comparison.
saves time and effort in creating data files
usually tests all date-related logic due to date-code coverage.
Advantages
Disadvantages
3-12
requires using production data, which is voluminous and possibly sensitive
consumes large amounts of processing time.

Test Types
In addition to determining the testing approach, your test strategy must identify
the types of tests that are required. These types, or phases as they are sometimes
called, are defined below.
Not all of the tests types are required for a given application
nor for all programs within an application.
A test strategys intent is to indicate where and on what portions of the application
the tests will be performed. The test types are described below.
Unit Testing
This test type is the testing of a single screen, unit or program to verify that it
operates correctly. Unit testing can be carried out by testing all of the logic paths
or by concentrating on the code that has been changed. Test all of the logic paths
for new development. You only need to concentrate on the changed code when
you are testing maintenance changes. You can implement unit testing in one of
several ways:
by the development or maintenance group normally responsible for the

application
by a core test team
by the renovation center or group making the changes.
When unit testing is performed at a renovation center, limited or selected test

cases are performed. By concentrating your effort, you increase the chance of
finding errors while expending minimal effort. This method is often referred to as
Touch Point testing. A clean compile is all that is necessary to progress to other
testing phases.
On the other hand, you may choose to employ full logic unit testing for critical
programs and/or test beds with high code coverage.
The method you choose is dependent upon what other test
types are being performed. This is one of the key purposes
for outlining a testing approach early in the planning cycle.
String Testing
This test type is the testing of a small or logical grouping of programs that are
tightly interrelated. Their relationship makes it difficult to individually test the
programs. Usually, the test consists of executing an on-line transaction or batch
job to verify that it functions correctly. Much compliance testing is performed in
3-13
this phase, particularly when the reverse transaction method is used to reset
files. String testing is often combined with unit testing and sometimes with
system testing.
Again, you can use the full logic or change concentration approach as in unit
testing.
System Testing
This test type is the testing of an application to validate that its functionality
performs as specified. Its objective is to prove that the whole system performs
correctly. You may perform system testing for the entire application or for logical
modules or subsystems such as daily, weekly and monthly. System testing is
often combined with acceptance testing and/or integration testing.
A part of system testing is to process the baseline that was created against the
renovated code to make sure that no problems were introduced during renovation.
Depending upon the size and complexity of the application, these baseline tests
can be performed in small groupings or for an entire application.
Integration Testing
This test type involves testing an applications external interfaces. The objective
is to make sure that data is correctly passed between applications and that any file
structure changes are accommodated. Integration testing is almost certain to be
required due to date and file format changes. In addition, bridges or other
temporary measures are more likely to be used and require testing. The inclusion
of these bridges adds processing and therefore, the potential for performance
testing. For this reason, integration and performance testing are often combined.
Acceptance Testing
This type of testing is used to ascertain whether an application completes those
functions required from a business perspective. Acceptance testing includes
ensuring that performance is at acceptable levels, and if functional deviations
occur, suitable alternatives are substantiated. It is often combined with system
and integration testing. Acceptance testing sometimes includes the generation of
special or unique outputs or interfaces.
Performance Testing
This test type is performed to determine if an application can operate as needed
when transactions, concurrent users or file sizes are larger than normally
expected. Transaction and response speeds, throughput and computer capacity are
measured to determine if they are within tolerance. Performance testing is done to
3-14

ensure that processing has not been significantly degraded. It is not intended to
test for optimizing the application.
Interoperability Testing
This test type is used to determine if the renovated application can operate in its
intended environment. Things like the new operating system, middleware (such
as DBMS) and other renovated applications must be installed and operational for
interoperability testing to take place. Unlike the other tests, interoperability
testing requires a separate region or logical partition (LPAR). Performing this
type of test is difficult and requires a lot of work. Consequently, interoperability
testing is only performed for critical applications. It is most often combined with
performance and implementation tests.
Retrofit Testing
This test type identifies any changes that may have occurred while an application
was being renovated and tested. As mentioned throughout this document,
changes should not be permitted or at very least, should be minimal during this
time. This is not always possible, however. Necessary changes must be made to
the renovated code and tested as they were for production. This test type includes
those tests. A retrofit test is usually run immediately before an application is
placed into production.
Implementation Testing
This test type tests the ancillary components of an application more than the code
itself. It ensures that there is sufficient time to implement the changes, that is,
time to convert files, execute utilities and move in the application changes.
Implementation testing includes such things as operating or production
procedures, production scheduling software (such as CA7) and file conversion
programs. Typically the test team is not directly involved with the test execution.
Test execution is performed by the group that normally places new or revised
applications into production. This testing is usually combined with
interoperability tests because it attempts to mimic the operating environment as
much as possible.
Back-Out Testing
This testing type is used to verify that a revised application can be taken out of
production and the previous version can be reinstalled. Its objective is to prove
that a fail-safe mechanism exists and can be invoked if necessary. This test is
typically run back-to-back with an implementation test. Back-out testing is not
usually combined with any other test type.

3-15
Note that in some instances, an applications implementation test is the real thing.
An application may be placed into production and remain there if certain results
are verified. When this occurs, the back-out test may be planned but never
executed. It would only be run if there were problems with the implementation
test.
Disaster Recovery Testing

Many organizations have plans for recovering an application in the event of a
disaster or when it needs to move to an alternate processing site. This is normally
the case for business critical applications. This test type modifies and exercises
the disaster recovery test for the newly renovated application. Disaster recovery
testing can be performed prior to placing the renovated application into
production or when the normally scheduled disaster recovery test. Regardless of
when it occurs, a disaster recovery test is required.
Roles and Responsibilities

There are five major testing functions and responsibilities. These can be further
broken down and combined in any fashion; also, additional and unique roles may
be added. The test strategy identifies these roles and the responsible group,
organization or in some cases, the individual. The major roles are:
test requirement specification
test region development
test planning
test performance
auditing.
Your test strategy should be as specific as possible regarding the responsibilities
expected to be performed by organizations external to the test team. These
responsibilities include technical support, security, applications development and
end users. In addition, the responsible organizations should provide estimates of
expected or average participation levels and time required in the role.
A key function of the test strategy is to help ensure that application experts, both
technical and end user, are available during the planned time frame. This includes
planning and setup activities, as well as the actual test performance activities.
As a rule of thumb, an application expert will need to spend
half of his or her time supporting the testing effort.
Participation follows a bell curve, starting out at a low level during planning,
rising to full time during the height of testing and tapering off in the later testing
phases. If application experts are not available (for example, they left the
3-16

company), the testing phase will take considerably longer, and the schedule will
need to be adjusted accordingly.
Open Items
The test strategy should identify any open items or issues. Since it is a
management document, include items that need to be addressed and to whose
responsibility the items need to be assigned. Avoid including detailed technical
issues. Typical items might include who will approve test plans, how the costs for
the test environment will be covered and how access to sensitive data will be
obtained.
Obtaining Test Strategy Sign-Off

Your test strategy is the guiding document for what gets tested and when. Those
involved should sign it. Normally those who should sign include the head of each
organization responsible for supplying resources. In a large organization, there
may be dozens of people involved, so considerable time may be required. This is
not to imply that work should not commence until test strategy sign-off has been
accomplished.
The primary reason for sign-off is to show concurrence and support for the plan.
The real objective is to obtain commitment.
It is strongly recommended that the test strategy be
presented and discussed in a series of meetings throughout
the entire organization.
A critical success factor is to have commitment and buy-in.
The test strategy could and should be signed off by the heads of the functional
departments, as well as the IT organization. These departments include:
Applications Development, Technical Support, Testing, Systems Support and
Security. But real concurrence comes from the entire organization knowing and
understanding what is involved. Do not rely on the people who sign-off on the
document to communicate its contents with their respective organizations.

3-17
Chapter 4-Creating Test Plans
Chapter 4 - Creating Test Plans

Introduction
This chapter explains how to develop the detailed test plans used in testing
applications.
A test plan identifies the tests required to meet criteria
identified in the test strategy and outlines the specific steps
that will be performed.
A test strategy outlines testing types and how to approach each test. Your test
plan should define:
how to perform the tests

the test cases to run
the expected test results.
Typically, you create a test plan for each test type you plan to perform.
Establishing a Test Plan

The following pages provide the detail on how to create a test plan, the objectives
of each area and the importance of each area of the test plan.
Application Description
Objective
Document the key information regarding the application to test. A thorough
understanding of the application, including its purpose, function, environment and
operational characteristics, is critical.

4-1

Developing a high-level business description of the application under test (AUT)
provides you with a basic understanding of the application and its business and
technical requirements. Determining if the application is developed internally,
vendor supplied or a combination provides a quick, high-level check against
requirements and helps to identify the test types required in the test plan.
Understanding the change control process and its impact upon the test effort are
important. They identify communication requirements and processes that may
need development and incorporation to ensure the project team is aware of
changes.
A high-level description of the technical environment in which the application
operates helps to identify test types required in the test plan and provides you with
a general understanding of the scope of the testing effort.
Objectives of the Test Effort

Objective
The purpose of the objectives section is to identify and document measurable
criteria for the goals set forth in the test strategy. The goals for the company, QA,
Development and Implementation teams are represented in the test effort.
To guide the testing activity and document how you established goals for this test
effort. In short, the goals help identify the testing activities. People can determine
what processes are needed to achieve the goals. This process also represents
management's interest in the AUT. Refer to test process assessment working
papers as necessary.
Understanding the different types of software releases helps identify goals. For
example, defect measurements may be more important in a maintenance release,
while maintaining existing functionality might not be applicable for a new release.
Testing activity can be initiated a number of ways; however, it is typically started
because of an impending software release. The following are some examples of
releases:
4-2
software upgrades
new site implementations/deployment of the software to multiple sites
new releases of software
minor releases of the software
high priority application fixes.
Documenting the changes made to the software for this release provides insight
into the change control process: reality versus the documented process. You need
to manage expectations at all levels:
the test team understands what to test.

managers know that the test team is testing functions that support the
business and technical requirements.
Equally important, measurement targets can be established against error

correction, added functionality and increased response times.
Scope of the Test Effort

Objective
To identify and document the scope of the test effort. Specifically, to identify the
sections of the AUT to be tested and those that will not be tested.
Identifying the scope of the test effort manages expectations at all levels by
focusing on completing work tasks that meet stated goals.
It prevents the assumption that everything will be tested; it focuses only on items
within the scope of this test effort.
It helps identify needed regression testing (functions or AUT components that are
not within the scope of this release may not need to be retested).
Test Type and Test Method

Objective
To document the type and method of testing. There will be a test plan for each
test type.
Defining the test type and method shows how the test goals will be achieved by
identifying the types of testing to use, such as unit, integration or regression
testing.
Documenting the test types needed for this effort defines the testing activity to
undertake. It also gives you an indication of the resources and test environments
required for the testing activity. Each of the test types should be considered;
however, not every type of test is typically employed. For example, it is not
necessary to do unit testing if the software is vendor-supplied.
4-3
Connecting test types to test goals assists in directing test activity and validates
that the testing is actually needed.
Entrance and Exit Criteria

Objective
To document the conditions that must be met to start testing and what constitutes
acceptance of the application for this type of testing based on quantifiable
measures.
This helps identify criteria required to begin the testing effort. This ensures that
the application meets any applicable contractual and regulatory requirements. For
example, food, pharmaceuticals and nuclear energy industries must meet
government regulations.
You can use these criteria to identify appropriate quality control activities and
measurements to include prior to the testing process.
Quality levels used to accept the software based on the test goals and test scope
allow the test team to determine and communicate a quantifiable state of the
AUT. These states may include coverage statistics, pass/fail statistics and others.
Measurements for the Test Effort

Objective
To document the types of measurements used during testing. These measurements
should supply data that can help determine if the AUT has successfully met its
goals.
Measurements answer questions. There is a Goal - Question - Metric paradigm
that is useful in developing measurements. Measurements should be based on
goals and should answer questions about the goals. Determining that
measurements can be used and applied correctly validates that the test goals are
measurable. Consequently, measurements can quantify progress toward the
achievement of the test goals and communicate when the goal(s) is met.
In addition to QualityPoint defined measurements, any defined measurements
should be incorporated to further supply data. By adding these measurements, you
have a familiar frame of reference when interpreting measurements.
If the measurements are long-term, the data can be used for trend analysis and
incorporated into the application readiness report. Short-term measurements
4-4

cannot be used for trend analysis. Attachment E shows a sample application

readiness report.
Measurements for trend analysis can make future project estimation activities
easier.
Test Schedule
Objective
To identify test plan details within the project planning schedule. This requires
you to update the preliminary test effort schedule from information gathered
during test case development.
The test schedule controls all activities during the test effort. It also provides
management with an explicit representation of all test activities.
Determining if the test effort can be done in the time allotted identifies potential
risk in actually meeting the expected schedule with the current level of effort.
If the current plan does allow all testing efforts to be completed within the allotted
time, time may still be available to identify alternative approaches to reduce the
schedule or to request additional time. Having to identify alternatives to reduce
the schedule late in the test effort could result in a solution that is not optimal or
does not meet the test goals.
Deadlines and fixed release dates are a major factor in determining potential test
schedule risks.
Resources for the Test Effort

Objective
To identify the human resources, testing tools, hardware and software necessary
to conduct the testing effort.
Identifies the required resources for the following processes:
test planning
planning and scheduling tools
test case development
test script development
test tool support

4-5
test environment preparation

test execution
test results analysis
management reporting.
Knowing the resources you need allows you to specify individual training needs
and create or buy training programs if available people do not meet required skill
and knowledge levels.
A schedule that includes resource assignments identifies when and how long
resources are needed. It also provides those who control the resources an
opportunity to plan appropriately, and can also identify areas where "creative"
staffing plans may need to be considered (contractors, co-ops).
Identifying resources early in the planning process improves the likelihood that
they will be available when needed. It also provides the opportunity to identify
alternatives if difficulties are encountered meeting the requirements.
Knowing resource requirements assists in forecasting testing effort costs such as
labor, materials and capital expenditure. It also identifies required test tools, test
environments and technical configurations.
Test Environment
Objective
To document the environment in which the tests are to be executed, procedures
for restoring the test environment, and environment procedures and processes that
could be automated based upon their repetition or lack of manual intervention.
Documenting the technical environment for test execution helps you identify
resource dependencies outside the control of the software testing organization; for
example, operations, help desk and facilities. Attachment C shows an example of
a test environment plan.
It provides an indication of the complexity of the environment, which in turn may
provide early warning signs of potential operational issues.
Identifying differences between the production environment and the testing
environment allows you to assess potential risks associated with the differences in
the environments.
4-6

Requirements Matrix Creation

Objective
To document the business requirements and technical requirements that are part
of the test effort and provide traceability from the tests to the requirements. This
activity can only begin during test planning and must be completed during test
case development.
A thorough understanding of the business and technical requirements helps ensure
test coverage and completeness of the test effort.
Designating the requirements that are in scope and out of scope helps manage
activities during the testing effort. The requirements matrix provides a visual
representation of both in scope and out of scope requirements. Ultimately, only
those business and technical requirements in scope will be tested.
Because the requirements matrix allows you to trace between requirements and
specific test cases, the requirements matrix assists in audit efforts as well.
Suspension and Resumption Criteria

Objective
To determine criteria which, if met, would stop the testing effort and the criteria
for test resumption.
Researching the conditions that would suspend testing efforts provides early
identification of issues. For example:
is the environment unstable?

is the test environment available for testing?
are the resources available for testing?
Defining the suspension and resumption criteria also improves resource utilization
by providing focus on value-added testing activities.
Test Case Development and Execution Process

Objective
To identify key items to use during test case development and test execution.

4-7

Identifying and documenting the process for defining the test cases communicates
testing activities to all levels. This process assists in the reporting and traceability
aspects of test execution.
Test Discrepancy Reporting Procedures and Associated Processes

Objective
To document the procedures to record and track issues discovered during testing.
Additionally, the high-level processes for requirements and configuration
management are identified.
Documenting the process for logging test execution discrepancies assists with
project reporting communication requirements such as quantity of defects by
category, by severity or associated with requirements, among others. It manages
results tracking, reporting and communication process expectations.
Identify Risks and Issues

Objective
To be prepared for issues that could hinder or stop completion of the testing
effort. While identifying risks and issues, the assumptions, constraints and
dependencies are identified.
Why is This Important?
This process determines different elements of risk to the testing efforts, including
risk identification, risk probability and risk impact.
will resources be available at the time they are needed?
are new processes being implemented?

will testing start early enough?
are requirements documented?
will training be held as necessary?
are there any verification activities prior to executing a test?
are their management changes?
Identifying risks and appropriate action plans can help avoid:
4-8
people spending time addressing problems that could have been avoided
requiring resources to address a problems at inopportune times when the

schedule does not allow for any slack.
Identifying risks also provides information and knowledge of consequences to

assist with decision making.
Mitigation planning can minimize the impact of a risk, should it occur.
Without contingency planning, alternatives to resolve problems may not be
optimal or may not achieve the objectives. Contingency planning minimizes the
disruption when a risk is realized; the team does not have to spend effort
determining what to do, and the cycle time to receive an authorization to proceed
should be minimal.
Other Test Plan Considerations

Access Requirements
One area that always seems to cause problems during testing is obtaining access
to needed functions and data. Based on the items previously outlined, you should
know what needs to be accessed: production data, other applications and libraries.
You need to identify and note several other things such as backup and restore
functions, library update capability and application security rights. Once these
items have been defined, you need to define how the testers will obtain access. If
they cannot have direct access, contact someone who has the authority to perform
the function.
Your test plan must include what access the testers need and
should also include how they will obtain the necessary
access.
Change Management
There are two aspects of change management you must contend with. One is the
inclusion of any other production changes that may have taken place while the
application was being renovated and tested. This is necessary because testing is
often a distinct and separate activity from other maintenance on the production
software. Therefore, you must identify any changes taking place and consider
them when placing the renovated application or new application into production.
A library control system (LCS) normally is used to perform this function and
should be used here as well. You may need to alter the process slightly to allow
for center renovation external to the LCS process.
Notify the test team of all production changes. Any test
cases and data used for the change should be provided to
them for use during any retrofit test.
4-9
The second aspect is the change management of the test environment. As

programs are tested, you will uncover problems, and the programs will undergo
additional changes. The source code is often tracked with the same LCS, but
changes may occur outside the responsible organization. Additionally, there
needs to be change management of other aspects of the test environment such as
test files, scripts and libraries.
All of these things require the employment of a change
management process. The use of an LCS is recommended.
Even when an LCS is being used, you must provide for the melding of the two
LCS processes.
Resetting or Initializing Files

To perform any type of test after establishing a baseline, it is necessary to reset
files to their initial state. You must also reset or initialize files to rerun a test after
a failure or after a fix has been applied.
There are two methods you can use to return files to their
original state: reversing transactions and restoring
(resetting) files.
With the reversing transaction method, all changes that were applied are reversed
or backed out. With the restoring files method, files are restored in their entirety
with a copy that was taken prior to running the test.
Many organizations rely on the reversing transaction method in the belief that it is
faster than restoring large databases. The method has several inherent flaws,
however. In many instances, transactions cannot be completely reversed, at least
not through any normal processing. The reverse transaction method also relies on
the tester to manually reverse everything that was done. This is not always easy
if a test fails in the middle. The restoring files method ensures that things are
properly reset and may require less time in the long run.
Testers should have the capability of taking backups of files
at their discretion.
This provides additional files to use for reinitializing as testing progresses and
creates additional baseline files.
Populating Data Stores

What is the best method of populating the necessary databases, files, tables and
transactions for testing? The following section explains how to populate data and
provides some alternative examples.
4-10

Use an existing test bedMany application development organizations already

have some form of test environment. In many cases, this is a good place to start.
There usually are test master files and databases that get around the problems of
large files and accessing sensitive data. It may be simply a matter of performing
the following tasks:
copy the existing test bed to the test environment

bring in production tables and control files
add sufficient transactions to perform the desired test cases.
Begin with ProductionBegin with production, reduce its volume and extract
needed data. Compuwares File and Data Management tools can be used to
extract data from existing files. Production transaction files can usually be edited
to include needed transactions and data fields, or edited to have redundant or
similar transactions deleted. If scripts of online activity were captured using
QAHiperstation, they can be trimmed down to have manageable yet complete
tests.
Use full files when it does not impact testingNumerous files are used by an
application for verification and are not processed. Therefore, they do not need to
be compared. Sometimes the files are small tables. Other times they are large,
such as vendor or part files, but are only used for verification. If they are not read
sequentially, using them in their entirety has little or no impact.
Use online scripts to drive testingThe processing of many applications is
driven by data that is entered via online modules. These modules need to be
scripted to have the test cases necessary to perform the required tests. By adding
some additional cases and processing the data through the remainder of the
application, you can populate databases with sufficient data for testing the
remaining programs.
Comparing Files
During testing, files often need to be compared to validate the results of a test.
You can perform comparisons either by manually viewing the outputs or by using
some utility or tool to automatically perform a comparison. Using a utility
requires that you save the file for later comparisons. Typically, all key outputs are
saved to form a part of a baseline that you can use to compare outputs after a
subsequent test has been performed. Many organizations manually view the
outputs because they think it is faster and less trouble than changing production to
save the desired files and reports. Production most likely has to be changed
anyway, so you can incorporate saving reports and temporary files to permanent
files into this effort.
Performing a manual compare is rarely faster and certainly never better than
performing comparisons with some type of automation. The major flaw with
manually comparing the output is that it is tedious and can produce errors in
4-11
proportion to the amount of work that is being done. Tools, such as

QAHiperstation, provide automatic comparison of online screen outputs. For file,
database and report comparisons, use Compuwares File-AID.
4-12

Chapter 5-Test Case Development
Chapter 5 - Test Case Developm ent

A test case is a test procedure combined with the data necessary to complete the
procedure. This includes the test steps plus the input and expected result values.
Attachment D shows a sample test case.
The key benefits of proper test case development include:
repeatable test cases become a strategic asset to the organization.

the test case templates designed for the environment can be leveraged for
continued use.
QA personnel gain knowledge of application functionality.
avoids disruption of the test environment.
Since test case development is a part of an overall testing process, it is assumed

that a testing process has been defined, test goals have been created and test
planning has been completed.
Create Decision Tree

Objective
To ensure that test case development supports the business and technical
requirements of the AUT.
Creating the decision tree assists in directing project resources. After creating the
decision tree, you then weigh the information and determine which test cases can
be executed during a test cycle. This determines the scope of the test cases and
helps ensure the test cases will be developed to support business and technical
requirements.

5-1
Prioritize the Test Cases

Objective
To prioritize the test cases.
Prioritizing the test cases assists in directing the project's resources. It also
provides the ability to assess risk by knowing the importance of what is not being
tested.
Document Test Procedures

Objective
To identify the steps necessary to set up, execute and evaluate test cases.
By documenting the test procedures, you identify discrete steps to complete the
testing. This, in turn, provides a level of repeatability during the testing effort.
Document Test Data Requirements

Objective
To document what data is necessary to execute the test cases and verify test case
results.
Identifying the data requirements for each test provides information for the test
environment preparation process. This effort will also uncover test case
dependencies so you can schedule test execution activities in the proper sequence.
Understanding and documenting the base data, transactional data and the expected
results provides validity and repeatability to the testing effort.
Identify Candidates for Automation

Objective
To identify which test cases are candidates for automation and which test cases
must be manually executed.
5-2


Documenting the candidates for test case automation identifies potential areas of
resource utilization efficiencies. Effort, schedule and resource estimates can be
refined based upon what test cases must be executed manually.
Automate Designated Test Cases

Objective
To plan for automating test cases.
Automating the designated test cases produces consistent test results and
repeatability. Test execution is more efficient in both time and resources, and
there is the potential for greater test coverage. Effort, schedule, cost and resource
estimates can be refined based upon (1) the activities required to automate the
designated test cases and (2) the effort required to execute the test cases that are
not automated.

5-3
Chapter 6 Test Environment Preparation
Chapter 6 - Test Environment

Preparation
When preparing to conduct application tests, preparing the test environment is a
main concern. This preparation includes:
understanding the types of testing that are being planned and their
associated test environment requirements
refining the test schedule to include environment preparation activities
arranging for operational and infrastructure support
developing the installation, backup and restoration procedures.
Only after these items are prepared can the test environment be considered ready
to accept input of test data, application installation and the initiation of the actual
test procedure.
Test environment preparation focuses on documenting the information necessary
to develop test environment preparation procedures.
The key benefits of developing test environment preparation procedures:
a stable, repeatable test environment protects validity of test results

technical requirements for test execution are met
accurate, repeatable, and meaningful test results
the following areas will assist you in developing your test environment
preparation process.
Document Initial Environmental Data

Objective
To identify the first data state required to begin testing.

6-1
Chapter 6 Test Environment Preparation

Identifying the initial environmental data requirements allows you to define the
application data needed before testing begins.
Document the Test Environment Configuration

Objective
To identify the specific environment in which test execution occurs. This is also
the section where you list the different workstation and server environments as
part of the test environment (Windows 95 and Windows NT workstations).
Documenting the physical components required for test execution helps identify
potential gaps in what is required and what actually exists; you can then develop
appropriate action plans. Consider such things as:
operating system version (remember to identify service packs, fix packs

and patches)
networking software and version
communications protocols
server and workstation machines
Identify Data Recovery Points

Objective
To identify and document data recovery points used during the test effort.
Identifying data recovery points saves time, re-work and re-execution of test cases
that have already proven to fulfill the requirements.
Establish Backup, Restoration and Construction Procedures

Objective
To establish backup and restoration procedures used after individual test cycles
and tests create data states.
6-2

Chapter 6-Test Environment Preparation

Having backup, restoration and construction procedures increases the probability
of having reliable, repeatable results should a problem occur that requires the
restoration of the test environment.
Creating documented procedures minimizes process time for completing a backup
or restoration, as well as data construction. Who is responsible and how they
complete the activities can be retrieved easily.
Create Initial Data State

Objective
To establish the initial test environment with the appropriate data, identify
additional data states and validate the test cases that will be used for the test runs.
This can also be thought of as debugging the test cases before the official
execution of tests. This minimizes errors that may occur during "live" test runs.
Actual backups of the data states will occur mainly when you implement test
execution key process areas.
Creating the initial data state means that the appropriate data is saved, and this
allows for restoration if needed. It provides a baseline to evaluate the results of
performed tests.

6-3
Chapter 7 Test Execution
Chapter 7 - Test Execution

Test execution includes running the test, documenting the results and validating
those results against performance criteria. Specific performance measurements
include application of test cases, input, output and documentation.
Test execution turns on creating a valid test execution plan.
The key benefits of developing a test execution plan:
test execution becomes a controlled, potentially automated process

test result documentation is consistent.
Review Test Execution Schedule from Test Plan

Objective
To analyze the projected and actual test schedules and document differences that
may affect the ability of the test group to complete the number and types of tests
to be executed.
It is necessary to review the existing test execution schedule because any changes
in the overall project schedule may impact the test execution schedule. It may be
necessary to reduce the scope of the test effort, expand on the time allotted for
testing or add resources to accommodate the modified schedule.
Restore Test Execution Environment

Objective
To execute the restoration procedures that prepare the test environment for the test
cases contained in the test plan.

7-1
Chapter 7 Test Execution

The test environment must be restored to its proper data state prior to executing
any tests; this is especially critical if testing uses multiple environments that must
be synchronized, and you need to compare and/or combine results.
Execute Test Cases

Objective
To apply the test cases defined in the test plan to the application.
You can analyze and measure the results to determine whether the tests fulfilled
business and technical requirements.
7-2

Chapter 8 Test Results Analysis
Chapter 8 - Test Results Analys is

Test results analysis focuses on documenting the information necessary to
develop results analysis procedures.
The key benefits of test results analysis:
establishes initial baseline metrics

you can identify risks associated with meeting the application's
requirements
establishes a standard, repeatable evaluation process
QA personnel gain knowledge of application quality
identifies potential problem areas.
Perform a Flow Analysis

Objective
The purpose of flow analysis is to confirm that the flow of events occurred as
expected and to assure there is a complete set of results to be analyzed.
Flow analysis helps ensure that a complete set of results exists. It also identifies
that the results were not only logged, but also safeguarded after the tests were
executed. As part of this analysis, it is important to correctly categorize any
defects so you can make an informed decision when evaluating test results. The
following categories used in this work program are:
valid results - Actual test results matched the value you expected.
test case defect - Actual test results do not match the value you expected
and the error was caused by a defect in the test case itself. The following
items list examples of test case defects:
test submitted an invalid value
test submitted an invalid command

8-1
Chapter 8 Test Results Analysis
test was performed out of context in regard to the system being tested.
application context - Actual test results do not match the value you
expected, and the error was caused by an invalid value or operation that
was submitted to the system being tested. The following items are
examples of application context errors:
user error
errors caused by other software
unsupported use of the system.
application defect - Actual test results do not match the value you
expected, and the error was caused by the software itself or an associated
component. The following items are examples of application defects:
documentation error
software defect
installation error
hardware error.
More mature organizations can include further breakdowns for recording test
failures. These can include design errors and requirement errors.
Failure Discovery
Objective
To document how the application performed in regard to failures.
By documenting the application's performance in regard to failures, you identify
and collect measurements. Resources focus on issues that they can resolve; for
example, business analysts address business process defects, and programmers
address application code defects.
Success Discovery
Objective
To identify which test cases show that the requirements are fulfilled.
By documenting the application's performance in regard to successes, you identify
and collect measurements. It also helps validate that any additional resources you
had available for further investigation or re-execution are not required.
8-2

Chapter 8-Test Results Analysis
Measurement Analysis
Objective
To review and analyze the test result measurements contained in the application
readiness report.
By reviewing and analyzing all of the test measurements and trends that you have
collected, you are better able to determine the application readiness status as well
as identify risks.

8-3
Chapter 9-Management Reporting
Chapter 9 - Management Repor ting

Management reports communicate test team conclusions and recommendations to
management, the application development team and to the user community.
Management reporting is one of the most important aspects of the test process.
With this information, the management team can make informed release decisions
based on the number, severity and types of defects reported and cleared, the
number of tests executed, pass/fail rates and other parameters. This information is
critical to application quality professionals who need to understand the
applications status.
Executive Summary
Objective
To document in narrative form the release risk statement based on the testing
effort.
Management reports assist a decision-maker in determining if a software
application should be released based on how the tests went. The reports do not
cover other release criteria such as availability of customer support personnel and
policy, readiness of user documentation or distribution and shipping capabilities.
Compuwares QualityPoint process uses a suite of reports to provide information
in regard to releasing software for distribution. The reports include:
trend for error detection

indicator of the testing activity in general
checkpoint on the quality of the product
identification of problem areas for the application by requirement
requirements coverage.
Management needs to review all of these reports to get an accurate picture of

product status as shown through testing results. The reports provide a
comprehensive assessment of the application, highlight variances from the
9-1
Chapter 9-Management Reporting
requirements and summarize testing activities. This permits the decision-maker to

manage by fact using the objective evidence in the reports. The application
readiness report is a combination of the executive summary and the suite of
reports.
Management reporting is the means to communicate testing results consistently
and effectively. It provides quantifiable information about the application status.
Consolidate Measurement Reports

Objective
To consolidate the measurement reports and include them as part of the
application readiness report. Most, if not all, of these reports will be generated by
the defect tracking or change control system.
Consolidating the measurement reports provides a summary of the test results to
management. Attachment F shows a sample test summary report.
9-2

Chapter 10-Productivity Tools and Professional Services
Chapter 10 - Productivity Tools a nd

Professional Services
Introduction
Some examples of Compuwares tools are presented in this chapter. These
examples are not intended to show all aspects and capabilities of the Compuware
products. Current information is available on the internet at
http://www.compuware.com. Contact your Compuware representative (800-5219353 or 248-737-7300) to obtain training, or call the Compuware Product Support
Hot Line (800-538-7822 or 248-865-5444) for further assistance.
The Compuware testing and implementation process, tools and professional
services are discussed in this chapter by the following areas of functionality:
NuMega Client/Server Products

The NuMega family of products for analysis, fault detection and performance
measurement is designed to assist Windows application developers to produce
high quality systems.
Automated testing tools

QACenter promotes software quality across the enterpriseautomating
testing tasks throughout the development life cycle and across multiple,
diverse platforms. More than just individual tools, QACenter is a powerful,
integrated solution of products and services that work together to help you
manage and implement the entire testing processfrom planning and test
script creation to test execution, analysis, defect tracking, load testing and
validation. QACenter has the right mix of products to assure resolution of
problems at "web speed" through faster root-cause analysis, to provide
confidence that applications are thoroughly tested and to provide confidence
that applications and environments will scale.
Interactive analysis and debugging tools

10-1
Featuring the XPEDITER family, these tools help debug program logic and
ensure test coverage. XPEDITER may also be used to change the instructions
in your program and see the results immediately without recompiling.
File and data management tools

File-AID, the industry leader in data management tools, helps application
developers and DBAs create test environments, resolve data-related
production problems and manage special projects with ease and confidence.
File-AID products enable developers to move, copy, create, extract, browse,
edit, reformat, convert, migrate and compare data, regardless of its platform
and data type. The File-AID family of mainframe and client/server products
plays an integral role in testing by assisting with the following tasks:
making global changes

generating test data
encrypting test data
comparison of test results
analyzing test data.
Fault management tools

Fault management products automate the fault resolution process to help
managers and problem solvers quickly resolve mission-critical application
problems in OS/390 environments. The products help optimize limited testing
time and reduce costly downtime in production.
Abend-AID Fault Manager provides real-time and historical reports and
graphs to help managers monitor, measure, control and improve the fault
resolution process.
Abend-AID XLS and CICS Abend-AID/FX provide problem solvers with the
tools to quickly analyze, diagnose and correct problems.
Professional Services and Training

Compuwares staff of over 11,000 professionals provides expertise,
methodology and training to clients around the world. Services include:
automated testing services featuring QASolutions
Training Services
10-2

Chapter 10- Productivity Tools and Professional Services
NuMega Tools for Quality Windows Applications

As a result of ever-expanding technology in Windows-based software
development, the need for easily accessible developer productivity tools and
desktop technical support resources is critical to the overall success of software
projects. Incorporating and managing problems associated with new technologies
such as Windows Distributed Network Architecture (DNA) can slow a project to
a crawl.
The NuMega family of products accelerates software development and improves
the reliability of Visual Basic, Visual C++ and Java applications by pinpointing
coding errors, performance bottlenecks and untested code. In addition, capability
is provided to manage the software defect tracking process as an integrated
element of software development and test process.
NuMega products include:
Multi-Language Studios:
DevPartner Studio
DevPartner Studio Enterprise Edition
Programming Language Suites:
DevPartner for Visual Basic

DevPartner for Visual C++
DevPartner Java Edition
Individual Applications:
BoundsChecker Run Time Error Detection for Visual C++

TrackRecordTracking and documentation system
DevPartner Studio
DevPartner Studio is a comprehensive suite of Windows-based, multi-language
software developer productivity tools. Easily integrated into the development
process and used in building the most challenging applications, including webenabled, e-commerce and distributed applications, these tools help developers
automatically detect, diagnose and facilitate resolution of software errors; take
full advantage of code performance; and optimize code coverage and testing.
DevPartner Studio Enterprise Edition
DevPartner Studio Enterprise Edition is a comprehensive suite of Windowsbased, multi-language software productivity tools, which coordinates and
accelerates the development of Windows, Web and other distributed applications
10-3
by integrating error detection, performance tuning and unit testing with

requirements and defect management.
DevPartner for Visual Basic
DevPartner for Visual Basic is a suite of software development productivity tools
that accelerate the development of distributed and other component-based
applications using Microsoft Visual Basic. DevPartner for Visual Basic helps
developers deliver reliable, high performance components on time by enabling
them to quickly identify potential coding issues, run-time errors and performance
problems.
DevPartner for Visual C++
DevPartner for Visual C++ is a suite of software development productivity tools
that accelerate the development of distributed and other component-based
applications using Microsoft Visual C++. DevPartner for Visual C++ helps
developers deliver reliable, high performance components on time by enabling
them to quickly identify potential coding issues, run-time errors and performance
problems.
DevPartner for Java
DevPartner for Java is a suite of software development productivity tools that help
developers build reliable, high-performance applications and components with
Java technology. Today's multi-tier e-business applications combine many
different technologies and are prone to problems with runtime performance,
memory utilization and multi-threading. DevPartner Java Edition helps
developers resolve these problems quickly and easily, producing robust Java
applications.
BoundsChecker Visual C++ Edition
BoundsChecker Visual C++ Edition is the premier run-time error detection and
debugging tool for Visual C++. It speeds development and time-to-market by
automating the debugging process right inside Visual C++ Developer Studio.
BoundsChecker provides clear, detailed analyses of programming errors, many of
which are unique to C++. It detects and diagnoses errors in static, stack and heap
memory, memory and resource leaks and validates over 8,700 APIs at run-time,
including the latest Windows APIs, ODBC, ActiveX, COM and Internet APIs.
TrackRecord
TrackRecord is the award-winning software development tool for tracking bugs,
features, releases and virtually any other information relating to software projects.
It gives individual developers, team leaders and QA engineers control over every
detail, resulting in improving time delivery and reliability.
10-4

Automated Testing Products

QACenter
Testing across the enterprise can be extremely complex, encompassing: online
applications; batch processing; third-party packages, including enterprise resource
planning (ERP); client/server systems; Internet and intranet applications; and
hierarchical and relational databases. QACenter provides the proper framework
for:
coordinating the entire process, including planning tests and tracking

defects
defining and creating test data
capturing and storing test scripts
automating test execution.
QACenter coordinates the testing of enterprise applications, including integration

points between programs. Its capabilities cross technologies easily and help you
test all of the critical applications in your environment. The QACenter solution
supports your core technologies; provides integrated, complementary testing
products; and helps maximize your investment across numerous test iterations.
The QACenter testing solution is process-oriented, providing tools to take you
easily through the complex tasks and multiple stages of testing. QACenter
enables you to establish comprehensive test suites that can be re-used as
applications are modified. Integration among QACenter products further
streamlines the testing process, so you can perform more testing and evaluate
more thoroughly in less time.
Figure 10-1: QACenter products simplify the challenges of testing across a variety of complex environments.

10-5
QADirector
Testing is a major undertaking that requires the coordination of many steps, many
testers, large suites of test scripts and multiple versions and releases of different
applications. QADirector gives you a framework for managing the entire testing
processfrom design to execution to analysis. Its distributed testing capabilities
and open architecture enable development and testing teams to control testing
activities across heterogeneous environments from a single point. QADirector
promotes thorough, consistent testing by allowing developers, testers and QA
managers to share test assets, current and historical information about tests and
test results.
Design
QADirector allows you to easily create test suites that cover an entire application
by logically grouping testing scripts in a visual tree structure. These test suites can
include automated and manual tests.
Execution
By automating test execution, QADirector overcomes the time restrictions that
often make thorough testing impossible. QADirector quickly and easily runs the
testing scripts and returns and stores status information when testing is complete.
Analysis
QADirector logs the results of each test run. From this summary, testers can easily
identify a failure, zero in on the problem and access application quality.
QADirector provides pre-defined reporting as well as the ability to export data to
many popular reporting tools. Results from failed tests are easily loaded into the
defect reports of TrackRecord.
Defect Tracking
Integration between QADirector and Compuwares TrackRecord enables defects
identified during the testing process to be easily loaded into TrackRecord. With
the press of a button, test results can be loaded from other sources, including
QARun, QAHiperstation, WebCheck and File-AID/CS. QADirector stores the
defect ID with the testing suite for easy follow-up.
ActiveAnalysis
With QADirectors new ActiveAnalysis feature, it is easier than ever to detect and
troubleshoot errors during the testing process, long before the application enters
production. ActiveAnalyis combines the key features of Compuwares QACenter,
NuMega, and EcoSYSTEMS tools to merge the testing process with error
detection, debugging and coverage analysis.
10-6

Figure 10-2: ActiveAnalysis Tab in QADirector.
If QADirector detects a problem with the application, the ActiveAnalysis tools

can provide more detailed information on where and why the problem occurred.
With this detailed information, you can resolve problems quickly and efficiently
to provide a quality application.
TrackRecord
TrackRecord, also available through the NuMega product line, is an enterprisewide defect tracking toolan integral part of QACenters family of automated
testing tools. TrackRecord helps organizations establish a systematic method for
tracking software defects, from detection through resolution and verification.
TrackRecord frees up application and QA managers, project managers, testers and
developers to focus on resolving problems and improving application quality.
Efficient, Integrated Defect Reporting
TrackRecord is easily implemented into your organizations automated testing
process, providing a tracking system that helps you take control of software defect
identification and resolution. In addition to software defects, TrackRecord can be
used to track releases, features, testing assets and any other information related to
development projects.
Prioritizing and assigning defects
TrackRecord provides a structure for setting priorities and assigning
responsibility for resolving critical defects. Testers, developers and project
managers can view real-time status of defects.

10-7
Remote access to information

TrackRecord can be accessed via an Internet browser, so developers and software
testers can obtain immediate information on defects quickly and efficiently and
report defects remotely. Anyone with a browser and the appropriate access
privileges can enter defects into the repository, update items and view reports. In
distributed workgroups, team members at multiple sites can easily share status
information.
Dynamic reporting
TrackRecord routes information directly to those who need it and presents the
information in organized, easy-to-read reports that contain the most current
information in the repository. You can zero in on information or evaluate the big
picture, and view defects by a variety of categories.
TrackRecord also keeps a change history, which provides an automated audit trail
of completed workwhen it was done and by whom.
Automatic notification
The TrackRecord AutoAlert module provides electronic mail notification of both
reported defects and any new information that enters the TrackRecord repository.
Customizable database
The TrackRecord repository stores defect information for a wide variety of preset
data types. In addition, TrackRecord provides an unparalleled level of
configurability, enabling you to easily customize a system that helps you take
control of your problem resolution process.
Integration with source code management
TrackRecord tracks the associations between low-level source file changes and
specific high-level modifications, such as bug fixes and feature additions.
QAHiperstation
QAHiperstation is QACenters S/390 testing solution. QAHiperstation streamlines
the testing of VTAM applications, LU6.2 (APPC) and TCP/IP data traffic.
10-8

Extended testing capabilities

The advanced record, playback and comparison functions provided by
QAHiperstation enhance unit, concurrency, integration, migration, capacity and
stress testing:
Record: Captures user sessions on a keystroke-by-keystroke basis. A

global recording function lets you capture any 3270-terminal activity,
APPC (LU6.2) or TCP/IP traffic in the VTAM network.
Playback: Re-executes previously recorded sessions in slow-motion or
full-speed interactive mode, or in a batch unattended.
Compare: Automatically compares the output of the current playback
session with that of the recorded session.
Testing Flexibility
QAHiperstation uses IBMs REXX as a script language to significantly expand its
playback facility in both the online and batch environments, allowing users to
customize scripts to meet their needs.
Superior Performance Testing
QAHiperstations Global Record and Unattended Playback provide unparalleled
capabilities for controlling and executing realistic stress or volume tests.
In environments where the S/390 functions as an enterprise server, the
QAHiperstation APPC (Advanced Program-to-Program Communications) option
helps determine how APPC applications will handle high volume traffic. From its
location on the S/390, the APPC option easily captures LU6.2 traffic, replicates it
to produce the desired volume and replays the traffic to the application.
Figure 10-3: Global Record and unattended playback enable successful deployment of critical applications through
superior performance testing capabilities.
Data Management
File Manager integrates the data management and automated testing processes
and performs complete and thorough testing without requiring assistance from
10-9
technical staff. Data Management provides the ability to record, play back and
compare VSAM and DB2 data from a CICS application.
Internet Reliability
QAHiperstations unique functionality helps ensure that data exchanges between
platforms properly. Because these data streams are invisible to the user interface,
they are rarely tested thoroughly. If data traffic or the S/390 business applications
are not operating as expected then serious, hard-to-find errors and malfunctions
can occur.
QAHiperstation is the only product that will record these data streams into scripts,
which can then be replayed, compared and used to complete a thorough and
focused test of your e-commerce business applications. By allowing you to test
the viability of the data as it flows across communications links, QAHiperstation
helps your business deploy e-commerce applications with confidence.
Script Processors
QAHiperstations script processors are a set of online facilities that automate the
process of making changes to recorded test scripts. Script processors enable you
to:
improve overall testing productivity because you can execute tests without
having to edit them for changes
add value and long-term re-usability to test assets by making it easy to
modify scripts in response to changes.
Euro Script Utility

The Euro Script Utility provides currency conversion and automates script
changes for euro application testing. It helps automate processes, enforce
standards and streamline testing so you can meet your euro testing deadlines.
QAHiperstation+
QAHiperstation+ is a Windows-based interface that works in conjunction with
QAHiperstation to test S/390 applications from a desktop.
Its versatile interface thoroughly, accurately and quickly captures and
synchronizes all test activities of mission-critical VTAM applications, including
those running in IMS/DC, IDMS/DC, CICS and TSO. QAHiperstation+ also
provides enhanced functionality in the areas of screen comparisons and script
modifications for the QAHiperstation user.
10-10

QABatch
QABatch is an S/390 automated testing solution that makes it feasible to easily
and quickly perform unit, integration, system and regression testing of large MVS
batch applications throughout the lifecycle. It automates the process of creating
new test environments for any number of modified applications, comparing them
with the baseline and documenting the results.
Structured Testing Methodology
Batch testing requires that testing steps be executed in a specific sequence.
QABatch incorporates a standard process commonly used in the industry for
manual batch testing. Its automatic, consistent approach can be used repeatedly
by technical and non-technical testers.
Create and Execute a Baseline Environment
QABatch creates the baseline environment by copying the production
environment to a new test (baseline) environment.
Create and Execute the Test Environment
QABatch copies and modifies the baseline environment to reflect test job names
and file names allowing tests to be re-executed multiple times to see how each
change of a multi-change project affects a single program or the entire system.
Compare Baseline and Test Results
QABatch creates JCL to compare output files and printed output, creates JCL to
unload and compare the DB2 data and executes the comparison.
Unique Documentation Capabilities
Thorough documentation of all baseline and test execution information can be
used for audit purposes and test validation. QABatch:
produces compare documentation that can be printed directly from the

QABatch compare menu
creates a log that includes the history of QABatch baseline and test
execution
optionally archives baseline and test output that can be compared after it
has been restored to disk.
Clean Up Test and Baseline Environments

QABatch enables the user to manage each test environment (baseline, test,
modified, compare) and its output by providing the option to clean up all or
10-11
portions of the testing environments reducing the potential for false mismatches
due to old test data.
QARun
QARun is a functional test automation tool that allows organizations to validate
business critical applications whether client/server, e-commerce or ERP. QARuns
automated capabilities help QA and application managers, technical directors and
webmasters verify that their applications will function as expected.
It includes support for
web-based applications including ActiveX, Java, HTML and DHTML

running under Internet Explorer or Netscape.
4GLs such as UNIFACE, Visual Basic and PowerBuilder
packaged applications such as SAP, Oracle and PeopleSoft.
QARun delivers a comprehensive, robust and powerful scripting language merged

with rapid, easy testing. The process is simplified by automating the creation of
test scripts through a QARun function that automatically translates user actions
into script commands. QARun provides a test development environment that
includes the following:
Script Editor for automatic script generation

Log Viewer for viewing test results and test report creation
Check Map for creating a wide range of checks for easy, extensive test
verification
Event definitions for automatic error recovery.
It records the users actions, translating each action and response into an objectoriented script that can be replayed. Advanced mapping techniques ensure that
the script will play back despite possible environmental changes. For instance,
scripts can be written to be independent of language changes to the application.
QARun also offers a built-in synchronization method that allows scripts to
synchronize automatically with the application under test.
Verification is done by inserting checks that can be predefined or can be created
during recording through the wizard interface. All the check information is held
externally to the scripts in a table where it can be re-used across multiple test
cases and is maintained only in one place. All the test assets are centrally stored in
a single repository to allow for ease of cross-referencing, low maintenance and
high security.
WebCheck
WebCheck is QACenters complete Web site testing solution. With the growing ecommerce market, it is very important that Web sites are fast and accurate, as the
competition is only a mouse click away. WebCheck allows organizations to scan
10-12

their Web sites for more than 50 potential types of problems such as broken links,
pages with stale content, orphaned (unused) files, pages with slow download
times and much more. Comprehensive graphical reports are generated that detail
errors to be fixed and performance information, including trends on your site over
time. WebCheck allows project teams to test the quality, usability and
accessibility of even the largest web sites.
WebCheck is integrated with QADirector, Compuware's test management solution
for full lifecycle testing of distributed large-scale applications. From a single point
of control, you can easily plan your tests, execute your tests, vary the test
environment, dynamically analyze the application under test, analyze the test
results and submit defects. QADirector and WebCheck allow testers, developers
and managers to thoroughly test an application with improved test repeatability,
sharing and ease of use.
WebCheck functionality is incorporated into QACenter's QARun automated
testing tool. QARun users can include site checks within QARun scripts,
providing site analysis and reporting. The advanced features available through the
WebCheck option include:
Test FORMS tags. Some of the most sophisticated web sites utilize
FORM tags as a method to secure access to entire sections of their site and
to prompt for user input (such as order forms). WebCheck can tell you if
these FORM tags work and if their links work.
Use WebCheck's graphical interface to navigate and analyze web site

problems. WebCheck allows you to interactively navigate through a web
site, performing site checks and viewing extensive site information and
results.
WebCheck has many options to help you manage your web site, including these
features: basic maintenance, mapping and organization, repair and analysis and
reporting features.
QALoad
QALoad is QACenters automated load and performance testing solution for
client/server, ERP and E-Commerce applications. QALoad helps pinpoint
problems, optimize system performance and ensure successful application
deployment by simulating hundreds or thousands of users performing key
business transactions on your system.
Combined with EcoSYSTEMS for monitoring the server, database and network
components of client/server applications, Compuware delivers the total end-toend solution, giving you the confidence to deploy mission critical applications
successfully.

10-13
QALoad can support load and performance testing of applications that use the
following middleware or protocol:
2-tier:
n-tier:
ERP:
e-commerce:
legacy:
ODBC, Oracle, Sybase, SQL Server, DB2 UDB

Tuxedo, DCOM, CORBA, UNIFACE
SAP R/3, PeopleSoft, Oracle
HTTP, SSL, digital certificates, IIOP, FTP
Windows Sockets, VT100-520 (Telnet)
Interactive Analysis, Testing and Debugging Tools

The XPEDITER suite of testing tools provides you with industry leading
capabilities to analyze, convert, debug and test application code. Supporting the
leading mainframe and client/server development environments, XPEDITER
gives you an inside look at source codehow it works, and how to fix it.
XPEDITER provides you with an interactive solution for analyzing, testing and
delivering quality applications. Using XPEDITER, you are equipped with a
toolset that enables you to:
create quality applications

quickly identify executed and unexecuted code
determine risk associated with moving new and modified applications into
production
better understand applications using graphical charts and diagrams
view and interact with source code online
set breakpoints
simulate future and past dates and times without dedicating a separate
system or affecting other jobs.
test, edit and debug in one session without recompiling
record and re-use test information
trace program logic.
The XPEDITER product suite includes:
10-14
XPEDITER/CICS
XPEDITER/TSO
XPEDITER/IMS
XPEDITER/Code Coverage
XPEDITER/DevEnterprise
XPEDITER for DB2
XPEDITER/Xchange
XPEDITER/SQL
DBPartner.
XPEDITER is accessed through a 3270 terminal in TSO, CICS and IMS/DC

environments and on a PC client in Windows 95/98/NT environments for
programs written in COBOL, PL/I, Assembler, PL/SQL and Transact SQL and
SQL. XPEDITER can access DB2, Oracle, IMS, Sybase and Microsoft SQL
Server databases.
XPEDITER/CICS
XPEDITER/CICS offers production support by providing you the ability to bring
the file utility into production to log any changes made. As organizations move
converted test code back into production, you are bound to encounter unforeseen
errors. With XPEDITER/CICS, you do not need to re-create errors in a test
region. You can solve problems in a fully secured Production environment using
three unique operations: Diagnosis, Utilities and Diagnosis/Utilities.
Diagnosis mode provides access to all of XPEDITERs problem diagnosis and
storage protection capabilities without the risk of data integrity violations.
Utilities mode provides access to XPEDITERs file, source listing and storage
display utilities without incurring overhead associated with XPEDITERs
debugging functions and user exits.
Diagnosis/Utilities, the most restricted mode, prevents you from tracing program
logic, trapping abends, monitoring execution or updating working storage. It
does, however, provide viewing capabilities for file, source listing and storage
display utilities.
XPEDITER/TSO
XPEDITER/TSO offers the following three modes:
interactive
batch connect
unattended batch.
You can test multiple steps within batch jobs in either attended or unattended
mode. Since time is often a critical factor in testing of projects large and small,
unattended mode allows you to use a predefined list of commands to test after
hours and on weekends.

10-15
Figure 10-4: Three Modes of XPEDITER/TSO Testing
XPEDITER/IMS
XPEDITER/IMS gives you a safe, easy method for testing IMS/DC programs.
XPEDITER can run tests in a live IMS/DC region, instead of simulating a
region through BTS providing a seamless testing environment.
XPEDITER/Code Coverage
XPEDITER/Code Coverage helps determine what percentage of a mainframe
application has been executed, allowing IT groups to intensify testing efforts
where needed. Code Coverage performs critical business function analysis in the
initial phases of application modification projects, such as web-enabling an
existing legacy application. As quality applications are becoming more important,
this advanced analysis and testing tool improves and documents the quality of
testing. Code Coverage enables programmers to:
10-16
analyze code executed for specific business functions

understand logic executed across business functions through data flow
analysis
identify the amount and area of an application exercised, and how often it
was exercised
determine the percentage of modified code tested
measure the relative risk points in a program
use graphical structure views to develop test data and user input
requirements
generate automatic breakpoints that help improve testing.
document testing efforts and create an audit trail of program test

execution.
With simple point-and-click navigation, programmers can move from a systemlevel view of programs into source code to assess the riskiest points in the
program, aiding and improving the testing effort. Code Coverage requires no
source code instrumentation, and integrates easily into current testing
methodologies.
As tests are run, Code Coverage collects test execution statistics. This
informationtaken from batch, IMS and CICS programsis used to populate a
Code Coverage repository. The collected information can then be used for a
single test execution, multiple tests or multiple tests run over a period of time.
Results are stored hierarchically by the following criteria: system grouping
(payroll, inventory, order entry), business function being tested (add order, update
order), user(s) performing the testing, load module, specific program and by
compile and optimization date for different versions of the same program.
Figure 10-5: The XPEDITER/DevEnterprise Program Analyzer allows programmers to view program source,
program structure charts and code coverage statistics easily.
XPEDITER/DevEnterprise
Offering an easy-to-use graphical interface, XPEDITER/DevEnterprise provides
programmers with a view of their application. The view works with other
DevEnterprise capabilities to enhance analysis and system testing. Although a

10-17
substantial portion of the product is graphical workstation based, DevEnterprise

fully integrates with XPEDITER on the mainframe.
The complete set of integrated tools provided by DevEnterprise:
an application analyzer
a data analyzer
a program analyzer
code segment creator
a sophisticated editor
code coverage documentation
batch and CICS debuggers
high-speed file transfer
easy-to-use compile and job manager.
DevEnterprise offers development capabilities to statically analyze business

functions, and presents the ability to easily change source code and re-compile.
Programmers set program stops and breakpoints within their source code,
enabling them to highlight and locate areas that need run-time analysis.
Application programs can be segmented easily, simplifying their re-use in other
applications.
Programmers & analysts can easily access detailed information on the testing
concentration of specific programs and systems with DevEnterprise. This
information includes detailed code coverage reports for risk assessment purposes,
allowing programmers a greater understanding of the possible risk in placing the
application, in its current state, into production.
DevEnterprise also gives its users an insight into how an application change will
impact the activity of that application and other applications it is linked to.
DevEnterprise provides businesses with the ability to direct testing activity to the
areas within the application where additional analysis and testing are needed.
Complex applications, such as those used to execute e-commerce activities,
benefit from DevEnterprise's thorough analysis, allowing businesses to move
applications into production with confidence.
10-18

Figure 10-6: The XPEDITER/DevEnterprise Application Analyzer allows programmers to quickly view
system connections and ascertain which programs are called when the data is used.
XPEDITER for DB2

XPEDITER for DB2 allows dynamic inserts, validation and EXPLAINs of SQL
statements without the need to recompile or rebind. XPEDITER identifies DB2
error codes, so that you can understand and resolve DB2 performance problems
without leaving your test session.
XPEDITER/Xchange
XPEDITER/Xchange allows you to simulate any date and time between 1900 and
2041 without requiring any changes to program modules or JCL. Moreover, you
can use XPEDITER/Xchange to test a programs reaction to any date and time
without dedicating a separate system or affecting other jobs running
simultaneously. An ISPF interface allows you to specify the date and time for
single or multiple jobs, steps, procedure steps and program names. You can also
exchange entire CICS regions or set individual requests by terminal, transaction
or program. XPEDITER/Xchange supports many environments and languages:
Language Environment/370
DB2
IMS/DC
CICS
COBOL, Assembler and PL/I.

10-19
To make identifying dates easier, XPEDITER has an easy to understand

journaling facility that records a list of all date exchanges. This independent file
can be used for reporting and for quality assurance.
XPEDITER/SQL
XPEDITER/SQL is an application development workbench for building and
testing Oracle, Sybase and Microsoft SQL Server stored procedures. It allows
developers to debug applications throughout the application lifecycle and also
provides interactive debugging capabilities for unit testing and integration testing
of stored procedures.
Using XPEDITER/SQL to navigate through database objects, developers can
create and edit stored procedures, as well as develop and test in a safe team
environment. XPEDITER/SQLs comprehensive debugger lets developers debug
procedures, functions and triggers. And, its Hook Up facility enables developers
to debug stored procedures in context to a calling application.
Figure 10-7: XPEDITER/SQL Debugging Screen.
XPEDITER/SQL is an integrated workbench comprising several key components:

The Object Browser displays a list of database objects grouped by object type and
schema. It allows users to launch a debugging session for a procedure or function,
10-20

and developers can navigate through object lists, letting users analyze
relationships among objects, group object shortcuts and check out objects.
XPEDITER/SQLs Team Management approach can be easily configured to
facilitate team development. Project folders can be created for individual use or
shared by team members. Sharing project folders gives project members the
ability to work together with a common view of the development environment.
The Template Editor function provides a graphical user interface that allows
teams to create custom templates according to specific project requirements. They
also have the option to revise existing, built-in templates based on their needs.
XPEDITER/SQLs debugging capabilities provide developers the ability to
interactively debug procedures executed within the product. The debugger
provides comprehensive support of all variable types and contains many
convenient extras, such as conditional breakpoints and lock monitoring.
The XPEDITER/SQL Hook Up facility allows developers to debug stored
procedures in context to the actual data parameters of the calling application and
to see how the stored procedure processes those variables.
With the Hook Up facility, developers can observe and change input or output
parameters being returned to the calling application. With the ability to perform
testing of both the procedure and calling application, XPEDITER/SQL provides
the most comprehensive distributed application testing available.
DBPartner for Oracle
DBPartner is an SQL analysis, tuning and debugging solution for Oracle
applications. Unlike any other product, DBPartner enables application developers
to tune and debug SQL and PL/SQL in context to an application. Developers can
use DBPartner throughout the application life cycle to gain a quick understanding
of the underlying processes taking place in their Windows applications. By giving
developers the ability to tune, DBPartner saves companies time and money by
addressing potential problems before production.
DBPartners unique capturing ability enables users to capture SQL statements
from multiple sources, including running programs, program source and the
database. Developers can understand exactly what is happening within their
application and identify what is causing performance bottlenecks.
The Tuning Wizard allows developers to isolate and then display the most
offensive SQL in graphical form, easily identifying problem areas. Its four-step
tuning process quickly finds and qualifies the worst performing SQL, creates
alternative versions that may perform more efficiently and compares each version
to determine the best solution. DBPartners Auto Tune feature makes it easy for
both experienced and novice developers to perform tuning.
DBAs can also use DBPartner to tune a production database.
10-21
Together with Compuwares XPEDITER/SQL for Oracle, DBPartner provides a

time-efficient way to tune and debug applications seamlessly during development,
increasing application reliability.
File and Data Management Tools

Compuwares File-AID products give you the ability to access and manage data
throughout the enterprise, regardless of its format, enabling you to optimize the
delivery and support of business-critical applications. Using a request-driven
interface, you can drive the File-AID products to quickly and easily find, create,
extract, transfer, convert, load, edit, compare, migrate and maintain data.
The File-AID line of products is designed to boost your productivity by allowing
direct, consistent, safe and simplified access regardless of the file type or database
structure. With File-AID you can browse, edit, copy, print, select, extract, load,
reformat, compare and allocate files through easy to use ISPF-like and GUI
interfaces, in both MVS mainframe and client/server environments. The file and
database types supported include:
QSAM
ISAM
PDS
IMS
BDAM
CICS/OS2
DB2
Oracle
Sybase
DB2 UDB
Informix
VSAM
XDB
Btrieve
Sequential
MS SQL Server
COBOL and PL/I record layouts and database structures are used to format the
data, allowing you to view, select and work with the data in the most intuitive
way possible. The File-AID products adhere to the rules defined by logical and
application relationships to allow safe editing and provide relationally intact
subsets.
File-AID/MVS
File-AID/MVS provides a wide range of file and data management capabilities
required to perform everyday data related tasks in a batch or interactive
environment. These capabilities include browse, edit, copy, reformat, compare
and print data from any standard MVS file type (VSAM, ISAM, sequential,
BDAM, PDS and IAM). File-AID/MVS also eliminates record length and file
size restrictions and provides an audit trail of all changes made during an edit
session. Existing COBOL and PL/I record layouts can be used as templates to
display data with the actual field names and formats.
10-22

File-AID/Data Solutions
File-AID/Data Solutions provides consistent, efficient methods to create accurate
test data and perform data conversion and analysis without having to write onetime programs or endure tedious manual efforts. With File-AID/Data Solutions,
developers can automatically generate test data, encrypt sensitive data, perform
results analysis, execute mass data changes, convert euro values and validate EDI
data values.
File-AID for DB2
File-AID for DB2 enables you to create, populate, browse, edit, refresh and where
appropriate, authorize DB2 objects. File-AID for DB2 enables you to work with
single or multiple tables, whether related or unrelated. File-AID for DB2 also
helps move applications into the test cycle faster by validating and EXPLAINing
program SQL prior to the compile and bind process.
File-AID for IMS
File-AID for IMS enables you to edit, browse, extract and load IMS databases
under TSO, IMS/DC or CICS. It provides you with the ability to copy data
between databases selectively and to customize the data to create efficient test
databases. The File-AID for IMS/FLEX option enables you to work with large
databases and schedule jobs for off-peak hours.
File-AID/Related Data XPERT (RDX)
File-AID/RDX easily extracts and loads complete subsets of related VSAM or
sequential files, IMS and DB2 data without coding SQL or writing programs.
With File-AID/RDX, you can create a small test environment that contains
exactly what you need to test an application thoroughly and effectively. Because
you are not testing with full-production copies, you can complete more test cycles
in less time.
DBA-XPERT for DB2
Designed as a complete solution for managing DB2 subsystems, DBA-XPERT
for DB2 automates and integrates all of the capabilities needed to analyze and
maintain the DB2 environment. For testing, DBA-XPERT for DB2 can clone, or
duplicate, a test database structure and its contents to create multiple test
environments for individual or groups of programmers as needed.
File-AID/CS
File-AID/CS is a test data management tool designed to help developers become
more efficient when working with data as they develop, test and support any
client/server applications. It allows developers to perform all data-related tasks
10-23
through a common, graphical interface. Like mainframe File-AID products, FileAID/CS eliminates the need to write programs or scripts, code SQL or use
multiple utilities.
File-AID/CS provides capabilities for copying, subsetting, converting, editing,
browsing, generating, comparing, reformatting, transferring and aging data. All
these capabilities save developers time when they have to create their baseline test
data, validate test data results, restore their baseline data and deal with any
changes to their test data environment.
File-AID/Express and File-AID/Express for MVS
File-AID/Express and File-AID/Express for MVS let developers quickly and
easily design and execute large-scale production data conversions and Enterprise
Application Integration (EAI) projects without writing programs. They have the
power and scalability to perform complex conversions between legacy and
relational data.
With their high performance, high capacity design, File-AID/Express and FileAID/Express for MVS can convert data for one-time migrations and for both
temporary and permanent data interfaces between applications in production.
Developers can also convert subsets of data required for testing the application or
the interface. By controlling the risks associated with data conversion and EAI,
File-AID/Express and File-AID/Express for MVS help projects stay on time and
within budget.
File-AID/Express for PeopleSoft
File-AID/Express for PeopleSoft simplifies large-scale conversions of data to a
PeopleSoft application or from it. It allows developers to work logically with
their PeopleSoft data, without any knowledge of the underlying physical database.
It helps ensure that any data loaded into the PeopleSoft application conforms to
all business rules and constraints imposed by the PeopleSoft application or the
clients business processes.
File and Data Management Tool Examples

Comparing Test Data with File-AID/MVS
Test results must be validated, a process that often involves a tedious manual
comparison of before and after test files. File-AID/MVSs full-featured data
comparison allows you to compare two files with different record layouts and any
file organization supported by File-AID/MVS.
10-24

Figure 10-8: Comparing Test Data with File-AID/MVS.
Comparing Test Data with File-AID/CS

File-AID/CS automatically compares the actual data results with the expected
results for accurate detection of differences.
Figure 10-9: Comparing Test Data with File-AID/CS
The comparison can be restricted to specific columns or rows in the tables,

allowing you to focus solely on the data that is important to a specific test.

10-25
Transforming data with File-AID/Express

The powerful auto-mapping capabilities of File-AID/Express reduce the manual
repetitive process of matching source to target data elements.
Figure 10-10: Transforming Data with File-AID/Express
Generating test data with File-AID/Data Solutions

File-AID/Data Solutions can be used to add new records and field values and fill
fields with valid values from another file. New values can be generated
sequentially, randomly or through a table look-up
Figure 10-11: Generating Test Data with File-AID/Data Solutions
10-26

Cloning a test database

With DBA-XPERT for DB2 you can clone a database and its contents to create
multiple test databases as needed.
Figure 10-12: Cloning Data
Edit DB2 data

File-AID for DB2 performs full-screen browse and edit of DB2 tables with ISPFlike simplicity. You can view column names, length and format in browse/edit
with optional display choices for primary and foreign keys. Insert, copy, delete or
repeat an individual row or blocks of rows of data using standard ISPF-like
commands. Global FIND and CHANGE commands are also available.
Figure 10-13: Edit DB2 Data

10-27
Subsetting data with File-AID/RDX

File-AID/RDX eliminates the need to use full table copies to maintain
relationships. Referential integrity information is automatically loaded, and easy
to use ISPF-like screens guide you in defining application relationships.
Figure 10-14: Subsetting Data with File-AID/RDX
Hierarchy diagramHIER command

File-AID for IMS can help you create, browse and edit data in your IMS
databases. It can also help you review the complicated IMS hierarchy to
determine the structure of the data you are working with.
Figure 10-15: Database Hierarchy Diagram
10-28

Fault Management Tools

IT management is constantly faced with challenges such as enterprise application
integration (EAI), e-commerce, the euro and application backlog.
Abend-AID products can help significantly with these challenges by automating
the fault resolution process to help managers and problem solvers quickly resolve
mission-critical application problems in test.
When a legacy application is modified (for EAI or e-commerce, for example), it
also will need to be tested. The more complex the modifications, the more
extensive the required testing. When testing reveals problems, product managers
and problem solvers without Abend-AID may be faced with manually:
identifying the problems

collecting relevant information
analyzing dumps (if available)
recreating problems
managing the fault resolution process without reliable information.
Testing and the ensuing problem solving are critical to prevent problems from
entering into production. But limited test time needs to be spent on testingnot
problem solvingto help ensure a quality application for production.
By automating the fault resolution process with Abend-AID products, you can
limit your testing time to evaluating an application thoroughly and you will spend
much less time on tedious problem-solving.
Abend-AID products help you proactively manage the problem solving process.
When an application fails, Abend-AID products automatically:
detect and log the problem

notify the problem-solving team that a problem has occurred so they can
take immediate action
provide detailed problem-solving information for tech support and
developers
track the problem solving process so managers know the exact status of
the problem at any time
provide management level status reporting to help managers first solve the
most important problems and also to efficiently allocate resources.
These capabilities mean that all problems are reported and are efficiently resolved
in the shortest possible timeleaving more time for testing.

10-29
Mainframe Products
Abend-AID Fault Manager
Abend-AID Fault Manager helps managers monitor, measure, control and
improve the fault resolution process for OS/390 environments. Fault Manager
also helps the problem-solving team members work more efficiently. These
capabilities significantly help to reduce costly downtime and provide a higher
level of service for an enterprise and its customers. The product is ideally suited
for projects such as e-commerce, enterprise application integration as well
meeting service level agreements. Fault Manager displays a real-time Graphical
User Interface (GUI) view of the fault resolution activity in an enterprise that
provides managers with the information they need to manage the problem
resolution process with confidence. In addition to real-time information, Fault
Manager also presents historical information for benchmark comparisons and
trend analysis to help improve the fault resolution process. For convenient access,
Fault Manager displays information either online or over the Internet. Fault
Manager readily shares data with help desk applications, word processors and
spreadsheets. The following sections are just a sample of Fault Managers
powerful capabilities.
Real-Time and Historical Graphical Analysis
Abend-AID Fault Manager automatically supplies the information required to
manage fault resolution problems, projects and resources. Fault Manager includes
pre-defined graphs and reports that can be customized to meet specific
requirements and saved for re-use.
Figure 10-16: Abend-AID Fault Manager's Graphical Analysis
10-30

Figure 10-16 shows a real-time status of the number of abends that occur by
specific application. Graphs can be compared over a period of time to confirm
improvements or disclose chronic problems. This type of high-level information
can be used in testing to identify quickly problem applications that might require
additional attention. This big picture information can be helpful in meeting
testing schedules.
Monitor the Process
Figure 10-17: Abend-AID Fault Managers Fault Resolution Process Monitoring
Figure 10-17 presents a high-level view of the problem resolution activity across
an entire system. This graph shows a real-time status of the fault resolution
process. The graph shows the problems automatically detected and logged, how
many are not reviewed, open, assigned, pending validation, pending
documentation or closed. While this graph depicts the status of an entire system, a
version of this graph can be run for a particular application to help drill down
on a specific issue. This big picture information is invaluable in meeting testing
schedules.

10-31
Detailed Reports Readily Available
Figure 10-18: Abend-AID Fault Managers Detailed Reports
Fault Manager provides detailed reports (Figure 10-18). This sample report lists
all Priority 1 problems. With Fault Manager problems will be listed
automatically as Priority 1, 2 or 3 when they occur. Priority can be configured
by criteria such as application, type of problem or program name. Problems can
be re-prioritized at anytime to upgrade or downgrade urgency. For all Priority 1
problems, this report lists the name of the person the problem has been assigned
to, the job name, program name, project (application) name, severity and
description. This report also shows which problems are affecting the most critical
applications and who is directly responsible for resolving them. A similar report
based on a specific application can also be created; for example, a report listing
Priority 1 problems for an Order/Entry application.
Reallocate Resources and Change Priorities with Confidence
After identifying problem areas and drilling down to detailed information, reassigning staff members to cover high priority problems might become necessary.
This graph report (Figure 10-19) displays a clear picture of the workload of each
staff member, so projects can be re-assigned to those with less critical projects or
lighter work loads, to keep projects on track.
10-32

Figure 10-19: Reallocating Resources
Abend-AID XLS
Abend-AID is the industry standard and most widely used product for the
diagnosis of mainframe application program failures across a variety of databases
(such as DB/2, IMS or CA-IDMS) and programming languages (such as COBOL,
PL/I, Assembler, FORTRAN or LE).
Abend-AID merges actual source code (such as COBOL Assembler or PL/I),
diagnostic information and program storage into a single report and provides online, menu-driven access for rapid problem resolution. The following sections are
just a sample of Abend-AID XLSs powerful capabilities.
The Abend-AID XLS Directory
The Abend-AID XLS Directory (Figure 10-20) provides an up-to-the-minute
overview of all fault activity in test and serves as a single point of control to help
you manage fault resolution. It lists all faults as they occur to help you quickly
identify them and prioritize their resolution. From the Directory you can focus in
on a specific fault for more detail. The directory also can provide hard-copy
documentation to certify test results and establish an audit trail.

10-33
Figure 10-20: Abend-AID XLS Directory
Abend-AID XLS Diagnostics

Abend-AID XLS automatically runs comprehensive diagnostics (Figure 10-21) on
an error when it occurs, and it then displays the results.
Figure 10-21: Abend-AID XLS Diagnostics
In most cases this is the only place you will need to go to resolve the fault. The
diagnostics focus on the incorrect program statement and explain the error
message. Diagnostics eliminate wasting time hunting for answers.
The diagnostics display the statement number and the actual statement in error.
Each field is displayed by field name, picture type and contents. The diagnostics
tell you what happened and where the failure occurred in the program, as well as
suggested corrective actions. Because the actual source code is merged in the
10-34

diagnostic report, programmers dont have to waste time locating the compile
listing and cross-referencing it.
For LE conditions, the diagnostics deliver pertinent in-depth information
including:
library mismatches
library name and version code for Run Time Library Services
contents of the LE control block in use at the time of the error
diagnostics for all LE COBOL and PL/1 messages and codes
first-level diagnostics of system and LE faults and condition codes for the
current release of OS/390
associated Language Environment and information on programs in the
calling chain.
Abend-AID XLS Menu

The Abend-AID XLS Online Menu (Figure 10-22) provides fast access to
pertinent information to help reduce the time spent on solving problems.
Figure 10-22: Abend-AID XLS Menu

10-35
Abend-AID XLS File Section

The Abend-AID XLS File Section (Figure 10-23) of the report identifies the paragraph
that contained the failing statement as well as the paragraph name and text of the last I/O
operation.
Figure 10-23: XLS File Section
Abend-AID XLS Source Code Listing

Abend-AID XLS identifies the statement and error and allows the problem solver
to immediately start analyzing the program logic (Figure 10-24).
Figure 10-24: Abend-AID XLS Program Source Code Listing
Abend-AID XLS provides the actual program source code (COBOL, PL/1 or
Assembler) to identify the instructions performed prior to the abending statement.
With Abend-AID XLS, the statement in error is identified so that programmers
10-36

can easily navigate to areas of interest, within the program listing section, by
using find commands to see program logic.
Abend-AID XLS Working Storage
Abend-AID XLS Working Storage (Figure 10-25) displays storage in familiar
COBOL , Assembler or PL/1 format as opposed to hexadecimal.
Figure 10-25: Abend-AID XLS Working Storage
You can check an applications field lengths and data. Formatted storage is
presented in a familiar format just the way a programmer would see it in the
program. Formatted storage identifies field name, definition and values. It also
displays, redefines and formats multi-dimensional tables and shows their index
values.

10-37
Abend-AID XLS DB2 Support

Abend-AID XLS for DB2 (Figure 10-26) generates detailed diagnostics for nonzero SQL codes, provides a complete diagnostic of the DB2 abend and suggests a
course of action.
Figure 10-26: Abend-AID XLS DB2 Support
Abend-AID XLS for DB2 also provides the location and text of the SQL
statement in error. For non-zero SQL codes, it identifies host variables and
contents and pinpoints the errors, as well as pertinent information from the DB2
catalogue. Abend-AID XLS for DB2 identifies and analyzes errors that occur in
stored procedures.
CICS Abend-AID/FX
For the CICS environment, Compuware provides a single comprehensive fault
management solution for CICS transactions, DB2 errors and region failures.
CICS Abend-AID/FX thoroughly analyzes the causes of CICS faults, which
enables application and system programmers to resolve critical problems quickly
regardless of a systems complexity or new technology. The following sections
are just a sample of CICS Abend-AID/FXs powerful capabilities.
10-38

The CICS Abend-AID/FX Directory

The CICS Abend-AID/FX Directory (Figure 10-27) provides an up-to-the-minute
overview of all fault activity including transaction and region failures in test and
serves as a single point of control to help you manage fault resolution. It lists all
faults, as they occur, to help you quickly identify them and prioritize their
resolution. From the Directory, you can focus on a specific fault for more detail.
The directory also can provide hard-copy documentation to certify test results and
establish an audit trail.
Figure 10-27: CICS Abend-AID/FX Directory

10-39
CICS Abend-AID/FX Diagnostics

CICS Abend-AID/FX automatically runs comprehensive diagnostics (Figure 1028) on an error when it occurs and displays the results.
Figure 10-28: CICS Abend-AID/FX Diagnostics
In most cases its the only place you will need to go to resolve the fault. The
diagnostics zero in on the program statement in error and explain the error
message. Diagnostics eliminate hunting for answers. The diagnostics display
statement number and the actual statement in error. Each field is displayed by
field name, picture type and contents. The diagnostics tell you what happened and
where the failure occurred in the program. Because the actual source code is
merged in the diagnostic report, programmers dont have to waste time locating
the compile listing and cross referencing it.
CICS Abend-AID/FX Menu
The CICS Abend-AID/FX Online Menu (Figure 10-29) provides fast access to
pertinent information to help reduce the time spent on solving problems.
10-40

Figure 10-29: CICS Abend-AID/FX Menu
CICS Abend-AID/FX Program Source Code Listing

CICS Abend-AID/FX identifies the statement and error and allows the problem
solver to immediately start analyzing the program logic (Figure 10-30).
Figure 10-30: CICS Abend-AID/FX Program Source Code Listing
CICS Abend-AID/FX provides the actual program source code (COBOL or PL/1)
to identify the instructions performed prior to the abending statement. With CICS
Abend-AID/FX, the statement in error is identified so that programmers can
easily navigate to areas of interest within the program listing section by using find
commands to see program logic.

10-41
CICS Abend-AID/FX Working Storage

CICS Abend-AID/FX Working Storage (Figure 10-31) displays storage in
familiar COBOL or PL/1 format as opposed to hexadecimal. You can check an
applications field lengths and data. Formatted storage is presented in a familiar
format just the way a programmer would see it in the program. Formatted storage
identifies field name, definition and values. It also displays, redefines and
formats multi-dimensional tables and shows their index values.
Figure 10-31: CICS Abend-AID/FX Work in Storage
CICS Abend-AID/FX DB2 Support

CICS Abend-AID/FX for DB2 (Figure 10-32) generates detailed diagnostics for
non-zero SQL codes, provides a complete diagnostic of the DB2 abend and a
suggested course of action.
Figure 10-32: CICS Abend-AID/FX DB2 Support
10-42

CICS Abend-AID/FX for DB2 also provides the location and text of the SQL
statement in error. For non-zero SQL codes it identifies host variables and
contents and pinpoints the errors. This allows application programmers to be
independent of DBAs freeing up resources.
Professional Services
Automated Testing ServicesQASolutions
QASolutions is a unique offering designed to help companies thoroughly test
business-critical applications and establish a proven and repeatable process in
working toward application quality.
Compuwares QASolutions consultants make use of QualityPoint when
implementing a testing process in a particular project area.
QualityPoint is a patent pending, trademarked, software testing methodology
which works as a Stand-Alone application used by our QASolutions consultants to
build a testing process customized to a particular client's needs. QualityPoint links
"readiness" metrics directly to the business requirements defined at the initiation
of a project. It provides a detailed testing roadmap and step-by-step procedures to
eliminate problems inherent in projects staffed by transient resources.
A full QualityPoint implementation consists of four major phases: testing
assessment, test project planning, test execution and project closure. The test
execution phase consists of six key process areas (KPAs): test planning, test case
development, test environment preparation, test execution, test results analysis
and management reporting. The result of a full QualityPoint implementation is a
repeatable testing process, represented in deliverables and via an intranet web site,
where your test process is fully described and available for re-use for other
projects.
Testing Assessment and Test Project Planning phases represent an all-inclusive
assessment of your businesss current testing methods, procedures and existing IT
skill set.
In the Test Execution and Management reporting phases, our QASolutions
testing experts will set up the testing environment, including the tools and
standards. At the core of the implementation of these phases are Compuwares
QACenter suite of Automated Testing Tools. These and other Compuware tools
provide the technical foundations on which the testing process is built. Once the
testing environment is established, Compuware testing experts will take the client
IT team through a full testing cycle, from definition of a testing strategy to
development and execution of testing scripts and analysis of the results.

10-43
Our QASolutions teams can take the results of a QualityPoint implementation and
expand it to meet the needs for testing and QA for multiple areas of our clients
organizations. In many cases, this may lead to the building of a testing center
within the organization, based on the repeatable testing process built using
QualityPoint.
Training Services
Compuwares Education Resources Group supports QASolutions with classes to
quickly educate you on Compuwares tools.
Abend-AID/Fault Management courses
Programmers and problem solvers are trained to resolve mission-critical
application problems quickly in the OS/390 environment. Compuware AbendAID fault management products help reduce testing time and costly application
downtime. Training on these real-time tools enables programmers and problem
solvers to manage and improve the fault-resolution process.
QACenter/Automated Testing courses
Testers learn how to use the Compuware QACenter suite of testing toolsto
automate testingthroughout the application life cycle. QACenter testing tools,
test-process management and testing services keep business-critical information
flowing across mainframe and client/server environments. By automating the
repetitive, time-consuming steps involved in testing, QACenter helps
programmers test quickly, accurately and cost-effectively and deliver quality
applications on time.
File-AID/File & Data Management courses
Developers learn quickly and easily how to use File-AID products to find, create,
extract, transfer, fix, convert, load, edit, age and compare data. Learning about
File-AID capabilities can help users create effective test data, identify and fix
production problems quickly and simplify the data management process
throughout the enterprise
XPEDITER/Interactive Analysis & Debugging courses
Developers learn how to use XPEDITER source-based, interactive testing and
debugging capabilities to understand and resolve program logic errors quickly.
XPEDITER tools can help improve application quality across multiple platforms
for developers who are building new e-commerce applications or extending
mainframe applications to the web environment.
10-44

NuMega courses
NuMega solutions are tightly integrated tool suites that enable developers to
debug, test, verify and accelerate the development of Windows-based, Java-based
and web-based applications for the enterprise and e-commerce. Participants learn
how to improve project management; increase productivity; raise code quality;
support workflow and communication standards; and facilitate the resolution of
software errors
CyberClass
CyberClass is an adaptation of the standard in-class, instructor-led client/server
training. You can attend a CyberClass from the comfort of your site, via phone
and Internet access. Your remotely located CyberClass instructor will use a
combination of telephone conferencing and online remote presentations to lead
you through conceptual topics, demonstrate features and techniques and guide
you through workshops.

10-45
Attachments
Attachment A Test Strategy Template

[Company Logo]
Your company logo goes here
[Project Name]
Name of Project/Product to be Tested
Test Strategy Template

[Document Identifier]
Referencing Identifier for Configuration Management
Release [x]
Release of the Software to be Tested
Date
Date the Test Strategy was initiated/changed

Attachments
Revision History
Revision
1.0
Comments
Initial Revision
Author
Date
MM/DD/YYYY
Distribution List
Name
Title
Approvals
Name
Title
Date

Attachments
1. Identifier
This is a comprehensive Test Strategy for the [product name to be released]. The identifier is
(xxxxxxx).
2. Introduction
2.1 Purpose
Describe the purpose of the test strategy. Initial implementation of Payroll testing.
2.2 Goals
Explain the goals of the testing effort.
2.3 Test Scope

Please refer to the [project] requirements document. Feature/function. Interfaces
another ABC accounting system. Not to include development level testing.
2.4 Test Environment Description

Give a high overview of the test environment and configurations that the application will
be tested on.
2.5 References
2.5.1 Acronyms
The following acronyms are used during the testing effort:
2.5.2 Definitions
The following definitions are used during the testing effort:
2.5.3 Documents
The following documents are sources of information to the testing effort:

Attachments
3. Organization
3.1 Roles and Responsibilities
The following groups that are defined below are participants in the [product name] testing
process.
Role
Responsibility
3.2 Staffing and Training

3.2.1 Staffing
Add staffing plan.
3.2.2 Training
Add training issues that need to be addressed for the test group.
application specific
test process training
tool training.
4. Test approach
4.1 Test Types
Show test types and their description:
Test Type
Description
Functional
Interface
Regression
Parallel Test

Attachments
4.2 Test Methods

Describe the testing methods to be used for the [project] (for example: automated and
manual testing methods).
4.3 Testing Key Process Areas

QualityPoint (QP) Key Process Area (KPA) implementation is designed to facilitate the
implementation of the Project Plan. This can be achieved using QP Key Process Areas:
Test Planning
Test Case Development
Test Environment Preparation
Test Execution
Test Results Analysis
Management Reporting.
4.3.1 Test Planning (KPA 1)

Describe the test planning activities associated with this project.
4.3.2 Test Case Development (KPA 2)
Describe how test cases are going to be developed.
4.3.3 Test Environment Preparation (KPA 3)
Describe any preparation that needs to be done before the test environment is ready for test
execution. May also describe software/hardware configurations.
4.3.4 Test Execution (KPA 4)
Describe Test Execution activities as they apply to test cases identified by the Test Plan.
4.3.5 Test Results Analysis (KPA 5)
The evaluation of test results and comparison of these results against the acceptance criteria (set
down in the Test Plan) will be used for Test Results Analysis.
4.3.6 Management Reporting (KPA 6)
Describe the Management Reports that will be used to communicate the conclusions and
recommendations of the test team to management, the application development team, and to the
user community.
4.4 Acceptance Criteria

Describe the acceptance criteria that will be used to determine if the testing effort is complete.
Attachments
5. Associated Processes
5.1 Requirements Management
Identify what type of requirements management (tools) are being used for this project.
5.2 Configuration Management

Describe Configuration Management activities for this project.
5.3 Defect Management

Describe what defect tool will be used to track defects and also summarize graphical
reports when complete.
6. Deliverables
Describe the deliverables for this testing effort.
7. Assumptions
Add any assumptions for this testing effort.
8. Constraints
Add any constraints for this testing effort.
9. Dependencies
Add any external dependencies (outside of the test group) for this testing effort.
10. Test Schedule

Add or refer to the test schedule.
11. Risks and issues

Add any risks and issues for this testing effort.

Attachments
Attachment B Test Plan Template

[Company Logo]
[Project Name]
Name of Project/Product to be tested
[Test Type] Test Plan Template
Release [x]
Release of the Software to be tested
Date
Date the Test Strategy was initiated/changed

Attachments
Revision History
Revision
1.0
Date
MM/DD/YYYY
Author
Comments
Initial Revision
Distribution List
Name
Title
Approvals
Name
Title
Date

Attachments
1. Identifier
This is a comprehensive [Test Type] Test Plan for the [product name to be released]. The
identifier is (xxxxxxx).
2. Introduction
2.1 Purpose
Describe the purpose of the test plan.
2.2 Objectives
Explain the objectives of the testing effort. These objectives should have a measurable
output.
2.3 Test Scope

Please refer to the [project] requirement document and the associated test type that is or is
not in scope for this testing effort.
2.4 Test Environment Configuration

Add any test environment configurations.
2.5 References
2.5.1 Acronyms
The following acronyms are used during the testing effort:
2.5.2 Definitions
The following definitions are used during the testing effort:
2.5.3 Documents
The following documents are sources of information to the testing effort:

Attachments
3. Staffing and training

3.1 Staffing
Add individuals who will be supporting this test planning effort.
3.2 Training
Add specific training requests in order to rollout this test planning effort.
4. [Test type] testing

4.1 Test Methods
Describe the testing methods to be used for the [project] (for example: automated and
manual testing methods).
10

Attachments
Black Box vs. White Box Testing.
4.2 Entrance Criteria

Describe any items that need to be addressed before testing can begin:
Unit/Integration Testing Complete
Test Environment Setup
Defect Tracking Setup.
4.3 Exit Criteria

Describe the exit criteria that will be used to determine if the testing effort is complete.
4.4 Test Case Development
Describe how test cases are going to be developed.

Show an example of a Test Case.
Give an example of a Decision Tree Template.

11
Attachments
4.4.1 Test Case Naming Convention

Describe how the test cases will be named.
4.4.2 Test Data
Describe the test data that will be used.
4.5 Peer Reviews

Discuss how the test cases will be reviewed by other testers, the client or management.
4.6 Test Environment

Describe what the test environment will be. May include a table showing the different
configurations.
4.7 Security Issues

Describe any security issues that may be encountered during this testing effort.
4.8 Communications
Describe any type of data communications between the hardware and software. For
example: using a standard IP Network.
4.9 Restoration Procedures

Discuss failure and recovery of systems.
4.10 Create Data States

Discuss how to recover data at any point in time during a test run.
5.Test execution
Describe Test Execution activities as they apply to test cases and /or test scripts.
6. Associated Processes
6.1 Requirements Management
Identify what type of requirements management (tools) are being used for this project.
12

Attachments
6.2 Configuration Management

Describe Configuration Management activities for this project.
6.3 Defect Management

Describe what defect tool will be used to track defects and also summarize graphical
reports when complete.
7. Suspension and resumption criteria

Describe any type of suspension that may occur during text execution and the resumption
criteria needed to re-execute tests.
8. Deliverables
Describe the deliverables for this testing effort.
9. Assumptions, Constraints And Dependencies

9.1 Assumptions
Add any assumptions for this testing effort:
9.2 Constraints
Add any constraints for this testing effort.
9.3 Dependencies
Add any external dependencies (outside of the test group) for this testing effort.
10. Test Schedule

Add or refer to the test schedule.
11. Risks and issues

Add any risks and issues for this testing effort.

13
Attachments
Attachment C Test Environment Requireme n ts

[Company Logo]
[Project Name]
Name of Project/Product to be tested
Test Environment Requirements
Release [x]
Release of the Software to be tested
Date
Date the Test Environment Preparation document was initiated/changed

15
Attachments
Revision History
Revision
1.0
Date
MM/DD/YYYY
Author
Comments
Initial Revision
Distribution List
Name
16
Title

Attachments
1.
Application Recommended Requirements
Document the applications environmental requirements. A table format could be used to

specify the requirement for each type of environment component such as operating
system, CPU, Video or just state in a paragraph or list.
Server Requirements
Quantity
2.
Hardware Configuration
Document each servers hardware requirements. A table format could be used to specify
the requirement for each type of server component such as operating system, CPU, Video
or just state in a paragraph or list.

17
Attachments
Disk Space/Storage
Software
Communications
3. Client Workstation Requirements
Quantity
Consider the expected number of users.
Hardware Configuration
18

Attachments
Software
Communications
4. Peripherals
5. Physical Access
6. Naming Conventions
7. Overall Process
Procedures to be defined and documented:
<procedure 1>
<procedure 2>
8. Security Requirements
9. Access and Privileges
10. Audit and Traceability
Software requirements related to providing an audit trail of data activity (adds, updates)
including accessibility requirements (online, nightly report) as well as the necessary data
and supporting information (user identification, date and time).
11. Data Synchronization and Reconciliation

Data sync and reconcile - Requirements related to periodic synchronization or
reconciliation of data with external applications and databases. This should include a
detailed description of all data transformations, validation, error handling as well as data
inputs and outputs.

19
Attachments
12. Data Archival and Retrieval

Software requirements related to periodic data archival and retrieval processes from the
primary storage to an alternate medium. This should include a description of the archive
(or purge) criteria, the retrieval process including initiation mechanism as well as the data
elements affected.
13. Backup
Permissions
Identify who has backup permissions.
Backup Method(s)
Determine appropriate backup method(s).
Normal:
backup of the entire system, to include all selected drives, directories and files,
regardless of whether or not they have been changed.
does not require a search for files to restore.
good for restoring an entire system since all of the most current information is
located on the backup.
can take longer than other types of backups since more data is being backed up.
Incremental:
backup of only the files that have been created or changed since the last Normal
or Incremental backup.
much less data storage space required since it only includes that data that has
changed.
usually takes less time to complete than normal backups.
files may be more difficult to find since they may be spread across multiple
backup media.
a full system restoration may be time consuming since the data will probably
reside on more than on backup media.
Differential:
backup of all files that have been created or changed since the last Normal
backup.
files may be easier to find.
20

Attachments
a full system restoration only requires access to two media: the latest normal
backup and the latest differential backup.
usually requires less time to complete than a normal backup.
time to create a differential backup gets greater each day since it includes all
changes since the last Normal backup; consequently, some redundancy exists
since the same data may be backed up each day, regardless of whether it had
changed.
Copy:
backups of all selected drives, directories, and files.
does not affect subsequent incremental or differential backups.
Daily:
backup of all files with todays date (created or changed today) and does not
affect the files backup status.
Media Rotation Scheme

Determine media rotation scheme -- how long can you use the media before it is worn
and it needs replacing?
Necessary to follow a strict schedule, else, the backup efforts could be useless.
Potential schemes:
Son:
Normal backup every day using the same media. Can only restore to the latest
backup.
Father/Son:
Combination of normal and incremental backups for a two-week schedule.
Incremental and differential backups are done for every day of the work week except
the last day. On the last day, a normal backup is done. The media used for the
incremental and differential backups are re-used each week, and the media used for
the normal backups are re-used every other week.
Grandfather:
Incremental backups done for every day of the work week except the last day. On the
last day, a normal backup is done; on the last day of the month, a normal backup is
also done, but these media are kept offsite. The incremental backup media is re-used
each week. The normal backups are not re-used until the following year.
Validation Process(es)
Identify and document processes for validating that the backup(s) was successful.
21
Attachments
Backup Chart
Create a backup chart including the date, time and the contents of each media.
Backup Location
Determine location of backup media.
consider offsite storage in case of fire and natural disasters.

ensure that backups are stored away from magnetic fields and electronic devices
such as video monitors, analog telephones.
Remote Backup Administration

Identify processes and requirements for remote backup administration.
Hardware Protection
Determine hardware protection requirements:
consider uninterruptible power supplies (UPS) to manage power failures,

fluctuations, surges, as well as surge protectors on servers and workstations.
security: password protection and network rights.
virus detection and protection program(s) or both.
Automation
Determine areas for automating backup processes; document and implement as
appropriate.
14. Restoration Processes

Permissions
Identify who has restoration permissions.
22

Attachments
Restoration Procedures
Identify and Document Restoration Procedures:
restoring data to its original location.

restoring data to a location other than where it was backed up from.
aborting a restoration.
restoration of selected files to include or exclude.
restoring compressed files.
restoring file permissions.
remote restorations.
restoring registry information.
Identify and document how the restored data is validated.
Archiving
Permissions
Identify who has archiving permissions.
Archiving Procedures
Identify and document data archiving procedures
Identify and document how the archives are validated.

23
Attachments
Attachment D Sample Test Case

QualityPoint Test Case Header
Example
Category
Date:
Tester:
Test Case ID:
Requirement#:
Requirement Document:
Function Area:
Interface(s):
Reports/Screens:
Test Case Dependencies:
Environment:
Revision Date:
Peer Review Date:
Regression Candidate?

Description/Location
11-Nov-99
Jake Swift
Add U.S. - 1.1.1.1
Add U.s. - 1.1.1.2
Employee Processing - Requirement No. 1
Payroll Business Requirements
Add Employees
N/A
Add Employee Screen
N/A
Win95, Visio, Project 98, Office97 SR2, IE 4.0
12-Nov-99
12-Nov-99
Yes
25
Attachments
QualityPoint
Test Case Body
Example 1
Step #
1
2
3
4
5
6
7
8
9
10
11
12
26
Steps to Perform
Data
Verify that the basic company
employee data is loaded in the
list view on the main payroll
screen
Select the Employee pull-down
Menu
Select Add from the pull-down
menu
Enter an name in the name Doe, John
field
Enter salary choice
1000
Select the "none" radio button
None
for the insurance choice
Select "Admin" for the Admin
Department Choice
Select "None" for the Benefits
None
choice
Select Okay to Add the
Employee
Close the payroll application
Reopen the payroll application
screen
Expected Results
Comments
Menu appears
Data field appears
Name entered and accepted
Salary entered and accepted
Radio button works
appropriately
Admin accepted
None accepted
All data has been accepted
Closed successfully
Opened successfully
Data loaded successfully

Attachments
13
14
15
16
17
18
19
20
21
Verify John Doe is listed as an

employee
Select Doe, John
Select the "View" pull-down
menu.
Select "detail" form the pulldown menu
Verify John Doe's salary
Verify John Doe's insurance
choice
Verify John Doe's department
choice
Verify John Doe's benefits
choice
Return to main payroll screen

John Doe is in the data field

for name
John Doe's salary should be

"1000"
John Doe's insurance choice
should be "None"
John Doe's department choice
should be "admin"
John Doe's benefit's choice is
"none"
Payroll Screen
27
Attachments
QualityPoint
Test Case Body
Example 2
Step #
1
2
3
4
5
6
7
8
9
10
11
12
28
Steps to Perform
Data
screen
Select the Employee pull-down
Menu
Select Add from the pull-down
menu
Enter an name in the name Doe, Jane
field
Enter salary choice
1000
Select the "none" radio button
None
for the insurance choice
Select "Finance" for the Finance
Department Choice
Select "None" for the Benefits
None
choice
Select Okay to Add the
Employee
Close the payroll application
Reopen the payroll application
screen
Expected Results
Comments
Menu appears
data field appears
Name entered and accepted
Salary entered and accepted
Radio button works
appropriately
Finance accepted
None accepted
All data has been accepted
Closed successfully
Opened successfully
Data loaded successfully

Attachments
13
14
15
16
17
18
19
20
21
Verify Jane Doe is listed as an

employee
Select Doe, Jane
Select the "View" pull-down
menu.
Select "detail" form the pulldown menu
Verify Jane Doe's salary
Verify Jane Doe's insurance
choice
Verify Jane Doe's department
choice
Verify Jane Doe's benefits
choice
Return to main payroll screen

Jane Doe is in the data field

for name
Jane Doe's salary should be

"1000"
Jane Doe's insurance choice
should be "None"
Jane Doe's department choice
should be "finance"
Jane Doe's benefit's choice is
"none"
Payroll Screen
29
Attachments
Attachment E Sample Application Readines s

Report
APPLICATION READINESS REPORT
ABC HARDWARE
PAYROLL 1.0 PROJECT
Chapter 1 - Executive Summary

This report will summarize the testing activities that have taken place
over the last 13 months on the Payroll Project. The basis for our risk
statement can be substantiated by the data contained within this report
and its supporting documents.
Based on the Acceptance Criteria, the Payroll application should NOT
be released into production at this time. We are further estimating that
the release date will be pushed back a minimum of two months.
The primary reason for this decision is the lack of test coverage. Only
one of the six business and technical requirements has been fully
tested. Current testing is on budget and needs to continue through the
remaining business and technical requirements. Once this is
completed, we will have a full understanding as to whether or not the
application meets the criteria for deployment to the production
environment.
Chapter 2 - Test Coverage Measurement

This measurement shows how many of the business and technical
requirements that were represented in the test plan have actually been
tested to this date.
There were a total of six business and technical requirements
identified. Of those six, only one has been fully tested. This represents
only a 16.6% test coverage measurement.

31
Attachments
The following chart graphically shows this information:

Tested?
Test Requirements
Yes
No
Business
Employee Processing
Payroll Calculation
Payroll Disbursement
Payroll Reporting
Technical
Graphical User Interface
System Performance
Total Test Coverage
16.6%
83.4%
One of the acceptance criteria for the test effort was 100% Business
and Technical Requirement Coverage. As you can tell from the chart,
only 16.6% of the business and technical requirements were tested;
therefore, an acceptance criterion was not met in this area.
Chapter 3 - Error Discovery Rate Measurement

The error discovery number shows how many detected defects there
are for the application at a given point in time. This discovery rate
trend should be on a downward slope as the release of the software
approaches. Our trend demonstrated that the application was stable
because the number of defects detected during testing did decrease
each month.
32

Attachments
The following chart depicts the number of detected defects over the life of the
Payroll project.
# o f D e te c te d D e fe c ts
80
60
# o f D e te c te d
D e fe c ts
40
20
0
1
11
13
Chapter 4 - Testing Activity Measurement

There were a total of 231 test cases for this effort. The test team was able to successfully
execute all 231 test cases. Therefore, our total activity was 100%.
This number represents an improvement over the last test effort where we were only able
to complete 78% of the test cases.
Chapter 5 - Success Status Measurement

Out of the 231 test cases, 216 were successful. That represents a 93.5% success ratio.
Therefore, we can be certain that the results we analyze for Employee Processing are
comprehensive enough to make an accurate business decision regarding the readiness of
that module.
Chapter 6 - Specific Risk Measurement

The following chart will show what specific areas of the application will be at risk if the
software is released without any further development or testing.

33
Attachments
Open Defects
Test Requirements
Severity
Severity
Severity
Business
Employee Processing
Payroll Calculation
Payroll Disbursement
Payroll Reporting
Technical
Graphical User Interface
System Performance
As the chart shows, there are no open defects for any business/technical requirement
except for Employee Processing. But, this is because the test team has not begun to
test the other areas of the application yet. Therefore, there is a huge risk in these areas
of the Payroll application.
Chapter 7 - Other Test Measurements

No other test measurements are being utilized at this time.
34

Attachments
Chapter 8 - Project Summary

Effort
Testing effort to date totals 90 days.
Outstanding Issues
Business and technical requirements 2, 3, 4, 5 and 6 still need to be tested.
Risks
There is a major risk in releasing the Payroll application in its current state. The risk
involves the unknowns associated, having not tested over 83% of the application in this
effort. The test team cannot reliably predict the problems we could face in the following
business areas: payroll calculation, payroll disbursement, payroll reporting, the
applications graphical user interface and system performance.
Chapter 9 Conclusion
Overall, this test effort has been a very positive experience. For the first time in our
companys history, we have an objective way of determining whether an application is
ready to be released into production. We have also established measures to better control
future testing projects and their associated costs.
All of the hard work and effort has demonstrated that the Payroll application is not ready
to be released into production.
The main reason for this recommendation is that only 16.6% of the application has been
tested to this date. Our further recommendation is to push the release back a minimum of
two months. This will allow enough time to finish the necessary development work and
to test the remaining modules of the application thoroughly.

35
Attachments
Attachment F Sample Test Summary Repor t

[Company Logo]
[Project Name]
[Test Type] Test Summary Report
Document Identifier: [Project Initials] TSR-01
Release [x]
Date

37
Attachments
Chapter 1 - Purpose
Explain the purpose of the Test Summary Report.
Explain the following:
applications tested
test type
software version
dates of test runs
environment.
Chapter 3 - Items Tested

Functions Tested
Add the functions that were tested
Functions Not Tested

Add the functions that were not tested.
Chapter 4 - Problems Reported

List any problems and their associated workarounds within this testing effort.
Chapter 5 - Summary of Results

Summary of test effort
Chapter 6 - Risks
List any risks that need to be addressed before exiting this test type.
Overall Evaluation
This evaluation is based upon the entrance/exit criteria of the associated test plan.

39
Attachments
Attachment G Test Planning KPA

Client Logo Goes
Here
Test Planning
KPA
Business and
Technical
Requirements
Test Strategy
Business and
Technical
Requirements
Refinement
Business and Technical

Requirements Review
1.12
No
Yes
Define/Refine Test
Objectives
1.13
Present Test
Objectives &
Scope
1.15
Define/Refine Test
Scope
1.14
Yes
Test Objectives
& Scope Approval
1.16
No
No
Create Initial Test

Plan
1.17
Test Plan
Walkthrough
1.18
Test Plan
Refinement
1.19
Test Plan
Approval
1.20
Yes
Approved Test Plan
Test Plan
Publication
1.21
Final Test Plan

Walkthrough
1.22
Test Plan Delivery

1.23
End Process
Prepared by
41
Attachments
Attachment H Test Case Development KPA

Test Case Development
KPA
Test Plan
Standards
Business
Process
Documentation
Test Data
Sources
Requirements
in Scope
Create
Decision Tree
Present Deci
Refine
Decision Tree
Decision
Tree
Approval
No
Yes
Prioritize Test
Cases
Document Test
Procedures
Document Test
Data
Requirements
Manual Test
Cases
Identify
Candidates for
Automation
No
Automate Test
Cases
Automated
Test Cases
Yes
Automation
Approval
Test Case
Delivery
End Process
Prepared by

43
Attachments
Attachment I Test Environment Preparation KPA

Test Environment
Preparation
KPA
Client Logo Goes

Here
Technical
Environment
Description
(Network)
Define/Refine
Technical
Environment
3.1
Technical
Environment
Description
(Workstation)
No
Define/Refine Test
Data Requirements
3.2
Base/
Master Data
Transaction
Test Data
Data Correct &

Complete?
3.3
Yes
Identify Restore
Points
3.4
Perform Data
Walkthrough
3.5
Data Flow Diagram
Define/Refine
Restoration
Procedures
3.6
DataBase
Restore
Procedures
3.7
Network
Restore
Procedures
3.8
Establish Test
Environment
3.10
Execute
Restoration
Procedures
3.11
Correct &
Complete
Environment
3.12
Worstation
Restore
Procedures
3.9
No
Yes
Verify Test
Environment
3.13
Environment
Walkthrough
3.14
Test Team Signoff

3.15
End Process
Prepared by

45
Attachments
Attachment J Test Execution KPA

Test Execution
KPA
Client Logo Goes

Here
Test Execution
Schedule
Test Plan
Information
Defined Test
Cases
Verified Test
Environment
4.1
Yes
Execute Test(s)
4.2
Logged Test
Results
No
Create Backup and
Identify Data State
4.9
Initial Results
Review
4.3
Logged
Discrepancies
Suspend Test
Execution?
4.4
Suspend the Test

4.5
Yes
Make a Data
State Backup?
4.8
Yes
Continue?
4.6
No
More Test
Cases?
4.7
No
End Process
No
Prepared by

47
Attachments
Attachment K Test Results Analysis KPA

Test Results Analysis
KPA
Client Logo Goes

Here
Logged Test
Results
Evaluate Results
5.1
Valid Result?
5.2
Yes
No
Test Case
Defect?
5.3
Yes
Refine/Define Test
Case Development
Procedure
5.4
No
Test Context
Defect?
5.5
Yes
Refine/Define Test
Execution
Procedure
5.6
No
Software Defect
5.7
Yes
Log to Defect
Tracking
5.8
Collect Metrics
5.9
End Process
Prepared by

49
Attachments
Attachment L Management Reporting KPA

Management
Reporting
KPA
Client Logo Goes

Here
Test Execution
Metrics
Map Software
Defects to
Business and
Technical
Requirements
6.1
Compare to Acceptance
Criteria
6.2
Prepare
Measurement
Reports
6.3
Determine Current
Status of existing
Business Issues
6.4
Create Application
Readiness Report
6.5
End Process
Prepared by

51
NORTH AMERICA
EUROPE
INTERNATIONAL
Compuware Corporation
Corporate Offices
31440 Northwestern Highway
Farmington Hills, MI 48334-2564
USA
Tel (248) 737-7300 (800) 521-9353
Fax (248) 737-7108
Compuware Europe BV
Compuware Corporation
International Operations
31440 Northwestern Highway
Farmington Hills, MI 48334-2564
USA
Tel +1 (248) 737-7639
Fax +1 (248) 737-7623
Hoogoorddreef 5
1101 BA Amsterdam Z-O
The Netherlands
Tel +31 20 3116222
Fax +31 20 3116200
www.compuware.com
Compuware, Abend-AID, BoundsChecker, DBA-XPERT for DB2, DBPartner, DevPartner, EcoSYSTEMS, File-AID, NuMega, QABatch, QACenter, QADirector,
QAHiperstation, QALoad, QARun, QASolutions, TrackRecord, WebCheck and XPEDITER are trademarks or registered trademarks of Compuware Corporation.
All other company or product names are trademarks of their respective owners.
435 4/00
2000 Compuware Corporation

Compuware Best Practices For Testing

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Compuware Best Practices For Testing

Uploaded by

Copyright:

Available Formats

Compuware Best Practices for Testing