E-book

Ten Steps To Effective

Mainframe Testing
It’s time to see if your
testing can be better
It’s time to think about changing the way you develop and test code for your
mainframe. It needs to be modernized and future-ready to ensure that your
mainframe systems deliver the quality and speed demanded by the marketplace
and by distributed technologies.

A recent Forrester Consulting study found that there’s still a way to go, stating,
“the lack of modern mainframe development and testing tools hinders development
teams’ ability to improve software quality and slows down ramp-up times,” which
leads to reduced quality and places the enterprise at risk. Forrester identifies several
challenges that must be faced, primary among them being detecting performance
issues and automated testing.

The Forrester discoveries reflect a common industry reality: modernization tools will
improve the quality of applications by a significant margin. Their report shows that
“nearly two thirds (62%) of developers said modern tools will increase the quality of
their organizations’ applications by 23%.”

The bottom line with this analysis, as well as our own, is that automated testing tools
are necessary and so too is improvement to the testing process itself, which must
enable testing earlier and more often.

02
Ten steps to
testing effectiveness
We have identified ten steps that will make testing of mainframe code much more
accurate, while being faster, more profitable, and far more practical, and this eBook
is your guide. So ask yourself, what if you could:

+ Be sure that all your code changes have been tested before moving to the next level

+ Ensure that fewer defects travel to the test environment

+ Run tests in minutes, rather than hours or days

+ Know exactly how many test cases will be needed

+ Automate testing so it is neither optional nor a nuisance

+ Tie production failures back to your changes so testing can improve

Our proposed shift in the way you can approach testing is summarized as “Analysis,
Data, and Testing.” Here are the ten steps which will put you and your team on a path
for better testing and better development outcomes.

03
Table Of Contents
05 Determine Your Current and Desired State (Analysis)

07 Modernize Your Mainframe Development

Environment (Analysis)

08 Map Your Code and Data Structures (Data)

09 Verify Testing Effectiveness (Testing, Data)

11 Create a Test Data Strategy (Testing)

14 Identify Changed Code (Testing)

16 Understand Testing Complexity (Testing)

17 Adopt Automated Unit Testing (Testing)

18 Adopt Automated System Testing (Testing)

19 Practice Continuous Improvement (Analysis)

20 What To Look For In Testing Solutions

for Mainframe Application Development
04
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 1:

Determine your current desired state

All planning and change management processes require an understanding of where you want to go,
especially in comparison with where you are at today. This becomes your plan and your roadmap.
Without it, every project runs the risk of going off track and having to start over at great expense.
The desired state (usually maturity and elite status) becomes the benchmark against which you can
measure your success when you arrive at Step 10.

As we described in our previous eBook, DORA 4 Metrics for Mainframe DevOps, organizations vary in
the pace of their work. Changes may be deployed extremely frequently, such as hourly, or on a longer
timescale, even annually, for items such as changes in an interest rate. In line with the deployment
frequency metric, organizations themselves are described in terms of maturity as low, medium, high,
and elite.

+ A low performer deploys once a month or once every six months, which is quite common in the
mainframe world.

+ A medium maturity performer deploys between once a week and once a month.

+ High performers deploy between once a day and once a week.

+ Elite status is given to organizations that deploy multiple times a day. Typically, these are
companies like Google, Facebook, and Netflix. They expect their teams to be able to push code
into production on day one. In terms of mean time to recover, an elite company can fix a problem
in less than one hour from the time it’s detected until it’s resolved.

05
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

Self-assessment
Tools that can help
+ It starts with a self-assessment that may involve a process analysis
to take a snapshot of how things are currently done: The Escaped Abends Ratio built into BMC
Compuware zAdviser assesses abilities, generates
+ Define your current testing processes reports, and establishes benchmarks that will be
useful in tracking your progress. It identifies abends in
+ Identify your technology stack production helping you establish a goal of decreasing
them over time.
+ Define your desired testing methodology and frameworks

+ Define your goals and success criteria

+ Identify your required capabilities such as the ability to obtain

DORA 4 metrics

Next, list your success indicators. Here are a few typical success indicators
(yours may be different):

+ Decrease your change failure rate – ability to add new test cases when
errors are found

+ Employ shift left – perform testing earlier in the process and increase
testing frequency, such as every sprint, or every night, or every compile

+ Automate testing so it isn’t optional

06
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 2:

Modernize your mainframe

development environment
The mainframe development environment refers to the tools at hand as well as the Actions that can help
culture of the workspace. For example, an organization that has a graphical experience
will be able to use tools like structure charting, with a graphical interface, as well as
Do an inventory:
DevOps tools and automation. Modern tooling allows developers to take advantage
of the best and most efficient techniques for development and testing.
+ How can you measure testing?
Modernizing also eliminates knowledge bottlenecks. For example, a senior developer
+ How can you do unit testing?
may have decades of experience and tricks available for every program that runs an
application or system. But if that person is not available – perhaps they are on vacation,
+ What automation tools do you have?
retired, taking time off, promoted elsewhere, or leaving for another company, then
the responsibility for troubleshooting problems falls to newer, less experienced
people, who might not even know which program to look at or pull down.
+ Do you have the tools you need?

This is obviously critical for testing because whoever is in the driver’s seat at any given Success indicators
moment, must understand what is being tested and how this impacts the overall
application, which itself might feed into Program B, which feeds into Program C. + Modern Developer Experience, IDE
Understanding the overall big picture, including the connections, when it comes time
to test, is critical. + Tooling that promotes agility

The modern development environment helps ensure that people have the right tests + Ability to scan for quality and measure testing
to test the data or the code. Having that in a modern interface, rather than a green
screen makes things much easier. + Modern source code management

07
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 3:

Map your code and data structures

Not mapping code and data structures leads to a simple problem: you can’t Success indicators
test if you don’t understand what your code is and what it does, and what
your data is and what it does. A surprising number of organizations do not
+ Gaining graphical, intuitive visibility into your codebase and data structure
understand all of their code or how it works. Nor do they fully understand
their data, in terms of how the data works, what data they’re calling, or how
+ Providing an inventory of your application listing programs, dependencies, data
that data is structured.

+ Being able to know exactly what, where and how to test

Mapping code and data structures means understanding the layout and
topology of your code. This means being able to quickly reference:

+ What to test Tools that can help

+ What is involved The BMC Compuware Topaz for Program Analysis Runtime
Visualizer provides a dynamic visualization of program and I/O calls
+ Programs and their call structure - is it online? What transactions does presented in the sequence in which they occur. There is no need to
it perform? How would you test it? Is it a DB2 program? IMS? Does it recompile your programs; simply just run them with Runtime Visualizer on.
have VSAM?

+ The structure of the data, such as Db2 tables and files

These type of questions boil down to the use of a program analysis tool,
which is an essential component of automating testing.

08
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6
STEP 4:
Verify testing effectiveness
Testing completeness using code coverage
Implement code coverage to understand how much of your code is being
executed. The benefit of increasing code coverage is that, whereas traditional
testing might leave large chunks of edited code unchecked – risking errors
being found in production – code coverage means getting a higher
percentage of the right code under control immediately.
How can you know whether or not you have tested the relevant pieces of
mainframe code? How do you know if your testing is effective or when you’re
done with testing? Code coverage delivers the metrics that ensure a test case
has covered the changed lines of code. The BMC Compuware Xpediter Code
Coverage product highlights the lines that were executed in green, and those
that were not executed show up in red. This means you can run a test case
through BMC Compuware Topaz for Total Test using code coverage and
generate a report that shows everything that was covered.
Code coverage is the game changer
Mainframe testing has traditionally followed a waterfall process, which is
essentially driven by its due date. Once the code is done, testing is assigned
to the last day or two before it goes out to production. At that point, people
throw everything at it – massive amounts of data – believing that more is
STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY
better. When the day to go to production arrives, testing is declared done.
Testing at the very end of the project means that errors are caught too late,
especially if the error requires going all the way back to the beginning, which
could be drastic. In earlier decades, this process relied on end users to report
bugs back to the developer, who would then fix them for the next release.
But the marketplace has no time or patience for such tradition any longer.
The market now demands testing within each two-week sprint, or even every
night, or after every change, which is impossible when a traditional test suite
takes hours to run, and must then be looked over. What needs to happen is a
shift left, in which testing is done earlier and throughout the development
process, using smaller amounts of data. Code coverage is the means to do
this effectively.
Traditional Model Shift Left Model
Testing during SDLC stages
Testing during SDLC stages
Requirement, Design, Development, Testing and Support Requirement, Design, Development, Testing and Support
09
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

Why is this smarter and more effective? Because you need the right balance between the
type of test (functional, regression, performance), and how often you will test (each compile,
each sprint before Prod). The amount of data you will need varies depending on what type of
testing you will be doing. You won’t need a million records to test a few lines of changed
code. But is it really possible to feel comfortable with just a small amount of test data?

Code coverage is the game changer. It allows you to say, “I ran it with 100 records, and I hit
everything.” By comparison, testing with a million records isn’t better just because it’s more.
In fact, it is more like testing blindly, since you can’t be sure that you have tested the changes.
Once developers see how code coverage can work, it becomes far easier to embrace.

Code coverage represents a shift left process that is essential for effective mainframe testing
overall. It’s one of the foundational pieces to getting testing done. It answers key questions
about knowing whether or not you have tested the relevant pieces of mainframe code, and
when you are done with testing.

Success indicators

+ Increased code coverage percentage

+ Increased quality and reduced change failure rate

+ Ability to reduce size of test data while increasing coverage

Even though you’ve hit every test case already,

hitting it 100 times isn’t going to be any better.
Code coverage, however, is a game changer.

10
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 5:

Create a test data strategy

As we have already established, working with large
sets of production data to run testing is cumbersome,
inefficient, and prone to further errors. To shift-left, it
is vital to have a strategy of first sub-setting from
production the types of data needed to satisfy the
changes being made to the application. This can then
Testing Strategies, Right Amount of Tests and Data
be privatized and sent to the test environment. It can
even be sub-set further to provide repeatable steps to Frequency of testing Amount of data needed
refresh a test environment. These tests can be
automated once they’re created.

The strategy comes from understanding the data,

creating a “gold copy” of test data, and then feeding it
Development Environment Test Environment Final Environment
into the various testing procedures. This is all part of
shifting left Without a strategy, the only thing that can Unit Test – only testing
specific code, right after
Integration and
Functional Tests
Performance and Acceptance
Tests – more data needed for
be done is to take a million+ sized file and run it against change, does it work? performance loads

the production environment and hope for the best.

Each type of testing has its own requirements for the amount of data needed.
This supports “Shift Left” testing to bring more of the testing to where errors are intorduced.
Because of old school waterfall tradition, a company’s
typical testing strategy often goes only to one level,
which is volume testing of millions or even billions of
lines of code. The new strategy requires working with
smaller subsets of data.

11
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

Actions that can help Minimal viable data: making smaller test data fles

Create a plan for your data:
+ Provide Minimum Viable Data (MVD) – this is because testing with
insufficient data or with too much data are both inefficient – not all
records are needed, just the right records
+ Procure production data on a regular basis and or as needed
+ Establish consistent standards for sub-setting and masking data
+ Stay compliant – Disguise Personally Identifiable Information (PII)
+ Develop the ability to subset and disguise data
MVD is the smallest set of data that is required to test the portions of code
where changes are being made. This is something that can only be known
once the data, its structures, and its relationships to other data, are fully
understood. This is especially important for bringing data down from the
gold copy. But this, too, is a process that developers don’t typically do, even
though it allows companies to move at a much faster speed.
You have two choices to produce smaller datasets:
1. Take your main driving dataset and cut it in half, check the code coverage,
then cut it in half again, check the coverage. Repeat until you get
something that won’t take long to run and still gets acceptable coverage.
2. Use BMC Compuware Test Data Optimization (TDO), a more scientific
method to get you to the minimal data necessary. TDO can help identify an
optimized subset of records that yield an acceptable execution percentage.

Test data optimization

Test data optimization means selecting the data that is needed to ensure the
highest level of code coverage. It is the exact number of records you need to
test your code.

12
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

Case study – A large North American bank: A client of ours told us that to test all
their capabilities, they would replicate their 15-to-20 million records in production
to multiple different databases at different levels, simply because they did not want to
go through the task of figuring out what data they needed at what area. Their testing
process took hours, days, and sometimes weeks. Imagine how much more efficient
their system would have been if, instead of 20 million records, they needed to identify
only the 10,000 records required. This is what optimization does. It makes things
simpler to run, simpler to understand, simpler to update, simpler to complete. It also
makes things faster, while reducing costly CPU usage.

Success indicators

+ Having “right sized” data for testing your code

+ Increasing testing velocity

+ Increasing testing frequency by improving testing efficiency

Tools that can help

TDO utilizes BMC Compuware Xpediter Code Coverage and

BMC Compuware File-AID to extract the minimum amount of
data necessary for the same coverage, significantly reducing
execution time.

13
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 6:

Identify changed code

This step focuses on the risks of not identifying change code. A hard fact of
this business is, “all of your changes will be tested eventually, either by you,
or by your end user.” This is the ultimate in shift left: shifting testing from the
end user back to your organization. At the very minimum, changes shouldn’t
leave the sprint if they haven’t been tested. It’s much more important to test
modules or changes that have just been made than to test a module that is
already many years old, not only at the beginning, but all the way through.
Code coverage metrics give you proof of execution, which is how you can
judge your test.
Identifying changed code can be done as simply as turning on an option in
your editor.
It can enter what you specify in columns 73-80 for each change you make.
Alternatively you can automatically identify changed code by using Compare
as part of the process after you do your compile. It automatically identifies
changed lines. The FLAG utility compares two versions of the program and
creates a new version that looks like the current version except that lines
inserted or modified since the baseline version are flagged. Flags consist of a
user-specified string of characters placed at a user-specified location on the
source line (within the range of columns 73-80).
FLAG modifies or creates a source listing (member in a DDIO file, source
database, or source shared directory) to show which of its source lines have
changed since a prior version of the program. Data definition source lines as
well as executable source lines are compared and flagged as appropriate.

Because everyone is always so busy, one of the chief goals of identifying

change is to provide focus. A code review keeps everyone focused on that
highest risk element: recently changed code.

14
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

All of your changes will be tested eventually,

either by you or by your end user.

Success indicators

+ Knowing if changed lines have been tested

+ Increased effective test cases

+ Reduced amount of data by eliminating redundant test cases

+ Knowing if new test cases are needed

+ Mapping code execution with solid metric

Tools that can help

The BMC Compuware Topaz Editor allows you to automatically

flag your changes as you make them. The BMC Compuware
Compare Utility can automatically flag your changes.

15
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 7:

Understand testing complexity

Testing can be complex when you consider all the paths the program can take, based on the input data.
It can be even more complex when you add in the error handling routines. The goal of testing is to test
all possible paths and conditions a program can take. Ideally, you strive to attain 100% coverage of your
program, meaning you have a test case for each possible scenario your program can handle. This is not
always easy with manual testing. With test automation, we can use tools to help identify what paths the
program can take, how many test cases are needed and the percentage of code that is covered by these
test cases.

The complexity of testing is made easier through the use of the McCabe Complexity metric, which
identifies the number of unique test cases, not for the whole program, but for modules or paragraphs. If
there are seven test cases needed to test it, it’s not too difficult to come up with seven test cases. But if
it’s 256, or 500 different test cases, then a different strategy is needed. The McCabe metric helps people
understand the complexity of the testing requirement, and might lead to a more significant refactoring.

Success indicators

Matching testing effort and test cases to code complexity

Tools that can help

The BMC Compuware Topaz for Program Analysis McCabe

Complexity Metric is available for each Perform Group (roughly
equivalent to a paragraph), and provides the number of unique test
cases you would need to test the code completely.

16
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 8:

Adopt automated unit testing

Unit testing, as discussed earlier, is the basis of shift left, and is as close to
the development of the code as possible. It tests just a part of the program,
as opposed to system integration testing, where all the pieces are put
together. It is the type of testing that happens earliest in the chain which
makes it important. When problems can be identified during the unit testing
phase, it makes it more feasible to fix them right then and there.
Automated testing is not optional, and manual testing
is not practical
Test driven development
Automated unit testing makes test driven development possible, which
involves test cases being written by developers before making changes to
code. It’s a proven pattern for development, in which tests are written that
will fail by default – because there have not yet been any changes to the
code, and then the code is written in a way that will make the tests pass.
Success indicators

+ Less time spent manually testing

Automated testing is vital to reaching elite maturity and DevOps. It is central
to achieving two-week sprints in place of traditional four-week sprints that
+ Developers get immediate feedback on their changes
include a week at the end for testing. Automated test cases that run on a
Nightly Batch cycle, for example, are far more likely to catch something than
+ Runs after every build process
the much riskier and time-consuming practice of manual testing.

Manual testing carries a further risk even when there is not necessarily a bug Tools that can help
within the code that has been updated. Manual testing adds a human factor,
BMC Compuware Topaz for Total Test enables developers and
and humans come with their own weaknesses such as fatigue, distraction,
testers at all skill levels to test programs and subprograms almost
and hunger. Things can get overlooked when testing is done manually, which
immediately after updating a piece of code to ensure unintended
adds greater risk and time to the project.
changes were not introduced into the codebase. It intelligently executes
only test cases related to changed programs in a CI/CD pipeline, and
By contrast, a codified, consistent, automated set of scripts runs at the same
ensures the consistency and accuracy of test data through an
time, it takes the same amount of time, and it hits all the pieces. It’s like the
integration with BMC Compuware Topaz for Enterprise Data.
expert’s perfect maxim: “all I can do is save you time and money.” Automated
testing does this for the organizations that deploy it.
17
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 9:

Adopt automated system testing

Although shift left involves moving much of the testing to much earlier in the Success indicators
software development lifecycle, there still remains a need to do system
testing at the end of the cycle. This is where the true functional regression,
+ Fewer defects in production
load performance and stress testing are assessed, and these need to be done
with the system as a whole. Whereas unit testing is done at the smallest
+ Testing moves up, sooner in the cycle
possible level of code and its changes, automated system testing tests the
application as a whole, and how new changes affect that system. So it must
exist within a production-like environment that has upstream and
downstream dependencies. It takes all the changes, sews them all together
Tools that can help
and says, “my unit tests were all really good, let’s see how we all did as a
team” by testing it against the system as it exists today. This helps see, for
example, how certain changes that might be fine unto themselves, might BMC Compuware Hiperstation for Automated Interactive
have broken dependencies downstream. Testing automates your testing, can be kicked off in a pipeline, and
will test your online applications. It integrates with BMC Compuware
Strobe for visibility into the actual performance of the application
Blending system testing and unit testing
being tested to ensure it meets specific criteria for production.
Hiperstation can produce the Pass/Fail results as well as other statistics
Developers need to unit test, taking small chunks of programming, and related to test execution in a user-friendly and consumable format.
testing them. Unit testing contributes to overall efficiency by helping ensure
the efficiency of Step 9, System testing. Topaz for Total Test provides the
ability to move quickly and automatically through unit test cases meaning
that by the time of the performance regression, big test – the things that
truly can’t be tested until everything else is all done – the upfront, small
errors, the logic errors, the coding errors, business logic errors, that are
shifted left and tested and debugged sooner, the less stress and scope is
required at the end.

18
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

STEP 10.

Practice continuous improvement

Continuous testing never has an endpoint. Everything that gets done will still
require a return to step 8, 7, maybe even step 6. As new changes are added,
there will be a continual need to improve, to find where there are choke
points, and to continue to up the percentage of code coverage, striving to hit
100%, but never quite getting there.
Test driven development may give way to behavior driven development,
which itself may give way to a newer framework or methodology, always
based on striving to get a little better. Tools also change, including those
developed by BMC Compuware. This includes new features and new
compatibilities – anything that will help customers improve their testing.

The move to Agile and DevOps has as a central area of focus, the idea of
continuous improvement through shift left and continuous testing, all of
which are intended to continuously improve a product during development,
but to also continuously improve an organization as it progresses through its
life. Continuous improvement is an ongoing process of tuning and tweaking
the system. In terms of testing, it’s about constantly refining the process to
improve code quality. It has no beginning and no end.

Specifically for mainframe testing, a continuous improvement approach

contributes to an organization’s ability to obtain DORA 4 metrics.

Success indicators

+ Continued measurable improvement

+ Testing closer to when code changes are made

+ Faster deliveries with increased quality to users

19
STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6 STEP 7 STEP 8 STEP 9 STEP 10 SUMMARY

What to look for in testing

solutions for mainframe
application development
To achieve the 10 steps outlined in this eBook, seek out automated testing solutions that are open and
that allow you to integrate with other tooling, with other capabilities, and with other processes. Take
advantage of innovation without worrying about vendor lock-in. Look for those things that allow you to
do a lot more plug and play. As you research improved testing methods for the mainframe, try to use the
same tooling that is being used on your distributed platform.

In terms of efficiency and quality, we’re all looking to move faster, and get the product out to the market
more quickly. But it’s important to not lose any of the efficiencies that have already been attained. Make
sure to maintain the relationship between quality and efficiency on the one hand and moving faster and
getting the product out the door, on the other.

In addition to the quality of the product being developed, it is advisable to also ensure you seek out
something that enhances the developer experience or makes the developer experience a delight. This
is part of the DevOps mantra and helps ensure your team members feel good about the changes and
developments, since they are the ones, ultimately, who are making progress happen for the organization.

Want to build a pragmatic approach to adopting efficient testing practices for your mainframe
application development?  Schedule a DevOps Transition Workshop with our BMC DevOps architects,
and they will help you develop a pragmatic approach to adopting DevOps practices for the mainframe. 

20
About BMC
BMC works with 86% of the Forbes Global 50 and customers and partners around the world to create their future. With our history of innovation, industry-
leading automation, operations, and service management solutions, combined with unmatched flexibility, we help organizations free up time and space to
become an Autonomous Digital Enterprise that conquers the opportunities ahead.

BMC—Run and Reinvent www.bmc.com

BMC, the BMC logo, and BMC’s other product names are the exclusive properties of BMC Software, Inc. or its affiliates, are registered or pending registration with the U.S. Patent
and Trademark Office, and may be registered or pending registration in other countries. All other trademarks or registered trademarks are the property of their respective owners. * 537032*
© Copyright 2022 BMC Software, Inc.