Best Practices For Workload Automation Ebook PDF

Best Practices for
Workload Automation in
the Bimodal Era
sponsored by
Dan Sullivan
Best Practices for Workload Automation in the Bimodal Era

Chapter 1: Why Workload Automation? ......................................................................................................... 1

The Challenges ...................................................................................................................................................... 2
IT Resource Gap ............................................................................................................................................... 2

IT Skills Gap Is a Top Concern Among Executives ............................................................................ 2
Think Like A CIO on Workload Management: Centralize and Consolidate ................................. 7
Bimodal IT and the Need for Workload Automation ......................................................................... 10

Automation Risks and Service Disruptions ........................................................................................... 11
Reducing Risk the Manual Way: Checklists ...................................................................................... 12
Reducing Risk the Automated Way: Workload Automation ..................................................... 13
Summary ............................................................................................................................................................... 13
Chapter 2: The Anatomy of a Workload Automation Solution .......................................................... 14

A Brief History: From Batch Scheduling to Workload Automation ............................................ 14
In the Beginning: Batch Processing ...................................................................................................... 15
Business Changes Drive Changes in Automation ........................................................................... 16
Changes in Business Require Changes in IT ................................................................................ 16
The Limitations of Custom Scripts for Workload Management .......................................... 17
The Emerging Demand for Workload Automation ....................................................................... 19
Reducing the Need for Custom Scripts .......................................................................................... 19
Leverage Library of Code to Execute Parameterized Jobs .................................................... 20
Reduces the Need for Custom Error Handling and Alerting Logic .................................... 22
Advantages of Workload Automation ...................................................................................................... 25
Best Practices in Workload Automation ................................................................................................. 26
Summary ............................................................................................................................................................... 27
Chapter 3: The Financial Impact of Workload Automation ................................................................. 28

Advantages of Consolidation and Coordination .................................................................................. 28
Fewer Skills Needed to Support Multiple Tools ............................................................................. 29
Consolidated Reporting ............................................................................................................................. 31
The Cost of Fractured Workload Environments ........................................................................ 31
Overhead Introduced by Fractured Workload Environments ............................................ 32
Reduced Custom Scripting ....................................................................................................................... 32
iii i

Continually Evolving Requirements ............................................................................................... 32

The Cost of Ongoing Maintenance ................................................................................................... 33
Decommissioning Unnecessary Software ......................................................................................... 33
Managing Licenses for Software ....................................................................................................... 34
Patching and Maintaining Workload Software .......................................................................... 34
Training Staff on Software .................................................................................................................. 34
Faster Builds and Deployments .................................................................................................................. 35
Libraries of Tested Code ........................................................................................................................... 35
Workflow APIs .............................................................................................................................................. 36
Self-Documenting Operations ................................................................................................................ 36
Visual Scripting Tools ................................................................................................................................. 36
Resource Optimization and IT Burden Reduction .............................................................................. 37
Next Steps Checklist ......................................................................................................................................... 39
Summary ............................................................................................................................................................... 40
iiiiii ii

Copyright Statement
© 2016 Realtime Publishers. All rights reserved. This site contains materials that have
been created, developed, or commissioned by, and published with the permission of,
Realtime Publishers (the “Materials”) and this site and any such Materials are protected
by international copyright and trademark laws.
THE MATERIALS ARE PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE,
TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice
and do not represent a commitment on the part of Realtime Publishers its web site
sponsors. In no event shall Realtime Publishers or its web site sponsors be held liable for
technical or editorial errors or omissions contained in the Materials, including without
limitation, for any direct, indirect, incidental, special, exemplary or consequential
damages whatsoever resulting from the use of any information contained in the Materials.
The Materials (including but not limited to the text, images, audio, and/or video) may not
be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any
way, in whole or in part, except that one copy may be downloaded for your personal, non-
commercial use on a single computer. In connection with such use, you may not modify
or obscure any copyright or other proprietary notice.
The Materials may contain trademarks, services marks and logos that are the property of
third parties. You are not permitted to use these trademarks, services marks or logos
without prior written consent of such third parties.
Realtime Publishers and the Realtime Publishers logo are registered in the US Patent &
Trademark Office. All other product or service names are the property of their respective
owners.
If you have any questions about these terms, or if you would like information about
licensing materials from Realtime Publishers, please contact us via e-mail at
info@realtimepublishers.com.
iiiiiiiii iii

Chapter 1: Why Workload Automation?

The IT departments of today have changed substantially from those of the past decades.
Waterfall methodologies that were once the go-to software development methodology
have been supplanted by a variety of agile practices. Long-term planning and capital
expenditure planning are still around, but they are accompanied by discussions about
avoiding new hardware purchases and moving applications to the cloud. The concept of
monolithic applications is giving way to the more fluid and amorphous concept of services.
Services are automated functions designed for incorporation with other automated
functions in compositions that are defined in response to changing business needs. To put
it simply, businesses do not have time to wait for IT practices of the past.
The practices that are emerging in IT are creating new challenges for IT managers, systems
administrators, and application developers. IT professionals have a long history of adapting
to new challenges by creating new tools, trying new methodologies, and learning from
mistakes. The current dynamic environment is no different. Although every organization
will have its own particular challenges and responses to those challenges, common
patterns have emerged.
This chapter will focus on two key areas that are fundamental to understanding how
workload automation can help address the evolving IT environment:
• Challenges facing IT executives trying to manage and leverage this evolving

landscape
• The rise of the bimodal IT department as a response to the disparate needs of those
focused on operations and those focused on innovation
As any seasoned IT professional knows there are no silver bullets that will streamline IT
operations and put an end to over-budget and delayed projects. Applications will have
bugs. Hardware will fail. The goal of this discussion is to demonstrate how to use workload
automation to control and manage processes that are amenable to more structured
processes. As a result, IT professionals have more time to focus on innovation and address
the unexpected, but inevitable, system problems that arise in any complex IT environment.

111 1

The Challenges
IT managers and executives face two constant challenges:
• How to acquire and develop staff with the skills needed to operate a dynamic IT
department
• How to centralize and consolidate IT operations to maintain a cost-effective
operation that delivers on the needs of the business
At first glance, these may seem like unrelated problems, but in fact, they are more
intertwined than first appearance suggests. This interconnection will become clear as we
delve into the details of each. At the same time, it will start to become apparent how the
role of workload automation can help address both of these challenges.
IT Resource Gap
IT professionals with in-demand skills have email inboxes littered with messages from
recruiters. Subject lines with phrases like “Immediate Need,” “Phone Interview Hire,” “Top
Salary for Sys Admins with knowledge of x,” where x is any of a long list of in-demand skills,
such as virtualization, cloud administration, and information security. This practice of
borderline-spamming of IT professionals is the canary in the coal mine facing many IT
organizations: they do not have the IT skills they need and the employees with top-valued
skills are sought after by competitors, collaborators, and other businesses alike.
IT Skills Gap Is a Top Concern Among Executives

The IT skills gap is having a direct impact on organizations’ ability to deliver IT services.
Problems range from delayed delivery because of unfilled position, loss of revenue,
increased workload on existing staff, and adverse impacts on customer service.
A recent study by an online recruiting and job placement firm highlights the problem:
• 60% of survey respondents are concerned about the impact of delays in filling open
positions
• 25% have suffered a loss in revenue due to unfilled positions
• 41% of respondents report increased workloads are driving down morale
• Customer service is adversely impacted in 30% of cases
• 25% of respondents report an increase in mistakes and lower quality work product

222 2

These statistics speak volumes about the difficult situations facing organizations and
employees, but one additional statistic stands out: 22% of respondents report high
turnover because of the added demands on staff due to unfilled vacancies.
Thus, the staff members whom organizations depend on to meet business needs and make
up for unfilled positions are increasingly at risk of leaving due to frustration, low morale,
and excessive workloads (see Figure 1.1). Given the fact that so many organizations are
facing similar skill gaps in their IT operations, those with skills in high demand will find
they likely have a range of options when looking for a new position. What was a problem
filling new positions is now compounded by the fact that businesses need to retain the
skilled staff they have in the face of stiff competition for those skills.

Figure 1.1: Fewer employees have the skills in demand to support IT operations,
creating a skills gap.
Evolving Technology Drives Demands for New Skills

Part of the problem is the disparity between the drop in the cost of technology and an
insufficient pool of skilled professionals who can leverage that technology (see Figure 1.2).
As the cost of technologies, such as servers and applications, drops, more businesses can
purchase these technologies. These purchases, in turn, drive up the demand for people with
the skills to manage them.
333 3

Figure 1.2: An IT resources gap is growing as hardware, software, and services are
increasingly available at relatively low costs while the growth in skills needed to
fully utilize those tools lags behind (Source: http://info.advsyscon.com/it-
automation-blog/addressing-the-growing-it-resources-gap).
Although it might take a team of well-trained software developers with years of experience
to make a useful tool, such as a business intelligence reporting tool, that same tool can be
applied by thousands of developers in businesses around the globe. Those developers must
be familiar with business intelligence reporting, capable of working with business
professionals to discover requirements, and have an understanding of state-of-the-art
methodologies, such as Agile, to deliver and sustain a finished product. Those skills are in
demand.
Leveraging Existing Staff with Automation
As the discussion so far has illustrated, executives and managers are facing difficult
decisions about how to allocate their staff resources to projects and operations. If they take
experienced IT professionals away from operational duties to develop new applications
and services, they risk disrupting operations by pulling experienced administrators from
crucial day-to-day operational tasks.
One way to deal with this challenge is to automate administrative tasks, such as running
regularly scheduled jobs. This kind of workload automation is commonplace in business.
Systems are backed up, data is exported from transaction processing systems and loaded
into data warehouses, inventory is analyzed, and new orders placed. These are the kinds of
operations that are well-structured business processes and amenable to automation.

444 4

They share common characteristics:
• Have a well-defined set of inputs and outputs

• Run on regular schedules or in response to specific events
• Require minimal manual intervention
The inputs and outputs of jobs described here are well defined. An application may
generate a file of order details that is read and processed by an order fulfillment process.
The order fulfillment process, in turn, makes use of shipping services to schedule pickup
and delivery of orders, and then writes additional details about the order fulfillment to a
customer support database that allows customers to check on the status of their orders.
This type of series of processes is readily scripted in a variety of languages and tools.
Windows administrators might choose one tool for scripting processes, while Linux
administrators might choose another, and a developer familiar with a particular
programming language might opt for her preferred language. You can imagine how this
variation turns out over the long term. There is a polyglot environment of scripts written
in:
• PowerShell in Windows environments

• Bash in Linux environments
• Perl, Python, and other “favorite” scripting languages of administrators
• Built-in application-specific schedulers
This siloed approach is understandable. Developers are under tight deadlines, and they are
working on multiple projects while trying to support ongoing operations. If they can
automate a workload by writing a script in Perl, then Perl is the tool of choice. For those
developers more familiar with .NET tools, PowerShell is the go-to solution. Their managers
are not going to fault them for getting the job done and moving on to the next top priority.
And so the cycle continues.
The problem with this all-too-common scenario is that the short-term technology needs are
emphasized over the long-term needs of the organization. Solving problems in isolation
leads to a complex unmanageable environment consisting of multiple workload
management tools, scripts, and systems (see Figure 1.3). Some are homegrown solutions
written in the favorite language of the developer who happened to be tasked with getting a
process up and running. Some of the tools are third-party applications purchased by a
department to put an end to the disparate collection of Perl, Python, and PowerShell scripts
that have accumulated in the department’s workloads. At some point, a department
manager might lay down the law and declare all processes be developed using a standard
tool—standard for that department anyway.
555 5

Figure 1.3: Silos of automation tools emerge when individual developers and
departments try to separately address the common problem of workload
automation.
Throwing Oil on the Fire: Multiple Automation Tools

At first glance, it might appear that automating, using whatever tools are available, is a
good thing. And it is, to a point. If your house is burning, there is not much incentive to ask
from where the flame-dousing water came. Short-term requirements to get a job done have
to be met. After the immediate need has been met, we sometimes find that we have made
choices that lead to unintended consequences. In the case of enterprise automation, we find
that we have silos of automation that actually exacerbate, rather than ameliorate, the
problem we were trying to solve.
The reason many organizations turn to automation tools is to find some relief from the
demands on developers and administrators to ensure existing processes and procedures
function as expected while adapting to changing business conditions and rolling out
innovative services to customers and users. Many organizations have trouble implementing
this setup because of the skills gap—there just are not enough people with the right skills
to meet demands. By automating, businesses can, in theory, free up the time of their best
developers to focus on more strategic needs. However, by letting individual developers, or
even departments, choose automaton tools, organizations are left supporting a mix of
different technologies that often don’t work well together or have a lot of overlap.
We can all imagine where this scenario leads. Businesses are running day-to-day
operations using scripts written in a variety of languages and tools that are understood by
a potentially small subset of their IT staff (see Figure 1.4). This setup creates, in effect,
another skills gap. Workload automation was supposed to solve the problem of lacking
skills but instead has effectively institutionalized the problem.

666 6

The skills gap problem can be addressed, to some degree, by workload automation, but not
if the decision-making is made at the level of individual developers or even at the
departmental level. A centralized approach that takes into account broad business
requirements and architecture is essential to avoiding, or at least mitigating, the challenges
described to this point.

Figure 1.4: Without a centralized workload strategy, organizations struggle to keep
disparate solutions and tools operating together.
Think Like A CIO on Workload Management: Centralize and

Consolidate
IT professionals approach the IT skills gap with different tactics. Some will try to hire to fill
the gaps while others will emphasize additional training for existing staff. Still others will
work out formal or even informal arrangements with other departments to share resources
and cross-train staff. All of these methods can help provide a buffer when someone with a
few key skills is needed in one area in a way that leaves another area without support. Here
is where workload automation can be of use.

777 7

As noted earlier, workload automation can help compensate for the skills gap. Tasks that
must be performed repeatedly can be automated in a variety of ways. The flexibility of
choosing tools for workload automation is both a benefit and a detriment. It is a benefit
because individuals and teams can rapidly deploy workload automation scripts using tools
they understand and can readily support. The downside emerges as this practice continues
with different individuals or teams choosing different tools.
From the perspective of a CIO, the organization has created a system that now depends on
a variety of tools requiring different skills sets. The choices each team made were no doubt
rational from their perspectives. Unfortunately, those perspectives are often limited to
their own project requirements, budget constraints, and scheduling deadlines. Gartner
research found that this pattern is so prevalent that it is not uncommon for organizations
to support three to eight different workload automation tools. If you are fortunate, your
organization supports only three workload automation tools, but some organizations are
supporting as many as eight workload automation tools. This conclusion seems out of line
with effective IT practices.
CIOs tend to take a broader view of the organization and the portfolio of services they have
to support. Thinking like a CIO with that broader perspective, you could see the advantage
of a consolidated approach to automation. Ideally, the workload automation solution will
serve the needs of multiple projects, operations, and departments. The benefits are clear:
• Fewer distinct tools to support

• Potentially lower costs with enterprise licensing instead of multiple, individual
department licenses
• Ability to leverage skills across teams and departments
• Ability to develop best practices and repositories of patterns and reusable code
• Faster time to deployment as skill sets improve
• Easier integration with other tools, such as log management and alerting systems
Perhaps the most important benefit is that a unified job scheduling and workload
automation tool provides a single point of control, as Figure 1.5 illustrates.
888 8

Scripts
Cloud
ERP
Orchestra2on
Centralized
Infrastructure Workload Managed File
Provisioning Automa2on Transfer

Security/ Data
Compliance Warehousing
Repor2ng
Figure 1.5: Centralized workload automation enables a single point of control and
problem resolution.
The ability to support a wide range of workload scenarios is especially important today
because IT organizations are shifting their structures to support both day-to-day
operations and more Agile development and deployment of new services. An organization’s
workload automation solution should support both constituents of this bimodal structure.
999 9

Bimodal IT and the Need for Workload Automation

Industry analyst firm Gartner has identified a new pattern of IT organization called
bimodal IT. As the name implies, it consists of two constituent parts. Mode 1 fills the role of
traditional IT with an emphasis on maintaining operations and ensuring day-to-day
activities carry on. Some of the more salient characteristics of Mode 1 are:
• Reliability
• Efficiency and price optimization
• Plan-driven approach
• Waterfall methodology
• Ability to work with long-term enterprise vendors
• Month-in-scale projects
• Tend to be IT centric
In short, Mode 1 “keeps the lights on.”
Mode 2 addresses the needs of business to respond rapidly to changing market conditions
and customer opportunities. Agility is the hallmark of Mode 2 IT. It is characterized by:
• Speed and responsiveness

• Attention to brand and customer experience
• Agile methodologies
• Continuous deployments
• A focus on short-term projects and working with vendors with specialized skill sets
• Operates well under uncertainty
• Produces incremental products in days and weeks
• Tends to be highly business-and customer-focused

101010 10

We might think mostly of Mode 1 operations when we focus on workload automation, but
that would be a mistake. Mode 2’s more agile approach introduces new kinds of complexity
into an organization. Mode 2 practitioners are experimenters. They may develop a new
process and modify it almost continually in attempts to optimize it. This kind of
experimenting and tuning would not be possible without workload automation tools.
Manually rerunning workload steps would be too time consuming and prone to error.
Gartner recommends that organizations deploy workload automation tools for Mode 1
operations that address the need for reliability and the ability to automate operations from
end to end. For Mode 2, the analysts recommend a tool that enables continuous
deployment. This kind of rapid deployment is feasible in Mode 2 operations but less
appropriate for Mode 1 where change management helps mitigate the risk of disrupting
operations.
Using Mode 1 and Mode 2 operation models is a way to balance competing needs within an
organization. Some tasks need to be executed in a highly controlled and reliable way.
Financial operations, such as generating ledgers and running payroll, fall into this category.
The cost of error can be so high in these areas that it is worth being cautious and slow in an
effort to minimize risk. Other areas, such as deploying a new version of a mobile
application, can tolerate higher levels of risk. If an application is rolled out to employees
and it causes problems for 10% of the users, a fix can be created and deployed rapidly. Yes,
there may be some slight disruption in the efficiency for those 10%, but the risk is worth it
because it is part of a larger innovative process that, over the long term, leads to more
efficient operations.
Automation Risks and Service Disruptions

To err is human. In IT, we are reminded of that over and over again. It is not that those of us
in IT are any more prone to making mistakes than anyone else, but when we deal with
complex systems, we have greater opportunities for something to go wrong. Analysts have
been able to quantify the extent of the problem with mistakes in IT operations that lead to
service outages. In 2010, Gartner estimated that through 2015, 80% of outages impacting
mission-critical services would be caused by people and process issues, with more than
50% of those outages caused by change/configuration/release integration and hand-off
issues.
The fact that people and process issues caused 80% of outages related to mission-critical
service is perhaps the single most effective argument for workload automation in Mode 1
operations. Mode 1 operations are designed to minimize risks of disruptions because the
cost of outages can be so high. When we depend on humans to perform a task over and
over again, the chance that an error will occur increases.
111111 11

Consider examples of the impact of disrupted IT services. On one day in July 2015, three
major service disruptions occurred:
• United Airlines experienced a router issue that caused flights to be suspended for 2
hours. By the time the incident was resolved, 800 flights had been delayed and 60
flights had been cancelled.
• The New York Stock Exchange, the largest in the world, shut down for more than 3
hours. The incident was related to a software update that was rolled out. It turned
out that some customer gateways were not properly configured for the new
software.
• The Wall Street Journal website was down for about 1 hour at the same time the
New York Stock Exchange experienced its outage.
These three incidents are not causally related, but the fact that they all happened on the
same day highlights the potential for service disruptions across a range of industries. Some
disruptions are relatively minor, such as a website being down for a short period of time,
while others are costly events that will attract the attention of C-level executives.
Given that service disruptions should be expected, the question becomes how should we as
IT professionals plan and prepare for them. We cannot anticipate all possible problems and
their root causes, but we can frame processes and workload management procedures with
an understanding that they need to be resilient to failure. Broadly speaking, we can
approach this challenge with manual procedures or with automated methods.
Reducing Risk the Manual Way: Checklists

There are ways to mitigate the risk of human error. Professionals ranging from airline
pilots to surgeons use checklists. The idea of checklists as a means of reducing error was
popularized by Atul Gawande in his book The Checklist Manifesto: How to Get Things Right.
The idea behind a checklist is that it helps us to execute multistep processes while avoiding
errors from skipping a step or executing steps out of order.
Checklists evolve over time. As we learn more efficient ways to do things or discover new
tasks that need to be included in our procedures, we can add them to the list. Checklists
become a repository of collective knowledge about how to perform a task efficiently and
reliably.
Checklists are well suited to some tasks that are inherently manual or static, but in the case
of dynamically changing IT and business processes, oftentimes manually checking off steps
simply isn’t enough.
121212 12

Reducing Risk the Automated Way: Workload Automation

Workload or IT automation is the best option for implementing efficient, reliable IT
processes. Workload automation tools have the advantages of checklists, but are flexible
and scalable to meet changing business needs. They
• Capture collective best practices

• Can be tuned over time
• Are reliable enough to support Mode 1 operations
• Are flexible enough to support Mode 2 operations
Modern workload automation tools can help mitigate the risk of disruption due to the
complexity of IT systems. They will not eliminate disruptions or mistakes, but they can help
minimize the risk of making the same mistake repeatedly.
Another benefit of workload automation solutions is that they can help address the IT skills
gap. By deploying a workload automation tool, organizations can develop a set of skills
across the organization that will serve operations spanning the business.
When evaluating workload automation tools, be sure to consider several essential

characteristics:
• Support both Mode 1 and Model 2 IT operations

• Avoid automation silos
• Leverage automation skills across the organization
• Provide a single point of control and management
• Reduce risk and cost of change
Summary
IT operations are crucial elements of business operations. Some operations must be
reliable and highly available while others are subject to near continual change in response
to new business opportunities. IT departments across industries face a critical skills gap
that leave us putting a premium on solutions that can be reused throughout the
organization. Workload automation tools and best practices can help address these
organizational needs.

131313 13

Chapter 2: The Anatomy of a Workload

Automation Solution
Workload automation has emerged as a solution to a long-standing but constantly evolving
need in IT: the ability to perform tasks repeatedly, consistently, and reliably. Batch
processing, the simplest workload processing model, is a staple of IT operations and has
existed almost as long as IT itself. The applications and tools used to create and run batch
jobs in the past are insufficient for today’s workloads, which are driven by dynamically
changing business requirements.
This chapter considers the composition of a modern workload automation solution,
focusing on:
• A brief history of automation from batch scheduling to workload automation

• Key requirements of workload automation tools
• Advantages of workload automation
• Best practices of workload automation
Throughout the discussion, business requirements will keep appearing as a key influencer
of workload automation.
A Brief History: From Batch Scheduling to Workload Automation

Businesses and other organizations execute their functions through a series of processes.
Banks manage the movement of funds across accounts. Manufacturers fill orders from
customers. Retail businesses stock and sell products. In many cases, there is a physical
component to operations, such as manufacturing a part or packaging a product for sale. In
other cases, the processes are more abstract, such as moving funds between accounts in the
case of financial institutions. In all cases, however, information processing is an essential
part of the business process.
Workload automation is the coordination and execution of a potentially complex array of
business operations focused on information processing. The earliest form of workload
automation was batch processing.
141414 14

In the Beginning: Batch Processing

Early information processing operations tended to focus on bulk operations needed in
large organizations. In business, early computers were used for accounting, payroll, and
other data-intensive (by standards of the time) operations. Similarly, governments would
apply computers to data-crunching tasks, such as calculating census statistics. These types
of operations shared several characteristics:
• Time-based processing
• Relatively stable set of jobs
• Reliable custom scripts
Closing the books and running payroll are the types of operations that run on a regular
schedule. Essentially, there is no unpredictability in these tasks. Fixed, time-based
processing was typical with early batch jobs. This setup was a significant advantage to
application developers and system operators. The tasks that were run last week or last
month would be run in the same order this week or this month. There was no need to
account for complex logic that would trigger the execution of some processes and not
others.
The time-based processing that characterized early batch jobs meant that businesses could
depend on a fairly stable set of jobs. Only so many business operations were automated,
and they tended to address core business operations, such as managing revenues and
payments. These types of business operations are stable and do not change much. In fact,
they do not change much today thanks to well-defined and agreed-upon accounting
procedures.
With a relatively stable set of jobs comes a stable set of scripts to run those jobs. After
programmers write and debug the code behind a batch job, it can continue to run for years
without change—as long as business requirements do not change. There may be some
minor changes in the business logic of an operation—for example, a change in payroll law
that requires a change in the way the process performs a calculation—but these do not
necessitate large restructuring of code or development of new batch jobs.
This description of batch processing sounds almost mythical in the way it describes a time
when simplicity was a defining characteristic of business operations. Of course, developers
had to deal with complex issues, but they centered more around working with scarce
computational resources and the lack of modern development tools. Managing workloads,
once developed and debugged, however, did not entail the challenges IT professionals face
today.
151515 15

Business Changes Drive Changes in Automation

Business processes today change more frequently than in the early days of information
processing. Twenty to fifty years ago, businesses could plan months, quarters, and years
ahead with reasonable confidence that they could foresee the major characteristics of near
future markets, competitive landscapes, and customer expectations.
Changes in Business Require Changes in IT

The pace of business change has accelerated significantly. In the era before online banking
and e-commerce, product offerings and competitive landscapes changed more slowly than
they do today. Businesses had more time to react and adapt business operations to new
opportunities. For example, bank deregulation in the 1990s created business opportunities
for financial institutions, but those institutions had long periods of time to prepare for such
changes.
Businesses now operate in a well-connected, online, and interactive environment.

Customers have become accustomed to around-the-clock access to business services
online. Do you feel like paying bills late at night? No problem, your bank’s bill pay service is
available online. Do you need a last-minute gift for a friend’s birthday? No problem, you can
place an order and have it delivered the next day.
Consumers are not the only ones growing accustomed to instant access to services and
information. Self-service portals are becoming very common in the B2B world. If you need
to track orders, review account statuses, or get reports of past activity, you can access this
information yourself through a customer-facing portal.
If we think about what must be going on behind the scenes of these operations, we can
pretty quickly see that the characteristics of batch processing no longer hold (see Figure
2.1). In particular, these new operations are not:
• Time-based processing; they are triggered in response to external events

• A relatively stable set of jobs; they are changing according to new services needed to
perform new business operations or support customer needs
• Supported by stable, custom scripts that change infrequently; they require new
programming as well as continuous maintenance and bug-fixing
The collective impact of these differences requires a change in the way IT operates.
Processes that worked well in the fairly stable world of batch processing are insufficient to
meet the dynamic needs of today’s business environment.
161616 16

Figure 2.1: Time-based batch processing is too limited for today’s workload
management requirements, which must be capable of responding to a wide array of
event triggers and file constraints.
The Limitations of Custom Scripts for Workload Management

In particular, custom scripts fail to keep pace with change. It is a relatively straightforward
process to write scripts. Developers do it all the time. Problems arise for three reasons:
• Dependency on limited set of skills

• Custom scripts are difficult to modify
• Custom scripts are prone to errors leading to job failures and the need for
debugging
Each of these reasons is difficult enough to address on its own, but these problems often
emerge at the same time, compounding their adverse impact.
171717 17

In addition to this, one of the biggest challenges IT managers face is finding and retaining
skilled staff. IT positions go unfilled and business requirements go unfulfilled. Existing staff
members are asked to continually do more with less resources to meet demands. One of
the needs that can get overlooked is the development of custom scripts to perform a new
set of business tasks in a controlled, repeatable, and dependable way.
In those cases where an individual develops custom scripts, those individuals are typically
also the ones who have to maintain them. When developers are expected to write scripts
on short schedules to meet a very specific requirement, it is no surprise to find the
resulting script is highly tailored to the existing requirements. As new requirements
emerge, the scripts must be modified. If the developer did not have time to anticipate the
types of changes that might be needed in the future, their scripts may be difficult to adapt
to the new requirements. In fact, it is not uncommon in such cases to simply start from
scratch and write a new script to meet the new requirement. This cycle creates a
proliferation of scripts that IT developers must continually maintain.
In the past, when businesses ran a limited set of batch jobs, there was time to develop and
debug code that one could reasonably expect to run for long periods of time. After
programmers wrote a batch script, the primary focus turned to making sure it ran as
expected and addressing problems as they arose (e.g., running out of storage space for an
unusually large set of data). Today, a developer might write a script and then turn her
attention to writing another script or to debugging one that a colleague wrote several
months ago that no longer works because of changes in an upstream process that generates
input for the now problematic script.
At this point, there is an unsustainable set of IT practices that can no longer meet the needs
of businesses. This argument is not a criticism of past practices but a recognition that
business demands on IT are changing. Custom scripting and batch processing is the best
practice when an organization needs to implement a small set of relatively stable,
repeatable processes. As the need for the number of custom scripts grows to meet
emerging business needs, the demands on IT grow in unsustainable ways. IT departments
must change the way they meet workload needs to leverage their staff skills, minimize
maintenance, and ensure reliability of operations. This reality creates the need for
workload automation.
181818 18

The Emerging Demand for Workload Automation

Workload automation is an IT practice that leverages programs and processing
frameworks to execute custom workloads without requiring custom solutions for each new
business task. In particular, workload automation:
• Reduces the need for custom scripts

• Leverages a library of code to execute parameterized jobs
• Reduces the need for custom error handling and alerting logic
Workloads have common requirements and workload automation tools take advantage of
those similarities to provide tools that reduce the time-consuming and error-prone parts of
custom script development.
Reducing the Need for Custom Scripts

If you were to look at tens or hundreds of custom scripts, you would see common patterns.
These patterns include:
• Definitions of constant values, such as names of working directories and control

parameters
• Specifications of input and output files
• Checkpoints to ensure input files exist and checkpoints to warn if an output file
already exists and may be overwritten
• Commands to invoke programs to process data
• Commands to write information about the state of the job to a log file
• Post-processing operations to finalize output so that it is ready to be used by the
next workload script
It would seem that custom scripts are not really needed after all if we can provide another
mechanism to implement common features, such as specifying inputs and writing log data.
Workload automation tools are designed to provide a framework that allows developers to
rapidly configure workload automation processes without having to implement operations
at the low levels of abstraction needed in custom scripts.

191919 19

Leverage Library of Code to Execute Parameterized Jobs

One of the ways workload automation tools can help developers is by providing code for
performing common tasks (see Figure 2.2). Libraries of code offer several advantages, in
particular, the code is typically:
• Designed for reuse

• Debugged
• Optimized for performance
• Easily integrated with other components of the workload automation platform

Figure 2.2: Example library component for performing secure file transfer.
202020 20

For example, a component for specifying an input file could accept a filename as a
parameter and perform a consistent set of operations, such as:
• Verifying the file exists

• Ensuring the file is of the correct format
• Locking the file at the operating system (OS) level to prevent other processes from
modifying it
• Writing metadata about the file (e.g., filename, size, file type) to the workload log file
Libraries with a rich set of components can help developers create processes faster by
eliminating, or at least reducing, the overhead of implementing commonly needed
operations and logic. For example, Figure 2.3 shows a drag-and-drop workflow designer
where users can simply pull from a content library of job steps on the left to easily create
complex workflows.

Figure 2.3: Example of a content-rich workflow designer.
212121 21

Reduces the Need for Custom Error Handling and Alerting Logic
In addition to streamlining the development of individual processes, workload automation
tools help businesses manage collections of these processes (see Figure 2.4). In particular,
by using a common set of error-handling and alerting logic, system operators can use a
single, centralized reporting and alerting tool to monitor the state of multiple workloads.
Businesses that depend on custom scripts can find themselves developing individualized
procedures to differentiate the error logging and alerting of those scripts.

Figure 2.4: Dashboards in workload automation can provide a consolidated view of
the state of multiple workloads.
A multitude of factors contribute to the value of workload automation solutions. No one

feature or small set of features realizes the full potential of workload automation, but a
solution that supports a full array of key requirements can go a long way to radically
improving workload management operations in a business.

222222 22

This section highlights some of the most important features to look for when selecting a
workload automation tool:
• IT organizations depend on a heterogeneous mix of server platforms and operating

systems. Workload automation tools should support the most popular platforms,
including:
o Windows
o Linux
o UNIX
o OpenVMS
o iSeries
o z/OS
Not every organization will have all of these platforms, but you should look for an
automation tool that supports all of your operating systems. Leaving even one OS
out of workload automation can diminish the overall value of the effort.
• Workload automation tools should support multitenant, secure environments. For

example, automation tools should ensure plans and jobs are protected and isolated
from editing and viewing by specific users or groups depending on access control
permissions.
• Documenting code is a best practice that is sometimes left unfinished. This situation
is especially problematic with custom scripts and is one of the reasons it is so
difficult to maintain and debug custom scripts. Workload automation tools can help
by generating self-documentation that is readily understood by developers.
• Business operations are subject to a range of regulations and policies. It is important
that a workload automation tool support compliance and governance requirements.
This is especially the case when businesses must demonstrate to auditors or other
departments they have implemented procedures and have controls in place to
protect the integrity of those procedures.
• A self-service portal is a necessary feature for any organization with multiple
workloads administered by different groups. Just as there are limited staff resources
to write custom scripts, there are limited system support personnel to perform
routine tasks that can be done by end users.

232323 23

• As noted earlier, time-based batch processing can address the needs of a small set of
workload automation requirements. Event-driven automation is an essential feature
of modern workload automation. Common events can trigger workload execution:
o Receipt of an email
o A message delivered in a message queue
o System startup or shutdown
o Web service execution
o Database trigger
Robust workload automation tools have mechanisms to detect and respond to
events across the IT ecosystem. Also look for automation tools that can balance
workloads. You might find, for example, that some servers are running at capacity
while others are underutilized. The ability to move jobs among resources can help
optimize workload execution (Figure 2.5).

Figure 2.5: Workload automation tools should allow administrators to move jobs
across server resources to execute workloads efficiently.
• The contents of the workload automation library strongly influence the overall value of
the platform. Libraries should include tools for common functions:
o Managing files and directories
o Transferring files between servers
o Logging workload information
o Integrating with databases, enterprise resource planning (ERP), customer
relationship management (CRM) and other enterprise applications
o Working with legacy scripts
o Invoking API functions

242424 24

• Workload management processes are defined in software and should be managed

like other software assets. Look for change management support tools, including
versioning and the ability to roll back to previous versions of scripts.
• Consider the way system support personnel and end users will interact with the
system. Mobile access is especially important for receiving alerts about unexpected
events as well as monitoring the status of workloads.
• Workload automation tools should provide a holistic view of jobs executed in the
platform. Reporting and analytics should be available to allow users and workload
administrators to understand overall usage patterns and demands on resources as
well as verify compliance with policies and procedures.
This collection of key features is not arbitrary but constitutes the set of features that
provides the collective advantages of workload automation.
Advantages of Workload Automation

Workload automation addresses several common challenges faced by businesses
responding to the dynamic demands of customers, business partners, and the market in
general. The benefits of workload automation stem from four core advantages:
• More reliable job execution

• Reduced demand on limited skills (improved use of limited skill sets)
• Improved reporting, alerting, and consolidated management
• Faster deployment of new business processes
Workload automation vendors invest design and development resources to produce
platforms that provide a robust execution environment for workload processing.
Developers writing custom scripts typically do not have the time to invest in high-
reliability features that may be used only in a single script. Vendors developing specialized
workload automation tools, however, have a different set of economic constraints and
incentives, and it is in their interest to invest in high-reliability features.
Another advantage is the reduced need for specialized skills. You do not need to find an
expert PowerShell developer or Bash script programmer each time you need to deploy a
new workload or modify an existing process. This idea is also important from an
opportunity-cost perspective: Even if you have the skilled staff on hand to write custom
scripts, is that the best use of their time?

252525 25

IT managers will appreciate the consolidated reporting, alerting, and management that
comes with a workload automation tool. This functionality is especially useful when self-
service portals and mobile device access are provided. Perhaps the biggest advantage of
using workload automation platforms is that they provide the ability to deploy new
business processes faster. As with any software tool, it is the combination of quality
software and best practices that help businesses realize the most benefit from their
investment.
Best Practices in Workload Automation

Over the long history of batch processing and more recently workload automation, IT
professionals have developed a set of best practices, including:
• Code Reuse. Code reuse is a resource multiplier. If one developer writes a

parameterized module for a common task, that module can be reused so that others
do not have to reinvent the proverbial wheel. Libraries also enable referencing of
code instead of copying. The advantage here is that if the library is updated with a
bug fix or performance enhancement, all users of the library code will benefit. If a
developer simply copied and pasted a piece of code, there are two versions of the
code that can drift out of sync.
• Streamlined Installation. You can leverage workload automation tools to ease
installation and upgrades of software throughout the enterprise. What might have
taken days in the past can be done in minutes using workload automation.
• A Focus on Architecture. Think in terms of integrated architectures and
information ecosystems. Begin with an architecture that supports workload
requirements across the enterprise and avoids silos of automation. Silos isolate code
and lead to duplicated efforts.
• Workload Automation Deployed Early and Often. Implement automation when
first starting a new business process. Do not treat automation as an add-on after the
fact. It is difficult to overestimate the benefits of starting a new business process
with workload automation in mind. This approach avoids, or at least reduces, the
need for code that will be thrown away when the process is properly automated. It
is likely that working with a workload automation tool will lead to a faster
deployment than using a custom set of scripts. Remember, even simple scripts can
have bugs, and the last thing any developer wants is to be debugging a script after it
has gone into production.

262626 26

• Using Version Control and Other Software Management Practices. Employ

version control and lifecycle management procedures with workload automation.
Workload automation scripts are valuable assets that need to be managed like other
software assets.
• Workflows Executed Based on Events. Time-based automation is a small subset of
the ways to trigger workload execution. Events are constantly occurring across an
IT infrastructure. Many of these events warrant some kind of action, such as starting
a workflow or reporting the event. Automating based on events, and not simply
time, enables businesses to respond more rapidly and efficiently than if processes
are queued to run only at predefined intervals.
To maximize the benefits of workload automation, be sure to consider best practices such
as these as you map your strategy.
Summary
This chapter has examined the composition of a workload automation solution with a focus
on framing workload automation in terms of the history of batch processing. Businesses
will realize the optimal benefits of automation by choosing a workload automation tool that
supports key requirements and combines that platform with a series of best practices.

272727 27

Chapter 3: The Financial Impact of

Workload Automation
As the past two chapters have illustrated, workload automation is obviously beneficial for
IT professionals. But what may be less obvious is just how important it is to the bottom
line. Streamlining IT operations in ways that free up IT staff, improve the consistency and
reliability of operations, and reduce barriers to innovation are clear advantages workload
automation brings to the IT side of business. These also have direct and indirect beneficial
impact on both costs and revenue streams of a business.
This final chapter of Best Practices for Workload Automation in the Bimodal Era outlines the
ways workload automation leads to financial benefits and provides a roadmap for
implementing workload automation. Specifically, this chapter will address:
• Advantages of consolidation and coordination

• Faster builds and deployments of the workload
• Resource optimization and IT burden reduction
• Next steps and how to get started with workload automation
It is important to note that these benefits span a multitude of different industries. Whether
your organization is a manufacturing business or a financial institution, a healthcare
provider or a retail sales operation, workload automation can help reduce operational
overhead and improve the quality of services in ways that directly affect both costs and
revenues.
Advantages of Consolidation and Coordination

Consider a typical scenario in midsized and large enterprises: Multiple departments across
the organization are constantly deploying new services and optimizing the use of long
running applications. These deployments often require some form of management and
automation. The development department may standardize on one tool, the operations
team chooses another, and both teams find themselves supporting legacy custom scripts
written by several former employees. This scenario can easily lead to fragmented
reporting, inefficient use of software developers, and brittle operations that leave systems
administrators hesitant to revise code that is only understood by an ex-employee who left
the company years ago. Consolidating these tools using a centralized automation platform
is an effective and efficient way to drive coordination and orchestration across the
organization and eliminate these all too common inefficiencies. And more importantly,
when consolidation isn’t an option, automation allows organizations to coordinate with
disparate applications and technologies.

282828 28

When businesses consolidate and centralize workload automation, they often see several
benefits:
• Improved coordination and orchestration

• Fewer skills needed to support multiple tools
• Consolidated reporting
• Reduced need for custom scripting
• Decommissioned software
Businesses that have departments and divisions deploying multiple tools can leverage the
benefits of consolidation to reduce costs.
Fewer Skills Needed to Support Multiple Tools

Someone unfamiliar with IT might think that IT developers and operators all do pretty
much the same thing. This misconception is understandable. Until you’ve worked in IT for a
while, you probably wouldn’t realize the wide range of diverse skills that are needed in a
modern IT department. For example, enterprises might have personnel that focus on
specialties such as:
• Web application design

• Mobile application development
• Database management
• Network operations
• Security and compliance
• User interface design
• Business intelligence
• Operations reporting
Now consider operations support. The staff members all take turns being on call in case a
process fails or there is some other problem with scheduled operations. One night the data
warehouse expert is on call and receives an alert that a job has failed. The failed job is
written in Perl. Unfortunately, the data warehouse expert doesn’t have much experience
with Perl and did not develop that particular job. Digging into the logs, he sees several
messages that give him a clue about the possible root cause but he decides he needs to
actually look into the code to understand the problem.

292929 29

He opens the Perl script and immediately realizes:
• Perl can be a cryptic language

• Perl can use programming patterns different from what one might find in
PowerShell
• The job script makes use of specialized libraries and the purpose of some functions
is unclear
• Error messages generated by the Perl script are fairly low-level errors in a library
with which the developer is unfamiliar
What happens next? The developer starts digging into library documentation and
searching online collaboration sites like Stack Overflow for hints and advice on deciphering
the meaning of these error messages. A seasoned Perl developer, especially the one who
wrote the script, could probably solve the problem in a much shorter period of time.
Unfortunately, that person is not available, so a problem that could have been solved in a
reasonable amount of time drags on until another developer can learn just enough Perl to
debug and correct the problem.
This scenario is repeated over and over and over again in IT shops that use many different
tools for automating their environments.
Perl Ruby
Developer Developer
PowerShell
Developer

Figure 3.1: Developers have overlapping skills and can sometimes work across
application frameworks but doing so often requires deep knowledge of a
programming language or tool to solve complex operational problems.
303030 30

Centralizing through the consolidation and coordination allows IT developers to

concentrate their efforts on mastering one solution and benefitting from the ease of
orchestration and coordination that comes from using one solution. Instead of having a
team with diverse skills but little overlap, this setup has the advantage of a team with
overlapping skills, so the business becomes less dependent on the need for a single hero or
champion with each of the different tools that might be in use. This kind of arrangement is
clearly a benefit to developers. They can go on vacation or work on other projects without
the risk of being pulled into an operational emergency because they are the only one with
sufficient specialized skills to address an operational emergency.
Consolidated Reporting
We are living in a world where human attention has become a scarce commodity. Never
before in history have we had so much information at our fingertips.
Despite this availability of data, businesses are struggling with ways to derive meaningful
insight from various sources. As more and more data becomes available for analysis, IT is
dealing with an abundance of applications, databases, and systems to manage, in addition
to the data dependencies between these systems.
Consolidated reporting is the source of another set of financial benefits. A consolidated

reporting system reduces the overhead associated with collecting data for operations,
management, and compliance reporting.
The Cost of Fractured Workload Environments
The key to this benefit is that workload administrators do not have to work with multiple
data sources to get a comprehensive view of operations. Consider a fractured workload
automation environment; you might find that:
• Some custom scripts write logs to the server on which the workload job is running
• Other custom scripts write to a consolidated logging system on the network
• Each custom script has its own pattern for logging different levels of information
• Log messages have a variety of structures that each requires custom code to parse
Now imagine a manager has asked for a status report that shows the results of a logical unit
of business work, such as fulfilling an order from the time it is entered into a web
application to the time it is shipped and delivered to the customer. One piece of the process
writes data to a custom log format from the web application to indicate the status of the
original order. The inventory system logs data about changes in inventory levels with
respect to the order. The shipping workflow interacts with a third-party service to schedule
delivery using the lowest cost carrier. Each of these pieces is written in a different
programming language using a different log format.
313131 31

Overhead Introduced by Fractured Workload Environments

Executives and managers do not want to work their way through separate log reports
trying to align pieces of information across log files as if it were a business process jigsaw
puzzle. They need a consolidated approach that captures the information across a business
process and presents it in a comprehensive report designed to highlight key business
operations. Anything that impedes this type of reporting imposes a drag on operations and
slows the ability to deliver services and products to customers.
A centralized workload automation tool provides a single source of reporting on business
processes. It can also provide a standard set of reports out of the box. This functionality
further reduces the demands on IT staff to write reports and associated reporting
infrastructure, such as tools to automate report generation, track recipients of reports, and
handle distribution of reports. The net result is less time spent by IT professionals on
workload automation tasks, allowing them to focus their efforts on higher priority business
needs.
Reduced Custom Scripting

Developers and systems administrators can develop a love-hate relationship with scripting.
They can love the ability to write code that does exactly what they want. They also
appreciate the ability to use the programming language of their choice. In addition, if
something goes wrong, they understand the code they have written and can reasonably
expect to solve the problem quickly. Unfortunately, scripting comes with a lot of downsides
for both individuals and organizations.
Continually Evolving Requirements
Writing custom scripts takes time. It is not uncommon to start with a minimal set of
requirements before writing a script. The script can often meet the first and most
important needs of a business user. The problem is that business is not static, and neither
are requirements. A developer might feel like a script is done and move onto another task
when a business user then adds to the original request or the process changes,
necessitating an update to the script. Now the script becomes another item on the
developer’s to-do list.
If such were the case with just one or even two scripts, it would probably be manageable.
That is not the norm. Most organizations have hundreds of scripts. These custom scripts
grow in number and complexity. If the person that originally wrote a script is not available,
someone else might have to modify the script. The second developer may not follow the
same coding style or use the same design patterns. After several iterations of different
developers adding new code, sometimes on tight deadlines, the script starts to lose its
integrity. It becomes more of a patchwork of different interwoven modules than a logically
structured script.

323232 32

The Cost of Ongoing Maintenance

This situation leads to the next problem with custom scripts: they become difficult to
maintain. We can think of this as somewhat analogous to the law of entropy in physics,
which states that over time, a system becomes more disordered. Custom scripts become
more complex as new features are added, often with an eye to finishing quickly instead of
following design patterns or maintaining structural integrity.
To fully understand the cost of custom scripting, we need to consider:
• Developer time needed to write the script

• Business users’ time working around the limitations of minimal scripts
• Business users’ time reporting bugs and feature requests
• Developer time to debug and add new features
• The context switch costs to developers who have to jump from working on one
script to another or, worse, from different types of systems administration to
scripting
Centralized workload automation helps avoid these issues. Workload automation tools
provide a rich library of production-ready Job Steps with built-in logic. This allows for
faster workflow creation as developers can simply drag and drop Job Steps, creating
dynamic workflows that are easily auditable and flexible to changing business needs. As a
result, developers can reuse, rather than rewrite, and reduce the amount of time spent on
researching, designing, testing, and modifying scripts.
Design patterns are more obvious and easier to follow in well-designed workload
automation tools than in “quick and dirty” scripts we sometimes need to write on the job.
In the long run, reducing dependence on custom scripting will reduce development and
maintenance costs while improving the quality of workload automation operations and
reporting.
Decommissioning Unnecessary Software

Every time we introduce a new piece of software to our organization, we introduce
additional overhead in terms of:
• Managing licenses for the software

• Patching and maintaining the software
• Training staff on the software

333333 33

Managing Licenses for Software

Do you know how many different automation tools are currently in use in your
organization? Do you know how many software licenses are available for those tools? Are
you using all of your licenses? Could you actually be unknowingly exceeding your number
of licenses?
These questions arise with any business software. Tracking licenses and optimizing the use
of licenses can be challenging. Different vendors have different licensing models. Some may
have per-user licenses while others have server-based licensing. Add the additional
considerations of running multiple virtual machines on a single server and the move
toward utility computing in the cloud and you begin to see the hidden overhead of using
multiple software tools for the same task.
Patching and Maintaining Workload Software
All software must be maintained. Even code we do not write ourselves still has some level
of maintenance overhead. This requirement usually means we set up tasks to check for
updates and apply those updates on a regular basis. If we are using a piece of software
throughout the organization, it is almost certainly worth the effort to standardize and
automate the patching process. When departments use specialized tools, such as different
automation tools, there may be a tendency to rely on the application administrator to
remember to manually update software on a regular schedule. Needless to say, the latter
strategy is prone to human error that can lead to unpatched software.
Training Staff on Software
Another problem with multiple applications is that they require multiple people to be
trained to use and maintain them. This problem is the same we have identified in previous
chapters. IT staff have limited time to develop skills and perform their jobs. The more
software we add to the enterprise portfolio, the more stress we put on IT personnel to keep
that software running efficiently. This, in turn, requires them to invest time to learn the
new software and understand how it integrates with existing applications, programs, and
devices.
One of the best ways to reduce the burden around managing software licenses, patching
and maintaining software, and training IT staff is to decommission unnecessary software.
Why should your organization deploy several software packages when one would suffice?
It is true that standardizing on one tool will require some migration effort. Scripts written
for decommissioned tools will need to be reimplemented in the consolidated workload
automation platform, but that is a one-time cost. The long-term benefits of a consolidated
approach will likely outweigh the short-term costs of reimplementing workload scripts.
343434 34

There are clear advantages of consolidating and centralizing workload automation:
• Reduced need for diverse and difficult-to-maintain skills for different workload
automation tools
• Consolidated reporting
• Reduced dependence on custom scripting
• The ability to decommission unnecessary software
The benefits do not end with these obvious personnel and financial benefits. There are
additional benefits that stem from increased ability to deploy workload automation
services more quickly and reliably to your organization.
Faster Builds and Deployments

A key advantage of using an established workload automation solution is that automation
vendors invest heavily in making their solutions competitive with each other as well as
with custom scripting. When evaluating workload automation tools, consider four key
features that enable IT personnel to develop and deploy workload automation scripts
faster:
• Libraries of tested code that allow for reuse, rather than rewriting
• Workflow application programming interfaces (APIs)
• Self-documenting operations
• Visual scripting tools
Each of these features can contribute significantly to the ability to build and maintain an
array of workload automation processes throughout your organization.
Libraries of Tested Code

The wheel has been invented; there is no need to reinvent it. The same can be said for
many workload automation functions. Although each business and IT process will have its
own particular requirements, they typically share a number of common types of
operations:
• Authenticating to services, such as databases and operating systems

• Transferring files
• Performing file integrity checks, such as checksums
• Calling Web services
• Writing process data to log files
• Sending notifications through email, SMS, and other channels

353535 35

Workload automation platforms provide libraries of tested code to perform these kinds of
operations and more. They are especially important for saving developer time. Not only do
in-house developers not need to write these scripts from scratch, they do not have to
maintain them. Developers can simply reuse these libraries of tested code, instead of
rewriting scripts over and over again. Also, workload automation tool vendors can invest
time to research the best ways to implement a function, such as authenticating users and
processes. This can lead to a more efficient and secure implementation than might be
possible if an in-house developer is under time constraints to get a functioning custom
script done quickly.
Workflow APIs
Workload automation solutions that support API accessibility promote more connected,
flexible environments. With API accessibility, organizations can incorporate methods and
properties of any technology or application into the automation solution’s content library.
As a result, organizations are virtually unlimited in scope: any technology with an API can
be easily incorporated into workflows, allowing developers to reuse rather than rewrite
scripts using pre-built Job Steps in the content library. As more organizations adopt
service-oriented architectures and the use of microservices, workload automation will fit
with that design model if APIs are readily available.
Self-Documenting Operations
Documentation is a necessary and valuable part of any software. Good documentation is
not always easy to come by. A developer on a tight schedule can get a script up and running
without writing documentation, but she cannot avoid writing necessary code. If forced to
choose between writing code to implement a feature and writing documentation describing
how the implementation works, most developers will choose the former.
Workload automation systems can help mitigate this problem by employing self-
documenting operations. This function can be employed through the implementation of
libraries, modules, and functions that use naming conventions for functions and
parameters that make clear the purpose of the function.
Visual Scripting Tools

Visual scripting tools are another important feature of a workload automation tool (see
Figure 3.2).
363636 36

Figure 3.2: Visual scripting tools help increase the speed in which workload scripts
are created.
Visual scripting tools can help speed development by supporting drag-and-drop access to
libraries and APIs, context-sensitive highlighting, and visual display of information that can
help developers debug workload processes.
Resource Optimization and IT Burden Reduction

The widespread adoption of cloud computing is an important factor in resource
optimization and IT cost control. One of the advantages of shifting workloads to public
clouds is that it allows for just-in-time provisioning of resources. Rather than purchasing
servers to run a specific set of workloads on-premise, businesses can rent virtual hardware
on an as-needed basis.
This approach helps reduce capital expenditures because companies do not have to
purchase servers. It can also help improve efficient use of resources because companies do
not have idle servers on-premise. Cloud resources can be simply shut down when they are
not needed, enabling companies to avoid paying for unneeded computing resources.

373737 37

Combining the advantages of on-demand computing with event-based scheduling of

workloads can further improve the overall efficiency of workload operations. For example,
event-based scheduling allows developers to trigger the start of a workload process after
another completes. This setup minimizes unnecessary overhead associated with
coordinating workload processes. For example, with event-based scheduling, developers
can avoid:
• Running short jobs to check whether another job has finished and resubmitting
themselves to keep checking manually
• Starting a complex workload process but leaving it idle while waiting for the
preceding job to finish
• Having a human operator monitor processes and manually start workload processes
as needed
Modern workload automation tools allow organizations to employ resources only when all
job dependencies are satisfied. They also leverage advantages of cloud computing and the
associated “pay as you go” model that is typical with cloud computing.
In addition to these obvious advantages, consolidated workloads provide for centralized
management and reporting. This functionality can help provide insights into operations
and allow systems administrators and application developers to find ways to improve
overall performance and optimize multiple workload processes (see Figure 3.3).

Figure 3.3: Consolidated workload automation reporting tools can help systems
administrators and application administrators optimize performance of multiple
workload processes.
383838 38

To summarize, centralized and consolidated workload automation systems reduce IT

burden in three specific ways:
• Reducing operational costs

• Improving resource utilization
• Reducing labor costs
The majority of this chapter has focused on the financial benefits of consolidated workload
automation. Assuming the discussion here has led you to believe a consolidated workload
automation tool may be beneficial for your organization, the question that follows is, how
does my company get started?
Next Steps Checklist

The best way to begin the process of deploying a workload automation tool is to begin with
an inventory of existing workload processes and tools and move from there in a logical
manner to migrating existing processes to the new platform. The following list provides the
specific steps to help you realize the benefits of consolidated workload automation
solutions:
1. Assess current set of workflow automation tools

2. Inventory custom scripts
3. Assess available skills, needed skills, and skills gap
4. Identify top workflow candidates for event-driven automation
5. Plan for architecture-based automation by replacing silos of automation with a
single enterprise workflow platform
6. Create a project schedule for migrating existing workflows to a centralized and
consolidated platform
The first step is understanding what workload tools are in place already. This step is
important for two reasons. First, some of the most important workload scripts in your
operations may be run through these platforms. At some point in the history of your
organization, a decision was made to deploy a workload automation tool. That was
probably motivated by the need for a common tool to implement multiple workloads.
It is also important to understand licensing costs around existing workloads. You may be
able to realize substantial savings by retiring multiple workload automation tools. That
kind of cost savings could significantly bolster the business case to implement a
consolidated workload automation tool.

393939 39

Be sure to inventory custom scripts as well. Doing so can be more difficult because there
may not be a central, up-to-date repository of information about custom scripts. A two-
pronged approach is suggested. First, work with application owners to create a
consolidated list of known workload scripts that are running to support ongoing
operations. Second, use scripts to scan servers for cron jobs or other batch-scheduled jobs
running on each server. In an ideal world, the list of workload scripts known by application
owners and systems administrators will match the list generated by the server scan. In
practice, there will likely be differences.
In addition to creating an inventory of workload scripts and tools, be sure to inventory the
skills available in your staff. Also consider what you expect to need in terms of skills and
assess the skills gap, if any. Plan training, if needed, to align with the deployment of the new
workload automation system.
This point in the process is a good time to think about the architecture of your future
workload automation tool. Think in terms of a consolidated infrastructure. Here is a chance
to eliminate silos of workload automation and custom scripts.
Finally, develop a project plan for migrating existing scripts to the new platform. Consider
the risk associated with any disruption in service. It often makes sense to migrate low-risk,
low-complexity workloads first. This approach allows your staff to ensure the platform is
installed and configured correctly without risking a high-impact disruption.
Summary
Centralized workload automation through consolidation and coordination can bring
substantial benefits to an organization. These stem from several factors, including the
benefits of consolidating and centralizing, improved speeds for building and deploying the
workload, optimization of resources, and IT burden reduction. As you plan to deploy a
workload automation tool, be sure to consider the next steps outline provided in this
chapter to help ensure a smooth and efficient transition to your new consolidated
workload automation platform.
404040 40

Best Practices For Workload Automation Ebook PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Best Practices For Workload Automation Ebook PDF

Uploaded by

Copyright:

Available Formats

Best Practices for

Chapter 1: Why Workload Automation? ......................................................................................................... 1

IT Resource Gap ............................................................................................................................................... 2

Bimodal IT and the Need for Workload Automation ......................................................................... 10

Chapter 2: The Anatomy of a Workload Automation Solution .......................................................... 14

Chapter 3: The Financial Impact of Workload Automation ................................................................. 28

Continually Evolving Requirements ............................................................................................... 32

Chapter 1: Why Workload Automation?

• Challenges facing IT executives trying to manage and leverage this evolving

IT Skills Gap Is a Top Concern Among Executives

Evolving Technology Drives Demands for New Skills

• Have a well-defined set of inputs and outputs

• PowerShell in Windows environments

Throwing Oil on the Fire: Multiple Automation Tools

Think Like A CIO on Workload Management: Centralize and

• Fewer distinct tools to support

Bimodal IT and the Need for Workload Automation

• Speed and responsiveness

Automation Risks and Service Disruptions

Reducing Risk the Manual Way: Checklists

Reducing Risk the Automated Way: Workload Automation

• Capture collective best practices

When evaluating workload automation tools, be sure to consider several essential

• Support both Mode 1 and Model 2 IT operations

Chapter 2: The Anatomy of a Workload

• A brief history of automation from batch scheduling to workload automation

A Brief History: From Batch Scheduling to Workload Automation

In the Beginning: Batch Processing

Business Changes Drive Changes in Automation

Changes in Business Require Changes in IT

Businesses now operate in a well-connected, online, and interactive environment.

• Time-based processing; they are triggered in response to external events

The Limitations of Custom Scripts for Workload Management

• Dependency on limited set of skills

The Emerging Demand for Workload Automation

• Reduces the need for custom scripts

Reducing the Need for Custom Scripts

• Definitions of constant values, such as names of working directories and control

Leverage Library of Code to Execute Parameterized Jobs

• Designed for reuse

• Verifying the file exists

A multitude of factors contribute to the value of workload automation solutions. No one

• IT organizations depend on a heterogeneous mix of server platforms and operating

• Workload automation tools should support multitenant, secure environments. For

• Workload management processes are defined in software and should be managed

Advantages of Workload Automation

• More reliable job execution

Best Practices in Workload Automation

• Code Reuse. Code reuse is a resource multiplier. If one developer writes a

• Using Version Control and Other Software Management Practices. Employ

Chapter 3: The Financial Impact of

• Advantages of consolidation and coordination

Advantages of Consolidation and Coordination

• Improved coordination and orchestration

Fewer Skills Needed to Support Multiple Tools

• Web application design

He opens the Perl script and immediately realizes:

• Perl can be a cryptic language

Centralizing through the consolidation and coordination allows IT developers to

Consolidated reporting is the source of another set of financial benefits. A consolidated

Overhead Introduced by Fractured Workload Environments

Reduced Custom Scripting

The Cost of Ongoing Maintenance