You are on page 1of 11

The Increasing Need to Automate Complex Workflows

A Look Into Automating FTP and Batch Processes

         

The Increasing Need to Automate Complex Workloads FTP & Batch Processes - Essential processes that never go away or end Hybrid Computing Increases Administrative & Management Complexity The Rise of Workload Automation Introducing Workload Automation The Benefits of Enterprise Job Scheduling & Complex, Hybrid Batch Processing Streamline Processing for Complex Environments Brief Technical Summary Integrated Jobs Library - Designed for Change Recommended Resources & Next Steps

turned to workload automation to orchestrate FTP operations and the management of accounting and banking processes for nearly 2.000 per year) with a 99. and resources. www. the need to automate and streamline becomes even more important. As a result. * Read the Success Story How would you streamline workflows involving more than 100 steps? SBA. needed to receive files daily that contain vast quantities of data used for marketing analysis and identify those files on the fly. Data and files must be moved. deliver them for processing. manually executed steps. an online marketing company. a cell phone tower operator. This entails inordinate amounts of scripting.The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes The Increasing Need to Automate Complex Workloads In the current era of distributed. and administration.ActiveBatch. and send out notifications to customers by combining workload automation and FTP. time. identifying and reporting failures. This takes people. Well known and long proven. and posting results.700 jobs per week (450. which open the organization to considerable risk. monitoring. monitoring the activity. * Read the Success Story Can you write and maintain scripts that handle 8. As a result. tasks must be run and validated and processes coordinated between disparate systems with an extremely low tolerance for errors to minimize costs. But the process has never been particularly efficient or reliable. the original free Internet protocol must be augmented through other tools. decrypt them. Scripts running on different platforms. And one of the key components to all the above mentioned processes is FTP.95% success rate? Precision Dialogue. . * Read the Success Story These aren’t new problems. companies have been relying on their IT organizations to handle tasks like those described above for decades. which are increasingly in short supply.000 vendors supporting the company’s global network of cellular transmission towers. real-time processing. the bulk of the work that IT organizations have to complete on a daily basis continues to be handled primarily through batch workloads. FTP also presents serious challenges in terms of the complexity of its administration and its security shortcomings. Could you reduce 100 SQL Server Agent jobs into 4 automated tasks? SQL Server was not intended for that level of automation but Cincinnati Bell found a way by using workload automation to automate their datacenter tasks that include FTP.

custom file naming. In doing so. PGP encryption. Supports multiple protocol access method while supporting FTPS-. an often overlooked critical component. and dependencies of key elements.ActiveBatch. FTP does not guarantee delivery or even notification of delivery. FTP is inherently insecure. 50-70% of all data processing is batch. This can create problems and delays when sending critical files. FTP’s complicated administration and lack of even the most rudimentary reporting challenge third party tools like those noted above. For example. Even FTP. and HTTPS-based file transfers in addition to . it doesn’t remove unneeded files and scripts once they have been sent. Today. orchestration. SFTP-. For example. And IT scheduling and automation. essentially increasing business risk. Telnet. Examples of common FTP management solutions Sysax A secure FTP client that augments the protocol with workflow capabilities that include zip/unzip. it doesn’t support encryption. which is far from streamlined or efficient. administrators have to manually stay on top of the resulting mess and delete it wherever they can find it. FTP is amazingly sloppy and lazy. ensuring data is passed as needed. but relies on scripting to automate such processes. Entire complex critical business processes can come to a sudden halt. many IT organizations rely on FTP management and automation tools. Enterprise FTP tools are difficult to use and require cumbersome programming and scripting for each operating system. having been developed and adopted before Internet security became a critical issue. the business processes dependent on it falter. can fail and when it does. According to industry estimates from both IDC and Forrester.The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes FTP & Batch Processes Essential processes that never go away or end File transfer and batch processes continue to serve as the cornerstone to enable critical business processes. backup of folders and support both Internet file transfer protocol (FTP) and network file transfer. Meanwhile file transfer provides the lifeblood of data to fuel business processes. scheduling is integral to the success of not only the FTP process but to the larger overall business process. which means that under normal conditions user names. and Secure Shell access. FTP commands. passwords. a well known and long accepted protocol. The resulting systems must be closely monitored and managed. e-mail notification. especially when SLAs are present. Fling FTP Sync and Upload An FTP tool to upload and transfer files and folders. Lastly. www. and connecting to multiple FTP servers with different protocols. Instead. handles the sequencing. and transferred files can be captured by anyone on the same network using just a packet sniffer.

simply is too fragmented. The usual solution is to use a set of specific FTP point solutions such as those listed above. This piecemeal approach to IT management and automation. You identify a specific automation problem and solve it with a specific or point automation solution. www. Cron. the solution has been to load up with platform. or a solution for the DBA looking to automate database tasks. These include serious risks associated with file transfer delays or failures. Most IT organizations manage these distributed environments through middleware-based integration and scripting coupled with a variety of automated tools for management and orchestration. a job scheduler to fulfill batch processing requirements. This same concept extends to process types as well. UNIX and mainframes. and on. Moreover. Then multiply it for different processes and. As noted above.and/or departmentalspecific scheduling tools. referred to as an elemental approach since it consists of multiple. such as implementing an automation tool for runbook automation. each intended to handle specific tasks or process types. The organization then seeks the appropriate point solution to be implemented. speed processes. The elemental approach starts easily enough. This hinders organizations trying to reduce IT costs.The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes FTP’s shortcomings expose any organization to significant business risks. the elemental approach simply is not sustainable. thereby increasing operational costs and diminishing staff productivity. This becomes very costly. In terms of automating the critical IT operational and business processes that span these heterogeneous environments. For example. They may have a mix of Windows. well you get the picture. require constant revision and resynchronization in the face of change. a DBA needs to automate database backups or a developer requires automation of a repetitive and time-consuming FTP task. Hybrid Computing Increases Administrative & Management Complexity This fragmented approach is largely the result of distributed computing environments. SQL Server Agent. . each tool requires a license and training. It also can jeopardize conventional batch processes that are dependent on successful FTP operations. And unless you undertake costly integration—which can fall apart the moment something new is added or something changes—the tools are unlikely to share information or play well with each other. It’s a piecemeal approach whereby the organization quickly ends up with a myriad of point automation tools. but these “point” solutions present another drawback: a fragmented approach that involves time-consuming and error-prone manual procedures spread across siloed tools/departments. whether a scripting tool. a Windows Task Scheduler. a tool for automating the FTP process. developed independently of each other. these uncoordinated automation solutions. non-interoperable elements. and increase efficiency.

The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes Required instead is an architected approach to automation and management. DeleteFile. when a bank would call inquiring about a missing or incomplete file. SFTP and MFT. For example. The architectural approach is designed to accommodate change. certificates and protocols. when something does go wrong. enterprise-wide scheduling and automation solution. For-Each-File. It meant. It will be aware. such as SSH. even worse. Rather than creating individual jobs for each file. Another frustration: SQL Server Agent itself had limited alerting capabilities. www. This certainly applies to FTP operations as well. which meant that a problem wouldn’t be discovered until a user logged in to manually check on the process or. of the need to automate compound and complex workloads running across different pieces of the hybrid environment. CreateFolder. In the process. In fact. reduced errors. these processes were handled as individual jobs in a SQL Server database and scheduled and executed via SQL Server Agent. which would allow it to simplify the repetitive and time-consuming task of developing and maintaining hundreds of individual jobs. IfFile and many more Securely send FTPs and the ability to automatically restart from a point of transmission failure and use a centralized alert and monitoring framework to alert IT a problem Use the completion of an FTP job to pass data downstream via a broad range of Job Steps as part of the Integrated Jobs Library The Rise of Workload Automation Workload automation recognizes that workloads consist of multiple process types that run across platforms and departments. Job Steps include: MoveFile. and able to support multiple platforms running different workloads and types of data. . it will bring the proven benefits of automation—streamlined operation. lower labor costs. The team found a workload automation product that had the ability to pass variables like filenames between job steps. a workload automation solution provides the ability to securely send FTPs and automatically restart from the point of transmission failure while providing a centralized alert and monitoring framework to alert IT Operations of a problem. CopyFile. Although the jobs shared many of the same properties there were enough small frustrating differences—maybe just the difference in the FTP file name— between the jobs to require the IT staff to treat each as an individual job. be extensible and flexible. As organizations increasingly deploy compound workloads running across disparate systems they need an architectural solution to ensure workload performance end-to-end. creating and maintaining over 100 individual jobs. The company realized that it needed a more robust automation tool. IT gains extended visibility and control over complex batch processes and the FTP operations that they are dependent on. Benefits of Using ActiveBatch® to Take an Architectural Approach to Automating FTP Operations Improve productivity by unifying point scheduling tools into a single. These processes included passing data and files between various internal accounting and transactional systems as well as sending and receiving data files from banks. By adopting an architectural approach and incorporating the management of FTP operations within your enterprise scheduling and automation solution. For example. At the time. by using a pre-defined FTP job step in the tool’s job library they could designate certain variables as part of the job step and share that variable with other jobs. out of the box. and greater efficiency. Cincinnati Bell needed a more robust automation solution to execute a series of DTS and SFTP jobs critical to the company’s banking and accounting processes. in addition to supporting secure encryption standards such as OpenPGP Support secure authentication sources.ActiveBatch. less reliance on manual processes and scripting Gain extended visibility and span of control over complex batch processes that are cross-dependent on FTP operations and other processes Drag and drop production-ready Job Steps for FTP operations into workflows that contain other datacenter and business processes Support standard-based file transfer protocols with predefined Job Steps that include FTP. SSL and TLS Common file system operations and flow control logic predefined Job Steps to allow for the end-to-end automation of FTP operations and dependent file operations.

enables organizations to:      Automate complex hybrid workloads Follow an architectural rather than elemental approach by combining FTP management scheduling. which supports the loading of web services.The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes they could create a single job and run it multiple times. a workload automation solution from Advanced Systems Concepts. As both Cincinnati Bell and SBA found. and technologies. . ActiveBatch allows IT organizations to take an architectural approach to IT automation by consolidating scripts and platform-specific scheduling tools into a single framework. and data movement in one toolset Eliminate the need to find. These job templates include processes for secure FTP. It was a similar story at SBA Communications. using a job variable to insert the correct file for each instance.Net Assemblies and more. ActiveBatch. The company had put in place a series of homegrown batch processes and custom scripts running via Windows Task Scheduler. thereby avoiding many problems and the time required to find and correct them. Introducing Workload Automation The tool both Cincinnati Bell and SBA found is ActiveBatch. The tool also let them verify the correct file through automated matching before an FTP job was sent. Inc. stored procedures. These batch jobs required considerable manual intervention and offered limited error-reporting capabilities. The ActiveBatch Integrated Jobs Library provides over 130 production-ready Job Steps. they consolidated 100 SQL Server Agent jobs into four job plans. the architectural tool. one of the largest independent owners and operators of cellular infrastructure across North and Central America. databases. As a result. SBA found an architectural tool that reduced the time spent when an error does occur from an hour to about 5-10 minutes. license. . they can construct workflows leveraging many of the more than 130 pre-defined yet customizable production-ready steps the tool provides from its jobs library. SBA realized it really needed a centralized scheduling tool with reporting and alerting capabilities that would allow IT operations to pinpoint errors quickly. This translates into a 50% productivity improvement for IT staff. reusable templates designed to simplify the development of business and IT operational processes composed of various applications. To automate these processes. batch. OpenPGP and data encryption. SBA. had been relying on a mix of disparate scheduling tools to automate the myriad of processes that support the building and upkeep of its cellular tower network. These are augmented by a Services Library. learn. The company also was able to encapsulate many of the custom scripts it already had in production as reusable jobs. The processing and movement of payment files is based on data that SBA’s accounting system produces. and manage multiple tools Address complex and compound process dependencies Deliver functionality through a central point of control www. In the end.ActiveBatch. The tool they eventually found serves as the scheduling engine to automate hundreds of accounting processes for securely sending and receiving payments with over 2000 vendors on a daily basis.

700 jobs per week (450. ActiveBatch automates all of this. and send out notifications to customers that the files were successfully received. deliver them for processing. Precision Dialogue ActiveBatch’s ability to automate mission-critical workflows across disparate technologies has paid big dividends for Precision Dialogue www. and serves as the hub of these processes and many more. We identify those files as they come in.The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes Precision Dialogue runs 8.Joe Schmidt.ActiveBatch.95% success rate.000 per year) . decrypt them. which has resulted in savings in both time and resources. through ActiveBatch with a Every day we receive FTPs from customers that contain vast quantities of data that we use for marketing analysis. Production Team Manager. .

automation. automation as delivered by ActiveBatch. and scheduling it is exactly what IT and business managers need to meet ever more demanding service levels while still containing costs and reducing risk. pre-defined Job Steps that support FTP protocols and OpenPGP  Improved Monitoring and Reporting: Automated job and process alerting and monitoring capabilities Streamline Processing for Complex Environments Companies will not suddenly stop running complex and compound workloads because that is how their business processes must work in today’s highly competitive and demanding environment.The The Increasing Increasing Need Need to to Automate Automate Complex Complex Workflows Workflows || A A Look Look Into Into Automating Automating FTP FTP and and Batch Batch Processes Processes The Benefits of Enterprise Job Scheduling & Complex. Hybrid Batch Processing ActiveBatch brings a number of compelling benefits to your core hybrid computing environment:  Cost Savings: Combined automated file transfer.ActiveBatch. www. hybrid workloads  Simplified Operations: Address process dependencies automatically  Higher Productivity: Script-less automation and integration through the Integrated Jobs Library  Minimized Skills/Training: . multi-process. By combining FTP. and scheduling  Error Reduction: Extended visibility and control over processes. And neither is FTP going away anytime soon. including FTP operations  Reduced Risk: Securely send files via FTP and automatically restart from failure  Increased Efficiency: Automatically execute compound. There is no going back. Fortunately. Hybrid systems and compound and complex processes will increasingly be the norm going forward. batch processing. simplifies complex hybrid processing while removing errors and the associated risk in the process.

Microsoft's Hyper V. FTP Embedded Within Automated Batch Scheduling Consolidated View Showing All Dependencies Built on Industry Standards. Extensibility. x64 and Itanium Microsoft Azure Amazon EC2 Hypervisors—VMware. Customizability                 Microsoft Windows .ActiveBatch. . Interface Support                  SAP NetWeaver BW SAP BusinessObjects Data Services Informatica PowerCenter Oracle PeopleSoft Oracle E-Business Suite IBM Cognos BI IBM InfoSphere DataStage Netezza Teradata Microsoft System Center Scripts Managed File Transfers Web Services . Protocol. Diverse Application. . . Vendor Platform-Neutral www.The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes Brief Technical Summary Object-Oriented Enables Reusability. . Customizability Integrated. Architectural Tools Approach Eliminates Siloed Tools Multi-Platform Enables Reusability.Net Assemblies Java OpenPGP Certificates FTP/SFTP/MFT much more . Xen Mac OS Linux SuSe Linux IBM AIX IBM iSeries (AS/400 – OS/400) IBM z/OS HP-UX HP Tru64 Unix HP NSK SCO UNIX Sun Solaris SPARC and x86 OpenVMS Alpha and Itanium much more . Extensibility.

The Integrated Job Library allows developers to more easily construct workflows from within the Library's drag-and-drop interface to integrate a wide range of IT technologies. FTP Workflow as seen in the ActiveBatch Integrated Jobs Library Learn More About IT/Business Process Automation and ActiveBatch's Integrated Jobs Library » .The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes Integrated Jobs Library .Designed for Change ActiveBatch offers over 130 production-ready Job Steps within its Integrated Jobs Library.ActiveBatch. tasks and functions without having to invest time and resources in creating custom scripts.

Learn how to achieve a near real-time data warehouse using job scheduling software.ActiveBatch. Why IT Organizations Need to Take an Architectural Approach to IT Automation • Consolidate islands of automation • Reduce the cost of IT operations • Implement a solution built for change • • • Improve productivity and reduce "slack" time Improve resource management Take a strategic approach to IT automation Interested in an Online IT Automation Consultation? Advanced Systems Concepts. We would be pleased to provide you and your team with a personalized IT automation consultation to discuss your automation requirements. Inc. has been working closely with customers in over 45 countries and assisting them in achieving their automation goals. Start Building Your IT Automation Strategy Get started with a personalized ActiveBatch IT Automation Consultation » www. • Service Level Agreements (SLAs) • Resource Management • Dealing with IT Complexity • Defining Job Requirements Solutions for Automating IT Job Scheduling Get insight to conquering challenges associated with trying to coordinate task across disparate technologies and solutions.ActiveBatch. • Making BI data useable • Reusing IT Jobs in a plans • How to manage multiple protocols • Reusing IT plans in a workflow How to Achieve a Near Real-Time Data Warehouse Data warehouses are no longer simply the repositories of business data. Visit .com or call +1-973-539-2660 to get started.The Increasing Need to Automate Complex Workflows | A Look Into Automating FTP and Batch Processes Recommended Resources & Next Steps The Shortcut Guide to IT Workload Automation and Job Scheduling This eBook offers help to professionals who must schedule jobs on multiple applications. servers and platforms within their organization and also serves as a workload automation best practices guide.