You are on page 1of 44

Batch modernization on z/OS An overview part 2

13 June 2012 GSE Belgium z/OS workgroup

De Smet Hendrik IT architect zEnterprise

Trademarks
This presentation contains trade-marked IBM products and technologies. Refer to the following Web site: http://www.ibm.com/legal/copytrade.shtml

2

© 2011 IBM Corporation

Agenda
Batch – Do you still need it? Accomodating new functional requirements in z/OS Batch Implementing dynamic and agile Batch Reducing the Batch window Reducing Batch complexity Batch Modernization with EGL Batch Schedule Performance Optimization Compilers on System z Summary and Q&A

3

© 2011 IBM Corporation

Recap Part 1
Using new functionalities in traditional programming languages

Executing Java Batch in traditional containers

Java stand-alone Batch

WebSphere XD Compute Grid

WebSphere Application Server Feature Pack for Modern Batch WOLA z/OS Batch Runtime (new in z/OS 1.13) PHP with BPXBATCH
4 © 2011 IBM Corporation

Agenda Batch – Do you still need it? Accomodating new functional requirements in z/OS Batch Implementing dynamic and agile Batch Reducing the Batch window Reducing Batch complexity Batch Modernization with EGL Batch Schedule Performance Optimization Compilers on System z Summary and Q&A 5 © 2011 IBM Corporation .

Business is changing faster and faster and responses are required faster and faster too Fast response times are becoming more important – Internet users are expecting direct results from their transactions over the web Business is very dynamic Operational Analytics and dynamic Data Warehouse There may be an increasing need to be able to run batch jobs any time during the day • Periodic interface or print Jobs • On-line triggering and direct execution of given Batch Jobs See also part 1: • WebSphere XD compute grid • CICS batch container • etc… 6 © 2011 IBM Corporation .

Solution Area 1 Optimizing the Information Management Architecture A New Approach to Leveraging Information Information On Demand to Optimize Real-Time Processes Dynamic Warehousing OLAP & Data Mining to Understand Why and Recommend Future Action Batch oriented Query & Reporting to Understand What Happened 7 Traditional Data Warehousing © 2011 IBM Corporation .

ETL challenges FTP Staging FTP BI 8 © 2011 IBM Corporation .

Solution Area 1 Optimizing the Information Management Architecture z/OS Data Warehouse. Transformation and Deployment Analytics & Access BI & BA Analysis and Reporting Linux on System z DW DB2 Data OLTP DB2 Data Data Repository Load Extract Data Integration ETL Pipeline & Parallelism Federation Batch pipes Replicate Event Publisher Changes Effective usage of Data Warehouses in Batch – can reduce access to OLTP Databases Leveraging ETL processes to replicate/duplicate/propagate OLTP data – can make operational data available for batch processing 9 Hypersockets Distributed Data Sources © 2011 IBM Corporation Extract Capture Change Data Integration Federation … Legacy Source Legacy Source Q Replication Extract/Query .

Solution Area 2 Optimizing DB2 Data Access Requirement: – Run Batch and OLTP in parallel on possibly same data Issue: – Parallel access from Batch and OLTP on same data Causing problems: – OLTP response times decreasing because of locks – Loss of efficiency Three areas of focus: – Caching mechanisms – Optimizing SQL Statements – Optimizing system functions: • DB2 for z/OS functions. for example using proper Isolation Level or Clone Table • I/O by means of MIDAW or DSNZPARMS • Optimizing JDBC access (in case of Java clients) • Optimizing checkpoints (commits) • Use DB2 accelerators: IDAA (IBM DB2 Analytic Accelerator) 10 © 2011 IBM Corporation .

. Operational Interfaces (e.g. .. security. DB2 Commands) DB2 for z/OS Log Manager IBM DB2 Analytics Accelerator Data Manager Buffer Manager .IBM DB2 Analytic Accelerator Applications Application Interfaces (standard SQL dialects) DBA Tools. IRLM Superior availability reliability. Workload management z/OS on System z Superior performance on analytic queries Powered by Netezza 11 © 2011 IBM Corporation .. z/OS Console..

Agenda Batch – Do you still need it? Accomodating new functional requirements in z/OS Batch Implementing dynamic and agile Batch Reducing the Batch window Reducing Batch complexity Batch Modernization with EGL Batch Schedule Performance Optimization Compilers on System z Summary and Q&A 12 © 2011 IBM Corporation .

in case of one time zone.. stores and branches.More Workload needs to run in a shorter Batch Window Batch 00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 Online 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 Batch 22:00 23:00 Globalization – Online window falls in different time zones. typically beyond the traditional 6PM deadline – Business continues through Internet and phone A longer OLTP window on most mainframes means a shorter Batch window..why? – data access – application dependencies The need to implement additional functions and to handle more data 13 © 2011 IBM Corporation .... creating the requirement the OLTP window actually needs to be 24 hours a day. there is pressure to make the OLTP window as long as possible – Longer opening times of traditional offices. Even.

Sort-Operation with Pipes 14 © 2011 IBM Corporation . writer ot both) Multiple readers und writers for one Pipe possible Pipes can exploit the CF in case of a Parallel Sysplex with multiple images (BatchPipePlex).MVS BatchPipes to achieve more Parallelism of Jobs Simultaneous read and write of data Jobs can run increasingly in parallel instead of sequential Jobs can use multiple Pipes (reader.

which allows the JZOS ZFile Java class to allocate pipes in a similar manner to regular sequential data sets. 15 © 2011 IBM Corporation . the BPXWDYN Dynamic Allocation service was enhanced to support the SUBSYS parameter.(Rather) New Pipes connectors DB2 Template Utility to specify a pipe as input to the DB2 LOAD utility. allowing transfer of data from transactions using the standard COMMAREA mechanism. CICS SupportPac CA1J provides a connector to CICS transactions. allow users of IBM Communications Server’s FTP functions to specify a pipe for PUT and GET requests.

Agenda Batch – Do you still need it? Accomodating new functional requirements in z/OS Batch Implementing dynamic and agile Batch Reducing the Batch window Reducing Batch complexity Batch Modernization with EGL Batch Schedule Performance Optimization Compilers on System z Summary and Q&A 16 © 2011 IBM Corporation .

Applications are becoming increasingly complex 17 © 2011 IBM Corporation .

Understand your application 18 © 2011 IBM Corporation .

19 © 2011 IBM Corporation .

20 © 2011 IBM Corporation .

21 © 2011 IBM Corporation .

22 © 2011 IBM Corporation . hurting the business‘ agility and time to market High complexity Recommendation: externalize business rules from legacy code and maintain/deploy them using a sophisticated process and tooling.Reducing Complexity: Usage of a Business Rule Management System Many business rules are „burried“ inside application programs – Business rules are hardly recognizable as such – No reuse of these business rules – Maintenance and deployment of business rules by IT staff Long cycle of maintenance.

Reducing Complexity: Usage of a Business Rule Management System Solutions: – IBM WebSphere ILOG JRules on System z with Rule Execution Server (JEE / J2SE) – IBM WebSphere ILOG Rules for COBOL Business Rule Developer Rule Studio Eclipse-based Rule IDE Rule design & development Benefits: – Development of business rules with business users and business user-friendly tools – Central management of business rules – Reuse of business rules – Cost benefits Batch applications – Invocation of external business rules as a service (either Java or Cobol) 23 Design Review Orchestrate Debug Author Deploy Compile & Deploy Rule Repository COBOL App Business Rule Development COBOL Rules CICS TS Business Professional Author Review Deploy Performance Scalability Security Availability Reliability System Management Rule Team Server Web-based console for Rule Maintenance Business Rule Management © 2011 IBM Corporation .

When using ILOG Rules for COBOL. – Batch programs running in stand-alone Java (eventually using JZOS) have easy access to the business rules too.Reducing Complexity: Usage of a Business Rule Management System When running in the Rules Execution Server (Java EE or J2SE) on z/OS: – Batch programs running in WebSphere XD Compute Grid will have easy access to ILOG business rules. 24 © 2011 IBM Corporation . – Traditional batch programs written in COBOL or PL/I and running with JCL see part 1 of this presentation. as the business rules are generated as executable COBOL modules. as both environments are natural Java EE environments. business rules can be easily integrated with existing COBOL batch programs.

. What is your comfort level with going down the custom middleware path? How will you keep your custom middleware from becoming isolated islands of functionality? We ask because the Java batch solutions have differing levels of support 25 © 2011 IBM Corporation . Batch code that is doing the core business processing against the data Batch processing function in support of the core business batch processing This is whatever the batch program does that directly contributes to the business objectives This isare function that provides These functions in support of the batch needs we discussed on the previous the business logic page: For thought or discussion .Reducing complexity: externalize support functions Business Function vs.. The core business function being done relies on support functionality to make it work. Support Function Batch processing does not take place in a vacuum.

especially in Batch – Today. 26 © 2011 IBM Corporation . there is an increasing requirement of transforming XML to z/OS native data and vice versa (see part 1 of this presentation) Development and maintenance of this transformation logic can be complex and expensive and does not add much value for the business Tools such as WebSphere Transformation Extender provide a solution for cost-effective development and maintenance of this transformation logic – Productive toolkit based on Eclipse to develop transformation maps – Transformation maps are deployed to z/OS and executed as separate modules from or in between batch programs Recommendation: externalize transformation logic from legacy code and maintain/deploy this logic using a sophisticated process and tooling.Reducing Complexity: Eliminate custom Transformation Code Large parts of application code deal with transformation of data.

etc. HIPAA. HL7. NCPDP Finance SWIFTNet FIN. SWIFTNet Funds EDI EDIFACT X12. EANCOM And others… Industry packs Map Transform Validate Deployment Options WebSphere ESB WebSphere Application Server 27 WebSphere Message Broker WebSphere Process Server Stand alone WebSphere DataPower System z (native) Windows. Used for both simple and highly complex maps.WebSphere Transformation Extender Transformation mapping = one of the biggest cost for integration projects WTX provides a “universal” mapper that can be used with all of IBM’s ESBs as well as 3rd-party ESBs.) Healthcare HIPAA. SWIFT. Unix. Linux © 2011 IBM Corporation .. Extensible with specific industry packs (e. SEPA.g.

data. whether it is batch. messaging systems. transformation. event-driven. using existing IT infrastructures Ready-to-use solutions for industry standards and regulatory compliance for operational and transactional data integration Multiple execution options to support right-time.WebSphere Transformation Extender: Benefits Highly automated transformation and routing of complex data across many points of integration in real time to support high message volumes Enterprise-wide interoperability supported by a services-oriented architecture (SOA) for seamless connectivity and interoperability across back-office systems Connectivity to a wide range of mainframe. or embedded 28 © 2011 IBM Corporation . and routing Seamless integration across the development. and enterprise applications. real-time. transactional environments to ensure completion and validation of transactions in real time A comprehensive library of more than 120 pre-built functions to reduce development time and simplify specification of rules for validation. and production layers of the enterprise. databases. right-style transformation. existing. and external information sources Support for high-performance.

ESB offerings from IBM WebSphere WebSphere ESB Built on WebSphere Application Server for an integrated SOA platform WebSphere Message Broker Built for universal connectivity and transformation in heterogeneous IT environments WebSphere DataPower Integration Appliance Purpose-built hardware ESB for simplified deployment and hardened security 29 WebSphere DataPower XI50z for zEnterprise Appliance in zBX © 2011 IBM Corporation .

ICEGENER.ibm.Reducing complexity: by exploiting DFSORT / ICETOOL Beyond sorting! You can use DFSORT for other functions beyond sorting. which is available at: http://www. which is available at: http://www. National Language Support. BLDINDEX. Symbols.ibm. Multiple Output. TOD and ETOD Date and Time Formats. Free Form Formats. The following resources provide details: The article Beyond Sorting. This paper reflects DFSORT features available as of October.com/support/docview. and copying data. OUTFIL Features. Continue or Terminate Controls. SAS Booster. and Sources of Information. Product Connections. Reformatting Features.wss?rs=114&uid=isg3T7000094 (Shows you clever ways to use DFSORT and ICETOOL to solve common problems) 30 © 2011 IBM Corporation . INCLUDE/OMIT Tests.com/support/docview. Topics included are: JOINKEYS. SMF. merging. Year 2000 Features.wss?rs=114&uid=isg3T7000091 (This paper highlights and shows examples of some of the features of DFSORT that can improve application performance and programmer productivity. ICETOOL. Floating Sign Format. Time-of-Day Controls. 2010.) – The article Smart DFSORT Tricks.

Agenda Batch – Do you still need it? Accomodating new functional requirements in z/OS Batch Implementing dynamic and agile Batch Reducing the Batch window Reducing Batch complexity Batch Modernization with EGL Batch Schedule Performance Optimization Compilers on System z Summary and Q&A 31 © 2011 IBM Corporation .

Indexed (VSAM) Other… IMS DB IMS DC COBOL RPG Java J2EE C / C++ PLI CICS Integration with and Generation of Existing Applications 32 © 2011 IBM Corporation .EGL Addresses all development needs Presentation Batch Batch 3270/5250 Web Rich Client* Reports Services Business Logic & Control EGL For developers who need to solve Business Problems not Technology Problems Data Access MQ Series DB2 Informix SQL Server Oracle MySQL Derby Informix Cloudscape Serial. Relative.

Agenda Batch – Do you still need it? Accomodating new functional requirements in z/OS Batch Implementing dynamic and agile Batch Reducing the Batch window Reducing Batch complexity Batch Modernization with EGL Batch Schedule Performance Optimization Compilers on System z Summary and Q&A 33 © 2011 IBM Corporation .

Batch Schedule Performance Optimization Do you recognize the following? •The Batch Schedule does not fit any more in the time line •Critical endpoints are not met •The batch schedule is to complex Then it is time to analyze the batch schedule process and analyze what is wrong. After the first quick analyses we can advise performing some steps of the seven step strategy or we advise to bring the batch schedule under governance and after that optimize the batch schedule using BSPO 34 © 2011 IBM Corporation .

.Batch Schedule Performance Optimization BSPO is: – You have your batch schedule under governance – You start optimizing your batch schedule using the anti-pattern versus pattern Strategy: – Incorrect usage of a special resource – Incorrect usage of a start time – Invalid relationship – Obsolete dummy jobs or dummy operations – Actual starttime not equal to endtime last predecessor – Workload Manager unhelpfully set up for batch – Redundant copies of files – Redundant image copies – Batch prevented from effectively using sysplex-wide resources . scripting language and data manipulation language) on the other hand.. And checks on anti patterns. 35 © 2011 IBM Corporation . – With BSPO IBM compares the schedule on one hand with the program sources (programming language.

Agenda Batch – Do you still need it? Accomodating new functional requirements in z/OS Batch Implementing dynamic and agile Batch Reducing the Batch window Reducing Batch complexity Batch Modernization with EGL Batch Schedule Performance Optimization Compilers on System z Summary and Q&A 36 © 2011 IBM Corporation .

Compilers on System z 37 © 2011 IBM Corporation .

Compilers on System z 38 © 2011 IBM Corporation .

2 G H z Single C P U Spe e d 5000 64-way 4000 4 . z 7 2 G 7 H 0 G z H M z H z 2000 2003 2005 2008 z900 z990 z9 EC z10 EC n.Compilers on System z 80-way 5. H 7 16-way1 . 54-way 4 1 32-way G .Wa y C a pa c it y 3000 MHz 2000 4 2 0 5 5 0 1000 3 0 0 M 0 H 1997 z G4 M M H H 1998 1999 z z G5 G6 2010 z196 Compilers technology and synergy with core technology 39 © 2011 IBM Corporation .

Agenda Batch – Do you still need it? Accomodating new functional requirements in z/OS Batch Implementing dynamic and agile Batch Reducing the Batch window Reducing Batch complexity Batch Modernization with EGL Batch Schedule Performance Optimization Compilers on System z Summary and Q&A 40 © 2011 IBM Corporation .

matured over 45 years With the help of new technologies in combination with the classic z/OS Batch strengths the most demanding applications and architectures can be deployed to z/OS. and there is no reason to believe that z/OS can‘t do it! Design Patterns Java PHP BatchPipes zAAP / zIIP ETL WMQ FTE WTX Dynamic Batch ILOG DFSORT © 2011 IBM Corporation WebSphere XD Compute Grid Data Warehousing 41 Proximity of data TWS EGL .Summary As before. Batch is an essential component of the IT landscape and old and new applications are dependent on batch z/OS is an extremely sophisticated and reliable platform for Batch.

com/ 42 © 2011 IBM Corporation .redbooks.ibm.Questions ??? http://www.

com/portals/systemz 43 © 2011 IBM Corporation .ibm.com/financing/us/lifecycle/acquire/zenterprise/ IBM Services for zEnterprise: http://www.redbooks.ibm.com/software/os/systemz/announcements IBM System Storage: http://www.html IBM zEnterprise 114 (z114): http://www.com/services/us/gts/zenterprise/index.html IBM zEnterprise Events Landing Page: http://www.ibm.ibm.More information on zEnterprise IBM zEnterprise landing page: http://www.ibm.com/systems/storage/product/z.ibm.html IBM zEnterprise / System z Redbooks Portal: http://www.ibm.com/systems/z/hardware/zenterprise/index.com/systems/breakthrough IBM software for zEnterprise: http://www.ibm.html IBM Global Financing: http://www.com/systems/z/hardware/zenterprise/z114.

Thai Traditional Chinese Russian Thank You English Bedankt Nederlands Merci French Arabic Obrigado Brazilian Portuguese Gracias! Spanish Danke German Simplified Chinese Japanese 44 © 2011 IBM Corporation .