DB2 Query CPU Parallelism & IIPHONORPRIORITY

By Jeff Lane, IDUG Content Committee July 29, 2013 Would you ever think that binding your on-line packages with DEGREE=ANY would cause your transactions to increase in elapsed time? Well its not just binding with DEGREE=ANY but a combination of available processors (GPs & zIIPs), and z/OS settings which can contribute to increasing transaction elapsed times. I encountered a situation in a small shop where the production LPAR was configured with 2 GPs (General Processor CPUs) and 1 zIIP engine. An overzealous DBA had rebound all the on-line packages with DEGREE=ANY thinking this couldnt hurt since there were multiple engines available to DB2 for query CPU parallelism. With DB2 query CPU parallelism, DB2 can break up a query into many subtasks (called child tasks) and these subtasks can then be run in parallel on the available CPUs (GPs & zIIPs). Since parallel query subtasks are zIIP eligible, not only does the number of GPs come into play when DB2 is determining the degree of parallelism it can achieve, but the number of zIIP engines are considered as well. What our anxious DBA may not have known was the systems group had the z/OS parameter IIPHONORPRIOIRTY set to NO. IIPHONORPRIORITY is set in the IEAOPTxx member of SYS1.PARMLIB and determines where zIIP eligible work will run. With IIPHONORPRIORITY=YES, zIIP eligible work can overflow to GPs if the zIIP engine(s) are busy. IIPHONORPRIORITY=NO forces all zIIP eligible work to run on zIIP engines. If the zIIP engines are busy, the tasks will wait. After rebinding with DEGREE=ANY, our on-line packages now have parallel tasks and the task are zIIP eligible. Keep in mind we have 1 zIIP engine, and IIPHONORPRIORITY is set to NO. As a result, our InDB2 times increased because DB2 would set up all the parallel subtasks at runtime, and the subtasks would run serially on the single zIIP engine. DB2 accounting records show: Class 2 CPU & Suspend times due to parallelism: CP CPU TIME PAR.TASKS=.000951 indicates parallel tasks running on CPs (GPs) SE CPU TIME (Specialty Engine CPU) PAR.TASK=.002367 indicates parallel tasks running on zIIPs SUSPEND TIME PAR.TASK=1.026066 is almost half the In-DB2 wait time NOT ACCOUNT.=.015151 can be attributed to CPU dispatch wait Class 3 Suspensions due to parallelism: SER.TASK SWTCH OTHER SERVICES = 2.002435 is from PAR.TASK suspend time

Page 1 of 4

DB2 Query CPU Parallelism & IIPHONORPRIORITY

AVERAGE APPL(CL.1) DB2 (CL.2) IFI (CL.5) ------------ ---------- ---------- ---------ELAPSED TIME 20.923280 1.061133 N/P NONNESTED 20.923280 1.061133 N/A STORED PROC 0.000000 0.000000 N/A UDF 0.000000 0.000000 N/A TRIGGER 0.000000 0.000000 N/A CP CPU TIME AGENT NONNESTED STORED PRC UDF TRIGGER PAR.TASKS SECP CPU SE CPU TIME NONNESTED STORED PROC UDF TRIGGER PAR.TASKS SUSPEND TIME AGENT PAR.TASKS STORED PROC UDF NOT ACCOUNT. DB2 ENT/EXIT EN/EX-STPROC EN/EX-UDF DCAPT.DESCR. LOG EXTRACT. 0.009507 0.008552 0.008552 0.000000 0.000000 0.000000 0.000955 0.000005 0.002371 0.000000 0.000000 0.000000 0.000000 0.002371 0.000000 N/A N/A 0.000000 0.000000 N/A N/A N/A N/A N/A N/A 0.009180 0.008229 0.008229 0.000000 0.000000 0.000000 0.000951 N/A 0.002367 0.000000 0.000000 0.000000 0.000000 0.002367 2.061452 1.035386 1.026066 N/A N/A 0.015151 59.43 0.00 0.00 N/A N/A N/P N/A N/P N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/P N/P CLASS 3 SUSPENSIONS AVERAGE TIME AV.EVENT -------------------- ------------ -------LOCK/LATCH(DB2+IRLM) 0.000266 1.43 IRLM LOCK+LATCH 0.000154 0.14 DB2 LATCH 0.000112 1.29 SYNCHRON. I/O 0.053917 35.90 DATABASE I/O 0.053917 35.90 LOG WRITE I/O 0.000000 0.00 OTHER READ I/O 0.004824 1.76 OTHER WRTE I/O 0.000000 0.00 SER.TASK SWTCH 2.002435 4.00 UPDATE COMMIT 0.000000 0.00 OPEN/CLOSE 0.000000 0.00 SYSLGRNG REC 0.000000 0.00 EXT/DEL/DEF 0.000000 0.00 OTHER SERVICE 2.002435 4.00 ARC.LOG(QUIES) 0.000000 0.00 LOG READ 0.000000 0.00 DRAIN LOCK 0.000000 0.00 CLAIM RELEASE 0.000000 0.00 PAGE LATCH 0.000000 0.05 NOTIFY MSGS 0.000000 0.00 GLOBAL CONTENTION 0.000009 0.10 COMMIT PH1 WRITE I/O 0.000000 0.00 ASYNCH CF REQUESTS 0.000000 0.00 TCP/IP LOB XML 0.000000 0.00 TOTAL CLASS 3 2.061452 43.24 HIGHLIGHTS -------------------------#OCCURRENCES : 21 #ALLIEDS : 21 #ALLIEDS DISTRIB: 0 #DBATS : 0 #DBATS DISTRIB. : 0 #NO PROGRAM DATA: 0 #NORMAL TERMINAT: 21 #DDFRRSAF ROLLUP: 0 #ABNORMAL TERMIN: 0 #CP/X PARALLEL. : 21 #IO PARALLELISM : 0 #INCREMENT. BIND: 0 #COMMITS : 21 #ROLLBACKS : 0 #SVPT REQUESTS : 0 #SVPT RELEASE : 0 #SVPT ROLLBACK : 0 MAX SQL CASC LVL: 0 UPDATE/COMMIT : 0.00 SYNCH I/O AVG. : 0.001502

Fig 1. DB2 Accounting report when package bound with DEGREE=ANY

When we rebound the package to DEGREE=1, all the extra In-DB2 (CPU & Suspend) times related to PAR.TASKS went away.
AVERAGE APPL(CL.1) DB2 (CL.2) IFI (CL.5) ------------ ---------- ---------- ---------ELAPSED TIME 0.137067 0.039487 N/P NONNESTED 0.137067 0.039487 N/A STORED PROC 0.000000 0.000000 N/A UDF 0.000000 0.000000 N/A TRIGGER 0.000000 0.000000 N/A CP CPU TIME AGENT NONNESTED STORED PRC UDF TRIGGER PAR.TASKS SECP CPU SE CPU TIME NONNESTED STORED PROC 0.004675 0.004675 0.004675 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.004378 0.004378 0.004378 0.000000 0.000000 0.000000 0.000000 N/A 0.000000 0.000000 0.000000 N/P N/A N/P N/A N/A N/A N/A N/A N/A N/A N/A CLASS 3 SUSPENSIONS AVERAGE TIME AV.EVENT -------------------- ------------ -------LOCK/LATCH(DB2+IRLM) 0.000000 0.00 IRLM LOCK+LATCH 0.000000 0.00 DB2 LATCH 0.000000 0.00 SYNCHRON. I/O 0.034098 7.00 DATABASE I/O 0.034098 7.00 LOG WRITE I/O 0.000000 0.00 OTHER READ I/O 0.000000 0.00 OTHER WRTE I/O 0.000000 0.00 SER.TASK SWTCH 0.000000 0.00 UPDATE COMMIT 0.000000 0.00 OPEN/CLOSE 0.000000 0.00 SYSLGRNG REC 0.000000 0.00 EXT/DEL/DEF 0.000000 0.00 OTHER SERVICE 0.000000 0.00 ARC.LOG(QUIES) 0.000000 0.00 LOG READ 0.000000 0.00 DRAIN LOCK 0.000000 0.00 CLAIM RELEASE 0.000000 0.00 PAGE LATCH 0.000000 0.00 HIGHLIGHTS -------------------------#OCCURRENCES : 1 #ALLIEDS : 1 #ALLIEDS DISTRIB: 0 #DBATS : 0 #DBATS DISTRIB. : 0 #NO PROGRAM DATA: 0 #NORMAL TERMINAT: 1 #DDFRRSAF ROLLUP: 0 #ABNORMAL TERMIN: 0 #CP/X PARALLEL. : 0 #IO PARALLELISM : 0 #INCREMENT. BIND: 0 #COMMITS : 1 #ROLLBACKS : 0 #SVPT REQUESTS : 0 #SVPT RELEASE : 0 #SVPT ROLLBACK : 0 MAX SQL CASC LVL: 0 UPDATE/COMMIT : 0.00

Page 2 of 4

DB2 Query CPU Parallelism & IIPHONORPRIORITY

UDF TRIGGER PAR.TASKS SUSPEND TIME AGENT PAR.TASKS STORED PROC UDF NOT ACCOUNT. DB2 ENT/EXIT EN/EX-STPROC EN/EX-UDF DCAPT.DESCR. LOG EXTRACT. 0.000000 0.000000 0.000000 0.000000 N/A N/A 0.000000 0.000000 N/A N/A N/A N/A N/A N/A 0.000000 0.000000 0.000000 0.034098 0.034098 0.000000 N/A N/A 0.001011 44.00 0.00 0.00 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/P N/P NOTIFY MSGS GLOBAL CONTENTION COMMIT PH1 WRITE I/O ASYNCH CF REQUESTS TCP/IP LOB XML TOTAL CLASS 3 0.000000 0.000000 0.000000 0.000000 0.000000 0.034098 0.00 0.00 0.00 0.00 0.00 7.00 SYNCH I/O AVG. : 0.004871

Fig 2. DB2 Accounting report when package bound with DEGREE=1

Rebinding with DEGREE=1 solved our In-DB2 elapsed time problem, but it presented the organization with a dilemma. Should the DB2 workloads run: 1) Faster (lower DB2 elapsed time) with no parallelism and utilizing all the GPs? 2) Or, run slower (& cheaper) with parallelism on the zIIP? From a technical DB2 standpoint, simply rebinding with DEGREE=1 and DEGREE=ANY, respectfully, answers the questions. But, in this case, rebinding isnt just a DBA decision. Changing to DEGREE=1 for our on-line packages will move workloads (DB2 query CPU parallelism tasks that were running on zIIP) back to the GPs. Moving DB2 zIIP eligible workload to GPs becomes a capacity and system tuning exercise and the setting of IIPHONORPRIORITY cannot be ignored. Capacity team has to understand: What the current CPU utilization is for both GPs & zIIPs Can the GPs handle the additional (DB2 query CPU parallelism) workload on average & during peak processing? How much workload can overflow & when? What are the ramifications of the workload moving to GPs?

We discussed leaving parallelism on and setting IIPHONORPRIORITY to YES. However, setting IIPHONORPRIORITY to YES would allow all zIIP eligible tasks to overflow to the GPs not just DB2 tasks. DB2 query CPU parallelism isnt the only DB2 feature, or product, which can utilize zIIP engines DB2 DDF workloads, some DB2 Utilities (Runstats, Index Rebuilds), SYNCSORT, TCP/IP, XML processing, and other products are now utilizing zIIPs. Answering the capacity questions (from above) is much more difficult when all zIIP eligible workloads are involved. Setting IIPHONORPRIORITY to YES brought other concerns as well. Moving zIIP eligible workloads to GPs can affect your monthly licensing costs (MLC). For example, if your GPs are highly utilized, moving zIIP workloads to GPs could potentially raise the 4 hour rolling average of CPU usage. This could cause additional software costs or, worse yet, a CPU upgrade.
Page 3 of 4

DB2 Query CPU Parallelism & IIPHONORPRIORITY

Furthermore, running on zIIP is usually cheaper than running on GPs. If charge back is in place and your customers are paying a reduced rate for zIIP, are they going to accept a higher charge if their workloads are moved from zIIP to GPs? Conclusion: In our case, In-DB2 elapsed times were elongated because of the combination of DEGREE=ANY, our number of GPs & zIIPs, and the setting of IIPHONORPRIORITY to NO. After many discussions, we opted to rebind all the on-line packages with DEGREE=1 mainly because query CPU parallelism isnt recommended for on-line transactions and it will allow us to re-baseline the application. But, this decision didnt come easily as we had to consider IIPHONORPRIORITY and other shifting workloads. Our technical dilemma became a business decision. CONSIDERATIONS FOR IIPHONORPRIORITY=YES: -

Default is YES (zIIP workloads can overflow to GPs) Workload will certainly go back to GPs, but is the customer willing to pay the cost increase and how much will they be willing to accept? Is there enough GP capacity to handle the workload coming from zIIP? If moving the workload back to the GPs causes an overall increase in the 4 hour rolling average, do we take a hit on any MLC charges?

CONSIDERATIONS FOR IIPHONORPRIORITY=NO: -

Is toggling IIPHONORPRIORITY to NO for batch processing, and using parallelism to reduce batch (GP) CPU costs worth investigating? Can some CICS transactions with batch type characteristics use DEGREE=ANY to reduce costs by running on zIIPs? Is there enough zIIP capacity to handle the eligible workload from all users? In DB2 v10, prefetch I/Os from DBM1 are scheduled on zIIP engines. Prefetch I/Os are read-ahead I/Os and we dont want DB2 to wait on I/Os. If IIPHONORPRIORITY is NO and the zIIP engine(s) are highly utilized, there is a chance DB2 prefetch tasks could be delayed.

Jeff Lane J. Lane Consulting Mainframe Relational Database Consulting jslane@msn.com Jeff Lane has been in IT for more than 30 years with more than 27 years in various DB2 capacities. Jeff is an independent consultant and specializes in DB2 for z/OS performance and tuning in large DB2 mainframe environments. He confers on a regular basis with many DB2 sites both locally and around the nation on technical DB2 issues. Jeff has participated as a query panel member and presenter in past IDUGs and regional user groups.

Page 4 of 4