You are on page 1of 41

Performance Tuning Oracle E-Business Suite

Part 2: Concurrent Processing and Tracing


1

Andy Tremayne, Senior Architect & Uday Moogala, Senior Principal


Applications Performance Oracle Development

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Agenda
Tuning the Concurrent Manager
Tracing

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

<Insert Picture Here>

Concurrent Manager
Introduction
50% of performance tuning is in the business!
Review long-running auto resubmitted jobs
Review short/long-running jobs selective parameters?
Dont purge jobs that the users then have to run again

Points to note
More managers/workers != more throughput
Avoid enabling an excessive number of managers/workers

New systems: a start point is 1-2 target process per CPU


Balance with resources required by online activity
3

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
General Points
Conflict Resolution Manager
Carefully consider reducing the CRM sleep time to 5 ~ 10 secs

Purge issues
2GB limit on any output files esp. reports.log (Doc ID 842850.1)
Set Concurrent: Force Local Output File Mode" to "Yes
Patch 7530490 for R12 or 7834670 for 11i (Doc ID 822368.1)

High throughput: Keep out and log on fast disks


Separate if throughput seems slow esp. on NFS Filers

Use IPC or bequeath if the CM and DB are co-located


4

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Pipes vs Queue
Use queues in RAC

System Profile: Concurrent:TM Transport Type: Pipe/Queue


Failover
Available in 11i (Doc ID 413636.1)
For R12 (or 11i post Oracle 10.1) AQ is largely self-tuning

Single instance can still use pipes


Some customers have reported better performance
Description of how pipes work Doc ID 105133.1

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Transaction Managers
Synchronous processing: PO Approvals
PO, CRP, INV, AR, and OE all ship TMs

Set the profile Concurrent:Wait for Available TM to 1 (second)


Set the sleep time on the TMs to a high number (e.g. 10 minutes)
Setting timeouts too low may return a TIMEOUT before completion
But the job then completes

Tracing: set 'Initialization SQL - Custom


fnd_transaction.synchronous(rpc_timeout,:2,:3,'PO','RCVTPO'.
WAIT #420: nam='pipe get' ela= 2929821 handle address=29374777744 buffer
length=4096 timeout=172800 obj#=4292320 tim=4590000269038

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager

Time

Concurrent
Requests

No W
ork

At least 1 job

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Worker 1
Lock?

Job

A
E
B

Job

Run

SelectARunnable
EB

Update
Set
Running
Select
for
Update
Set
AE
Complete
Select
AUpdate

Job

Run

SelectARunnable
B

A
B

Select for
Update

SelectSet
forBUpdate
Update
Running

Job

Run

SelectC
Runnable
Select
ARunnable
B
D

C
A
D
B

Select
SelectSet
for
forCUpdate
Update
Update
Running

Job

Run

ARunnable
B
C
D
SelectNone

Worker 3
Lock?

X
Worker 4
Lock?

XX
A
C
Sleep
X
B
D

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

A
B

Worker 2
Lock?

FND_CONCURRENT_REQUESTS

Select
for Update

Select for
Select
for Update
Update

Update
Set
DUpdate
Running
Select
for
Update
Select
for
Update
Set
D
Complete

Status

Running
Complete
Running
Running
Complete
Running
Running

D
E

Concurrent Manager
Workload Management Strategy
Fast
Medium
Slow

Avg. Duration
10s
5 mins
30 mins
2 hours +
N/A

AvoidLong
short sleep times

Workers
Lots
A few
Not many
Very few
?

Sleep
15 - 20
60
60
60
30 or 60

Cache
Enough*
2xworkers
= 2xworkers**
= workers**
?

Critical for a couple of minutes of work


*Enough
**Set cache size to at least twice the number of workers

Specialization rules and work shifts


Bind specific jobs to specific time windows
Profile Option: Concurrent: Active Request Limit
Specialize if there are too many jobs in a specific category
9

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
How do you know your strategy needs review?

More workers = more accesses


9301929 (fix 12.1.4) modifies fnd_concurrent_requests_u2
Adds program_application_id and concurrent_program_id
10

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Implementation Example: The Problem
11.5.10
4 Node RAC: 40 CPUs per node
5,000 concurrent users
In addition to standard processing
400,000 concurrent requests per day (24,000 per hour peak)
Typically run times 2 ~ 3 seconds

Using Fast, Medium and Slow strategy


CPU running at 85% on all nodes and jobs were queuing

11

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Implementation Example: The Workload
4 Specialized Managers
15 Workers, Cache 30, Sleep15s
24,000 jobs/hr = 400 jobs/worker/hr = ~800~1200 secs/hr/worker

4 Managers/15 Workers = 1 manager/60 workers

12

Each scan caches 30 jobs = 60~90 secs work


Divide that by 60 processes avg. 1 sec work/worker
Add back the other managers
Scan 2+ million row table 2 3 times/second

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Implementation Example: The Strategy
Get enough work for a few minutes of work
Number

Manager

Workers

Cache

Sleep

Description

Fast

30

100

15

Jobs < 2 mins

Intermediate

15

70

60

Jobs >2 <40 mins

Slow

20

180

Jobs > 40 mins

4
3

Very Fast
Standard managers

30
30

400
20

10
60

Fast specialized < 3secs


Mixed workload

Other specialized

50

60

Other Specialized jobs

20

Other (specialized)

50

Various

Customer still had things to work out (in red)


CPU Improvement: 85% => 25% with no backlog

13

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Affinity
Goal: Direct some workload components to specific RAC
nodes to maximize scalability by minimizing inter-node
communication and synchronization work
Put parent and all child processes on the same RAC node
RAC/PCP Doc ID1359612.1

May cause an overload one node


Decision: load balance or affinitize
Can also happen on Failover
Recommended: set the profile option Concurrent: PCP Instance
Check to OFF
14

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Affinity
PCP: 1 1 correspondence of CM nodes to RAC nodes
Set <s_cp_twotask> per CM node to tie to a specific RAC instance
Define primary/secondary CM nodes for failover

Use dedicated queues


Use specialization rules for programs with high scalability needs
Group by application, program, user, request_type; examples:
OM, Receivables Super User, Create Accounting
WF background engines Node to ITEM_TYPE assignment

12.1 define node affinity at the program level


Concurrent | Program | Define ( See Doc ID 1129203.1)
15

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Extending Affinity to the Applications Tier
Forms
Set Profile Option "Database Instance" at App or Resp level
which can be tied to two_task value or to a service

Self Service
Set Profile Options App%Agent to web tier hosts configured
for specific services or
Set Profile Option "Applications Database ID" to node specific
DBC file name

16

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Affinity Examples
16TB DB 4 Node Cluster, R12, Oracle 11.1.0.7
15K concurrent peak users
Node 1

Node 3

Order Management
Node 2

Install Base
Shipping

Payroll

Education

HR (with self-service)

Contracts

Benefits
OTL (self-service)

17

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Financials
GL, AP, PO, FA
Self-Service
iProc, Expenses
DBI

Node 4
Accounts Receivable
Project Accounting
Imaging

Concurrent Manager
Affinity Examples
Large AR Telco system: Batch Affinity
Node 1: AutoInvoice,Create Accounting-Invoices
Node 2: AutoReceipt, Remittance, Create Accounting- Receipts

Order Management Benchmark: Batch Affinity


Node 1: Order Management, Shipping, WF background- OE Item Type
Node 2: Financials, Procurement, WF for PO item types

Incentive Comp system: No Affinity

18

4 node cluster, commodity HW; nodes too small to handle required OIC batch
workload in one node
Scaled to customers needs with RAC to required throughputs, BUT
Needed custom partitioning, and several development and custom code
optimizations, index changes/drops and other tuning

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Interconnect Messaging Performance Instance Level
Baseline & Monitor AWR
Global Cache Load Profile: Estimated Interconnect Traffic
Global Cache and Enqueue Services - Messaging Statistics
Average message sent queue time on ksxp
IPC efficiency > 2ms = interconnect/CPU capacity/issue
Eg) Network Traffic received
= 232.31*8k=1.9 Mb/sec

19

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Interconnect Messaging Performance Instance Level

E-Business Suite Technology-Related Sessions at OpenWorld 2011


https://blogs.oracle.com/stevenChan/entry/e_business_suite_technology_related
20

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Interconnect Messaging Performance Statement Level

21

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Enhancing The Concurrent Manager (Doc ID 164085.1)
Much of the functionality has now been built into 12.1
Procedure: How to get the REQUEST_ID into a shell script

22

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Tracing
Program Level
Not recommended
Enable the trace/profiling option in the concurrent program definition

Request Level
Ensure Concurrent: Allow Debugging is set to Yes
Use the Debug Options button in the Request Submission Form

Tracing Documents
280391.1 How to Retrieve SID Information For a Running Request

23

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Concurrent Manager
Useful References
Doc ID
1359612.1
1367676.1
164085.1
1057802.1
1304305.1

24

Document
Webcast 'E-Business Suite - RAC & Parallel Concurrent Processing
Webcast 'E-Business Suite - Concurrent Manager Performance
Enhancing and Automating Oracle Applications Concurrent Processing
Best Practices for Performance for Concurrent Managers
E-Business Concurrent Processing Information Center

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Agenda
Tuning the Concurrent Manager
Tracing

25

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

<Insert
<Insert Picture
Picture Here>
Here>

Tracing
General Points
Good idea to create a user called Trace
Collecting Diagnostic Data for Performance Issues in Oracle EBusiness Suite - Doc ID 1121043.1
Enhanced Performance Diagnostics for Oracle E-Business Suite
Release 12.1.3 Doc ID 1362660.1

For 11g apply tracing related patches


7522002, 8342329, 11687336, and 8328200

SQL Trace Level 8 (with waits)


Raw trace and TKPROF
26

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Catch-All
Forms: Help Diagnostics Trace
OA HTML: Diagnostics Tracing
Catch-All
Set the profile: Initialization SQL Statement - Custom"
BEGIN
FND_CTL.FND_SESS_CTL('','','TRUE','TRUE','','ALTER SESSION
SET TRACEFILE_IDENTIFIER=''TESTING''
MAX_DUMP_FILE_SIZE=unlimited STATISTICS_LEVEL = ALL
EVENTS=''10046 TRACE NAME CONTEXT FOREVER, LEVEL 8''');
END;

27

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Parameters
max_dump_file_size=unlimited
Ensure there is sufficient disk space in the tracing directory

user_dump_dest or if diagnostic_dest is set in Oracle11g


(diagnostic_dest)/diag/rdbms/(dbname)/(instname)/trace
background_dump_dest for parallel slave processes

STATISTICS_LEVEL=ALL

28

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Plan Completeness
Run TKPROF from the correct Oracle home
Use the sort options = fchela,exeela,prsela

Checks
Check file contains runtime execution plans and row source stats
Check the last line: elapsed seconds in trace file

29

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Plan Completeness
Rows
------0
1
1
0
0

Row Source Operation Actual runtime plan needed for diagnosis


--------------------------------------------------NESTED LOOPS (cr=5 pr=0 pw=0 time=0 us cost=4 size=71 card=1)
TABLE ACCESS BY INDEX ROWID FND_PROFILE_OPTIONS (cr=3 pr=0 pw=0 time=0 us cost=2 size=44 card=1)
INDEX UNIQUE SCAN FND_PROFILE_OPTIONS_U2 (cr=2 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 215619)
TABLE ACCESS BY INDEX ROWID FND_PROFILE_OPTION_VALUES (cr=2 pr=0 pw=0 time=0 us cost=2 size=27 card=1)
INDEX RANGE SCAN FND_PROFILE_OPTION_VALUES_U1 (cr=2 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 224902)

Rows
------0
0
1
1
0
0

Explain option == explain plan for on SQL*Plus


Execution Plan
--------------------------------------------------SELECT STATEMENT
MODE: ALL_ROWS
NESTED LOOPS
TABLE ACCESS
MODE: ANALYZED (BY INDEX ROWID) OF 'FND_PROFILE_OPTIONS' (TABLE)
INDEX
MODE: ANALYZED (UNIQUE SCAN) OF 'FND_PROFILE_OPTIONS_XX_1' (INDEX (UNIQUE))
TABLE ACCESS
MODE: ANALYZED (BY INDEX ROWID) OF 'FND_PROFILE_OPTION_VALUES' (TABLE)
INDEX
MODE: ANALYZED (RANGE SCAN) OF 'FND_PROFILE_OPTION_VALUES_U1' (INDEX (UNIQUE))

New tracing option: plan_stat=adaptive (patch 8328200)


Sampled statistics are dumped for every minute of DB time
30

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Event 10046
Trace Analyzer (TRCA)
Doc ID 224270.1
Reads an Event 10046 SQL Trace file
Provides a comprehensive report for performance analysis and
tuning

Interpreting Raw SQL_TRACE files Doc ID 39817.1

31

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
SQLT XTRACT for Poorly Performing SQL
SQLTXPLAIN (aka SQLT)
Maintained by Oracle Support - often requested
Analyses why the execution plan was chosen

Always use the latest version


Doc ID 215187.1

XTRACT report is preferred output


Focuses on actual SQL execution runtime data

32

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
SQLT XTRACT for Poorly Performing SQL

33

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
SQLT XTRACT - Completeness
With
Statistics_level = ALL

34

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Enhanced Tracing in R12.1.3
Doc ID 1362660.1/Patch 12544073
Trace a session
Run Gather Diagnostic Data for Traced Sessions

AWR and ADDM Reports


ASH Report
Session Statistics
Trace Analyzer output, which includes SQLT output for top SQLs
Object statistics when a SQL statement is traced, but SQLT output is not chosen or not installed
SQLT output (requires SQLT tool Doc ID 215187.1
Real-Time SQL Monitor Report when a SQL statement is traced (in Oracle Database 11g and higher)
SQL Detail Report when a SQL Statement is traced (in Oracle Database 11gR2 and higher)

Also use to capture information for individual SQL(s)


35

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
System Level Performance Information
AWR (Automatic Workload Repository) Report
ADDM Report
ASH (Active Session History) Report
Real-Time SQL Monitoring Report (11g+)
Parallel queries and queries taking > 5 seconds

Operating system statistics


OSW Document Doc ID 301137.1
OSWg Document Doc ID 461053.1

36

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Real-Time SQL Monitoring Report

37

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Takeaway
AWR is to an entire SYSTEM
TRCA is to a PROCESS
SQLT is to ONE SQL

ASH
Session, Module, SQL etc

Process
SQLTrace followed by TKPROF and/or TRCA

ONE SQL
SQLT using XTRACT method
38

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Tracing
Useful References

39

Doc ID

Document

1121043.1

Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite

1362660.1

Enhanced Performance Diagnostics for Oracle E-Business Suite Release 12.1.3

301137.1

OSW Document

461053.1

OSWg Document

215187.1

SQLTXPLAIN (SQLT)

224270.1

Trace Analyzer (TRCA)

39817.1

Interpreting Raw SQL_TRACE files

280391.1

How to Retrieve SID Information For a Running Request

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

PC and Network Performance


March 22nd
PC
Windows 7 vs Windows XP
Longevity of old clients

Network
Real world network maps
WAN emulation
WAN and VSAT optimization

40

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Q&A

41

Copyright 2012, Oracle and/or its affiliates. All rights reserved.