1

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

E-Business Suite Release 12.1 Upgrade Best Practices Technical Insight
Isam Alyousfi
Senior Director, Applications Performance

Samer Barakat
Senior Architect, Applications Performance

2

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.

3

3 Copyright © Oracle 2012, Oracle its affiliates. All reserved. rights Copyright © 2012, and/or and/or its affiliates. All rights reserved.

<Insert Here>

Introduction

4

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

All rights reserved. Oracle and/or its affiliates.Introduction Context within Performance Management Configuration Requirements & Best Practices Operational Best Practices Patching & Upgrades Performance Testing & Capacity Planning Workload Management Performance Triage & Resolution Performance Monitoring & Trending People Management 5 Copyright © 2012. .

Agenda • Introduction to Performance Triage Approach • Tuning Application Modules • Tuning the Concurrent Manager • Tuning the Database Tier 6 Copyright © 2012. Oracle and/or its affiliates. All rights reserved. .

Performance Triage 7 Copyright © 2012. Oracle and/or its affiliates. All rights reserved. .

All rights reserved. Oracle and/or its affiliates. . possibly gather additional data  Search for a KNOWN SOLUTION or workaround that addresses the root cause of the problem  If it is a product issue.Performance Triage & Resolution How to Approach a Performance Issue  DEFINE the problem clearly  GATHER the right data to analyze the issue  Identify the ROOT CAUSE of the problem. PASS ON the right information to support/dev through the regular channels  Try to identify a TEMPORARY WORKAROUND to alleviate the issue while you get a permanent fix 8 Copyright © 2012.

Oracle and/or its affiliates. . All rights reserved.Performance Triage & Resolution Define the Problem  Get a clear understanding & quantitative definition of the issue  Where is the time going ? Get the right diagnostics – What? – Where? – When? – Why? – How?  Identify the Techstack components in-play  Refer to the architecture 9 Copyright © 2012.

All rights reserved. .Performance Triage & Resolution Gather  ~ 80% of issues are DB processing related  If non-database (client side) processing occurs within the flow. Oracle and/or its affiliates. you may see DB wait events like SQL*Net Message from client TCP Socket% PLSQL Lock Timer  Keep a watchful eye on OS metrics and resource consumption in all 3 tiers 10 Copyright © 2012.

278868.1) • FRD Log (mos: 445166.1) • Middletier Tuning • JVM Logs • JVM Sizing/Tuning (mos: 362851.OSWatcher (mos: 301137.1) • ADDM report (mos: 250655.1) 11 Copyright © 2012.1) • Database Tuning • AWR Report (mos: 748642.1) • Reports Tracing (mos: 111311. All rights reserved.1.1) • 11g SQL Monitor Report • AWR SQL Report (awrsqrpt.1) • Trace Analyzer (mos: 224270.1) • AWR Report (mos: 748642. Oracle and/or its affiliates.sql) • PL/SQL Tuning • Product logs • PL/SQL Profiler (mos: 808005.1) • OS .1 for How-to & Best Practices • SQL Tuning • Trace files • SQLT output (mos: 215187.1) • Generic note (mos: 438652.Performance Triage & Resolution Gather Diagnostic Data Start with MOS Doc 1121043.1) • Active Session History (ASH) • Forms Tuning • Forms Tracing (mos: 373548. .

All rights reserved.1) •Gather as much diagnostic data as possible in one round trip to customer.3 as a standalone patch# 12544073 (Mandatory Pre-Requisite AD Patch# 12991557) •Enhancement to FND_TRACE : New procedure “set_preferences” enables you to •Enable tracing for sessions •Enable tracing for one or more SQL statements using SQL_ID (*) •Generate CBO trace when tracing SQL statements (*) •Trace PL/SQL using PL/SQL Hierarchical Profiler (*) •Set Statistics Level (*) .Performance Triage & Resolution Gather Enhanced Performance Diagnostics for Oracle E-Business Suite (mos: 1362660. •Available on 12.Only in Oracle Database 11g+ 12 Copyright © 2012. Oracle and/or its affiliates. .1.

1) •One or more of the following outputs can generated automatically after the end of traced session.1.1+ . 13 Copyright © 2012. (** )Oracle DB 11g+ (*** )Oracle DB 11. Oracle and/or its affiliates.2. but SQLT output is not chosen or not installed •Real-Time SQL Monitor Report when a SQL statement is traced (**) •SQL Detail Report when a SQL Statement is traced (***) (* )Requires SQLT tool .0. •AWR. All rights reserved. The concurrent program 'Gather Diagnostic Data for Traced Sessions' is automatically submitted via 'System Administrator responsibility to gather this output.Note 215187. ADDM and ASH Reports •Session Statistcs •SQLT output (*) •Trace Analyzer output. which includes SQLT output for top SQLs •Objects statistics when a SQL statement is traced.Performance Triage & Resolution Gather Enhanced Performance Diagnostics for Oracle E-Business Suite (mos: 1362660.

. Oracle and/or its affiliates.Performance Triage & Resolution 14 Copyright © 2012. All rights reserved.

All rights reserved.ora : Notes 396009.Performance Triage & Resolution  Check for Known Issues & Available Fixes/Workarounds – MOS  Check the basics – Version levels – Required Configuration » Init. .oracle.com/stevenChan/ 15 Copyright © 2012.1 (11i) – Latest patch levels – Recommended Patch lists in MOS .1 (R12) & 216205. Oracle and/or its affiliates. OAM – Also check for product specific notes  Best Practices in Performance for EBS – http://blogs.

Oracle and/or its affiliates.Tuning the Applications 16 Copyright © 2012. . All rights reserved.

Oracle and/or its affiliates. All rights reserved. .0 .3: Patch 9239090 : MOS Doc: 1080973.1)  OAM Patch Wizard – “Recommended Performance Patches for the Oracle E-Business Suite”.RUP6: Patch 6728000 : MOS Doc: 743368.1  Recommended performance patches for all the modules and tech stack components are consolidated in this note 17 Copyright © 2012.1  Release 12.Tuning the Applications Patch Current – Apply latest Release Update Packs  Release 12. MOS Doc 244040.1 – Apply the latest EBS Family Packs + Recommended Patch List  Patching & Maintenance Advisor: (EBS) 11i and R12 (MOS Doc 313.1.

for R12 => Database 11.2.5. JDK 6.oracle.html 18 Copyright © 2012.0.Tuning the Applications Stay Current – Upgrade techstack components to the latest certified levels For example (as of Aug/2012).3.1. etc  http://blogs.com/stevenChan/resource/certifications. OC4J 10.1.2.3. . Forms 10.3. All rights reserved. Oracle and/or its affiliates.

All rights reserved. Oracle and/or its affiliates. initiated and monitored » Set the execution frequency as well view history of purge programs – R12: programs tagged with the “Purge” program type 19 Copyright © 2012.Tuning the Applications Performance Best Practices – Purge & Archive  Implement a regular purge process including transient data as well as obsolete transactional data  Purge old data before performing upgrades  Many documented standard programs available – ~260 purge programs in R12  Purge Portal – Purge programs can be configured. .

Oracle and/or its affiliates. .Tuning the Applications System Administrator > Oracle Applications Manager >Purging/Critical Activities 20 Copyright © 2012. All rights reserved.

there could be exceptions and errors which need to be investigated 21 Copyright © 2012. Oracle and/or its affiliates. .Set to Unexpected (Level=6)  Query FND_LOG_MESSAGES to check the logging traffic  With above profile configuration – If the query shows a large number of rows being created.0 RUP3+.Tuning the Applications Performance Best Practices – Logging  FND: Debug Log Enabled .Set it to ‘Yes’ (12. All rights reserved.x)  FND: Debug Log Level .1. 12.

set the ‘Process Stuck’ parameter to ‘No’ – Identification of stuck workflows is resource intensive – Start a separate background engine to handle stuck/timed out processes with a low frequency –i. All rights reserved. PO Document Approval.  Purge runtime data (Conc Program: FNDWFPR) – Can run by ITEM_TYPE – Monitor/Troubleshoot old workflows not properly closed – Purge frequently 22 Copyright © 2012. once a day  Use deferred activities to improve online response times for flows such as Scheduling. Oracle and/or its affiliates. etc. .e.Tuning the Applications Performance Best Practices – Workflow  For background engines via Concurrent Manager.

Tuning the Applications Performance Best Practices – Workflow  Run $FND_TOP/sql/wffngen.ALTER_QUEUE(queue_name=>:b1.sql to translate the activity function calls into static calls – Edit & add heavily used ITEM_TYPES itemtypeList_t (‘WFSTD’.pls in utl_file_dir to re-create pkg  Disable retention on Workflow Queues – DBMS_AQADM. All rights reserved.1  For high volume batch processing in RAC use ITEM_TYPE to Node affinity 23 Copyright © 2012.retention_time=>0).’FNDFFWF’) – Generates wffncal2b. . Oracle and/or its affiliates.  Partition runtime tables for higher scalability – MOS Doc: 260884.

Oracle and/or its affiliates. . All rights reserved. reducing WF data significantly  Use Deferred tax & credit check  For online booking speed.1 & 130511.1  Use streamlined workflows – Consider seeded workflow: “Line Flow Generic: Performance” – Removes unnecessary activities and sub-processes. consider deferring scheduling –MOS Doc 421314.Tuning the Applications Performance Best Practices – Order Management  R12 OM Performance Fixes & Configuration Options to speed up processing: MOS Docs 849060.1  Configurator should be setup to use a separate pool of JVMs 24 Copyright © 2012.

set to Request Viewer Off –WSH: Debug Enabled. All rights reserved. Oracle and/or its affiliates. set to 0 for OFF  Set “MRP: Calculate Supply Demand” to No –Only enable for a user session if detailed supply / demand data needs to be reviewed in an ATP Inquiry 25 Copyright © 2012. use the Quick Sales Order Form – OEXOETEL  Disable Debugging –OM: Debug Level.Tuning the Applications Performance Best Practices – Order Management  For online order entry. set to 0 for OFF –QP: Debug Mode. . set to No –WSH: Debug Level.

Tuning the Applications Performance Best Practices – Financials  Account Analysis & Journal Entries Report – MOS Doc 983063. Processing Unit Size & Event Class ordering – Payables specific fixes in MOS Doc 1271365. Oracle and/or its affiliates. .MOS Doc 791049.1 – Troubleshoot & Tune XML generation – BI Publisher – Recommended Performance Fixes – Debug. Scalable Flags and Memory setting recommendations –OOM Issues  Subledger Accounting & Accounting Hub .1 26 Copyright © 2012. All rights reserved.1 – Recommended Performance Fixes – Custom Indexing guidelines – Optimize # of Processors.

. Oracle and/or its affiliates. AutoReceipts/Remittance.Tuning the Applications Performance Best Practices – Financials AR  Profile Option : AR: AutoInvoice Gather Statistics – Set to "NO" once representative stats have been gathered for RA_INTERFACE* tables  Index trx flexfields for headers & lines used in validation –interface_line_attributeN columns  Parallelize Batch flow – By supplied parameters where possible » e. All rights reserved.g. Rev Rec. Autoinvoice – Manually by data sets if needed » e.g. Auto-accounting (R12) 27 Copyright © 2012.

Tuning the Applications
Runtime Performance Testing Tips

Use Automated, scripted tools
 EBS Test Started Kits (Winrunner/QTP) – Bundled QA based automated scripts for EBS testing - Patch 8408886  Oracle Applications Testing Suite (Accelerators for EBS) – Web and Forms based flows

– Complement with user participation tests and batch load tests with frequent and critical

jobs
– References

http://blogs.oracle.com/stevenChan/2009/10/oats_ebs_certified.html http://blogs.oracle.com/stevenChan/2009/08/evolutionary_steps_automated_testing_ebs.html
28
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Tuning the Concurrent Manager

29

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Tuning the Concurrent Manager

CM General Best Practices Workload Management Strategy CM Configuration Best Practices Parallel Concurrent Processing (PCP) Best Practices Concurrent Manager Tracing Best Practices

30

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Tuning the Concurrent Manager
General Best Practices
– Manage CM Jobs Lifecycle: 50% of performance tuning is in the business!
   

Review long-running auto resubmitted jobs Review short/long-running jobs – selective parameters? Don’t purge jobs that the users then have to run again Trim the fat: Review and eliminate concurrent jobs that are not required and/or are not being used by users.

– Manage CM Managers : More managers/workers != more throughput
  

Avoid enabling an excessive number of standard or specialized managers.
A common guideline is between 1-2 target process per CPU, but this needs to be balanced with resources required by online activity Utilize Parallel Concurrent Processing (PCP) to leverage the Cluster (RAC) .

31

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

log on fast disks.out and *. . All rights reserved. Watch for 2GB limit on any output files (Doc ID 842850. Avoid scheduling resource intensive batch requests during peak activity.Tuning the Concurrent Manager General Best Practices – Manage CM State Data:   Purge the FND tables on a regular basis using the “Purge Concurrent Request and Manager Data” program Patch 7530490 for R12 or 7834670 for 11i (MOS Doc ID 822368. – Manage CM Schedules:     32 Copyright © 2012.log file in log directory. Use Separate disks if throughput seems slow especially on NFS Filers Truncate the reports.1)   High throughput: Keep *. Oracle and/or its affiliates. Define Workload Management Strategy based on job average duration and system usage profile.1) Use specialization rules and work shifts to bind specific jobs to specific time windows . Reschedule some programs to run when the concurrent managers have excess capacity.

Oracle and/or its affiliates.Tuning the Concurrent Manager Workload Management Strategy – • • – • Avoid Short Sleep Times *Enough for a couple of minutes of work **Set cache size to at least twice the number of workers Specialization rules and work shifts Bind specific jobs to specific time windows • • 33 Profile Option: Concurrent: Active Request Limit can be used restrict the number of concurrent requests that may be run simultaneously by each user Specialize if there are too many jobs in a specific category Copyright © 2012. . All rights reserved.

Payroll). Auto Invoice. consider reducing the sleep time of the Conflict Resolution Manager (CRM) Default is 60s.Tuning the Concurrent Manager Configuration Best Practices – Conflict Resolution Manager Sleep Time:  To maximize throughput for jobs which spawn parallel workers (i. . Oracle and/or its affiliates. All rights reserved. this avoids constant polls to check for shutdown requests. The profile sets the total time to wait for a TM before switchover to next available TM Set Sleep time on Transaction Managers to 30 minutes.e. consider 5 or 10 seconds  – Concurrent Manager Sleep Time:  Define dedicated queues for short and frequent requests and increase the sleep times for managers which do not require near real-time job execution Reducing sleep time to a very low value many cause excessive CPU utilization. Set the profile “Concurrent:Wait for Available TM” to 1 (second) to minimize TM latency.  – Transaction Manager Sleep Time:   34 Copyright © 2012.

. Tune the sleep time for queues with a large number of target processes. Oracle and/or its affiliates. 35 Copyright © 2012. The cache size specifies the number of requests that will be cached each time the concurrent manager reads from the FND_CONCURRENT_REQUESTS table – Cache Size:    Increase the cache size (number of requests cached) to at least twice the number of target processes Increasing the cache size will boost the throughput of the managers by attempting to avoid sleep time. All rights reserved.Tuning the Concurrent Manager Configuration Best Practices – CM Queues :    Consider dedicating certain concurrent managers to process either short or long running programs to avoid queue backup Add more queues if all queues are running at maximum capacity only if resources permit.

1) AQ is largely self-tuning 36 Copyright © 2012.     Set System Profile: “Concurrent:TM Transport Type”: to either Pipe/Queue Use queues in RAC for easier setup while supporting Failover. . All rights reserved.1) For R12 (or 11i post Oracle 10. Oracle and/or its affiliates.Tuning the Concurrent Manager Configuration Best Practices –   Transaction Managers (Pipes vs Advanced Queue) : Transaction Managers are used for synchronous online processing (ex:Inventory Transactions) Transaction Managers can use Pipes or queues (AQ). Queues are Available in 11i (Doc ID 413636.

program.    For RAC and PCP refer to MOS Doc ID1359612. All rights reserved. Receivables Super User. Oracle and/or its affiliates. . examples: – –  OM. user. request_type.Tuning the Concurrent Manager Parallel Concurrent Processing (PCP) Best Practices –   Utilize Parallel Concurrent Processing (PCP) to leverage the Cluster (RAC).1 Recommended: set the profile option ‘Concurrent: PCP Instance Check ‘ to OFF Configure Multiple Concurrent Manager Instances. Each CM Instance points to a specific DB instance 37 Copyright © 2012. Use dedicated CM queues with specialization rules (inclusion/exclusion) for programs with high scalability needs Group by application. Create Accounting WF background engines Node to ITEM_TYPE assignment Use Affinity : Direct some workload components to specific RAC nodes to maximize scalability by minimizing inter-node communication and synchronization work.

1) 12. .1. Define primary/secondary CM nodes for Failover –   12.1 define node affinity at the program level Concurrent | Program | Define ( See MOS Doc ID 1129203.Tuning the Concurrent Manager Parallel Concurrent Processing (PCP) Best Practices –    PCP: 1 – 1 correspondence of CM nodes to RAC nodes Set <s_cp_twotask> per CM node to tie to a specific RAC instance. All rights reserved. Oracle and/or its affiliates. The request may hang if the sessions are load balanced.3 introduced option to define node affinity at the program level – Concurrent -> Program -> Define 38 Copyright © 2012. Do not use a load balanced TNS entry for the value of s_cp_twotask.

Tuning the Concurrent Manager Parallel Concurrent Processing (PCP) Best Practices –  Extending Affinity to the Applications Tier (Forms) Set Profile Option "Database Instance" at App or Resp level which can be tied to two_task value or to a service –   Extending Affinity to the Applications Tier (Self-service) Set Profile Options “App%Agent” to web tier hosts configured for specific services …or Set Profile Option "Applications Database ID" to node specific DBC file name 39 Copyright © 2012. Oracle and/or its affiliates. All rights reserved. .

FA Self-Service – Shipping Education Contracts Payroll HR (with self-service) Benefits OTL (self-service) Node 4 Accounts Receivable Project Accounting Imaging iProc. AP.0. 11. All rights reserved. .Tuning the Concurrent Manager Parallel Concurrent Processing (PCP) Best Practices Application Affinity Example • 16TB DB 4 Node Cluster. Oracle and/or its affiliates.1.7. PO. Expenses DBI 40 Copyright © 2012. 15K concurrent peak users Node 1 Order Management Install Base Node 2 Node 3 Financials – GL.

Tuning the Concurrent Manager Concurrent Manager Tracing Best Practices –   Program Level Not recommended Enable the trace/profiling option in the concurrent program definition –   Request Level Ensure “Concurrent: Allow Debugging” is set to ‘Yes’ Use the Debug Options button in the Request Submission Form –  Tracing Documents 280391. All rights reserved. .1 How to Retrieve SID Information For a Running Request 41 Copyright © 2012. Oracle and/or its affiliates.

1 Best Practices for Performance for Concurrent Managers • 1304305.1 Enhancing and Automating Oracle Applications Concurrent Processing 1057802.RAC & Parallel Concurrent Processing 1367676.1 E-Business Concurrent Processing Information Center 42 Copyright © 2012. All rights reserved.1 Webcast 'E-Business Suite . Oracle and/or its affiliates. .1 Webcast 'E-Business Suite .Concurrent Manager Performance 164085.Tuning the Concurrent Manager References (MOS Docs) • • • • 1359612.

Oracle and/or its affiliates.Tuning the Database 43 Copyright © 2012. . All rights reserved.

. All rights reserved. Oracle and/or its affiliates.Tuning the Database Tier Configuration Best Practices CBO – Statistics Gathering Performance Management Common corrective actions to maximize scalability 11g New Features 44 Copyright © 2012.

Oracle and/or its affiliates. All rights reserved. – MOS Docs 396009.1 (R12 & 11i respectively)  Apply Required Patches & Fixes/Workarounds for Known Issues – Recommended Performance Patches -MOS Doc 244040.1 & 216205.ora parameters are set correctly.1 45 Copyright © 2012.ora Parameters: – Ensure mandatory init.1 – Required Patching – Interoperability Notes »11gR1 Certified with 11i and R12 – MOS Docs 452783.1 & 735276.1 »11gR2 Certified with 11i – MOS Doc 881505. .Tuning the Database Tier Configuration Best Practices  Tune Init.

For example on Linux HugePages – MOS Doc 744769. . 46 Copyright © 2012. so more SGA memory will be needed.1 – Only SGA allocation benefits from large/huge pages. Oracle and/or its affiliates. All rights reserved. and much less PGA will be required. PGA allocation does not.ora parameter) »For shared server processes. – Consider both SGA and PGA memory requirement when sizing your RAM »For dedicated server processes plan for PGA allocation of 30-40MB per shadow process (correlated to the process init. user connections allocate almost all of their memory from the SGA.Tuning the Database Tier Configuration Best Practices  Properly Size SGA/PGA Memory : – Use Large/Huge pages.

CALIBRATE_IO – MOS Doc 727062.A. S. Oracle and/or its affiliates.M.Tuning the Database Tier Configuration Best Practices  Tune Disk IO: – Ensure kernel asynchronous I/O is enabled and supported for the underlying file system (if file systems are being used for the DB files) – Configure & verify direct IO is used (if file systems are being used for the DB files) »Improves performance/scalability by simulating raw devices. avoiding file system cache traffic – Ensure your IO subsystem can handle your peak IO load & IOPS »Consider RAID. configuration with sufficient spindles »Measure with ORION tool or in 11g DBMS_RESOURCE_MANAGER.1 – Reduce the IO workload »Tune expensive SQL »Trim the workload of unnecessary tasks »Do 3X less IO on average with 11g Advanced Compression Option 47 Copyright © 2012. .E. All rights reserved.

Migration utility can be used to migrate a schema at a time 48 Copyright © 2012. All rights reserved. This is achieved by supporting locally managed tablespaces as opposed to the dictionary managed tablespace OATM Helps avoid frequent space management operations.Tuning the Database Tier Configuration Best Practices  Convert to the OATM Tablespace Model – OATM Consolidates # of Apps tablespaces from 400 to ~12  MOS Doc 248857. OATM supports uniform extent allocation and auto allocate extent (AUTOALLOCATE ) management OATM provides efficient space utilization. Oracle and/or its affiliates. .1 – OATM Uses automatic segment management (ASSM) along with uniform extent – – – – sizes for tablespaces containing large transaction tables/indexes.

.sid and oc. 49 Copyright © 2012.sid.oc.Tuning the Database Tier Configuration Best Practices  Tune Backend Code Execution Path – Utilize PL/SQL Native Compilation.sid.hash_value.oc.count(*) from v$open_cursor oc. – Check for cursor leaks select oc.sql_text not like '%COMMIT%' group by oc.oc. v$session ss where oc.sql_text having count(*) > 5 order by 4.sid = ss. All rights reserved. – Pin Top Apps PL/SQL packages by executions helps avoid shared pool fragmentation and reloads.hash_value. Oracle and/or its affiliates.sql_text.oc.

.PARALLEL_FORCE_LOCAL =true – Tune sequence cache sizes for hot sequences (e. – On RAC ensure PX requests do not span instances  10g .g.Set instance specific values for INSTANCE_GROUPS.Tuning the Database Tier Configuration Best Practices  Tune Backend Code Execution Path – Check for SQL statements using literals with a large number of occurrences. PARALLEL_INSTANCE_GROUP  11gR1 .000 or higher) in order to minimize index key contention for the Apps surrogate keys. – Cursor leaks/literals can result in ORA-4031 errors.Can be done as above or via services  11gR2 . Oracle and/or its affiliates. 50 Copyright © 2012. 1. All rights reserved. – Make sure to use FNS_STATS to gather statistics.

– When Gather Schema Statistics concurrent program is used fnd_stats does the bookkeeping for the run. . Oracle and/or its affiliates.Tuning the Database Tier CBO – Statistics Gathering  Disable the 10g/11g automatic job to gather stats  Do not gather statistics excessively on entire schemas or the entire database such as nightly or weekly  Use the GATHER_AUTO option to gather incrementally  Use only FND_STATS or the Gather Schema / Table Statistics Concurrent Programs – Do NOT USE the analyze or dbms_stats command directly. 51 Copyright © 2012. It is not supported. and can result in sub-optimal plans. All rights reserved.

 Analyze all schemas at 10%. Oracle and/or its affiliates. can use value of zero (AUTO_SAMPLE_SIZE)  Consider locking stats for very volatile tables once a representative set is gathered – Some Interface tables – Other examples: ap_selected_invoices.load_histogram_cols.LOAD_XCLUD_TAB to skip the table 52 Copyright © 2012. – Histograms can be seeded by calling to fnd_stats. . some tables benefit from higher sampling % – For 11g. All rights reserved. then for some specific objects use higher sampling x% – Due to data skew. wsh_pr_workers – Use FND_STATS.Tuning the Database Tier CBO – Statistics Gathering  FND_STATS supports column histograms.

. FND_STATS Schema/Table stats will take care of the gathering stats on Multi-Column stats. The Multi-Column Statistics are used by Oracle to estimate the combined selectivity of the predicates. 53 Copyright © 2012.LOAD_EXTNSTATS_COLS.  11g Extended Stats FND_STATS Patch 9470196.  Multi-Column Stats can be seeded by calling FND_STATS. Also. All rights reserved.Tuning the Database Tier CBO – Statistics Gathering  FND_STATS now supports Database 11g Extended Stats. Oracle and/or its affiliates. in patch set 9962815.

Tuning the Database Tier Performance Management • Establish baselines for different workloads by collecting system level performance information – – – – AWR (Automatic Workload Repository) Report ADDM (Automatic Database Diagnostic Monitor) Report ASH (Active Session History) Report Real-Time SQL Monitoring Report (11g+) • Parallel queries and queries taking > 5 seconds – Operating system statistics • OSW Document Doc ID 301137. Oracle and/or its affiliates. .1 • OSWg Document Doc ID 461053. All rights reserved.1 54 Copyright © 2012.

Tuning the Database Tier Performance Management • Monitor and trace critical flows & concurrent programs – ASH (Active Session History) Report – SQL Trace Level 8 (with waits) or Level 12 (with waits and binds) followed by TKPROF • Check both raw trace and TKPROF • Interpreting Raw SQL_TRACE files MOS Doc ID 39817. Oracle and/or its affiliates.1 – Trace Analyzer (TRCA) reads an Event 10046 SQL Trace file and provides a comprehensive report for performance analysis and tuning MOS Doc ID 224270. All rights reserved. .1 • Maintained by Oracle Support .1 • Monitor and trace specific SQLs – SQLTXPLAIN (aka SQLT) MOS Doc ID 215187.often requested • Collects information on why the execution plan was chosen 55 Copyright © 2012.

Oracle and/or its affiliates. Use FND_CTL. .Custom“.Tuning the Database Tier Performance Management • Trace/TKPROF … Get the complete session story Set Trace Parameters – max_dump_file_size=unlimited • Ensure there is sufficient disk space in the tracing directory – user_dump_dest or if diagnostic_dest is set in Oracle11g • (diagnostic_dest)/diag/rdbms/(dbname)/(instname)/trace • background_dump_dest for parallel slave processes – STATISTICS_LEVEL=ALL Generate Trace – Forms: Help -> Diagnostics -> Trace – OA HTML: Diagnostics -> Tracing – Set the profile: “Initialization SQL Statement .FND_SESS_CTL 56 Copyright © 2012. All rights reserved.

1 (Also review Appendix B – Getting In-Memory partial runtime stats) 57 Copyright © 2012.Tuning the Database Tier Performance Management • Trace/TKPROF … Get the complete session story – Raw trace must be • A complete. . non-truncated trace file *** DUMP FILE SIZE IS LIMITED TO 12345 BYTES*** – From before the user action began To the point the DB session ends** – Run TKPROF from the correct Oracle home • Use the sort options = fchela.exeela. All rights reserved. detailed how-to steps & recommended tracing patches • MOS Doc 1121043. Oracle and/or its affiliates.prsela – TKPROF output MUST have • Runtime Execution Plans and Row Counts & Row Source Stats •Run Time Execution Plan = “Row Source Operation” != “Explain Plan” – Best practices.

---------0.00 0 0 0 780.79 728. All rights reserved.70 0 1 0 6328.78 6757.---------.---------.---------.52 225987 153846533 0 rows ---------0 0 7674758 ---------7674758 Misses in library cache during parse: 1 Misses in library cache during execute: 1 Optimizer mode: ALL_ROWS Parsing user id: 173 (recursive depth: 2) Rows Row Source Operation ------. 58 Copyright © 2012.00 0. 7675089 TABLE ACCESS BY INDEX ROWID RA_CUSTOMER_TRX_LINES_ALL (cr=30700356 pr=63178 pw=0 time=539908369 us) 7675089 INDEX UNIQUE SCAN RA_CUSTOMER_TRX_LINES_U1 (cr=23025267 pr=2820 pw=0 time=196155435 us)(object .58 7486.-------total 15349517 cpu elapsed disk query current -------..---------.---------.---------7009.---------..Tuning the Database Tier Performance Management call count -----------Parse 1 Execute 7674758 Fetch 7674758 ------. . Oracle and/or its affiliates.--------------------------------------------------7674758 SORT AGGREGATE (cr=153846532 pr=225987 pw=0 time=7174772779 us) 7675089 NESTED LOOPS (cr=153846532 pr=225987 pw=0 time=6919378560 us) 7675089 SORT UNIQUE (cr=123146176 pr=162809 pw=0 time=6083080435 us) 55878643 TABLE ACCESS BY INDEX ROWID AR_TRX_COGS_GT (cr=123146176 pr=162809 pw=0 time=5577872837 us) 136390387 INDEX RANGE SCAN AR_TRX_COGS_N1 (cr=23439707 pr=37706 pw=0 time=2136416980 us)(object id.80 225987 153846532 0 -------.

All rights reserved. Oracle and/or its affiliates. Monitor and correlate top SQL in AWR to business flows.sql Review the Advisory sections in AWR to fine tune SGA & PGA. 59 Copyright © 2012. . Review Load Profile and Top 5 Timed Events section to determine utilization & bottlenecks and review relevant detailed sections for these events.Tuning the Database Tier Performance Management • Regularly Review Automatic Workload Repository (AWR) data for different workloads – – – – – Use DBMS_WORKLOAD_REPOSITORY or Enterprise Manager to create snapshots Generate reports using $ORACLE_HOME/rdbms/admin/awrrpt.

row lock & ITL waits 60 Copyright © 2012. dynamic object creation/removal. • Review the latch Statistics section to determine the hot latches • Trace waiter and holder sessions to determine actual cause – For I/O related waits • Review SQL sections by Logical/Physical reads and the Segment Statistics sections by I/O – For Enqueue related or buffer busy waits • Review the following sections: enqueue. All rights reserved. etc. sub-optimal SQL which performs full table or full index scans.Tuning the Database Tier Performance Management • Regularly Review Automatic Workload Repository (AWR) data for different workloads – For Latch related waits • Often due to: non-sharable SQL. Oracle and/or its affiliates. . segment statistics by buffer busy waits.

Tuning the Database Tier Performance Management • RAC AWR Analysis – Verify SQL execution is optimized – 10046 trace/SQL Monitoring – Collect AWR data from all instances • awrgrpt. . Oracle and/or its affiliates.sql (cluster-wide base and diff reports) – Verify IO. CPU utilization and Interconnect messaging performance are within acceptable limits – Drill down on cluster wait events + “SQL order by Cluster Waits” – Correlate SQL executions on all instances accessing/modifying top segments in Segment Statistics by • Global Cache Buffer Busy & Buffer Busy Waits • CR/Current Blocks Received • Logical Reads 61 Copyright © 2012.sql and awrgdrpt. All rights reserved.

Tuning the Database Tier Performance Management with EM-DB Console 62 Copyright © 2012. . All rights reserved. Oracle and/or its affiliates.

. Oracle and/or its affiliates. All rights reserved.Tuning the Database Tier Performance Management with EM-DB Console 63 Copyright © 2012.

. log file sync. free buffer waits. 64 Copyright © 2012. – IO Sub-system:  RAID 10 still preferred most often for high end requirements Ideal avg.) – Tune Top SQL is always the first step – Make sure that system has updated statistics. use quick IO technologies. allocate generously to buffer cache. etc. review AWR advisories & monitor swapping & paging. – Use ASM or alternatively use the SAME methodology for db files.. – Maximize Memory availability.. – Check for excessive/redundant indexing. .Tuning the Database Tier Common corrective actions to maximize scalability  If bottleneck is IO related.. response times of < 10ms. db file scattered read. Oracle and/or its affiliates. – Configure async IO. All rights reserved. ` (db file sequential read.

. – If the program allows. Oracle and/or its affiliates. etc… ) – Use global hash-partitioned indexes for hot leaf blocks . – Work with support/consulting to evaluate if table/index partitioning will help . latch%.Tuning the Database Tier Common corrective actions to maximize scalability  If the bottleneck is concurrency related (enq%.e. more frequent commits (batch size) can help by reducing the CR (consistent read) work required. All rights reserved. – Increase INITRANS to alleviate ITL contention. :latch: cache buffer chains. buffer busy%. 65 Copyright © 2012. %buffer busy waits.Identify via AWR "Top logical IOs by Segment" – Some of these waits are commonly caused by bad SQL execution plans i. read by other session – Make sure to use ASSM and OATM tablespace model. – Increase sequence caches.

. Oracle and/or its affiliates. (gc%..Tuning the Database Tier Common corrective actions to maximize scalability  If the bottleneck is RAC related .... global%.Use Jumbo frames.) – First do all the steps mentioned previously. – Do not use parallel query/DML across nodes  Use instance_groups & parallel_instance_groups settings – Review Parallel Concurrent Processing & Application Affinity configurations  12. etc.1..3 introduced option to define node affinity at the program level – Concurrent -> Program -> Define – Perform RAC focused AWR analysis … 66 Copyright © 2012. – Check interconnect configuration -. . All rights reserved.

.oracle. Oracle and/or its affiliates.Tuning the Database Tier Why upgrade EBS to – Internal tests show 10-20% improvement in EBS batch & reporting workloads – Significant RAC Performance improvements – Oracle internal’s production EBS system on 11gR2.pdf »http://www. All rights reserved.com/us/solutions/benchmark/apps-benchmark/ 67 Copyright © 2012. lower-risk upgrade with SPM/RAT/SPA … – Best Practices/Tips white papers »twp_upgrading_10g_to_11g_what_to_expect_from_optimizer. – Lower cost with Advanced compression. better/automated diagnostics.

Tuning the Database Tier 11g SQL Tracing Enhancements – New tracing parameter PLAN_STAT  first_execution | all_executions | adaptive* – Key tracing patches* listed on MOS Doc 1121043. All rights reserved. Oracle and/or its affiliates.planstat=never | first_execution | all_executionss|adaptive level = 1| 4| 8| 12 | 16 68 Copyright © 2012. .Sql_trace wait=true | false.1 – Turn trace on for just 1 or a list of sql_id’s – With real-time row source stats monitoring from v$sql_plan_statistics – no more waiting hours for a complete trace to get the right trace data – Expanded event syntax : .alter session set events ’sql_trace [sql:a5ks9fhw2v9s1|56bs32ukywdsq] … . bind=true | false.

Oracle and/or its affiliates.Tuning the Database Tier 11g Automatic SQL Monitoring & Tuning 69 Copyright © 2012. . All rights reserved.

. . All rights reserved. Oracle and/or its affiliates. 70 Copyright © 2012.Tuning the Database Tier 11g Performance Enhancements  Optimizer improvements – SPM (SQL Plan Management)– quick relief while permanent solution arrives – Better density estimates for skewed data – Faster. more accurate statistics with AUTO_SAMPLE_SIZE – Extended statistics .address predicate correlation – Adaptive cursor sharing – to address bind peeking issues  Some others… – Interval partitioning. Additional partitioning schemes – Text indexing performance enhancements – RAC scales better & AWR is more RAC aware – Many others.

1.1 OSW Document 461053. Oracle and/or its affiliates.1 OSWg Document • • • • 215187.3 301137. All rights reserved.1 Trace Analyzer (TRCA) 39817. .Tuning the Database Tier References (MOS Docs) • • • • 1121043.1 Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite 1362660.1 Interpreting Raw SQL_TRACE files 280391.1 SQLTXPLAIN (SQLT) 224270.1 Enhanced Performance Diagnostics for Oracle E-Business Suite Release 12.1 How to Retrieve SID Information For a Running Request 71 Copyright © 2012.

.Other References 72 Copyright © 2012. All rights reserved. Oracle and/or its affiliates.

1 upgrade (761570.1) AD Command Line Options for Release R12 (1078973.1.1.1 Info center (806593.References          R12. 225165.1 documentation roadmap (790942. 384248. 236469.1) OAM “Patch Wizard” overview and FAQ (976188.1) Recommended Performance Fixes (244040. All rights reserved.1) 73 Copyright © 2012. Oracle and/or its affiliates.1) Staged or shared APPL_TOP and distributed AD (734025.1) Database preparation guidelines for R12.1.1) Patching FAQs (459156. 976688.1) R12 Upgrade Sizing & Best Practices (399362. .1) Oracle E-Business Suite Release 12.1.

. Oracle and/or its affiliates.Additional Resources 74 Copyright © 2012. All rights reserved.

jspa?forumID=395&start=0 75 Copyright © 2012.1 Upgrade related Reports – – – EBS 12.1) White paper – – –  Have Upgrade questions ? Please post on OTN R12 upgrade forum http://forums. .Additional Resources  EBS R12. All rights reserved.oracle.1) –  EBS pre-install patches Report (1448102.1) EBS ATG Seed Data Comparison Report (1327399.1.1) EBS File Comparison Report (1446430.com/forums/forum.1) R12 Upgrade considerations by product: Financials (889733.1) Oracle E-Business Suite Upgrades and Platform Migration (1377213.3 Data Model Comparison Report (1290886.1 (987516. Oracle and/or its affiliates.1) Planning Your Oracle E-Business Suite Upgrade from Release 11i to Release 12.

All rights reserved.1 Technical Upgrade Best Practices Advanced Architecture and Technology Stack for Oracle E-Business Suite W-066 W-067 76 Copyright © 2012. Oracle and/or its affiliates. .Oracle Demos Moscone West End-to-End Management of Oracle E-Business Suite SOA-Based Integration for Oracle E-Business Suite W-063 W-064 W-065 New User Productivity Capabilities in Oracle E-Business Suite Oracle E-Business Suite 12.

Oracle and/or its affiliates.Upgrading Your Customizations of Oracle E-Business Suite 12. All rights reserved.Oracle E-Business Suite 12. Udayan Parvate CON9032 . .1 Sara Woodhull CON9259 .Related Sessions Wednesday 10:15 am CON8809 .Oracle E-Business Suite Internationalization and Multilingual Features Maher Al-Nubani CON9029 .1 Upgrade Best Practices: Technical Insight Isam Alyousfi.Oracle E-Business Suite Directions: Slashing Downtimes with Online Patching Kevin Hudson Moscone West 3011 Moscone West 2016 Moscone West 2018 Moscone West 2016 10:15 am 10:15 am 11:45 am 77 Copyright © 2012.

Advanced Oracle E-Business Suite Architectures: Maximum Availability. Nishit Rao Moscone West 2016 3:30 pm Moscone West 2016 Moscone West 2016 5:00 pm 78 Copyright © 2012. and More Elke Phelps CON9047 .Efficiently Scaling Oracle E-Business Suite on Oracle Exadata and Oracle Exalogic Isam Alyousfi. Oracle and/or its affiliates.Personalize and Extend Oracle E-Business Suite Applications with Rich Mashups Gustavo Jimenez. All rights reserved. Security.Related Sessions Wednesday 1:15 pm CON9027 . Padmaprabodh Ambale CON9036 . .

Integrating Oracle E-Business Suite with Oracle Identity Management Solutions Sunil Ghosh. Samer Barakat CON3429 . Oracle and/or its affiliates.Web Services and SOA Integration Options for Oracle E-Business Suite Rekha Ayothi. Juan Camilo Ruiz. Sara Woodhull CON9020 . All rights reserved. .Using Oracle ADF with Oracle E-Business Suite: The Full Integration View Siva Puthurkattil.Recommendations for Oracle E-Business Suite Performance Tuning Isam Alyousfi. Veshaal Singh CON9030 . Elke Phelps Moscone West 2016 Moscone West 2018 Moscone West 3003 Moscone West 2016 11:15 am 11:15 am 12:45 pm 79 Copyright © 2012.Related Sessions Thursday 11:15 am CON8716 .

Troubleshooting. All rights reserved. Erik Graversen Moscone West 2016 Moscone West 2018 2:15 pm 80 Copyright © 2012. Diagnosing.Related Sessions Thursday 2:15 pm CON9019 . Oracle and/or its affiliates. . and Optimizing Oracle E-Business Suite Technology Gustavo Jimenez CON9031 .The Top 10 Things You Can Do to Secure Your Oracle E-Business Suite Instance Eric Bing.

Oracle and/or its affiliates.Q&A 81 Copyright © 2012. . All rights reserved.

. All rights reserved.82 Copyright © 2012. Oracle and/or its affiliates.

83 Copyright © 2012. All rights reserved. Oracle and/or its affiliates. .

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.