<Insert Picture Here>

Database Tuning Best Practices and Related Diagnostic Tools

Soumendra Paik
Senior Principal Technical Support Engineer, DB Performance Team, USA
 Optimizer Setting

 Optimizer is one of the most important component of

the database
 OPTIMIZER_FEATURES_ENABLE should be set
correctly
 Set it to lower value than the DB Version if application not
tested and tuned well for a recently upgraded DB
 Lot of SQL performance issues seen because of wrong
setting
 Considered an umbrella parameter to control optimizer
(CBO) behavior

2
 Performance Baselines

 Create AWR Baselines from Good Performance Period

 Generate awrinfo.sql script report
 Review the report periodically to keep SYSAUX space usage
under control
 Use SQL Plan Baselines (SPM) for Plan Stability
 Use it for a small set of SQLs
 Helps to maintain better performance and SLA Goal

3
 SQL Performance Tuning Tools

 SQL Tuning Advisor (STA)

 Use proactively to generate Tuning Advices on slower running SQLs
 Review the Advice and Implement it when it provides significant
Performance benefit
 SQLT – Highly recommended to Install ASAP
 Preferred tool to review SQL Tuning problems
 Helps to provide Testcase without Data to Oracle Support
 XTRACT is the preferred Method used in most cases
 XPLORE is another very useful Method

4
 SQL Performance Tuning Tools

 SQLHC (SQL Health Check) script

 Alternative tool when SQLT cannot be installed/used
 Need no installation
 Runs faster than SQLT
 awrsqrpt.sql script
 Helps to analyze SQL performance very fast
 Provides least amount of details compared to SQLT
XTRACT/XECUTE method or SQLHC script
 Available under $ORACLE_HOME/rdbms/admin

5
How to Create a SQL-testcase Using the DBMS_SQLDIAG Package [Video] (Doc ID 727863.1)
How to Create a SQL-testcase Using the DBMS_SQLDIAG Package [Video] (Doc ID 727863.1)
How to Create a SQL-testcase Using the DBMS_SQLDIAG Package [Video] (Doc ID 727863.1)

SQL Performance Tuning Tools

 Testcase Builder (using DBMS_SQLDIAG package)

 Most useful tool to provide Testcase to Oracle Support for SQL Tuning
problems
 It can generate the Testcase with or without data
 How to Create a SQL-testcase Using the DBMS_SQLDIAG Package
[Video] (Doc ID 727863.1)
 Expdp and Impdp
 Useful commands to provide Testcase data for Wrong Result and other
SQL problems which must need data to reproduce the problem

6
 SQL Performance Tuning Tools

 10046 Event Trace / tkprof command

 Used to diagnose slower SQL or Batch Job performance
 To verify progress for apparent Hung session
 Use LEVEL 12 from start to end of operation
 Use tkprof command with
SORT=PRSELA,FCHELA,EXEELA option
 Use tkprof with AGGREGATE=NO and SYS=NO to understand TX
lock waits and to understand per execution performance of a SQL
 Use NOTE 376442.1

7
 Database Performance Tuning Tools

 Errorstack Trace
 Event tracing for specific errors like ORA-4020, ORA-4021, ORA-942, etc.
 Useful to identify the SQL causing the error when such data is not
available otherwise
 Useful to collect call stack data to understand what the process is doing

8
 Database Performance Tuning Tools

 SYSTEMSTATE Dump Trace File

 Very useful Diagnostic Data Collector
 Used to Diagnose Very Slow, near Hang or Hang Problem in
the database
 Use it to understand probable cause of a Real Hang problem
before restarting the database

9
 Database Performance Tuning Tools

 HANGANALYZE Trace File

 Use it as a complimentary Diagnostic Data Collector with
SYSTEMSTATE dump trace command
 Run it from a separate sqlplus session
 Generates much smaller file than systemstate dump trace file
 LEVEL 4 is actually better than level 3 and can avoid generation of
Systemstate dump trace file

10
 Database Hang Problem

 Hang is NOT a very common problem

 It is a very slow performing DB in most cases
 Restart is the only option for a real hang condition after
collecting SYSTEMSTATE dump trace with –PRELIM
option
 Hang Waiting on 'LGWR worker group ordering' with Deadlock
Between LGWR (Log Writer) Slaves (LG0n) when Using Multiple
LGWR Processes (Doc ID 1968563.1)
 Set "_use_single_log_writer"=TRUE and restart the database

11
 Database Performance Tuning Tools

 awrinfo.sql script
 Very useful to validate SYSAUX Tablespace space usage
problem
 Slower performance on AWR SQLs
 Slower Stats gathering on AWR objects
 AWR repository cleanup problems

12
 Database Performance Tuning Tools

 AWR/ASH/ADDM Reports
 Generate AWR and ASH reports in HTML format, ADDM report
in TEXT format for upto 1 hour duration
 Generate Individual AWR reports from each RAC Instance
instead of Global AWR report
 Generate multiple reports from Slower Performance Periods in
HTML format
 ASH Report provides Top 5 slower running SQLs
 ADDM report provides built-in Cause-Effect analysis for any
performance period for a quick check

13
 Database Performance Tuning Tools

 Version Count Script

 Useful to investigate why a SQL is creating hundreds or even
thousands of Child Cursors ( versions) causing Mutex, Latch
and library cache lock waits (Doc ID 438755.1)
 Row Chaining and Migration
 Causes excessive IO and slower performance for SQLs
when IO is not the cause of performance problem
 De-fragmenting the table and related indexes improve
performance (Doc ID 122020.1)

14
 Serverside Recommended Tools

 OS Watcher
 Used to verify Server side performance
 Highly recommended to be pre-installed
 Uses available OS specific commands
 truss, tusc, strace, etc.
 Used to trace a slower process
 Use options to get DELTA Time in calls (i.e., -D with truss)
 Rda  Used to collect Basic OS data
 Orachk  Used to verify s/w configuration

15
 Application Testing and Tuning Tools

 Real Application Testing (RAT)

 Workload Capture in Source DB
 Process Captured Workload in Destination DB
 Replay Processed Workload in Destination DB
 SQL Performance Analyzer (SPA)
 RAT and SPA usage needs a separate License
 You will need some Training before using it effectively

16
 Database Parameter Tuning Recommendations

 Consider to set following parameters

 These few parameter changes can help alleviate most of Mutex, Latch and Library Cache
Lock waits in any database
 AMM Feature (MEMORY_TARGET > 0) is not Compatible with HugePages or LargePages
features of the OS
 ASMM Feature usage Highly recommended (SGA_TARGET > 0)

17
 System Statistics Recommendation

 Gather System Statistics under good workload in the database

for upto 1 hour duration

18
 System Statistics Recommendation

 Sample System Statistics Output

column PVAL2 format A35
select * from aux_stats$;

19
 HugePages Recommendations

 Highly recommended for Linux OS

 Transparent HugePages not recommended for Linux 6
 Highly recommended for SGA size of 8GB and higher
 Reduces Paging and Swapping in the Server
 Improves Connection and DB Performance
 Related Notes:
 HugePages on Oracle Linux 64-bit (Doc ID 361468.1)
 HugePages on Linux: What It Is... and What It Is Not...
(Doc ID 361323.1)
 ALERT: Disable Transparent HugePages on SLES11, RHEL6, OEL6 and
UEK2 Kernels (Doc ID 1557478.1)

20
 LargePages Recommendations

 Recommended for AIX and Windows x64 OS

 Provides similar performance benefit like HugePages
 Reduces Paging and Swapping in the server
 Related Notes:
 How to enable Large Page Feature on AIX-Based Systems (Doc ID
372157.1)
 Using Large Memory Pages on 64-Bit Windows Systems (Doc ID
422844.1)
 Oracle NUMA Usage Recommendation (Doc ID 759565.1)
 NUMA should be disabled unless the underlying Hardware supports it

21
$ sqlplus / as sysdba EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS (); EXEC DBMS_STATS.GATHER_DICTIONARY_STATS ();

Metadata Statistics Gathering

 Gather Fixed Objects Stats

• Gather it once a month or so
$ sqlplus / as sysdba
EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS ();

 Gather Data Dictionary Stats

$ sqlplus / as sysdba
EXEC DBMS_STATS.GATHER_DICTIONARY_STATS ();
• Gather it after lot of new objects have been added to or deleted from the
database
• Gather once a month or so

22
 Database Statistics and Histogram

 Gather Statistics and Histogram for Application Schema Objects

 AUTO Estimate is OK for most objects
 SIZE AUTO in METHOD_OPT is OK in most cases
 Use ESTIMEATE_PERCENT to 20% or so to gather stats faster and
works OK in most cases
 Use METHOD_OPT with SIZE 2048 in 12c to gather maximum number
of histogram buckets to capture data skew in columns
 Use NO_INVALIDATE=> FALSE when you need to use the last gathered
stats ASAP

23
 Schema Objects Statistics and Histogram

 Schema Statistics and Histogram Example

24
 Oracle Application Specific Stats Gathering

 PeopleSoft Apps
 Improving Statistics in Oracle RDBMS for PeopleSoft Enterprise
(Doc ID 1322888.1)
 Siebel CRM Apps
 Performance Tuning Guidelines for Siebel CRM Application on
Oracle Database (Doc ID 781927.1)
 EBS Apps
 Best Practices for Gathering Statistics with Oracle E-Business
Suite (Doc ID 1586374.1)

25
 Lock Objects Statistics

 Intermediate Tables with varying amount of data

 Delete current stats
 Lock table stats using DBMS_STATS
 Set OPTIMIZER_DYNAMIC_SAMPLING to non-zero value or 11
in 12c
 Dynamic Sampling/Statistics will be used by CBO

26
 Keep up with Auto Stats Gathering

 Gather Statistics Manually or with additional Scheduled

Jobs
 Gather stats for objects with NO Statistics
$ sqlplus / as sysdba
BEGIN
DBMS_STATS.GATHER_DATABASE_STATS (
cascade=> TRUE,
options=> 'GATHER EMPTY',
no_invalidate=> FALSE);
END;
/

27
 Keep up with Auto Stats Gathering

 Gather Statistics Manually or with additional Scheduled

Jobs
 Gather stats for objects with Stale Statistics
$ sqlplus / as sysdba
BEGIN
DBMS_STATS.GATHER_DATABASE_STATS (
cascade=> TRUE,
options=> 'GATHER STALE',
no_invalidate=> FALSE);
END;
/

28
 Backup Strategy

 RMAN Backup
 IO intensive operation for a Large Database
 Use incremental backup strategy using Block Change Tracking
(BCT) option
 Incremental Full Backup
 Backup in Level 0 and 1
 Reduces lot of unnecessary IO activities in the Storage System to
maintain acceptable IO performance (<= 10 ms IO service time per
IO call)

29
 Patch Recommendations

 Highly recommended to apply Quarterly PSU patches ASAP

 Make sure to apply recommended patches based on – Things to
Consider Notes …
 Things to Consider Before Upgrading to 12.1.0.2 to Avoid Poor
Performance or Wrong Results (Doc ID 2034610.1)
 Things to Consider Before Upgrading to 11.2.0.4 to Avoid Poor
Performance or Wrong Results (Doc ID 1645862.1)
 Oracle Recommended Patches -- Previous Oracle Database
Recommended Patches (Doc ID 802103.1)

30
 Reference Documents

 A Few Useful Notes

 Patch Set Update - Bugs fixed in each PSU (Doc ID 1123863.1)
 SQLT (SQLTXPLAIN) - Tool that helps to diagnose SQL statements performing poorly (Doc ID 215187.1)
 SQL Tuning Health-Check Script (SQLHC) (Doc ID 1366133.1)
 Best Practices: Proactively Avoiding Database and Query Performance Issues (Doc ID 1482811.1)
 Best Practices and Recommendations for RAC databases using very large SGA (e.g. 100 GB) (Doc ID 1619155.1)
 Oracle Sun Database Machine X2-2/X2-8, X3-2/X3-8 and X4-2 Performance Best Practices (Doc ID 1274475.1)
 Things to Consider Before Upgrading to 12.1.0.2 to Avoid Poor Performance or Wrong Results (Doc ID 2034610.1)
 How To Gather Statistics for the Cost Based Optimizer (Doc ID 1226841.1)
 Best Practices: Proactive Data Collection for Performance Issues (Doc ID 1477599.1)
 How to Use SQL Plan Management (SPM) - Plan Stability Worked Example (Doc ID 456518.1)
 How to Identify Hot Blocks Within the Database Buffer Cache that may be Associated with 'latch: cache buffers
chains' Wait Contention (Doc ID 163424.1)

31
Questions?

Q&A

32