You are on page 1of 49

Real-World

Performance
What the Real-World Performance Team Learns from Your
Automa8c Workload Repository Report
Andrew Holdsworth Graham Wood
Architect
Vice President
Real-World
Performance,
Real-World Performance,

Server Technologies Server Technologies
September 29, 2014 September 29, 2014

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Safe Harbor Statement


The following is intended to outline our general product direcOon. It is
intended for informaOon purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code,
or funcOonality, and should not be relied upon in making purchasing
decisions. The development, release, and Oming of any features or
funcOonality described for Oracles products remains at the sole
discreOon of Oracle.

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Real-World
Performance

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Real-World Performance Root Causes

The database is not being


used as it was designed to
be used

The applica8on
architecture/code design
is sub-op8mal

There is a sub op8mal


algorithm in the database

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

To Fix Root Causes Requires


CHANGE

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

CHANGE
is scary and somebody will always
get upset

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Are you willing to be an agent of


CHANGE ?

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

How Does Your OrganizaOon Do Performance ?


ConvenOonal

Real-World

Focus is on Good Enough or


What the Business Needs
Process Orientated/Part of
Quality Assurance
Spends most of the Ome on
Pla`orm Tuning Issues
Only changes things within
limited scope
Boaom up tuning approach
Looking for incremental gains

Focus on excellence and what


the Hardware and Socware
can do
Innovate excellent
performance and add
intellectual property to your
code
Everything is within scope
HolisOc top down approach
Focus on orders of magnitude
gains

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Recent Results 1000X Project


Baseline:

4.3 Hours

Baseline:

2.4 Days

Baseline:

2.5 Hours

Code Changes:

4.3 Hours

Code Changes:

27 Mins

Code Changes:

2.5 Hours

Correct Usage:

29 Secs Correct Usage: 7.5 Mins

Correct Usage: 4 Secs

Bug Fixes:

12 Secs

Bug Fixes: 4.5 Mins

Bug Fixes:

0.90 Secs

Final:

12 Secs

Final:

4.5 Mins

Final:

0.90 Secs

Speed up:

1355.57

Speed up:

1002

Speed up:

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

1024

Yes we are the Crazy


Ones !

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Program Agenda
1

Real-World Performance

OLTP

Batch

Real-World Performance Programs

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR Architecture Analysis


More than just wait events and top SQL

Large amount of data in the AWR report


Tells us about the way that the system has been architected
and designed as well as about how it is performing
Ocen see common mistakes
Hyperlinks in slides are links to RWP video series
hap://oracle.com/goto/oll/rwp

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR from online


system

Ready for Black


Friday?
Copyright 2014, Oracle and/or its aliates. All rights reserved. |


AWR from Online system
TesOng system for Black Friday readiness
Cannot generate load expected on test system
Do you see any problems with this system scaling up from
this test?
Will we survive Black Friday ?

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR Header
First-level bulleted text is
Calibri
28 pt
Second-level text (press tab
key) is
24 pt

Calibri is the only font used


in
the template
All bulleted text is sentence
case (capitalize rst leaer of
rst word)
Use bold, red, or both to
highlight ONLY KEY text

32 Cores available
Over processed
Sessions is 100x cores
Session count growing
Session leak
Dynamic connecOon pools

Cursors per session growing


Cursor leakage

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Load Prole
~260 sessions acOve on
average
~40 on CPU
Only have 32 cores
System CPU limited

10 logons per second


In a stable system?
Session leaks
Dynamic connecOon pools

40% of user txns are


rollbacks
Coding for failure

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Init.ora
Underscore parameters
Db_block_size=16384
Cursor_sharing=FORCE
Db_le_mulOblock_read_co
unt=32

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Init.ora
Db_writer_processes=12
On a system that supports
asynchIO?

Open_cursors=2000
Per session limit
Implies cursor leaking

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Init.ora
OpOmizer_index_cost_adj=5
0
Classic hack parameter

Processes=5500
Sessions=8320

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Top events
Where is the Ome going?

Concurrency waits > 35% of


Ome
Library cache: mutex X
Latch:row cache objects
Typical of high CPU load
A symptom, not the problem

Row lock contenOon 18% of


Ome
IO with 7ms avg read Ome
CPU only 15% of DB Time
Log le sync?
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Top SQL
Where is the Ome going?

Top statement
SELECT /*SHOP*/
YFS_ORDER_HEADER.*
FROM YFS_ORDER_HEADER
WHERE (ORDER_HEADER_KEY = :1 )
FOR UPDATE

12% of load
2 million execuOons
Average execuOon 0.1 sec

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Top SQL
Top statement
SELECT /*SHOP*/
YFS_ORDER_HEADER.*
FROM YFS_ORDER_HEADER
WHERE (ORDER_HEADER_KEY = :1 )
FOR UPDATE

40% of RLWs are on that object

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Top SQL
Next two statements
Call of
DBMS_APPLICATION_INFO
ApplicaOon instrumentaOon

14% of load
26M execuOons each
InstrumentaOon is a good thing
BUT
Not needed since Oracle 10g
Use parameters to OCI or Java
instead
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Other SQL

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Online Summary
Not looking good for Black Friday

System is CPU bound at test load levels


System seems to be leaking both cursors and sessions (and
maybe locks)
System is running far too many processes
High overhead applicaOon instrumentaOon

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR from batch


system

Maxing out the Batch


Window?
Copyright 2014, Oracle and/or its aliates. All rights reserved. |


AWR from batch system
Night Ome batch workload approaching Ome limit
AlternaOve pla`orms proposed to increase throughput

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR Header
First-level bulleted text is
Calibri
28 pt
Second-level text (press tab
key) is
24 pt

44 Cores available
Over processed
Sessions is 30x cores

Calibri is the only font used


in
the template
All bulleted text is sentence
case (capitalize rst leaer of
rst word)
Use bold, red, or both to
highlight ONLY KEY text

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Load Prole
~55 sessions acOve on
average
~10 on CPU
Only 25% of CPU resource used
Scope for signicantly more
thoughput

~5 logons per second

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Init.ora
Underscore parameters
Cell_ooad_processing=FAL
SE

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Init.ora
Unused pools congured
Java pool and large pool 3.5GB
each

Open_cursors=5000
Per session limit
Implies cursor leaking

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Top events
Where is the Ome going?

Row lock contenOon


> 4.5sec average wait Ome
In batch?
Ocen applicaOon parallelism
Load balancing vs anity

9.5m IOs with 2ms avg read


Ome
CPU only 18% of DB Time
CPU usage underreported on
AIX7 when processor
mulOthreading enabled
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Top SQL
Where is the Ome going?

PL/SQL calls with 64 and 128


execs
ApplicaOon parallel threads?

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Top SQL
Top statement by
execuOons
3.2 M execuOons
0 rows returned

Large number of frequently


executed statements
All process a single row per
execute
Row by row batch processing

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Top SQL
Top statement
UPDATE CH_ACCT_MAST
SET BAL_AVAILABLE = BAL_AVAILABLE
+ :B2 , CTR_UPDAT_SRLNO =
CTR_UPDAT_SRLNO + 1
WHERE COD_ACCT_NO = :B1

22% of DB Time
Not spending Ome on CPU or
disk IO

Object accounts for > 70% of


RLWs
Object is parOOoned
AaempOng to reduce
contenOon?

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR Batch Summary


Plenty of resources on current hardware pla`orm
ContenOon caused by sub-opOmal applicaOon parallelism
Row by row processing of millions of rows giving away
orders of magnitude in performance vs set based processing

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Where do we learn
more ?

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Real-World Performance
Programs

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Online
EducaOon
11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Online Video Series



hap://www.oracle.com/goto/oll/rwp

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

One Day Real-World Road


Show

11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Real-World Performance Tour in AssociaOon


with IOUG

Europe 2015

Feb 9th Finland


Feb 11th Benelux
Feb 13th Norway
Mar 17th Slovenia/CroaOa
Mar 19th England
Mar 23rd Denmark
Mar 25th Austria
Mar 27th Turkey

USA tbd
ASIA tbd
South America tbd
11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Real World Performance


Training
11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Real-World Performance Classroom Training


4 Day Class of Intensive Performance Training

Topics: SQL, OLTP, DW/BI


Classroom, Demos, Hands On, Test and Quizzes
Training given by Real World Performance Engineers
Designed for Architects, Developers and DBAs
4 months training in 4 days

Contact RWP and your local Oracle team to apply


11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Real-World Performance Classroom Training


Quotes
Fantas'c such a good course. Why cant all training be like this?
The demos schooled me
Content matches the real worldtotally dierent than courses oered by Oracle
University
Your understanding of Oracle real life problems, familiarity with computer science
basics and procedures, teaching excellence, and interpersonal skills all contributed to
a successful trainingVenkat Dulla, Qualcomm
Anyone out there claiming that he/she is an Oracle DBA must aPend RWP classes. We
are highly experienced team of DBAs and we thought weve seen it all un'l this class !
Real problems that we are dealing day-to-day, like connec'on storms, was turned
into training scenarios and explained so clearly with cause and solu'ons along with a
diagnos'c methodologyit was also an eye-opener for OLTP DBAs that I/O is no
longer a boPleneck with exadata and it is 'me to start using set opera'ons to
achieve order of magnitude performance gains Me'n Ylimaz, Turkcell
11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Safe Harbor Statement


The preceding is intended to outline our general product direcOon. It is
intended for informaOon purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code,
or funcOonality, and should not be relied upon in making purchasing
decisions. The development, release, and Oming of any features or
funcOonality described for Oracles products remains at the sole
discreOon of Oracle.
Note: The speaker notes for this slide
include instrucOons for when to use
Safe Harbor Statement slides.
Tip! Remember to remove this text
box.
Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Copyright 2014, Oracle and/or its aliates. All rights reserved. |