Creating\ue000high-performance\ue000as\ue000an\ue000attribute\ue000of\ue000complex\ue000software\ue000is\ue000extremely\ue000difficult\ue000 business\ue000 for\ue000 developers,\ue000 technology\ue000 administrators,\ue000 architects,\ue000 system\ue000 analysts,\ue000 and\ue000 project\ue000 managers.\ue000 However,\ue000 by\ue000 understanding\ue000 some\ue000 fundamental principles,\ue000 performance\ue000 problem\ue000 solving\ue000 and\ue000 prevention\ue000 can\ue000 be\ue000 made\ue000 far simpler and more reliable.\ue000 This\ue000 paper\ue000 describes\ue000 those\ue000 principles,\ue000 linking\ue000 them\ue000 together\ue000 in\ue000 a\ue000 coherent\ue000 journey\ue000 covering\ue000 the\ue000 goals,\ue000 the\ue000 terms,\ue000 the\ue000 tools,\ue000 and the decisions that you\ue000 need\ue000 to\ue000 maximize\ue000 your\ue000 application\u2019s\ue000 chance\ue000 of\ue000 having\ue000 a\ue000 long, productive, high-performance\ue000 life.\ue000Examples\ue000in\ue000this\ue000paper\ue000touch\ue000upon\ue000Oracle\ue000experiences, but the scope of the\ue000paper\ue000 is\ue000not\ue000restricted\ue000to\ue000Oracle\ue000products.\ue000

1\ue000 An\ue000Axiomatic\ue000Approach.................................................... 1\ue000 2\ue000 What\ue000is\ue000Performance?....................................................... 2\ue000 3\ue000 Response\ue000Time\ue000vs\ue000Throughput..................................... 2\ue000 4\ue000 Percentile\ue000Specifications.................................................. 3\ue000 5\ue000 Problem\ue000Diagnosis.............................................................. 3\ue000 6\ue000 The\ue000Sequence\ue000Diagram..................................................... 4\ue000 7\ue000 The\ue000Profile.............................................................................. 5\ue000 8\ue000 Amdahl\u2019s\ue000Law........................................................................ 5\ue000 9\ue000 Minimizing\ue000Risk................................................................... 6\ue000 10\ue000 Load........................................................................................... 7\ue000 11\ue000 Queueing\ue000Delay.................................................................... 8\ue000 12\ue000 The\ue000Knee................................................................................. 8\ue000 13\ue000 Relevance\ue000of\ue000the\ue000Knee....................................................... 9\ue000 14\ue000 Capacity\ue000Planning..............................................................10\ue000 15\ue000 Random\ue000Arrivals................................................................10\ue000 16\ue000 Coherency\ue000Delay................................................................10\ue000 17\ue000 Performance\ue000Testing.......................................................11\ue000 18\ue000 Measuring.............................................................................11\ue000 19\ue000 Performance\ue000is\ue000a\ue000Feature...............................................12\ue000 20\ue000 Acknowledgments............................................................12\ue000 21\ue000 About\ue000the\ue000Author...............................................................13\ue000

When\ue000I\ue000joined\ue000Oracle\ue000Corporation\ue000in\ue0001989,\ue000
performance\u2014what\ue000everyone\ue000called\ue000\u201cOracle\ue000tuning\u201d\u2014
was\ue000difficult.\ue000Only\ue000a\ue000few\ue000people\ue000claimed\ue000they\ue000could\ue000do\ue000
it\ue000very\ue000well,\ue000and\ue000those\ue000people\ue000commanded\ue000nice,\ue000high
consulting\ue000rates.\ue000When\ue000circumstances\ue000thrust\ue000me\ue000into
the\ue000\u201cOracle\ue000tuning\u201d\ue000arena,\ue000I\ue000was\ue000quite\ue000unprepared.\ue000
Recently,\ue000I\u2019ve\ue000been\ue000introduced\ue000into\ue000the\ue000world\ue000of\ue000
\u201cMySQL\ue000tuning,\u201d\ue000and\ue000the\ue000situation\ue000seems\ue000very\ue000similar
to\ue000what\ue000I\ue000saw\ue000in\ue000Oracle\ue000over\ue000twenty\ue000years\ue000ago.\ue000

It\ue000reminds\ue000me\ue000a\ue000lot\ue000of\ue000how\ue000difficult\ue000I\ue000would\ue000have\ue000told
you\ue000that\ue000beginning\ue000algebra\ue000was,\ue000if\ue000you\ue000had\ue000asked\ue000me
when\ue000I\ue000was\ue000about\ue00013\ue000years\ue000old.\ue000At\ue000that\ue000age,\ue000I\ue000had\ue000to\ue000
appeal\ue000heavily\ue000to\ue000my\ue000\u201cmathematical\ue000instincts\u201d\ue000to\ue000solve\ue000
equations\ue000like\ue0003x\ue000+\ue0004\ue000=\ue00013.\ue000The\ue000problem\ue000with\ue000that\ue000is\ue000
that\ue000many\ue000of\ue000us\ue000didn\u2019t\ue000have\ue000mathematical\ue000instincts.\ue000I\ue000
can\ue000remember\ue000looking\ue000at\ue000a\ue000problem\ue000like\ue000\u201c3x\ue000+\ue0004\ue000=\ue00013;\ue000
find\ue000x\u201d\ue000and\ue000basically\ue000stumbling\ue000upon\ue000the\ue000answer\ue000x\ue000=\ue0003\ue000
using\ue000trial\ue000and\ue000error.\ue000Maybe\ue000I\ue000tried\ue000x\ue000=\ue0002\ue000first,\ue000yielding\ue000
a\ue000left-hand\ue000side\ue000value\ue000of\ue00010,\ue000which\ue000was\ue000smaller\ue000than\ue000
13,\ue000so\ue000I\ue000tried\ue000a\ue000larger\ue000number.\ue000And,\ue000happily,\ue000x\ue000=\ue0003\ue000
happened\ue000to\ue000work.\ue000

The\ue000trial-and-error\ue000method\ue000of\ue000feeling\ue000my\ue000way\ue000through\ue000
algebra\ue000problems\ue000worked\u2014albeit\ue000slowly\ue000and\ue000
uncomfortably\u2014for\ue000easy\ue000equations,\ue000but\ue000it\ue000didn\u2019t\ue000scale\ue000
as\ue000the\ue000problems\ue000got\ue000tougher,\ue000like\ue000\u201c3.1x\ue000+\ue0004\ue000=\ue00013.\u201d\ue000Now\ue000
what?\ue000It\ue000was\ue000hard.\ue000My\ue000problem\ue000was\ue000that\ue000I\ue000wasn\u2019t\ue000
\u201cthinking\ue000clearly\u201d\ue000yet\ue000about\ue000algebra.\ue000My\ue000introduction\ue000
at\ue000age\ue000fifteen\ue000to\ue000Jimmy\ue000Harkey\ue000put\ue000me\ue000on\ue000the\ue000road\ue000to\ue000
solving\ue000that\ue000problem.\ue000

set\ue000of\ue000steps\ue000that\ue000worked\ue000every\ue000time\ue000(and\ue000he\ue000gave\ue000us\ue000
plenty\ue000of\ue000homework\ue000to\ue000practice\ue000on).\ue000In\ue000addition\ue000to\ue000
working\ue000every\ue000time,\ue000by\ue000executing\ue000those\ue000steps,\ue000we\ue000
necessarily\ue000documented\ue000our\ue000thinking\ue000as\ue000we\ue000worked.\ue000
Not\ue000only\ue000were\ue000we\ue000thinking\ue000clearly,\ue000using\ue000a\ue000reliable\ue000and
repeatable\ue000sequence\ue000of\ue000steps,\ue000we\ue000were\ue000proving\ue000to\ue000
anyone\ue000who\ue000read\ue000our\ue000work\ue000that\ue000we\ue000were\ue000thinking\ue000
clearly.\ue000

This\ue000was\ue000Mr.\ue000Harkey\u2019s\ue000axiomatic\ue000approach\ue000to\ue000algebra,\ue000
geometry,\ue000trigonometry,\ue000and\ue000calculus:\ue000one\ue000small,\ue000
logical,\ue000provable,\ue000and\ue000auditable\ue000step\ue000at\ue000a\ue000time.\ue000It\u2019s\ue000the\ue000
first\ue000time\ue000I\ue000ever\ue000really\ue000got\ue000mathematics.\ue000

in\ue000the\ue000world\ue000after\ue000school.\ue000In\ue000life,\ue000I\u2019ve\ue000found\ue000that\ue000it\u2019s\ue000not
nearly\ue000as\ue000important\ue000to\ue000be\ue000right\ue000as\ue000it\ue000is\ue000to\ue000be\ue000able\ue000to\ue000
convince\ue000others\ue000that\ue000you\u2019re\ue000right.\ue000

My\ue000goal\ue000since\ue000the\ue000mid-1990s\ue000has\ue000been\ue000to\ue000create\ue000a\ue000
similarly\ue000rigorous\ue000approach\ue000to\ue000Oracle\ue000performance\ue000
optimization.\ue000Lately,\ue000I\ue000am\ue000expanding\ue000the\ue000scope\ue000of\ue000that\ue000
goal\ue000beyond\ue000Oracle,\ue000to:\ue000\u201cCreate\ue000an\ue000axiomatic\ue000approach\ue000
to\ue000computer\ue000software\ue000performance\ue000optimization.\u201d\ue000I\u2019ve\ue000
found\ue000that\ue000not\ue000many\ue000people\ue000really\ue000like\ue000it\ue000when\ue000I\ue000talk\ue000
like\ue000that,\ue000so\ue000let\u2019s\ue000say\ue000it\ue000like\ue000this:\ue000

My\ue000goal\ue000is\ue000to\ue000help\ue000you\ue000think\ue000clearly\ue000about\ue000how\ue000to\ue000
optimize\ue000the\ue000performance\ue000of\ue000your\ue000computer\ue000
software.\ue000

If\ue000you\ue000google\ue000for\ue000the\ue000word\ue000performance,\ue000you\ue000get\ue000over\ue000a\ue000
half\ue000a\ue000billion\ue000hits\ue000on\ue000concepts\ue000ranging\ue000from\ue000bicycle\ue000
racing\ue000to\ue000the\ue000dreaded\ue000employee\ue000review\ue000process\ue000that\ue000
many\ue000companies\ue000these\ue000days\ue000are\ue000learning\ue000to\ue000avoid.\ue000
When\ue000I\ue000googled\ue000for\ue000performance,\ue000most\ue000of\ue000the\ue000top\ue000hits\ue000
relate\ue000to\ue000the\ue000subject\ue000of\ue000this\ue000paper:\ue000the\ue000time\ue000it\ue000takes\ue000for\ue000
computer\ue000software\ue000to\ue000perform\ue000whatever\ue000task\ue000you\ue000ask\ue000
it\ue000to\ue000do.\ue000

execute\ue000some\ue000task.\ue000Response\ue000time\ue000is\ue000the\ue000execution\ue000
duration\ue000of\ue000a\ue000task,\ue000measured\ue000in\ue000time\ue000per\ue000task,\ue000like\ue000
\u201cseconds\ue000per\ue000click.\u201d\ue000For\ue000example,\ue000my\ue000Google\ue000search\ue000for\ue000
the\ue000word\ue000performance\ue000had\ue000a\ue000response\ue000time\ue000of\ue000
0.24\ue000seconds.\ue000The\ue000Google\ue000web\ue000page\ue000rendered\ue000that\ue000
measurement\ue000right\ue000in\ue000my\ue000browser.\ue000That\ue000is\ue000evidence,\ue000to\ue000
me,\ue000that\ue000Google\ue000values\ue000my\ue000perception\ue000of\ue000Google\ue000
performance.\ue000

Some\ue000people\ue000are\ue000interested\ue000in\ue000another\ue000performance
measure:\ue000Throughput\ue000is\ue000the\ue000count\ue000of\ue000task\ue000executions\ue000
that\ue000complete\ue000within\ue000a\ue000specified\ue000time\ue000interval,\ue000like\ue000
\u201cclicks\ue000per\ue000second.\u201d\ue000In\ue000general,\ue000people\ue000who\ue000are\ue000

responsible\ue000for\ue000the\ue000performance\ue000of\ue000groups\ue000of\ue000people\ue000 worry\ue000more\ue000about\ue000throughput\ue000than\ue000people\ue000who\ue000work in\ue000a\ue000solo\ue000contributor\ue000role.\ue000For\ue000example,\ue000an\ue000individual accountant\ue000is\ue000usually\ue000more\ue000concerned\ue000about\ue000whether the\ue000response\ue000time\ue000of\ue000a\ue000daily\ue000report\ue000will\ue000require\ue000him to stay\ue000late\ue000after\ue000work\ue000today.\ue000The\ue000manager\ue000of\ue000a\ue000group\ue000of\ue000 accounts\ue000is\ue000additionally\ue000concerned\ue000about\ue000whether\ue000the system\ue000is\ue000capable\ue000of\ue000processing\ue000all\ue000the\ue000data\ue000that\ue000all\ue000of\ue000 her\ue000accountants\ue000will\ue000be\ue000processing\ue000today.\ue000

Throughput\ue000and\ue000response\ue000time\ue000have\ue000a\ue000generally\ue000
reciprocal\ue000type\ue000of\ue000relationship,\ue000but\ue000not\ue000exactly.\ue000The\ue000
real\ue000relationship\ue000is\ue000subtly\ue000complex.\ue000

throughput\ue000at\ue0001,000\ue000tasks\ue000per\ue000second\ue000for\ue000some\ue000 benchmark.\ue000What,\ue000then,\ue000is\ue000your\ue000users\u2019\ue000average\ue000 response\ue000time?\ue000

It\u2019s\ue000tempting\ue000to\ue000say\ue000that\ue000your\ue000average\ue000response\ue000time\ue000
was\ue0001/1,000\ue000=\ue000.001\ue000seconds\ue000per\ue000task.\ue000But\ue000it\u2019s\ue000not\ue000
necessarily\ue000so.\ue000

Imagine\ue000that\ue000your\ue000system\ue000processing\ue000this\ue000throughput\ue000
had\ue0001,000\ue000parallel,\ue000independent,\ue000homogeneous\ue000
service\ue000channels\ue000inside\ue000it\ue000(that\ue000is,\ue000it\u2019s\ue000a\ue000system\ue000with\ue000
1,000\ue000independent,\ue000equally\ue000competent\ue000service\ue000
providers\ue000inside\ue000it,\ue000each\ue000awaiting\ue000your\ue000request\ue000for\ue000
service).\ue000In\ue000this\ue000case,\ue000it\ue000is\ue000possible\ue000that\ue000each\ue000request\ue000
consumed\ue000exactly\ue0001\ue000second.\ue000

Now,\ue000we\ue000can\ue000know\ue000that\ue000average\ue000response\ue000time\ue000was\ue000
somewhere\ue000between\ue0000\ue000seconds\ue000per\ue000task\ue000and\ue000
1\ue000second\ue000per\ue000task.\ue000But\ue000you\ue000cannot\ue000derive\ue000response\ue000
time\ue000exclusively1\ue000from\ue000a\ue000throughput\ue000measurement.\ue000
You\ue000have\ue000to\ue000measure\ue000it\ue000separately.\ue000

because\ue000there\ue000are\ue000mathematical\ue000models\ue000that\ue000can\ue000compute
response\ue000time\ue000for\ue000a\ue000given\ue000throughput,\ue000but\ue000the\ue000models\ue000
require\ue000more\ue000input\ue000than\ue000just\ue000throughput.\ue000

The\ue000subtlety\ue000works\ue000in\ue000the\ue000other\ue000direction,\ue000too.\ue000You\ue000can certainly\ue000flip\ue000the\ue000example\ue000I\ue000just\ue000gave\ue000around\ue000and\ue000prove\ue000 it.\ue000However,\ue000a\ue000scarier\ue000example\ue000will\ue000be\ue000more\ue000fun.\ue000

programming\ue000to\ue000deliver\ue000a\ue000throughput\ue000of\ue000100\ue000tasks\ue000
per\ue000second\ue000on\ue000a\ue000single-CPU\ue000computer.\ue000Imagine\ue000that\ue000
the\ue000new\ue000task\ue000you\u2019ve\ue000written\ue000executes\ue000in\ue000just\ue000
.001\ue000seconds\ue000on\ue000the\ue000client\u2019s\ue000system.\ue000Will\ue000your\ue000
program\ue000yield\ue000the\ue000throughput\ue000that\ue000the\ue000client\ue000
requires?\ue000

It\u2019s\ue000tempting\ue000to\ue000say\ue000that\ue000if\ue000you\ue000can\ue000run\ue000the\ue000task\ue000once\ue000
in\ue000just\ue000a\ue000thousandth\ue000of\ue000a\ue000second,\ue000then\ue000surely\ue000you\u2019ll\ue000be\ue000
able\ue000to\ue000run\ue000that\ue000task\ue000at\ue000least\ue000a\ue000hundred\ue000times\ue000in\ue000the\ue000
span\ue000of\ue000a\ue000full\ue000second.\ue000And\ue000you\u2019re\ue000right,\ue000if\ue000the\ue000task\ue000
requests\ue000are\ue000nicely\ue000serialized,\ue000for\ue000example,\ue000so\ue000that\ue000
your\ue000program\ue000can\ue000process\ue000all\ue000100\ue000of\ue000the\ue000client\u2019s\ue000
required\ue000task\ue000executions\ue000inside\ue000a\ue000loop,\ue000one\ue000after\ue000the\ue000
other.\ue000

But\ue000what\ue000if\ue000the\ue000100\ue000tasks\ue000per\ue000second\ue000come\ue000at\ue000your\ue000
system\ue000at\ue000random,\ue000from\ue000100\ue000different\ue000users\ue000logged\ue000
into\ue000your\ue000client\u2019s\ue000single-CPU\ue000computer?\ue000Then\ue000the\ue000
gruesome\ue000realities\ue000of\ue000CPU\ue000schedulers\ue000and\ue000serialized\ue000
resources\ue000(like\ue000Oracle\ue000latches\ue000and\ue000locks\ue000and\ue000writable\ue000
access\ue000to\ue000buffers\ue000in\ue000memory)\ue000may\ue000restrict\ue000your\ue000
throughput\ue000to\ue000quantities\ue000much\ue000less\ue000than\ue000the\ue000required\ue000
100\ue000tasks\ue000per\ue000second.\ue000

It\ue000might\ue000work.\ue000It\ue000might\ue000not.\ue000You\ue000cannot\ue000derive\ue000
throughput\ue000exclusively\ue000from\ue000a\ue000response\ue000time\ue000
measurement.\ue000You\ue000have\ue000to\ue000measure\ue000it\ue000separately.\ue000

Response\ue000time\ue000and\ue000throughput\ue000are\ue000not\ue000necessarily\ue000
reciprocals.\ue000To\ue000know\ue000them\ue000both,\ue000you\ue000need\ue000to\ue000measure
them\ue000both.\ue000

So,\ue000which\ue000is\ue000more\ue000important:\ue000response\ue000time,\ue000or\ue000
throughput?\ue000For\ue000a\ue000given\ue000situation,\ue000you\ue000might\ue000answer
legitimately\ue000in\ue000either\ue000direction.\ue000In\ue000many\ue000
circumstances,\ue000the\ue000answer\ue000is\ue000that\ue000both\ue000are\ue000vital\ue000
measurements\ue000requiring\ue000management.\ue000For\ue000example,\ue000a\ue000
system\ue000owner\ue000may\ue000have\ue000a\ue000business\ue000requirement\ue000that\ue000
response\ue000time\ue000must\ue000be\ue0001.0\ue000seconds\ue000or\ue000less\ue000for\ue000a\ue000given\ue000
task\ue000in\ue00099%\ue000or\ue000more\ue000of\ue000executions\ue000and\ue000the\ue000system\ue000
must\ue000support\ue000a\ue000sustained\ue000throughput\ue000of\ue000
1,000\ue000executions\ue000of\ue000the\ue000task\ue000within\ue000a\ue00010-minute\ue000
interval.\ue000

In\ue000the\ue000prior\ue000section,\ue000I\ue000used\ue000the\ue000phrase\ue000\u201cin\ue00099%\ue000or\ue000more\ue000
of\ue000executions\u201d\ue000to\ue000qualify\ue000a\ue000response\ue000time\ue000expectation.
Many\ue000people\ue000are\ue000more\ue000accustomed\ue000to\ue000statements\ue000like,
\u201caverage\ue000response\ue000time\ue000must\ue000be\ue000r\ue000seconds\ue000or\ue000less.\u201d\ue000The\ue000
percentile\ue000way\ue000of\ue000stating\ue000requirements\ue000maps\ue000better,
though,\ue000to\ue000the\ue000human\ue000experience.\ue000

computer\ue000every\ue000day.\ue000Further\ue000imagine\ue000that\ue000the\ue000lists\ue000of\ue000
numbers\ue000shown\ue000in\ue000Exhibit\ue0001\ue000represent\ue000the\ue000measured\ue000
response\ue000times\ue000of\ue000ten\ue000executions\ue000of\ue000that\ue000task.\ue000The\ue000
average\ue000response\ue000time\ue000for\ue000each\ue000list\ue000is\ue0001.000\ue000seconds.\ue000
Which\ue000one\ue000do\ue000you\ue000think\ue000you\u2019d\ue000like\ue000better?\ue000

You\ue000can\ue000see\ue000that\ue000although\ue000the\ue000two\ue000lists\ue000have\ue000the\ue000same\ue000
average\ue000response\ue000time,\ue000the\ue000lists\ue000are\ue000quite\ue000different\ue000in\ue000
character.\ue000In\ue000List\ue000A,\ue00090%\ue000of\ue000response\ue000times\ue000were\ue000
1\ue000second\ue000or\ue000less.\ue000In\ue000List\ue000B,\ue000only\ue00060%\ue000of\ue000response\ue000
times\ue000were\ue0001\ue000second\ue000or\ue000less.\ue000Stated\ue000in\ue000the\ue000opposite\ue000
way,\ue000List\ue000B\ue000represents\ue000a\ue000set\ue000of\ue000user\ue000experiences\ue000of\ue000
which\ue00040%\ue000were\ue000dissatisfactory,\ue000but\ue000List\ue000A\ue000(having\ue000
the\ue000same\ue000average\ue000response\ue000time\ue000as\ue000List\ue000B)\ue000represents\ue000
only\ue000a\ue00010%\ue000dissatisfaction\ue000rate.\ue000

In\ue000List\ue000A,\ue000the\ue00090th\ue000percentile\ue000response\ue000time\ue000is\ue000
.987\ue000seconds.\ue000In\ue000List\ue000B,\ue000the\ue00090th\ue000percentile\ue000response\ue000
time\ue000is\ue0001.273\ue000seconds.\ue000These\ue000statements\ue000about\ue000
percentiles\ue000are\ue000more\ue000informative\ue000than\ue000merely\ue000saying\ue000
that\ue000each\ue000list\ue000represents\ue000an\ue000average\ue000response\ue000time\ue000of\ue000
1.000\ue000seconds.\ue000

As\ue000GE\ue000says,\ue000\u201cOur\ue000customers\ue000feel\ue000the\ue000variance,\ue000not\ue000the
mean.\u201d2\ue000Expressing\ue000response\ue000time\ue000goals\ue000as\ue000percentiles\ue000
make\ue000for\ue000much\ue000more\ue000compelling\ue000requirement\ue000
specifications\ue000that\ue000match\ue000with\ue000end\ue000user\ue000expectations:\ue000

In\ue000nearly\ue000every\ue000performance\ue000problem\ue000I\u2019ve\ue000been\ue000invited\ue000
to\ue000repair,\ue000the\ue000problem\ue000statement\ue000has\ue000been\ue000a\ue000statement
about\ue000response\ue000time.\ue000\u201cIt\ue000used\ue000to\ue000take\ue000less\ue000than\ue000a\ue000
second\ue000to\ue000do\ue000X;\ue000now\ue000it\ue000sometimes\ue000takes\ue00020+.\u201d\ue000Of\ue000
course,\ue000a\ue000specific\ue000problem\ue000statement\ue000like\ue000that\ue000is\ue000often
buried\ue000behind\ue000veneers\ue000of\ue000other\ue000problem\ue000statements,\ue000