Oracle SQL Parallel Execution

An Oracle White Paper June 2008

Oracle SQL Parallel Execution

Executive Summary...............................................................................4 Introduction...........................................................................................4 Why Parallel Execution?.......................................................................6 The ultimate goal: scalability............................................................6 Shared everything – the Oracle advantage.......................................7 Fundamental Concepts of Oracle's Parallel Execution..........................8 Processing parallel SQL statements..................................................9 Query Coordinator (QC) and parallel servers..............................10 Producer/consumer model...........................................................12 Granules.......................................................................................13 Data redistribution.......................................................................14 Enabling parallel execution in Oracle.............................................19 Controlling SQL Parallel Execution in Oracle................................21 Understand your target workload................................................22 Controlling the degree of parallelism..........................................22 Controlling the usage of parallelism............................................24 Oracle SQL Parallel Execution best practices.....................................25 Start with a balanced system...........................................................26 Calibrate your configuration........................................................26 Stripe And Mirror Everything (S.A.M.E.) – use ASM................27 Set database initialization parameters for good performance..........28 Memory allocation ......................................................................28 Controlling parallel servers.........................................................29 Enabling efficient I/O throughput ...............................................30 Use parallel execution with common sense.....................................31 Don't enable parallelism for small objects...................................31 Use parallelism to achieve your goals, not to exceed them.........31 Avoid using hints.........................................................................31 Combine parallel execution with Oracle Partitioning.....................32 Ensure statistics are good enough....................................................32 Monitor parallel execution activity..................................................32 Whether or not to use parallel execution in RAC............................33 Use Database Resource Manager....................................................33 Don't try to solve hardware deficiencies with other features...........33 Don't ignore other features..............................................................34 Monitoring SQL Parallel Execution....................................................34

Oracle SQL Parallel Execution

Page 2

(G)V$ parallel execution views.......................................................34 Interpreting parallel SQL execution plans.......................................35 Parallel plan without partitioning................................................35 Parallel plan with partitioning and partition-wise join................36 Oracle Enterprise Manager..............................................................38 Wait events..................................................................................38 Input/Output (I/O) monitoring.....................................................40 Parallel execution monitoring......................................................40 SQL monitoring...........................................................................41 Upgrade considerations coming from Oracle Database 9i..................43 More parallel operations..................................................................43 If on Oracle Database 9i you used hints to enable SQL parallel execution......................................................................................44 If on Oracle Database 9i you used session settings to enable SQL parallel execution.........................................................................44 If on Oracle Database 9i you used object level settings to enable SQL parallel execution................................................................44 Execution plan changes...................................................................44 Changes in database defaults...........................................................45 Use Resource Manager....................................................................45 Conclusion...........................................................................................46

Oracle SQL Parallel Execution

Page 3

Oracle SQL Parallel Execution


Parallel execution is one of the fundamental database technologies that enable organizations to manage and access tens – if not even hundreds of terabytes of data. Without parallelism, these large databases, commonly used for data warehouses but increasingly found in operational systems as well, would not exist. Parallel execution is the ability to apply multiple CPU and I/O resources to the execution of a single database operation. While every major database vendor today provides parallel capabilities, there remain key differences in the architectures provided by the various vendors. SQL parallel execution was first introduced in Oracle more than a decade ago1 and has been enriched and improved since. This paper discusses the parallel execution architecture of Oracle Database 11g and shows its superiority over alternative architectures for real-world applications. This paper also touches on how to control and monitor parallel execution; lastly, it gives an insight into on upgrade considerations when migrating from earlier versions of Oracle. While the focus of this paper is on Oracle Database 11g, the fundamental concepts are also applicable to earlier versions of Oracle.


Databases today, irrespective of whether they are data warehouses, operational data stores, or OLTP systems, contain a wealth of information. However, finding and presenting the right information in a timely fashion can be a challenge because of the vast quantity of data involved. Parallel execution is the capability that addresses this challenge. Using parallelism, terabytes of data can be processed in minutes or even less, not hours or days. Parallel execution uses multiple processes to accomplish a single task – to complete a SQL statement in the case of SQL parallel execution. The more effectively the database software can leverage all hardware resources – multiple cores, multiple I/O channels, or even multiple nodes in a cluster - the more efficiently queries and other database operations will be processed.
1 Parallel execution was first introduced in Oracle Version 7.3 in 1996

Oracle SQL Parallel Execution

Page 4

such as batch operations. Oracle SQL parallel execution requires Oracle Database 11g Enterprise Edition. – – – Oracle SQL Parallel Execution Page 5 . the reader will become familiar with Oracle's parallel architecture. The paper covers four main topics: – The first section discusses the fundamental concepts of parallel processing of the Oracle database. The fourth section focuses on upgrade considerations when migrating an environment from an earlier release of Oracle to Oracle Database 11g. Specific operations in OLTP applications. leveraging either SQL or Oracle Enterprise Manager Database/Grid Control. and understand the basics of how to control and identify parallel SQL processing. The second section focuses on best practices around parallel execution to ensure the most optimal usage of your hardware resources The third section provides an insight into how to monitor an environment using parallel execution.Examples of resource-intensive database operations include: – Large (long-running) queries: for example data warehouse analysis comparing one year's results with the results of the year prior Building indexes on large tables Gathering statistics in a large database Loading a large amount of data into a database Taking a database backup – – – – Large data warehouses should always use parallel execution to achieve good performance. learn Oracle-specific terminology around parallel execution. can also significantly benefit from parallel execution.

For example. Scenario 2: If your friend is available then the two of you could start on opposite ends of the street. right? Look again: it shows the absolute processing time.WHY PARALLEL EXECUTION? The ultimate goal: scalability Imagine that your task is to count the number of cars in your street. using 2x the resources reduces the processing time from 360 to 180. both cases of linear scalability. If this is the case then your operations scales linearly. If you allocate twice the number of resources and achieve a processing time that is half of what it was with the original amount of resources. The database is not very different from the counting cars example. you expect to complete the task of counting all cars in a street in approximately half the time compared to when you perform the job all by yourself. The graph does not look linear to you. – Assuming your friend counts equally fast as you do.43 45 40 36 200 150 100 50 0 1x 2x 3x 4x 5x 6x 7x 8x 9x 10x Resources in units of x Figure 1: Processing time as a function of resources for linear scalability. count cars until you meet each other and add the results of both counts to complete the task. 2x the number of resources halves the total processing time. It's just that the absolute performance gain Oracle SQL Parallel Execution Page 6 . and from 2x to 4x down to 90. not a relative speedup factor. 1x 2x 3x 4x 5x 6x 7x 8x 9x 10x 400 350 300 Relative processing time 250 360 180 120 90 72 60 51. – Scenario 1: You can go through the street by yourself and count the number of cars. Figure 1 Below shows graphically how the processing time decreases for a linearly scalable operation. then the operation scales linearly.

The main differentiation is whether or not the physical data layout is used as a base – and static pre-requisite – for dividing. For database processing you may experience a lack of scalability if you don't allocate resources in the correct quantities across the various components. Figure 2: Shared everything versus shared nothing In a shared nothing system CPU cores are solely responsible for individual data sets and the only way to access a specific piece of data you have to use the CPU core that owns this subset of data2. for the sake of simplicity we will discuss them as one core. memory and Input/Output (I/O) all collaborate together. Shared everything – the Oracle advantage Traditionally. if you add CPU resources but you don't add I/O resources then the CPUs may not be able to retrieve the data fast enough to keep processing at full speed. two approaches have been used for the implementation of parallel execution in database systems. In a database there are multiple components involved in processing a query. Now imagine your friend gets tired easily and has to rest regularly throughout the job. the work. thus decreasing with higher number of resources. Most notably CPUs. each having its own maximum processing power. Of course the total amount of time it takes to count the total number of cars reduces. These fundamental approaches are known as shared everything architecture and shared nothing architecture. Maybe you spend two thirds of the original processing time to complete the task. but doubling the resources does not half the processing time. such systems are are also commonly known 2 Some implementations allow a static small number of cores as smallest unit. but we will come back to this in the best practices section. In this case the operation does not scale as well: doubling the resources does not give the expected linear reduction in processing time. the architectural trade-offs are identical Oracle SQL Parallel Execution Page 7 . For example.

as MPP (Massively Parallel Processing) systems. fixed parallelism in their systems in order to perform operations that involve table scans. the data has been pre-partitioned (using Oracle Partitioning). Oracle can use the same optimizations and algorithms shared nothing vendors claim. Oracle can parallelize almost every operation. however. Oracle SQL Parallel Execution Page 8 . In order to achieve a good workload distribution MPP systems have to use a hash algorithm to distribute (partition) data evenly across available CPU cores. independent of the underlying data distribution. Most nonOracle data warehouse systems are MPP systems. If. Thanks to Oracle's shared everything architecture the Oracle Database does not require any pre-defined data partitioning to enable parallelism. As a result MPP systems introduce mandatory. using a superset of parallel execution capabilities over shared nothing vendors. the fixed parallelism completely relies on a fixed static data partitioning at database or object creation time. Oracle's shared everything architecture enables flexible parallel execution and high concurrency without overloading the system.

Operations that can be executed in parallel include: – – – – – – SQL loader and SQL-based data loads Queries RMAN backups Index builds Gathering statistics And more This paper focuses on SQL parallel execution only. The statement returns the total number of customers in the CUSTOMERS table: select count(*) from customers c.a. While the paper focuses on Oracle Database 11g. serial plan Oracle SQL Parallel Execution Page 9 .FUNDAMENTAL CONCEPTS OF ORACLE'S PARALLEL EXECUTION The Oracle Database provides functionality to perform a complex task in parallel. Processing parallel SQL statements When you execute a SQL statement in the Oracle Database it is decomposed into individual steps (a. identified as separate lines in an execution plan. which consists of parallel query. rowsources). parallel DML (Data Manipulation Language) and parallel DDL (Data Dictionary Language). Below is an example of a simple serial SQL statement and its execution plan. ---------------------------------------| Id | Operation | Name | ---------------------------------------| 0 | SELECT STATEMENT | | | 1 | SORT AGGREGATE | | | 2 | TABLE ACCESS FULL| CUSTOMERS | ---------------------------------------- Figure 3: customer count. the information in this paper also applies to Oracle Database 10g and higher. unless explicitly stated. without manual intervention.k.

name.01 Q1.amount from customers c. showing all customer purchase information is shown below: select c.00 Q1. parallel plan Oracle SQL Parallel Execution Page 10 .01 Q1. s.00 | P->S | QC (RAND) Q1.01 Q1.00 | PCWC | Q1.00 Q1.purchase_date. The two plans shown above will change as follows: ------------------------------------------------------------------------------| Id | | | | | | | | Operation 0 | SELECT STATEMENT 1 | 2 | 3 | 4 | 5 | 6 | SORT AGGREGATE PX COORDINATOR SORT AGGREGATE | | | | | Name | | | | | | | | TQ |IN-OUT| PQ Distrib | | | | | | | | | | | | | | ------------------------------------------------------------------------------- PX SEND QC (RANDOM) | :TQ10000 PX BLOCK ITERATOR | TABLE ACCESS FULL| CUSTOMERS Q1.00 Q1. serial plan If you execute a statement in parallel (via mechanisms described later).01 | | | | | | | | | | | | P->S | QC (RAND) | PCWP | | PCWP | | P->P | BROADCAST | PCWC | | PCWP | | PCWC | | PCWP | ----------------------------------------------------------------------------------------- Figure 6: customer purchase .00 | PCWP | ------------------------------------------------------------------------------- Figure 5: customer count. the Oracle Database will parallelize as many of the individual steps in the execution plan as possible and reflects this in the execution plan.A serial example.customer_id = c.00 | PCWP | Q1.01 Q1. s. parallel plan ---------------------------------------------------------------------------------------| Id | Operation | Name | TQ |IN-OUT| PQ Distrib | ---------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | | | | | | | | | | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | PX COORDINATOR PX SEND QC (RANDOM) HASH JOIN PX RECEIVE PX SEND BROADCAST PX BLOCK ITERATOR TABLE ACCESS FULL PX BLOCK ITERATOR TABLE ACCESS FULL | | :TQ10001 | | | :TQ10000 | | CUSTOMERS | | SALES | | | | | | | | | Q1. ---------------------------------------| Id | Operation | Name | ---------------------------------------| 0 | SELECT STATEMENT | | |* 1 | HASH JOIN | | | 2 | TABLE ACCESS FULL| CUSTOMERS | | 3 | TABLE ACCESS FULL| SALES | ---------------------------------------- Figure 4: customer purchase information. sales s where s.

SQL parallel execution in the Oracle database is based on a few fundamental concepts. All the work shown in a parallel plan BELOW the QC in our sample parallel plans (Figure 5. Query Coordinator (QC) and parallel servers SQL parallel execution in the Oracle Database is based on the principles of a coordinator (often called the Query Coordinator – QC for short) and parallel servers. The QC is the session that initiates the parallel SQL statement and the parallel servers are the individual sessions that perform work in parallel. The process acting as the QC of a parallel SQL operation is the actual user session process itself. Oracle SQL Parallel Execution Page 11 .. The QC is easily identified in the parallel execution plans above as 'PX COORDINATOR' (for example ID 1 in Figure 6 shown above). Parallel server processes can be easily identified on the OS level. mainly because we are having additional “logistical” processing steps due to the parallel processing that we did not have before3. The parallel servers are taken from a pool of globally available parallel server processes and assigned to a given operation (the setup is discussed in a later section). The following section discusses these concepts that help you understand the parallel execution setup in your database and read the basics of parallel SQL execution plans. Figure 7: parallel server processes seen on the OS level using 'ps -ef' 3 Parallel plans will look different in versions prior to Oracle Database 10g.. The QC distributes the work to the parallel servers and may have to perform a minimal – mostly logistical – portion of the work that cannot be executed in parallel. for example on Linux they are the oracle processes ORA_P***: oracle oracle oracle oracle 23473 23475 23477 23479 1 1 1 1 0 17:46 ? 0 17:46 ? 0 17:46 ? 0 17:46 ? 00:00:00 ora_p000_linux111 00:00:00 ora_p001_linux111 00:00:00 ora_p002_linux111 00:00:00 ora_p003_linux111 .These plans look quite a bit different than before. For example a parallel query with a SUM() operation requires adding the individual sub-totals calculated by each parallel server. Figure 6) is done by the parallel servers.

imagine the job is to count the total number of cars per car color you go ahead and count the cars. To parallelize Oracle SQL Parallel Execution Page 12 . each one of you potentially sees the same colors and gets a subtotal for the colors. this is equivalent to the operation with the ID 2 in Figure 5.Going back to the example of counting the cars. but not the complete result for the street. if you and your friend are going to cover one side of the road each. this is equivalent to the operations with the ID 4. each of you tells the third person your individual subtotals (ID 3) and he would add them up to the final result (ID 1). memorize all this information and tell it back to the third person (the “person in charge”). and ID 6. – You can do exactly the same on the road that is being done internally in the database with the SQL and execution plan shown in Figure 8: You and your friend will go ahead and count only the cars on your own side. but this poor individual then has to sum up all of the results by himself – what if all cars in the street had a different color? The third person would redo exactly the same work as you and your friend. You could go ahead. – QC gets the subtotals and adds them up Parallel servers retrieve subtotals Figure 8: QC and parallel servers Producer/consumer model Continuing with our car counting example. Finally. illustrated in Figure 8.telling you and your friend – the two parallel servers . there would be a third person – the QC . where ID 5 is the equivalent to tell each one of you to only count the cars on your side of the road (details to follow in the granule section). This is the hand-over from the parallel servers (processes doing the actual work) to the QC for final “assembly” of the result for returning it to the user process. ID 5. Well.

They both walk in the middle of the road. but matches exactly the number of people that are counting cars. the first slave set (Q1. redistribute it based on the color information. you ask two more friends to help you out4. the other one of all bright colors (assuming this “car color separation” is approximately splitting the information in half). Oracle SQL Parallel Execution Page 13 . you tell the person that is in charge of this color about the new encounter – you produce the information. each with two friends doing a part of the job. both color counting friends tell their result the person in charge and you're done.00) is reading table CUSTOMERS in parallel and producing rows that are sent to slave set 2 (Q1.assuming that all “car scanners” have equally distributed incremental results on a continuous base . That's how the database works: In order to execute a statement in parallel efficiently sets of parallel servers work in pairs: one set is producing rows (producer) and one set is consuming the rows (consumer). we had two sets. As shown in Figure 9.01) that consumes these records and joins it with table SALES. At the end. as shown in Figure 9. rows have to be redistributed based on the join key to make sure that matching join keys from both tables are sent to the same parallel server process doing the join. Whenever data is distributed from 4 Note that the number of additional friends is not related to the number of distinct car colors. working hand in hand. using more friends to count the car colors would leave all three of them without work for 30% of their time (on average).the counting. For example for the parallel join between the SALES and CUSTOMERS tables. Slave set 2 “consumes” the records and joins it with table sales Slave set 1 “produces” rows from table customers Figure 9: Producer and Consumer Operations (rowsources) that are processed by the same set of parallel servers can be identified in an execution plan by looking in the 'TQ' column. and the color counter is consuming the information. We want to use our additional friends in the most optimal manner and . Whenever you count a new car.having as many “car color counters” keeps them continuously busy as well. one of them taking count of all dark colors. In this example one set of parallel servers reads and sends the data from table CUSTOMERS (producer) and another set receives the data (consumer) and joins it with table SALES.

'Block Iterator' is the operation name for block-based granules Figure 10: Block-based granule in the customer count example. Unlike all other systems. Granules A granule is the smallest unit of work when accessing data. The basic mechanism the Oracle Database uses to distribute work for parallel execution is block ranges on disk – so-called block-based granules. Oracle can – and will . Access to the underlying objects is divided into a large number of granules which are given out to parallel servers to work on (and when a parallel server finishes the work for one granule the next one is given out). This methodology is unique to Oracle and is not dependent on whether the underlying objects have been partitioned.a producers to consumers you will also see an entry of the form :TQxxxxx (Table Queue x) in the 'NAME' column.a. this is the most fundamental architectural difference between Oracle and all other major database products on the market. the number of parallel servers working on an individual task). slave set) for a parallel operation. The only case when parallel servers do not work in pairs is if the statement is so basic that one set of parallel servers can complete the entire statement in parallel. then 8 parallel server processes will be used for this statement. This has a very important consequence for the number of parallel servers that are spawned for a given parallel operation: the producer/consumer model expects two sets of parallel servers (a.k.choose this smallest unit of work solely dependent on a query's requirements. so the number of parallel server processes is twice the number of the requested Degree Of Parallelism (DOP. The number of granules is always much higher than the Oracle SQL Parallel Execution Page 14 . For example. which from a storage perspective means that any CPU core in a configuration can access any piece of data. For example select count(*) from customers requires only one parallel server set (see Figure 5). Please disregard the content of the other columns for now. Oracle Database uses a shared everything architecture. if the parallel join in Figure 9 runs with DOP 4.

The existing data volume – the street – is subdivided into physical pieces on which the parallel servers – you and your friend – are working on independently. there are some operations that can benefit from the underlying data structure and leverage individual partitions as granules of work. or. however. The most common operations that use partitionbased granules are partition-wise joins which will be discussed later. Remember the last car example? The car color mattered. this is one of the most fundamental principles of parallel processing.requested DOP in order to get a good distribution of the workload between parallel servers. The fundamental difference and advantage of Oracle's capabilities. Data redistribution is not unique to the Oracle Database. in the case of parallel execution across multiple machines in a Real Application Clusters (RAC) database. You redistributed the information about the amount of cars per color to the additional two friends based on their color responsibility. Although block-based granules are the basis to enable parallel execution for almost any operation. but you don't know – or even control – what color car is parked where on the street. the operation 'PX BLOCK ITERATOR ' shown in Figure 11 literally is an iteration over all generated block range granules. is that parallel data access (discussed in the granules section earlier) and therefore the necessary data Oracle SQL Parallel Execution Page 15 . enabling them to do the total counting for the colors they're in charge of. Of course in the latter case interconnect communication is used for the data redistribution while shared-memory is used for the former. As the first parallel step of the parallel processing. Data redistribution is required in order to perform operations such as parallel sorts. you can not influence this behavior. aggregations and joins. With partition-based granules only one parallel server performs the work for all data in a single partition. At the block-granule level there is no notion of knowledge about the actual data content of an individual granule. Data redistribution takes place between individual parallel servers either within a single machine. Data has to be redistributed as soon as a subsequent operation relies on the actual content.could be considered the equivalent of a block-based granule. the Oracle Database decides whether block-based on partition-based granules lead to a more optimal execution. one side of the street – or even a block of a long street . Data redistribution Parallel operations – except for the most basic ones – typically require data redistribution. In the car counting example. being used by every product that provides parallel capabilities. Based on the SQL statement and the degree of parallelism. In fact. between parallel servers on multiple machines. The Oracle Optimizer considers partition-based granules if the number of (sub)partitions accessed in the operation is at least equal to the DOP (and ideally much higher if there may be skew in the sizes of the individual (sub)partitions).

parallel operations do not always have to use interconnect communication. Shared-nothing (MPP) database systems also require data redistribution unless operations can take advantage of partition-wise joins (as explained further down in this section). Figure 11: Serial join based on two full table scans. The database uses full table scans to access both tables. accessible to any parallel server. Figure 11 depicts the serial join5. This complexity has deliberately been left out from the images. thus avoiding a potential bottleneck at the interconnect channel. 5 Please note that the figures in this section represent logical diagrams to explain data redistribution. such as indexes or materialized views.redistribution are not constrained by any given hardware architecture or database setup. In shared-nothing systems parallel operations that cannot benefit from a partition-wise join – such as a simple three-way table join on two different join keys . Oracle SQL Parallel Execution Page 16 .always make heavy use of interconnect communication. In an actual database environment data would typically be striped across multiple physical disks. The following section will explain Oracle's data redistribution capabilities using the simple example of table joins without any secondary data structures. Because the Oracle Database also enables parallel execution within the context of a node. In this example we assume two large tables CUSTOMERS and SALES are involved in the join. For a serial join the single serial session (red arrows) can perform the full join because all matching values from the CUSTOMERS table are read by one process. Serial join In a serial join a single session reads both tables and performs the join.

Individual parallel servers work on data ranges so that the QC does not have to do any sorting but only to present the individual parallel server results in the correct order. Hash (re)distribution is the basic parallel execution enabling mechanism for most data warehouse database system. a redistribution of rows will become necessary. Instead-of redistributing rows from both result sets the database sends the smaller result set to all parallel servers in order to guarantee the individual servers are able to complete their join operation. BROADCAST: Broadcast redistribution happens when one of the two result sets in a join operation is much smaller than the other result set. RANGE: Range redistribution is generally used for parallel sort operations.Parallel joins Processing the same simple join in parallel. – – Oracle SQL Parallel Execution Page 17 . Parallel servers scan parts of either table based on block ranges and in order to complete the join. Both tables are read in parallel by both the red and green process (using block-range granules) and then each parallel server has to redistribute its result set based on the join key to the subsequent parallel join operator. The small result set may be produced in serial or in parallel. Figure 12: Data redistribution for a simple parallel join. There are many data redistribution methods. rows have to be distributed between parallel servers. Figure 12 depicts the data redistribution for a parallel join at a DOP 2. represented by the green and red arrow respectively. The following 5 are the most common ones: – HASH: Hash redistribution is very common in parallel execution in order to achieve an equal distribution of work for individual parallel servers based on a hash distribution. most notably MPP systems.

LOCAL redistribution is an optimization in RAC to minimize interconnect traffic for inter-node parallel queries. If both tables are equipartitioned on the join key the database may use a full partition-wise join. It can also be used in an early stage of a query when no redistribution constraints are required. For example you may see a HASH LOCAL redistribution in an execution plan indicating that the row set is produced on the local node and only sent to the parallel servers on that node.– KEY: Key redistribution ensures result sets for individual key values to be clumped together. Otherwise a partial partition-wise join may be used in which one of the tables is dynamically partitioned in memory followed by a full partition-wise join. A partition-wise join does not require any data redistribution because individual parallel servers will work on the equivalent partitions of both joined tables. – As a variation on the data redistribution methods you may see a LOCAL suffix in a parallel execution plan on a Real Application Clusters (RAC) database. ROUND ROBIN: Round-robin data redistribution can be the final redistribution operation before sending data to the requesting process. The execution plan for the simple parallel join illustrated in Figure 13. This is an optimization that is primarily used for partial partition-wise joins (see further down) to ensure only one side in the join has to be redistributed. Parallel partition-wise joins If at least one of the tables accessed in the join has been partitioned on the join key the database may decide to use a partition-wise join. Oracle SQL Parallel Execution Page 18 . Data redistribution is shown in the SQL execution plan in the 'PQ Distrib' column. HASH redistribution on join column HASH redistribution on join column Figure 13: Data redistribution for a simple parallel join using a HASH redistribution.

-------ID NOT NULL NAME NOT NULL YEAR_OF_BIRTH EMAIL_ADDRESS STREET_NUMBER STREET_NAME CITY STATE_PROVINCE ZIP_CODE COUNTRY NOT NULL Type -----------NUMBER(38) VARCHAR2(60) NUMBER(38) VARCHAR2(50) VARCHAR2(10) VARCHAR2(60) VARCHAR2(60) VARCHAR2(40) VARCHAR2(10) VARCHAR2(40) Oracle SQL Parallel Execution Page 19 . As shown in Figure 14. Operations that do not use partition-wise operations in an MPP system often do not scale well. and for any pair of partitions of these two tables. the choice of partitioning (distribution) in a shared nothing system is critical as well as the access path to the tables. As a result. The same is true the green parallel server process. too. the equipartitioning of both tables on the join key guarantees that there will no matching rows for the join outside of these two partitions. Note that partition-wise joins use partitionbased granules rather than block-based granules. Following are the relevant table definitions: SQL> desc customers Name Null? ----------------.Figure 14: Full partition-wise joins do not require data redistribution. Enabling parallel execution in Oracle Consider the following example. Shared nothing systems typically scale well as long as they can take advantage of partition-wise joins. The partition-wise join is the fundamental enabler for shared nothing systems. the red parallel process reads data partition one of the CUSTOMERS table AND data partition one of the SALES table. Your database stores historical sales data and customer data. The red parallel process will always be able to complete the full join by reading just these matching partitions.

By default the Oracle Database is configured to support parallel execution out-of-the-box. In order to execute an operation in parallel. is to execute in parallel. not in use by another parallel operation).SQL> desc sales Name ----------------PURCHASE_DATE ITEM_ID CUSTOMER_ID STORE_ID QUANTITY AMOUNT TAX Null? -------NOT NULL NOT NULL NOT NULL Type -----------DATE NUMBER(38) NUMBER(38) NUMBER(38) NUMBER(38) NOT NULL NUMBER(7.purchase_date between to_date('01-NOV-2007'.country = 'United States of America' group by c. parallel servers must be available (i. You want to know the total revenue for the last two months of 2007 in the United States. parallel_min_servers: the minimum number of parallel servers that are always started when the database instance is running.'DD-MON-YYYY') and to_date('31-DEC-2007'.amount) revenue from customers c .id and s. assuming there are surplus resources available.state_province / You run the query without enabling parallel execution and let's say it takes 10 minutes to execute the query.2) NUMBER(7. sum(s. By default the value for parallel_max_servers is derived from other database settings and will be discussed later in this paper.2) The tables are initially not partitioned and there are no indexes on the tables.'DD-MON-YYYY') and c. by state. Going back to the example of counting cars and using help from friends: parallel_max_servers is the maximum number of friends that you can call for help. parallel_min_servers enables you to avoid any delay in the – Oracle SQL Parallel Execution Page 20 .customer_id = c.e. The end user who runs the query expects a faster response time (less than 3 minutes) and one way to achieve this. The most relevant database initialization parameters are: – parallel_max_servers: the maximum number of parallel servers that can be started by the database instance.state_province . sales s where s. The following query retrieves this result: select c.

alter table customers parallel .state_province / This method is mainly useful for testing purposes. sum(s. 3) Use alter session force parallel query . 1) Enable the table(s) for parallel execution: alter table sales parallel . Again going back to the example of counting cars: parallel_min_servers is the number of friends that are there with you that you don't have to call in order to start the job of counting the cars. This method is useful if your application always runs in serial except for this particular session that you want to execute in parallel.purchase_date between to_date('01-JAN-2007'.'DD-MON-YYYY') and to_date('31-DEC-2007'. Use this method if you generally want to execute operations accessing these tables in parallel.state_province . or if you have a particular statement or few statements that you want to execute in parallel. but most statements run in and s.----------parallel_max_servers integer VALUE -------80 There are three ways to enable a query to execute in parallel. Verify that parallel execution is enabled for your database instance (connect to the database as a DBA or SYSDBA): SQL> show parameter parallel_max_servers NAME TYPE --------------------.amount) revenue from customers c . sales s where s.execution of a parallel operation for the startup of parallel servers. A batch operation in an OLTP application may fall into this category. Oracle SQL Parallel Execution Page 21 .'DD-MON-YYYY') and = 'United States' group by c.customer_id = c. 2) Use a parallel hint. select /*+ parallel(c) parallel(s) */ c.

but if too many operations take this approach. Multi-user concurrent workload Most production environments have a multi-user workload. All processes in the database require resources. CPU and I/O resources. so if you run your query on an environment with 2 CPU cores the DOP will be 4. End-users will expect a fair amount of resources to be allocated to their operation in order to get predictable response times. Consider the workload to which you want to apply parallel execution to get optimum use of the system while satisfying your requirements.All of these three methods enable the so-called DEFAULT parallel capabilities where Oracle chooses the DOP. Database initialization parameter parallel_max_servers is a good example of one of these limits. The original query which initially took 10 minutes to complete should complete within less than 3 minutes at DOP of 4. you may wonder where's the limit of parallel processing. In a single-user workload all resources can be allocated to improve performance for the single operation. Obviously. Controlling SQL Parallel Execution in Oracle Now that you know how to enable parallel execution and you know the concepts behind Oracle's parallel execution model. Oracle will spawn two parallel server processes per each core on most can't use more resources than you have. Oracle Database has built-in limits and settings to prevent system overload and ensure the database remains available to applications. Understand your target workload Parallel execution can enable a single operation to utilize all system resources. An example for this type of workload is a large overnight batch load that populates database tables or gathers statistics. The system will not allocate more parallel servers to users than the setting of this initialization parameter. While this may not be a problem in certain scenarios there are many cases in which this would not be desirable. assuming sufficient resources are available. Also benchmark situations often measure maximum performance in a single-user workload. including memory and while active. In a multi-user environment. workload resources must be divided amongst concurrent operations. Users concurrently execute queries – often ad-hoc type queries – and/or concurrent data load operations take place. By default. Oracle SQL Parallel Execution Page 22 . Single-user workload The single-user workload is a workload in which there is a single operation executing on the database and the objective is for this operation to finish as fast as possible. the system may soon be starved for resources . you can use more resources to speed up response times.

Controlling the degree of parallelism Oracle's parallel execution framework enables you to either explicitly chose . Adaptive parallelism When using Oracle's adaptive parallelism capabilities. a specific DOP can be requested from the Oracle database.or even enforce . alter table sales parallel 16 .ora parameter parallel_threads_per_cpu. a discussion of these exceptions is beyond the scope of this paper. Fixed Degree Of Parallelism (DOP) Unlike the DEFAULT parallelism. DEFAULT parallelism uses a formula to determine the DOP based on the system configuration6 (typically the DOP is 2 x [number of CPU cores]. you can set a fixed DOP at a table or index level: alter table customers parallel 8 . In a system that makes aggressive use of parallel execution by using a high DOP the adaptive algorithm will throttle down with only few operations running in parallel. So. DEFAULT parallelism In the earlier example of our parallel query we used so-called DEFAULT parallelism. on a four node cluster with each node having 8 CPU cores. users may experience inconsistent response times. the default DOP would be 2 x 8 x 4 = 64. In this case queries accessing just the customers table use a requested DOP of 8. and queries accessing the sales table request a DOP of 16. Oracle is using the higher DOP7. an OS specific parameter that is set to two on most platforms Some statements do not fall under this rule. such as a parallel CREATE TABLE AS SELECT. the database will use an algorithm at SQL execution time to determine whether a parallel operation should receive the requested DOP or be throttled down to a lower DOP. DEFAULT parallelism targets the single-user workload. Oracle SQL Parallel Execution Page 23 . whenever different DOPs are specified. Using solely the adaptive 6 7 We are oversimplifying here for the purpose of an easy explanation. The DEFAULT algorithm was designed to use maximum resources assuming the operation will finish faster if you use more resources. For example. A query accessing both the sales and the customers table will be processed with a DOP of 16 and potentially allocate 32 parallel servers (producer/consumer). The multiplication factor of two is derived by the init. While the algorithm will still ensure optimal resource utilization. in a cluster configuration 2 x [number of CPU cores] x [number of nodes]). In a multi-user environment DEFAULT parallelism will rapidly starve system resources leaving no available resources for other users to execute in parallel.a specific degree of parallelism (DOP) or to rely on Oracle to control it.

128) */ count(*) from sales s * ERROR at line 1: ORA-12827: insufficient parallel query slaves available If there are insufficient parallel query servers available – in this example less than 64 parallel servers for a simple SQL statement (or less than 128 slaves for a more complex operation. However if you start at a low DOP – either as a result of adaptive parallel execution or because there were simply not enough parallel servers available . Adaptive parallelism is controlled through the database initialization parameter will see ORA-12827 and the statement will not execute.parallelism capabilities in an environment that requires deterministic response times is not advised. Controlling the usage of parallelism Depending on your expected workload pattern you might want to ensure that Oracle's parallel execution capabilities are used most optimally for your environment. SQL> select /*+ parallel(s. This parameter controls the minimal percentage of parallel server processes that must be available to start the operation. use the initialization parameter parallel_min_percent. meaning that Oracle will always execute the statement. involving producers and consumers) . Guaranteeing a minimal DOP Once a SQL statement starts execution at a certain DOP it will not change the DOP throughout its execution.128) */ count(*) from sales s .it may take a very long time to complete the execution of the SQL statement. select /*+ parallel(s. irrespective of the number of available parallel server processes. If the completion of a statement is time-critical then you may want to either guarantee a minimal DOP or not execute at all (and maybe warn the DBA or programmatically try again later when the system is less loaded). it defaults to 0. (a) to control the usage of parallelism and (b) to ensure that the system does not get overloaded while adhering to the Oracle SQL Parallel Execution Page 24 . You can capture this error in your code and retry later. if you want to ensure to get at least 50% of the requested parallel server processes for a statement: SQL> alter session set parallel_min_percent=50 . This implies two basic tasks. For example. To guarantee a minimal DOP.

Tables and/or indexes in the select statement accessed have the parallel degree setting at the object level. The requested DOP for this query is 16. Oracle SQL Parallel Execution Page 25 . For example.potential different priorities for different user classes in the case of mixed workload environments. For example: alter session force parallel query parallel. if your Figure 15: Restricting parallel execution in Oracle Database Control. For a query that processes objects with different DOP settings. and restrict parallel execution for some users. If objects have a DEFAULT setting then the database determines the DOP value that belongs to DEFAULT. The requested DOP for any operation in that session will be DEFAULT parallelism. and no user in a resource group (using a specific resource plan) will ever be able to run with a higher DOP than the resource group's maximum. in the following priority order: – Parallelism was requested in a hint: select /*+ parallel(s. – – Using Oracle Database Resource Manager Oracle Database Resource Manager (DBRM) enables you to group users based on characteristics. The DOP was requested in an alter session command. DBRM is the ultimate last instance in determining the maximum degree of parallelism.16) */ count(*) from sales s . the object with the highest parallel degree setting accessed in the query determines the requested DOP. Whether or not a SQL operation is running parallel and what DOP is chosen is determined based on the following rules.

Parallel execution is very I/O intensive by nature. Figure 15 shows an Enterprise Manager Database Control screenshot restricting parallel execution to a DOP of 4 for a resource plan named 'DW_USERS'. incl. I/O and memory. CPUs. your SQL will run with a DOP of 4. making Oracle SQL Parallel Execution Page 26 . SQL parallel execution is also a heavy consumer of memory. Start with a balanced system A good foundation is the basis for successful use of SQL parallel execution. All system resources. For example.then you have to size the interconnect appropriately. DBRM can control the maximum number of active sessions for a given resource group. So for the shown resource plan 'DW_USERS' a maximum of 4 sessions are allowed to be active. if you want to keep 4 CPU cores busy in such a configuration. resulting in a total maximum resource consumption of 4 (sessions) x 4 (DOP) x 2 (slave sets) = 32 parallel server processes. In the case of SQL parallel execution the foundation consists of the hardware configuration that you use to run your database. Note that I/O throughput requirement has to be guaranteed throughout the whole hardware system: the Host Bus Adapters (HBAs) in the compute nodes. For example. If you use RAC and you use inter-node parallel query – parallel operations that spawn multiple nodes . any switches you use. then the entire I/O subsystem should be able to support 800 MB/s sustained for optimum performance. The weakest link is going to limit the performance and scalability of operations in this configuration. or that you may revisit if you already use it to ensure you get the optimum performance out of your system. If you rely on storage shared with other applications then the throughput performance for your database is not guaranteed and you will likely see inconsistent response times for your parallel operations. Per CPU core you should have at least 4 GB of RAM. If you use Real Application Clusters (RAC) then you have to also size the interconnect appropriately. then plan the system conservatively. if your system is intended to run an I/O intensive workload. and the I/O subsystem. storage controllers and physical spindles. Furthermore. so every built-in imbalance in a system may have a bigger and more visible impact on the overall scalability of the hardware platform than for less I/O intensive workloads.resource plan has a policy of using a maximum DOP of 4 and you request a DOP of 16 via a hint. ORACLE SQL PARALLEL EXECUTION BEST PRACTICES This section lists best practices that you should bear in mind when you consider using SQL parallel execution. should be able to support the use of SQL parallel execution. some of the data redistribution is going to happen over the interconnect. assuming that every CPU core can process approximately 200 MB/s sustained.

The Oracle Database software will not achieve better performance than the hardware configuration can achieve. SQL parallel execution is typically very I/O intensive. any physical disk may be able to sustain 20-30 MB/s for large random reads.A.A.) – use ASM Conservatively. you should realize that you need a lot of physical spindles to get good performance for database operations running in parallel.M. and use it as a baseline if later you think the performance is insufficient. so you want to measure the maximum I/O performance you can achieve without the Oracle Database.e. For high availability you should use a RAID configuration (storage-based RAID1 or RAID5 are commonly used) to ensure you can survive the failure of a single disk. Oracle has even more optimizations to minimize interconnect traffic as classical shared nothing architectures – such as choosing to run a parallel operation or a subset of it within a single node .com/technology/software/tech/orion/index.10 physical disks). http://www. this might work well for your single-user home video archive. You can use ORION8 (ORacle I/O Number calibration as crucial as the overall I/O capabilities. Make sure to calibrate the configuration in the way Oracle will use it (how the data will be laid out across storage devices) and use a calibration workload that resembles the type of workload the Oracle Database will perform when running SQL statements in parallel (typically large random I/Os). Do not use a single 1 TB disk for your 800 GB database.) methodology using a stripe size of 1 because you will not get good performance running operations in parallel against the database. Stripe And Mirror Everything (S. Calibrate your configuration You should set a baseline for the performance you expect out of the Oracle Database. 8 . Use (multiple) 10 GigE or Infiniband interconnect if you plan to use inter-node parallel query. For many years Oracle has recommended its users to use the Stripe And Mirror Everything (S.html Oracle SQL Parallel Execution Page 27 . The way to utilize multiple physical spindles with Oracle's shared everything architecture is to stripe across multiple devices.E. but not for a database leveraging parallel query with multiple users. Hence you should know what the operating system can achieve before you introduce the Oracle software. Considering that you need about 200 MB/s to keep a single CPU core busy (i.but in the worst case the throughput required on the interconnect for good scalability is at least equal to the throughput going to disk. Such a configuration is 8 Orion is downloadable from the Oracle Technology Network. a free Oracleprovided utility designed to simulate Oracle I/O workloads) or basic operation system utilities (such as the Linux/Unix dd command) to measure the I/O performance for your system. Work with your hardware vendor and Oracle representative to ensure you start with a good foundation.E.

if and when you expand your configuration. shared_pool_size Parallel servers communicate among themselves and with the Query Coordinator by passing messages. data warehouse). so that you will always benefit from all storage devices in your configuration. If you are doing inter-node parallel operations (((2 + (cpu_count X parallel_threads_per_cpu)) X 2) X (cpu_count X parallel_threads_per_cpu)) X parallel_execution_message_size X # concurrent queries Oracle SQL Parallel Execution Page 28 . The messages are passed via memory buffers that are allocated from the shared pool. You should also bear in mind that the majority of operations that execute in parallel bypass the buffer cache. Set database initialization parameters for good performance Once you have ensured a balanced system you install the Oracle Database software and create a database. ASM implements Oracle's S. ASM can be used to store Oracle Database files (including online redo log files and archivelog files). There are a few parameters that you should pay attention to when it comes down to achieving good performance for SQL parallel execution. In order to size your shared pool appropriate you should use the following formulas to calculate the additional overhead parallel servers will put on the shared pool. Memory allocation Large parallel operations may use a lot of execution memory. ASM can also be used to mirror data or it can be used with hardware RAID configurations. When a parallel server is started it will allocate buffers in the shared pool so it can communicate. if there is not enough free space in the shared pool to allocate the buffers the parallel server will fail to start. If the object size is less than 2% of the buffer cache then the cost of the checkpoint to start the direct read is deemed more expensive than just reading the blocks into the cache. ASM will automatically re-balance (re-stripe) the data across all devices. Use ASM if you are using Oracle Database 10g or higher. A parallel operation will only use the buffer cache if the object has been either explicitly created with the CACHE option or if the object size is smaller than 2% of the buffer cache. and you should take this into account when allocating memory to the database. Most importantly.A. ASM will stripe across all devices you present to it in the context of a disk group. included with the database.M. methodology and automatically maintains it as devices are added or removed. reporting.E. without running into hot spots in the storage configuration.relatively simple to set up. Starting with Oracle Database 10g you can use Oracle's Automatic Storage Manager (ASM). providing good performance for pretty much any workload (OLTP.

These rules apply irrespective of whether you use shared_pool_size directly. Do not change the value of this parameter. Oracle attempts to keep the amount of private memory below the target you specified by adapting the size of the work areas. pga_aggregate_target The pga_aggregate_target parameter controls the total amount of execution memory that can be allocated by Oracle. more memoryintensive operations are able to run fully in memory and less will work their way over to disk. parallel_execution_message_size As mentioned above the Parallel servers communicate among themselves and with the Query Coordinator by passing messages via memory buffers. their memory buffers will be allocated “on the fly” from the shared pool. a larger value for parallel_execution_message_size will increase the memory requirement for the shared_pool so if you increase it from 2K to 16K your parallel server memory requirement will be 8 X more. Consequently. Oracle SQL Parallel Execution Page 29 . or sga_target (10g and higher) or memory_target (starting with 11g). For environments that run a lot of parallel operations you should set pga_aggregate_target as large as possible. you indirectly increase the memory allotted to work areas. it’s advisable to reduce the messaging latency by increasing the parallel_execution_message_size (the size of the buffers). cpu_count CPU count is an automatically derived parameter by the Oracle system and is used to determine the default number of parallel servers and the default degree of parallelism for an object. If there are no parallel servers available the operation will actually be executed serially. (((2+ (cpu_count X 2)) X 4) X cpu_count X 2)) X parallel_execution_message_size X # concurrent queries Note the results are returned in bytes. Controlling parallel servers In order for a parallel operation to execute in an optimal fashion there has to be enough parallel servers available. As additional parallel servers are needed.or when you use cross instance parallel operation in a RAC environment. By default the message size is 2K. Ideally you should increase it to 16k (16384). When you increase the value of this parameter. Only the memory needed for the parallel_min_servers will be preallocated from the shard_pool at database startup. A good rule of thumb is to have a minimum of 100MB X parallel_max_servers. However. If you execute a lot of large operations in parallel.

However. A good rule of thumb is to ensure parallel_max_servers is set to a number greater than the “maximum number of concurrent queries * maximum degree of parallelism need by a query”. parallel_max_servers This parameter determines the maximum number of parallel servers that may be started for a database instance. which can significantly impact the execution time. By default the value is 0. Depending on the workload and the user expectations you should set this parameter to true or false. This will ensure that there are ample parallel server processes available for the majority of the queries executed on the system and queries will not suffer any additional overhead of having to spawn extra parallel servers. parallel_adaptive_multi_user This parameter controls whether or not Oracle automatically downgrades parallel operations to proactively to prevent an overloading of the system. The default value on Oracle Database 10g and higher is 10 * cpu_count * parallel_threads_per_cpu. Oracle SQL Parallel Execution Page 30 . parallel_min_servers This parameter determines the number of parallel servers that will be started during database startup. Bear in mind that any additional parallel server processes that are spawned above parallel_min_servers will be killed after they have been inactive for a certain about of time and will have to be re-spawned if they are need again in the future. It is used to calculate the default degree of parallelism for the instance and determines the maximum number of parallel servers if parallel_max_servers is not set. if extra parallel servers are required for additional queries above you average workload they can be spawn “on the fly” up to the value of parallel_max_servers. For predictable response times on a busy server it is better to set this parameter to false. The default is platform-dependent and is adequate in most cases (two on most platforms). should there be demand for them.parallel_threads_per_cpu The parameter describes the number of parallel execution processes or threads that a CPU can handle during parallel execution. Realize that if you set the parameter to true. It is recommended that you set parallel_min_servers to “average number of concurrent queries * maximum degree of parallelism need by a query”. then parallel operations may be downgraded aggressively. By doing this you will ensure every query gets the appropriate number of parallel servers.

Best practices for customers that are using object sizes as the main driving factor for parallelism are commonly aligning the DOP with some kind of step function for parallelism. Set db_file_multiblock_read_count such that when it is multiplied by the block size you end up with 1 MB. then it will not change its DOP throughout the execution. Use parallel execution with common sense While parallel execution provides a very powerful and scalable framework to speed up SQL operations. for example when doing a full table scan. Since parallel execution will bypass the buffer cache and access data directly from disk you want each I/O to be as efficient as possible. and using large I/Os is a way to reduce latency. Use parallelism to achieve your goals. Remember also that once an operation starts at a certain DOP. use db_file_multiblock_read_count=128. You cannot use more resources than you have available. e. for 8K block size.and highly depend on your target workload and business requirements only. whereas they would use parallel servers that you want to be available for operations accessing large tables. If a certain class of queries has to run within 2 minutes don't increase the Oracle SQL Parallel Execution Page 31 . Operations that only hit small tables will not benefit much from executing in parallel. This is the default value for the majority of platforms. disk_async_io For optimum performance make sure you use asynchronous I/Os. you should not forget use some common sense rules.g. – – – objects smaller than 200 MB will not use any parallelism objects between 200 MB and 5GB are using a DOP of 4 objects beyond 5GB are getting a DOP of 32 Needless to say that your personal optimal settings may vary .g.either in size range or DOP . not to exceed them Use parallelism to achieve your business requirements. up to 10s of data blocks) should never be enabled for parallel execution. never forget that while parallel execution might buy you an additional incremental performance boost. it requires more resources and might also have side effects on other users or operations on the same system.Enabling efficient I/O throughput db_file_multiblock_read_count SQL parallel execution is generally used for queries that will access a lot of data. E. not to over-achieve them. Don't enable parallelism for small objects Small tables/indexes (up to thousands of records.

There are specific optimizations between SQL parallel execution and Oracle Partitioning that you should bear in mind when you plan to use these functionalities together: For example.DOP to run them in 30 seconds. The data placement is controlled with additional information about the object. The basis for a good computation is good information about table sizes. such as ranges of order data or hash buckets of customer id information. getting four times the work done and still adhering to your business goals (obviously this is somewhat of a simplification since you should plan for some head room. data distribution etc. If you execute in parallel then using the wrong execution plan may exaggerate the performance issue. Parallel table joins between SALES and CUSTOMERS can now take advantage of full partition-wise joins. Gathering statistics timely is the key to get the statistics right so that the optimizer can generate a good execution plan. Partitioning enables you to store one logical object – a table or index – transparently in several independent physical segments. Remember Figure 1? Assuming linear scalability. The Oracle Database will compute the optimal execution plan for any SQL operation. 9 Oracle Partitioning is an extra licensable option of Oracle Enterprise Edition Oracle SQL Parallel Execution Page 32 . For example: consider two large tables SALES and CUSTOMERS. Ideally (sub)partitions are similar in size which can be achieved by using hash (sub)partitioning on a unique or almost unique column with the number of hash (sub)partitions a power of 2. Hints are hard to maintain and may not give the right behavior over time when objects and business requirements change. and to achieve good performance when accessing large database objects. but the message should be clear). you need four times the number of parallel processes for this speedup example – resources you could give to three additional queries of the same class. Partition the SALES table using composite partitioning RANGE on ORDER_DATE. Ensure statistics are good enough Executing a SQL statement with the wrong execution plan usually results in poor execution performance. Partition the CUSTOMERS table using HASH partitioning on CUSTOMER_ID. Avoid using hints In general you should avoid using hints to enable parallel execution. HASH on CUSTOMER_ID. Combine parallel execution with Oracle Partitioning Oracle Partitioning9 is powerful database functionality that is useful to manage large database objects. two large partitioned tables that can take advantage of parallel partial or full partition-wise joins (as discussed on page 18) can be joined faster than if no partitioning is involved.

Do take into account that inter-node parallel execution may result in a lot of interconnect traffic. then you may be better of restricting parallel execution to a single node or to a limited number of nodes. your interconnect must provide the total I/O throughput of all nodes in a cluster (since all nodes can distribute data at the same point in time with the speed data is read from disk). Use the Enterprise Manager performance page in Database Control or Grid Control to monitor wait events. inter-node parallel execution will not scale with an undersized interconnect. to the query optimizer. It is not recommended to use inter-node parallel execution unless your interconnect satisfies this requirement (or comes very close). The parameter instance_groups is going to be deprecated and only retained for backwards compatibility reasons. relative to the I/O bandwidth from the server to the storage configuration. Alternatively you can use statspack (on Oracle Database 9i) and AWR reports (Oracle Database 10g and higher) to analyze system performance. If you use a relatively weak interconnect. Oracle SQL Parallel Execution Page 33 . so ensure you size the interconnect appropriately. so. By default the Oracle database enables inter-node parallel execution (parallel execution of a single statement involving more than one node). As a general rule of thumb. such as I/O and CPU performance and utilization. the interconnect must be able to support 4 x 1GB/sec = 4GB/sec to scale linearly for operations involving inter-node parallel execution. Monitor parallel execution activity Use database utilities to monitor the activity on your system. focusing on SQL parallel execution if you suspect problems in that area. System statistics describe the system's hardware characteristics. enabling the query optimizer to choose a better execution plan. It is recommended to use services beginning with Oracle database 11g.Starting with Oracle Database 10g also make sure to gather system statistics. each node being able to read 1GB/sec from the I/O subsystem. For more information also see the following section about parallel execution monitoring. Use instance_groups and parallel_instance_groups or database services (starting with Oracle Database 11g) to limit inter-node parallel execution. if you have a four node cluster. System statistics enable the query optimizer to more accurately estimate I/O and CPU costs. Whether or not to use parallel execution in RAC RAC provides an excellent architecture to incrementally scale your hardware configuration as you require system resources. You can use the additional resources to support additional users (and hence reduce the load on the other servers) and/or use the additional resources to directly improve the performance of the operations running on the database.

MONITORING SQL PARALLEL EXECUTION There are several ways to monitor parallel execution. This section discusses various options. the GV$ views are useful in a Real Application Cluster (RAC) environment to extract cluster-wide information. Rather than trying to address the underlying problem by implementing a balanced system. Don't try to solve hardware deficiencies with other features The most common “problem” of parallel execution (besides overloading a system) is that people try to achieve scalability with parallel execution on unbalanced systems – which obviously will not work. Database Resource Manager is an excellent tool to guarantee resources for operations that require a certain response time. not to adhere to a religious approach to only use one set of functionality. people often fight the symptoms. so use specific features to leverage its strengths and to solve your business requirements. but not for building a whole house. While such measurements might alleviate existing deficiencies in the short term. nothing less). Don't ignore other features On the other hand. using a wrench might work for one or two nails. having a hammer and making everything look like a nail is as bad as not having the hammer at all. All of Oracle's warehousing functionality is working together in harmony.g. For example. (G)V$ parallel execution views Specific parallel execution performance views start with (G)V$_PQ and (G)V$_PX. e. While the so-called V$ views give you an instance-specific view. When you need a hammer because you have a nail. but do not forget that there may be other functionality that is equally if not more appropriate to achieve better performance for specific business requirements. they will not fix them. SQL parallel execution is a great way to get better performance for expensive database operations.Use Database Resource Manager Database Resource Manager ultimately decides the final DOP for a parallel SQL operation before executing it. Consider using the Database Resource Manager if you want to restrict users from using unlimited parallelism (and hence overload the system). are creating indexes or additional summary tables. For Oracle SQL Parallel Execution Page 34 . but rather introduce unnecessary complexity and delay solving the problem. embedding a cube-organized materialized view for your multi-dimensional reporting and analysis might deliver a level of performance that would require an orders of magnitude larger hardware to satisfy the same queries using parallel execution and the detail data records. In addition to the columns in the equivalent V$ view the GV$ view contains the instance ID (nothing more.

for a given query there is a single cursor that is executed by all parallel servers. so we will discuss how to identify and interpret the most fundamental parallel execution = 'United States of America' group by c. e. You can get the parallel plan information through various mechanisms. INST_ID ---------1 1 2 2 3 3 4 4 STATUS PX_SERVERS# --------. count(1) px_servers# from gv$px_process group by and s. sum(s.----------AVAILABLE 4 IN USE 12 AVAILABLE 8 IN USE 8 AVAILABLE 6 IN USE 10 AVAILABLE 2 IN USE 14 Interpreting parallel SQL execution plans Starting with Oracle Database 10g.amount) revenue from customers c. or use the advanced workload repository. status order by inst_id.'DD-MON-YYYY') and to_date('31-DEC-2007'. The following shows a portion of the execution plan. sales s where s. The basic plan information will be the same for all these mechanisms. using the EXPLAIN PLAN utility. All parallel execution information is in the single execution plan that is used by every parallel server process.'DD-MON-YYYY') and c.purchase_date between to_date('01-NOV-2007'.10 10 Note that some columns in the execution plan have been removed to improve the readability of this example. status .example if you wanted to know parallel execution activity across a cluster.customer_id = c. you could run: select inst_id . Oracle SQL Parallel Execution Page 35 .state_province . explain plan for select c. select from the cursor cache. Parallel plan without partitioning Initially tables SALES and CUSTOMERS are not partitioned.display).g. select * from table(dbms_xplan. status . namely a partition-wise join.state_province.

the data is redistributed using a HASH redistribution (ID 5) on the group by column. Results are returned to the query coordinator in random order (ID 2). parallel plan Using all the information discussed in the concept section of this paper. – – – Parallel plan with partitioning and partition-wise join Large databases and particularly data warehouses – the types of databases that mostly use parallel execution – should always use Oracle Partitioning. Hash join and hash group by take place in parallel without a need for redistribution (ID 6 and ID 7). whenever a parallel server finishes the computation of its incremental result it is returned to the QC. Partitioning can provide great performance improvements because of partition elimination (pruning) capabilities. After the join.-------------------------------------------------------------| Id | Operation | Name | PQ Distrib | -------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 1 | PX COORDINATOR | | | | 2 | PX SEND QC (RANDOM) | :TQ10002 | QC (RAND) | | 3 | HASH GROUP BY | | | | 4 | PX RECEIVE | | | | 5 | PX SEND HASH | :TQ10001 | HASH | | 6 | HASH GROUP BY | | | | 7 | HASH JOIN | | | | 8 | PX RECEIVE | | | | 9 | PX SEND BROADCAST | :TQ10000 | BROADCAST | | 10 | PX BLOCK ITERATOR | | | | 11 | TABLE ACCESS FULL| CUSTOMERS | | | 12 | PX BLOCK ITERATOR | | | | 13 | TABLE ACCESS FULL | SALES | | -------------------------------------------------------------- Figure 16: customer purchase information per state. Table SALES and CUSTOMERS are now equi-partitioned on the join column – Oracle SQL Parallel Execution Page 36 . since no order by was specified in the SQL statement. but also because parallel execution plans can take advantage of partitioning. Every parallel server process is doing the incremental aggregation of their disjoint data set. Let's recreate the tables SALES and CUSTOMERS as follows: – HASH partitioning on the ID column for the CUSTOMERS table using 128 partitions. you will be able to identify the following parallel processing steps: – The CUSTOMERS table is read in parallel (ID 11) and is then broadcasted to all parallel servers (ID 9) who read the SALES table.

00 | | ---------------------------------------------------------------------------------------- Figure 17: customer purchase information. you do not see the granules for table SALES and CUSTOMERS right away in the plan. from partition 1 to 128 (identified through columns 'Pstart' and 'Pstop') For each HASH partition pair. – Tables SALES and CUSTOMERS are accessed in parallel. a parallel server process joins the table CUSTOMERS and SALES.00 | | | 8 | HASH JOIN | | | | Q1.01 | | | 4 | PX RECEIVE | | | | Q1.00 | HASH | | 6 | SORT GROUP BY | | | | Q1.01 | QC (RAND) | | 3 | SORT GROUP BY | | | | Q1. so Oracle does not have to partition the data for parallel access at runtime. the partitionbased granule iterator is ABOVE the hash join operation in the execution plan. we are joining two equi-partitioned tables leveraging a partitionwise join.00 | | | 10 | TABLE ACCESS FULL | SALES | 1 | 128 | Q1. – No data redistribution is taking place to join tables SALES and CUSTOMERS.would behave like a shared nothing system for this operation.00 | | | 9 | TABLE ACCESS FULL | CUSTOMERS | 1 | 128 | Q1.01 | | | 5 | PX SEND HASH | :TQ10000 | | | Q1. one parallel server process is working on one equivalent partition pair at a given point in time. Besides the known processing steps of parallel execution this new behavior of a partition-wise join is seen in the execution plan in Figure 17. iterating over the existing equi-partitioned hash partition-based granules (ID 7). parallel plan . there would be no data transfer necessary between the compute nodes. In the case of inter-node parallel query.-------------------------------------------------------------------------------------------------- | Id | Operation | Name | Pstart| Pstop| TQ | PQ Distrib | -------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | | | 1 | PX COORDINATOR | | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | | | Q1. A set of parallel servers is working on n partitions at a time (n equals the DOP). the database simply has to iterate over existing partitions. Unlike in previous examples. The simple reason for this because we are now using partition-based granules. Furthermore. The partition-based granules are not only identical for both tables. Oracle SQL Parallel Execution Page 37 . hash partitioning with partition-wise joins Figure 17 shows the execution plan for the same query using the now hash partitioned tables.00 | | | 7 | PX PARTITION HASH ALL| | 1 | 128 | Q1. You can read this operation as “ loop over all hash partitions and process the operations below”. Consequently. but the iteration (processing) of granules is now a processing of pairs of partitions that includes the join as well. and the Oracle database – although built on the shared everything paradigm .

Partition-wise joins can also be leveraged when joining REF Partitioned tables or as so-called partial partition-wise joins when a small table is joined with a significantly larger table and the database enforces a data redistribution to match the partitioning strategy of the larger table. partitioned by month) and CUSTOMER_ID for hash subpartitioning using 128 sub-partitions we still adhere to the condition for a full partition-wise join and the plan would look change only slightly. using PURCHASE_DATE for range partitioning (7 years worth of data. 'syyyy-mm-dd hh24:mi:ss') AND "TIME_ID"<=TO_DATE(' 2007-12-31 00:00:00'. For the sake of focusing on parallel execution only we will not further discuss partition-wise joins for REF partitioned table nor do we discuss partial partition-wise joins."CUSTOMER_ID"="C". providing appr. 'syyyy-mm-dd hh24:mi:ss')) Figure 18: customer purchase information. Besides the benefits from the parallel full partition-wise join a big performance improvement is achieved through partition elimination. the Oracle database analyzes all existing predicates in the query to see whether some partitions can be ruled out from the processing completely. shown in pstart/pstop of ID 10)."ID") 9 .filter("TIME_ID">=TO_DATE(' 2007-11-01 00:00:00'.-------------------------------------------------------------------------------------| Id | Operation | Name | Pstart| Pstop | PQ Distrib | -------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | | 1 | PX COORDINATOR | | | | | | 2 | PX SEND QC (RANDOM) | :TQ10001 | | | QC (RAND) | | 3 | HASH GROUP BY | | | | | | 4 | PX RECEIVE | | | | | | 5 | PX SEND HASH | :TQ10000 | | | HASH | | 6 | HASH GROUP BY | | | | | | 7 | PX PARTITION HASH ALL | | 1 | 128 | | |* 8 | HASH JOIN | | | | | |* 9 | TABLE ACCESS FULL | CUSTOMERS | 1 | 128 | | | 10 | PX PARTITION RANGE ITERATOR| | 72 | 73 | | |* 11 | TABLE ACCESS FULL | SALES | 9089 | 9344 | | -------------------------------------------------------------------------------------Predicate Information (identified by operation id): --------------------------------------------------8 – access("S".752 partitioned. However the query against the new partitioned tables returns even faster than before. a 40x performance improvement."COUNTRY"='United States of America') 10 . we only have to access 256 out of 10.752 subpartitions in total. the composite range-hash partitioned table SALES has 84 x 128 =10. Oracle SQL Parallel Execution Page 38 . Analyzing the filter predicate on purchase_date leads to a reduction down to two range partitions (#72 and #73. In our case. as shown in Figure 18 above. If we change the SALES table to become a composite RANGE–HASH partitioned table. parallel plan with PWJ A full partition-wise join only requires the partitioning strategy of the join column(s) to be identical.filter("C".

wait events. The most common PX events deal with the message (data) exchange of the producer/consumer model: in order to mitigate the waits the parallel execution infrastructure uses buffers: producers fill a buffer and consumers read the buffer. If a significant portion of the workload consists of SQL statements executing in parallel then it is typical to see a high CPU utilization and/or significant user I/O waits. The mechanism works both ways to ensure efficient processing. This screen is useful to identify what the system workload looks like at any point in time. It is easy to figure out whether the system is using a lot of CPU resources or whether it is waiting on a particular resource and if so. Wait events The main performance screen in Oracle Enterprise Manager Database Control or Grid Control – starting with Oracle Database 10g – shows a graph of wait events over time. As a result of this model you will very likely see wait events in the database instance that are due to producers waiting for consumers to accept data (PX Deq Credit: send blkd) or consumers waiting for producers to produce data (PX Deq Credit: need buffer).Oracle Enterprise Manager Oracle Enterprise Manager Database Control 11g provides new monitoring capabilities useful from a parallel execution perspective. The wait events due to the producer/consumer Oracle SQL Parallel Execution Page 39 . what resource that is. Figure 19: Oracle Enterprise Manager Database Console 11g performance page . The functionality will also be available in Oracle Enterprise Manager Grid Control 11g. The parallel execution workload shows a lot of I/O waits and not a very high CPU utilization on the system. Figure 19 shows an Oracle Enterprise Manager Database Control screenshot of the performance page focused on the graph with wait events.

. Input/Output (I/O) monitoring Almost all SQL statements executing in parallel will read data directly from disk rather than out of memory. Figure 20: Detailed I/O page in OEM 11g Database Console for a parallel DML workload. Oracle SQL Parallel Execution Page 40 . An increase of the idle parallel execution events can often be considered a symptom of a performance problem rather than the cause. and for a coordinator to be able to get the parallel servers it needs. such as I/O waits. Other wait events that you might see are related to parallel server startup/shutdown. an increase of consumers waiting for producers to produce data (PX Deq Credit: need buffer) very likely indicates a performance problem of slow IO. As a result parallel statements can be very I/O intensive. For example.model are unavoidable to a large extent and don't really hurt performance (the wait events fall in the “idle” wait class). As mentioned earlier. Oracle Enterprise Manager Database Control 11g provides I/O throughput information on the main performance page – on the “I/O tab” – as well as on the detailed I/O pages. In a statspack output or an Analytic Workload Repository (AWR) report you will see all parallel execution wait events reported. in case of having a producer operation that involves disk IO (e. a parallel full table scan). unavoidable events due to the additional process communication in a parallel environment. These wait events should be rare and should not take up a lot of time in a production environment. or high CPU utilization. Generally it is not parallel execution specific wait events that may cause slow system performance but rather waits introduced by the workload running in parallel.g. most of the parallel execution (PX) events are either idle wait events or non-tunable.

.The example in Figure 20 shows the I/O page for a parallel DML workload. Oracle SQL Parallel Execution Page 41 . Figure 21: Parallel execution monitoring in OEM 11g Database Console. Figure 21 shows a screenshot of the Parallel Execution tab on the performance page in Oracle Enterprise Manager 11g Database Control. If parallel SQL operations are bottlenecked by I/O it is usually because the maximum throughput (MB/s) has been reached rather than the maximum I/O operations per second (IOPS). Parallel execution monitoring Oracle Enterprise Manager Database Control 11g also introduced parallel execution monitoring on the performance page. The screens help you identify whether the system is running a large number of statements in parallel and whether the majority of the resources are used for few statements running at a large DOP versus a large number of statements running at a lower DOP. For a predominantly parallel query environment you expect the majority of the throughput (in MB/s or GB/s) from large reads. A lot of the I/Os per second are for the database writer and a significant portion of the throughput is large writes.

7 is not yet available. Figure 22: Monitoring a parallel execution query in near real-time. Oracle Enterprise Manager Grid Control 11g will also provide the graphical interface. 12 As of publication Oracle Database 11. A parallel statement shows the parallel server sets. The example shows screenshots of an early version of database console on a development version. The SQL Monitoring screen shows the execution plan of a long-running statement or a statement that is running in parallel.SQL monitoring Oracle Database 11g introduced a new dynamic view GV$SQL_MONITOR11.1.7 there is a graphical interface to GV$SQL_MONITOR. In near real-time (the default refresh cycle is 5 seconds) you can monitor which step in the execution plan is being worked on and if there are any waits (see Figure 22). The SQL Monitoring screens also provide information about the parallel server sets and work distribution between individual parallel servers on the “Parallel” tab (see Figure 23).0. Starting with Oracle Enterprise Manager database console 11. With Oracle Database 11. The SQL Monitor output is extremely valuable to identify which parts of an execution plan are expensive throughout the total execution of a SQL statement.1.6 you can only use textual output from the view.0.0. The examples and screenshots in this section show Oracle Enterprise Manager 11.1.7 database console on a single instance 2 CPU database server12. Oracle SQL Parallel Execution Page 42 .0. 11 Oracle Database Enterprise Manager Tuning Pack must be licensed in order to access (G)V$SQL_MONITOR. This view enables real-time monitoring of long-running SQL statements and all parallel SQL statements without any overhead.1.

Figure 23: Parallel server sets and work distribution in SQL Monitoring. Oracle SQL Parallel Execution Page 43 . Ideally you see an equal distribution of work across the parallel servers. Use this information to identify at statement level what resources are used most intensely. The third tab in the SQL Monitoring interface shows the activity for the statement over time in near real-time (see Figure 24). Figure 24: Wait activity in SQL Monitoring. The statement will have to wait for the parallel server performing most work to complete. If there is a skew in the distribution of work between parallel servers in one parallel server set then you have not achieved optimal performance.

or even be serialized. if any. When you plan to upgrade Oracle Database 9i you should review your SQL parallel execution settings. but it also means that the system will end up using a lot more parallel resources than it used to. If on Oracle Database 9i you used hints to enable SQL parallel execution If you always use hints. In any and all cases you should validate the parallel execution behavior between Oracle Database 9i and another release through a representative test of the workload on your production system. In the worst case operations that would run in parallel in Oracle Database 9i are now going to be starved for parallel resources and may be either be running at a lower DOP. then you should be aware of some changes in the SQL parallel execution infrastructure. to enable SQL parallel execution on Oracle Database 9i then there is little to worry about when upgrading. then you should look at the operations that are executed in the sessions that enable or force parallel execution. More parallel operations The internal code rewrite introduced with Oracle Database 10g lifted a number of parallel execution restrictions that existed in Oracle Database 9i. If on Oracle Database 9i you used session settings to enable SQL parallel execution If you always use only the session setting to enable parallel execution. but if it does. mainly in terms of getting more statements parallelized and the chance of using more parallel resources on a system. it will do so in Oracle Database 10g and beyond as well. Many parallel execution restrictions that existed in Oracle Database 9i have been lifted starting with Oracle Database 10g. and you plan to upgrade to Oracle Database 10g or beyond. Expect more operations to execute in parallel after an upgrade to Oracle Database 10g or beyond.UPGRADE CONSIDERATIONS COMING FROM ORACLE DATABASE 9I Oracle Database 10g introduced a completely rewritten internal parallel execution infrastructure. As a result you might see that some operations that were running in serial are now executed in parallel when you use parallel settings at the table level. after an upgrade. and nothing but hints. that can lead to different execution times for parallel operation or a different system utilization between Oracle Database 9i and higher releases. You should verify whether every operation with parallel hints actually runs in parallel in Oracle Database 9i. If there are only parallel operations on Oracle Database 9i in your parallel enabled sessions then you would expect minimal changes. This may be great for the execution time of these operations that did not run in parallel before. If you are using SQL parallel execution on Oracle Database 9i. Oracle SQL Parallel Execution Page 44 . These changes may result in unexpected changes. This problem is even exaggerated if your system already runs close to the resource limit with Oracle Database 9i.

and reset the parallel setting on small database objects to noparallel (database objects with fewer than thousands of records and/or few database blocks in size). Expect some operations that access parallel enabled objects which would not execute in parallel on Oracle Database 9i to run in parallel after an upgrade. However. Furthermore. If your system was heavily loaded on Oracle Database 9i with some operations Oracle SQL Parallel Execution Page 45 . Execution plan changes As mentioned before in this paper. Note that the fact of changing to a single cursor model by itself will not have any impact on the operation of your system. For Oracle Database 10g and higher. The most notable changes are: parallel_max_servers The default value in Oracle Database 9i was 10. Carefully review the parallel settings at the table level. if any. if you automate the comparison of execution plans between the old database release and the new database release.If on Oracle Database 9i you used object level settings to enable SQL parallel execution If you set the parallel properties at the table or index level in order to enable parallel execution. only relates to more parallel capabilities in Oracle database 10g and beyond. an impact. you use pga_aggregate_target or starting with Oracle Database 11g memory_target) the default equates to 10 * cpu_count. due to the change to a single cursor model you will only see multiple parallel servers executing the actual single cursor for the parallel execution plan instead of seeing different SQL statements representing fragments of the parallel plan (a. Operations that complete in a few seconds or less when running in serial benefit little from executing in parallel. which means that SQL parallel execution may end up using a lot more system resources. You should understand where these changes come from and you may have to manually compare the execution plans to ensure they do not change for the worse.k.e. Changes in database defaults Some of the default values for database initialization parameters for SQL parallel execution have changed from Oracle Database 9i.a. so they way of how to monitor and analyze parallel execution will change. slave SQL in version prior to Oracle database 10g). you will only see a single execution plan for a parallel statement in Oracle Database 10g and beyond that us used by all parallel servers. As a result the execution plan is easier to read. then you will see changes. then you will face the highest likelihood to experience changes. Generally 10 * cpu_count equates to a lot more than 10. Rather you want operations that take minutes or even hours to complete in serial to benefit from parallel execution. assuming you use automatic memory management for execution memory (i.

The remedy for this change is to manually set parallel_max_servers to 10 in the database initialization file pfile or spfile. In Oracle Database 10g and beyond parallel_adaptive_multi_user equates to true. potentially starving operations that do require parallel execution in order to complete in a reasonable amount of time. Use Resource Manager Consider the use of Resource Manager beyond Oracle Database 9i to ensure operations get the resources they need when they need them. Resource availability is the most important prerequisite for scalable parallel execution.running in parallel. As a result the database will aggressively reduce the DOP for SQL parallel operations when some other statements already use SQL parallel servers. The Oracle Database provides a powerful SQL parallel execution engine that can run almost any SQL-based operation – DDL. parallel_adaptive_multi_user In Oracle Database 9i parallel_adaptive_multi_user was by default derived from parallel_automatic_tuning and defaulted to false. This paper explained how to enable SQL parallel execution and provided some best practices to ensure its successful use. DML and queries – in the Oracle Database in parallel. you may see the overall system throughput go down when you upgrade to Oracle Database 10g or beyond. CONCLUSION The objective of parallel execution is to reduce the total execution time of an operation by using multiple resources concurrently. If you did not explicitly change parallel_automatic_tuning or parallel_adaptive_multi_user on Oracle Database 9i. If there is a class of user or a type of application that should never execute in parallel. Oracle SQL Parallel Execution Page 46 . That way the application will not unexpectedly consume parallel resources. consider ensuring that this application cannot execute in parallel using a specific consumer group and an appropriate resource plan in Resource Manager. then you should explicitly set parallel_adaptive_multi_user to false when you upgrade to Oracle Database 10g or beyond.

650. JD Edwards. including implied warranties and conditions of merchantability or fitness for a particular purpose. Maria Colgan Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores. Worldwide Inquiries: Phone: +1.506.506. All rights reserved. CA 94065 U. Oracle. and Siebel are registered trademarks of Oracle Corporation and/or its affiliates. for any purpose. nor subject to any other warranties or conditions. without our prior written permission.7000 Fax: +1. This document is provided for information purposes only and the contents hereof are subject to change without Copyright © 2008. Oracle. . PeopleSoft. Other names may be trademarks of their respective owners. whether expressed orally or implied in law.Oracle SQL Parallel Execution June 2008 Author: Mark Van de Wiel.7200 oracle. Hermann Baer Contributing Authors: Thierry Cruanes.650. electronic or mechanical. This document is not warranted to be error-free.A. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means.S.

Sign up to vote on this title
UsefulNot useful