Advanced Queuing Internals

Julian Dyke Independent Consultant

Web Version - November 2008
1

2008 Julian Dyke

juliandyke.com

Agenda

Introduction Single Consumer Queues Multiple Consumer Queues Recipients Subscribers Exception Queues Array Payloads Buffered Messages Spilled Messages Performance

2008 Julian Dyke

juliandyke.com

Introduction Advanced Queuing

Advanced Queuing Introduced in Oracle 8.0 Extended and enhanced in most subsequent versions Supports Oracle Streams in Oracle 9.2 and above Supports buffered messages in Oracle 10.2 and above Allows messages to be enqueued and dequeued from queues that are managed by the database Each queue is associated with a queue table Properties of queue table specify behaviour of associated queues Each queue has a payload which can be: RAW - only messages of type RAW can be enqueued Object type- only messages of the specified type can be enqueued ANYDATA - messages with any object type can be enqueued

2008 Julian Dyke

juliandyke.com

Introduction Advanced Queuing

By default messages are dequeued in the order they are enqueued Default behaviour can be overridden in several ways Messages can be persistent or buffered Persistent messages Stored in queue table Survive an instance restart

Buffered messages Stored in SGA Can be spilled to queue table Lost during instance restart

Messages can be immediate or on-commit Immediate messages are committed immediately when they are enqueued/dequeued On-commit messages are committed with the enqueuing transaction Buffered messages can only be immediate.

2008 Julian Dyke

juliandyke.com

Introduction Advanced Queuing

Queue tables can be created for single or multiple consumers Messages in single consumer queue tables can only be dequeued once Messages in multiple consumer queue tables can be dequeued multiple times by multiple consumers Multiple consumer queue tables can be associated with Multiple recipients Multiple subcribers Subscribers can: Specify rules to control which messages they dequene Specify transformations to be performed against dequeued data

2008 Julian Dyke

juliandyke.com

Advanced Queuing Introduction

Sessions can listen for messages on multiple queues Session is notified when a message arrives on any of the target queues Queue messages can be propagated from one queue to another In the same database In different databases Propagation can be immediate or at specified intervals Transformation converts payload from one object type to another Queue data can be transformed when messages are: Enqueued Propagated Dequeued Messages must be transformed using DBMS_TRANSFORM API

2008 Julian Dyke

juliandyke.com

Introduction Payloads

Queue payloads can be RAW Abstract data types ANYDATA Abstract data types Maximum number of attributes is limited to 900 For example: CREATE TYPE type1 AS OBJECT ( c1 NUMBER, c2 NUMBER, c3 NUMBER ); /

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Queue Tables

Every queue must be associated with a queue table Queue table defines properties of queue Reported in DBA_QUEUE_TABLES Managed using DBMS_AQADM For example: DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','RAW'); DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');

By default single consumer queues will be created Each queue table can contain multiple queues Queues inherit properties of queue table Each queue table block contains blocks for one queue

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Queue Table Columns (RAW Payload)

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','RAW');

Queue table QT1 contains the following columns:

Data Type Col# Intcol# Column Name 16 17 18 19 20 21 16 DEQ_TIME 17 DEQ_UID 18 DEQ_TID 19 RETRY_COUNT Data Type TIMESTAMP(6) VARCHAR2(30) VARCHAR2(30) NUMBER

Col# Intcol# Column Name

1
2 3 4 5 6 7 8 9 10 11

1 Q_NAME
2 MSG_ID 3 CORRID 4 PRIORITY 5 STATE 6 DELAY 7 EXPIRATION

VARCHAR2(30)
RAW(16) VARCHAR2(128) NUMBER NUMBER TIMESTAMP(6) NUMBER

20 EXCEPTION_QSCHEMA VARCHAR2(30) 21 EXCEPTION_QUEUE VARCHAR2(30)

22
23 24 25 26 27 28 29

22 STEP_NO
23 RECIPIENT_KEY 24 DEQUEUE_MSG_ID 25 SENDER_NAME 26 SENDER_ADDRESS 27 SENDER_PROTOCOL 28 USER_DATA 29 USER_PROP

NUMBER
NUMBER RAW(16) VARCHAR2(30) VARCHAR2(1024) NUMBER BLOB SYS.ANYDATA

8 TIME_MANAGER_INFO TIMESTAMP(6) 9 LOCAL_ORDER_NO 10 CHAIN_NO 11 CSCN NUMBER NUMBER NUMBER

12
13 14 15

12 DSCN
13 ENQ_TIME 14 ENQ_UID 15 ENQ_TID

NUMBER
TIMESTAMP(6) VARCHAR2(30) VARCHAR2(30)

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Queue Table Columns (Object Payload)

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');

Queue table QT1 contains the following columns:

Data Type Col# Intcol# Column Name 17 18 19 20 21 22 17 DEQ_UID 18 DEQ_TID 19 RETRY_COUNT Data Type VARCHAR2(30) VARCHAR2(30) NUMBER

Col# Intcol# Column Name

1
2 3 4 5 6 7 8 9 10 11

1 Q_NAME
2 MSG_ID 3 CORRID 4 PRIORITY 5 STATE 6 DELAY 7 EXPIRATION

VARCHAR2(30)
RAW(16) VARCHAR2(128) NUMBER NUMBER TIMESTAMP(6) NUMBER

20 EXCEPTION_QSCHEMA VARCHAR2(30) 21 EXCEPTION_QUEUE 22 STEP_NO VARCHAR2(30) NUMBER

23
24 25 26 27 28 28 28 28 29

23 RECIPIENT_KEY
24 DEQUEUE_MSG_ID 25 SENDER_NAME 26 SENDER_ADDRESS 27 SENDER_PROTOCOL 28 USER_DATA 29 SYS_NC00029$ 30 SYS_NC00030$ 31 SYS_NC00031$ 32 USER_PROP

NUMBER
RAW(16) VARCHAR2(30) VARCHAR2(1024) NUMBER TYPE1 NUMBER NUMBER NUMBER

8 TIME_MANAGER_INFO TIMESTAMP(6) 9 LOCAL_ORDER_NO 10 CHAIN_NO 11 CSCN NUMBER NUMBER NUMBER

12
13 14 15 16

12 DSCN
13 ENQ_TIME 14 ENQ_UID 15 ENQ_TID 16 DEQ_TIME

NUMBER
TIMESTAMP(6) VARCHAR2(30) VARCHAR2(30) TIMESTAMP(6)

10

2008 Julian Dyke

juliandyke.com

SYS.ANYDATA

Single Consumer Queues Database Objects (RAW payload)

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','RAW');

The following objects will be created (object IDs and constraint IDs will vary):
Object ID 70581 70582 70583 70584 70585 Object Name QT1 SYS_LOB0000070581C00028$$ SYS_IL0000070581C00028$$ SYS_LOB0000070581C00029$$ SYS_IL0000070581C00029$$ Object Type TABLE LOB LOB INDEX LOB LOB INDEX

70586
70587 70588 70589 70590 70591 70592

SYS_C009433
AQ$_QT1_T AQ$_QT1_I QT70581_BUFFER AQ$QT1 AQ$_QT1_F AQ$_QT1_E

INDEX
INDEX INDEX VIEW VIEW VIEW QUEUE

LOB columns are used for USER_DATA and USER_PROP columns

11

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Database Objects (Object Payload)

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');

The following objects will be created (object IDs and constraint IDs will vary):
Object ID
70581 70582 70583

Object Name
QT1 SYS_LOB0000070581C00032$$ SYS_IL0000070581C00032$$

Object Type
TABLE LOB LOB INDEX

70584
70585 70586 70587 70589 70590 70591

SYS_C009433
AQ$_QT1_T AQ$_QT1_I QT70581_BUFFER AQ$QT1 AQ$_QT1_F AQ$_QT1_E

INDEX
INDEX INDEX VIEW VIEW VIEW QUEUE

LOB column is used for USER_PROP column

12

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Index Columns

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');

The following indexes will be created by default (constraint IDs will vary):
Index Name
SYS_C009436 AQ$_QT1_I

Column #
1 1 2

Column Name
MSGID Q_NAME STATE

3
4 5 6 AQ$_QT1_T 1

ENQ_TIME
STEP_NO CHAIN_NO LOCAL_ORDER_NO TIME_MANAGER_INFO

13

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Sort Lists

The columns indexed by AQ$_QTI are determined by the SORT_LIST parameter. Possible values are enq_time (default) DBMS_AQADM.CREATE_QUEUE_TABLE priority ('QT1','TYPE1',SORT_LIST=>"priority,enq_time"); priority,enq_time enq_time,priority Must be defined when queue table is created Cannot be subsequently altered
enq_time Q_NAME priority Q_NAME priority,enq_time Q_NAME enq_time,priority Q_NAME

Column # 1

2
3 4 5 6 7 14

STATE
ENQ_TIME STEP_NO CHAIN_NO LOCAL_ORDER_NO

STATE
PRIORITY CHAIN_NO LOCAL_ORDER_NO

STATE
PRIORITY ENQ_TIME STEP_NO CHAIN_NO LOCAL_ORDER_NO

STATE
ENQ_TIME STEP_NO PRIORITY CHAIN_NO LOCAL_ORDER_NO

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Views

Two views are created for each queue table For example for QT1 (object ID =70581):

QT<object_id>_BUFFER e.g. QT70581_BUFFER based on X$BUFFER2

AQ$_<queue_table_name>_F e,g. AQ$_QT1_F based on QT1 and ALL_DEQUEUE_QUEUES

15

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Views

Two views are created for each queue table For example for QT1 (object ID =70581) QT70581_BUFFER AQ$_QT1_F

QT70581_BUFFER is defined as follows:

SELECT "ADDR", "INDX", "INST_ID", "OBJNO", "QUEUE_ID", "MSGID", "CORRID", "SEQUENCE_NUM", "MSG_NUM", "STATE", "PRIORITY", "EXPIRATION", "ENQ_TIME", "ENQ_UID", "ENQ_USER_NAME", "RETRY_COUNT", "SENDER_NAME", "SENDER_ADDRESS", "SENDER_PROTOCOL", "DEQUEUE_MSGID", "SRCSEQUENCE_NUM", "SUBSCRIBER_ID", "EXCEPTIONQ_SCHEMA", "EXCEPTIONQ_NAME" FROM X$BUFFER2 WHERE objno = 70581;

16

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Views

AQ$_QT1_F is defined as follows:

SELECT qt.q_name Q_NAME, qt.rowid ROW_ID, qt.msgid MSGID, qt.corrid CORRID, qt.priority PRIORITY, qt.state STATE, qt.delay DELAY, qt.expiration EXPIRATION, qt.enq_time ENQ_TIME, qt.enq_uid ENQ_UID, qt.enq_tid ENQ_TID, qt.deq_time DEQ_TIME, qt.deq_uid DEQ_UID, qt.deq_tid DEQ_TID, qt.retry_count RETRY_COUNT, qt.exception_qschema EXCEPTION_QSCHEMA, qt.exception_queue EXCEPTION_QUEUE, qt.cscn CSCN, qt.dscn DSCN, qt.chain_no CHAIN_NO, qt.local_order_no LOCAL_ORDER_NO, qt.time_manager_info TIME_MANAGER_INFO, qt.step_no STEP_NO, qt.user_data USER_DATA , qt.sender_name SENDER_NAME, qt.sender_address SENDER_ADDRESS, qt.sender_protocol SENDER_PROTOCOL, qt.dequeue_msgid DEQUEUE_MSGID, 'PERSISTENT' DELIVERY_MODE, 0 SEQUENCE_NUM, 0 MSG_NUM, qo.qid QUEUE_ID, qt.user_prop USER_PROP FROM "QT1" qt, ALL_DEQUEUE_QUEUES qo WHERE qt.q_name = qo.name AND qo.owner = 'US01' WITH READ ONLY;

17

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Queues

Every queue must be associated with a queue table Queue table must exist before queue can be created
DBMS_AQADM.CREATE_QUEUE (queue_name => 'Q1',queue_table => 'QT1');

This statement might create the following object:

Object ID 70793 Object Name Q1 Object Type QUEUE

Every queue has a type which can be: NORMAL (default) EXCEPTION NON PERSISTENT Non persistent queues are deprecated in Oracle 10.2 Use buffered messages instead

18

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Enqueue

The following code enqueues a message of TYPE1 on a single consumer queue:

DECLARE message TYPE1; msgprop dbms_aq.message_properties_t; enqopt dbms_aq.enqueue_options_t; enq_msgid RAW(16); BEGIN message := new TYPE1 (10001,20001,30001); msgprop.expiration :=DBMS_AQ.NEVER dbms_aq.enqueue ( queue_name => 'Q1', enqueue_options => enqopt, message_properties => msgprop, payload => message, msgid => enq_msgid ); END;

19

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Enqueue

The enqueue processes executes the following recursive statement:

insert into "US01"."QT1" (q_name, msgid, corrid, priority, state, delay, expiration, time_manager_info, local_order_no, chain_no, enq_time, step_no, enq_uid, enq_tid, retry_count, exception_qschema, exception_queue, recipient_key, dequeue_msgid, user_data, sender_name, sender_address, sender_protocol, user_prop, cscn, dscn) values (:1, :2, :3, :4, :5, :6, :7, :8, :9, :10, :11, :12, :13, :14, 0, :15,:16, :17, :18, :19, :20, :21, :22, :23, :24, :25)

In Oracle 11.1 this statement uses the LOAD TABLE CONVENTIONAL operation
STAT #3 id=1 cnt=0 pid=0 pos=1 obj=0 op='LOAD TABLE CONVENTIONAL (cr=1 pr=5 pw=5 time=0 us)')

For this statement the following objects are modified

Object ID 70581 70586 70588 Object Name QT1 SYS_C009433 AQ$_QT1_1 Object Type TABLE INDEX INDEX

20

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Dequeue

The following code dequeues a message of TYPE1 from a single consumer queue:
DECLARE message TYPE1; msgprop dbms_aq.message_properties_t; deqopt dbms_aq.dequeue_options_t; deq_msgid RAW(16); BEGIN dbms_aq.dequeue ( queue_name => 'Q1', dequeue_options => deqopt, message_properties => msgprop, payload => message, msgid => deq_msgid ); END;

21

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Dequeue

The dequeue processes executes the following recursive statement:

select /*+ FIRST_ROWS(1) */ tab.rowid, tab.msgid, tab.corrid, tab.priority, tab.delay, tab.expiration, tab.retry_count, tab.exception_qschema, tab.exception_queue, tab.chain_no, tab.local_order_no, tab.enq_time, tab.time_manager_info, tab.state, tab.enq_tid, tab.step_no, tab.sender_name, tab.sender_address, tab.sender_protocol, tab.dequeue_msgid, tab.user_prop, tab.user_data from "US01"."QT1" tab where q_name = :1 and (state = :2 ) order by q_name, state, enq_time, step_no, chain_no, local_order_no for update skip locked

The statement selects all rows in the queue specified by :1 with a state of :2 The FIRST_ROWS(1) hint is used to optimize the plan The statement locks any rows to be deleted This will generate undo/redo The statement uses the FOR UPDATE SKIP LOCKED clause to skip any rows still locked by ongoing transactions

22

2008 Julian Dyke

juliandyke.com

Single Consumer Queues Dequeue

Execution plan for SELECT FOR UPDATE statement is:

STAT #3 id=1 cnt=1 pid=0 pos=1 obj=0 op='FOR UPDATE (cr=7 pr=2 pw=2 time=0 us)' STAT #3 id=2 cnt=1 pid=1 pos=1 obj=0 op='SORT ORDER BY (cr=7 pr=0 pw=0 time=0 us cost=4 size=2759 card=1)' STAT #3 id=3 cnt=1 pid=2 pos=1 obj=70581 op='TABLE ACCESS FULL QT1 (cr=7 pr=0 pw=0 time=0 us cost=3 size=2759 card=1)'

As queue grows, object statistics must be gathered to ensure AQ$_QT1_I index is used prevent full table scans on QT1
Rows identified by SELECT FOR UPDATE are deleted using:
delete /*+ CACHE_CB("QT9") */ from "US01"."QT1" where rowid = :1

Execution plan for DELETE statement is:

STAT #7 id=1 cnt=1 pid=0 pos=1 obj=0 op='DELETE QT1 (cr=1 pr=2 pw=2 time=0 us)' STAT #7 id=2 cnt=1 pid=1 pos=1 obj=70581 op='TABLE ACCESS BY USER ROWID QT1 (cr=1 pr=0)'

23

2008 Julian Dyke

juliandyke.com

Queues Exception Queues

A default exception queue is created for each queue table Exception messages will be moved to default queue unless a userdefined exception queue has been specified when the message is enqueued For example to create a user-defined exception queue
DBMS_AQADM.CREATE_QUEUE (queue_name => 'Q1',queue_table => 'QT1'); DBMS_AQADM.CREATE_QUEUE (queue_name => 'Q1E',queue_table => 'QT1' queue_type => DBMS_AQADM.EXCEPTION_QUEUE);
Object ID 70793 70794 Object Name Q1 Q1E Object Type QUEUE QUEUE Queue Type NORMAL EXCEPTION

To check number of rows in each queue:

SELECT q_name, COUNT(*) FROM qt1 GROUP BY q_name;

24

2008 Julian Dyke

juliandyke.com

Queues Exception Queues

Exceptions will be written to user-defined exception queue if it is specified during enqueue operation
DECLARE l_payload TYPE1; l_msgprop dbms_aq.message_properties_t; l_enqopt dbms_aq.enqueue_options_t; l_enq_msgid RAW(16); BEGIN l_payload := new TYPE1 (10001,20001,30001); l_msgprop.expiration := 60; l_msgprop.exception_queue := 'Q1E';

dbms_aq.enqueue ( queue_name => 'Q1', enqueue_options => l_enqopt, message_properties => l_msgprop, payload => l_payload, msgid => l_enq_msgid ); END;

Message will expire after 60 seconds Expired message will be move to exception queue Q1E by queue monitor

25

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Introduction

There are two ways to use multiple consumer queues Multiple Recipients Multiple Subscribers The same queue definitions are used for both examples:
BEGIN dbms_aqadm.create_queue_table ('QT3','TYPE1',multiple_consumers=>TRUE) dbms_aqadm.create_queue ('Q3','QT3'); dbms_aqadm.start_queue ('Q3'); END; /

26

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Database Objects

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1',MULTIPLE_CONSUMERS=>TRUE);

The following objects will be created (object IDs and constraint IDs will vary):
Object Name QT1 SYS_LOB0000070581C00032$$ SYS_IL0000070581C00032$$ SYS_C009457 AQ$_QT1_S Object Type TABLE LOB LOB INDEX INDEX TABLE Object ID 70767 70768 70769 70770 70771 Object Name AQ$_QT1_H SYS_IOT_TOP_70767 AQ$_QT1_G SYS_IOT_OVER_70769 SYS_IOT_TOP_70769 Object Type TABLE INDEX TABLE TABLE INDEX

Object ID 70756 70757 70758 70759 70760

70761
70762 70763 70764 70765 70766

SYS_C009460
AQ$_QT1_N AQ$QT1_S AQ$_QT1_V AQ$_QT1_T SYS_IOT_TOP_70765

INDEX
SEQUENCE VIEW EVAL CTXT TABLE INDEX

70772
70773 70774 70775 70776 70777

AQ$_QT1_I
SYS_IOT_TOP_70772 QT70756_BUFFER AQ$QT1 AQ$_QT1_F AQ$_QT1_E

TABLE
INDEX VIEW VIEW VIEW QUEUE

27

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Tables

AQ$_<queue_table_name>_T e.g AQ$_QT3_T IOT used queue monitor to manage timed operations Single consumer queues use TIME_MANAGER_INFO column only AQ$_<queue_table_name>_I IOT that maintains state for dequeue operations One row per message per recipient/subscriber AQ$_<queue_table_name>_S Heap table containing information about subscribers AQ$_<queue_table_name>_H IOT used to store dequeue history One row per message per recipient/subscriber AQ$_<queue_table_name>_G IOT correlating messages to subscriber signatures

28

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Tables

AQ$_<queue_table_name>_T IOT used queue monitor to manage timed operations e.g. AQ$_QT3_T
Column Name NEXT_DATE Data Type TIMESTAMP

TXN_ID
MSGID ACTION

VARCHAR2(30)
RAW(16) NUMBER

First 3 columns form primary key Values for the ACTION column include: 0 - delay 1 - expiration 2 - delay Single consumer queues use TIME_MANAGER_INFO column only

29

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Tables

AQ$_<queue_table_name>_I IOT that maintains state for dequeue operations

Column Name Data Type

SUBSCRIBER
NAME QUEUE# MSG_ENQ_TIME MSG_STEP_NO MSG_CHAIN_NO MSG_LOCAL_ORDER_NO

NUMBER
VARCHAR2(30) NUMBER TIMESTAMP NUMBER NUMBER NUMBER

MSG_ID
HINT SPARE

RAW(16)
ROWID RAW(16)

First eight columns form primary key HINT and SPARE columns are stored in IOT overflow segment

30

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Tables

AQ$_<queue_table_name>_S Heap table containing information about subscribers

Column Name SUBSCRIBER_ID QUEUE_NAME NAME ADDRESS PROTOCOL SUBSCRIBER_TYPE RULE_NAME TRANS_NAME RULESET_NAME NEGATIVE_RULESET_NAME CREATION_TIME MODIFICATION_TIME DELETION_TIME SCN_AT_REMOVE Data Type NUMBER VARCHAR2(30) VARCHAR2(30) VARCHAR2(1024) NUMBER NUMBER VARCHAR2(30) VARCHAR2(65) VARCHAR2(65) VARCHAR2(65) TIMESTAMP(6) TIMESTAMP(6) TIMESTAMP(6) NUMBER

31

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Tables

AQ$_<queue_table_name>_H IOT used to store dequeue history

Column Name MSGID SUBSCRIBER# NAME ADDRESS# DEQUEUE_TIME TRANSACTION_ID DEQUEUE_USER PROPAGATED_MSGID RETRY_COUNT HINT SPARE Data Type RAW(16) NUMBER VARCHAR2(30) NUMBER TIMESTAMP VARCHAR2(30) VARCHAR2(30) RAW(16) NUMBER ROWID RAW(16)

First four columns form primary key No IOT overflow segment

32

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Tables

AQ$_<queue_table_name>_G IOT correlating messages to subscriber signatures

Column Name NAME ADDRESS# SIGN DBS_SIGN Data Type VARCHAR2(30) NUMBER SYS.AQ$_SIG_PROP SYS.AQ$_SIG_PROP

All columns form primary key

33

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Indexes

By default six indexes are created for each queue table. For example:
Index Name SYS_C009457 Index Type NORMAL # Columns 1 Table Name QT3

SYS_C009460
SYS_IOT_TOP_70765 SYS_IOT_TOP_70767 SYS_IOT_TOP_70769 SYS_IOT_TOP_70772

NORMAL
IOT IOT IOT IOT

1
3 4 4 8

AQ$_QT3_S
AQ$_QT3_T AQ$_QT3_H AQ$_QT3_G AQ$_QT3_I

Index columns for NORMAL indexes are:

Index Name SYS_C009457 SYS_C009460 Column # 1 1 Column Name MSGID SUBSCRIBER_ID

Index columns for IOT indexes are shown on previous slides

34

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Views

Three views are created for each queue table For example for QT3 (object ID = 70756) QT70756_BUFFER AQ$_QT3_F AQ$QT3 <queue_object_Id>_BUFFER e.g QT70756_BUFFER Similar for single and multiple consumers AQ$_<queue_table_name>_F e.g AQ$_QT3_F Similar for single and multiple consumers AQ$<queue_table_name> views e.g. AQ$QT3 Based on: Queue table (QT3) History IOT (AQ$_QT3_H) Subscriber table (AQ$_QT3_S)

35

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Views

AQ$<queue_table_name> views (AQ$QT3) are based on: Queue table (QT3) History IOT (AQ$_QT3_H) Subscriber table (AQ$_QT3_S) Abbreviated definition is as follows:
SELECT <column_list> FROM "QT8" qt, "AQ$_QT8_H" h, "AQ$_QT8_S" s WHERE qt.msgid = h.msgid AND ((h.subscriber# != 0 AND h.subscriber# = s.subscriber_id) OR (h.subscriber# = 0 AND h.address# = s.subscriber_id)) AND (qt.state != 7 OR qt.state != 9) WITH READ ONLY;

Best view to understand current state of queue for all subscribers

36

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Views

AQ$QT3 contains the following columns

Data Type VARCHAR2(30) RAW(16) VARCHAR2(128) NUMBER VARCHAR2(16) DATE TIMESTAMP(6) NUMBER DATE TIMESTAMP(6) VARCHAR2(30) VARCHAR2(30) DATE TIMESTAMP(6) VARCHAR2(30) VARCHAR2(30) Column Name RETRY_COUNT EXCEPTION_QUEUE_OWNER EXCEPTION_QUEUE USER_DATA PROPAGATED_MSGID SENDER_NAME SENDER_ADDRESS Data Type NUMBER VARCHAR2(30) VARCHAR2(30) TYPE1 RAW(16) VARCHAR2(30) VARCHAR2(1024)

Column Name QUEUE MSG_ID CORR_ID MSG_PRIORITY MSG_STATE DELAY DELAY_TIMESTAMP EXPIRATION ENQ_TIME ENQ_TIMESTAMP ENQ_USER_ID ENQ_TXN_ID DEQ_TIME DEQ_TIMESTAMP DEQ_USER_ID DEQ_TXN_ID 37

SENDER_PROTOCOL
ORIGINAL_MSGID ORIGINAL_QUEUE_NAME ORIGINAL_QUEUE_OWNER EXPIRATION_REASON CONSUMER_NAME ADDRESS PROTOCOL

NUMBER
RAW(16) VARCHAR2(30) VARCHAR2(30) VARCHAR2(31) VARCHAR2(30) VARCHAR2(1024) NUMBER

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Recipients

The following code enqueues a message for three named recipients

DECLARE l_payload type1; l_msgprop dbms_aq.message_properties_t; l_enqopt dbms_aq.enqueue_options_t; l_enq_msgid RAW(16); l_recipient_list dbms_aq.aq$_recipient_list_t; BEGIN l_recipient_list(1) := sys.aq$_agent ('CONSUMER1',NULL,NULL); l_recipient_list(2) := sys.aq$_agent ('CONSUMER2',NULL,NULL); l_recipient_list(3) := sys.aq$_agent ('CONSUMER3',NULL,NULL); l_msgprop.recipient_list := l_recipient_list; l_msgprop.expiration := DBMS_AQ.NEVER; l_payload := new TYPE1 (10001,20001,30001); dbms_aq.enqueue ( queue_name => 'Q3', enqueue_options => l_enqopt, message_properties => l_msgprop, payload => l_payload, msgid => l_enq_msgid ); END;

38

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Recipients

A recipient list is constructed using AQ$_AGENT objects In the example all recipients are in the local database The enqueue operation performs the following actions: Inserts one row in the queue table (QT3) Inserts three rows in the queue status table (AQ$_QT3_I) Inserts three rows in the queue history table (AQ$_QT3_H)

39

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Recipients

The following code dequeues a message for one of the named recipients
DECLARE l_payload TYPE1; l_msgprop dbms_aq.message_properties_t; l_deqopt dbms_aq.dequeue_options_t; l_deq_msgid RAW(16); BEGIN l_deqopt.consumer_name := 'CONSUMER2'; dbms_aq.dequeue ( queue_name => 'Q3', dequeue_options => l_deqopt, message_properties => l_msgprop, payload => l_payload, msgid => l_deq_msgid ); END;

Notes A consumer name MUST be specified The message must have been enqueued specifically for that consumer

40

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Recipients

The dequeue operation performs the following actions Deletes one row from the queue status IOT (AQ$_QT3_I) Updates the following columns in one row of the queue history IOT (AQ$_QT3_H) DEQUEUE_TIME TRANSACTION_ID DEQUEUE_USER Inserts one row into the queue timer table (AQ$_QT3_T) The queue monitor (QMNC) process asynchronously checks the timer table (AQ$_QT3_T) for actions If any actions are found these are sent to the queue monitor slaves (Q001, Q002 etc) When last recipient has dequeued message, queue monitor slaves perform the following actions Delete all rows for message in queue history table (AQ$_QT3_H) Delete row in queue table (QT3) for message

41

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Subscribers

Subscribers must exist for the queue before messages can be enqueued The following code creates two subscribers for queue Q3
DECLARE l_subscriber sys.aq$_agent; BEGIN l_subscriber := sys.aq$_agent ('SUBSCRIBER1',NULL,NULL); DBMS_AQADM.ADD_SUBSCRIBER ( queue_name => 'Q3', subscriber => l_subscriber ); l_subscriber := sys.aq$_agent ('SUBSCRIBER2',NULL,NULL); DBMS_AQADM.ADD_SUBSCRIBER ( queue_name => 'Q3', subscriber => l_subscriber ); END;

Creating a subscriber inserts one row in the AQ$_QT3_S table

42

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Subscribers

The following code enqueues ten messages on Q3

DECLARE l_payload TYPE1; l_msgprop dbms_aq.message_properties_t; l_enqopt dbms_aq.enqueue_options_t; l_enq_msgid RAW(16); BEGIN FOR f IN 1..10 LOOP l_payload := new TYPE1 (10000 + f,20000 + f,30000 + f);
l_msgprop.expiration := DBMS_AQ.NEVER; dbms_aq.enqueue ( queue_name => 'Q3', enqueue_options => l_enqopt, message_properties => l_msgprop, payload => l_payload, msgid => l_enq_msgid ); END LOOP; END;

43

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Subscribers

The following code dequeues a message from Q3 for SUBSCRIBER1

SET SERVEROUTPUT ON
DECLARE l_payload TYPE1; l_msgprop dbms_aq.message_properties_t; l_deqopt dbms_aq.dequeue_options_t; l_deq_msgid RAW(16); BEGIN l_deqopt.consumer_name := 'SUBSCRIBER1'; dbms_aq.dequeue ( queue_name => 'Q3', dequeue_options => l_deqopt, message_properties => l_msgprop, payload => l_payload, msgid => l_deq_msgid ); DBMS_OUTPUT.PUT_LINE ('C1 = '||TO_CHAR (l_payload.c1)); DBMS_OUTPUT.PUT_LINE ('C2 = '||TO_CHAR (l_payload.c2)); DBMS_OUTPUT.PUT_LINE ('C3 = '||TO_CHAR (l_payload.c3)); END;

44

2008 Julian Dyke

juliandyke.com

Multiple Consumer Queues Subscribers

Subscribers can subsequently be added and deleted dynamically

DECLARE l_subscriber sys.aq$_agent; BEGIN l_subscriber := sys.aq$_agent ('SUBSCRIBER3',NULL,NULL);
DBMS_AQADM.ADD_SUBSCRIBER ( queue_name => 'Q3', subscriber => l_subscriber ); DBMS_AQADM.REMOVE_SUBSCRIBER ( queue_name => 'Q3', subscriber => l_subscriber ); END;

New subscribers will only be allowed to dequeue messages that have been enqueued after the subscriber was added

45

2008 Julian Dyke

juliandyke.com

Array Payloads Introduction

Payload of a queue can optionally be a VARRAY of object types For example:

CREATE OR REPLACE TYPE type2 AS VARRAY (10) OF type1; / CREATE OR REPLACE TYPE type3 AS OBJECT (c1 type2); /

Queue table can be created with a TYPE3 payload It is not possible to create a queue table with a TYPE2 payload For example:
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT3','TYPE3');
DBMS_AQADM.CREATE_QUEUE ('Q3','QT3'); DBMS_AQADM.START_QUEUE ('Q3');

46

2008 Julian Dyke

juliandyke.com

Array Payloads Enqueue

DECLARE l_payload TYPE3; msgprop dbms_aq.message_properties_t; enqopt dbms_aq.enqueue_options_t; enq_msgid RAW(16); BEGIN l_payload := new TYPE3 (TYPE2 ( TYPE1 (10001,20001,30001), TYPE1 (10002,20002,30002), TYPE1 (10003,20003,30003), TYPE1 (10004,20004,30004) )); msgprop.expiration := DBMS_AQ.NEVER; dbms_aq.enqueue ( queue_name => 'Q4', enqueue_options => enqopt, message_properties => msgprop, payload => l_payload, msgid => enq_msgid ); END;

47

2008 Julian Dyke

juliandyke.com

Array Payloads Dequeue

SET SERVEROUTPUT ON DECLARE l_payload TYPE3; msgprop dbms_aq.message_properties_t; deqopt dbms_aq.dequeue_options_t; deq_msgid RAW(16); BEGIN dbms_aq.dequeue ( queue_name => 'Q4', dequeue_options => deqopt, message_properties => msgprop, payload => l_payload, msgid => deq_msgid );

FOR i IN 1..message.c1.COUNT LOOP DBMS_OUTPUT.PUT ('C1 = '||TO_CHAR (l_payload.c1(i).c1)||' '); DBMS_OUTPUT.PUT ('C2 = '||TO_CHAR (l_payload.c1(i).c2)||' '); DBMS_OUTPUT.PUT ('C3 = '||TO_CHAR (l_payload.c1(i).c3)); DBMS_OUTPUT.NEW_LINE (); END LOOP; END;
48

2008 Julian Dyke

juliandyke.com

Buffered Messages Introduction

In Oracle 10.2 and above messages can be buffered in the SGA Messages will not be written to database immediately Messages are spillled to database if: Number of messages exceeds threshold value Messages not dequeued within 10 minutes Buffered messages Are much faster than persistent queues Do not guarantee reliability Cannot form part of a transaction Do not support (Oracle 11.1) Message retention / delay Transaction grouping Array enqueue / dequeue Message export / import

49

2008 Julian Dyke

juliandyke.com

Buffered Messages Introduction

Buffering is specified at message level Queues can contain both persistent and buffered messages Payload can be ADT, XML, ANYDATA or RAW Support for LOB payloads is restricted The following definitions are used with the examples in this section
BEGIN dbms_aqadm.create_queue_table ('QT1','TYPE1') dbms_aqadm.create_queue ('Q1','QT1'); dbms_aqadm.start_queue ('Q1'); END;

Note that all queue tables support buffered messages No additional attributes are specified for the queue or the queue table

50

2008 Julian Dyke

juliandyke.com

Buffered Messages Enqueue

The following code enqueues a buffered message

DECLARE l_payload TYPE1; l_msgprop dbms_aq.message_properties_t; l_enqopt dbms_aq.enqueue_options_t; l_enq_msgid RAW(16);
BEGIN l_payload := new TYPE1 (10001,20001,30001); l_msgprop.expiration := DBMS_AQ.NEVER; l_enqopt.visibility := DBMS_AQ.IMMEDIATE; l_enqopt.delivery_mode := DBMS_AQ.BUFFERED; dbms_aq.enqueue ( queue_name => 'Q1', enqueue_options => l_enqopt, message_properties => l_msgprop, payload => l_payload, msgid => l_enq_msgid ); END;

51

2008 Julian Dyke

juliandyke.com

Buffered Messages Dequeue

The following code dequeues a buffered message:

SET SERVEROUTPUT ON DECLARE l_payload TYPE1; l_msgprop dbms_aq.message_properties_t; l_deqopt dbms_aq.dequeue_options_t; l_deq_msgid RAW(16); BEGIN l_msgprop.expiration := DBMS_AQ.NEVER; l_deqopt.visibility := DBMS_AQ.IMMEDIATE; l_deqopt.delivery_mode := DBMS_AQ.BUFFERED; dbms_aq.dequeue ( queue_name => 'Q1', dequeue_options => l_deqopt, message_properties => l_msgprop, payload => l_payload, msgid => l_deq_msgid ); DBMS_OUTPUT.PUT_LINE ('C1 = '||TO_CHAR (l_payload.c1)); DBMS_OUTPUT.PUT_LINE ('C2 = '||TO_CHAR (l_payload.c2)); DBMS_OUTPUT.PUT_LINE ('C3 = '||TO_CHAR (l_payload.c3)); END;

52

2008 Julian Dyke

juliandyke.com

Buffered Messages Memory Usage

Memory is allocated from the Streams Pool The following table shows the amount of streams pool memory required to enqueue 5101 messages with the TYPE1 payload:
Before After 10,324,448 2,448,480 428,512 387,692 224,444 88,060 19,796 16,384 3,936 1,648 recov_kgqmsub kwqbsinfy:cco kwqbsinfy:sta spilled:kwqbl fixed allocation callback kgqmsub deqtree_kgqmctx substree_kgqmctx time manager index msgtree_kgqmctx Before 336 332 208 216 256 144 136 120 120 120 After 504 332 312 288 256 216 192 160 160 160

kodpaih3 kwqbsinfy:mpr image handles kwqbsinfy:bms kggmem_fl_1 kggbt_alloc_block Sender info recov_kgqbtctx kwqbcqini:spilledovermsgs kwqbsinfy:bqg

0 480 84 72 44 2,072 14,140 12,288 2,952 1,236

recov_kggmctx
53

924

1,232

name_kgqmsub

32

48

2008 Julian Dyke

juliandyke.com

Buffered Messages Database Objects

Additional database objects are created the first time a buffered message is enqueued on a queue table This will cause elapsed time of first enqueue operation to be high For example the following objects might be created
Object ID Object Name Object Type

72638 72639 72640 72641

AQ$_QT3_P SYS_LOB0000072638C00032$$ SYS_IL0000072638C00032$$ SYS_C0010003

TABLE LOB LOB INDEX INDEX

The enqueuing session also creates a service for the queue For example SYS$US01.Q3.TEST where US01 is the queue owner Q3 is the queue name TEST is the database name

54

2008 Julian Dyke

juliandyke.com

Buffered Messages Database Objects

AQ$_<table_queue_name>_P contains the following columns

Column Name Q_NAME MSGID CORRID PRIORITY STATE DELAY EXPIRATION TIME_MANAGER_INFO LOCAL_ORDER_NO CHAIN_NO Data Type VARCHAR2(30) RAW(16) VARCHAR2(128) NUMBER VARCHAR2(16) DATE NUMBER TIMESTAMP(6) NUMBER NUMBER Column Name DEQ_TIME DEQ_UID DEQ_TID RETRY_COUNT EXCEPTION_QSCHEMA EXCEPTION_QUEUE Data Type DATE VARCHAR2(30) VARCHAR2(30) NUMBER VARCHAR2(30) VARCHAR2(30)

STEP_NO
RECIPIENT_KEY DEQUEUE_MSGID SENDER_NAME SENDER_ADDRESS SENDER_PROTOCOL USER_DATA USER_PROP

NUMBER
NUMBER RAW(16) VARCHAR2(30) VARCHAR2(1024) NUMBER TYPE1 SYS.ANYDATA

CSCN
DSCN ENQ_TIME ENQ_UID ENQ_TID 55

NUMBER
NUMBER DATE VARCHAR2(30) VARCHAR2(30)

2008 Julian Dyke

juliandyke.com

Buffered Messages Database Objects

The AQ$_<queue_table_name>_P table has one primary key index on Q_NAME MSGID Two view definitions are also updated when the first buffered message is enqueued: AQ$<queue_table_name> e.g. AQ$QT3 reports all messages in persistent and buffered queues AQ$_<queue_table_name>_F e.g. AQ$_QT3_F reports all messages that have not yet been dequeued in both persistent and buffered queues

56

2008 Julian Dyke

juliandyke.com

Buffered Messages Database Objects

The queue monitor slaves write spilled messages to AQ$_<queue_table_name>_P Rows are inserted individually; no array operation is used For example
INSERT INTO "us01"."aq$_qt3_p" ( q_name, msgid, corrid, priority,state, delay, expiration, time_manager_info, local_order_no, chain_no, enq_time, step_no, enq_uid, enq_tid, retry_count, exception_qschema, exception_queue, recipient_key, dequeue_msgid, user_data, sender_name, sender_address, sender_protocol, dscn, cscn ) VALUES (:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,:14,0,:15,:16,:17,:18,:19,:20,:21,:22,:23,:24)

Messages are asynchronously deleted from AQ$_<queue_table_name>_P by queue monitor slaves Messages are deleted using an array size of 32 For example
DELETE FROM us01.aq$_qt24_p WHERE q_name = :1 AND msgid = :2

57

2008 Julian Dyke

juliandyke.com

Buffered Messages Database Objects

AQ$_<table_queue_name>_D contains the following columns

Column Name OID MSGNUM MSGID SUB SEQNUM RSUBS Data Type NUMBER NUMBER RAW(16) NUMBER NUMBER SYS.AQ$_RECIPIENTS

The RSUBS column is stored as a LOB

58

2008 Julian Dyke

juliandyke.com

Buffered Messages Spillage

If flow control is enabled then number of buffered messages that can be enqueued on any queue is limited Subsequent attempts to enqueue messages will be rejected

Set _BUFQ_STOP_FLOW_CONTROL parameter to TRUE to disable flow control completely Limited to 5000 buffered messages 15000 captured messages Can be overridden in 10.2.0.3 by applying Patch 5093060 and setting Event 10867 for buffered messages (level is # messages) Event 10868 for captured messages (level is # messages) Can be fixed in 10.2.0.4 onwards by setting: _BUFFERED_PUBLISHER_FLOW_CONTROL_THRESHOLD _CAPTURED_PUBLISHER_FLOW_CONTROL_THRESHOLD

59

2008 Julian Dyke

juliandyke.com

Buffered Messages Database Objects

For a multiple consumer queue the following objects will be created when the first buffered message is enqueued:
Object ID Object Name Object Type

72638

AQ$_QT3_P

TABLE

72639
72640 72641 72642 72643 72644 72645 72646

SYS_LOB0000072638C00032$$
SYS_IL0000072638C00032$$ SYS_C0010003 AQ$_QT3_D SYS_IOT_OVER_72642 SYS_LOB0000072642C00006$$ SYS_IL0000072642C00006$$ SYS_IOT_TOP_72642

LOB
LOB INDEX INDEX TABLE TABLE LOB INDEX INDEX

60

2008 Julian Dyke

juliandyke.com

Performance Elapsed Times

Enqueue
No Commit Single Consumer PERSISTENT ON COMMIT Single Consumer PERSISTENT IMMEDIATE Single Consumer BUFFERED IMMEDIATE Single Consumer PERSISTENT ON COMMIT VARRAY(10) OF TYPE1 Multi Consumer PERSISTENT ON COMMIT 2 recipients Multi Consumer PERSISTENT ON COMMIT 2 subscribers 4.77 Commit 10.99

Dequeue
No Commit 5.75 Commit 9.62

10.80

11.41

8.77

9.78

2.32

2.60

1.53

2.13

0.66

1.38

1.00

1.35

6.40

14.45

6.36

11.20

6.02

14.59

6.54

11.40

61

10000 TYPE1 messages enqueued then 10000 messages dequeued. Average of 5 runs. Oracle 10.2 on RHEL4.5 x86

2008 Julian Dyke

juliandyke.com

Performance Redo Generation

Enqueue
No Commit Single Consumer PERSISTENT ON COMMIT Single Consumer PERSISTENT IMMEDIATE Single Consumer BUFFERED IMMEDIATE Single Consumer PERSISTENT ON COMMIT VARRAY(10) OF TYPE1 Multi Consumer PERSISTENT ON COMMIT 2 recipients Multi Consumer PERSISTENT ON COMMIT 2 subscribers 9223 Commit 15491

Dequeue
No Commit 10806 Commit 15521

15165

15485

14936

14904

1211

1831

1381

1832

16459

23324

8102

12521

15832

23404

7934

12953

10 TYPE1 messages enqueued then 10 messages dequeued. Average of 5 runs. Oracle 10.2 on RHEL4.5 x86

62

2008 Julian Dyke

juliandyke.com

Conclusion

Several single queues may be more efficient than Multiple recipients Multiple subscribers Use ON_COMMIT visibility where possible No transaction overhead for queuing operations Reduces undo / redo generation IMMEDIATE is much more expensive Buffered messages give best performance Provided they do not spill regularly Array payloads are very efficient Message overhead is reduced

63

2008 Julian Dyke

juliandyke.com

Thank you for your interest

info@juliandyke.com
64

2008 Julian Dyke

juliandyke.com