PeopleTools Integration Performance

and Tuning
An Oracle White Paper July 19, 2010

Contains:
IB Architecture
Performance and Tuning
Configuration
Additional Documentation
This material has not been submitted to any formal Oracle test and is published AS IS. It has not been the subject of
rigorous review. Oracle Corporation assumes no responsibility for its accuracy or completeness. The use of this
information or the implementation of any of these techniques is a customer responsibility and depends upon the
customer's ability to evaluate and integrate them into the customers operational environment.

Peopletools Integration Performance and Tuning White Paper 7/19/2010

Table of Contents

TABLE OF CONTENTS ............................................................................................................................................... 2

CHAPTER 1 - INTRODUCTION ................................................................................................................................... 3
Structure of this White Paper

Related Materials

CHAPTER 2 - IB ARCHITECTURE ............................................................................................................................. 4

2.1 Asynchronous Processing

2.2 Synchronous Processing

12

CHAPTER 3 PERFORMANCE AND TUNING ........................................................................................................... 14

3.1 Asynchronous Processing

14

3.2 Synchronous Processing

34

3.3 Tools to determine bottlenecks

40

3.4 Gateway Configuration

41

CHAPTER 4 CONFIGURATION ................................................................................................................................ 43

CHAPTER 4 ADDITIONAL DOCUMENTATION ........................................................................................................ 45
5.1 Customer Bench Marks

45

5.2 PeopleTools Performance Study: Integration Broker

46

APPENDIX A SPECIAL NOTICES .......................................................................................................................... 76

APPENDIX B VALIDATION AND FEEDBACK ....................................................................................................... 76
Customer Validation

77

Field Validation

77

APPENDIX C REVISION HISTORY ........................................................................................................................ 77

Chapter 1 - Introduction
This White Paper is a practical guide for technical users, installers, system administrators, and programmers who
implement, maintain, or develop applications for your PeopleSoft system. In this White Paper we discuss guidelines
for performance and tuning of the Integration Broker architecture. In addition, a performance study is included to
validate these recommendations.

STRUCTURE OF THIS WHITE PAPER

This White Paper provides guidance for performance and tuning of the Integration Broker framework.
Keep in mind that PeopleSoft updates this document as needed so that it reflects the most current feedback we
receive from the field. Therefore, the structure, headings, content, and length of this document is likely to vary with
each posted version. To see if the document has been updated since you last downloaded it, compare the date of
your version to the date of the version posted on My Oracle Support.

RELATED MATERIALS
This paper is not a general introduction to Integration Broker or Messaging. We assume that our readers shall
consult additional reference material for an in-depth understanding of the subject. To take full advantage of the
information covered in this document, we recommend that you have a basic understanding of Integration Broker
implemented in PeopleTools release 8.48 or greater.
This document is not intended to replace the documentation delivered with the PeopleTools 8.48, 8.49 or 8.50
PeopleBooks. We recommend that before you read this document, you read the Integration Broker related
information in the PeopleTools PeopleBooks to ensure that you have a well-rounded understanding of our
Integration Broker technology. Note: Much of the information in this document will eventually get incorporated into
subsequent versions of PeopleBooks.
Many of the fundamental concepts related to Integration Broker are discussed in the following PeopleSoft
PeopleBooks:
PeopleSoft Integration Broker (PeopleBooks|Enterprise PeopleTools 8.48 PeopleBook:Integration Broker)
Application Messaging (Integration Tools|Application Messaging)
PeopleCode (Development Tools|PeopleCode Reference)
Application Designer (Development Tools|Application Designer)

Chapter 2 - IB Architecture
The IB architecture consists mainly of an asynchronous and a synchronous framework. The Integration Broker
Gateway is a common component that handles both Asynch and Synch requests. This section will examine in detail
the various parts and responsibilities for each type of processing.

2.1 ASYNCHRONOUS PROCESSING

The Asynchronous architecture is based on a publish/subscribe model using the concept of dispatchers and
handlers.
Just as you have other separate server processes on the application server, such as PSAPPSRV, designed to
handle specific types of transaction requests, Integration Broker has six server processes related specifically to
asynchronous processing. These are the servers that are enabled when selecting Pub/Sub Servers from the
PSADMIN menu. Going forward, these servers will be referred to as the pub/sub servers.

PSBRKDSP Message Broker Dispatcher

PSBRKHND Message Broker Handler
PSSUBDSP Subscription Contract Dispatcher
PSSUBHND Subscription Contract Handler
PSPUBDSP Publication Contract Dispatcher
PSPUBHND Publication Contract Handler

Oracle provides a default for each of these server processes. The default server processes have _dflt at the end
of the name, for example PSBRKDSP_dflt. Additional server processes can be created via PSADMIN. These
server processes operate in sets of six and within that set of six there are three sets of two server processes that
work as a unit. When a dedicated server is created for a specific queue set, a dispatcher and handler will
automatically be created.

This graphic shows the flow of a messagein this case as a provider of a Service Operation.

2.1.1 Dispatchers and Handlers

Dispatchers
Dispatchers are Tuxedo based processes that receive Tuxedo asynchronous calls (TPA calls) from other processes
such as PSAPPRV or other handler processes. The dispatchers primary purpose is to determine queue order of all
messages and send (dispatch) them to its associated handler(s). The dispatcher gets initialized with a queue list
determined from PSADMIN configuration files. The contents of this list are the only queues that the dispatcher will
attempt to dispatch. The dispatcher builds queues in memory based on Queue and Sub-Queue names from
database selects it performs on its associated database table, based on the queue received as part of the TPA call.
The dispatcher first synchronizes its in-memory queue with the contents of the associated database table. It then
determines what message(s) can be dispatched based on that queue name. If there are messages to dispatch, it
will create TPA calls for each request. The last call in the request will have an additional flag to indicate that the
handler processing this request needs to send a TPA call back to the dispatcher so it can start a new dispatch cycle
based on that queue name. These TPA requests are added to a Tuxedo queue where handlers assigned to that
queue process the request(s). These queued Tuxedo requests are a key indicator as to how you should attempt to
configure your dispatcher/handlers.
Handlers
Handlers process the Tuxedo request (TPA) from the dispatcher. There can be multiple handlers per domain.
Each handler processes one request at a time. The more handlers you have, the more concurrent processing you
can have however, that means more system resources taken up by their footprint. The actual number of handlers
5

configured needs to be balanced with the amount of CPU and memory consumed by the process. For instance, if a
message is consumed by a handler that is processing a notification event that uses a Component Interface with a
substantial amount of business logic, then more processing power is required to consume that message. That
means you dont want to have a large part of the machines resources tied up just managing a large number of
handlers; there wont be enough resources left to handle the spike from concurrent notifications of this type. In this
case, fewer handlers (3 - 5) are better than more handlers. If the notification process is fairly lightweight, then more
handlers can be used for faster slice of time processing.
To determine the balance of handlers to resources, look in PSADMIN when messages are actually being
processed. If there are messages waiting to be processed and the machine is being under utilized, then configure
more handlers.
The variety of message notifications for a given number of inbound messages can affect messaging performance.
If all of the messages coming in have the same CI or DB tables as the end result of their notification, then
messaging performance will not be as good if the same number of input messages are being consumed by a variety
of CIs or tables on the app side. Row level locking helps, but depending on what is being done in the notification, it
might not be the answer.

Dispatcher/ Handler Types

The Integration Broker engine consists of 6 types of server processes that work in conjunction to process all
asynchronous messages. The specific processing of each will now be examined.
Message Broker
The primary responsibility of the Message Broker is for routing messages to the publication and/or subscription
contract queues.
Dispatcher (PSBRKDSP)
This dispatcher uses the database table PSAPMSGPUBHDR to build its in-memory queue(s). The Tuxedo
requests (TPA) calls can come from a PSAPPSRV process as part of a PeopleCode Publish or a request from its
associated handler. Moreover if the dispatcher is sitting idle for more than the configured scan interval rate (in
PSADMIN), the dispatcher will check to see if any queues have been updated since the last synchronization and
process those messages accordingly.

The process flow for this dispatcher is the following:

1.
2.
3.
4.

Lock out other same type dispatcher(s) for a specific Queue Name via PSPUBLOCK.
Perform in-memory synchronization of the database table PSAPMSGPUBHDR.
Create a dispatch list of all message(s) that can be processed.
Dispatch message(s) via TPA calls and update message status.

Note that if the actual PeopleCode Publish from a PSAPPSRV is not in the same domain as these pub/sub servers,
a Tuxedo request will not get queued for this dispatcher. On-Idle processing is the only way to pick up this
published message. Therefore, if you configure these pub/sub servers as stand-alone (i.e. on a different machine
then where the PIA domains are configured), you should set the scan interval to 1 in order to pick up these requests
as soon as possible.
Handler (PSBRKHND)
The primary purpose of this handler is to determine the appropriate routings based on the Tuxedo request received
and update the appropriate database tables (queues) accordingly. This process also executes the OnRoute
PeopleCode events (OnRouteSend or OnRouteReceive) and Inbound Transformations.
6

The process flow for this handler is the following:

1.
2.
3.
4.
5.

Create routing lists for publication contracts and/ or subscription contracts.

Run OnRoute PeopleCode events.
Run any inbound transformations.
Update publication contract and/or subscription contract queues.
Send Tuxedo request(s) to publication contract and/or subscription contract dispatchers.

Note that the processing time to determine the routings is much faster then actually processing a PeopleCode
event. The number of OnRoute and Transform events typically is very low compared to the overall number of
different messages. If physical resources are a problem, the number of these type of handlers can be possibly
reduced without impacting performance.
Publication Contractors
The primary responsibility of the Publication Contractor is to take the information submitted by the Message Broker
and perform an HTTP post of the message to the Integration Broker Gateway web servlet.
Dispatcher (PSPUBDSP)
This dispatcher uses the database table PSAPMSGPUBCON to build its in-memory queue(s). The Tuxedo
requests (TPA) calls can come from a PSBRKHND process or a request from its associated handler. Moreover if
the dispatcher is sitting idle for more than the configured scan interval rate (in PSADMIN), the dispatcher will check
to see if any queues have been updated since the last synchronization and process those messages accordingly.
The process flow for this dispatcher is the following:
1.
2.
3.
4.

Lock out other same type dispatcher(s) for a specific queue name via PSPUBCLOCK.
Perform in-memory synchronization of the database table PSAPMSGPUBCON.
Create a dispatch list of all message(s) that can be processed.
Dispatch message(s) via TPA calls and update message status.

This dispatcher will also attempt to ping all nodes that have messages in retry status as indicated by the entries in
the PSNODESDOWN table as part of on-idle processing.
Handler (PSPUBHND)
The primary purpose of this handler is to perform the actual HTTP request to the Integration Broker Gateway. This
process also executes the OnSend and OnAckReceive PeopleCode events and Outbound Transformations.
The process flow for this handler is the following:
1.
2.
3.
4.
5.

Run OnSend PeopleCode event.

Run any outbound transformation(s).
Send HTTP request to IB Gateway.
Run OnAckReceive PeopleCode EVENT.
Send Tuxedo Request back to publication contract dispatcher.

Note that the handler will wait for a response from the IB Gateway before it will continue processing. This could be
up to 3 minutes based on HTTP timeout configuration setting.
Subscription Contractors
The primary responsibility of the Subscription Contractor is to process IB Notification events.
Dispatcher (PSSUBDSP)

This dispatcher uses the database table PSAPMSGSUBCON to build its in-memory queue(s). The Tuxedo
requests (TPA) calls can come from a PSBRKHND process or a request from its associated handler. Moreover if
the dispatcher is sitting idle for more than the configured scan interval rate (in PSADMIN), the dispatcher will check
to see if any queues have been updated since the last synchronization and process those messages accordingly.
The process flow for this dispatcher is the following:
1.
2.
3.
4.

Lock out other same type dispatcher(s) for a specific queue name via PSSUBCLOCK.
Perform in-memory synchronization of the database table PSAPMSGSUBCON.
Create a dispatch list of all message(s) that can be processed.
Dispatch message(s) via TPA calls and update message status.

Handler (PSSUBHND)
The primary purpose of this handler is to process the IB Notification PeopleCode event.
The process flow for this handler is the following:
5.
6.

Run OnNotification PeopleCode event.

Send Tuxedo request back to subscription contract dispatcher.

Note that handler processes perform in-memory caching, therefore handlers like PSSUBHND can grow to a very
large size like PSAPPSRV processes.

2.1.2 Message Queue vs. Message Consumption

There is a difference between the messaging system being able to receive a given number of messages per slice of
time and being able to process the message data (i.e. a notification process) per slice of time. For example, if a
non-PeopleSoft system needs to send messages at a rate of x number per minute, the PeopleSoft messaging
system can receive and queue them at this rate. But if the notification process involves a lot of business logic or
actions that take up a lot of server resources, the messages may not be able to be consumed by the application at
the same rate they are queued by the messaging system (given a particular hardware configuration).
An evaluation should be done to determine what the real requirement is for slice of time throughput. Is it the ability
to have the messages queued at x number per minute (with the data from x messages being processed in more
than that minute), or does the message data from x messages absolutely have to be run through the business
logic within that minute slice of time? If the messages just need to be queued (i.e. receive them as fast as other
systems are sending them), then additional configuration options can be explored for optimization. You might even
look at deferring message consumption (running the business logic) until certain times of the day or night when
additional system resources may be available. If the data from x messages absolutely has to be run through the
business logic in that one minute slice of time, then you may have to look at what is happening in the notification
event and optimize that.

2.1.3 Queue Partitioning

One of the biggest problems that affects Integration Broker Asynchronous performance is in the area of partitioning.
Messages (Service Operations) should be designed to take full advantage of as much partitioning as possible. One
area of concern is the Oracle-delivered full sync Service Operations. These particular Service Operations do not
take advantage of partitioning due to the way they currently have to chunk the content data and process the
notifications. Integration Broker has come up with an alternative to this current chunking mechanism by the use of
Message Segments. Message Segments are covered in a later section of this document.

This section will explain how the pub/sub servers use partitioning and the obvious benefits. It will give you more
insight to how these queues should be constructed and when new queues should created.

The dispatchers build in-memory queues based on Queue and Sub-Queue names. The queue name is a queue
(no sub queue name) and all distinct sub queue names are also their own queue. For example, a Message is
partitioned on employee ID in a queue called ABC. Messages are published sequentially within queue ABC and
have the following sub queue names: SUB_QUEUE_A and SUB_QUEUE_B. The following picture shows how
these messages are partitioned. The dispatcher will create requests for the latest timestamp-based message in
each actual queue.

Therefore if a queue is not partitioned, the dispatcher will only be able to process one message per cycle and only if
that message has been completed (DONE or CANCEL message status). This leads to blocked queues with many
messages in NEW status, leading to less then desired throughput of the Integration Broker.
Partitioning Rowset-based and Nonrowset-based Messages
Partitioning of rowset-based and nonrowset-based messages now occurs in the Queue page under the Integration
Setup menu. You can now alias partitioned fields with mix case for both rowset and nonrowsetbased messages.
The Parser used to parse and find elements is case sensitive.
Partitioning occurs in two places in the Integration Brokerat the time of publish and for inbound requests.

At the time of publish for rowset-based messages, the partition happens within the rowsettherefore the partition
data will always be upper case, even if the fields were defined with mixed case aliases. For nonrowset-based
messages, the parser first attempts to parse by uppercase, then by lower case, and finally by alias name if
applicable.
For inbound requests, the partition happens before the data is actually written to the database. If the sub queue
field is empty, the queue assigned to the Service Operation will be used to attempt to partition the data based on
the criteria described above. You should try to avoid this partition point as the data is compressed at this time.
The data would have to be decompressed, parsed, partition logic run then compressed again. Therefore every
attempt to partition the data at the source (time of publish) should be investigated. For 3 rd party systems a query
parameter on the HTTP header can be used to pass in a value for the sub queue field.

Note: For inbound transformations, partitioning will be re-attempted after the transform logic is run, if the sub queue
field is blank.
9

One Message vs. Multiple Messages per Queue

Another design-time consideration when creating queues is whether to have one message per queue or many
messages in one queue. The answer to this question depends on many factors. For example, if there are 20
unrelated messages in a queue , the dispatcher will try to process the 20 messages in the queue assuming these
messages are partitioned. If you create a queue for each message, there would be 20 queues. The dispatcher
would have to traverse all 20 queues to process those 20 messages. This means degraded performance due to
more dispatch cycles and database reads/writes.
Therefore, ideally you should put related messages in one queue and partition accordingly. For high volume
transaction messages, create a queue for each message, as these queues can then be part of a dedicated
messaging server. Dedicated Messaging servers are discussed in a later section.

2.1.4 PSADMIN Parms

There are many parms that are configurable in PSADMIN that have significant impact on performance of
asynchronous-based messaging. This section will discuss each of these parms in detail, describing what it does
and its impact on performance.
Tuxedo Queue Size This is the actual Tuxedo message queue size. For NT its a registry parameter. For Unix
systems its a kernel parameter. This value is used for Tuxedo queue threshold determination by the pub/sub
dispatchers. As Tuxedo requests get queued there is a point where the queue can get full and any additional
requests get disregarded. Prior to 8.43 the number of Tuxedo requests generated by each dispatcher was
unbounded. In other words, if there were 300 requests to create, it would do so disregarding that the Tuxedo queue
was potentially full. With 8.43 the dispatcher reads the number of queued requests and based on this parameter,
determines if any or all requests should be sent out for this cycle. Therefore its important that this value accurately
reflects the actual size. Note that for AIX, the Tuxedo queue size is dynamic, therefore an arbitrary value should be
used otherwise the benefits of throttling are not realized. This value should be 1 or 2 Meg instead of using the
default of 65535.
Dynamic Recycle of IB Handlers This parameter is an alternative to Recycle Count parameter and allows you to
automatically recycle a handler based on Memory Growth. The default is 20 for Percentage of Memory Growth,
which means that after the process as built up its memory cache a 20 percent increase would result in the process
automatically restarting. The Memory Growth Interval parm determines how many requests to process between
checks, the default is 100.
Dispatcher Parms
Recycle Count The dispatcher will automatically recycle itself when this value is reached. By default the recycle
count is set to 0. This count is based on the number of Tuxedo requests. In general you should not have to cycle
the dispatcher. However if a recycle count is used, it will affect performance since initialization will be performed
which includes rebuilding all the in-memory queues for each active queue assigned to that dispatcher.
Dispatch List Multiplier This is a parm used to throttle the number of requests sent to its associated handlers.
The actual list is the number of associated handlers multiplied by this value. The current default is set to 10. This
value was obtained by many performance tests. This value should not have to change as it scales very well.

Scan Interval This is the interval that the dispatcher will run its on-idle processing. The current value is set to 15,
which is fine if Pub/Sub is running in the same domain as the PIA appservers. However, if the Pub/Sub servers are
stand-alone, this value should be set to 1, as this is the only mechanism to initially poll the database queue for work.
Dispatcher Queue Max Queue Size This value is the maximum number of items (messages) per queue that the
dispatcher will keep in memory. The current default is set to 1000, which was obtained by many performance tests.
This value should not have to change as it scales very well.

10

Memory Queue Refresh Rate This is the number of actual dispatches that the dispatcher will automatically
rebuild its in-memory queue for a particular queue. The queues should not get corrupted, however the current
default value of 1000 is set at such a high level that it does not impact performance and is recommended based on
performance tests. This value should not have to change as it scales very well.
Restart Period This is the time that the dispatcher will attempt to re-dispatch messages that are still in the START
status. This can potentially have a big impact on overall performance of the messaging system. When the
dispatcher dispatches a request, it sets the status of the message to START. The Tuxedo request is queued and
the next available handler will attempt to process this request and set the status to WORK. However, when the
message system is under configured (i.e. not enough handlers to process all the requests) the request stays
queued. The dispatcher will again send the request after the restart period has elapsed. This potentially leads to a
lot of redundant requests that the available handlers have to cycle through. This leads to the Tuxedo queue
overflowing and potentially losing requests, which would have to be picked up when the restart period is reached.
However, you do not want to set this value too high, as messages would not be restarted in case of a handler crash.
A good guideline is to use the number of incoming requests per second divided by the number of associated
handlers, multiplied by the average processing time per second.
((Incoming requests per second)/ (# of associated handlers)) * (average processing time per request)
PSPUBDSP only
Ping Rate This parameteralong with the scan intervaldetermine how often a node that is in the
PSNODESDOWN table should be pinged to see if it is in a valid state to send a request to. Part of the on-idle
processing performs these ping requests. When there are a lot of nodes that are down due to improper
configuration of routings on Service Operations, many CPU cycles are spent performing these pings. This value
allows for a longer time between subsequent pings. The algorithm used to determine the interval is:
Attempts * Ping Rate * Scan Interval)

Maximum Ping Interval This is the maximum time between subsequent pings for a node that is in the
PSNODESDOWN table. This value is in hours.
Handler Parms:
Min Instances and Max Instances These values should always be the same. If the Min Instance is not the same
as the Max Instance then under load the system has to wait for the process (Handler) to boot up and initialize before
it can be used. It is better to boot all at one time, avoiding allocations during max load.
Recycle Count Since the handlers run IB Events that contain PeopleCode, there is a good chance of memory
leaks. Therefore you should have a high recycle count and monitor it to determine if it grows to an undesirable size.
The recycle count by default is 20000; however depending on the PeopleCode being run, this value could be set
higher or lower. The problem with a recycle count on a handler is that with proper load balancing from Tuxedo, all
associated handlers recycle at approximately the same time. If this becomes a problem, set the recycle count to 0
and create an automated script that will stagger the recycling of the handler(s) by using Tuxedo command line
arguments to specify the specific handler.
Max Retries - This value represent the number of times the handler will attempt to process a message before the
status is set to TIMEOUT. Therefore, if the PeopleCode being run causes a crash of the process, it will attempt to
process it again. This value should be set to a low value5 or lowerto limit the amount of handler crashes for
one specific bad message.
PSPUBHND only
Thread Pool Size This represents the number of concurrent threads the handler can spawn for HTTP requests.
Note that with performance benchmark testing in this area, with a tread count set to 5, one handler was able to
replace 3 non-threaded handlers. This is something to consider if memory resources are scarce.

11

2.2 SYNCHRONOUS PROCESSING

All Integration Broker Synchronous processing occurs in the PSAPPSRV process. By default this process is single
threaded; therefore only one request can be made at a time. The PSAPPSRV process must wait for a response
from the Integration gateway before continuing processing.

For the most part, the work done by the Integration Gateway corresponds to the work done by the PIA web server,
and the rest of the SyncRequest processing takes place on the same app servers used for regular PIA requests.
PIA requests result in serialized HTML; Sync Requests result in serialized XML.
In general, the overhead of a SyncRequest should be no higher than a PIA request, as long as transforms arent
used. In some cases, it may even be lighter, since the Integration Broker doesnt have to deal with state
management. However, it does have to authenticate each message (prior to 8.48).
Prior to 8.48 the authentication happened as a single jolt request to get user context (GetCertificate service),
followed the by the actual service request (Integration Srv). This would happen for every request from the gateway
to the application server. The overhead to make the GetCertificate call was 300 msec. In 8.48, user context was
cached, thereby eliminating the GetCertificate call. User authentication would occur on the application server using
managed objects.
With tools 8.46, multi-threading was introduce to reduce the overhead of multiple sync request calls in one
transaction. This topic is discussed further in section 3.2.

2.2.1 PSADMIN Parms

These are the configurable PSADMIN parms for synchronous processing that can have an effect on performance,
found in the Integration Broker section.
Min Message Size For Compression Message content data size that will cause the Integration Broker to
compress the data prior to posting to the Integration Gateway. Information on the benefits of compression is
discussed in section 3.2.3.

12

Thread Pool Size This represents the number of concurrent threads one PSAPPSRV process can spawn for
HTTP requests. The default is set to 5 based on performance benchmarks. Performance benchmark results can
be found in the section 5. Additional Documents.

13

Chapter 3 Performance and Tuning

3.1 ASYNCHRONOUS PROCESSING
This section will discuss in detail several ways to increase asynchronous performance using the IB Framework.

3.1.1 Dedicated Message Servers

It is best to run pub/sub (asynchronous) messaging activity on a dedicated server(s) when processing a high
message volume. These domains should have application server processes (PSAPPSRV) configured in addition to
pub/ sub servers. The advantage of having the application server processes on the pub/sub machine is that if the
server only has pub and sub services defined, it needs to poll the database queue to determine if there is work to be
done. With the (additional) application servers configured on the same machine as the pub and sub services, you
get event-driven notification of work to be done without the latency of polling. The gateway should point to this
application server domain for message processing. PIA should not be accessed on the dedicated machine. To
access PIA for configuring messaging, use the PIA install on the other online application server(s).
Dedicated messaging servers are assigned a certain number of queue names that result in its queue list. This list
needs to be unique across messaging servers pointing at the same database. The default messaging server
defined by _dflt at the end of each dispatcher process creates its own queue list based on all queues in the
database less the queues defined for the same type dedicated server process. Therefore, to configure multiple
dedicated servers on different domains (to spread across different machines), ensure that the queue lists are
indeed unique for the different dispatcher types, otherwise there will be database contention and the overall
performance will be significantly impacted.
In addition, add only the server type needed for the queue. This means if you have long running notifications, create
only a subscription contract server. Moreover, remove all queues no longer in use as these will consume additional
resources.
For information on how to configure dedicated servers please reference the Integration Broker PeopleBooks.

3.1.2 Multiple Domains

Pub/Sub Domains for high volume messaging should usually be running on a box other than those used for online
users. This will help prevent messaging traffic from being affected by the number of online users, and vice versa.
Separate pub/sub domains only work for Asynch messaging. Synch messaging runs as a service in the online app
server. When running Pub/Sub on a dedicated box, adjust the Scan Interval for each dispatcher in PSADMIN to 1,
as this will be the fastest initial poll rate from the database queue.
Within a dedicated Pub/Sub machine, you can configure one or more domains. There are few scenarios in which
having multiple domains on one box will result in performance advantages. The reason is that each domain has a
publish dispatcher and a subscription dispatcher. The roll of the subscription dispatcher is to take messages from
the database queue and create Tuxedo calls to get them processed. The dispatcher is where the partitioning logic
is applied. The dispatcher looks at the queue, figures out how many messages can be processed next (based on
partitioning), and then puts them in the Tuxedo in-memory queue to be processed. The dispatcher locks the rows in
the DB message queue until the Tuxedo calls come back with either success or failure. The dispatcher doesnt wait
for the return of the first set of calls before moving on. It will continue to read the DB queue and make the Tuxedo
calls even before the first set of calls have come back. The second dispatcher may not be able to read from the
DB queue since the first dispatcher will have rows locked while processing. The second dispatcher can also add
complications in debugging. Adding another domain takes up systems resources; if you are going to consume
resources, then add handlers rather than a new domain. If you are tying to configure for failover, configure using IB
failover. See Domain failover section below. .
14

In general, you should usually add a new domain when you have another box to add. Then you have more systems
resources that can be used for adding additional handlers. In terms of failover, adding a new box as a domain helps
guard against hardware failure. Other than that, Tuxedo can handle one domain on one box by automatically
restarting failed processes.:
The following is a post in Metalink about performance or messaging problems when running multiple PUBSUB
domains due to inproper configuration.
SOLUTION 201058206 E-IB: Performance or messaging problems when running multiple PUBSUB domains
SPECIFIC TO: Enterprise, PeopleTools, Integration Broker, All Releases
ISSUE:
In an effort to achieve either scalability or redundancy in a Integration Broker infrastructure, customers may be
tempted to simply add additional PUBSUB domains. Setting up the system this way represents an invalid
configuration that can cause severe performance degradation and messaging problems.
It is critical to understand that a message queue should be actively served by only ONE of each type of dispatcher,
i.e. broker, publication, subscription. By design, a given dispatchers populates their in-memory queues with SQL
that assumed that it is the only active dispatcher of that type that is gathering work from a given message channel.
The performance degradation caused by the competing locks when multiple dispatchers are attempting to retrieve
work from a given channel is so significant that the overall PUBSUB throughput typically goes DOWN noticeably.
EXAMPLES
1.) By default, each PUBSUB domain will launch default ("_dflt") dispatcher/handler combinations. If multiple
domains actively run the default dispatchers, it will set up contention as they all go after the same work.
2.) You can generate contention if you use default dispatcher/handlers on one domain and dedicated
dispatcher/handlers on another active domain.
3.) If you use dedicated dispatcher/handlers configured on multiple domains that simultaneously draw work from the
same channel, this too will cause this contention.
DETAILS
a.) In the process of getting work, the broker dispatchers, i.e. PSBRKDSP, will place critical locks on PSPUBLOCK,
PSAPMSGPUBHDR. If multiple broker dispatchers are configured to draw work from the same message channel,
these locks will cause significant performance degradation.
b.) In the process of getting work, the publication dispatchers, i.e. PSPUBDSP, will place critical locks on
PSPUBCLOCK, PSAPMSGPUBCON. If multiple publication dispatchers are configured to draw work from the
same message channel, these locks will cause significant performance degradation.
c.) In the process of getting work, the subscription dispatchers, i.e. PSSUBDSP, will place critical locks on
PSSUBCLOCK, PSAPMSGSUBCON. If multiple subscription dispatchers are configured to draw work from the
same message channel, these locks will cause significant performance degradation.
SOLUTION:
SCALABILITY and REDUNDANCY CONSIDERATIONS
Only one PUBSUB domain should be active for a given database unless you implement PeopleTools delivered
functions for scalability and redundancy.
For redundancy, PeopleTools has provided Integration Broker Failover Groups. For Scalability, PeopleTools has
provided Master/Slave PUBSUB domains and dedicated handlers. Additionally, within a PUBSUB domain you can
increase the number of handler instances. Please consult PeopleBooks for setup and configuration details
15

Master Slave
The Integration Broker will now support the ability to create dedicated Slave Pub/Sub Servers called Static Slaves.
You would create a domain(s) with nothing but dedicated slave pub/sub servers. These servers would register
themselves as slaves along with additional configurable information (number of handlers booted, etc) so that the
appropriate master server can use that information to allocate work (messages to process) to these slave server(s).
With 8.48 PeopleTools, the master dispatcher can actually assign work to more than one slave domain.
Dispatcher in Master Mode
1. When the dispatcher is initialized it will determine the mode to run in by reading the PSADMIN parm Dispatcher
Slave Mode. It will then initialize and determine its queue set and register the active queues associated with this
dispatcher accordingly. (See the Queue Registration section below for further details).
2. When the dispatcher starts processing to determine its dispatch list, it will call the method SlaveEvaluation,
passing in the queue name. This method will determine corresponding active slave dispatchers with a queue set
that includes the passed in queue name. It will return a list of slaves with the following structure {ProcessName,
MachineName, DispatchCount, and SlaveBusy}. It will use these values to determine the actual size list to send
to that slave process.
3. The master dispatcher will have responsibility to restart any messages designated for a slave dispatcher.
Dispatcher in Slave Mode
1. When the dispatcher is initialized it will determine the mode to run in by reading the PSADMIN parm Dispatcher
Slave Mode. It will then initialize and determine its queue set and register the active queue names associated
with this dispatcher accordingly. In this case it will determine that it is in slave mode.
2. On-Idle processing is different when in slave mode. On-Idle processing will only look for messages that are in
STARTED status for that particular processidentified by machine and process name.
3. Synchronization for queues while in slave mode will consist of building in-memory queues for messages that have
the same dispatcher name and status of STARTED or WORKING. There is no need to perform an update of the
queue synchronization in memory based on last update date time check, due to the relatively small size and the
fact that the Master is really the ultimate controller of the message.
4. The slave dispatcher will determine if the number of started messages in the queue exceeds that of the throttle
value (#of handlers * Dispatcher List Multiplier). If the value is exceeded, the slave dispatcher will notify the
Master that it cannot accept any more messages. The master dispatcher will then stop assigning messages to that
slave dispatcher. The slave dispatcher will again notify the Master that it can again receive messages when it is
finally below the threshold point.
5. Cancelled and Resubmit messages are all handled by the master dispatcher. A message that was assigned to a
slave and did not process successfully or had to be restarted will become the responsibility of the master
dispatcher.
PSADMIN Parm
The following parm is under the section Settings for PUB/SUB servers
Dispatcher Slave Mode=0
The default will be 0 indicating that it is not in Slave Mode.
Queue Registration
All dispatchers will now register their active queue set with the database to be used for validation and verification
with respect to failover and for Master/ Slave processing. This data can be viewed from the Service Monitor
Failover page. This is a good place to verify that all queue lists are indeed unique across multiple pub/sub domains.
The following Functionality: Load Balancing, Deffered Processing and Slave Templates are avaliable only in
PeopleTools 8.50 . Access to this functionality is via the Domain Status page.

16

Load Balancing
Selecting the Master/ Slave Load Balance hyper link will bring up a secondary page which will have all domains that
are currently registered for that database. Each domain will have a weighting of 1 by default. The user can modify
this number according to the processing power of the machine that the domain resides on. Therefore the higher
the number the more requests that domain (master or slave) will be allocated.
This is a relative weighting, by that it depends on who the master and actual active slaves used for that dispatch
cycle to determine the number of request to send to each domain. For example for a particular dispatch cycle if the
master is weighted at 2 (with 3 handlers) and the there are two slaves weighted at 3 (with 3 handlers), then the
number of requests to be processed by the master will be 33% less then the max number of requests if not
weighting applied (slaves avg = 3, master 2 overall number used by master .66 of maximum requests). The 2 other
slaves would process max requests based on handlers*multiplier.

Alternatively, if the master is weighted at a 3 and the two slaves are weighted at 2 then the avg of the slaves is 2 as
compared to the master weighting of 3 therefore, the max number of requests will be set to the master first, followed
by 33% less of max to each slave.
Deffered Processing

17

The drop down list box for Master Processing Status displays the following options: Enabled (the default selection),
Deferred All Queues and Deferred Unordered Queues. Status of Enabled is normal processing in that the
Master will process its appropriate share of requests. A status of Deferred All Queues when selected, will cause
the Master to not send any requests to its respective handler(s) as long as there is at least one active slave that
could be used for that dispatch cycle. This frees up processing on the Master domains machine to be allocated to
other processes such as additional PIA processing. A Status of Deferred Unordered Queues will cause the
Master to not send any requests in an unordered queue to its respective handler(s) as long as there is at least one
active slave that could be used for that dispatch cycle.
Slave Templates
The idea here is to be able to dynamically add slave domains without performing any configuration changes in
psadmin. As the need to add more processing power for increased throughput with respect to async processing,
one should be able to just use a standard psadmin template and create in a sense virtual domains as needed to
meet the user desired processing requirements. In order to create Static Slave Domains that use a pre-configured
queue list create these domains by importing the IB Master Configuration file. Therefore, when a domain is created
that uses the PUB/SUB processes with optional Dedicated Servers configured. This configuration file should be
used to build all Slave Domains. When configuring these Slave Domains, select the Import domain configuration
option on the PeopleSoft Application Server Administration page in psadmin.

This will bring up the PeopleSoft Import Application Server Configuration page. Select option 2 Import IB Master
Configuration.

This will give you the same options like the standard import domain configuration. You can import the configuration
from a file or from the master domain directly. When the configuration is imported, all the PUB/SUB processes will
be configured identical to the Master configuration file along with PSWATCHSRV and PSMONITOR servers. Note
that other processes will not be included by default (PSAPPSRV, PSSAMSRV etc.) as this Domain is specifically
used for Integration Broker. With that said one can modify the configuration file to include these additional
processes if warranted. These Domain(s) when booted will appear in the Domain Status Page as a Template
Slave.

18

Selecting the hyper link Slave Template Setup will bring up a secondary page displaying a read only, comma
separated Queue list for each default or dedicated Dispatcher Type (Broker, Publication, and Subscription).

19

20

Inactivating all domains results in the Add/ Remove Queues hyper link becoming active (be sure to select the Force
Reset Button as this will reset any contacts in a Started or Working state). The user can then select the appropriate
Add/Remove Queues hyper link to update the respective scroll of all Queues defined. The user can add any number
of queues to each Dispatcher type, which will be used to configure the Queue list for that type Dispatcher when the
domain is configured as a Slave Template. Selecting the Update button will also update any currently configured
Slave Template domain with that queue set for the appropriate Dispatcher type.

21

Selecting the Slave Template Cleanup hyperlink (enabled when all domains are inactive) will bring a secondary
page which will clean up the appropriate template dispatcher queues. Selecting a dispatcher type will remove the
queue list from the database and will automatically be re-configured with the default queue list if applicable.

Note: Slave Template Domains when configured and booted for the initial time will come up with Domain Status as
Active. This will allow Domains when configured and booted in Slave Template Mode to be used even with failover
enabled.
Multi-Queue (Tools 8.5)
The ability to assign multiple queues (unordered) and auto load balance the Service Operation with respect to the
assigned queues can be a significant performance gain based on configuration. This should be considered when
processing of a single Service Operation requires an extremely high throughput. This feature can be used with
Master/ Slave, Failover, etc. to give additional configuration options to meet your throughput goals.
In order to enable Multi-queue processing, select the Enable Multi-queue checkbox on the Service Configuration
page.

22

A hyper-link called Multi-queue will be added to the Service Operation page when the Enable Multi-queue check box
is selected.

When selected, one can assign any number of unordered queues to the Service Operation. The queue shown in
the Queue Name edit box will be the default queue when Mutli-Queue is disabled on the Service Configuration page
or the Service Operation Save pushbutton is selected. The rotation of queues will insure a proper load balance
between all queues with respect to transactions for this Service Operation. Note that this can also be used for
23

inbound transactions (provider Web Services). The queue is assigned as part of the deserialization of the data
upon acceptance. The same load balancing takes place, therefore if a queue is not passed in as part of the request
then, queue load balancing will automatically take effect.

Domain Failover
Failover is an important fault tolerance function of any mission-critical system that relies on constant accessibility.
You can choose to enable failover capability for the Integration Broker if there are backup domains available.
Domains are assigned to a group. If the primary domain of a group fails to process for any reason, the failover
mechanism will automatically start, and switch to the backup domain of that group. It will then continue to process
messages without any manual intervention. You can set up groups, and prioritize the domains in each group. You
can also dynamically set the threshold for when the failover mechanism should be invoked.

There are two parts to enabling domain failover.

4.
5.

First, on the Domain Status page, failover is enabled/disabled and domain failover group and priorities are set.
Second, the polling mechanism is set up to actually perform the failover processing.

Domain Status Page

The Domain status page provides a Set Up Failover link to transfer to the Failover Configuration page. It also
shows the current failover status (enabled/disabled), failover group and failover priority.

24

Failover Group: A failover group consists of 1 to n domains that are logically grouped together representing an
independent failover. Failover Group(s) start with a default of 1 and must be sequential when adding additional
groups.
Failover Priority: Within a Failover Group, the domains are sequentially ordered from 1 to n number of domains in
the group. When Failover is enabled the domain that has a failover priority equal to 1 will become the active
domain. All other domains in the group will become inactive. When a failover occurs, the next domain in priority
(sequence of 1,2,3) will become the active domain.
Setting up failover into groups allows the primary domain (failover priority = 1) in each group to become an Active
domain. When failover is enabled, all domains in a group that are not the primary domain will become Inactive.
Domains not assigned to a group will also become Inactive.
Failover enabled: This page indicates that Failover is enabled. Note that the option to select a Domain Status has
been removed. You cannot select the check boxes All Domains Active or All Domains Inactive as they will have
no effect on the current status. The Grace Period fields will become disabled. Failover processing does not use
Grace Periods.
If another Domain with Pub/Sub servers is booted up with Failover Enabled, the Domain will come up Inactive and
not have a group or priority. In other words that domain will not be used for failover. The user must update the
failover configuration page (adding a failover group and priority) for the domain to become part of the failover
process.
Failover Configuration Page
The IB Failover page automatically validates domains associated to a particular group. Upon request, this page
also shows the current queue sets associated with each dispatcher within a domain. The domains identified as
Slave domains will be shown on the bottom of the page in the Static Slave Domains grid. These slave domains will
not have a group or priority associated with them since they can be the slave of any domain in any group or priority
that uses any queue name in its queue list.
The Failover logic for Slave Dispatcher(s) will not actually perform a failover of that domain if the Slave Dispatcher
has not updated its own health timestamp. However an email will be constructed and sent out signifying that the
slave domain is not running correctly.
Use this page to set up the parameters for failover. The page has a default of one group (Failover Group = 1).
When failover is enabled, this will become the Active Domain. If you want to create groups of domains because

25

there are domains configured with dedicated pub/sub servers, the groups must be sequential. There are edit
checks to insure that the group sequence is sequential and a failover priority is entered.

IB Failover Time (minutes): This refers to the elapsed time that dispatchers must register themselves before an
actual failover will commence. The dispatchers register themselves after each dispatched cycle. This includes onidle processing. Therefore the scan interval must always be lower then the Failover Time entered. A failover time
should be long enough to insure that the dispatchers can successfully rebuild their in-memory queues as part of a
memory queue refresh.
Email To: The data in the e-mail section is used to send e-mail when an actual failover takes place. In order for an
e-mail to be sent, the SMTP server must be configured via PSADMIN. This sever has no dependency on
Integration Broker. The only required field to send an email is an address. If no address is entered, no e-mail will
be sent.

Enable Failover checkbox: You can change parameters and save updates when failover is NOT enabled. Note
that group and priority validation checks will still occur.
If the Enable Failover checkbox is selected and saved and the current Active domains are not Failover Priority 1, the
current active Domains will be set to Inactive, and the same logic that the Force Reset Button performs today will be
executed. This will insure that all the runtime data is in a proper state for the new Active Domain to process. The
Failover Priority 1 Domain for each group will become the Active Domain.
If failover is enabled and the primary Failover Priority is changed for any group, the same logic that initially enabled
failover will be performed.
Check Group Validity: This hyperlink will take all the domains in a group and perform validation on all queue sets.
It will return an invalid queue set, if there is one. If there is more than one, it will still only bring back the first one.
26

You should make the necessary modifications (remove from group, add a queue name, etc.) and revalidate. If the
group is validated correctly (no miss-matches of queue sets) no queue set will be returned.
View Group Queue Sets: This hyperlink shows all queue lists for all domains in a group. Here you can determine
what the queue lists are for dedicated servers.
Dynamic Slave Option: Another nice feature of using IB failover is creating a dynamic slave domain for each
failover group by selecting the Dynamic Slave checkbox. When this checkbox is selected, the second domain in
each group automatically (dynamically) becomes a slave domain. These dynamic slave domains have the same
functionality as the static slaves. When a failover occurs within the group, the dynamic slave will become a master
again and the next domain in the priority for that group will become a dynamic slave. All of the benefits of
Master/Slave are available by simply selecting a check box.
Priority Reset checkbox (8.50 tools): When enabled will attempt to failover back to the highest priority domain
instead of using the normal round-robin type failover.
Additional Notes:
It is highly recommended that domains assigned to the same group have identical queue sets and that the complete
list of queues assigned to each type of dispatcher (Broker, Subscription, Publication) be the same. This will prevent
the following situation: A queue that has been processing messages in the primary domain becomes inactive due to
a failover. If the new active domain does not have that queue defined, the messages in that queue will not be
processed.
Domains that do not have a group assigned (and by default a priority) will become inactive when failover is enabled.
Typical Questions:
How do I deactivate the failover ability? In other words, even if a failover event was detected, how do I prevent the
system from actually performing a failover?

To disable failover at any time, simply deselect the Enable Failover check box on the Failover Configuration
page.

If a primary domain must be shut down in a group, what are the steps to re-assign a different domain as primary?

If failover is enabled, simply reassign the failover priority in the group and click the Save button. This will
automatically inactivate the domain to be shutdown and reset the run-time data so that the current primary domain
can then start processing.

If Domain B were the next in priority for failover in a group, how would I change the priority to Domain C instead
because Domain B has to be shutdown?

Change the failover priority in the group and click the Save button. As long as the primary failover priority was
not changed, no additional inactive/active processing is necessary. The next in priority will only be used at the
time of actual failover.

Failover Monitor Process

The Tuxedo server called PSMONITOR is used as a polling mechanism for IB Failover. Originally designed to
monitor long running queries, the framework has been extended to also perform Domain Failover processing. The
polling interval is currently not configurable and is set to poll every second. Each Application Server domain will
have a PSMONITOR server that will boot as part of the application server boot-up process.
API & Command-line Specifications for failover
A Component Interface called IBFAILOVER_CI is used to generate a java object adapter template, a Visual Basic
template or a C template. To create the appropriate template, access the CI IBFAILOVER_CI in Application
Designer. Right click on Properties and a menu will be displayed. Select the desired template from the menu. This
will automatically create the file and place it in a TEMP directory called IBFAILOVER_CI. *.
Refer to PeopleBooks describing the Component Interface technology and how to build, set up, generate and
implement APIs in java, C++ and COM environments. Samples are also provided for each API type.

27

3.1.3 Multi-Treading of Publication Contract Handler

The Publication Contract Handler can now accept a message, spawn a thread and return. The handler will continue
to accept messages until he maximum number of configurable spawned threads has been reached (PSADMIN
parm). The Publication Contract Handler will automatically poll to determine completed processing on each thread
and complete post thread processing of the message accordingly. To increase throughput or reduce the footprint of
IB, change the Thread Pool Size parm from the default of 1. Refer to the Performance and Benchmark documents
for specific numbers on performance.

3.1.4 Message Segments

Message segments provide a way to process and send large amounts of data (Gbytes) without impacting
performance due PeopleCode processing, or running out of memory.
Attempting to process large amounts of data via a rowset or SQL insert has shown to either crash the server or take
an extraordinary amount of time. The IB framework itself can handle large amounts of data, as it only passes a
reference to this data in the database. Moreover this data is compressed and is sent that way to the Integration
Gateway. The problem occurs when the data is actually used in an IB PeopleCode event such as OnNotification.
The data has to be decompressed and loaded in to either a rowset or xmldoc. The problem is compounded when
that same data is used in a SQL object to insert this application data into the database.
Another problem is loading a rowset with large amounts of data. You can use message chunking for full sync. The
problem here is that the messages are chunked and published. However they are all in the same queue and not
partitioned due to the header and trailer messages that are associated with the processing. This leads to
processing these messages one at a time in sequence, not taking advantage of parallel processing from the IB
framework.
Message segments allow for a single message to be loaded with all the data segmented by any chunkable size
desired. After one segment is populated based on a configuration parm, or overridden by PeopleCode, that segment
will be serialized to XML and inserted into the IB database queue compressed. The next segment will then be
available for processing. This type of loading continues until the message is completely loaded with all the desired
data. The message can be sent either as one message (Ordered Segments) or multiple messages (Unordered
Segments). The actual data is sent chunked by segments to the gateway and received by the target system.
These segments should be complete stand-alone data structures. Only one segment can be decompressed and
loaded into a rowset at one time. The message object is responsible for this memory management. Please refer to
the Performance and Benchmark documents for specific numbers on performance.

The consumption of a segmented message in PeopleCode is straightforward. One segment at a time is

decompressed and serialized into a rowset. After the data is inserted into the database, the rowset is destroyed
freeing up memory and the next segment is then loaded. This process repeats for the number of segments in the
message.
Refer to IB PeopleBooks for Message PeopleCode APIs with respect to Message Segments.
3rd Party and Message Segments
rd

A 3 Party can send in data chunks of a transaction by including the following Parms as http properties, query
arguments or SOAP Headers: DataChunk, DataChunkCount, and TransactionID. Note that only the HTTP
Listening Connector will look for these parms. No other listening connector type will have this enhancement.
rd

Example: 3 Party to send 3 chunks of data as one transaction.

First request -the following parms should be populated:

28

DataChunk = 1 (indicating the first chunk) , DataChunkCount = 3 (indicating total number of chunks to be sent for
this transaction), TransactionID (N/A for first request). When the request is received by PeopleSoft the data chunk
will be saved in the database as a segment and the status of the transaction will be set to HOLD. The response will
send back the TransactionID which needs to be included as part of all subsequent requests for this transaction.
Second request -the following parms should be populated:
DataChunk = 2 (indicating the second chunk) , DataChunkCount = 3 (indicating total number of chunks to be sent
nd
for this transaction), TransactionID (TransactionID from initial response). When the 2 request is received by
PeopleSoft the data chunk will be saved in the database as a segment and the status of the transaction will remain
HOLD.
Third request -the following parms should be populated:
DataChunk = 3 (indicating the second chunk) , DataChunkCount = 3 (indicating total number of chunks to be sent
rd
for this transaction), TransactionID (TransactionID from initial response). When the 3 request is received by
PeopleSoft the data chunk will be saved in the database as a segment and the status of the transaction will be
changed to a NEW status. The transaction can then be processed like any other transaction at this point. The
PeopleCode to read these datachunks (segments) would be the Message Segment API.
rd

For the case where a segmented message is sent out of PeopleSoft to a 3 party (via the HTTP Connector) these
rd
parms will also be populated accordingly. This will allow the 3 party to potentially handle multiple segments of data
rd
as one transaction. The designated 3 party Target Node must have the segment aware checkbox enabled for
multiple segments to be sent to the gateway. The gateway will then populate the parms accordingly based on the
number of content sections in the request.

3.1.5 Load Balancing

Ping other than On Idle
Ping other than On-Idle allows transactions that failed due to a framework (connection) problem to be retried
periodically in a pub/sub system that has long periods of processing time (i.e. On-Idle is not run very often). The
benefit of this is to unblock a queue and have it able to process in a more load balancing way, then for all the other
queues to finish processing, thereby allowing On-Idle to run and clear the blocked queue.
PSADMIN Parm
The following parm is under the section Settings for PUB/SUB servers
Load Balance Interval=0
This value (in minutes) determines the time interval between dispatcher load balance processing. The publication
contract dispatcher performs an equivalent of an On-Idle ping on all down nodes without the delay.
Note: Only the Default Publication Contract Dispatcher (_dflt) will be used to ping these nodes. When the Interval
is exceeded, the default publication contract dispatcher will perform the basic On-Idle processing of generating node
pings as is used today. If the actual On-Idle processing is run before the load balance interval is exceeded, then the
load balance time will be reset.
Load Balancing of Queues
This functionality also helps to load balance queue processing on an active Pub/ Sub system. For example, if there
is no load balancing and there are 3 queues that have messages to process, only one queue will be processed at a
time. TPA calls will still be generated for that dispatcher to call back in the same queue resulting in that queue
performing most of the processing. This is evident when publishing in batch mode. The initial TPA call to prime the
pub/sub engine comes from On-Idle processing; therefore as long as there are messages in that queue, no other
queue will be processed. With load balancing enabled, the dispatcher will check the load balance interval, and if
exceeded, will perform the equivalent of a dispatch all queues. This means that other queues that will be processed
at least partially for that interval time designated.

29

The parm in PSADMIN is the same as what is used of On-Idle Ping. The concept is the same, however an On-Idle
ping will not happen if the dispatcher is not the default publication contract dispatcher.

3.1.6 DMS Handler

The DMS handler is a type of Notification Handler that can process PeopleSoft rowset-based messages. This type
of handler is ideal for bulk loading large amounts of data into a few database tables, similar to current Full Sync
processing. Invoking PeopleCode that uses a SQL Object to insert/update database tables has large overhead and
is very inefficient for large volumes of data. Instead, use the DMS handler. All DMS Handler processing occurs in
the C++ layer and uses principles based on DMS (Data Mover Script) to insert/update database tables. Currently
only destructive loads can be processed. This means that the record data will first be deleted prior to the bulk insert
of the data. If an error occurs during bulk insert the data cannot be rolled back.
In 8.50, there will be a size limit specified to determine if the data can be rolled back in an event of an error.
Moreover, with 8.50 one can also use the DMS handler for PeopleSoft rowset based Part messages and non-rowset
based messages.
The functionality for a container message will be similar to a rowset based message in that the message structure
will determine DMS processing. For a non-rowset based message, the user will define the structure to use for
DMS.
The type of processing is destructive load meaning that the data is first deleted from the table then bulk inserted.
If there is an error there is no transactional rollback. The proposed functionality will allow for transactional rollback
based on database transaction.
When a Data Mover Handler is created, there will now be a details link available.

Selecting this details link will bring up the DMS Handler Details Page.

All Databases that PeopleSoft applications supports provide transactional rollback capability, however transactional
rollback is dependent on bulk size of the data to be updated. Therefore, when the Rollback Enabled check box is
selected, a list box will appear. This list box will store the roll back size as configured on the database in kilobytes.
By default the value will appear as 32 (kilobytes).
Warning: that if the value entered is greater that what the database is configured for transactional roll back, then
the database could restart thereby losing connection to any process currently attached.
30

If the Rollback Enabled checkbox is selected then a check of the data size will be determined if transactional
rollback can be enabled based on the value entered. If the value is greater then the Rollback Size entered, then
transactional rollback will be disabled for that particular transaction thereby causing a destructive load will take
place.

For a non-rowset based message defined on the Service Operation, the user can now add a data structure defining
the message content with respect to database records. This allows for basically any shaped data to be mapped to
the records and fields of the database.

To add records, select the Add Record to Root hyperlink in order to select the desired record. The functionality to
build out the hierarchy is the same as on the Message builder. Note that unlike in the Message Builder, multiple
records can be at the root level (level 0). Similar to the Message Builder, aliasing of the database record(s) and
fields, and excluded of fields are allowed.

Selecting the OK pushbutton will save the data structure for that Data Mover handler. The user at any time can
modify this data structure. The functionality to add and delete records is exactly like the functionality in the Message
Builder.

31

3.1.7 Throttle Threshold

When the receiving system cannot process incoming requests at the throughput possible of the IB Asynch runtime
system, it becomes necessary to throttle the number of requests. Throttling the number of messages sent to a
receiver has both a benefit for the sender and receiver. For the sender, the IB system will not waste resources
waiting for a reply that in some cases can exceed the http timeout limit. Depending on how the service operations
are partitioned, this can lead to blocked queues due to this timeout. The benefit to the receiver is that the messages
sent can be processed in a timely manner and they are not saturated with incoming requests.
Integration Broker throttle threshold is node-based and thereby configured on the Nodes component in PIA. Enter
an integer in the IB Throttle Threshold field to indicate the maximum number of requests the IB Asynch runtime will
send to that node per dispatcher cycle. Master/Slave processing will also honor this threshold valuetherefore if
multiple Slaves are configured the Master Dispatcher will throttle the number of requests sent to its available Slaves.

3.1.8 Application Engine Handler

A new Service Operation Handler will be available for Notification events in PeopleTools 8.5 called Application
Engine Handler. This handler will schedule an AE program to run within the process scheduler. The process can
be tracked by viewing the Subscription Contract in the Service Operation Monitor. This handler is a good alternative
for long running programs or Service Operations that have large amounts of data to process in that processing
would be performed by the Process Scheduler and not the IB Runtime.
The limitation of programmatically scheduling an AE program via the AE handler is that the TransactionID (used to
get message data) must be retrieved from within the AE program. The run control record PSIBRUNCNTLwill be
populated with the TransactionID prior to scheduling the actual AE Program. Note that it is not required to retrieve
this TransactionID therefore any AE program can be scheduled that does not require initialization parameters via a
run control table.
Note: For Asynchronous Request/ Response transactions, one can optionally add the Application Class information
(based off the INotificationHandler) that will then be used by the IB framework to automatically invoke once the
Application Engine process is successfully completed. The Status flow is as follows; The Subscription Contract
status would change to NEW instead of a DONE status. The Subscription Contract would process just like a typical
Application Class notification. It is up to the developer to publish the response message accordingly.
A Subscription Contract processing an Application Engine Handler cannot be Cancelled or Resubmitted from the
Service Operations page. This must be done on the Process Monitor Page. The status will then be updated on the
Subscription Contract by the IB runtime, at that time a refresh of the Service Operation Monitor page will result in
displaying this update status. Note that there could be a delay from when the Application Engine Program is
Cancelled or Resubmitted from the Process Monitor and when the update is actually seen on the Service
Operations Monitor. At least one Pub/Sub Domain must be active for this status update to happen.
Refer to IB PeopleBooks for Implementation details.

3.1.9 Application Considerations

Attributes used for Content based Routing
Setting attributes prior to invocation of a Service Operation and reading these attributes to perform content based
routing decisions for the OnRouteSend and OnSend IB PeopleCode events will have significant performance
improvement impacts. The rowset or xmldoc object would not have to be serialized and subsequently loaded into
memory and traversed to retrieve applicable data.
Refer to IB PeopleBooks for Implementation details.

32

Component Interfaces
If service operation notification messages use a Component Interface, and the data from x number of service
operations absolutely has to be processed in the one-minute slice of time, the notification process, not the IB
framework, will most likely be the bottleneck. Component Interfaces (CIs) are an easy way to reuse existing
business logic, and allow access to components just like in online transactions. When a CI is invoked, the page
processor has to gather information about the component and make it available to the PeopleSoft system, just like
when requesting a page for an online transaction. Performance will be slower when accessing a CI built on a large
component than when accessing a smaller component; there are more page and record definitions to gather
information on in order to be ready to accept a transaction. If the service operation notification process that is using
a CI does not use all of the CI functionality/business logic, or if the CI is built on a large component, then alternative
steps should be considered if the notification process is a bottleneck.
For incremental messages, the next steps to performance tuning a CI notification process are to:
6.
7.
8.

Evaluate the business logic in the component that is being used.

Identify the PeopleCode functions in the component that make up that business logic.
Rewrite the Notification PeopleCode make funclib calls to that component PeopleCode instead of using a CI.

An alternative for batch-oriented or chunked messaging is to load the messages into a staging table, and then run
an Application Engine job to gain the advantages of set-based SQL. The Application Engine job could be run very
often (i.e. once per minute) to minimize data latency.
Batch Processing
When a non-PeopleSoft system is going to send messages to PeopleSoft in more of a batch mode (i.e. not
triggered from online user transactions), individual message transactions should be grouped (chunked) together in
a single XML document and HTTP post. For example, one message coming in to PeopleSoft can have several
(maybe hundreds) of transactions in them, wherein each transaction was originally considered to be an individual
message. In an incremental/single transaction message, there is only one <Transaction> node under the
<MsgData> node. With multiple transactions, there are multiple <Transaction> nodes under the <MsgData> node.
There can be noticeable performance gain by doing this; since less work is done when making connections to the
application server, calling application server processes, etc. It is generally recommended that if the message size is
around three megabytes, a new message should be created.
Guidelines for Asynch Messaging
Messages should be partitioned as much as possible. Third-party clients should post to PeopleSoft in parallel. To
reduce delays due to posting/queuing, use large messages (multi-transaction) for full sync processes, and use
compression.
When publishing from PeopleCode, use the enqueue option as much as possible, or publish the message as late in
the transaction as possible. If a message is published early in a transaction, and the transaction takes a long time
to complete, a lock will be maintained on the queue sequence ID table, blocking other messages in the same
queue.
Notification PeopleCode should be as light as possible. For multi-transaction messages, commits should be
frequent to minimize contention. CIs should not be used for full sync processes. They may be used in incremental
sync processes where volumes are not expected to be high. It should be noted that CI performance is dependent
on the underlying component. If the component is light, it is possible to have a very fast CI. The heavier the base
component is, the slower the CI.
GetMessage is a relatively heavy call. The message actually passed into the method should be used and
referenced whenever possible in PeopleCode. Transform code (either PeopleCode or XSLT) should be as light as
possible. In general, XSLT transforms will be much faster than PeopleCode transforms.

33

rd

Message compression is automatic for PeopleSoft to PeopleSoft transactions. For 3 party applications, set the
HTTP target connector property sendUncompressed to N. Compression can reduce response time by 20 to 80%
rd
based on message size. By adding a simple relay servlet, 3 party messaging can take advantage of compression.
Areas to review when using the Full Sync EIPs:
Database Tuning Create database statistics in a test environment from test runs and export them before
production runs. Full Sync EIPs are used for initial database creation. Empty databases do not have database
statistics. If EIPs are using temp tables, make sure the indexes are available for reads. After temp table
population, create the database statistics for these tables.
10. Archiving Enable the archive option on the Queue page for queues designated for EIPs. This will cause the
message data to be archived into other IB PeopleTool tables. The Full Syncs are one-time in nature and hence do
not need to be archived.
11. Database Layout Plan database layout by splitting PeopleTool tables from application tables for better
throughput.
12. Additional Tips:
Enable archiving for Full Sync or any process where archiving is not required.
Deactivate all message queues that are not being used.
After Full Sync is run, the DBA should update the database statistics for the new (grossly
changed) table sizes.
Keep commits at the transaction level and not at the message level. This will reduce locking and
improve performance.
9.

Guidelines for Sync Messaging

Response time tends to be important in synchronous requests, so transforms and OnRequest PeopleCode should
to be light. CIs should only be used if the base components are light and have quick response times.
Keep the number of sync requests from a client to a minimum or use synchronous threading. For example, if you
have a certain amount of work to be done on a remote system, pack as much of it into as few of calls as possible.
This will reduce the amount of PeopleCode message overhead to instantiate request/response messages.
In general, sync messages should be small to improve response time. Third-party clients should use compression,
if possible (we do this by default between PSFT applications). Partitioning is not an issue with sync messaging.
Sync messaging should typically be used for remote queries, not for remote inserts/updates/deletes. If they are
used for inserts/updates/deletes, SyncRequests do not share a transaction context with the requesting system. If a
SyncRequest has been completed, and the client transaction rolls back, the SyncRequest will not be rolled back.
(Publishes will be rolled back).
Also, a component should not depend on a SyncRequest being successful. If the remote system is down, the
SyncRequest will fail. The requesting application should be prepared to deal with this by using exception logic, or
should use the Asynch system. If the remote system is down, the Asynch system will retry. If you need to send
data to multiple systems at the same time, use threaded sync requests or possibly Asynch messaging. The Asynch
system is much more efficient at fanning out information to multiple target systems.

3.2 SYNCHRONOUS PROCESSING

This section will discuss several ways to increase synchronous performance using the IB Framework.

3.2.1 Multi-Threading Sync Requests

Multi-threading of the SyncRequest Handler allows multiple PeopleCode SyncRequests to be performed
simultaneously within one PeopleCode SyncRequest call. Previously the PeopleCode SyncRequest call pertained
to one IB transaction and was a blocking call. In order to perform multiple SyncRequests, the invocation had to be
done one after the other. The SyncRequest PeopleCode can now accept an array of messages and return an array
34

of messages. However if one request fails, the entire component transaction will be rolled back. With tools 8.48,
the developer can interrogate the error response message for each SyncRequest message in the array (or single
message), and determine to continue processing the array of response messages and component transaction.
Please refer to Integration Broker PeopleBooks for examples of how to code threaded SyncRequests.
Processing SyncRequests
This is the syntax for a threaded SyncRequest call: (Array of Response Messages = %IntBroker.SyncRequest(Array
of Request Messages)). The IntBroker object will be responsible for managing the messages, instantiation of the
SyncRequest Handler and calling the Send method for each request message. The IntBroker object will then poll
the SyncRequest Handler object to determine when all processing is complete. At that time status and error
checking will be performed and creation of the response message objects will take place, packaging these
messages as an array and returned to the calling method.

3.2.2 Exception Handling

This functionality available in 8.48 tools will allow for sync requests that fail to throw a framework exception leading
to a message box error and subsequent component roll back of the transaction. This type exception handling would
also work with threaded sync requests. For example, if 10 sync requests are performed in parallel (threaded sync
request), the developer will have the option to select the User Exception check box on the routing. This means that
if any of the sync requests error, the component will not be rolled back. The developer can check each sync
request to determine if there is an error and actually read the associated error message. The developer can then
throw an exception or go on to process the next sync request in the array.
Synchronous service operations can use this feature. This pertains only to outbound sync request calls originating
within the application server (i.e. %IntBroker.SyncRequest( )).
On the routing page, select the User Exception check box: Here is example PeopleCode to read the exception:

Local Rowset &FLIGHTPLAN, &FLIGHTPLAN_RETURN;

Local array of Message &messages;
Local array of Message &return_mesages;
&messages = CreateArrayRept(&MSG, 0);
&return_mesages = CreateArrayRept(&MSG, 0);
QE_FLIGHTDATA.QE_ACNUMBER.Value = QE_FLIGHTDATA.QE_ACNUMBER + 1;
&FLIGHT_PROFILE = GetLevel0();
&rs1 = &FLIGHT_PROFILE.GetRow(1).GetRowset(Scroll.QE_NAVIGATION);
&rs2 = &FLIGHT_PROFILE.GetRow(1).GetRowset(Scroll.QE_RADAR_PRESET);
&rs3 = &FLIGHT_PROFILE.GetRow(1).GetRowset(Scroll.QE_ARMAMENT);
&messages [1] = CreateMessage(Operation.SYNC_PARTS);
For &i = 1 To &messages [1].PartCount
If &i = 1 Then
&rs1.CopyTO(&messages [1].GetPartRowset(&i));
End-If;

35

If &i = 2 Then
&rs2.CopyTO(&messages [1].GetPartRowset(&i));
End-If;
If &i = 3 Then
&rs3.CopyTO(&messages [1].GetPartRowset(&i));
End-If;
End-For;
&messages [2] = CreateMessage(Operation.SYNC_PARTS);
For &i = 1 To &messages [2].PartCount
If &i = 1 Then
&rs1.CopyTO(&messages [2].GetPartRowset(&i));
End-If;
If &i = 2 Then
&rs3.CopyTO(&messages [2].GetPartRowset(&i));
End-If;
If &i = 3 Then
&rs2.CopyTO(&messages [2].GetPartRowset(&i));
End-If;
End-For;
&return_mesages = %IntBroker.SyncRequest(&messages);
If &return_mesages [1].ResponseStatus = %IB_Status_Success Then
For &i = 1 To &return_mesages [1].PartCount
//perform local processing on response data
End-For;
Else
&nMsgNumber = &return_mesages [1].IBException.MessageNumber;
&nMsgSetNumber &return_mesages [1].IBException.MessageSetNumber;
&exceptString = &return_mesages [1].IBException.ToString();
// Evaluate exception and throw error if necessary
End-If;
If &return_mesages [2].ResponseStatus = %IB_Status_Success Then

36

For &i = 1 To &return_mesages [2].PartCount

//perform local processing on response data

End-For;

Else
&nMsgNumber = &return_mesages [2].IBException.MessageNumber;
&nMsgSetNumber &return_mesages [2].IBException.MessageSetNumber;
&exceptString = &return_mesages [2].IBException.ToString();
// Evaluate exception and throw error if necessary
End-If;

37

3.2.3 Transactional Sync Requests

For local sync requests to insure transactional boundary separation the request is sent out to the gateway and then
returned to run the OnRequest event under a different application server process. However, for cases where
running the OnRequest event under the same transactional boundary is desirable then one can use the
Transactional sync request option. In this case the request is NOT sent to the gateway and the OnRequest event is
run on the same process saving the overhead of sending the request to the gateway and authentication/ validation
of the subsequent inbound request (to process the OnRequest event).
When the Local-to-Local check box is selected the Transactional check box will be available for selection. By
default, the Transactional check box is not enabled. This means that a local synchronous transaction operates as
before (since 8.4). If the Transactional check box is selected then the OnRequest event will run under the existing
transaction boundary of the component.

WARNING: If outbound routings where created prior to saving the Service Operation with local-to-local and
Transactional check box true, these routings will become inactive. Creating an outbound routing (local to remote)
with Transactional check box true will always be saved inactive.

38

3.2.4 Message Size vs. Compression

In general, Sync messages should be as small as possible to improve response time. Third-parties should always
use compression if possible. Compression from PeopleSoft is enabled based on message size as defined by the
PSADMIN parm (Min Message Size For Compression). The following graphic shows how compression can reduce
the overall message size.

3.2.5 Compress override capability for sync messages

The configuration of determining when to compress sync request content data is not granular enough. The
customer needs the option to always compress or not compress at the transaction level, regardless on configured
compressed size as defined in psadmin. Note that compress option on the HTTP target connector will still be
honored.
&MSG.IBInfo.CompressionOverride = %IntBroker_Compress;

Set/Get property for Compression override. Valid parms : %IntBroker_Compress,

%IntBroker_UnCompress, %IntBroker_Compress_Reset

3.2.6 Tuning the App Server

Consumer of Sync Service Operations
The consumer (source) can have multiple domains only if the online application uses multiple domains.
SyncRequest calls share domains with online requests. Therefore you should include the potential SyncRequest
message load when determining proper number of PSAPPSRVs to configure. This includes response times from
rd
the requests. Also consider if response times are high due to 3 -party processing or transmission latency.
Provider of Sync Service Operations
A dedicated application server domain should be configured. This should be separate from any online requests.
Dedicated resources can then be used and the estimated number of PSAPPSRV processes can be determined
based on Sync message load and response time characteristics.
Jolt Handlers
Size should be based on the number of messaging users per domain. Optimally try to run 20 users per handler.
The single handler footprint is about 2M. Here is an example of PSAPPSRV.cfg settings:
[JOLT Listener]

39

;=========================================================================
; Settings for JOLT Listener
;=========================================================================
Address=%PS_MACH%
Port=9000
Encryption=0
Min Handlers=60
Max Handlers=80
Max Clients per Handler=20
Client Cleanup Timeout=6
Init Timeout=5
Client Connection Mode=ANY
Jolt Compression Threshold=1000000

Use the number of CPUs to determine the number of PSAPPSRV processes to configure. For tuning, consider the
pq and psr Tuxedo commands while processing requests to determine the proper size of the domain. Note that
Tuxedo queuing is OK as long as it clears up quickly.

3.3 TOOLS TO DETERMINE BOTTLENECKS

There a few tools that can be used to identify and locate bottle necks within the IB system. This section covers the
tools that are part of the delivered product.

3.3.1 Tuxedo commands

pq This command shows the current Tuxedo queuing on the domain.
psr This command shows the current service request for each server on the domain and also the load (number of
requests) that the server processed.
Tuxedo settings
Message Segment Size Increase size from default 24 to 8K or 16K
Message Queue Size Increase from 64K to 1M or 2M
Max Message Size - Increase from 64K to 1M or 2M

40

3.3.2 Profile and Statistics

Use the Statistics component in Service Operations Monitor to collect profile information during runtime. Collection
of these statistics is enabled by selecting a check box on the Monitor Setup Options.

Please refer to IB PeopleBooks for complete explanation on how to use this component.

3.4 GATEWAY CONFIGURATION

The gateway is fairly thin in terms of overall processing time. Load balancing across three gateways is better than
having one gateway dedicated for messaging. A disadvantage of having one dedicated gateway is when the
messaging activity doesnt consume a majority of the system resources, you have idle processing power that is not
being used for online users.
The following case illustrates an exception. Throughput per slice of time for messages is very important. The
gateways are used for PIA access to self-service pages. There is a benefit enrollment window of time wherein a
number of employees are accessing PIA, causing an unusually large resource hit on the web servers. In this case,
throughput used for messages will be adversely affected.
Settings to use when configuring a dedicated Integration Gateway:
Use of concurrent Peak Message Load to set system threads.
WebLogic
Weblogic Server performs dynamic thread allocation, therefore there is no need to set Minimum or Maximum
number of threads.

WebSphere

41

Set MinimumSize and MaximumSize to be the same.

JVM Heap Settings

Set MinimumHeapSize and MaximumHeapSize the same. This will improve garbage collection and lesser resource
allocation during runtime. Set HeapSize to a minimum of 512Mb, subject to real memory on the server.
JDK 1.3 Hotspot has many performance knobs. By default, thread memory is 512K for SPARC, 256K Intel.
Therefore set this to 1M.
Set OS File Descriptor to 100*Thread Count
TCP Timeout Settings
Messaging is socket intensive, given the IB connector architecture. Once a session is completed, a Socket is
marked for closure. Sockets disconnect only when Socket Timeout expires. The defaults range from 4 to 11
minutes. Therefore reduce related OS TCP Time Wait Parameter to between 30 seconds and 1 minute.
Servlet Reload
This setting is internal to JVM to automatically reload the servlets. In production environments, the servlets do not
change very often. Therefore disable this parameter, so that reload does not occur.

42

Chapter 4 Configuration
The following diagrams show a few ways that IB can be configured to maximize performance.

4.1 Single Gateway

Domain 1
Gateway

Application
Server

Domain 3

Pub/Sub
Master

Pub/Sub
Slave

Failover Domain 2
Application
Server

Domain 4

Pub/Sub
Master

Pub/Sub
Slave

This is a basic configuration and the easiest to implement. Create multiple domains with pub/sub booted, on
different machines.

Domain 1 should be on the machine that has the most CPU power and is the primary domain to process messages.
Domain 2 should be configured on another machine, which is designated as the failover domain.
Domains 3 and 4 (optional, only needed if performance number still not met) can be on additional machines and
configured as static slaves.
When failover is configured, make sure the dynamic failover check box is selected making Domain 2 a dynamic
slave.

4.2 Multiple Gateways, Dedicated Servers

Domain 1
Gateway

Application
Server

Domain 4
Pub/Sub
Master

Pub/Sub
Slave

Queues A-E
Queues A-Z

Domain 2
Gateway

Application
Server

Domain 5
Pub/Sub
Master

Pub/Sub
Slave

Queues F-Q
Queues A-Z

Domain 3
Gateway

Application
Server

Domain 6
Pub/Sub
Master

Pub/Sub
Slave

Queues R-Z
Queues A-Z

43

This configuration uses domains that have dedicated pub/sub servers enabled. Create dedicated pub/sub servers
along with the appropriate amount of PSAPPSRV processes to handle the inflow of messages. These PSAPPSRV
processes are for Integration Broker use onlydo not configure PIA using these domains. The URL used by the
source system (consumer) should be the appropriate gateway address for the message queue being processed. At
this point it easy to scale by using failover where Domains 1, 2 and 3 are in their own group. The static slave
Domains 4, 5, and 6 do not have to be dedicated serversuse the default pub/sub configuration.

4.3 Gateway/Application Server Load Balancing

Application
Application
Server
Server

Domain 1
Domain 4
Pub/Sub
Master

Gateway
Application
Application
Server
Server
Gateway
Load Balancer

Gateway

Application
Server
Load Balancer

Domain 2
Domain 5
Pub/Sub
Master

Application
Application
Server
Server

Gateway

Gateway
Application
Application
Server
Server

Pub/Sub
Slave

Pub/Sub
Slave

Failover
Domain 3

Domain 6

Pub/Sub
Master

Pub/Sub
Slave

Failover

In this case the load is so high that load balancers (gateway/application server) are needed to handle the extreme
high volume of messaging. The pub/sub servers are not part of the incoming request domains. The pub/sub
servers will use on-idle processing to pick up the data initially. The scan interval should be set to 1. For processing
high volumes of data you should also consider using the load balance feature if the high volume messages (service
operations) are in different queues. Scalability can be accomplished by adding additional Slaves or by creating
dedicated servers in combination with failover.

44

Chapter 4 Additional Documentation

The IB PeopleBooks are a great source to understand all the components of the Integration Broker. They also
contain many examples on how to use the performance features of IB such as Message Segments, Sync MultiThreading, and Master Slave.

5.1 CUSTOMER BENCH MARKS

The following numbers provide some idea of whats possible with the Integration Broker system. These are actual
numbers from tests conducted at a customer site on Tools 8.48 on a 4CPU/4GB RAM Sun/Solaris app server
machine. The base infrastructure is capable of running at 7600 transactions/minute (no transform or subscription
processing at all). The messages are prewritten to the queue in unordered queues, which means that theres no
transport or delivery-sequencing overhead.
When transforms were turned on, only 187 transactions could be processed each minute, and when notifications
were turned on (without transforms) only 395 transactions could be processed in a minute. The messages (~50KB
uncompressed) can be written to the queue at the rate of 700 transactions per minutethis may actually be limited
by the load generator, not the app server.
A complete end-to-end run (with everything turned on, and posting from a client Load Runner script) could process
~120 transactions per minute.
In summary, for the test scenario, the message posting/queuing portion of the test took up 14% of the processing
time, transforms took up 58%, subscription PeopleCode (theyre not using CIs,) took up 27%, and the base
infrastructure took up 1%.
Outbound publishes processed at the rate of 1500 transactions per minute (50KB, compressed, no transforms, no
ordering, posting to a file on the gateway).

45

5.2 PEOPLETOOLS PERFORMANCE STUDY: INTEGRATION BROKER

1/9/2006 Synchronous Messaging
Hardware

Appservers
Source Domain

Target Domain

OS: Win2003 Server 5.2

OS: Win2003 Server 5.2

Processor: 4 x 3 GHz

Processor: 4 x 3 GHz

Memory: 4 GB

Memory: 4 GB

Domain: F8481073
Gateway

Domain: F8481073

OS: Win2003 Server 5.2

Processor: 4 x 3 GHz
Memory: 4 GB

DBServers
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB
Databases: F8481073 & F8481073

LoadDriver
N/A

Test Scenario 1
Messages Sent: 10
Target PSAPPSRV Processes: 10
Thread Pool Sizes: 5, 10, 15, 20
Target Processing Delays (sec): 1, 5, 10
Iterations: 20

Load: The PeopleCode behind PeopleTools Quality QE IB Test QE Flight Data Sync was modified to pass
an array of 10 messages to a single call of %IntBroker.SyncRequest as follows:
For &curIteration = 1 To &iterations
&start = %PerfTime;
&return_mesages = %IntBroker.SyncRequest(&messages);
&etime = %PerfTime - &start;
&log.WriteLine(&msgCount | " messages: " | NumberToString("10.3", &etime) | "
seconds.");

46

End-For;

&iterations were set to 20 resulting in 200 messages sent from a single ICPANEL request in the source domain.
The target domain had 10 PSAPPSRV processes running to process the incoming messages. Elapsed times were
gathered by calling %PerfTime before and after the call to SyncRequest() as shown in the code snippet above. The
test was repeated for 5, 10, 15, and 20 threads in the source domain and for 1, 5, and 10-second processing delays
in the target domain.
Results Overview

Thread Pool Size

Delay

10

15

20

1 sec

3.077

2.066

2.089

2.057

5 sec

11.092

6.056

6.070

6.083

N/A

11.088

11.056

11.060

10 sec

Sending 10 messages that take 10 seconds to process with only 5 threads in the source domain could not complete
before the ICPANEL request timeout was exceeded. However messages with 1 second and 5 second processing
times finished in approximately 3 and 11 seconds respectively. Reading across a row for a given processing delay
you can see there is significant gain when increasing from 5 to 10 source threads, but as expected, little to no gain
in having more threads than messages in the burst.

Detailed Results
1 Second Processing Delay, 10 Message Burst

1 Sec

Thread Pool Size

47

Delay

10

15

20

3.297

2.344

2.250

2.094

3.094

2.094

2.078

2.125

3.000

1.969

2.110

2.031

2.968

1.984

2.015

2.063

3.094

1.985

2.079

1.984

2.985

2.078

2.015

2.078

3.109

2.062

2.078

1.969

3.109

2.078

1.985

2.078

3.047

2.063

2.078

1.984

10

3.110

2.078

1.969

1.985

11

3.078

1.984

2.078

2.094

12

3.078

2.063

2.406

2.078

13

3.094

1.969

2.078

2.140

14

3.109

2.093

2.078

1.985

15

3.016

2.110

2.078

2.031

16

2.968

2.078

2.094

2.078

17

3.078

2.062

2.078

2.078

18

3.125

2.079

2.094

2.079

19

3.110

2.078

2.078

2.109

20

3.062

2.078

2.063

2.078

Avg

3.077

2.066

2.089

2.057

SyncRequest Elapsed Time by Iteration

3.5
3.3

3.5

48
sed Time (sec)

Avg SyncRequest Time

3.1
2.9
2.7
2.5
2.3

5 Threads
10 Threads
15 Threads

5 Second Processing Delay, 10 Message Burst

Thread Pool Size

5 Sec
Delay

10

15

20

11.344

6.078

6.297

6.250

11.141

6.109

6.094

6.062

11.094

6.063

6.062

5.969

11.079

6.109

6.094

6.062

11.141

6.094

6.078

6.094

11.000

6.078

5.984

5.984

11.079

6.079

6.094

6.094

11.078

6.078

6.094

6.094

11.157

6.062

6.078

6.062

10

11.062

5.969

6.078

6.063

11

10.969

6.063

6.063

6.078

12

11.110

5.953

6.078

6.094

13

11.109

5.984

6.078

6.109

14

11.000

6.047

6.062

6.078
49

15

11.110

5.969

5.969

6.125

16

11.109

6.109

5.984

6.078

17

11.079

6.078

6.094

6.079

18

11.047

6.063

6.078

6.078

19

11.031

6.062

6.032

6.109

20

11.094

6.063

6.015

6.094

Avg

11.092

6.056

6.070

6.083

SyncRequest Elapsed Time by Iteration

13.5

Avg SyncRequest Time

Elapsed Time (sec)

11.5
12.0

Elapsed Time (sec)

10.0

8.0

9.5

5 Threads
10 Threads

7.5

15 Threads
5.5

20 Threads

3.5
1.5
1

6.0

11

Iteration
4.0

5 Threads
2.0

10 Threads
15 Threads
20 Threads

0.0

Thread Pool Size

10 Second Processing Delay, 10 Message Burst

10
Sec
Delay

Thread Pool Size

5

10

15

20

N/A

11.375

11.125

11.203

N/A

11.094

11.000

11.078

N/A

11.141

11.078

10.969

N/A

11.109

11.000

11.062

50

13

15

17

19

N/A

11.094

11.063

11.000

N/A

11.094

11.078

11.063

N/A

11.109

11.062

11.078

N/A

11.062

10.985

11.078

N/A

11.079

11.078

11.078

10

N/A

11.078

11.093

11.094

11

N/A

11.078

11.079

11.078

12

N/A

11.078

11.078

11.078

13

N/A

10.984

11.093

11.078

14

N/A

11.078

10.985

11.110

15

N/A

10.985

10.969

10.968

16

N/A

11.109

11.125

11.078

17

N/A

11.000

11.078

11.094

18

N/A

11.063

11.093

10.985

19

N/A

11.078

10.985

11.062

20

N/A

11.078

11.078

10.969

Avg

N/A

11.088

11.056

11.060

SyncRequest Elapsed Time by Iteration

11.5

Avg SyncRequest Time

Elapsed Time (sec)

11.1

11.1

Elapsed Time (sec)

11.4
11.1

11.3
10 Threads

11.2

15 Threads

11.1

20 Threads

11.0
10.9
10.8

11.1

11

13

15

17

19

Iteration
11.1

51
11.1

10 Threads

Test Scenario 2:
Messages Sent: 20
Target PSAPPSRV Processes: 20
Thread Pool Sizes: 5, 10, 15, 20
Target Processing Delays (sec): 1, 5, 10
Iterations: 20
Test scenario 2 is very similar to Test 1 but is designed to exploit thread counts greater than 10 by sending 20
messages in a burst with 20 target PSAPPSRV processes.
Results Overview

Source Threads
Delay

10

15

20

1 sec

5.874

3.612

3.227

2.932

5 sec

N/A

11.591

11.138

7.051

10 sec

N/A

21.712

21.145

11.876

In this case you can see increasing to 20 threads in the source domain greatly improves throughput for 20-message
bursts while there is little difference for the 10 and 15 thread configurations. Also with 20 messages in the burst we
were unable to complete 20 iterations with only 5 threads and with 5 and 10 second delays on the target before the
ICPANEL timeout was surpassed.

1 Second Processing Delay, 20 Message Burst

1 Sec
Delay
1

Thread Pool Size

5

10

15

20

6.047

3.828

3.469

3.219

52

5.844

3.672

3.235

3.046

5.937

3.672

3.218

2.954

5.938

3.640

3.172

2.953

5.844

3.594

3.063

2.843

5.953

3.516

3.172

2.860

5.859

3.625

3.265

2.953

5.844

3.515

3.141

2.844

5.875

3.594

3.156

2.937

10

5.891

3.609

3.172

2.938

11

5.750

3.594

4.266

2.890

12

5.813

3.656

3.062

2.953

13

5.859

3.485

3.250

2.844

14

5.813

3.672

3.156

2.828

15

5.750

3.593

3.141

2.953

16

5.859

3.641

3.047

2.938

17

5.875

3.547

3.156

2.922

18

5.875

3.609

3.172

2.828

19

5.922

3.578

3.172

2.953

20

5.938

3.594

3.047

2.984

Avg

5.874

3.612

3.227

2.932

(1 Second Processing Delay, 20 Message Burst continued)

53

Avg SyncRequest Time

7.0

6.0

Elapsed Time (sec)

5.0

4.0

3.0

2.0
5 Threads
10 Threads

1.0

15 Threads
20 Threads

0.0
Thread Pool Size

SyncRequest Elapsed Time by Iteration

8.0

Elapsed Time (sec)

7.0
6.0

5 Threads
10 Threads

5.0

15 Threads
4.0

20 Threads

3.0
2.0
1

11

13

15

17

Iteration

5 Second Processing Delay, 20 Message Burst

54

19

Thread Pool Size

5 Sec
Delay

10

15

20

N/A

11.891

11.422

7.047

N/A

11.578

11.031

6.969

N/A

11.625

11.047

7.047

N/A

11.609

11.141

7.062

N/A

11.500

11.172

6.938

N/A

11.563

11.047

6.953

N/A

11.578

11.078

7.031

N/A

11.531

11.250

6.922

N/A

11.578

11.140

7.047

10

N/A

11.625

11.141

8.015

11

N/A

11.594

11.047

6.954

12

N/A

11.578

11.078

6.937

13

N/A

11.641

11.031

7.047

14

N/A

11.578

11.172

7.031

15

N/A

11.500

11.141

6.906

16

N/A

11.625

11.172

6.922

17

N/A

11.500

11.125

7.141

18

N/A

11.484

11.140

7.031

19

N/A

11.625

11.235

7.031

20

N/A

11.625

11.140

6.985

Avg

N/A

11.591

11.138

7.051

SyncRequest Elapsed Time by Iteration

12.0

(sec)

11.0
10.0

55

Avg SyncRequest Time

14.0

12.0

Elapsed Time (sec)

10.0

8.0

6.0

4.0
10 Threads
2.0

15 Threads
20 Threads

0.0
Thread Pool Size

10 Second Processing Delay, 20 Message Burst

10
Sec
Delay

Thread Pool Size

5

10

15

20

N/A

21.907

21.531

12.187

N/A

21.531

21.078

11.875

N/A

21.640

21.141

11.938

N/A

21.625

21.172

11.828

N/A

21.704

21.078

11.875

N/A

21.937

21.078

11.859

N/A

21.609

21.219

11.735

N/A

21.610

21.093

11.859

N/A

21.609

21.188

11.891

10

N/A

21.594

21.094

11.859

11

N/A

21.735

21.153

11.871

56

12

N/A

21.548

21.087

11.758

13

N/A

21.724

21.114

11.983

14

N/A

21.824

21.127

11.882

15

N/A

21.629

21.087

11.857

16

N/A

21.824

21.087

11.895

17

N/A

21.693

21.291

11.753

18

N/A

21.817

21.039

11.895

19

N/A

21.735

21.188

11.819

20

N/A

21.954

21.049

11.895

Avg

N/A

21.712

21.145

11.876

SyncRequest Elapsed Time by Iteration

26.0

Avg SyncRequest Time

24.0

Elapsed Time (sec)

25.0

Elapsed Time (sec)

20.0

15.0

22.0
20.0

10 Threads

18.0

15 Threads

16.0

20 Threads

14.0
12.0
10.0
1

10.0

11

13

15

17

19

Iteration

5.0

10 Threads
15 Threads
20 Threads

0.0
Thread Pool Size

1/9/2006 Asynchronous Messaging

Hardware

Appservers
Source Domain
Name: PBNT0017

Target Domain
Name: PBNT0016
57

OS: Win2003 Server 5.2

Processor: 4 x 3 GHz
Memory: 4 GB
Domain: F8481073
Gateway

OS: Win2003 Server 5.2

Processor: 4 x 3 GHz
Memory: 4 GB
Domain: F8481073

Name: PBNT0017
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB

DBServers
Name: PBNT0016 & PBNT0017
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB
Databases: F8481073 & F8481073

LoadDriver
N/A

Test 1
Load: 1000 Message Publication Contracts
Source: 1 PSPUBHND process
rd

Target: 10 PSAPPSRV processes 1 Second Delay in processing (simuate 3 party processing)

Variable: Thread Pool Size for PSPUBHND process

Description
We first populated PSAPMSGPUBCON with 1000 publication contracts for the QE_FLIGHTPLAN message. For
each iteration of the test we inactivated the messaging domain and reset the state of the contracts to New. We
than activated the domain and let PSPUBHND churn through the 1000 messages. Results were gathered by
computing the difference between the LastUpdateTime on the last message sent and the first message sent. We
completed six iterations for each Thread Pool Size.

SQL for resetting the contract state:

update PSAPMSGPUBCON set PUBCONSTATUS=1, STATUSSTRING='NEW', RETRYCOUNT=1

SQL for gathering the data:

select count(*) as 'Done', 1000 - count(*) as 'Remaining', min(LASTUPDDTTM) as 'Start',
max(LASTUPDDTTM) as 'Last Done', datediff(ms, min(LASTUPDDTTM), max(LASTUPDDTTM)) as
'Elapsed' from PSAPMSGPUBCON where STATUSSTRING = 'DONE'

Results:
58

Time in ms to send 1000 messages

PSPUBHND Thread Pool Size
1 Thread

5 Threads

10 Threads

15 Threads

20 Threads

1184546

359013

421720

405170

402856

1182826

242513

413610

409483

402046

1184543

238093

420156

401983

408733

1183306

238546

404580

382546

396483

1184216

241860

422766

463003

415220

1184016

239783

406966

409050

397203

Average

1183.909

259.968

414.966

411.873

403.757

Sec/Msg

1.184

0.260

0.415

0.412

0.404

Total time to send 1000 messages

Seconds/Message

1320000
1.400

1120000
1 thread

920000

5 threads

720000

1.200

10 threads
15 threads

520000

1.000

20 threads

0.800

320000
120000

0.600

6
1 thread

0.400

5 threads
10 threads

0.200

15 threads
20 threads

Test 2

0.000
1

Load: 1000 Message Publication Contracts

Source: 1 PSPUBHND process
Target: 10 PSAPPSRV processes, 5 Second Delay in processing
Variable: Thread Pool Size for PSPUBHND process
Test 2 is identical to Test 1, except a 5 second delay was added to PSAPPSRV in the target domain.

59

Results:
PSPUBHND Thread Pool Size
1 Thread

5 Threads

10 Threads

15 Threads

20 Threads

5181176

1035096

523850

523060

707110

5182783

1033426

520063

523406

745406

5190126

1033830

516140

526766

733656

5186314

1033636

518486

523986

572360

5187086

1033300

519843

517626

670296

5180970

1032953

518843

519796

748670

Average

5184.743

1033.707

519.538

522.440

696.250

Sec/Msg

5.185

1.034

0.520

0.522

0.696
Seconds/Message

Total time to send 1000 messages

1.200

1120000
1020000

1.000

920000
5 threads

820000

10 threads

720000

15 threads

0.800

20 threads

620000
0.600

520000
420000
1

0.400

6
5 threads
10 threads

0.200

15 threads
20 threads
0.000
1

1/16/2006 Message Segments

Hardware

Appservers
Source Domain
Name: PBNT0017
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB
Domain: F8481073

Target Domain
Name: PBNT0016
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB
Domain: F8481073
60

Gateway
Name: PBNT0017
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB

DBServers
Name: PBNT0016 & PBNT0017
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB
Databases: F8481073 & F8481073

LoadDriver
N/A

Description
The Segments test cases were designed to explore elapsed time to move 40 megabytes from the source to the
target domain using message segmentation. The data was organized first as 20 distinct 2 MB messages, second
as a 2 MB message replicated into 20 ordered segments, and finally as a 2 MB message replicated into 20
unordered segments.
Each case was executed for 6 iterations. Elapsed time was calculated by looking at the time the source domain
created the publication contract and the time the target domain completed servicing the subscription contract.

For case one (20 2MB messages) the elapsed time is found by subtracting the LASTUPDDTTM timestamp on the
first row on PSAPMSGPUBHDR in the source database from the LASTUPDDTTM in the 20 th row on
PSAPMSGSUBCON in the target database.
In the 2nd case (1 message, 20 ordered segments), only a single subscription contract is needed so the time is found
by simply subtracting the corresponding row on PSAPMSGPUBHDR in the source database.
In the 3rd case (1 message broken into 20 unordered segments), 20 distinct subscription contracts are created, one
for each segment, for only one row on PSAPMSGPUBHDR (1 message) so the elapsed times for each iteration
was found by taking the 1st row on the source from the 20th row on the target, the 2nd row on the source from the
40th row on the target, etc, for all 6 iterations.

First making the source messaging domain inactive, and then resetting the STATUSSTRING on
PSAPMSGPUBHDR row(s) to NEW, and finally reactivating the messaging domain ran iterations.
Results
20 Messages

20 Ordered
Segments

20 Unordered
Segments

80.326

20.753

18.59

79.500

20.793

19.706

77.840

21.007

18.163

79.377

21.173

18.773

79.710

22.977

18.993

61

80.040

21.266

19.36

Avg

79.466

21.328

18.931

MB/Sec

0.503

1.875

2.113

Megabytes/Second

Avg Tim e (sec) 40 MB Transfer

3
90
80

70

MB/Sec

Seconds

60
50
40
30

Messages

20

Ordered Segments

2
1
1

Unordered Segments

10

Total Time for 40 MB Transfer

90.0
80.0

Seconds

70.0
60.0

Messages

50.0

Ordered Segments

40.0

Unordered Segments

30.0
20.0
10.0
0.0
1

Iterations

1/12/2006 Message Parts

Hardware

Appservers
Source Domain
Name: PBNT0017
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB
Domain: F8481073
Gateway

Target Domain
Name: PBNT0016
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB
Domain: F8481073

Name: PBNT0017
62

OS: Win2003 Server 5.2

Processor: 4 x 3 GHz
Memory: 4 GB

DBServers
Name: PBNT0016 & PBNT0017
OS: Win2003 Server 5.2
Processor: 4 x 3 GHz
Memory: 4 GB
Databases: F8481073 & F8481073

LoadDriver
N/A

Description
Testing Message Parts was a simple repeat of the Synchronous Messaging tests with the Message Parts feature
enabled for 3-part messages. PeopleCode was also added to the messages OnRequest method on the target to
better simulate a real world implementation.

Test Scenario
Messages Sent: 20
Target PSAPPSRV Processes: 20
Thread Pool Sizes: 1, 5, 10, 15, 20
Target Processing Delays (sec): 1, 5, 10
Iterations: 10

Results

Thread Pool Size

Delay

10

15

20

1 sec

24.999

5.849

3.662

3.399

3.352

5 sec

N/A

21.843

11.595

11.169

7.278

10 sec

N/A

N/A

21.561

21.232

12.239

63

As with simple Synchronous Messaging, we were unable to complete a request with 10 iterations of 20 message
bursts with few threads and longer processing delays before the ICPANEL timeout was reached. Also, sending with
Parts came out a little slower than sending without Parts.
1 Second Delay, 20 Message Burst

1
thread

5
threads

10
threads

15
threads

20
threads

25.266

5.906

3.937

3.438

3.547

25.000

5.859

3.656

3.406

3.500

25.126

5.797

3.641

3.500

3.297

25.047

5.813

3.687

3.407

3.344

24.938

5.906

3.672

3.437

3.281

24.906

5.782

3.547

3.328

3.282

24.907

5.922

3.546

3.360

3.375

24.906

5.828

3.656

3.312

3.234

24.954

5.812

3.625

3.391

3.281

10

24.938

5.860

3.656

3.406

3.375

Avg

24.999

5.849

3.662

3.399

3.352

SyncRequest Elapsed Time by Iteration

Avg SyncRequest Time

25

Elapsed Time (sec)

26
21
1 thread
16

5 threads
10 threads

11

15 threads
20 threads

Elapsed Time (sec)

20
1
1

15

Iteration
10
1 thread
5 threads
5

10 threads
15 threads

64

10

5 Second Processing Delay, 20 Message Burst

5
threads

10
threads

15
threads

20
threads

22.235

11.703

11.234

7.344

21.844

11.609

11.235

7.312

21.828

11.562

11.219

7.250

21.813

11.610

11.203

7.282

21.813

11.578

11.125

7.265

21.735

11.593

11.203

7.172

21.828

11.563

11.219

7.297

21.766

11.594

11.094

7.297

21.719

11.578

11.079

7.282

10

21.844

11.562

11.078

7.281

Avg

21.843

11.595

11.169

7.278

SyncRequest Elapsed Time by Iteration

Elapsed Time (sec)

26
21
5 threads

16

10 threads
15 threads

11

20 threads
6
1
1

Avg SyncRequest Time

25

10

Iteration

65

Above: Shows the total time SyncRequest() blocked for each iteration for each thread pool size.

10 Second Processing Delay, 20 Message Burst

5 threads

10 threads

15 threads

20 threads

N/A

21.563

21.515

12.266

N/A

21.624

21.235

12.297

N/A

21.563

21.235

12.172

N/A

21.562

21.219

12.188

N/A

21.484

21.109

12.313

N/A

21.563

21.204

12.219

N/A

21.562

21.234

12.203

N/A

21.547

21.360

12.250

66

N/A

21.594

21.094

12.188

10

N/A

21.547

21.110

12.297

Avg

N/A

21.561

21.232

12.239

SyncRequest Elapsed Time by Iteration

Avg SyncRequest Time

25

Elapsed Time (sec)

26

Elapsed Time (sec)

20

21
16

10 threads
15 threads

11

20 threads

6
1
1

15

10

Iteration
10
10 threads
5

15 threads
20 threads

0
Thread Pool Size

Above: Shows the total time SyncRequest() blocked for each iteration for each thread pool size.

1/16/2006 Master/ Slave

The Master/Slave tests looked at processing times starting with 1000 messages in Operation Instance. The 1000
rows on PSAPMSGPUBHDR generated 1000 publication contracts on PSAPMSGPUBCON and subsequently 2000
subscription contracts (2 notification handlers) on PSPMSGSUBCON. The final numbers reported are the sum of
the total time to process the 1000 messages on PUBHDR, the 1000 publication contracts and the 2000 subscription
contracts. The numbers for were calculated by looking at the LASTUPDDTTM on the first and last rows of each
table.
Two different hardware scenarios were used and multiple configurations of the hardware for each scenario. Two
iterations of the test were run for each configuration within each scenario.

The first scenario used an IBM laptop as the Master and a Compaq desktop as well as an HP multiprocessor
desktop as the Slaves.
The second scenario used the HP multiprocessor as the Master and the IBM laptop and Compaq desktop as
Slaves.

Scenario 1
Master: IBM laptop
Slaves: Compaq Desktop, HP dual processor desktop
67

Test 1a : Master Only

A single IBM laptop was used as the Master. No Slaves.

Avg

PUBHDR

PUBCON

SUBCON

Total
Time

993

1927

1887

4807

1010

1970

1880

4860

1002

1949

1884

4834

Test 1b: Master + 1 Slave

The same IBM laptop Master plus one Compaq desktop as a Slave.

Avg

PUBHDR

PUBCON

SUBCON

Total
Time

656

1157

1105

2918

645

1176

1140

2961

651

1167

1123

2940

Test 1c: Master + 2 Slaves

IBM laptop Master, one Compaq desktop Slave, and one HP dual processor Slave.

Avg

Total
Time

PUBHDR

PUBCON

SUBCON

540

811

707

2058

540

812

710

2062

540

812

709

2060

This data simply shows us adding a single Slave improves performance over using only a Master, and adding
multiple Slaves improves performance over a single Slave. In this case performance improved by more than 114%
when comparing 2 Slaves to Master-only.
Scenario 2
Master: HP dual processor desktop
68

Slaves: Compaq Desktop, IBM laptop

Test 1a: Master Only
The dual processor HP was used as the Master. No Slaves.

Avg

PUBHDR

PUBCON

SUBCON

Total
Time

371

680

676

1727

391

675

641

1707

381

678

659

1717

Test 1b: Master + 1 Slave

Dual processor HP Master plus one Compaq desktop as a Slave.

Avg

PUBHDR

PUBCON

SUBCON

Total
Time

281

523

478

1282

270

512

489

1271

276

518

484

1277

Test 1c: Master + 1 Slave

Dual processor HP Master plus one IBM laptop as a Slave.

Avg

PUBHDR

PUBCON

SUBCON

Total
Time

357

603

556

1516

320

625

527

1472

339

614

542

1494

PUBHDR

PUBCON

SUBCON

Total
Time

252

415

332

999

Test 1d: Master + 2 Slaves

Dual processor HP Master plus both Slaves.

69

Avg

248

428

297

973

250

422

315

986

By looking at the results from Scenario 1/Test 1c and the results from Scenario 2/Test 2d you can see Master/Slave
performance has a sensitive dependence on the quality and configuration of the underlying hardware. The same
three boxes were used in both tests (Master + 2 Slaves) but Scenario 2 outperformed Scenario 1 by more than
104% simply because we used the most powerful hardware as the Master. Given the magnitude of the
performance improvement, it warrants adding intelligence in 8.49 to allow underlying hardware to be used as
efficiently as possible.

11/19/2007 8.50 Performance features

All tests conducted with messages already queued up in a NEW status. This is the worst case scenario as there
are no tuxedo (tpa) calls queued on the dispatcher at the start of the test. This will test maximum throughput from
the start under a worst case scenario of no queued tuxedo (tpa) calls for the dispatcher.
The initial series of tests (1a-c) are used to get a bench mark (processing power) of each type of machine that will
be used for the following tests.
Setup: 1000 MESSAGES queued, 1 DISPATCHER/ 1 HANDLER, Time is in seconds.
Test 1a: Master Only
HP Desktop was used as the Master. No Slaves.

Avg

PUBHDR

PUBCON

SUBCON

Total
Time

220

320

270

810

226

322

274

822

223

321

272

816

70

Test 1b: Master Only

Compaq Desktop was used as the Master. No Slaves.

Avg

PUBHDR

PUBCON

SUBCON

Total
Time

214

345

253

812

220

350

258

828

217

348

256

820

Test 1c: Master Only

Dell D620 Laptop was used as the Master. No Slaves.

Avg

PUBHDR

PUBCON

SUBCON

Total
Time

117

320

240

677

119

316

238

673

118

318

239

675

Based on above result the Dell outperformed both the HP and Compaq desktops. The HP and Compaq desktops
will be treated as inter-changeable with respect to the following tests.
LOAD BALANCING
This first series of tests will demonstrate the new load balancing feature for 8.5. Prior to 8.5 the overall throughput
with respect to Master/ Slave was significantly impacted by the Processing Power of the Master. If the Slave was
the higher performing machine, throughput suffered significantly due to the Master performing equal requests in
addition to overall queue orchestration.
By configuring the Load balance parameter according to processing power of the machines, one can achieve
significantly better throughput in the case where the Master is not the higher performing machine with respect to the
Slave.
Test 2a:Master + 1 Slave
HP Master plus one Dell Laptop as a Slave.

Load
Balance
(M,S)

PUBHDR

PUBCON

SUBCON

Total Time

163

177

150

1,1

490

159

149

140

1,2

448

154

131

127

1,5

412

71

140

125

1,10

115

380

Test 2a shows that load balancing has a significant impact on performance vs the old 8.48 design (> 22%)
Test 2b: Master + 2 Slaves
HP Master plus one Compaq desktop as a Slave and one Dell laptop as a Slave.

Load
Balance
(M,S,S)

PUBHDR

PUBCON

SUBCON

Total Time

74

120

106

1,1,1

300

72

123

102

1,1,2

297

66

114

93

1,1,5

273

60

103

100

1,1,10

263

This test shows that Load balancing for Slaves does have a positive effect on overall throughput.

DEFERRED MASTER PROCESSING

The next series of tests will demonstrate the advantages of using Deferred Master Processing (new feature for 8.5).
This feature allows the user to defer all Handler processing to Slaves. This is a powerful feature in that now
Pub/Sub can be configured in the same domain as the on-line (PIA) domain taking advantage of tuxedo
communication between processes; yet allowing the CPU intensive work to be done by the slave(s) themselves.
Test 3a:Master (Deferred) + 2 Slaves
HP Master (deferred) plus one Compaq desktop Slave plus one Dell Laptop as a Slave.

Load
Balance
(S,S)

PUBHDR

PUBCON

SUBCON

108

151

139

1,1

398

90

140

128

1,2

358

80

163

106

1,5

349

69

145

83

1,10

297

Test 3b:Master (Deferred) + 1 Slave

HP Master (deferred) plus one Dell Laptop as a Slave.

72

Total Time

PUBHDR

PUBCON

SUBCON

89

183

128

Load
Balance
(S,S)

Total Time

1,1

400

These test show that deferred processing is significantly faster then processing of tradition Master/ Slave.
Comparing the results from 3a with 2a shows an increase of overall throughput as much as 20%. Moreover,
comparing 1A with 3b also shows throughput to increase 40% by processing the requests on Slave vs. a Master.

TEMPLATE TESTING:
The next series of tests will demonstrate the use of Slaves as Templates (new feature for 8.50) when more then
one Slave is configured with/ without Load balancing. For these tests, processing was isolated to subscription
processing only.
Setup: HP hosting database. 4000 MESSAGES: 1DISPATCHER/ 1 HANDLER
th

Master HP / all Slaves Dell Laptops 1-4 slaves, 5 Slave Compaq (Time in Minutes: Seconds)

Time
(mm:ss)

# of
Slaves

Load
Balance

9:25

5:07

3:32

2:40

2:10

1:52

Time
(mm:ss)
9:25

# of
Slaves
0

Load
Balance
1

73

4:38

3:12

2:15

1:53

1:41

These test show that additional Slaves used in concert with Load Balancing can dramatically increase overall
throughput.

Deferred processing (using Templates)

Setup: HP hosting database. 15,000 MESSAGES: 1DISP/ 3 HANDLERS
th

Master HP / all Slaves Dell Laptops 1-4 slaves, 5 Slave Compaq (Time in Minutes: Seconds)

Time
(mm:ss)

# of
Slaves

Load
Balance

25:29

14:30

10:21

8:28

7:30

7:20

th

A Dell laptop was used for the Master, Dell Laptops as Slaves 1-3, and 4 Slave Hp. The configuration was 1
Dispatcher, 3 Handlers, processing 15,000 messages.

Time
(mm:ss)

# of
Slaves
0

13:38

74

Load
Balance
5

13:50

6:20

4:32

5:30

Processing on the Slaves really becomes apparent here. Distributing request (via load balance) to a less then
performing Slave would not improve performance due the request taking longer on the slave box (less performing)
which in this case does not outweigh just sending more to the higher performing slave(s) with respect to overall
time.

75

Appendix A Special Notices

All material contained in this documentation is proprietary and confidential to Oracle Corporation, Inc., is protected
by copyright laws, and subject to the nondisclosure provisions of the applicable Oracle agreement. No part of this
documentation may be reproduced, stored in a retrieval system, or transmitted in any form or by any means,
including, but not limited to, electronic, graphic, mechanical, photocopying, recording, or otherwise without the prior
written permission of Oracle Corporation, Inc.
This documentation is subject to change without notice, and Oracle Corporation, Inc. does not warrant that the
material contained in this documentation is free of errors. Any errors found in this document should be reported to
Oracle Corporation, Inc. in writing.
The copyrighted software that accompanies this documentation is licensed for use only in strict accordance with the
applicable license agreement, which should be read carefully as it governs the terms of use of the software and this
documentation, including the disclosure thereof. See Customer Connection or PeopleBooks for more information
about what publications are considered to be product documentation.
Oracle, the PeopleSoft logo, PeopleTools, PS/nVision, PeopleCode, PeopleBooks, and Vantive are registered
trademarks, and PeopleTalk and "People power the internet." are trademarks of Oracle Corporation, Inc. All other
company and product names may be trademarks of their respective owners. The information contained herein is
subject to change without notice.
Information in this book was developed in conjunction with use of the product specified, and is limited in application
to those specific hardware and software products and levels.
Oracle Corporation may have patents or pending patent applications covering subject matter in this document. The
furnishing of this document does not give you any license to these patents.
The information contained in this document has not been submitted to any formal Oracle test and is distributed AS
IS. The use of this information or the implementation of any of these techniques is a customer responsibility and
depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While
each item may have been reviewed by Oracle for accuracy in a specific situation, there is no guarantee that the
same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own
environments do so at their own risk. Any pointers in this publication to external Web sites are provided for
convenience only and do not in any manner serve as an endorsement of these Web sites.

Appendix B Validation and Feedback

76

This section documents that real-world validation that this White Paper has received.

CUSTOMER VALIDATION
Oracle Corporation is working with PeopleSoft customers to get feedback and validation on this document. Lessons
learned from these customer experiences will be posted here.

FIELD VALIDATION
Oracle Corporation is working with PeopleSoft Global Services to get feedback and validation on this document.
Lessons learned from these field experiences will be posted here.

Appendix C Revision History

77

Revision History
1. July 19. 2010 Posted Document

78