BPEL PM 11g Performance Tuning - Appendies

1
2

Contents
APPENDIX A: WEBLOGIC SERVER OVERVIEW ......................................................................................6
1 WLS CONFIGURATION .......................................................................................................................6
2 DOMAIN ........................................................................................................................................6
3 SERVER ..........................................................................................................................................7
4 ADMINISTRATION SERVER ..................................................................................................................8
5 MANAGED SERVER ...........................................................................................................................9
6 ADMINISTRATION SERVER TO MANAGED SERVER INTERACTION ................................................................. 10
7 CLUSTER ....................................................................................................................................... 10
8 NODE MANAGER ............................................................................................................................ 12
9 MACHINE ..................................................................................................................................... 13
APPENDIX B: AUDITING IN BPEL PM ................................................................................................. 15
1 AUDIT LEVELS ................................................................................................................................ 15
2 ORDER OF PRECEDENCE FOR AUDIT LEVEL SETTINGS ................................................................................. 15
APPENDIX C: ANTI PATTERNS ........................................................................................................... 18
1 SYNCHRONOUS ASYNCHRONOUS ..................................................................................................... 18
2 OVER USE OF ASYNCHRONOUS PROCESSES ............................................................................................ 19
3 OVER USE OF DURABLE PROCESSES ...................................................................................................... 19
4 NO FAULT HANDLING ...................................................................................................................... 19
5 SYNCHRONOUS FAULT HANDLING ....................................................................................................... 19
6 TO MANY RETRIES ........................................................................................................................... 20
7 CHATTING BPEL PROCESS (CALL BACK) ................................................................................................ 20
8 OVER USE OF FLOWN ...................................................................................................................... 20
9 LOOPS AND MORE LOOPS.................................................................................................................. 20
10 SYNCHRONOUS AND ASYNCHRONOUS PROCESSES ON SAME MANAGED SERVER/CLUSTER ............................... 20
11 DURABLE AND TRANSIENT PROCESSES ON SAME MANAGED SERVER/CLUSTER ............................................... 21
12 STICKY LOAD BALANCER .................................................................................................................. 21
13 NOT KEEPING ASPECT RATIO ............................................................................................................ 21
APPENDIX D: SQL QUERIES ............................................................................................................... 22
1 EM CONSOLE SQL QUERIES .............................................................................................................. 22
1.1 RECOVERY CONSOLE QUERIES ................................................................................................................... 22
1.2 RECENT FAULT AND REJECTED MESSAGES QUERY .......................................................................................... 22
1.3 RECENT COMPOSITE INSTANCE QUERY ........................................................................................................ 23
1.4 INSTANCE TAB PAGE QUERY ...................................................................................................................... 23
3

1.5 INSTANCE TAB PAGE SEARCH QUERY BASED ON NAME VS TITLE QUERY ............................................................. 23
1.6 FAULT AND REJECTED MESSAGE TAB PAGE QUERIES ...................................................................................... 23
1.6.1 Parent query ................................................................................................................................... 23
1.6.2 Child query ...................................................................................................................................... 24
2 MISCELLANEOUS ............................................................................................................................ 24
2.1 STORED PROCEDURE TO CONVERT BLOB IN STRING ....................................................................................... 24
2.2 QUERY TO FIND PERCENTAGE OF FREE SPACE ............................................................................................... 25
2.3 QUERY TO FIND THE WAIT EVENTS FOR LGWR USING ITS SID ........................................................................ 25
2.4 QUERY TO MONITOR REDO BUFFER ALLOCATION RETRIES ............................................................................. 25
2.5 SQL STATEMENT TO RECLAIM SPACE AFTER PURGING.................................................................................... 25
2.6 QUERY TO FIND OUT TOTAL SESSIONS ON A DATABASE .................................................................................. 25
2.7 QUERY TO FIND OUT UTILIZATION OF PROCESSES AND SESSIONS IN A DATABASE ................................................ 25
2.8 FIND OUT THE PROCESS INSTANCE FROM A CONVERSATION ID WHEN THERE IS NO INSTANCE NUMBER SHOWING IN
THE LOG FILE (BPEL INSTANCE ID FOR A TIMES OUT ITEM) .................................................................................... 26
2.9 QUERY TO GET AUDIT DETAILS FROM AUDIT_DETAILS TABLE ........................................................................... 26
2.10 QUERY TO GET AUDIT DETAILS FROM AUDIT_TRAIL TABLE ............................................................................ 26
2.11 QUERY TO GET XML MESSAGE WITH THE GIVEN INSTANCE ID ...................................................................... 27
2.12 QUERY TO GET XML MESSAGE WITH A GIVEN INSTANCE NAME ..................................................................... 27
2.13 QUERY TO GET PAYLOAD SIZE OF MESSAGE ................................................................................................ 27
2.14 QUERY TO GET EXECUTION TIME OF BPEL INSTANCES ................................................................................. 27
2.15 QUERY TO GET THE EXECUTION TIME OF BPEL INSTANCES AND TO FIND THE PARENT THAT HAS INITIALIZED THE
COMPOSITE .................................................................................................................................................... 27
2.16 QUERY TO IDENTIFY ALL THE FAULTS FOR THE MESSAGES THAT WERE SITTING IN BPEL ENGINE LEVEL RECOVERY AS
UNDELIVERED INVOKES ..................................................................................................................................... 28
APPENDIX E: BIG OR LARGE OR HUGE PAGES .................................................................................... 29
1 LINUX .......................................................................................................................................... 29
2 WINDOWS .................................................................................................................................... 30
3 SOLARIS ....................................................................................................................................... 30
4 REFERENCE: .................................................................................................................................. 30
APPENDIX F: ORA-01438: VALUE LARGER THAN SPECIFIED PRECISION ALLOWED ............................... 31
5 WHAT IS THE ERROR IN LOGS? ........................................................................................................... 31
6 EFFECTS........................................................................................................................................ 31
7 CAUSE ......................................................................................................................................... 31
8 SOLUTION ..................................................................................................................................... 32
APPENDIX G: LOBS IN THE SOAINFRA SCHEMA ................................................................................. 33
APPENDIX H: AWR, ADDM, & ASH REPORTS ..................................................................................... 36
4

1 AWR REPORT ............................................................................................................................... 36
2 ADDM REPORT ............................................................................................................................. 36
3 ASH REPORT ................................................................................................................................. 36
4 AWR REPORT ANALYSIS .................................................................................................................. 37
4.1 SQL STATEMENTS ORDERED BY ELAPSED TIME ............................................................................................ 37
4.2 SQL STATEMENTS ORDERED BY CPU TIME .................................................................................................. 37
4.3 SQL STATEMENTS ORDERED BY GETS ......................................................................................................... 38
4.4 SQL STATEMENTS ORDERED BY READS ....................................................................................................... 38
4.5 SQL STATEMENTS ORDERED BY EXECUTIONS ............................................................................................... 38
4.6 SQL STATEMENTS ORDERED BY PARSE CALLS .............................................................................................. 38
5 REFERENCE ................................................................................................................................... 39
APPENDIX I: MONITORING SCRIPTS .................................................................................................. 40
1 DATABASE MONITORING .................................................................................................................. 40
2 JMS MONITORING ......................................................................................................................... 41
3 AQ MONITORING ........................................................................................................................... 44
APPENDIX J: HOW TO MONITOR SOA SERVER MEMORY USAGE ........................................................ 48
1 SETUP: JCONSOLE OR VISUALVM (INSTALLED LOCALLY) ............................................................................ 48
2 SETUP: JVISUALVM (INSTALLED AT REMOTE MACHINE) ............................................................................ 52
3 SETUP: JROCKIT MISSION CONTROL (INSTALLED AT REMOTE MACHINE) ....................................................... 54
4 REFERENCE ................................................................................................................................... 55
APPENDIX K: HEAP DUMP FILES ANALYSIS: JROCKIT AND HOTSPOT JVMS ......................................... 56
1 EXAMPLE ANALYSIS OF A HEAP DUMP FILE USING ECLIPSE MEMORY ANALYZER ............................................ 56
2 REFERENCE ................................................................................................................................... 62
APPENDIX L: CAPACITY PLANNING .................................................................................................... 63
1 CAPACITY PLANNING FOR BPEL PM ................................................................................................... 63
1.1 DETERMINING PERFORMANCE GOALS AND OBJECTIVES CURRENT & FUTURE ................................................. 64
1.2 MEASURING PERFORMANCE METRICS ....................................................................................................... 65
1.3 IDENTIFYING BOTTLENECKS ...................................................................................................................... 65
1.4 IMPLEMENTING A CAPACITY MANAGEMENT PLAN ....................................................................................... 65
2 REFERENCE ................................................................................................................................... 66

5

Exhibits
Exhibit 82: WebLogic Homes ........................................................................................................ 14
Exhibit 83: Synchronous Asynchronous - 1 ................................................................................ 18
Exhibit 84: Synchronous Asynchronous - 2 ................................................................................ 19
Exhibit 85: Database Monitoring .................................................................................................. 40
Exhibit 86: JMS Monitoring ........................................................................................................... 42
Exhibit 87: AQ Monitoring ............................................................................................................ 45
Exhibit 88: Setup jConsole or visualVM (installed locally) - 1 ....................................................... 49
Exhibit 92: Heap Dump Files analysis - 1 ...................................................................................... 57

6

Appendix A: WebLogic Server Overview
1 WLS Configuration
WLS configuration is segmented by domain
Each domain represents a logically related group of WLS instances that you
manage from a single set of configuration files.
Each domain has one Administration Server, and can have multiple managed
servers and clusters
Administration Server - Central configuration controller for the entire domain
Managed Server - A running instance that hosts applications and resources needed by
those applications - The real work horses in a WebLogic domain
Cluster - group of Managed Servers running simultaneously and working together to
provide increased scalability and reliability
Node Manager is a per-machine process used to start and stop WLS instances.
Independent of all domains
2 Domain
What is it?
A logically related group of WLS instances that you manage from a single set of
configuration artifacts.
Whats in a domain?
Servers
Clusters of servers
Rules:
All WLS instances within the same domain must be at the same major and minor
version.
Servers within a domain can be at different Maintenance Pack levels as long as the
Administration Server is at the same Maintenance Pack Level or higher than its Managed
Servers.
7

Domain Restrictions
Each domain requires its own Administration Server for performing management activities.
When you use the Administration Console to perform management and monitoring tasks, you
can switch back and forth between domains, but in doing so, you are connecting to different
Administration Servers.
All Managed Servers in a cluster must reside in the same domain; you cannot split a cluster over
multiple domains.
All Managed Servers in a domain must run the same version of the WLS software. The
Administration Server may run either the same version as the Managed Servers in the domain,
or a later service pack.
If multiple domains created, each domain must reference its own database schema. Domains
cannot share configured resources or subsystem. For example, if one creates a JDBC data
source in one domain, one cannot use it with a Managed Server or cluster in another domain.
Instead, one must create a similar data source in the second domain. Furthermore, two or more
system resources cannot have the same name.
3 Server
What is it?
A configured instance to host applications and resources
WebApps, Enterprise Apps, Web Services,
JMS, JDBC, Diagnostics,
What types of servers are there?
Administration Server
Managed Server
All instances of WLS use a root directory to store their working copy of the domain's
configuration files, to store runtime data, and to provide the context for any relative
pathnames in the server's configuration. An Administration Server always uses the domain
directory as its root directory. A Managed Server can use the domain directory but can also use
any other directory that you define.
Applications can use the following resources and services:
Security providers, which are modular components that handle specific aspects of
security, such as authentication and authorization.
8

Resource adapters, which are system libraries specific to Enterprise Information Systems
(EIS) and provide connectivity to an EIS.
Diagnostics and monitoring services.
JDBC data sources, which enable applications to connect to databases.
Mail sessions.
XML entity caches and registry of XML parsers and transformer factories
Messaging services such as JMS servers and store-and-forward services
Persistent store, which is a physical repository for storing data, such as persistent JMS
messages. It can be either a JDBC-accessible database or a disk-based file.
Startup classes, which are Java programs that you create to provide custom, system-
wide services for your applications.
Work Managers, which determine how an application prioritizes the execution of its
work based on rules you define and by monitoring actual runtime performance. You can
create Work Mangers for entire WebLogic Server domains or for specific application
components.
Work Contexts, which enable applications to pass properties to a remote context
without including the properties in a remote call.
4 Administration Server
What is it?
Central configuration controller for the entire domain
What else does it do?
Hosts the Administration Console
Enables you to start and stop servers from a central location
Enables you to migrate servers and services within the domain
Enables you to deploy applications within the domain
Guidelines:
There must be exactly one* Administration Server in domain
An Administration Server controls only one domain.
For production use, we recommend not hosting application logic or resources on
the Administration Server
9

The Administration Server does not need to run at all times, but is required for making
configuration and deployment changes to a running domain.
The Administration Server operates as the central control entity for the configuration of the
entire domain. It maintains the domain's configuration documents and distributes changes in
the configuration documents to Managed Servers. One can also use the Administration Server
as a central location from which to monitor all resources in a domain.
To interact with the Administration Server, one can use the Administration Console, WLST, or
custom JMX client.
Each WLS domain must have one server instance that acts as the Administration Server.
In each domain, one WLS instance acts as the Administration Serverthe server instance which
configures, manages, and monitors all other server instances and resources in the domain. Each
Administration Server manages one domain only. If a domain contains multiple clusters, each
cluster in the domain has the same Administration Server.
5 Managed Server
What is it?
A running instance that hosts applications and resources needed by those
applications - The real work horses in a WebLogic domain
Managed Servers host business applications, application components, Web
services, and their associated resources. To optimize performance, Managed
Servers maintain a read-only copy of the domain's configuration document.
Each Managed Server is independent of all other Managed Servers in the domain
(unless they are in a cluster, defined later)
You can have as many Managed Servers in a domain as you need
Individual Managed Servers are typically added for capacity and application
isolation
Managed Servers can use the following resources:
Machine definitions that identify a particular, physical piece of hardware. A machine
definition is used to associate a computer with the Managed Servers it hosts. This
information is used by Node Manager in restarting a failed Managed Server, and by a
clustered Managed Server in selecting the best location for storing replicated session
data.
10

Network channels that define default ports, protocols, and protocol settings that a
Managed Server uses to communicate with clients. After creating a network channel,
you can assign it to any number of Managed Servers and clusters in the domain.
Virtual hosting, which defines a set of host names to which WLS instances (servers) or
clusters respond. When you use virtual hosting, you use DNS to specify one or more
host names that map to the IP address of a server or cluster. One also specifies which
Web applications are served by each virtual host.
6 Administration Server to Managed Server Interaction
The Administration Server stores the master copy of the domain configuration, including
the configuration for all managed servers in the domain
Each Managed Server stores a local copy of its configuration.
When a Managed Server starts, it connects to the Administration Server to synchronize
the configuration
When configuration is changed, the Administration Server sends changed configuration
to Managed Servers
7 Cluster
What is it?
A cluster is a group of Managed Servers running simultaneously and working
together to provide increased scalability and reliability
Scalability: through parallelism
Reliability/Availability: through replication and redundancy
A cluster appears as a single instance to most clients.
Clusters enable some advanced features, such as Whole Server Migration,
Service Migration, and clustered JMS destinations.
The server instances that constitute a cluster can run on the same machine, or be located on
different machines. You can increase a cluster's capacity by adding additional server instances
to the cluster on an existing machine, or you can add machines to the cluster to host the
incremental server instances. Each server instance in a cluster must run the same version of
WLS Software.
How Does a Cluster Relate to a Domain?
11

A cluster is part of a particular WLS domain.
A domain is an interrelated set of WLS resources that are managed as a unit. A domain includes
one or more WLS instances, which can be clustered, non-clustered, or a combination of
clustered and non-clustered instances. A domain can include multiple clusters. A domain also
contains the application components deployed in the domain, and the resources and services
required by those application components and the server instances in the domain. Examples of
the resources and services used by applications and server instances include machine
definitions, optional network channels, connectors, and startup classes.
One can use a variety of criteria for organizing WLS instances into domains. For instance, one
might choose to allocate resources to multiple domains based on logical divisions of the hosted
application, geographical considerations, or the number or complexity of the resources under
management.
In each domain, one WLS instance acts as the Administration Serverthe server instance which
configures, manages, and monitors all other server instances and resources in the domain. Each
Administration Server manages one domain only. If a domain contains multiple clusters, each
cluster in the domain has the same Administration Server.
All server instances in a cluster must reside in the same domain; one cannot "split" a cluster
over multiple domains. Similarly, one cannot share a configured resource or subsystem
between domains. For example, if one creates a JDBC connection pool in one domain, one
cannot use it with a server instance or cluster in another domain. (Instead, one must create a
similar connection pool in the second domain.)
Clustered WLS instances behave similarly to non-clustered instances, except that they provide
failover and load balancing. The process and tools used to configure clustered WLS instances
are the same as those used to configure non-clustered instances. However, to achieve the load
balancing and failover benefits that clustering enables, one must adhere to certain guidelines
for cluster configuration.
Cluster Guidelines
All servers in a cluster must also be in the same domain.
All servers within a cluster must be at the same Maintenance Pack level.
Clustered servers can be on the same or different machines.
You can have multiple clusters in a domain.
Communication in a Cluster
Peer to Peer using Sockets - used for:
12

Accessing non-clustered objects deployed to another clustered server instance
on a different machine.
Replicating HTTP session states and stateful session EJB states between a
primary and secondary server instance.
Accessing clustered objects that reside on a remote server instance.
Peer to Peer using Unicast or Multicast - used for:
Cluster-wide JNDI updates
Heartbeats
Cluster-wide JNDI tree
Lists local resources and resources available throughout the cluster
List is maintained on all servers in the cluster
8 Node Manager
What is it?
Utility/process running on a physical server that enables you to start, stop,
suspend, and restart WebLogic Server instances remotely
Must run on each physical server that hosts WebLogic Server instances that you
want to control with Node Manager
Not associated with a domain. Can start any server instance that resides on the
same physical server.
Optional, but required to start/stop servers using the Administration Console
Required for Whole Server Migration and for some configurations of Automatic
Service Migration
Server instances in a WLS production environment are often distributed across multiple
domains, machines, and geographic locations. Node Manager is a WLS utility that enables one
to start, shut down, and restart Administration Server and Managed Server instances from a
remote location. Although Node Manager is optional, it is recommended for high availability
environment.
A Node Manager process is not associated with a specific WebLogic domain but with a machine.
One can use the same Node Manager process to control server instances in any WLS domain, as
13

long as the server instances reside on the same machine as the Node Manager process. Node
Manager must run on each computer that hosts WLS instances -- whether Administration
Server or Managed Server -- that one wants to control with Node Manager.
9 Machine
What is it?
A definition that identifies a particular, physical piece of hardware.
A machine definition is used to associate a computer with the Managed Servers it
hosts.
Used by Node Manager in restarting a failed Managed Server
Used by a clustered Managed Server in selecting the best location for storing
replicated session data

14

Exhibit 1: WebLogic Homes

15

Appendix B: Auditing in BPEL PM
1 Audit levels
Although audit levels can be configured in various areas, they mostly (but not in every case) fall
under one of these four levels:
Off: Absolutely no composite instance or payload information is collected. Although this
is the best in terms of performance, it severely limits the visibility as no information is
logged, rendering it an option that is not recommended in most cases. Instances are
created, but nothing is logged to the database or displayed on the console, which may
lead to difficulties in fault diagnosis and incident analysis.
Development: Both composite instance and payload information is collected. Though
this option provides the most detail, it is the worst performing of all the audit levels. It is
to be set in development environments for debugging purposes, but not in production
environments, except for transactions that specifically require that degree of auditing.
Audit levels set to development capture payloads throughout most activities, such as
Assign and Transform
Production: Although composite instance information is collected, most payload
details are not. This is the typical setting for production environments and provides the
best balance between performance and visibility. If payload details need to be captured,
it is best to consider setting the audit level to development only for specific
composites, components, or services, as we shall describe later.
Inherit: Audit levels are inherited from the parent level.
2 Order of precedence for audit level settings
Audit levels can be manipulated across each of these. It is possible to set the audit level at the
component level, the composite level, or even the service engine level. But if the audit level is
set at the composite level as development and at the SOA Infrastructure level as development,
which one takes precedence?
At a high level, the order of precedence is as follows:
Component > Composite > Service Engine > SOA Infrastructure
For example, if the audit level is set to development at the composite level and production at
the SOA Infrastructure level, setting at the composite level overrides that of the SOA
Infrastructure.
If the composite audit level is set to inherit, it will inherit the settings from the applicable
service engine. If the service engine is also set to inherit, it will inherit the settings from the
SOA Infrastructure.
16

As a best practice, set all audit level settings to inherit and controlling it at the SOA
Infrastructure level. Then, as the need for different levels of auditing are required, start
manipulating the service engine, composite, and component audit levels as needed.
But as one start making changes in audit settings at different levels, simple level of order of
precedence become complex and murky.
Examples of Order of Precedence
Component Composite
Service
Engine
SOA
Infrastructure Who Takes Precedence?
No property Off Production Development Composite.
The audit level is set to Off. The service
engine and SOA Infrastructure audit
levels do not take effect.
No property Inherit Developme
nt
Production Service engine.
The audit level is set to Development.
The payload is shown in the assign
activity. The SOA Infrastructure audit
level does not take effect.
No property Inherit Inherit Production SOA Infrastructure.
The audit level is set to Production.
No property Inherit Production
/Developm
ent/Off/In
herit
Off The overall audit is not shown.
The composite inherits the audit level
from the SOA Infrastructure. The
payload is shown in the assign activity
based on the service engine audit level
setting.
Developmen
t
Off Production Development Composite.
Since the composite audit level is set to
Off, the overall audit is not shown. The
service engine audit level is shown, but
the Development setting for the
component takes precedence.
The payload is shown in the assign
activity based on the component audit
level setting of Development.
17

Component Composite
Service
Engine
SOA
Infrastructure Who Takes Precedence?
Inherit Off Production Development Composite.
Since the composite audit level is set to
Off, the overall audit is not shown. The
service engine audit level is not shown
because Off is inherited from the
composite.

Notes:
When the composite audit level is set to Off, there is no audit trail generated for this
composite and all service engines used within the composite.
When the composite audit level is set to Inherit, it always inherits the settings of the
SOA Infrastructure.
When the composite audit level is set to Off, the component inherits the service engine
settings.

18

Appendix C: Anti Patterns
While developing BPEL processes/composites, one must avoid few anti patterns from
performance perspective. List includes from design of BPEL Process/composite perspective as
well as other areas as well.
Synchronous Asynchronous
Over use of asynchronous processes
Over use of durable processes
No fault Handling
Synchronous Fault Handling
To many retries
Chatting BPEL Process (call back)
Over use of FlowN
Loops and more loops
Synchronous and Asynchronous processes on same managed server/cluster
Durable and transient processes on same managed server/cluster
Sticky load balancer
Not keeping aspect ratio: Increasing capacity for a component (say WLS) but not
increasing others (like database).
1 Synchronous Asynchronous
Never call an asynchronous process from a synchronous process directly (web service). It is a
very bad design decision. In this kind of design, calling process (synchronous one) will slow
down significantly.

Exhibit 2: Synchronous Asynchronous - 1
Situation become worse if calling process is synchronous and transient while called process is
asynchronous and durable.
19

Exhibit 3: Synchronous Asynchronous - 2
If synchronous to asynchronous call cannot be avoided, use some decoupling trick like
introducing JMS queue and Durable Asynchronous process in between.
2 Over use of asynchronous processes
By design, asynchronous processes interact with data base multiple times. This very nature of
asynchronous process makes them resource intensive.
In any deployment, excessive usages of asynchronous processes strain the available resources
and may affect performance of overall system.
3 Over use of durable processes
A durable process interacts with SOAINFRA schema multiple time which puts strain on WLS as
well as on database.
In any deployment, excessive usages of durable processes strain the available resources and
may affect performance of overall system.
4 No fault Handling
No fault handling in BPEL process affects the system at multiple levels. BPEL Engine consumes
resources to process fault, log files writing eats up I/O resources, increased size aand number of
log files may affect availability of disk space and increased interaction with SOAINFRA tables to
log increased number of audit information.
Do appropriate and sufficient fault handling.
5 Synchronous Fault Handling
One of the most common bad designs is synchronous fault handling. In case of fault,
synchronous fault handling will slow down the BPEL process.
Develop a separate set of BPEL processes to log faults and exceptions, also send notifications
and storage of fault data.
20

6 To many retries
Avoid large number of retries in case of exception/fault in BPEL processes. Most of the time
retries does not help but loads the system excessively which even may lead to collapse of WLS.
While setting number of retries in BPEL process, always consider the consequences of downing
of underlying WLS.
7 Chatting BPEL Process (call back)
As famous mathematician Pythagoras has said - Silence is better than unmeaning words. Any
chatting BPEL Process is burden on resources. Avoid call back if not required.
8 Over use of FlowN
Branches in flowN activity are executed serially in a single thread (that is, the Nth branch is
executed only after N-1 execution has completed). Execution is not completely parallel. This is
because the branches do not execute in concurrent threads in this mode. Instead, one thread
starts executing a flow branch until it reaches a blocking activity (for example, a synchronous
invoke). At this point, a new thread is created that starts executing the other branch, and the
process continues. This creates the impression that the flow branches are executing in parallel.
In this mode, however, if the flow branches do not define a blocking activity, the branches still
execute serially.
The very nature of FlowN slows down the BPEL PM and excessive use of FlowN slows down
whole system.
9 Loops and more loops
forEach activity executes similar to FlowN. While loop executes enclosed activities repeatedly.
Most of the time number of loops to be executed are unknown at design time which introduces
big unknown in estimating resources required for any deployment.
Avoid using loops and especially embedded loops inside loops.
10 Synchronous and Asynchronous processes on same managed
server/cluster
Avoid deployment of synchronous and asynchronous BPEL processes on same managed server.
Tuning of BPEL engine which is hosting synchronous processes is slightly different from BPEL
engine which is hosting asynchronous.
21

11 Durable and transient processes on same managed server/cluster
Avoid deployment of transient and durable BPEL processes on same managed server. Tuning of
BPEL engine which is hosting transient processes is slightly different from BPEL engine which is
hosting durable.
12 Sticky load balancer
In clustered environment, load balancer plays big role. If load balancer is sticky one (especially
IP sticky), it may fail whole premises of clustered environment. A sticky load balancer may load
few servers in cluster while leaving other with inadequate load. This imbalance will affect
overall performance of system.
13 Not keeping aspect ratio
Over the time period with change in load profile and due to non-technical reasons, one part of
infrastructures capacity will increase while other may cripple. For example increasing capacity
for a component (say WLS) but not increasing others (like database) will affect performance of
the system.

22

Appendix D: SQL Queries
1 EM Console SQL Queries
1.1 Recovery console queries
SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum FROM (SELECT
MESSAGE_GUID AS a1, DLV_TYPE AS a2, CIKEY AS a3, CLUSTER_NODE_ID AS a4,
CLUSTER_NODE_KEY AS a5, COMPONENT_NAME AS a6, COMPONENT_TYPE AS a7,
COMPOSITE_LABEL AS a8, COMPOSITE_NAME AS a9, COMPOSITE_REVISION AS a10,
CONV_ID AS a11, DOMAIN_NAME AS a12, ECID AS a13, EVENT_NAME AS a14, EXT_INT1
AS a15, EXT_STRING1 AS a16, EXT_STRING2 AS a17, HEADER_PROPERTIES_BIN_FORMAT
AS a18, HEADERS_REF_ID AS a19, OPERATION_NAME AS a20, PARTNER_LINK AS a21,
PRIORITY AS a22, PROPERTIES AS a23, RECEIVE_DATE AS a24, RECOVER_COUNT AS
a25, STATE AS a26, TENANT_ID AS a27, CONV_TYPE AS a28, RES_SUBSCRIBER AS a29
FROM DLV_MESSAGE WHERE ((((COMPONENT_TYPE = ?) AND (STATE IN (?, ?))) AND
(RECEIVE_DATE <= ?)) AND (DLV_TYPE = ?)) ORDER BY RECEIVE_DATE DESC) a WHERE
ROWNUM <= ?) WHERE rnum > ?

SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum FROM (SELECT
MESSAGE_GUID AS a1, DLV_TYPE AS a2, CIKEY AS a3, CLUSTER_NODE_ID AS a4,
CLUSTER_NODE_KEY AS a5, COMPONENT_NAME AS a6, COMPONENT_TYPE AS a7,
COMPOSITE_LABEL AS a8, COMPOSITE_NAME AS a9, COMPOSITE_REVISION AS a10,
CONV_ID AS a11, DOMAIN_NAME AS a12, ECID AS a13, EVENT_NAME AS a14, EXT_INT1
AS a15, EXT_STRING1 AS a16, EXT_STRING2 AS a17, HEADER_PROPERTIES_BIN_FORMAT
AS a18, HEADERS_REF_ID AS a19, OPERATION_NAME AS a20, PARTNER_LINK AS a21,
PRIORITY AS a22, PROPERTIES AS a23, RECEIVE_DATE AS a24, RECOVER_COUNT AS
a25, STATE AS a26, TENANT_ID AS a27, MASTER_CONV_ID AS a28 FROM DLV_MESSAGE
WHERE ((((COMPONENT_TYPE = ?) AND (STATE IN (?, ?))) AND (RECEIVE_DATE <= ?))
AND (DLV_TYPE = ?)) ORDER BY RECEIVE_DATE DESC) a WHERE ROWNUM <= ?) WHERE
rnum > ?
1.2 Recent fault and rejected messages query
SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum from
(SELECT f.CIKEY, f.NODE_ID, f.SCOPE_ID, f.COUNT_ID, f.FAULT_NAME,
f.FAULT_TYPE, f.POLICY_NAME, f.POLICY_VERSION, f.POLICY_CATEGORY,
f.POLICY_ECID, f.CREATION_DATE, f.MODIFY_DATE, f.MESSAGE, wi.LABEL,
wi.CREATOR, wi.MODIFIER, wi.STATE, ci.ECID, ci.CMPST_ID, ci.DOMAIN_NAME,
ci.COMPOSITE_NAME, ci.COMPONENT_NAME, ci.COMPOSITE_REVISION,
ci.COMPOSITE_LABEL FROM CUBE_INSTANCE ci, WI_FAULT f LEFT JOIN WORK_ITEM wi
ON f.CIKEY = wi.CIKEY AND f.NODE_ID = wi.NODE_ID AND f.SCOPE_ID = wi.SCOPE_ID
AND f.COUNT_ID = wi.COUNT_ID WHERE ci.CIKEY = f.CIKEY AND ci.COMPONENTTYPE =
? AND (f.FAULT_TYPE is null OR f.FAULT_TYPE = ?) AND ci.STATE != 9 AND
(wi.STATE is null OR wi.STATE IN (9, 4, 13, 3)) ORDER BY f.CREATION_DATE
DESC) a where ROWNUM < ? ) where rnum > ?

23

1.3 Recent composite Instance query
SELECT /*+ FIRST_ROWS(40) */ ID, CONVERSATION_ID, HAS_ASSOC, PARENT_ID,
UPDATED_TIME, CREATED_BY, ECID, TITLE, TEST_RUN_NAME, INDEX5, TEST_RUN_ID,
INDEX3, TEST_SUITE, INDEX1, TEST_CASE, BATCH_INDEX, SOURCE_NAME,
COMPOSITE_DN, SOURCE_TYPE, CREATED_TIME, SOURCE_ACTION_TYPE, INDEX6,
SOURCE_ACTION_NAME, INDEX2, STATE, BATCH_ID, LIVE_INSTANCES, TAGS,
STATE_COUNT, BUSINESS_STATUS, VERSION, INDEX4, PARTITION_DATE, UPDATED_BY,
TENANT_ID FROM COMPOSITE_INSTANCE WHERE (CREATED_TIME >= :1 ) ORDER BY
CREATED_TIME DESC
1.4 Instance tab page query
TENANT_ID FROM COMPOSITE_INSTANCE WHERE (CREATED_TIME >= :1 ) ORDER BY
CREATED_TIME DESC
1.5 Instance tab page search query based on name vs title query
TENANT_ID FROM COMPOSITE_INSTANCE WHERE (TITLE LIKE :1 ) ORDER BY
CREATED_TIME DESC
1.6 Fault and Rejected message tab page queries
There is one parent query fetches initial 40 records and then for each records its run another
child sql query.
1.6.1 Parent query
SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum from
(SELECT f.CIKEY, f.NODE_ID, f.SCOPE_ID, f.COUNT_ID, f.FAULT_NAME,
f.FAULT_TYPE, f.POLICY_NAME, f.POLICY_VERSION, f.POLICY_CATEGORY,
f.POLICY_ECID, f.CREATION_DATE, f.MODIFY_DATE, f.MESSAGE, wi.LABEL,
wi.CREATOR, wi.MODIFIER, wi.STATE, ci.ECID, ci.CMPST_ID, ci.DOMAIN_NAME,
ci.COMPOSITE_NAME, ci.COMPONENT_NAME, ci.COMPOSITE_REVISION,
ci.COMPOSITE_LABEL
FROM CUBE_INSTANCE ci, WI_FAULT f LEFT
JOIN WORK_ITEM wi ON f.CIKEY = wi.CIKEY AND f.NODE_ID = wi.NODE_ID AND
f.SCOPE_ID = wi.SCOPE_ID AND f.COUNT_ID = wi.COUNT_ID WHERE ci.CIKEY
= f.CIKEY AND ci.COMPONENTTYPE = ? AND ci.STATE != 9 AND (wi.STATE is null OR
wi.STATE IN (9, 4, 13, 3)) ORDER BY f.CREATION_DATE DESC) a where ROWNUM <
? ) where rnum > ?
24

1.6.2 Child query
SELECT F.CIKEY, F.NODE_ID, F.SCOPE_ID, F.COUNT_ID, F.FAULT_NAME,
F.FAULT_TYPE,F.CREATION_DATE,F.MODIFY_DATE,
F.MESSAGE, WI.LABEL,
WI.CREATOR,WI.MODIFIER,WI.STATE,CI.ECID,CI.CMPST_ID
FROM WI_FAULT F LEFT JOIN WORK_ITEM WI ON F.CIKEY = WI.CIKEY
AND F.NODE_ID = WI.NODE_ID AND F.SCOPE_ID = WI.SCOPE_ID AND
F.COUNT_ID = WI.COUNT_ID
LEFT JOIN CUBE_INSTANCE CI ON F.CIKEY = CI.CIKEY WHERE
F.CIKEY = ? AND F.NODE_ID = ?
AND F.SCOPE_ID = ? AND F.COUNT_ID = ?
bind => [2050981, BpInv1, BpSeq3.17, 2]
SELECT F.CIKEY, F.NODE_ID, F.SCOPE_ID, F.COUNT_ID, F.FAULT_NAME,
F.FAULT_TYPE,F.CREATION_DATE,F.MODIFY_DATE,
F.MESSAGE, WI.LABEL,
WI.CREATOR,WI.MODIFIER,WI.STATE,CI.ECID,CI.CMPST_ID
FROM WI_FAULT F LEFT JOIN WORK_ITEM WI ON F.CIKEY = WI.CIKEY
AND F.NODE_ID = WI.NODE_ID AND F.SCOPE_ID = WI.SCOPE_ID AND
F.COUNT_ID = WI.COUNT_ID
LEFT JOIN CUBE_INSTANCE CI ON F.CIKEY = CI.CIKEY WHERE
F.CIKEY = ? AND F.NODE_ID = ?
AND F.SCOPE_ID = ? AND F.COUNT_ID = ?
2 Miscellaneous
2.1 Stored procedure to convert Blob in string
create or replace function BLOBCONVERTER(B BLOB)
return clob is
c clob;
n number;
begin
if (b is null) then
return null;
end if;
if (length(b)=0) then
return empty_clob();
end if;
dbms_lob.createtemporary(c,true);
25

n:=1;
while (n+32767<=length(b)) loop

dbms_lob.writeappend(c,32767,utl_raw.cast_to_varchar2(dbms_lob.substr(b,32767
,n)));
n:=n+32767;
end loop;
dbms_lob.writeappend(c,length(b)-
n+1,utl_raw.cast_to_varchar2(dbms_lob.substr(b,length(b)-n+1,n)));
return c;
end;
2.2 Query to find percentage of free space
select df.tablespace_name TABLESPACE, df.total_space TOTAL_SPACE,
fs.free_space FREE_SPACE, df.total_space_mb TOTAL_SPACE_MB,
(df.total_space_mb - fs.free_space_mb) USED_SPACE_MB,
fs.free_space_mb FREE_SPACE_MB,
ROUND(100 * (fs.free_space / df.total_space),2) PCT_FREE
from (select tablespace_name, SUM(bytes) TOTAL_SPACE,
ROUND(SUM(bytes) / 1048576) TOTAL_SPACE_MB
from dba_data_files
group by tablespace_name) df,
(select tablespace_name, SUM(bytes) FREE_SPACE,
ROUND(SUM(bytes) / 1048576) FREE_SPACE_MB
from dba_free_space
group by tablespace_name) fs
where df.tablespace_name = fs.tablespace_name(+)
and df.tablespace_name like '%SOA%'
order by fs.tablespace_name;
2.3 Query to find the wait events for LGWR using its SID
select sid, event, time_waited, time_waited_micro
from v$session_event
where sid in (select sid from v$session WHERE type!='USER' and program like
'%LGWR%' )
order by time_waited;
2.4 Query to monitor redo buffer allocation retries
select name, value from V$SYSSTAT where name = 'redo buffer allocation
retries'
2.5 SQL statement to reclaim space after purging
alter table CUBE_SCOPE enable row movement;
alter table CUBE_SCOPE shrink space;
alter table CUBE_SCOPE modify lob (SCOPE_BIN) (shrink space);
alter table CUBE_SCOPE disable row movement;
2.6 Query to find out total sessions on a database
select username, status, machine, count(*) as TOTAL_SESSIONS from v$session
group by username, machine, status
2.7 Query to find out utilization of processes and sessions in a database
select resource_name,current_utilization,max_utilization,limit_value from
v$resource_limit where resource_name in ('processes','sessions');
26

2.8 Find out the process instance from a conversation ID when there is no
instance number showing in the log file (BPEL Instance ID for a Times Out
Item)

Error

com.oracle.bpel.client.delivery.ReceiveTimeOutException: Waiting for response
has timed out. The conversation id is LocalGUID:54s013a1a13963a8:-
3e1457ad:279ee837833:-39ca. Please check the process instance for detail.

Solution

select ds.cikey from dlv_message dm, dlv_subscription ds where dm.properties
like '%<LocalGUID>%' and dm.conv_id = ds.conv_id;
The cikey is the process instance ID.
2.9 Query to get audit details from audit_details table
select UTL_COMPRESS.LZ_UNCOMPRESS(bin) from audit_details where cikey =
<<CIKEY>>
2.10 Query to get audit details from audit_trail table
Step 1: Ensure there is a function to bring together all blocks pertaining to an instance key and
then uncompress it (log column is RAW data type) and return as BLOB.
create or replace function get_Audit_Trail_Log(cikey in integer) return blob
is
cursor cu_log(cikey_1 integer) is
select * from audit_trail atr where cikey = cikey_1 order by count_id;
bl blob;
begin
dbms_lob.createtemporary (bl, true);
for r1_log in cu_log(cikey)
loop
dbms_lob.append (bl,r1_log.log);
end loop;
return(bl);
end;
Step 2: use UTL_COMPRESS.LZ_UNCOMPRESS to uncompress the returned BLOB.
27

select UTL_COMPRESS.LZ_UNCOMPRESS(get_Audit_Trail_Log (cikey)) from
cube_instance where cikey = <<CIKEY>>
2.11 Query to get XML message with the given instance ID
select xd.Document_ID, UTL_COMPRESS.LZ_UNCOMPRESS(xd.DOCUMENT) from
XML_DOCUMENT_REF xdr left join XML_DOCUMENT xd on xdr.DOCUMENT_ID =
xd.DOCUMENT_ID where xdr.COMPOSITE_INSTANCE_ID = '<<InstanceID>>'
2.12 Query to get XML message with a given instance name
select xd.Document_ID, UTL_COMPRESS.LZ_UNCOMPRESS(xd.DOCUMENT) from
XML_DOCUMENT_REF xdr left join XML_DOCUMENT xd on xdr.DOCUMENT_ID =
xd.DOCUMENT_ID where xdr.COMPOSITE_DN like '%<<compositeName>>%'
2.13 Query to get payload size of message
select document_id , DOCUMENT_TYPE, DBMS_LOB.GETLENGTH(document) from
XML_DOCUMENT
2.14 Query to get execution time of BPEL instances
select composite_name COMPOSITENAME,A.CMPST_ID
COMPOSITE_INSTANCE_ID,creation_date BEGIN_TIME,modify_date END_TIME,
(extract(day from modify_date - creation_date)*86400+ extract(hour from
modify_date - creation_date)*3600+ extract(minute from modify_date -
creation_date)*60+
extract(second from modify_date - creation_date)) duration_in_second,A.*
from cube_instance A where state = 5 and creation_date
between TO_DATE('<<YYYY-MM-DD HH24:MM:SS>>','YYYY-MM-DD HH24:MI:SS')and
TO_DATE('<<YYYY-MM-DD HH24:MM:SS>>','YYYY-MM-DD HH24:MI:SS')
and composite_name in (composite_name)
2.15 Query to get the execution time of BPEL instances and to find the parent
that has initialized the composite
select
qu.row_id,qu.ID,qu.composite_id,cu.STATUS,qu.created,qu.duration_in_second
from CUBE_INSTANCE cu,
(
select cmpinst.TITLE as row_id,cube.CIKEY as ID,cube.CMPST_ID as
composite_id,
cube.CREATION_DATE as created,(extract(day from cube.modify_date -
cube.creation_date)*86400 +
extract(hour from cube.modify_date - cube.creation_date)*3600 +
extract(minute from cube.modify_date - cube.creation_date)*60 +
extract(second from cube.modify_date - cube.creation_date))
duration_in_second
28

from CUBE_INSTANCE cube,COMPOSITE_INSTANCE cmpinst
where cube.CMPST_ID=cmpinst.ID and cube.COMPOSITE_NAME='COMPOSITE_NAME' and
cube.COMPONENT_NAME='COMPONENT_NAME' and cube.STATE>4 and
cube.CREATION_DATE between TO_DATE('<<YYYY-MM-DD HH24:MM:SS>>', 'YYYY-MM-
DD HH24:MI:SS')
and TO_DATE('<<YYYY-MM-DD HH24:MM:SS>>', 'YYYY-MM-DD HH24:MI:SS')) qu
where
cu.CMPST_ID=qu.composite_id and cu.PARENT_ID=concat('bpel:',qu.ID)
2.16 Query to identify all the faults for the messages that were sitting in BPEL
Engine Level recovery as undelivered Invokes
select wif.* from WI_FAULT WIF where wif.cikey IN
(select ci.cikey from Dlv_Message dlv, cube_instance ci where dlv.State = 0
and dlv.dlv_type = 1 and dlv.ecid= ci.ecid and dlv.conv_id =
ci.conversation_id
and dlv.receive_date >TO_DATE(<<YYYY-MM-DD HH24:MM:SS>>', 'YYYY-MM-DD
HH24:MI:SS'))

29

Appendix E: Big or Large or Huge Pages
A page, memory page, or virtual page is a fixed-length contiguous block of virtual memory, and
it is the smallest unit of data for the following:
memory allocation performed by the operating system for a program; and
transfer between main memory and any other auxiliary store, such as a hard disk drive.
Virtual memory allows a page that does not currently reside in main memory to be addressed
and used. If a program tries to access a location in such a page, an exception called a page
fault is generated. The hardware or operating system is notified and loads the required page
from the auxiliary store (hard disk) automatically. A program addressing the memory has no
knowledge of a page fault or a process following it. Thus a program can address more (virtual)
RAM than physically exists in the computer. Virtual memory is a scheme that gives users the
illusion of working with a large block of contiguous memory space (perhaps even larger than
real memory), when in actuality most of their work is on auxiliary storage (disk). Fixed-size
blocks (pages) or variable-size blocks of the job are read into main memory as needed.
A transfer of pages between main memory and an auxiliary store, such as a hard disk drive, is
referred to as paging or swapping.
It is not recommended to allocate too many Huge Pages. These pre-allocated pages can only
be used for shared memory. This means that unused Huge Pages will not be available for
other use than for shared memory allocations even if the system runs out of memory and
starts swapping.
Every Operating System has different way to configure Large Pages.
1 Linux
1. To enable Reserve memory to be used for large pages, executing following command:
echo n > /proc/sys/vm/nr_hugepages
Where n is the number of desired pages.
Execute this step as soon as possible after the machine has been started since
ongoing memory usage creates fragmentation and Linux might be unable to
allocate the number of specified pages.
2. To enable the JRockit JVM to allocate large pages, make a hugetblfs file system
available by using following command:
mount -t hugetlbfs nodev /mnt/hugepages
3. Grant the user executing the Java application read and write permission to the file
system. This can be achieved by either the mount command or with the chmod and
chown commands.
30

2 Windows
1. As Administrator, give the user who will run the application the permission to lock
pages in memory by opening the Start menu and selecting:
Control Panel Administrative Tools Local Security Policy
Local Policies User Rights Assignments Lock pages in memory.
2. Select Lock pages in memory.
3. Make enough free consecutive memory available by either logging off from computer
or rebooting it.
3 Solaris
Nothing has to be configured in the O/S to enable an application to use large pages.
4 Reference:
Page (computer memory) Retrieved July 26, 2013:
https://en.wikipedia.org/wiki/Page_(computer_memory)

31

Appendix F: ORA-01438: value larger
than specified precision allowed
Some time while running through the logs (SOA managed server out and diagnostic log file); one
may encounter ORA-01438 error. This error is due to exiting bug in BPEL PM.
5 What is the error in logs?

Error while invoking bean "cube delivery": Exception not handled
by the Collaxa Cube system.[[
an unhandled exception has been thrown in the Collaxa Cube
systemr; exception reported is: "ORABPEL-00000

Exception not handled by the Collaxa Cube system.
an unhandled exception has been thrown in the Collaxa Cube
systemr; exception reported is: "java.sql.SQLDataException: ORA-
01438: value larger than specified precision allowed for this
column

at
oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMappi
ng.java:83)
at
oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.j
ava:135)
at
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError
.java:210)
6 Effects

Server spends additional processing cycles on logging errors, which results in degraded
performance. Apart from performance issue, lots of disk space will be utilized by log files.
7 Cause

This is an existing bug (12621337: ORA-01438 ON LIVE_INSTANCES COLUMN). The issue is with
the precision of the column COMPOSITE_INSTANCE.LIVE_INSTANCES. The currently defined
data type is NUMBER(3) which can at the most hold a value up to 999.
32

8 Solution

The solution is simple, increase precision of column to NUMBER(38).

Steps to apply solution:

1. Stop the SOA domain
2. Log in to the SOAINFRA schema as SYSDBA
3. Modify the COMPOSITE_INSTANCE table to increase precision

alter table soainfra.composite_instance modify (live_instances NUMBER(38))

33

Appendix G: LOBs in the SOAINFRA
schema
TABLE LOB COLUMN
BPM_CUBE_ACTIVITY_INSTANCE ERRORMESSAGE
BPM_CUBE_ACTIVITY SOURCECODE
BPM_CASE_DATA DATA
BPM_AUDIT_QUERY AUDIT_COMMENT
BPM_ACTIVITY_INSTANCE ERRORMESSAGE
BPM_ACTIVITY SOURCECODE
SOAQTZ_TRIGGERS JOB_DATA
BPM_CUBE_PROCESS DEPLOYMENTINFO
BPM_CUBE_AUDITINSTANCE BUSINESSINDICATORS
B2B_WIRE_MESSAGE ERROR_TEXT_CLOB
B2B_WIRE_MESSAGE ERROR_DESCRIPTION_CLOB
B2B_TRANSPORT_MANAGER STATS
B2B_EXT_BUSINESS_MESSAGE PROTOCOL_WORK_AREA
B2B_EXT_BUSINESS_MESSAGE ERROR_TEXT_CLOB
B2B_EXT_BUSINESS_MESSAGE ERROR_DESCRIPTION_CLOB
B2B_EXT_BUSINESS_MESSAGE ERROR_DETAIL_CLOB
MDS_LARGE_ATTRIBUTES ATT_LONG_VALUE
MDS_METADATA_DOCS MD_CONTENTS
MDS_STREAMED_DOCS SD_CONTENTS
WFRULEDICTIONARY DICTIONARY
WFROUTINGSLIP ROUTINGSLIP
WFMESSAGEATTRIBUTE BLOBVALUE
WFHEADERPROPS PROPERTIES
WFEVIDENCE SIGNATURE
WFEVIDENCE PLAINTEXT
WFCERTIFICATEREVOKED VALIDATIONDATA
WFCERTIFICATE CERTIFICATE
WFATTACHMENT CONTENT
BPELNOTIFICATION MESSAGE
PC_TASKPAYLOAD PAYLOAD
PC_TASKATTACHMENT CONTENT
XML_DOCUMENT DOCUMENT
TEST_INSTANCE RESULTS
REJECTED_MSG_NATIVE_PAYLOAD PAYLOAD
REJECTED_MESSAGE ERROR_MESSAGE
REJECTED_MESSAGE STACK_TRACE
REFERENCE_INSTANCE ERROR_MESSAGE
34

REFERENCE_INSTANCE STACK_TRACE
COMPOSITE_INSTANCE_FAULT ERROR_MESSAGE
COMPOSITE_INSTANCE_FAULT STACK_TRACE
ATTACHMENT ATTACHMENT
EDN_LOG_MESSAGES MSG
EDN_EVENT_ERROR_STORE PAYLOAD
BPM_PML_HS METADATA_CHANGE
BPM_PML_HS CHANGES
WLI_QS_REPORT_DATA DATA_VALUE
MEDIATOR_PAYLOAD BIN
MEDIATOR_CASE_INSTANCE FAULT_OBJ
MEDIATOR_CASE_DETAIL AUDIT_TRAIL
MEDIATOR_AUDIT_DOCUMENT DOCUMENT
WFUSERTASKVIEW DEFINITION
BPM_USERAPPLICATIONDATA APPLICATIONDATA
BPM_PRESENTATION DEFINITION
B2B_BAM_QTAB USER_DATA"."HEADER"."PROPERTIES
B2B_BAM_QTAB "USER_DATA"."TEXT_LOB"
B2B_BAM_QTAB USER_PROP
VARIABLE_SENSOR_VALUES BLOB_VALUE
VARIABLE_SENSOR_VALUES CLOB_VALUE
FAULT_SENSOR_VALUES MESSAGE
COMPOSITE_SENSOR_VALUE CLOB_VALUE
COMPOSITE_SENSOR_VALUE BLOB_VALUE
ACTIVITY_SENSOR_VALUES ERROR_MESSAGE
BRDECISIONUNITOFWORK UOW_DATA
BRDECISIONINSTANCE DECISION_TRACE
BRDECISIONFAULT MESSAGE
CUBE_SCOPE SCOPE_BIN
AUDIT_DETAILS BIN
EDN_OAOO_DELIVERY_TABLE SYS_NC00032$(*)
EDN_OAOO_DELIVERY_TABLE USER_PROP
EDN_EVENT_QUEUE_TABLE SYS_NC00032(*)
EDN_EVENT_QUEUE_TABLE USER_PROP
TASK_NOTIFICATION_Q_T USER_DATA."HEADER"."PROPERTIES
TASK_NOTIFICATION_Q_T USER_DATA."TEXT_LOB"
TASK_NOTIFICATION_Q_T USER_PROP
B2B_APP_MESSAGE APP_MESSAGE_PROPS
B2B_APP_MESSAGE ERROR_TEXT_CLOB
B2B_APP_MESSAGE ERROR_DESCRIPTION_CLOB
B2B_DATA_STORAGE CLOB_VALUE
B2B_DATA_STORAGE BLOB_VALUE
AG_INSTANCE MILESTONE_STATE
35

BPM_PROJECTSHAREDATA VALUEBLOB
SOAQTZ_JOB_DETAILS JOB_DATA
SOAQTZ_CALENDARS CALENDAR
SOAQTZ_BLOB_TRIGGERS BLOB_DATA
WI_FAULT MESSAGE
TEST_DETAILS TEST_RESULT
TEST_DEFINITIONS DEFINITION

36

Appendix H: AWR, ADDM, & ASH Reports
Database has Automatic Workload Repository (AWR), Automatic Database Diagnostic Monitor
(ADDM) and Active Session History (ASH) to gather and analyze database performance statistics
which helps optimize database and SQL statements.
1 AWR Report

AWR report provides performance summary that include:

CPU bottlenecks
Undersized Memory Structures
I/O capacity issues
High load SQL statements
High load PL/SQL execution and compilation, and high-load Java usage
Oracle RAC specific issues
Sub-optimal use of Oracle Database by the application
Database configuration issues
Concurrency issues, Hot objects

2 ADDM Report
ADDM report provides information pertaining to areas of database which are consuming most
time:

Lock contention
Excessive parsing
I/O capacity
Incorrect sizing of SGA, PGA

ADDM report also provides:

Recommends solutions and quantifies expected performance benefits.
Recommends changes to hardware, database configuration, database schema, or
applications
Estimated impact on performance
3 ASH Report

ASH report provides information about:
37

Historical view of active sessions Sampled every second
Top User Events (frequent wait events)
Load Profile
Details to the wait events
Top Queries and PL/SQL
Top Sessions
Top Blocking Sessions
Top DB Objects (Objects/Files/Latches)
Activity Over Time
Enables after the fact analysis
Drilldown to a more granular period
P1, P2, P3 values
List details of only a session, SQL, Wait class service, module over a period of a time
Determines blocking session, hot segment

ASH is helpful when there's a sudden performance degradation of the database.
4 AWR Report Analysis
AWR Report has various sections but from performance perspective take a deep dive in
following sections:
SQL statements ordered by Elapsed Time
SQL statements ordered by CPU Time
SQL statements ordered by Gets
SQL statements ordered by Reads
SQL statements ordered by Executions
SQL statements ordered by Parse Calls
4.1 SQL statements ordered by Elapsed Time
This section lists top SQL statements which has large elapsed time (CPU time plus any other
wait times) in descending order of Total Elapsed time. Check out the SQL statements which are
taking large time to execute. Take help from DBA to find out the reason.
If the query execution is taking long time, the JDBC calls made from SOA will wait till it gets the
response back which could cause stuck threads, timeout issues etc.
4.2 SQL statements ordered by CPU Time
Most of the times the SQL statements from the Elapsed Time list also find their slot as top SQL
statements in the CPU Time list as well. But it's not in all cases. If the SQL exists in Elapsed Time
list but not in CPU Time list, then there is an issue with recursion associated with this SQL
statement.
38

Larger CPU time usually caused by excessive function calls, missing filter columns in the SQL
statement which results full table scan.

If SQL statement under consideration is select type then indexing might help to improve
performance.
4.3 SQL statements ordered by Gets
SQL statements listed in this section due to large logical reads from DB Block buffers. Usually
reading from logical read is desirable but excessive reading from logical also can cause
performance issues.
High logical read results due to missing indexes on the filter columns or full table scan. Setting
proper filter columns and/or indexes might help to improve performance by reducing the
logical read time.

4.4 SQL statements ordered by Reads
SQL statements listed in this section due to larger disk reads (physical read) rather than reading
from buffers. Disk reads are undesirable. Excessive disk reads can cause performance issues.
Usually larger disk reads is due to missing indexes.
To improve performance, increase the buffer cache (If the memory is not a constraint at
database server) and/or tune the query with additional filters and/or tune table by adding
additional indexes.
4.5 SQL statements ordered by Executions
SQL statements listed in this section are executed most number of time in the given time
period. Usually it is OK if application design or load pattern warrants repeated execution of few
SQL statements execution.
Ensure that SQL statements listed here must not be in above lists. SQL statements listed here
must be very efficient.
4.6 SQL statements ordered by Parse Calls
SQL statements listed in this section are parsed/re-parsed repeatedly. Whenever a SQL
statement contains unsafe bind variables, then it get parsed each time. It leads to larger
execution time.

Check for SQL statements listed here and in Elapsed Time list as well. If one is found in both lists
then investigate, why this statement is parsed multiple times and whether it's normal.
39

5 Reference
Using Automatic Workload Repository for Database Tuning: Tips for Expert DBAs:
http://www.oracle.com/technetwork/database/manageability/diag-pack-ow09-
133950.pdf
Oracle Database Performance Tuning Guide:
http://docs.oracle.com/cd/B28359_01/server.111/b28274/toc.htm
Beginning Performance Tuning: Active Session History:
http://www.oracle.com/technetwork/issue-archive/2013/13-jan/o13dba-1871177.html
Oracle Database 2 Day + Performance Tuning Guide:
http://docs.oracle.com/cd/B28359_01/server.111/b28275/toc.htm

40

Appendix I: Monitoring Scripts
In a typical deployment, during LnP (even in production) test few of the resources need to be
monitored. One can take help of reference scripts to monitor few of the critical resources.
Before using these monitoring scripts, one should be aware that these scripts consume
resources which will affect performance of the system.
1 Database Monitoring
This script monitors database connections in use at a given time. One can schedule this script to
run continually (say with gap of 5 min) during LnP test.
Later, one can compile the report from Email and log files generated and perform trend analysis
with load pattern.

Exhibit 4: Database Monitoring
SQL files
SQL files lists statements which will be executed by sh file periodically.
dbmon.sh
This file does heavy lifting by invoking SQL files, sending emails and writing logfiles.
ORACLE_HOME: Set Oracle home in place of <<OracleHome>>
41

LD_LIBRARY_PATH: Put down path of LD_LIBRARY in place of <<PathToLDLibrary>>
PATH: Ensure that Oracle database client library is in path. Replace
<<PathToOracleClientLibrary>> with location of Oracle database client library
DBMON_HOME: This is home for this script. Replace
<<SharedLocationForMonitoringScripts>> with location where monitoring scripts are
kept.

dbmon.properties

DBNAME: Database name. Replace <<Databasename>> with database name.
DBHOST: Database host name. Replace <<HostName>> with database host name.
DBPORT: Database listening port. Replace <<port>> with database listening port
number.
DBSERVICE: database Service name. Replace <<ServiceName>> with database
service name.
DBUSER: Database user name. Replace <<userName>> with database user name.
DBPASSWORD: Database password. Replace <<password>> with database password.
SOAENV: SOA environment, under consideration. Replace <envName>> with SOA
environment under consideration.
MAILFROM: From where e-mail has been sent. Replace aqmonitor@example.com
with an e-mail ID who should send this mail.
MAILTO: Comma separated list of e-mail IDs to whom email should be sent. Replace
support@example.com,LnP@example.com with comma separated list of e-mail
IDs.
MAILCC: Comma separated list of e-mail IDs to whom email should be sent as
carbon copy reciepient. Replace
supportAdmin@example.com,supportDatabase@example.com with comma
separated list of e-mail IDs.
DBSESSIONCOUNTTHRESHOLD: Threshold for session count. Replace
<<ValueOfDatabaseSessionThreshold>> with positive integer value.

output.html and dbmon.log
These files will store the log information HTML and text format in logs folder.
2 JMS Monitoring
This script monitors pending messages on JMS queue under consideration. One can schedule
this script to run continually (say with gap of 5 min) during LnP test.
with load pattern.
42

Exhibit 5: JMS Monitoring
There are couples of config files which need to be modified as required. But lets talk about
jmsmon.sh.
jmsmon.sh
43

In this file one need to replace <<SharedLocationForMonitoringScripts>> with
location where monitoring scripts are kept.

Files in config and sub folder
jmsmon.properties
This file contains details of JMS server. Queues on this JMS server will be monitored.
hostname: Host name of computer where JMS server is hosted. Replace
<<HostName>> with hostname.
portString: Port number where JMS server is listening. Replace <<port>> with
port number.
Username: User name of JMS server. Replace <<userName>> with user name.
Password: Password of JMS server. Replace <<password>> with password.
MailTo: Email ID where email should be sent in case of failure. Replace
<<EmailID>> with Email ID.
Servers: Semicolon separated list of servers, if JMS has cluster. Replace
=<<Semicolon separated list of JMS servers>> with semi-colon separated list of
servers. For example JMSServer1;JMSServer2;JMSServer3.
monitorconfig.csv
This file contains list of queues (comma separated) which need to be monitored. Here one
needs to list each queue with value for:
QueueName
WarningThreshold-LC
CriticalThreshold-LC
WarningThreshold-PendingMsgCount
CriticalThreshold-PendingMsgCount
WarningThreshold-CurrentMsgCount
CriticalThreshold-CurrentMsgCount
domains.csv
This file contains list of (comma separated):
Partition
Service Name
Weblogic Domain or System Name
Destination Name
destinations.emailcc
44

This file lists email IDs with respect to each domain under consideration.
mailcctmp
This file lists domains for which carbon copy (cc) mail is sent.
dntmailtmp
This file lists domains for which carbon copy (cc) mail is not sent.
bin
This folder contains java code. This java code is tested for three server cluster. One may need to
modify if there different number of servers in cluster.
lib
This folder contains java libraries required by java code.
logs
This folder will contain log file.
3 AQ Monitoring
This script monitors pending messages on JMS queue under consideration. One can schedule
this script to run continually (say with gap of 5 min) during LnP test.
with load pattern.
This script is consists of few file which are dedicated to do specific work. Let us go through each
piece one by one.
45

Exhibit 6: AQ Monitoring
DB_1_aq_monitor.sql and DB_2_aq_monitor.sql

These files contain SQL statements which query AQ tables. These SQL statements are executed
by commands listed in aqmon.sh file.

There are two sample files but one can add as much as many required. This count is determined
by number of databases to be connected for AQ monitoring.

In each sample file set SQL queries are listed as per the AQs to be monitored in this schema. In
DB_1_aq_monitor there are 3 SQL statements while in DB_2_aq_monitor has two. It also
means that first database has three AQs to be monitored while second database has two.

Add SQL files as required and also keep SQL statements in each files as required.

In the sample files replace <<SchemaName>> with schema name where AQ tables are. Also
replace <<TableName>> with AQ table.

aqmon.sh

This file contains script which picks up SQL statements from DB_1_aq_monitor.sql and
DB_2_aq_monitor.sql files, write log files and sends email.

This file should be called from scheduler (or cron job). In this file one need to modify certain
values as per deployment under consideration.
46

ORACLE_HOME: Set Oracle home in place of <<OracleHome>>
LD_LIBRARY_PATH: Put down path of LD_LIBRARY in place of <<PathToLDLibrary>>
PATH: Ensure that Oracle database client library is in path. Replace
<<PathToOracleClientLibrary>> with location of Oracle database client library
AQMON_HOME: This is home for this script. Replace
<<SharedLocationForMonitoringScripts>> with location where monitoring scripts are
kept.
To add new databases, refer header comments in file.

aqmon.properties

This file contains details of database to be connected and e-mail details.

DBNAME_1: Database name. Replace <<Databasename>> with database name.
DBHOST_1: Database host name. Replace <<HostName>> with database host name.
DBPORT_1: Database listening port. Replace <<port>> with database listening port
number.
DBSERVICE_1: database Service name. Replace <<ServiceName>> with database
service name.
DBUSER_1: Database user name. Replace <<userName>> with database user name.
DBPASSWORD_1: Database password. Replace <<password>> with database
password.
On similar line update DBNAME_2, DBHOST_2, DBPORT_2, DBSERVICE_2,
DBUSER_2, and DBPASSWORD_2.
To add new database details add DBNAME_3, DBHOST_3, DBPORT_3, DBSERVICE_3,
DBUSER_3, and DBPASSWORD_3.
SOAENV: SOA environment, under consideration. Replace <envName>> with SOA
environment under consideration.
MAILFROM: From where e-mail has been sent. Replace aqmonitor@example.com
with an e-mail ID who should send this mail.
MAILTO: Comma separated list of e-mail IDs to whom email should be sent. Replace
support@example.com,LnP@example.com with comma separated list of e-mail
IDs.
MAILCC: Comma separated list of e-mail IDs to whom email should be sent as
carbon copy reciepient. Replace
supportAdmin@example.com,supportDatabase@example.com with comma
separated list of e-mail IDs.

aq_monitor.html and aqmon.log
These files will store the log information HTML and text format in logs folder.

47

48

Appendix J: How to Monitor SOA Server
Memory Usage
During LnP test monitoring of SOA server for JVM memory usage, garbage collection, memory
related error is very important because of obvious reasons. After LnP analysis of memory usage,
heap analysis and thread dump analysis are required to take a deep dive to understand causes
and solution of issues.
For this purpose, there are several tools but following might be sufficient from WLS and BPEL
PM perspective.
jConsole
visaulVM or JvisualVM
JRockit Mission Control (JRMC)
1 Setup: jConsole or visualVM (installed locally)
Since most of the time BPEL PM and other servers might be running on servers which do not
have any window component, so running UI application is not possible. In this case one needs
to export display to a computer which has window component. To do this follow following
steps:
a. Ensure that jConsole or visualVM or/and JRockit Mission control is/are installed on the
computer which need to be monitored.
b. Ensure that target machine where one needs to open UI has putty and XMing installed
c. Start XMing on target machine
d. Configure Putty:
In the box below Host Name (or IP Address) key-in server name or IP address.
Make sure SSH is selected.
Key-in server name or IP address in the box below Saved Sessions.
49

Exhibit 7: Setup jConsole or visualVM (installed locally) - 1
Navigate till Connection SSH X11 and click on X11
50

Select Enable X11 forwarding check box.
Scroll the left hand window back to the top and click on the Session heading
51

Click the Save button. The host name entered should now appear below Default
Settings. In the future, one will be able to connect by simply double-clicking this host
name.
Click the Open button.
In case of PuTTy Security Alert window click on Yes
SSH window should appear. Provide credentials and navigate to folder where
jConsole or visualVM is present.
Start jConsole (./jconsole &) or visualVM (./visualvm &) as background process
Find out process ID (PID) of the server to be monitored
By this time jConsole or visualVMs UI should be available at target machine.
In jConsole or visualVM, using PID monitor Heap usage and garbage collection.
In ideal scenario, one should get saw tooth shaped memory usage and garbage collection
pattern.
52

2 Setup: JvisualVM (installed at remote machine)

JVisualVM comes with HotSpot JVM and can be found at $HotSpot_JDK_INSTALL/bin.
JVisualVM should be installed on separate machine to minimize the monitoring overhead. On
the runtime server, one needs to provide the JMX parameter to JVM.
To configure a JMX port, one should edit setSOADomainEnv.sh file in
$Middleware_Home/user_projects/domains/<<DomainName>>/bin to add the -
Dcom.sun.management.jmxremote parameters.
Define this parameter for a specific server in the domain since reuse of the JMX port number
between servers is not allowed.
if [[ "${SERVER_NAME}" = "soa_server1" ]]
then
PORT_MEM_ARGS="-Xms1024m -Xmx2048m -XX:PermSize=256m -XX:MaxPermSize=512m -
Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -
Dcom.sun.management.jmxremote.authenticate=false -
Dcom.sun.management.jmxremote.ssl=false"
echo "DEBUG! (soa_server1) USER_MEM_ARGS: ${USER_MEM_ARGS}"
fi
Server needs restart, for this parameter to take into effect.
53

To start jVisualVM GUI, execute
./jvisualvm -J-XX:MaxPermSize=256m
from $HotSpot_JDK_INSTALL/bin.
To ensure that jVisualVM installation has required plugins, download all plugins.
Select Tools Plugins download and install all of the available plugins.
Restart JVisualVM.
Select the Add Remote Host icon from menu.

Key in the Host name and Display name where the SOA managed server is running and select
OK.

Right click the new Remote connection and select Add JMX Connection...

In the Connection field enter the defined JMX port of 9004, this is from -
Dcom.sun.management.jmxremote.port=9004, also check the Display name box.

Select OK, open the + to the left of the Remote connection and double click the JMX
connection, verify the following tabs are displayed.

Note: It is possible that the Visual GC tab will only show the message "Not supported for this
JVM." You will get this if the JDK running JVisualVM and the SOA server are not the same
version or if the operating systems do not match. One way to get this tab to display is to run
JVisualVM from the same JDK as the SOA server.
After double left clicking the SOA managed server , verify the following tabs are displayed.
Monitor Tab
It provides the graphs of the running system.

Also there is a Perform GC button to force a garbage collection and a Heap Dump button to
cause a heap dump. The resulting heap dump will be eventually loaded into JVisualVM in a new
tab where it can be analyzed. The loading may be slow and not detailed enough.
One can take heap dump from the command line from the JDK installation where the SOA
server is running:
$HotSpot_JDK_INSTALL/bin/jmap -dump:live,file=heap.dump.out,format=b <<pid of
the SOA managed server>>
54

The resulting heap dump file *.out can be viewed from Eclipse Memory Analyzer (MAT).
Visual GC Tab
It provides a visual representation of the memory Spaces being used in real-time in the
PermGen, Old Gen, Eden Space, and Survivor Spaces (S0 & S1). This gives an idea of how how
full each partition is at any given time. One can use this to scale the defined memory for the
spaces based on actual load.

The Graphs section also provides information on the maximum and current sizes of the spaces
and their garbage collection statistics.

Tracer Tab
It allows to choose which metrics to keep track of in a running graph. After selecting the Start
button the graphing of each metric starts.

After desired time tracing can be stopped and can be exported as .csv file ( Select floppy disk
button "Export all data").
3 Setup: JRockit Mission Control (installed at remote machine)
JRMC is used to monitor JRockit JVM. JRMC should be installed on a separate machine from the
SOA runtime server to minimize the monitoring overhead. On the runtime server, one need to
setup the JMX monitoring port.

In 11g SOA server installation one can edit the setSOADomainEnv.sh file in
$Middleware_Home/user_projects/domains/<<myManagedDomain>>/bin to add the -
Xmanagement parameters. Add following to this file:
if [[ "${SERVER_NAME}" = "soa_server1" ]]
then
PORT_MEM_ARGS="-Xms2048m -Xmx2048m -
Xmanagement:ssl=false,authenticate=false,port=7093"
echo "DEBUG! (soa_server1) USER_MEM_ARGS: ${USER_MEM_ARGS}"
fi
These parameters must be defined for a specific server in the domain since reuse of the JMX
port number between servers is not allowed.
The -Xmanagement parameters will setup the JVM for the JRMC connection. To come in effect
these parameters, soa_server1 need to be restarted.
55

Start JRMC. Right click the Connectors folder and select New Connection:

Fill out the Host of the SOA server and the Port as defined in -Xmanagement:port=7093,
selecting "Test Connection" should show OK

Select Finish and the new connection should appear under Connectors, highlight the new
connection and select to start a console which will monitor the JVM in real-time.

This will depict memory, CPU usage, threads, and methods are being used.

memleak depicts the type of java objects present, growth rate, percentage of heap, instances,
and total size from largest to smallest. This view helps in determining which objects are taking
up the most memory:

To keep record of for comparison purpose, use Flight recording option. Data is kept in .jfr file.
4 Reference
visualVM: http://visualvm.java.net
jConsole: http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html
Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server:
http://docs.oracle.com/cd/E21764_01/web.1111/e13814/wldf.htm
Oracle JRockit Mission Control Introduction to Mission Control Client:
http://docs.oracle.com/cd/E15289_01/doc.40/e15067/toc.htm
Oracle JRockit Documentation: http://docs.oracle.com/cd/E15289_01/index.htm
Putty: http://www.putty.org
XMing: http://sourceforge.net/projects/xming

56

Appendix K: Heap Dump Files analysis:
JRockit and HotSpot JVMs
Eclipse Memory Analyzer (MAT) and Java Heap Analysis Tool (jhat) can be used to analyze of
heap dump of JRockit and HotSpot.
Heap dump files from JRockit and HotSpot has following name format:
JRockit: jrockit_<process id>.hprof
HotSpot: java_pid<process id>.hprof
One can download MAT from eclipse web site and jhat can be found at
$JDK_INSTALL_LOCATION/bin/jhat.

A heap dump file usually is larger in size than the maximum heap size defined for the server in
the -Xmx parameter. The large heap dump file may exceed the memory capacity of MAT. In this
scenario MAT may run out of memory or MAT may truncate the file or corrupt the file.
In this scenario, configure MAT to run with more memory by editing the MemoryAnalyzer.ini
file. To configure MAT to run with 7GB of memory, just add following lines to the file:
-vmargs
-Xmx7168m

Restart MAT.
Maximum memory setting for MAT depends upon:
Operating system (32 or 64 bit)
Physical RAM available (installed and sharing with other applications running)
1 Example Analysis of a Heap Dump File Using Eclipse Memory
Analyzer
Here is a practical example of a heap dump collected at the time of out-of-memory using a
HotSpot JVM when opened using Eclipse Memory Analyzer Version 1.0.1. This was collected
from an 11g SOA installation where Microsoft SQLServer Database was being used as the
57

dehydration store.

The problem in this case was that the Data Direct Microsoft SQLServer JDBC Drivers caused a
memory leak in the container. After some time the leak caused the java heap allocation to run
out of memory. The solution was to change from the Data Direct SQLServer JDBC Drivers to the
Microsoft JDBC Drivers.
Analysis of the heap dump in Eclipse Memory Analyzer led to the conclusion that there was a
problem with the Drivers and that a change in Drivers would most likely resolve the memory
leak problem.

These are some screen shots from the heap dump analysis done in Eclipse Memory Analyzer
(MAT) for this case.
A. Open the *.hprof file in MAT: File -> Open File -> java_pid8368.hprof:

Exhibit 11: Heap Dump Files analysis - 1
B. Select "Leak Suspects Report" and then click on Finish:
58

C. This will produce an Overview Tab of the Biggest Java Objects by Retained Size and lists of
Actions and Reports:


D. The other tab is "default_report org.eclipse.mat.ami:suspects" which contains the Leak
Suspects. This shows that MAT has detected two leak suspects:
59


If there are no leaks suspects are shown then there are three possibilities:
There are no leak suspects
Max. heap size is not large enough
Heap size is not tuned
To confirm one of the possibilities, one should try some alternative settings of JVM
E. Scrolling Down on the Same Tab Shows More Detail on the Leak Suspects:
60


These are two instances of the same Leak Suspect. The "weblogic.sqlserverutil.ddo" is a hint that the
problem is related to a MSSQL Server database connection.
F. The idea that the problem is with a MSSQL Server Database Connection is reinforced by
selecting the details link and then opening the "Shortest Paths to the Accumulation Point":
61


This looks to point to a SQLServer data source. In a SOA installation a SQLServer datasource may be
allocated for use by a database adapter or as the database where the overall SOA Suite database
repository has been installed by the running of the Repository Creation Utility (RCU). In this example the
SQLServer data sources were used for the SOA Suite database repository. After contacting WebLogic
JDBC support it was recommended to change the Microsoft SQLServer driver from Data Direct to
Microsoft. This will end up in resolving the problem.
G. If it is not clear from the Leak Suspects Report as to what is the cause of the out-of-memory,
make sure to look into the other Actions and Reports provided at the Overview tab:
Histogram
Dominator Tree
Top Consumers
Duplicate Classes
Top Components

These Reports and Actions are explained in the Overview tab. Close examination of all of the Reports
and Actions may lead to finding the cause of the memory leak.
62

2 Reference
1. Eclipse Memory Analyzer: http://www.eclipse.org/mat
2. JVM Memory Monitoring, Tuning, Garbage Collection, Out of Memory, and Heap Dump Analysis
For SOA Suite Integration 11g (Doc ID 1358719.1):
https://support.oracle.com/epmos/faces/ui/km/SearchDocDisplay.jspx?returnToSrId=&srnum=
&_afrLoop=327168651925814&type=DOCUMENT&id=1358719.1&displayIndex=1&_afrWindow
Mode=0&_adf.ctrl-state=r1sv9bzh9_202

63

Appendix L: Capacity Planning
Capacity planning is the process of determining the software and hardware capacity needed by
a system to serve dynamic load patterns at satisfactory performance/throughput over a pre
decided time period.
Like performance planning, capacity planning is an iterative process. An effective and efficient
capacity management plan is based on monitoring and measuring load over time, forecast and
implementing flexible solutions to handle variances without impacting performance.
1 Capacity Planning for BPEL PM
While performance tuning is for the optimizing existing system for better performance and
throughput, capacity planning is for system needs (and when it needs it) to maintain
performance in both steady-state and peak usage periods.
Capacity Planning involves designing solution and testing the configuration, as well as
identifying business expectations, periodic fluctuations in demand, and application constraints.
One needs to plan carefully, test methodically, and incorporate design principles that focus on
performance and scalability. For any capacity planning activity, LnP testing is essential part.
Capacity Planning Factors
For Capacity Plan creation one need understanding of BPEL PM & supporting sub-systems and
data to defend deployment strategy.
Before and/or during capacity planning for BPEL PM, following questions must be asked and
data collected to be used thereof for a successful end result.
Table X.X Capacity Planning Factors to Consider
Question Response
What are current performance goals and objectives?

What will be future performance goals and objectives?
How many processes will run concurrently?
What is current load pattern under various business conditions?
How load pattern is expected to change over time period on existing process?
What is expected growth in terms of new processes over time period on
existing hardware/network/software resources?

64

Question Response
Is the simulated workload adequate for current and future load patterns?
Is the Fusion Middleware deployment configured to support clustering and
other high availability factors?

Does the hardware meet the configuration requirements?
Does number of is adequate JVMs to support load?
Is the database a limiting factor?
How is the weakest link in the chain in terms of load carrying capacity?
What is purging strategy?
With increased load how hardware/network/software assets will scale?
What analytic modeling will be used for Capacity planning?
Is there enough network bandwidth?

Creating an effective Capacity Management plan includes following steps:
Step 1: Identifying Performance Goals and Objectives
Step 2: Measuring Performance Metrics
Step 3: Identifying Bottlenecks
Step 4: Capacity Management Plan Implementation

1.1 Determining Performance Goals and Objectives Current & Future
The first step in creating an effective capacity management plan is to find out network load and
performance objectives current and future. One need to understand the processes deployed
and the environmental constraints placed on the system. One should have information about
the levels of activity that components of the system are expected to meet, such as:
The anticipated load pattern
The number of concurrent sessions
The number of SSL connections at a given point of time and in total
The size of messages in the system
The amount of data and its consistency.
Target CPU utilization
Target memory utilization
Performance objectives are limited by constraints, such as
65

The configuration of hardware and software such as CPU type, disk size, disk speed,
memory, cache, network connectivity, etc.
The ability to interoperate between domains, use legacy systems, support legacy data.
The security requirements and use of SSL. SSL involves intensive computing operations
and supporting the cryptography operations in the SSL protocol can impact the
performance of the WLS.
Development, implementation, and maintenance costs.
One can use this information to set realistic performance objectives for system in terms of
response times and throughput.
1.2 Measuring Performance Metrics
Conduct series of LnP tests under various conditions and measure the metrics of interest to
quantify the performance objectives. Benchmarking key performance indicators provides a
performance baseline.
1.3 Identifying Bottlenecks
Bottlenecks must be identified and addressed while developing capacity management plan.
Profile the system and processes to pinpoint bottlenecks and improve application performance.
Use statistical analysis tools on the data collected while measuring performance metrics. On the
basis of results of statistical analysis design new LnP tests and isolate the bottlenecks.
1.4 Implementing a Capacity Management Plan
Once step 1, 2 and 3 are performed on iterative basis, create and implement a capacity
management plan. The goal of the plan should be at least meet or exceed performance
objectives (especially during peak usage periods) and to allow for future workload increases.
To achieve defined performance objectives, implement capacity management plan and then
continuously monitor the performance metrics to incorporate any changes required.
Since each deployment is unique, so there is no one fit all capacity management plan but all
capacity management plan must include followings:
Hardware Requirements
Network Requirements
CPU Requirements
Memory Requirements
RAM Requirements
Cache Requirements
JVM Requirements
Management Requirements
66

Database Requirements
Deployment Topology
LnP Testing Requirements and Strategy
2 Reference
Capacity Planning at Wikipedia: http://en.wikipedia.org/wiki/Capacity_planning
Oracle Fusion Middleware Performance and Tuning Guide:
http://docs.oracle.com/cd/E23943_01/core.1111/e10108/capacity.htm

BPEL PM 11g Performance Tuning - Appendies

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BPEL PM 11g Performance Tuning - Appendies

Uploaded by

Copyright:

Available Formats

1

You might also like