TOCF(EE) High Availability

T24 R12

User Guide

Page 1
Information in this document is subject to change without notice.
No part of this document may be reproduced or transmitted in any form or by any means,
electronic or mechanical, for any purpose, without the express written permission of TEMENOS Holdings NV.

Copyright 2002-2006 TEMENOS Holdings NV. All rights reserved.

 Document revisions
Date Author Changes Made
13/07/2010 Lucas Gut Initial revision

Page 2
Contents
Introduction.................................................................................................................................................4
High Availability architectures................................................................................................................4
Weblogic 10.3.2 High Availability.............................................................................................................8
Web-tier: Apache HTTP Server 2.2 with WL proxy plug-in..................................................................8
Presentation-tier: Weblogic cluster – T24Brower & T24RA..................................................................8
Object-tier: Weblogic cluster – TOCFListeners & DeliveryMDB.......................................................11
Websphere 7.0 High Availability..............................................................................................................15
Web-tier: IBM HTTP Server 7.0 with proxy plug-in............................................................................15
Presentation-tier: Websphere cluster – T24Browser & T24RA............................................................15
JBOSS High Availability..........................................................................................................................17
Web-tier: Apache HTTP Server 7.0 with proxy plug-in.......................................................................17
Presentation-tier: JBOSS cluster – T24Browser & T24RA..................................................................18
Testing.......................................................................................................................................................20
Web & Presentation tiers.......................................................................................................................20
Fail-over scenarios.............................................................................................................................20
Data injection test results...................................................................................................................22
Object tier..............................................................................................................................................22
Resources...................................................................................................................................................23

Page 3
Introduction

Systems requiring high availability are required to provide a fail-over mechanism to a secondary system in case of
system failure or scheduled downtime. Highly available systems should also be able to load balance requests to
prevent overloading servers and to ensure acceptable response times. Fail-over and load-balancing must be
addressed in every layer of multi-tiered system architectures (e.g. Presentation layer, Application layer and
Database layer). This document focuses on the connectivity requirements used to ensure high availability from
the presentation layer down to the application layer.

This documents assumes a good understanding of the TOCF(EE) components and the T24 Browser web
application. The high availability architectures proposed in this document require the following components:

 Latest T24 Browser R12 release

 TOCF(EE) R12.1.1 or higher

For earlier versions of R12, please refer to the R09 high availability documentation.

Starting with version TOCF(EE) R12.1.1, TOCF(EE) includes the T24 Resource Adapter (T24RA), a new
component enabling T24 Browser high availability without the need of JMS messaging. T24 Browser is now
preconfigured to use this resource adapter and provide direct connectivity between T24 Browser and T24.

The T24RA is a JCA 1.5 compliant resource adapter enabling client applications to send OFS requests to T24.
This component extends the functionality of the former jRemote Resource Adapter adding the following features:

 Enables users to specify multiple servers to provide high availability

 Load balancing option to distribute allocated connections evenly among the set of available servers
 Option to disable/enable inbound listener for CALLJEE calls (disabled by default)

The T24 resource adapter (T24RA) does not guarantee high availability by itself since it does not automatically
fail-over to other jAgent connections in case of connection failure. A connection failure triggers the resource
adapter to throw an exception which should be caught by the client application which is using the T24RA. The
latest version of T24 Browser catches these connection exceptions and attempts to resend the request.

The application server provides T24 Browser with a valid connection when it attempts to send a request. Broken
connections are automatically removed from the connection pool. Even though automatic fail-over does not take
place in the resource adapter, when the T24 RA is asked to open a connection, it will do this to the next available
jAgent server. This mechanism ensures that a request that is resent due to a connection failure is now handled by
a valid jAgent connection. The ordering of the ‘next available’ server when opening a new connection depends on
whether load balancing is enabled or disabled.

High Availability architectures

Highly available systems provide load-balancing and fail-over at different layers in case of multi-tiered
architectures. The following diagram describes a typical system architecture with caching proxies, HTTP server,
web servers, T24 server, other application server and a database server:

Page 4
Page 5
Scalability and high availability is achieved by using load balancers and by clustering applications into different
tiers. These tiers are logical divisions of different system components and not necessarily physical divisions of
hardware/software components:

 Web tier
Provides static web content (e.g. HTML) to web clients. The HTTP servers used in this document
are the Apache HTTP Server and IBM HTTP server but other type of server may be used (e.g.
Netscape Enterprise Server, IIS).
 Presentation tier
Provides dynamic web content (e.g. jsp) to web clients. This content is managed by the
application server itself and accessed from the web-tier via a reverse proxy mechanism provided
by the application server’s proxy plug-in.
 Object tier
Provides the business logic and java objects like EJB and JDBC connections.

The following diagram focuses on these three layers and represents the system architecture described
throughout this document:

Fail-over and load-balancing at the web-tier is provided by the reverse proxy plugin and the application server
cluster. The application server cluster must be configured to replicate HTTP session in order to ensure HA. The
application servers hosting the TOCF Listeners and JMS queues used to interface external systems do not

Page 6
necessarily require fail-over servers or to be in a cluster since JMS ensures message delivery even if the system
is temporarily down. However, in case this delivery delay is not acceptable, it will be necessary to configure an
application server cluster providing JMS failover to a redundant backup server. Configuring application server
clusters is specific to each type of application server but all of them described in this document follow the same
principle.

Page 7
Weblogic 10.3.2 High Availability

The following sections only focus on the Web-tier (HTTP server cluster), Presentation-tier (Weblogic cluster for
T24 Browser) and the Weblogic cluster in the Object-tier used to interface T24 to external systems.

Web-tier: Apache HTTP Server 2.2 with WL proxy plug-in

1. Copy mod_wl_22.so to %APACHE_HOME%\modules directory
2. Configure httpd.conf
LoadModule weblogic_module modules/mod_wl_22.so
…
# Weblogic proxy plugin
<IfModule mod_weblogic.c>
WebLogicCluster 10.44.1.72:7002,10.44.1.36:7003
Debug ON
WLLogFile c:/temp/apache_wl_proxy.log
DebugConfigInfo ON
KeepAliveEnabled ON
KeepAliveSecs 15
</IfModule>
<Location /BrowserWeb>
SetHandler weblogic-handler
</Location>

Presentation-tier: Weblogic cluster – T24Brower & T24RA

Machines (Environment  Machines)
 MyMachine1
o Machine OS: Other/UNIX
o Listen Address: 10.44.1.72
o Listen Port: 5556
 MyMachine2
o Machine OS: Other/UNIX
o Listen Address: 10.44.1.36
o Listen Port: 5556

 Check under Monitoring  Node Manager Status that the node manager is reachable.

Clusters (Environment  Clusters)

 MyWebCluster
o Migration Basis: Consensus
o Candidate Machines for migratable services: MyMachine1, MyMachine2

Page 8
Servers (Environment  Servers)
 MyWebServer1
o Machine: MyMachine1
o Cluster: MyWebCluster
o Listen port: 7002
 Server Start
 Arguments: -Xms256m -Xmx512m -XX:CompileThreshold=8000
-XX:PermSize=48m -XX:MaxPermSize=128m
 ClassPath: .;C:\Oracle\Middleware\wlserver_10.3\common\nodemanager\co
nfig\lib\jremote.jar;C:\Oracle\Middleware\wlserver_10.3\common\
nodemanager\config\lib\tocfT24ra.jar;C:\Oracle\Middleware\wlser
ver_10.3\server\lib\weblogic_sp.jar;C:\Oracle\Middleware\wlserve
r_10.3\server\lib\weblogic.jar

 MyWebServer2
o Machine: MyMachine2
o Cluster: MyWebCluster
o Listen port: 7003
 Server Start
 Arguments: -Xms256m -Xmx512m -XX:CompileThreshold=8000
-XX:PermSize=48m -XX:MaxPermSize=128m
 ClassPath: .;C:\Oracle\Middleware\wlserver_10.3\common\nodemanager\co
nfig\lib\jremote.jar;C:\Oracle\Middleware\wlserver_10.3\common\
nodemanager\config\lib\tocfT24ra.jar;C:\Oracle\Middleware\wlser
ver_10.3\server\lib\weblogic_sp.jar;C:\Oracle\Middleware\wlserve
r_10.3\server\lib\weblogic.jar

 Copy jremote.jar and tocfT24ra.jar to the specified location on each machine. I.e. on remote
machines under <WL home>\wlserver_10.3\common\nodemanager\config\lib.

Note: The server logs on remote machine are found under

<WL home>\wlserver_10.3\common\nodemanager\servers\<WL server>\logs

Deployments
 tocfT24ra-ra.rar
o Target: MyWebServer1, MyWebServer2
 Configure and copy the following deployment descriptor into
tocfT24ra-ra.rar!META-INF/weblogic-ra.xml
Weblogic-ra.xml
<?xml version='1.0' encoding='UTF-8'?>
<weblogic-connector xmlns="http://www.bea.com/ns/weblogic/weblogic-connector"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bea.com/ns/weblogic/weblogic-
connector http://www.bea.com/ns/weblogic/weblogic-connector/1.0/weblogic-connector.xsd">
<native-libdir></native-libdir>
<jndi-name>jca/t24ResourceAdapter</jndi-name>
<enable-access-outside-app>false</enable-access-outside-app>
<enable-global-access-to-classes>true</enable-global-access-to-classes>
<security></security>
<outbound-resource-adapter>
<default-connection-properties>
<pool-params></pool-params>
<logging></logging>
</default-connection-properties>

Page 9
 <connection-definition-group>
<connection-factory-interface>com.temenos.tocf.t24ra.T24ConnectionFactory</connection-factory-interface>
<default-connection-properties>
<pool-params></pool-params>
<logging></logging>
</default-connection-properties>
<connection-instance>
<jndi-name>jca/t24ConnectionFactory</jndi-name>
<connection-properties>
<pool-params></pool-params>
<logging></logging>
<properties>
<property>
<name>actionTimeout</name>
<value>90</value>
</property>
<property>
<name>envVariables</name>
<value>OFS_SOURCE=GCS</value>
</property>
<property>
<name>hosts</name>
<value>10.44.1.72,10.44.1.72</value>
</property>
<property>
<name>loadBalancing</name>
<value>true</value>
</property>
<property>
<name>ports</name>
<value>20002,20003</value>
</property>
</properties>
</connection-properties>
</connection-instance>
</connection-definition-group>
</outbound-resource-adapter>
</weblogic-connector>

Weblogic automatically creates this file as part of the deployment plan when deploying the resource
adapter. Adding this file to tocfT24ra-ra.rar simplifies the deployment process, in particular in
configurations where components are copied to different physical servers.

 BrowserWeb.war
o Target: MyWebServer1, MyWebServer2
o Weblogic.xml: Enable In-memory HTTP session replication
<session-descriptor>
<persistent-store-type>replicated</persistent-store-type>
</session-descriptor>

 The default server connection method configured in browserParameters.xml is set to AGENT. This
method triggers T24 Browser to use the T24RA. For backwards compatibility, this property can be set
to AGENT_JREMOTE in order to use the deprecated jRemote RA.

Object-tier: Weblogic cluster – TOCFListeners & DeliveryMDB

Machines (Environment  Machines)

Page 10
  MyMachine3
o Machine OS: Other/UNIX
o Listen Address: 10.44.1.72
o Listen Port: 5556
 MyMachine4
o Machine OS: Other/UNIX
o Listen Address: 10.44.1.36
o Listen Port: 5556

 Check under Monitoring  Node Manager Status that the node manager is reachable.

Clusters (Environment  Clusters)

 MyAppCluster
o Migration Basis:Consensus
o Candidate Machines for migratable services: MyMachine3, MyMachine4

Servers (Environment  Servers)

 MyAppServer1
o Machine: MyMachine3
o Cluster: MyAppCluster
o Listen port: 7004
 Server Start
 Arguments: -Xms256m -Xmx512m -XX:CompileThreshold=8000
-XX:PermSize=48m -XX:MaxPermSize=128m
 ClassPath: .;C:\Oracle\Middleware\wlserver_10.3\common\nodemanager\co
nfig\lib\jremote.jar;C:\Oracle\Middleware\wlserver_10.3\common\
nodemanager\config\lib\tocfT24ra.jar;C:\Oracle\Middleware\wlser
ver_10.3\server\lib\weblogic_sp.jar;C:\Oracle\Middleware\wlserve
r_10.3\server\lib\weblogic.jar

 MyAppServer2
o Machine: MyMachine4
o Cluster: MyAppCluster
o Listen port: 7005
 Server Start
 Arguments: -Xms256m -Xmx512m -XX:CompileThreshold=8000
-XX:PermSize=48m -XX:MaxPermSize=128m
 ClassPath: .;C:\Oracle\Middleware\wlserver_10.3\common\nodemanager\co
nfig\lib\jremote.jar;C:\Oracle\Middleware\wlserver_10.3\common\
nodemanager\config\lib\tocfT24ra.jar;C:\Oracle\Middleware\wlser
ver_10.3\server\lib\weblogic_sp.jar;C:\Oracle\Middleware\wlserve
r_10.3\server\lib\weblogic.jar

 Copy jremote.jar and tocfT24ra.jar to the specified location on each machine.

Persistent stores (Services  Persistent Stores [File Store])

 MyFileStore1
o Target: MyAppServer1
o Directory: <Path to file store directory>
 MyFileStore2

Page 11
 o Target: MyAppServer2
o Directory: <Path to file store directory>

N.B: Use JDBC stores for increased resilience

JMS Servers (Services  Messaging)

 MyJMSServer1
o Target: MyAppServer1
o Persistent Store: MyFileStore1
 MyJMSServer2
o Target: MyAppServer2
o Directory: MyFileStore2

JMS Modules (Services  Messaging)

 MyJMSModule
o Target: MyAppCluster (All servers in the cluster)
o Subdeployments
 MyJMSServerModules
 Target: MyJMSServer1, MyJMSServer2
o T24 JMS Queue Connection Factory
 JNDI: jms/t24QueueCxFactory
 Type: Connection Factory
 Subdeployment: MyJMSServerModules
o T24 OFS Queue
 JNDI: jms/t24OFSQueue
 Type: Distributed Queue
 Subdeployment: MyJMSServerModules
o T24 OFS Reply Queue
 JNDI: jms/t24OFSReplyQueue
 Type: Distributed Queue
 Subdeployment: MyJMSServerModules
o Request Queue
 JNDI: jms/requestQueue
 Type: Distributed Queue
 Subdeployment: MyJMSServerModules
o Response Queue
 JNDI: jms/responseQueue
 Type: Distributed Queue
 Subdeployment: MyJMSServerModules

N.B:
We have to ensure that queue consumers are able to access the response messages. This can be done by either
1. Configuring the reply queues as singleton queues instead of distributed queues
2. Configure MDB to connect to all queue members in the cluster
tocfee.ear!tocfOFSmessage-ejb.jar!weblogic-ejb-jar.xml
<?xml version="1.0"?>
<weblogic-ejb-jar>
<weblogic-enterprise-bean>
<ejb-name>OFSMessageMDB</ejb-name>
<message-driven-descriptor>
<destination-jndi-name>jms/t24OFSQueue</destination-jndi-name>
<distributed-destination-connection>EveryMember</distributed-destination-connection>
</message-driven-descriptor>

Page 12
 <transaction-descriptor>
<trans-timeout-seconds>20</trans-timeout-seconds>
</transaction-descriptor>
<iiop-security-descriptor>
</iiop-security-descriptor>

<resource-description>
<res-ref-name>jca/t24ConnectionFactory</res-ref-name>
<jndi-name>jca/t24ConnectionFactory</jndi-name>
</resource-description>
<resource-description>
<res-ref-name>jms/jmsConnectionFactory</res-ref-name>
<jndi-name>jms/t24QueueCxFactory</jndi-name>
</resource-description>
<resource-description>
<res-ref-name>queue/t24OFSReplyQueue</res-ref-name>
<jndi-name>jms/t24OFSReplyQueue</jndi-name>
</resource-description>
<jndi-name>t24OfsMessageMdb</jndi-name>
</weblogic-enterprise-bean>
</weblogic-ejb-jar>
3. Using another type of alias mechanism to ensure that a response message ends up on the JMS server
which has sent the initial request.

Using distributed reply queues appears to work due to server affinity but this cannot be guaranteed at all times.

Deployments
 tocfT24ra-ra.rar
o Target: MyAppServer1, MyAppServer2
 Configure and copy the following deployment descriptor into
tocfT24ra-ra.rar!META-INF/weblogic-ra.xml
Weblogic-ra.xml
<?xml version='1.0' encoding='UTF-8'?>
<weblogic-connector xmlns="http://xmlns.oracle.com/weblogic/weblogic-connector"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-
connector http://xmlns.oracle.com/weblogic/weblogic-connector/1.0/weblogic-connector.xsd">
<native-libdir></native-libdir>
<jndi-name>jca/t24ResourceAdapter</jndi-name>
<enable-access-outside-app>false</enable-access-outside-app>
<enable-global-access-to-classes>false</enable-global-access-to-classes>
<security></security>
<outbound-resource-adapter>
<default-connection-properties>
<pool-params></pool-params>
<logging></logging>
</default-connection-properties>
<connection-definition-group>
<connection-factory-interface>com.temenos.tocf.t24ra.T24ConnectionFactory</connection-factory-interface>
<default-connection-properties>
<pool-params></pool-params>
<logging></logging>
</default-connection-properties>
<connection-instance>
<jndi-name>jca/t24ConnectionFactory</jndi-name>
<connection-properties>
<pool-params></pool-params>
<logging></logging>
<properties>
<property>
<name>actionTimeout</name>
<value>120</value>
</property>

Page 13
 <property>
<name>envVariables</name>
<value>OFS_SOURCE=GCS</value>
</property>
<property>
<name>hosts</name>
<value>10.44.1.72,10.44.1.72</value>
</property>
<property>
<name>loadBalancing</name>
<value>true</value>
</property>
<property>
<name>ports</name>
<value>20002,20003</value>
</property>
</properties>
</connection-properties>
</connection-instance>
</connection-definition-group>
</outbound-resource-adapter>
</weblogic-connector>

Weblogic automatically creates this file as part of the deployment plan when deploying the resource
adapter. Adding this file to tocfT24ra-ra.rar simplifies the deployment process, in particular in
configurations where components are copied to different physical servers.

 Set the TCP port for the inbound listener in tocfT24ra-ra.rar!META-INF/ra.xml:

<config-property>
<config-property-name>ListenPort</config-property-name>
<config-property-type>java.lang.Integer</config-property-type>
<!-- Inbound listener is disabled by default (port=0). Set to e.g. 55006 to enable inbound
listener -->
<config-property-value>55006</config-property-value>
</config-property>

 tocfee.ear
o Target: MyAppServer1, MyAppServer2

Page 14
Websphere 7.0 High Availability

The following sections focus on the Web-tier (HTTP server cluster), Presentation-tier (Websphere cluster for T24
Browser) and the Websphere cluster in the Object-tier used to interface T24 to external systems.

Web-tier: IBM HTTP Server 7.0 with proxy plug-in

 Install IBM HTTP Server including the Websphere proxy plugins and start the HIS.

Presentation-tier: Websphere cluster – T24Browser & T24RA

Websphere application server clusters (Servers  Clusters)
 MyWebCluster
o Configure HTTP session memory-to-memory replication: true
o First member: MyWebServer1
o Second member: MyWebServer2

Web servers (Servers  Server types)

 T24WebServer
o Type: IBM HTTP Server
o Plugin installation location: <IHS directory>\Plugins
o Application mapping to webserver: All

 Generate and Propagate the plugin to this web server and restart the HTTP server.

NOTE: After propagating the plugins, <HIS directory>\conf\httpd.conf should have the following entries (ensure it
is using the correct web server):
LoadModule was_ap22_module "C:\IBM\HTTPServer1\Plugins\bin\32bits\mod_was_ap22_http.dll"
WebSpherePluginConfig "<HIS directory>\Plugins\config\T24WebServer\plugin-cfg.xml"

Resource Adapters (Resources  Resource Adapters)

 Install the T24 resource adapter (tocfT24ra-ra.rar) using default values
 J2C activation specification
 Set activation specification which is used by the RA to configure a specific endpoint instance
- NAME: T24 Resource Adapter
- JNDI NAME: jca/t24ResourceAdapter

 J2C connection factories

 Define new JCA connection factory
- NAME: T24 JCA Connection Factory
- JNDI NAME: jca/t24ConnectionFactory

 J2C connection factories

o T24 JCA Connection Factory
 Customer properties

Page 15
  Define new JCA connection factory

Websphere enterprise applications (Applications  Application Types)

 Install the T24 Browser (BrowserWeb.war) archive using default values

NOTE:
1. The default server connection method configured in browserParameters.xml is set to AGENT.
This method triggers T24 Browser to use the T24RA. For backwards compatibility, this property
can be set to AGENT_JREMOTE in order to use the former, non-HA connectivity via the jRemote
RA.
2. Context root might need to be changed if already used (to e.g.: /BrowserWeb)

Page 16
JBOSS High Availability

The following sections focus on the Web-tier (HTTP server cluster), Presentation-tier (JBOSS cluster for T24
Browser) and the JBOSS cluster in the Object-tier used to interface T24 to external systems.

Web-tier: Apache HTTP Server 7.0 with proxy plug-in

1. Copy mod_jk.so to %APACHE_HOME%\modules directory
2. Configure %APACHE_HOME%\conf\httpd.conf
…
# JBOSS proxy plugin
Include conf/mod_jk.conf
3. Create %APACHE_HOME%\conf\mod_jk.conf
# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so

# Where to find workers.properties

JkWorkersFile conf/workers.properties

# Where to put jk logs

JkLogFile logs/mod_jk.log

# Set the jk log level [debug/error/info]

JkLogLevel info

# Select the log format

JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# JkRequestLogFormat
JkRequestLogFormat "%w %V %T"

# Mount your applications

JkMount /BrowserWeb/* loadbalancer

# Add shared memory.

# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile logs/jk.shm

# Add jkstatus for managing runtime data

<Location /jkstatus/>
JkMount status
Order deny,allow
Deny from all
Allow from 0.0.0.0
</Location>
4. Create %APACHE_HOME%\conf\workers.properties
# List of workers
worker.list=loadbalancer,status

# Node1 AJP port

worker.MyWebServer1.port=8009
worker.MyWebServer1.host=10.44.1.72
worker.MyWebServer1.type=ajp13
worker.MyWebServer1.lbfactor=1
worker.MyWebServer1.cachesize=10

Page 17
 # Node2 AJP port
worker.MyWebServer2.port=8009
worker.MyWebServer2.host=10.44.1.36
worker.MyWebServer2.type=ajp13
worker.MyWebServer2.lbfactor=1
worker.MyWebServer2.cachesize=10

# Load-balancing
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=MyWebServer1,MyWebServer2
worker.loadbalancer.sticky_session=1
#worker.list=loadbalancer

# Status worker for managing load balancer

worker.status.type=status

Presentation-tier: JBOSS cluster – T24Browser & T24RA

This JBOSS cluster has two members, running on separate JBOSS instances. To start:
run –c <JBOSS configuration> –b 0.0.0.0

e.g.: run –c MyWebServer1 –b 0.0.0.0

JBOSS configurations
 Cluster Member 1 (e.g. 10.44.1.72):
cp –R <jboss instance 1>/server/all <jboss instance 1>/server/MyWebServer1
 Cluster Member 2 (e.g. 10.44.1.36):
cp –R <jboss instance 2>/server/all <jboss instance 2>/server/MyWebServer2

JBOSS session replication

 Configure session replication on every JBOSS instance

E.g. on JBOSS instance 1:

 <jboss instance 1>/server/MyWebServer1/deploy/jboss-web.deployer/server.xml
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3"
emptySessionPath="true" enableLookups="false" redirectPort="8443" />
<Engine name="jboss.web" defaultHost="localhost" jvmRoute="MyWebServer1">
…
</Engine>
 <jboss instance 1>/server/MyWebServer1/deploy/jboss-web.deployer/META-INF/jboss-service.xml
<attribute name="UseJK">true</attribute>

Deployments
 Copy the following deployments into JBOSS_HOME/server/MyWebServer1/farm.
 tocfT24ra-ra.rar
 t24-ds.xml
<?xml version="1.0" encoding="UTF-8"?>
<!-- ================================================================================= -->
<!-- -->
<!-- JBoss Server Configuration -->
<!-- -->
<!-- TOCF T24 Resource Adapter deployment for JBoss, configured for local connection -->
<!-- -->
<!-- ================================================================================= -->
<connection-factories>
<tx-connection-factory>

Page 18
 <jndi-name>jca/t24ConnectionFactory</jndi-name>

<rar-name>tocfT24ra-ra.rar</rar-name>
<connection-definition>com.temenos.tocf.t24ra.T24ConnectionFactory</connection-definition>

<config-property name="hosts" type="java.lang.String">10.44.1.72,10.44.1.72</config-property>

<config-property name="ports" type="java.lang.String">20002,20003</config-property>

<config-property name="loadBalancing" type="java.lang.Boolean">true</config-property>

<config-property name="allowInput" type="java.lang.Boolean">false</config-property>
<config-property name="compression" type="java.lang.Boolean">true</config-property>
<config-property name="compressionThreshold" type="java.lang.Integer">2048</config-property>
<config-property name="envVariables" type="java.lang.String">OFS_SOURCE=GCS</config-property>
<config-property name="actionTimeout" type="java.lang.Integer">120</config-property>
<config-property name="charset" type="java.lang.String">ISO-8859-1</config-property>

<min-pool-size>5</min-pool-size>
<max-pool-size>20</max-pool-size>
<idle-timeout-minutes>15</idle-timeout-minutes>

<!--
<config-property name="debugPort" type="java.lang.Integer">30097</config-property>

<config-property name="SSL" type="java.lang.Boolean">true</config-property>

<config-property name="NaiveTrustManager" type="java.lang.Boolean">true</config-property>

<config-property name="account" type="java.lang.String">MYACCOUNT</config-property>

<config-property name="user" type="java.lang.String">test</config-property>
<config-property name="password" type="java.lang.String">account_password</config-property>
-->
</tx-connection-factory>

</connection-factories>

 BrowserWeb.war

Page 19
Testing

Web & Presentation tiers

This section describes the fail-over scenarios to be considered when testing a highly available system and
presents the results of testing a system in which T24 Browser is accessed by multiple users simultaneously.

Fail-over scenarios
1. HTTP server is down
No fail-over will occur because only a single HTTP server exists in this sample architecture.
Adding additional HTTP servers provide increased availability.
To reproduce: Kill the HTTP server process and trigger T24 Browser to request data.

2. Presentation-tier server goes down

The weblogic proxy plug-in detects the server failure and redirects requests to other servers in
the presentation tier.
To reproduce:
1. Login into T24
2. Kill the presentation-tier Weblogic server process that has received the last HTTP request.
3. Execute a T24 enquiry.
4. Check the Weblogic proxy plug-in log and ensure that request are now sent to the other
server.
Wed Apr 07 21:00:26 2010 <5672127067042693> trying connect to PRIMARY '10.44.1.72'/7002/0
Wed Apr 07 21:00:26 2010 <5672127067042693> getPooledConn: No more connections in the pool for Host[10.44.1.72] Port[7002]
SecurePort[0]
Wed Apr 07 21:00:26 2010 <5672127067042693> INFO: New NON-SSL URL
Wed Apr 07 21:00:26 2010 <5672127067042693> Connect returns -1, and error no set to 10035, msg 'Unknown error'
Wed Apr 07 21:00:26 2010 <5672127067042693> EINPROGRESS in connect() - selecting
Wed Apr 07 21:00:27 2010 <5672127067042693> Error in delayed connection(), error 10061
Wed Apr 07 21:00:27 2010 <5672127067042693> *******Exception type [CONNECTION_REFUSED] (Error connecting to host
10.44.1.72:7002) raised at line 1739 of ../nsapi/URL.cpp
Wed Apr 07 21:00:27 2010 <5672127067042693> Trying a pooled connection for '10.44.1.36/7003/0'
Wed Apr 07 21:00:27 2010 <5672127067042693> getPooledConn: No more connections in the pool for Host[10.44.1.36] Port[7003]
SecurePort[0]
Wed Apr 07 21:00:27 2010 <5672127067042693> general list: trying connect to '10.44.1.36'/7003/0 at line 2635 for
'/BrowserWeb/servlet/BrowserServlet'

3. Presentation-tier server goes down while processing a request

We use in-memory session replication to ensure session data is replicated on other servers in the
presentation tier. The weblogic proxy plug-in will resend HTTP header and post data.
To reproduce:
1. Login into T24
2. Execute a T24 enquiry
3. Kill the presentation-tier Weblogic server process while it is processing the request.
4. Check that the other Weblogic server is now processing the result received from T24 and
that it displays the result of the enquiry.

Killed server:
<07-Apr-2010 22:34:13> <INFO> <NodeManager> <Server failed so attempting to restart (restart count = 1)>

Proxy-plugin:

Page 20
 Wed Apr 07 22:34:06 2010 <56721270676046169> created a new connection to preferred server '10.44.1.72/7002' for
'/BrowserWeb/servlet/BrowserServlet', Local port:4366
Wed Apr 07 22:34:12 2010 <56721270676046169> *******Exception type [READ_ERROR_FROM_SERVER] (socket read failure)
raised at line 207 of ../nsapi/Reader.cpp
Wed Apr 07 22:34:12 2010 <56721270676046169> caught exception in readStatus: READ_ERROR_FROM_SERVER [os
error=10054, line 207 of ../nsapi/Reader.cpp]: socket read failure at line 828
Wed Apr 07 22:34:12 2010 <56721270676046169> PROTOCOL_ERROR: Backend Server not responding - isRecycled:0
Wed Apr 07 22:34:12 2010 <56721270676046169> *******Exception type [PROTOCOL_ERROR] (Backend Server not responding)
raised at line 848 of ../nsapi/URL.cpp
Wed Apr 07 22:34:12 2010 <56721270676046169> Marking 10.44.1.72:7002 as bad
Wed Apr 07 22:34:12 2010 <56721270676046169> got exception in sendRequest phase: PROTOCOL_ERROR [line 848 of
../nsapi/URL.cpp]: Backend Server not responding at line 3062
Wed Apr 07 22:34:12 2010 <56721270676046169> Failing over after sendRequest() exception: PROTOCOL_ERROR as
Idempotent is set to ON
Wed Apr 07 22:34:12 2010 <56721270676046169> attempt #1 out of a max of 5
Wed Apr 07 22:34:12 2010 <56721270676046169> general list: trying connect to '10.44.1.36'/7003/0 at line 2635 for
'/BrowserWeb/servlet/BrowserServlet'

Failover server:
####<07-Apr-2010 22:34:12 o'clock BST> <Info> <Cluster> <ukhml-lgut> <MyWebServer2> <[ACTIVE] ExecuteThread: '1' for
queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1270676052493> <BEA-000113> <Removing
MyWebServer1 jvmid:353823264820287612S:10.44.1.72:[7002,7002,-1,-1,-1,-1,-1]:T24HA_PROD_domain:MyWebServer1 from
cluster view due to PeerGone.>
####<07-Apr-2010 22:34:12 o'clock BST> <Info> <Cluster> <ukhml-lgut> <MyWebServer2> <[ACTIVE] ExecuteThread: '1' for
queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1270676052493> <BEA-000144> <Managed server
MyWebServer1 has been suspended or shutdown.>

4. T24 server is down

The application server will invalidate the broken jAgent connection and use another connection or
ask the T24 resource adapter to open a connection to an available T24 server.
To reproduce:
1. Login into T24
2. Stop one of the jAgent instances.
3. Execute a T24 enquiry
4. Check that all requests are sent to the other jAgent instance.

5. All T24 servers are down

T24 Browser will retry sending the request until it times out. The first retry takes place
immediately, consecutive retries every 2 seconds.
To reproduce:
5. Login into T24
6. Stop all the jAgent instances.
7. Execute a T24 enquiry, wait a few seconds and then restart one of the jAgent instances.
8. Check that T24 Browser retries sending the request and finally display the result of the
enquiry .
2010-04-07 21:48:04,000 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN
com.temenos.t24browser.beans.TemenosBean - Attempting to re-send the request [1. retry] due to connection error:
Unable to connect to remote jAgent instances 10.44.1.72:20002,10.44.1.72:20003.

2010-04-07 21:48:06,000 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN
com.temenos.t24browser.beans.TemenosBean - Attempting to re-send the request [2. retry] due to connection error:
Unable to connect to remote jAgent instances 10.44.1.72:20002,10.44.1.72:20003.

………………………

6. T24 server processing the current request goes down

T24 Browser will retry sending the OFS request until it receives a response from T24 or until it
times out.
N.B:

Page 21
 The T24 RA is XA enabled and is able to propagate transaction contexts to T24. However, T24
should currently not be enlisted into external transactions and for this reason it is possible for T24
to receive a request twice. T24 ensures that transactions are not duplicated. If the first request
has committed a transaction, the response for the retry will result in a “Duplicate OFS Message”
message.
To reproduce:
1. Login into T24
2. Execute a slow T24 enquiry
3. Kill the jAgent connection process that is processing this request.
4. Check that T24 Browser retries sending the request and finally displays the result of the
enquiry .
07-Apr-2010 22:14:13 com.jbase.jremote.io.JRemoteSocketConnection receive
WARNING: Failed to receive message
2010-04-07 22:14:13,972 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN
com.temenos.t24browser.beans.TemenosBean - Attempting to re-send the request [1. retry] due to connection error:
Failed to receive message - Server disconnected
07-Apr-2010 22:14:13 com.jbase.jremote.io.JRemoteSocketConnection receive
WARNING: Failed to receive message

Data injection test results

The system configuration proposed in this document has been tested by means of running JMeter scripts. These
scripts simulate 10 simultaneous T24 users creating a total of 10000 new customer entries.

During data injection, multiple jAgent connections have been killed to ensure the request does get processed.
08-Apr-2010 17:24:38 com.jbase.jremote.io.JRemoteSocketConnection receive
WARNING: Failed to receive message
2010-04-08 17:24:38,387 [[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN
com.temenos.t24browser.beans.TemenosBean - Attempting to re-send the request [1. retry] due to connection error: Failed to
receive message - Server disconnected
08-Apr-2010 17:24:38 com.jbase.jremote.io.JRemoteSocketConnection receive
WARNING: Failed to receive message

Additionally, the MyWebServer1 and MyWebServer2 java processes have been killed at different moments in
time to verify the correct behavior of session replication in case of failover. It was observed that the CPU load on
the failover server increments immediately as expected and the data is resent to T24.

Due to these failovers, a number of “DUPLICATE OFS MESSAGE” response message have been received. As
mentioned in the previous section, these messages can be ignored. The final number of customer entries has
been increase by 10000.

Object tier
The TOCF Listeners have been tested by means of copying a BFL file into the corresponding INPUT directory on
each server and verifying the response file.

The DeliveryMDB has been tested by means of deploying a sample application which consumes a message and
sends a response message back to T24. The application invoking a CALLJEE call must implement the retry
mechanism to ensure high availability. The DeliveryMDB uses transacted JMS session to dispatch the messages,
triggering CALLJEE to return an error if the delivery failed.

Page 22
Resources
Java JDK http://java.sun.com

J2EE Connector Architecture (JCA) Specification http://java.sun.com/j2ee/connector/

Weblogic http://www.oracle.com/appserver/weblogic/weblogic-suite.html

Page 23