Jboss Clustering: High Availability and Load Balancing

Professional Open Source™
JBoss Clustering
High Availability and Load Balancing
© JBoss, Inc. 2003-2005. 8/2/2005 1
Clustering Topics
– Features
– How to use it
– Design overview
– Future
© JBoss, Inc. 2003-2005. 2
1
Load Balancing != Fault Tolerance
Load balancing is used for scalability, not for fault tolerance
© JBoss, Inc. 2003-2005. 3
Load Balancing != Fault Tolerance

Failover and state replication is used for fault tolerance
State Replication
Why do you want to use clustering?
© JBoss, Inc. 2003-2005. 4
2
Clustering Has a Cost
Clustering always has a cost

– fail-over with state state replication serialization
What about the cache?

– In a cluster, you never have an exclusive access to the DB
Raw Performance != Scalability

– (1 + 1) != 2
© JBoss, Inc. 2003-2005. 5
Cluster Management: Installation

To install a cluster you can either:

– Manually install & configure each node separately
– Bootstrap all nodes from a central location NetBoot
NetBoot
– Microkernel installed on every machine
– JBoss configuration & applications on a central web server
– Supply URL to micro-kernel
• JBoss boots up from micro-kernel
© JBoss, Inc. 2003-2005. 6
3
Cluster Manager: Net Boot
NetBoot Client NetBoot Client
NetBoot Client
NetBoot Client
NetBoot Server
Configuration
Files
Deployment
Packages
Web Server
Administrator
© JBoss, Inc. 2003-2005. 7
Cluster Management: Deployment

Remember JBoss’ deployment easiness?

– Simply copy a file in /deploy
The farm service brings this simplicity to clustering

– We see the cluster as farm of nodes Farm Service
Farm Service Features:

– Hot-deploy across cluster
– Undeploy across cluster
– 24x7 availability: simple redeployment across cluster
© JBoss, Inc. 2003-2005. 8
4
Cluster Manager: Farming
Package Replication
Deploy Packages
on Single Cluster Node
© JBoss, Inc. 2003-2005. 9
Cluster Management: Auto Discovery

How do you “build” a cluster?

– A cluster is not statically defined ( e.g. C1 = {N1, N2, N3} )
– A cluster is identified by a name (Partition Name)
– All nodes that assert to belong to a given partition name dynamically
extend the cluster with this name
• Dynamic composition of a cluster
Behaviour
– On boot up, new cluster node detects existing group members
– Group members are notified on node start/stop/failures
– No configuration of topology needed.
© JBoss, Inc. 2003-2005. 10
5
Cluster Fault Tolerance – Server Side
What does “fail-over” mean on the server?
Component state is always the issue!

– Replicated state for stateful session beans
– Replicated HTTP sessions
– No replicated state for entity beans
• Must use commit-option ‘B’ or ‘C’ or cache invalidation with commit-
option ‘A’ (for non-critical read-mostly data)
– Global cluster-wide, replicated JNDI tree (HA-JNDI)
© JBoss, Inc. 2003-2005. 11
Cluster Fault Tolerance – Client Side

Java Client (RMI)

– Use of RMI smart proxies
– Transparent failover of client proxies: All EJB types, RMI, JNDI
HTTP client (browser)

– You need a “tool” to failover sessions
– Hardware solutions
• Cisco Content Switch, F5 BigIP, Alteon, etc.
– Software solutions
• Apache Mod_JK 1.2.x
© JBoss, Inc. 2003-2005. 12
6
Cluster Load Balancing
Load balancing for all RMI proxy types:

– EJBs, RMI, and JNDI
– Pluggable load balance policies
• Round Robin, First Available, Random
• You can write your own
• SFSB, Entities: First Available (sticky)
• SLSB – Round Robin (no state involved)
• All Home invocations: Round Robin
HTTP LoadBalancing
– Extension of Failover mechanism
• Hardware solution: Cisco, etc.
• Software solutions: Apache, etc.
© JBoss, Inc. 2003-2005. 13
What to Cluster?
Which components do you really need to cluster?
Apache
Apache
AJP13
AJP13
HTTP
HTTP Session
Session HTTP
HTTP Session
Session
SLSB
SLSB SLSB
SLSB
EB
EB EB
EB
Everything should be homogeneously deployed on all nodes

– Optimize local calls
Why would you cluster the SLSB and EB?
– If they fail, the whole node will fail anyway (JVM process failure, operating
system failure or hardware failure)
© JBoss, Inc. 2003-2005. 14
7
Specific Optimisations – SFSB
With SFSB, the bean state must be replicated to other nodes after
each invocation
– Replication requires serialization which has a high cost
Was the state really modified?

– This decision cannot be made by the container
• By default, state must always be replicated
– The developer can help the container
• The developer will tell the container if the bean has really been
modified
public boolean isModified();
• If defined on the bean implementation class, this method will be called

before the replication. If the bean state has not been modified, not
replication occurs.
© JBoss, Inc. 2003-2005. 15
The Big Picture – HTTP

Net
Net Boot
Boot
Apache
Apache Load
Load Balancing
Balancing
AJP13
AJP13 HTTP
HTTP Failover
Failover
Replicated
Replicated HTTP
HTTP Session
Session
Global
Global JNDI
JNDI Context
Context
Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat
EJB
EJB EJB
EJB EJB
EJB
Database
Database Commit
Commit Option
Option ‘B’/’C’
‘B’/’C’
© JBoss, Inc. 2003-2005. 16
8
Using Clustering – Basic Steps
Use JBoss’ “all” configuration

– run.bat -c all
– ./run.sh -c all
The “all” configuration contains everything needed for clustering:

– Clustered beans
– HA-JNDI
– HTTP session replications
– Farming
– HA-JMS
© JBoss, Inc. 2003-2005. 17
Using Clustering
deploy/cluster-service.xml
<?xml
<?xmlversion="1.0"
version="1.0"encoding="UTF-8"?>
encoding="UTF-8"?>
<server>
<server>
<classpath
<classpatharchives="jbossha.jar"/>
archives="jbossha.jar"/>
<depends>JBOSS-SYSTEM:service=Naming</depends>
<depends>JBOSS-SYSTEM:service=Naming</depends>
<mbean
<mbeancode="org.jboss.ha.framework.server.ClusterPartition”
code="org.jboss.ha.framework.server.ClusterPartition”
name="JBOSS-SYSTEM:service=DefaultPartition"/>
name="JBOSS-SYSTEM:service=DefaultPartition"/>
<mbean
<mbeancode="org.jboss.ha.jndi.HANamingService"
code="org.jboss.ha.jndi.HANamingService"name="JBOSS-SYSTEM:service=HAJNDI">
name="JBOSS-SYSTEM:service=HAJNDI">
<attribute
<attributename="Port">1100</attribute>
name="Port">1100</attribute>
</mbean>
</mbean>
<mbean
<mbeancode="org.jboss.ha.framework.server.FarmMemberService“
code="org.jboss.ha.framework.server.FarmMemberService“
name="jboss:service=FarmMember,partition=DefaultPartition"
name="jboss:service=FarmMember,partition=DefaultPartition">>
<depends>jboss:service=DefaultPartition</depends>
<depends>jboss:service=DefaultPartition</depends>
<attribute
<attributename="PartitionName">DefaultPartition</attribute>
name="PartitionName">DefaultPartition</attribute>
<attribute
<attributename="FarmDeployDirectory">./farm</attribute>
name="FarmDeployDirectory">./farm</attribute>
<attribute
<attributename="ScannerName">jboss.deployment:type=DeploymentScanner,flavor=URL</attribute>
name="ScannerName">jboss.deployment:type=DeploymentScanner,flavor=URL</attribute>
</mbean>
</mbean>
</server>
</server>
© JBoss, Inc. 2003-2005. 18
9
Using Clustering – Protocol Stack
<attribute
<attributename="PartitionConfig">
name="PartitionConfig">
<Config>
<Config>

-->

-->
<UDP
<UDPmcast_addr="228.1.2.3"
mcast_addr="228.1.2.3"bind_addr="192.168.0.20"
bind_addr="192.168.0.20"mcast_port="45566"
mcast_port="45566"
ip_ttl=”32"
ip_ttl=”32"ip_mcast="true"
ip_mcast="true"
mcast_send_buf_size="150000"
mcast_send_buf_size="150000"mcast_recv_buf_size="80000"
mcast_recv_buf_size="80000"
ucast_send_buf_size="150000"
ucast_send_buf_size="150000"ucast_recv_buf_size="80000"
ucast_recv_buf_size="80000"loopback="false"
loopback="false"/>/>
<PING
<PINGtimeout="2000"
timeout="2000"num_initial_members="3"
num_initial_members="3" up_thread="true"
up_thread="true"down_thread="true"
down_thread="true"/>/>
<MERGE2
<MERGE2min_interval="5000"
min_interval="5000"max_interval="10000"
max_interval="10000"/>/>
<FD
<FDshun="true"
shun="true"up_thread="true"
down_thread="true"timeout="2500"
timeout="2500"max_tries="5"
max_tries="5"/>/>
<VERIFY_SUSPECT
<VERIFY_SUSPECTtimeout="3000"
timeout="3000"num_msgs="3”
num_msgs="3”up_thread="true"
<pbcast.NAKACK
<pbcast.NAKACKgc_lag="50"
gc_lag="50"retransmit_timeout="300,600,1200,2400,4800"
retransmit_timeout="300,600,1200,2400,4800"
max_xmit_size="8192"
max_xmit_size="8192"use_mcast_xmit="true"
use_mcast_xmit="true"up_thread="true"
<UNICAST
<UNICASTtimeout=”600,1200,2400"
timeout=”600,1200,2400"window_size="100"
window_size="100"min_threshold="10"
min_threshold="10"down_thread="true"
<pbcast.STABLE
<pbcast.STABLEdesired_avg_gossip="20000”
desired_avg_gossip="20000”up_thread="true"
<FRAG
<FRAGfrag_size="8192"
frag_size="8192"down_thread="true"
down_thread="true"up_thread="true"
up_thread="true"/>/>
<pbcast.GMS
<pbcast.GMSjoin_timeout="5000"
join_timeout="5000"join_retry_timeout="2000"
join_retry_timeout="2000"shun="true"
shun="true"print_local_addr="true"
print_local_addr="true"/>/>
<pbcast.STATE_TRANSFER
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>/>
up_thread="true" down_thread="true"
</Config>
</Config>
</attribute>
</attribute>
© JBoss, Inc. 2003-2005. 19
JBossCluster – Using It…

EJB Configuration
jboss.xml
jboss.xml
<jboss>
<jboss>
<secure>false</secure>
<secure>false</secure>
<enterprise-beans>
<enterprise-beans>
<session>
<session>
<ejb-name>SLSB</ejb-name>
<ejb-name>SLSB</ejb-name>
<jndi-name>SLSB</jndi-name>
<jndi-name>SLSB</jndi-name>
<clustered>True</clustered>
</session>
</session>
</enterprise-beans>
</enterprise-beans>
</jboss>
</jboss>
© JBoss, Inc. 2003-2005. 20
10
JBossCluster – Using It…
EJB Configuration
jboss.xml
jboss.xml
<enterprise-beans>
<enterprise-beans>
<session>
<session>
<ejb-name>SFSB</ejb-name>
<ejb-name>SFSB</ejb-name>
<jndi-name>SFSB</jndi-name>
<jndi-name>SFSB</jndi-name>
<cluster-config>
<cluster-config>
<partition-name>MyPartition</partition-name>
<partition-name>MyPartition</partition-name>
<home-load-balance-policy>
<home-load-balance-policy>
org.jboss.ha.framework.interfaces.RoundRobin
org.jboss.ha.framework.interfaces.RoundRobin
</home-load-balance-policy>
</home-load-balance-policy>
<bean-load-balance-policy>
<bean-load-balance-policy>
org.jboss.ha.framework.interfaces.FirstAvailable
org.jboss.ha.framework.interfaces.FirstAvailable
</bean-load-balance-policy>
</bean-load-balance-policy>
</cluster-config>
</cluster-config>
</session>
</session>
© JBoss, Inc. 2003-2005. 21
HTTPSession Replication
How to activate HTTP session clustering

– Set your application as distributable in WEB-INF/web.xml
<distributable/>
HTTP Session Replication Strategy: WEB-INF/jboss-web.xml
<jboss-web>
<jboss-web>
<replication-config>
<replication-config>
<replication-trigger>SET</replication-trigger>
<replication-trigger>SET</replication-trigger>
<replication-type>ASYNCH</replication-type>
<replication-type>ASYNCH</replication-type>
</replication-config>
</replication-config>
</jboss-web>
</jboss-web>
© JBoss, Inc. 2003-2005. 22
11
HTTP Session Replication
Replication Trigger
– SET_AND_GET
– SET_AND_NON_PRIMITIVE_GET
– SET
Replication Type
– SYNCH – synchronous
– ASYNCH - asynchronous
© JBoss, Inc. 2003-2005. 23
JBossCluster Uses Client Side Proxy

HAInvokerProxy
– Proxy behavior is enhanced to deal with clustered invokers
– Interceptor contains list of available nodes
– Interceptor chooses server to invoke on through load-balance policy
List
List of
of
nodes
nodes
HA
RMI JMX
Cluster
Client
Container EJB Tx Security Client
Interceptor HA
RMI JMX
© JBoss, Inc. 2003-2005. 24
12
JBossCluster Uses Client Side Proxy
HARMIResponse
– Client-side cluster interceptor maintains a list of nodes the MBean (EJB,
JNDI, etc…) is available.
– If cluster topology changes, new list is piggy-backed with invocation
response to client.
– HA invoker manages this.
List
List of
of nodes
nodes
HA
RMI JMX
Cluster
Client
Container EJB Tx Security Client
Interceptor HA
RMI JMX
© JBoss, Inc. 2003-2005. 25
Client Side Proxy for Clustering

package
package org.jboss.invocation.jrmp.interfaces;
org.jboss.invocation.jrmp.interfaces;
/**
/**
** JRMPInvokerProxy,
JRMPInvokerProxy, local local toto the
the proxy
proxyandand is is capable
capable of
ofdelegating
delegating to
to local
local and
and JRMP
JRMP
implementations
implementations
*/*/
public
public classclassJRMPInvokerProxyHA
JRMPInvokerProxyHAextends extends JRMPInvokerProxy
JRMPInvokerProxyimplements
implements Externalizable
Externalizable
{{
//// Public --------------------------------------------------------
Public --------------------------------------------------------
protected
protected ArrayList
ArrayListtargets
targets ==null;
null;
protected
protected LoadBalancePolicy
LoadBalancePolicyloadBalancePolicy;
loadBalancePolicy; The
The HA
HA version
version
contains
contains aa list
list of
of
protected
protected transient
transientlong
long currentViewId
currentViewId ==0;
0; invoker
invoker proxies
proxies
public
public static
static final
final HashSet
HashSetcolocation
colocation == new
new HashSet();
HashSet();
© JBoss, Inc. 2003-2005. 26
13
public
publicObject
ObjectgetRemoteTarget()
getRemoteTarget(){{
ifif(targets.size()
(targets.size()==
==0)
0)
return
returnnull;
null;
synchronized
synchronized(targets)
(targets)
Pluggable
Pluggable load
load
return balance
balance policies
policies are
are
returnloadBalancePolicy.chooseTarget(targets);
loadBalancePolicy.chooseTarget(targets);
}} responsible
responsible forfor
picking
picking the
the target
target
/**
/**
**Returns
Returnswether
wetherweweare
arelocal
localto
tothe
theoriginating
originatingcontainer
containeror
ornot.
not.
*/*/
public
publicboolean
booleanisLocal(Invocation
isLocal(Invocationinvocation)
invocation) {{
return
returncolocation.contains(invocation.getObjectName());
colocation.contains(invocation.getObjectName());
}}
© JBoss, Inc. 2003-2005. 27

public
publicObject
Objectinvoke(Invocation
invoke(Invocationinvocation)
invocation)throws throwsException
Exception
{{
////we
wegive
givethe
theopportunity,
opportunity,to
toany
anyserver
serverinterceptor,
interceptor,to
toknow
knowififthis
thisaa
////first
firstinvocation
invocationtotoaanode
nodeor
orififititis
isaafailovered
failoveredcall
call
int
intfailoverCounter
failoverCounter==0; 0;
invocation.setValue
invocation.setValue("FAILOVER_COUNTER",
("FAILOVER_COUNTER",new newInteger(failoverCounter),
Integer(failoverCounter),invocation.AS_IS);
invocation.AS_IS);
////optimize
optimizeififcalling
callinganother
anotherbean
beanin
insame
sameEJB-application
EJB-application
ifif(isLocal(invocation))
(isLocal(invocation)){{
return
returnInvokerInterceptor.getLocal().invoke(invocation); We
InvokerInterceptor.getLocal().invoke(invocation); We still
still collocate
collocate ifif we
we are
are
}} local
local
else
else
{{
////We
Weare
aregoing
goingto togo
gothrough
throughaaRemote
Remoteinvocation,
invocation,switch
switchto
toaaMarshalled
MarshalledInvocation
Invocation
MarshalledInvocation
MarshalledInvocationmi mi==new
newMarshalledInvocation(invocation);
MarshalledInvocation(invocation);
////Set
Setthe
thetransaction
transactionpropagation
propagationcontext
context
mi.setTransactionPropagationContext(getTransactionPropagationContext());
mi.setTransactionPropagationContext(getTransactionPropagationContext());
mi.setValue("CLUSTER_VIEW_ID",
mi.setValue("CLUSTER_VIEW_ID",new newLong(currentViewId));
Long(currentViewId));
Invoker
Invokertarget
target==(Invoker)getRemoteTarget();
(Invoker)getRemoteTarget();
© JBoss, Inc. 2003-2005. 28
14
while
while(target
(target!= !=null)
null){{
try
try{{
HARMIResponse
HARMIResponsersp rsp==(HARMIResponse)((MarshalledObject)target.invoke(mi)).get();
(HARMIResponse)((MarshalledObject)target.invoke(mi)).get();
ifif(rsp.newReplicants
(rsp.newReplicants!= !=null)
null){{
setTargets(rsp.newReplicants);
setTargets(rsp.newReplicants);
currentViewId
currentViewId==rsp.currentViewId;
rsp.currentViewId; Server
Server piggy-backs
piggy-backs
}} information
information about
about the
the
return rsp.response;
return rsp.response; topology
topology
}}
////ALL
ALLCATCH
CATCHJUST JUSTGOES
GOESON ON
catch
catch(java.rmi.ConnectException
(java.rmi.ConnectExceptionce){}ce){}
////IfIfwe
We fail over here by taking
We fail over here by taking
wereach
reachhere,
here,this
thismeans
meansthat
thatwe
wemust
mustfail-over
fail-over aa new
remoteTargetHasFailed(target); new target
target and
and looping
looping on
on
remoteTargetHasFailed(target);
target the
the while
while (target
(target !=null)
!=null)
target==(Invoker)getRemoteTarget();
(Invoker)getRemoteTarget();
failoverCounter++;
failoverCounter++;
mi.setValue
mi.setValue("FAILOVER_COUNTER",
("FAILOVER_COUNTER",new newInteger(failoverCounter),
Integer(failoverCounter),invocation.AS_IS);
invocation.AS_IS);
}}
////ififwe
weget
gethere
herethis
thismeans
meanslist
listwas
wasexhausted
exhausted
throw
thrownew newjava.rmi.RemoteException("Service
java.rmi.RemoteException("Serviceunavailable.");
unavailable.");
}}
}}
© JBoss, Inc. 2003-2005. 29
Conclusion
Understand why you want to cluster:

– Fail-over?
– Load-balancing?
State replication is expensive
– How often do you want to send the state across nodes?
HTTP load balancing is based on:
– Hardware
– Mod_jk 1.2
Stand-alone client load-balancing is easy because of the smart
proxies
© JBoss, Inc. 2003-2005. 30
15

Jboss Clustering: High Availability and Load Balancing

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Jboss Clustering: High Availability and Load Balancing

Uploaded by

Copyright:

Available Formats

Professional Open Source™

© JBoss, Inc. 2003-2005. 8/2/2005 1

© JBoss, Inc. 2003-2005. 2

Load balancing is used for scalability, not for fault tolerance

© JBoss, Inc. 2003-2005. 3

Load Balancing != Fault Tolerance

Failover and state replication is used for fault tolerance

Why do you want to use clustering?

© JBoss, Inc. 2003-2005. 4

Clustering always has a cost

What about the cache?

Raw Performance != Scalability

© JBoss, Inc. 2003-2005. 5

Cluster Management: Installation

To install a cluster you can either:

© JBoss, Inc. 2003-2005. 6

NetBoot Client NetBoot Client

© JBoss, Inc. 2003-2005. 7

Cluster Management: Deployment

Remember JBoss’ deployment easiness?

The farm service brings this simplicity to clustering

Farm Service Features:

© JBoss, Inc. 2003-2005. 8

© JBoss, Inc. 2003-2005. 9

Cluster Management: Auto Discovery

How do you “build” a cluster?

© JBoss, Inc. 2003-2005. 10

What does “fail-over” mean on the server?

Component state is always the issue!

© JBoss, Inc. 2003-2005. 11

Cluster Fault Tolerance – Client Side

Java Client (RMI)

HTTP client (browser)

© JBoss, Inc. 2003-2005. 12

Load balancing for all RMI proxy types:

© JBoss, Inc. 2003-2005. 13

Which components do you really need to cluster?

Everything should be homogeneously deployed on all nodes

© JBoss, Inc. 2003-2005. 14

Was the state really modified?

public boolean isModified();

• If defined on the bean implementation class, this method will be called

© JBoss, Inc. 2003-2005. 15

The Big Picture – HTTP

© JBoss, Inc. 2003-2005. 16

Use JBoss’ “all” configuration

The “all” configuration contains everything needed for clustering:

© JBoss, Inc. 2003-2005. 17

© JBoss, Inc. 2003-2005. 18

© JBoss, Inc. 2003-2005. 19

JBossCluster – Using It…

© JBoss, Inc. 2003-2005. 20

© JBoss, Inc. 2003-2005. 21

How to activate HTTP session clustering

HTTP Session Replication Strategy: WEB-INF/jboss-web.xml

© JBoss, Inc. 2003-2005. 22

© JBoss, Inc. 2003-2005. 23

JBossCluster Uses Client Side Proxy

© JBoss, Inc. 2003-2005. 24

© JBoss, Inc. 2003-2005. 25

Client Side Proxy for Clustering

© JBoss, Inc. 2003-2005. 26

© JBoss, Inc. 2003-2005. 27