You are on page 1of 15

Cluster Control Protocol Reference

NG FP3

For additional technical information about Check Point products, consult Check Point’s
SecureKnowledge database at
http://support.checkpoint.com/kb/

Overview The introduction of ClusterXL into an existing network. This document is not meant as an installation guide. the enclosed sections will explore the following topics: ■ ■ ■ Implementation Planning Cluster Control Protocol Overview Cluster Control Protocol Logic . often implies that certain changes to that network be made. Although parts of the Cluster Control Protocol are also used by OPSEC High Availability products. adequate comprehension of the ClusterXL decision making process is needed. this aspect will not be covered. as well as the Cluster Control Protocol itself. Such alterations are needed to accommodate both the clustered topology. there will be no context in which to place observed cluster behavior. implementation. Therefore. Moreover.Preface Introduction This document explores various technical aspects of the Cluster Control Protocol as utilized by ClusterXL. Understanding these changes is important for the purposes of planning. and troubleshooting. monitoring. Otherwise. and assumes the reader has a working knowledge of the ClusterXL product.

In keeping with multicast standards. A layer two switch connected to non-secured interfaces. a separate VLAN. and model. A HotFix is also available from Check Point support which allows this configuration to be supported in FP3. If the connecting switch is incapable of forwarding multicast. it offers all the topology advantages of Load Sharing. there are additional concerns with this configuration which make it currently unsupportable. Legacy HA will not be covered in the following sections. It is acceptable that the switch forward such traffic to all ports within the given VLAN. Doing so will cause the connecting switch ports to flap. However. and Load Sharing configurations. . this multicast address is used only as the destination. must be capable of forwarding multicast packets to ports within that VLAN. makes use of layer two multicast. or to an isolated VLAN. and is used in all CCP packets sent on "non-secured" interfaces.Implementation Planning NOTE: Due to the enhanced Hot/Standby configuration available in FP3. While the above mentioned HotFix may be used. If such a need exists. Simply referred to as "New Mode". The steps needed to enable multicast support will vary according to the switch vendor. To toggle between these two modes use the command (mode survives a reboot): 'cphaconf set_ccp broadcast/multicast' VLAN Configuration It is not recommended to connect the non-secured interfaces of multiple clusters to the same VLAN. Please check your switch documentation for details. Therefore. it is considered more efficient to forward to only those ports connecting cluster members. High Availability (New Mode) FP3 introduces a new form of operation for High Availability. it is best to connect the secured interfaces of a given cluster via a crossover link when possible. CCP can be changed to use broadcast instead. Connecting together the secured interfaces of multiple clusters is also not recommended for the same reason. and/or switch will be needed for each cluster. while maintaining a Hot/Standby orientation. Important factors to consider while planning for a New Mode implementation are: ■ ■ ■ ■ Switch support/configuration for layer two multicast forwarding VLAN configuration IP address migration SmartCenter/CMA location Switch Support The Cluster Control Protocol used by both New Mode.

Therefore. as the SmartCenter/CMA Server can now reside on any given IP segment. or cluster addresses when feasible. If not. In the latter case. One only needs to ensure that the general tab IP address of the cluster member object is reachable. This design affords a great level of flexibility when using either New Mode. and IPSec connections will need to be altered to accommodate the new clustered landscape.IP Address Migration It is reasonable to assume that many ClusterXL installs will be either for a new VPN-1/FireWall-1 cluster. existing NAT. or to replace a different clustering solution. However. It does so by installing the Security Policy to each cluster member. simply choose one which will be accessible to the SmartCenter/CMA Server. it is recommended to take the existing IP addresses from the current gateway. as well keep Hide NAT configurations the same in many cases. SmartCenter/CMA Location A SmartCenter/CMA Server. This is true regardless of the IP address(es) of the cluster object itself. or Load Sharing configurations. . Doing so will avoid altering current IPSec endpoint identities. it is also reasonable to assume that many will be to provide high availability to an existing single gateway configuration. can install a Security Policy to one or more clusters with only a single install action. and make these the cluster owned VIP's. using the general tab IP of each cluster member object as the recipient.

If the connecting switch is incapable of forwarding multicast. VLAN Configuration It is not recommended to connect the non-secured interfaces of multiple clusters to the same VLAN.Load Sharing Important factors to consider while planning for a Load Sharing implementation are: ■ ■ ■ ■ ■ Switch support/configuration for layer two multicast forwarding Router support for multicast VLAN configuration IP address migration SmartCenter/CMA location Switch Support The Cluster Control Protocol used by both New Mode. and some Nortel models (Passport 1200 or XLR) which will not accept this type of static ARP entry. Doing so will cause the connecting switch ports to flap. Please check your switch documentation for details. ARP replies sent by a cluster member will indicate that the unicast cluster IP. and is used in all CCP packets sent on "non-secured" interfaces. adding a static ARP entry for the cluster IP on the routing device will solve the issue. While the above mentioned HotFix may be used. The steps needed to enable multicast support will vary according to the switch vendor. CCP can be changed to use broadcast instead. Even still there are some routers such as Extreme routers. is reachable via a multicast MAC. but without the need of multicast for the cluster addresses. ClusterXL FP4 will introduce a new mode of operation referred to as Pivot mode. It is acceptable that the switch forward such traffic to all ports within the given VLAN. and Load Sharing configurations. Pivot mode operates as a Load Sharing cluster. must be capable of forwarding multicast packets to ports within that VLAN. Connecting together the secured interfaces of multiple clusters is also not recommended for the same reason. If such a need exists. . this multicast address is used only as the destination. Load Sharing associates a multicast MAC for each configured cluster IP. A HotFix is also available from Check Point support which allows this configuration to be supported. it is best to connect the secured interfaces of a given cluster via a crossover link when possible. For instance. A layer two switch connected to non-secured interfaces. Avia routers. For those cases. However. Therefore. To toggle between these two modes use the command (mode survives a reboot): 'cphaconf set_ccp broadcast/multicast' Router Support In addition to the use of multicast by CCP. and model. a separate VLAN. all versions of Cisco IOS do not include such support. In keeping with multicast standards. Therefore. Some routing devices are incapable of receiving such ARP replies. makes use of layer two multicast. In such cases. or to an isolated VLAN. it is considered more efficient to forward to only those ports connecting cluster members. there are additional concerns with this configuration which make it currently unsupportable. and/or switch will be needed for each cluster. This design ensures that traffic destined to the cluster is received by all members.

It does so by installing the Security Policy to each cluster member. can install a Security Policy to one or more clusters with only a single install action. as well keep Hide NAT configurations the same in many cases. or cluster addresses when feasible. using the general tab IP of each cluster member object as the recipient. or Load Sharing configurations. This design affords a great level of flexibility when using either New Mode. simply choose one which will be accessible to the SmartCenter/CMA Server. Doing so will avoid altering current IPSec endpoint identities. existing NAT.IP Address Migration It is reasonable to assume that many ClusterXL installs will be either for a new VPN-1/FireWall-1 cluster. or to replace a different clustering solution. . it is also reasonable to assume that many will be to provide high availability to an existing single gateway configuration. Therefore. it is recommended to take the existing IP addresses from the current gateway. as the SmartCenter/CMA Server can reside on any given IP segment. However. This is true regardless of the IP address(es) of the cluster object itself. If not. and IPSec connections will need to be altered to accommodate the new clustered landscape. SmartCenter/CMA Location A SmartCenter/CMA Server. and make these the cluster owned VIP's. One only needs to ensure that the general tab IP address of the cluster member object is reachable. In the ladder case.

CCP will probe that segment in an attempt to illicit a response. These reports contain state of the transmitting cluster member. per interface. connection information is updated between cluster members on the defined secured interface. The outcome of this probe will determine what action is taken next. Specifically.Cluster Control Protocol CCP Overview The Cluster Control Protocol serves an integral role to the operation of ClusterXL. such as with a reboot. Cluster Member Probing If a cluster member fails to receive status for another member on a given segment. Querying Cluster Membership When a cluster member comes online. it will send as series of CCP query/response messages to gain knowledge of it's cluster membership. State Table Synchronization When state synchronization is enabled. . CCP is responsible for the following: ■ ■ ■ ■ ■ Health status reports Cluster member probing State change commands Querying for cluster membership Sate table synchronization Health Status Reports CCP will report the status of a cluster member roughly three times a second. the command to do so takes place on the defined secured interface. The purpose of such probes is to detect the nature of possible interface failures. as well as the presumed state of other cluster members. and to determine which module has the problem. State Change Commands If a cluster member wishes to change state.

Policy ID change request/notification FWHAP_SYNC .Interface active check request FWHA_IF_PROBE_RPLY . the most important of which is: ■ ■ ■ ■ ■ Cluster ID .Report source machine's state FWHA_Query_STATE .member identification according to configured priority.Interface active check reply FWHA_IFCONF_REQ . General Heading This portion contains information necessary for the processing of the encapsulated message type. and content.Query other machine's state FWHA_IF_PROBE_REQ .version and Feature Pack revision Source Interface .CCP Message Format The Cluster Control Protocol payload is made up a general heading. with each having it's own unique purpose.unique identifier shared amongst all members of a given cluster Protocol Version .New Sync packet .Interface configuration request FWHA_IFCONF_REPLY .Last two bytes of MD4 Policy ID Message Types Below is a complete listing of the possible CCP message types with description: ■ ■ ■ ■ ■ ■ ■ ■ FWHA_MY_STATE . Calculated as priority -1=ID Policy ID . format. and one of a series of message types.Interface configuration reply FWHA_POLICY_CHANGE .transmitting interface number as recognized by the OS kernel Source Machine ID .

The last octet of the source address 0:0:0:0:fe:0. both the destination MAC. CCP packets sent by the highest priority machine will look like this: 0:0:0:0:fe:0 1:0:5e:3:dc:67 ip 78: 0. In case of a second. CCP transmits by default as follows: ■ ■ ■ ■ Source MAC .0.ff:ff:ff:ff:ff:ff (all hosts broadcast) Destination IP . lets assume a scenario in which New Mode is being used.00:00:00:00:fe:<Source Machine ID> Source IP .0 Destination MAC . CCP transmits it's packets by default with layer two multicast.01:00:5e:<cluster IP concatenation of bits 9-24> Destination IP .0.0.0 > 10.3. this source address will reflect the members priority such as: 0:0:0:0:fe:1 0:0:0:0:fe:2 Using this design. the cluster IP is 10.network broadcast address As an example.0.220. with the rest corresponding to the last three octets of the cluster address.00:00:00:00:fe:<Source Machine ID> Source IP . in this case this primary.103/27. 1:0:5e:3:dc:67 corresponds to the destination MAC. and destination IP address will change per IP segment.0.220. indicates the Machine ID of the transmitting member.CCP Transmission Non-Secured Interfaces For interfaces not defined as secured (non synchronization interfaces).3. The addressable fields are as follows: ■ ■ ■ ■ Source MAC . This is true irrespective of the interface type. with both the source and destination port set to 8116. On a given segment.0.0. according to both the cluster.96 Here.0.network broadcast address Port usage CCP uses UDP as the transmission protocol.0 Destination MAC . . and indicates that the OID is multicast (1:0:5e:). and network address. or third member. Secured Interfaces For interfaces defined as secured (synchronization interfaces).

Sync (Secured) network .220.8 .96 .3.168.250.96 .ClusterXL cluster P1_Primary .External segment Net_10.1 will be assumed. Note: The following examples are given in general terms. We will do so by looking at several common failures. and how ClusterXL responds to such scenarios.Corporate network Net_192.2.ClusterXL Decision Logic Topology The following section will explore the logic utilized by ClusterXL.Managing CMA Net_10. A separate section will be dedicated to both New Mode High Availability.96 .1 Topology Legend ■ ■ ■ ■ ■ ■ Dallab_Cluster . Figure 1.Admin network Net_10.1.220.220. and do not represent a per packet analysis. and to Load Sharing configurations. The topology represented by Figure 1.

This causes the primary to then announce on all other segments via FWHA_MY_STATE. the secondary concludes that it's own interfaces are up and working. 4. 1. With this report from the secondary. 2. The primary will also note that no CCP responses have been received on the failed interface. the primary concludes the issue is with it's own interface. and N-1 of them are down. 6. in which the external interface of the primary has failed. it announces via FWHA_MY_STATE. and move to the "Down/Dead" status.e. As such.High Availability (New Mode) Interface Failure This scenario assumes two cluster members. that there may be an issue in the inbound direction with one of it's interfaces. When more than two members are present. At the same time as the above events. but the ping requests are being answered. and that the interface of the primary has failed. the secondary will recognize that no CCP packets have been received. This will happen when there are N cluster members. that all of it's own interfaces are operational. Therefore. 3. This is done in an attempt to diagnose which member has the problem. and cluster address per IP segment. . the secondary will attempt ARP requests to hosts belonging to the affected segment. Since no FWHA_PROBE_RPLY message is received as a response. In addition. The secondary issues gratuitous ARP's for both the physical. 5. it will announce via FWHA_MY_STATE on all other segments. and moves to the "Active/Active-Attention" state. and begins sending FWHA_PROBE_REQ messages on the affected segment. such pings will only be issued if all other cluster members do not respond to CCP probing. At the point of failure. and will begin pinging those hosts which respond. the primary will recognize that no CCP messages have been heard on the failed interface. CCP packets) that the interface is alive. The pings will continue as long as we cannot identify by other means (i. that the outbound direction for one of the it's interfaces is in question as well.

The primary at this stage announces itself as "Down/Dead". and assumes responsibility for processing all connections. 11. otherwise from the management server. Though the primary is now considered "Down/dead". 3. Once this occurs. When more than two members are present. is will begin sending FWHA_IFCONF_REPLY packets regularly. and now moves to the "Active/Active-Attention" state. it will still be able to send/receive CCP packets until its' interfaces are brought down. and cluster IP for each segment. but before the policy is loaded. As the primary goes down. . It does so by sending as series of gratuitous ARP's for both it's physical IP.e. 12. 8. The secondary will do several things in an effort to ascertain why no CCP packets are being received. 4. so a random ID is used. First. This will update all necessary hosts/routers on each segment with the relevant updated MAC address information.Once this state has been acknowledged. 2. the primary issues gratuitous ARP's for both the physical. and ping those hosts which respond. The primary now initiates full synchronization on the secured interface via the FW1 protocol 10. and cluster IP for each clustered segment. 9. CCP packets) that the interface is alive. the primary moves to the "Ready" state. the secondary. will make notice of the fact that no CCP packets are being received. The pings will continue as long as we cannot identify by other means (i. and announces this as part of FWHA_MY_STATE. After the Primary learns the cluster ID. it changes it's state to "Down/dead". The secondary will ARP on each segment for IP's belonging to that segment. it sends FWHA_IF_PROB_REQ packets on all segments in which no CCP packets have been heard. 7. 6. The secondary now moves to the "Active/Active-Attention" state. This triggers the secondary to prepare itself to become the active member. This is to illicit a response from any member capable of responding. Once synchronization is complete. it begins announcing FWHA_MY_STATE. which is now in the "Active/Active-Attention" state. such pings will only be issued if all other cluster members do not respond to CCP probing.Once the primary has rebooted. and N-1 of them are down. 5. The secondary acknowledges this by moving to the "Standby" state. It does so without knowing what Cluster it belongs to. This will happen when there are N cluster members.Primary Reboot 1. The primary fetches the policy from another cluster member if possible.

the secondary will not make any attempts to diagnose interface related issues such as the pinging of hosts.Registered Device Failure This scenario assumes two cluster members. the primary is still able to send CCP hello packets. Because of this. this is detected by Cluster XL as the device is no longer reporting state. The secondary now moves to the "Active/Active-Attention" state. The primary changes it's state to "Down/dead". This differs from an interface failure where CCP messages would not be received. 3. and cluster IP for each segment. . in which the fwd daemon has failed on the primary. and will continue to do so. 1. which would trigger such a diagnosis using our topology. and announces this as part of FWHA_MY_STATE. Once the fwd daemon has died. This will update all necessary hosts/routers on each segment with the relevant updated MAC address information. and assumes responsibility for processing all connections. In this case. 2. This triggers the secondary to prepare itself to become the active member. It does so by sending as series of gratuitous ARP's for both the physical.

Dual Failure This scenario assumes a dual failure by both cluster members of the secured (synchronization) interface connected via a crossover link. 2. and for the secondary to report itself as "Down/Dead". both members will become aware of this fact via CCP. the secondary will resume the "Standby" status. Both members will announce as part of FWHA_MY_STATE that N-1 interfaces are up. 4. the failure of one interface will bring down the line protocol of the other resulting in a dual failure. 3. The solution is for the highest priority member to remain in the "Active/Active Attention" state. a decision must be made as to what action to take next. Since both members have suffered the loss of a single interface. but some level of disturbance as already occurred. 1. Upon recovery of the secured link. Since it is assumed that the secured interfaces are connected via a crossover link. and announced as part of FWHA_MY_STATE. Once this occurs. . The necessary state changes are made. Bringing both members down will result in a total failure.

there will be no issuance of gratuitous ARP's for any cluster address during a failure. However. In New Mode. 4. 1. that members state will be changed to "Down/Dead". Therefore. there are some important differences which should be noted. 3. in Load Sharing mode. closely resembles those covered thus far in the previous New Mode section. and continure to process connections. it is necessary to issue gratuitous ARP's during a failure. 2. while all other members will remain in "Active/Active Attention". Although CCP will advertise the configured priority of the sending cluster member. Upon the failure of a member. all members of a Load Sharing cluster will remain in the "Active/Active Attention" state during normal operation. This is not to be confused with the multicast MAC used by CCP for message transmission. This is necessary to ensure that each cluster member receives every packet. each cluster address is associated with the corresponding physical MAC address of the active member.Load Sharing The events carried out by CCP during various failures in Load Sharing mode. is also used as the MAC address for the purposes of packet forwarding. for the purposes of packet forwarding. the multicast MAC used per segment by CCP. As opposed to New Mode. . these priority labels do not dictate a level of seniority in Load Sharing during normal operation as they do in New Mode configurations. For this reason. For this reason. However. a complete analysis will not be given.