ClusterXL Troubleshooting

Clustering
©2003-2007 Check Point Software Technologies Ltd. All rights reserved. Proprietary and confidential.
puresecurity
Course Agenda
FW-1 General Concept

User Space Debug
Kernel Debug
fw monitor
NAT Debugging
Clustering
VPN Troubleshooting
Smart Defense and Web Intelligence
puresecurity ©2003-2007 Check Point Software Technologies Ltd. All rights reserved. Proprietary and confidential. 2
Agenda
Understanding the Cluster Elements

– CCP
– State synchronization
– Pnote
Check Point Solutions:
ClusterXL (HA ,Load Sharing)
Advanced features
3rd party clusters
Troubleshooting
Annex: problematic scenarios
1
Cluster Elements - CCP
Cluster Control Protocol runs on UDP 8116.

CCP is running on all interfaces (in Cluster XL).
In charge for the following:
– Health status reports
– Cluster member probing
– State change commands
– Querying for cluster membership
– State table synchronization
• Note: When VLAN are used CCP will run only on the lowest VLAN ID
(Not true for VSX)
CCP can run in Multicast mode or Broadcast (by

default it is set to Multicast).
It can be changed by:
cphaconf set_ccp [multicast | broadcast]
The file located: $FWDIR/boot/ha_boot.conf
The MAC address used for the multicast is
determine with a special algorithm.
On a network that has a cluster IP x.y.z.w :

If y<=127, the dst multicast MAC would be 01:00:5e:y:z:w.
For example: 01:00:5e:5A:0A:64 Æ 192.90.10.100
If y>127, the dst multicast MAC would be 01:00:5e:(y-128):z:w.
For example: 01:00:5e:28:0A:64 Æ192.168.10.100
(168-128=40 = 28 in hex).
Note: This MAC can be changed via GUI only in LS multicast.
Regarding monitored interfaces that don’t hold a cluster IP but we

still need a destination MAC address for CCP - we use an internal
default cluster IP which looks like x.y.z.250 (when x.y.z is taken
from the interface IP) in order to calculate the MAC address. The
exact translation procedure is the same as for 1, it always like -
01:00:5e:??:??:FA (as FA is 250 in hex).
2
In order to view the addresses run the following command:

cphaconf debug_data
---- Multicast table ----
eth0: Address: 192.168.0.57

Cluster/default multicast IP: 194.29.43.63, MAC address: 01-00-5e-1d-
2b-3f
eth1: Address: 172.30.130.57
Cluster/default multicast IP: 172.30.130.63, MAC address: 01-00-5e-1e-
82-3f
eth2: Address: 10.10.10.57
Cluster/default multicast IP: 10.10.10.250, MAC address: 01-00-5e-0a-
0a-fa
------------------------
NOTE: This was taken from HA configuration using multicast for CCP.
The output will appear in the /var/log/messages.
ipmaddr is another tool.
Viewing the CCP state:
cphaprob –a if
Required interfaces: 3
Required secured interfaces: 1
eth0 UP non sync(non secured), multicast

eth1 UP non sync(non secured), multicast
eth2 UP sync(secured), multicast
Virtual cluster interfaces: 2
eth0 194.29.43.63
eth1 172.30.130.63
State Synchronization
Used to “duplicate” kernel table information between

cluster members
Composed of two phases:
– Full sync: Upon boot
Does not have to be on the Sync interface (fwd
on TCP/256)
– Delta sync: Replicate updates in kernel tables
incrementally. Triggered by every operation done
to a synchronized kernel table (Done by the FW
kernel).
3
Active Member
Standby
Member
member Down
Member
starting
Full sync request
Connection1 Connection1
Connection 2 Connection 2
Connection 3 Connection 3
New New
connection (4) connection (4)
Connection Table Connection Table

Generic
Automatic
Serves both ClusterXL and third parties without

significant changes
Does not sync user mode applications information.
Might cause some performance hit
Selective sync:
4
To see the sync parameters run “fw ctl pstat”
Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 209693, retransmitted : 166, retrans reqs : 129, acks : 54
Sync packets received:
total : 134755, were queued : 221, dropped by net : 101
retrans reqs : 29, received 26 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
Callback statistics: handled 11 cb, average delay : 1, max delay : 1
Cluster Elements – Pnotes
A critical device is also known as a Problem

Notification (pnote).
If a critical device stops functioning, this is
defined as a Failure.
fwd , cphad are predefined as pnotes.
Filter (policy) , sync and interface active check
as well.
To register a new pnote:

cphaprob -d <device> -t <timeout(sec)> -s
<ok|init|problem> [-p] register
To view the pnote status run :

cphaprob list
5
Built-in Devices:
Device Name: Interface Active Check

Current state: OK
Registered Devices:
Device Name: cphad

Registration number: 2
Timeout: 2 sec
Current state: OK
Time since last report: 0 sec
Device Name: fwd

Registration number: 3
Timeout: 2 sec
Current state: OK
Time since last report: 0.8 sec
Clusters basic requirements
The following must match between

members:
– Operating System
– FW-1 version and HFA level
– Installed products
NOTE : Check Point recommends that customers use the same hardware.
ClusterXL
CP clustering product
Includes both HA and LS solutions (AKA CPHA
and CPLS)
Works over the state sync
Supports Solaris, SPLAT and Linux, does not
run on Nokia.
4 modes of operation
– HA – Legacy and New
– LS – Multicast and unicast
6
ClusterXL – HA (New mode)
62.90.111.3
Who isis 62.90.111.3
at 00:80:24:01:01:01
?
ISP router
Internet
62.90.111.3
62.90.111.1 62.90.111.3 is now at 00:80:24:01:01:02

62.90.111.2
00:80:24:01:01:01 00:80:24:01:01:02 Gratuitous ARP
X
172.168.1.1
A Synchronization S
A
172.168.1.2
00:80:24:01:02:01 00:80:24:01:02:02 Internal router
172.168.1.3
Cluster XL (new mode)
cphaprob stat :
Cluster Mode: New High Availability (Active Up)
Number Unique Address Assigned Load State
1 (local) 172.18.100.5 100% active

2 172.18.100.6 0% standby
ClusterXL – Load Sharing (multicast mode)
62.90.111.3 is at 01:00:5e:5a:6f:03
Internet
Who is 62.90.111.3
62.90.111.3
62.90.111.1
62.90.111.2
A1 Synchronization A2
172.168.1.1 172.168.1.2
172.168.1.3
7
Cluster XL multicast mode
cphaprob stat
Cluster Mode: Load Sharing (Multicast)
1 192.168.255.21 33% active
2 192.168.255.22 33% active
3 (local) 192.168.255.20 33% active
Unicast (Pivot) mode
62.90.111.3 is at 00:80:00:01:02:03
Internet
62.90.111.3
62.90.111.1
62.90.111.2
Packet forwarding
P Synchronization
172.168.1.1 172.168.1.2
172.168.1.3
Unicast (Pivot) mode
cphaprob stat
Cluster Mode: Load Sharing (Unicast)
1 (local) 10.10.10.57 30% active (pivot)

2 10.10.10.61 70% active
8
Cluster XL unicast mode
Forwarding will be done on the receiving

network, changing the source mac address to
be the opposite:
Original MAC: 00:d0:b7:be:17:7e

Inversed original MAC: ff:2f:48:41:e8:81
The destination mac will be the non pivot mac.
NOTE: this is not shown some times in tcpdump
Clustering: Advanced features
Asymmetric Routing
Session from standby (Forwarding).
IGMP snooping
Block new Conns
Different subnet
Mac Magic
Disconnected interfaces
Freeze mechanism
Pinging the VIP
Asymmetric Routing
Asymmetric routing is any situation in which the C2S

packet goes through one cluster member, while the
S2C packet goes through another.
C2S – Hash = X
S2C – Hash = Y
9
Asymmetric Routing – cont.
Why bother?
– Race conditions (Is the other member synched?)
– Features without sync (Security Servers)
When will this happen ?
– NATed and encrypted connections
– Data connections
How ClusterXL solves it ?
– Flush and Ack mechanism – hold a packet that made
a change in the kernel table until the change is
synced successfully
– NGX – SDF
Flush and Ack mechanism
Held
packet
Packet
Sync Acks
Decision Function (DF)
The Decision Function

– Calculated for each packet going through the cluster.
– Decides which member should handle the packet.
There are three different types of decision functions (DF).

The desired one can be chosen in the Policy Editor:
1. IP addresses only
2. IP addresses and ports
3. IP addresses, ports and SPIs (Full DF)
Sharing vs. Stickiness
10
Decision Function (DF)
Sticky Decision Function
Should be used when C2S and S2C needs to be

inspected on the same cluster member
Activate when…
– Using Cluster XL Load Sharing and a-symmetric
routing exists
– A feature that need stickiness is used
Drawback
– Performance
– Supported platforms
The Solution - SDF
Active
Sticky
Connection
decision
info
function
Flush&Ack
Sync
Sticky
decision
function
Active
11
3rd party clusters
There are many vendors :

Nortel ASF.
Crossbeam.
RainWall.
Nokia.
StoneBeat.
Check Point is doing only Sync and stateful inspection.
3rd party clusters
The same command line is used.
Cluster Mode: Sync only (OPSEC)Number

Unique Address Firewall State (*)
1 172.17.5.193 active
2 (local) 172.17.5.194 active
3rd party clusters
The Clustering is in most cases the same logic as

ClusterXL meaning there is a VIP and MAC address that
bounded to this VIP , some times that MAC is physical ,
other times it is logical.
In VRRP clusters VIP can sometimes use a VRRP MAC

or interface MAC depending on the platform
12
3rd party clusters
3rd party clusters
Support for Non Sticky connections:

It is using enabling the Flush And Ack mechanism.
Hide Cluster members
When a packet goes from the member it will have the
cluster IP.
Forward cluster incoming packets
Will send the packets coming to the VIP to the member
IP address.
Cluster states
Active - Everything is OK.

Active attention -
A problem has been detected, but the cluster member is
still forwarding packets because it is the only machine in
the cluster or there is no other active machines in the
cluster. In any other situation the state of the machine
would be down.
Down One of the critical devices is down.
13
Cluster states
Ready - Can occur in following scenarios :
1. When a cluster is upgraded from one version of VPN-1 Pro

to another, and the cluster members have different versions
of VPN-1 Pro, the members with a new version have the
ready state and the members with the previous version have
the active state.
2. Before a cluster member becomes active, it sends a

message to the rest of the cluster, and then expects to
receive confirmations from the other cluster members
agreeing that it will become active. In the period of time
before it receives the confirmations, the machine is in the
ready state.
Cluster states
Standby - Applies only to a High Availability

configuration, and means the member is waiting for an
active machine to fail in order to start packet forwarding.
Initializing - An initial and transient state of the cluster

member. The cluster member is booting up, and
ClusterXL product is already running, but VPN-1 Pro is
not yet ready.
Debugging tools
cphaprob
– list (look at pnotes)
– -a if (show interfaces)
– state (show cluster and members’ state)
– syncstat
fw ctl pstat (check sync data)
14
Debugging tools
fw ctl debug –m cluster xxx
Debugging flags [ -m cluster + ]:
conf – Configuration related debug messages

if - Interface tracking and validation
stat - Cluster module state change
select - Packet selection including DF
CCP – Cluster control packet handeling
pnote - Pnote device
Debugging tools (cont)
mac - mac address sync

forward – forwarding layer debug
df – decision function
drop - drops caused by SDF.
There is also the sync flag which is in the fw
topic and shows debug that is related to sync
only.
Other tips
Snoop/tcpdump (using UDP port 8116 traffic)

– It is recommended to look at Ethernet headers
fw monitor (enforcement, cluster hide/fold)

– forwarded packets may cause confusion
15
LAB exercise 1: configuration
Configure your cluster to High Availability

Install policy
Run kernel debug
– fw ctl debug –buf 32000
– fw ctl debug –m cluster + conf stat
– fw ctl kdebug –f &> clust_debug.txt &
Change the cluster configuration to Load Sharing
Install policy
Trace the debug
Note: You may also turn on the fw module flag “filter” to monitor the
policy installation but that will increace the debug output.
LAB exercise 2: status
Configure you cluster to Load Sharing

Install policy
Run kernel debug:
– fw ctl debug –m cluster + pnote stat if log
– fw ctl kdebug –f clust_debug2.txt &
Open a connection through the cluster
Bring one member down
– Use the cecommand clusterXL_admin down
Trace the debug output
Bring the member back up
– Use the cecommand clusterXL_admin up
Trace the debug output
LAB exercise 3: Decision Function
Configure your cluster to Load Sharing

Select the option for Sticky Decision Function
Install policy
Run kernel debug:
– fw ctl debug –m cluster + df select forward drop
– fw ctl debug –m fw + conn
– fw ctl kdebug –f clust_debug2.txt &
Pass several connection through the cluster
See the drops made by SDF in the debug
16
Annex:
Problematic scenarios
When a session starts from the Standby it will

have the following path:
To the server it will go directly from the standby.
From the Server it will go to the Active member ,
then the active member will forward the
connection to the standby.
fw ctl debug + drop

fw ctl debug –m cluster + forward
============on the active=============
FW-1: Inspecting connection to 0xc21d2b3c on ifn 0. local
ifn 0xc21d2b3d
fwha_forward_msg_wrapper(0 0 2142240768 0):
forwardingFW-1: fwha_forward_send_msg: Forwarding
packet to id 0
fwha_forw_flush_callback: Forwarded successfully.
Dropping chainfw_log_drop: Packet proto=6
194.29.42.32:21 -> 194.29.43.60:46659 dropped by
fwhaforw.c Line 644
17
On the standby
==========================
FW-1: FORWARDED Packet : fwha_select_packet: (IF

eth2 (2) at 520238.0 sec) using magic ether header
(0x865900b2):
IGMP
IGMP - allows layer 2 devices to make intelligent

multicast forwarding decisions by examining the contents
of each frame’s Layer 3 IP header.
A station that wishes to become a receiver sends an

IGMP “group join” message to that group’s transmitter
The transmitter will forwards IP multicast traffic only to

those hosts that have requested it
IGMP Snooping
IGMP snooping - Layer 2 devices can “listen in” on IGMP

conversations between hosts and routers.
When a switch “hears” a group join message from a host

on a specific port, it adds that port to the group.
Similarly, when a Layer 2 switch hears a group leave

message or a timer expiration, the port will be removed
from the group.
18
IGMP snooping - the problem
The snooping mechanism relies on listening to IGMP

packets for decision making.
When ‘cphamcset’ starts it opens multicast sockets

which will trigger IGMP messages.
Unfortunately, after 10 minuets there is a refresh time in

which the router sends a question if there are still clients
registered for the multicast group.
clusterXL does not conform to this RFC and therefore

doesn’t answer this query.
Eventually the router gives up and the switch which did

not hear a reply will stop forwarding multicast to the
cluster ports.
IGMP Snooping – the solution
Configure on the switch the static MACs that it needs to

expect and to what ports to forward these MACs.
Work in ClusterXL Load Sharing in Unicast mode
Upgrade to HFA05 on R60, when it is possible to

configure join messages interval in cphaconf.
Block new conns
The reason for blocking new connections is that new

connections are the main source of new synchronization
traffic, and synchronization may be put at risk if new
traffic continues to be processed at this rate.
A related error message is: “FW-1: State synchronization
is at risk. Please examine your synchronization network
to avoid further problems!”
Reducing the amount of traffic passing through VPN-1
Pro protects the synchronization mechanism.
fw_sync_block_new_conns allows VPN-1 Pro to detect
heavy loads and start blocking new connections. Load is
considered heavy when the synchronization transmit
queue of the firewall starts to fill beyond the
fw_sync_buffer_threshold.
19
Block New conns
To enable load detection, set to 0.

To disable load detection, set to -1.
Note that blocking new connections when sync is busy is
only recommended for Load Sharing ClusterXL
deployments. While it is possible to block new
connections in High Availability mode, doing so does not
solve inconsistencies in sync, as High Availability mode
precludes that from happening.
FW-1 default is -1 , VSX the default is 0.
Different subnets feature
172.16.6.10
172.16.6.2 172.16.6.1
1.1.1.1 Non-routable IPs

1.1.1.2
192.168.0.1
(routable IP)
Internet
Different Subnet
create static routes on each cluster

member, which determine the interface connected to the
cluster’s network (the subnet to which the cluster IP
belongs).
Explicitly specify which member subnet is associated

with the cluster IP(In the cluster topology tab).
20
Different Subnet
Note that not all features are supported:
Connection between the members:

You need to add on each member an arp entry for all
other members , since the arp request is issued with the
VIP.
Automatic ARP is not supported , needs to be defined in
$FWDIR/conf/local.arp.
You may need to add a route to get to the members
since their address is not routable.
Mac magic and Forward magic
When more than one cluster is connected to the same

VLAN, if CCP and forwarding layer traffic uses multicast,
this traffic reaches only the intended cluster.
However, if broadcast is used for CCP and forwarding

layer traffic (and in certain other cases), cluster traffic
intended for one cluster is seen by all connected
clusters, and is processed by the wrong cluster, which
causes communication problems.
To ensure that the source MAC address in packets from

different clusters that are connected to the same VLAN
can be distinguished, change the MAC source address
of the cluster interface that is connected to the VLAN in
all but one of the clusters.
Use the following module configuration parameters to set
more than one cluster on the same VLAN. These
parameters apply to both ClusterXL and OPSEC certified
clustering products.
Parameter Default value

fwha_mac_magic 254
fwha_mac_forward_magic 253
21
Disconnected Interfaces
Disconnected Interfaces are interfaces that will

not run CCP.
Sync Interface must not be disconnected.
In 3rd party all interfaces except for the sync
interface.
Defined interfaces can be changed (up/down)
and will not cause a failover.
Disconnected Interfaces
R60 and above , you can define them form the

GUI in the topology tab.
R55 (also in R60) you can add a file in
$FWDIR/conf/discntd.if
the format is : ethX
ethY
machine has to be rebooted.
22
Freeze mechanism
Sometimes during a policy installation , when the

load on the active member is high a failover will
occur , this failover can issue some traffic lose
and in “Switch to higher priority” it will cause a
double failover.
Freeze mechanism
When installing policy the state will stay as it was before

the policy installation.
fwha_freeze_state_machine_timeout=30
Dec 22 19:20:33 amir2 kernel: FW-1: fwha_state_freeze:

FREEZING state machine at ACTIVE
(time=2723329,caller=fwha_set_conf)
Dec 22 19:20:39 amir2 kernel: FW-1: fwha_state_freeze:
ENABLING state machine at ACTIVE
(time=2723385,caller=policy change - finished changes
(fwha_start))
Pinging the VIP
When opening the connection to the VIP , it stay in the

connection table as:
Client -> VIP
Client -> processing member.
When a failover occur other member will handle the
connection but will try to pass it to the other member.
It depends now on the status of the failed member if a
connection will work.
23
Pinging the VIP
Solution :
fw _ allow _ simultaneous _ ping
Firewall-1 is treating ICMP Request & Reply packets as
a single connection.
It uses identifier as destination port and zero as source
port, so that sequential ICMP request are treated as the
same connection.
To make continuous ping to work after failover we have
to treat ICMP request/reply pair as a single connection.
By using ICMP Sequence Number as source port , it will
allow to implement the above.
24

ClusterXL Troubleshooting

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ClusterXL Troubleshooting

Uploaded by

Copyright:

Available Formats

Clustering

 FW-1 General Concept

 Understanding the Cluster Elements

 Cluster Control Protocol runs on UDP 8116.

Cluster Elements - CCP

 CCP can run in Multicast mode or Broadcast (by

Cluster Elements - CCP

On a network that has a cluster IP x.y.z.w :

Note: This MAC can be changed via GUI only in LS multicast.

 Regarding monitored interfaces that don’t hold a cluster IP but we

In order to view the addresses run the following command:

---- Multicast table ----

eth0: Address: 192.168.0.57

Cluster Elements - CCP

Viewing the CCP state:

eth0 UP non sync(non secured), multicast

Virtual cluster interfaces: 2

 Used to “duplicate” kernel table information between

Connection Table Connection Table

 Serves both ClusterXL and third parties without

 Does not sync user mode applications information.

 Might cause some performance hit

 To see the sync parameters run “fw ctl pstat”

Cluster Elements – Pnotes

 A critical device is also known as a Problem

Cluster Elements – Pnotes

 To register a new pnote:

 To view the pnote status run :

Device Name: Interface Active Check

Device Name: cphad

Device Name: fwd

Clusters basic requirements

 The following must match between

62.90.111.1 62.90.111.3 is now at 00:80:24:01:01:02

Cluster XL (new mode)

Cluster Mode: New High Availability (Active Up)

Number Unique Address Assigned Load State

1 (local) 172.18.100.5 100% active

ClusterXL – Load Sharing (multicast mode)

Unicast (Pivot) mode

Unicast (Pivot) mode

Cluster Mode: Load Sharing (Unicast)

Number Unique Address Assigned Load State

1 (local) 10.10.10.57 30% active (pivot)

Forwarding will be done on the receiving

Original MAC: 00:d0:b7:be:17:7e

Clustering: Advanced features

 Asymmetric routing is any situation in which the C2S

Flush and Ack mechanism

Decision Function (DF)

 The Decision Function

There are three different types of decision functions (DF).

 Sharing vs. Stickiness

Sticky Decision Function

 Should be used when C2S and S2C needs to be

The Solution - SDF

 There are many vendors :

 Check Point is doing only Sync and stateful inspection.

3rd party clusters

 The same command line is used.

 Cluster Mode: Sync only (OPSEC)Number

3rd party clusters

 The Clustering is in most cases the same logic as

FW-1 General Concept

Understanding the Cluster Elements

Cluster Control Protocol runs on UDP 8116.

CCP can run in Multicast mode or Broadcast (by

Regarding monitored interfaces that don’t hold a cluster IP but we

Used to “duplicate” kernel table information between

Serves both ClusterXL and third parties without

Does not sync user mode applications information.

Might cause some performance hit

To see the sync parameters run “fw ctl pstat”

A critical device is also known as a Problem

To register a new pnote:

To view the pnote status run :

The following must match between

Asymmetric routing is any situation in which the C2S

The Decision Function

Sharing vs. Stickiness

Should be used when C2S and S2C needs to be

There are many vendors :

Check Point is doing only Sync and stateful inspection.

The same command line is used.

Cluster Mode: Sync only (OPSEC)Number

The Clustering is in most cases the same logic as

In VRRP clusters VIP can sometimes use a VRRP MAC

Support for Non Sticky connections:

Active - Everything is OK.

Ready - Can occur in following scenarios :

Standby - Applies only to a High Availability

Initializing - An initial and transient state of the cluster

fw ctl pstat (check sync data)

fw ctl debug –m cluster xxx

Debugging flags [ -m cluster + ]:

mac - mac address sync

Snoop/tcpdump (using UDP port 8116 traffic)

fw monitor (enforcement, cluster hide/fold)

Configure your cluster to High Availability

Configure you cluster to Load Sharing

Configure your cluster to Load Sharing

When a session starts from the Standby it will

fw ctl debug + drop

IGMP - allows layer 2 devices to make intelligent

A station that wishes to become a receiver sends an

The transmitter will forwards IP multicast traffic only to

IGMP snooping - Layer 2 devices can “listen in” on IGMP

When a switch “hears” a group join message from a host

Similarly, when a Layer 2 switch hears a group leave

The snooping mechanism relies on listening to IGMP

When ‘cphamcset’ starts it opens multicast sockets

Unfortunately, after 10 minuets there is a refresh time in

clusterXL does not conform to this RFC and therefore

Eventually the router gives up and the switch which did

Configure on the switch the static MACs that it needs to

Work in ClusterXL Load Sharing in Unicast mode

Upgrade to HFA05 on R60, when it is possible to

The reason for blocking new connections is that new

To enable load detection, set to 0.

FW-1 default is -1 , VSX the default is 0.

create static routes on each cluster

Explicitly specify which member subnet is associated

Connection between the members:

When more than one cluster is connected to the same

However, if broadcast is used for CCP and forwarding

To ensure that the source MAC address in packets from

Parameter Default value

Disconnected Interfaces are interfaces that will

R60 and above , you can define them form the

Sometimes during a policy installation , when the

When installing policy the state will stay as it was before

Dec 22 19:20:33 amir2 kernel: FW-1: fwha_state_freeze:

When opening the connection to the VIP , it stay in the