VPN TROUBLESHOOTING:

REFFER: http://www.cpug.org/forums/vpns-virtual-private-networks/4764-vpntrouble-shooting.html
Basics:
IKE negotiation consists of two phases - Phase I (Main mode which is six packets) and
Phase II (Quick Mode which is three packets).

The

$FWDIR/log/ike.elg file contains this information (once debugging is enabled).

To enable debugging, you need to login to your firewall and enter the command "vpn debug on
Check Point have a tool called

vpn debug ikeon" or "vpn debug trunc".

IKEView.exe which parses the information of ike.elg into a GUI making this easier to view.

Note that another useful tool is "vpn debug on mon" which writes all of the IKE captured data into a file ikemonitor.snoop which you can open with wireshark or ethereal.

PHASE1:
negotiates encryption methods (DES/3DES/AES etc), the key length, the hash Algorithm (MD5/SHA1) and
creates a key to protect the messages of the exchange .
It does this in 5 stages:
1. Peers Authenticate using Certificates or a pre-shared secret.

2. Each peer generates a private Diffie-Hellman key from random bits and from that derives a DH public key. These are then exchanged.

3. Each peer generates a shared secret from its private key and its peers public key, this is the DH key.

4. The peers exchange DH Key material (random bits and mathematical data) and methods for PhaseII are agreed for encryption and integrity.

5. Each side generates a symmetric key (based upon the DH key and key material exchanged).

In IkeView under the IP address of the peer , open the

> "P1 Main Mode ==>" for outgoing or "P1 Main Mode <==" for incoming

Main Mode Packet 1 - expand :

>MM Packet 1

>Security Association

>Prop1 PROTO_ISAKMP

>Tran1 KEY_IKE

proposed Encryption Algorithm, Key Length, Hash Algorithm, Authentication Method, DH Group, and SA
renegotiation params (life type - usually secs and duration).
You should then be able see the

If your

encryption fails in Main Mode Packet 1, then you need to check your VPN communities.

Packet 2 ( MM Packet 2 in the trace ) is from the responder to agree on one encryption and hash algorithm
Packets 3 and 4 arent usually used when troublshooting. They perform key exchanges and include a large number called a NONCE. The NONCE is a set of never before used random numbers
sent to the other part, signed and returned to prove the parties identity.
Packets 5 and 6 perform the authentication between the peers. The peers IP address shows in the ID field under MM packet 5. Packet 6 shows that the peer has agreed to the proposal and has
authorised the host initiating the key exchange.

If your encryption fails in Main Mode Packet 5, then you need to check the authentication - Certificates or preshared secrets

Phase II
IPSec Security Associations (SAs) are negotiated, the shared secret key material used for the SA is determined and there is an
additional DH exchange.
Phase II failures are generally due to a misconfigured VPN domain.

Phase II occurs in 3 stages:

1. Peers exchange key material and agree encryption and integrity methods for IPSec.

2. The DH key is combined with the key material to produce the symmetrical IPSec key.

3. Symmetric IPSec keys are generated.

In IkeView under the IP address of the peer , expand Quick

> "P2 Quick Mode ==>" for outgoing or "P2 Quick Mode <==" for incoming

Mode packet 1:

> QM Packet 1

> Security Association

> prop1 PROTO_IPSEC_ESP

> tran1 ESP_AES (for an AES encrypted tunnel)

You should be able to see the SA life Type, Duration, Authentication Alg,
If your encryption fails here, it is one of the above Phase II settings that needs to be looked at.

Encapsulation Mode and Key length .

There are two ID feilds in a QM packet. Under

> QM Packet 1

> ID
You should be able to see the initiators

VPN Domain configuration including the type (ID_IPV4_ADDR_SUBNET) and data (ID Data field).

Under the second ID field you should be able to see the peers VPN Domain configuration.
Packet 2 from the responder agrees to its own subnet or host ID, encryption and hash algorithm.
Packet 3 completes the IKE negotiation.
If all of this works without any errors, then you may have previously initiated an invalid tunnel previously. You can use the

A.

VPN tunnel utility "vpn tu" to remove SA keys from the table.

If there is any additional information regarding two other frequent problems one way only traffic and tunnel disconnections ?
One way only traffic is generally the result of one peer not having correctly established a security association.
Most frequently this is due to the way in which Check Point combines adjacent IP address networks together into supernets. ie, if you have 192.168.0.0/24 and 192.168.1.0/24, Check Point will supernet this into
192.168.0.0/23.
This is done to reduce the number of keys required and hence reduce the load on the VPN gateway. However, other VPN devices do not follow this methodology, so depending upon the version of VPN-1 you are
using, you may need to set IKE_use_largest_possible_subnets or correctly configure the VPN communities tunnel management (one vpn per pair of hosts, per subnet pair or per gateway). See
SecureKnowledgebase article Solution ID: #sk26336.
Tunnel disconnections can be caused either a physical connectivity problem or routing problems or once again, a mismatch in the VPN security associations. Be particularly careful with VPNs to Cisco in this regards.
Plenty of times I've seen people confused between seconds and minutes! I've also seen that sometimes the Cisco ends of VPNs dont want to reset the SAs when told to by the Check Point end.
B. If there is any information regarding ike.elg + vpn.elg explanation. I familiar with the ike/ipsec processes but those 2 files are still no easy to understand (I know there is a tool called ikeview but I dont work for
organization considered as CSP so its seems like Ill never put my hands on this tool) ?

From my answer above, you can see that my statement about "Most VPN debugging consists of looking at the IKE negotiation" to be true. And it is most unlikely that you will need to look into the vpnd.elg file. With
respect to obtaining ikeview.exe.

IMPORTANT PATHS:
1.Where are database revision control files stored?

$FWDIR/conf/db_versions/repository/

Cisco ASA order of operations

1.

FLOW-LOOKUP- This will check for existing connections. I a connection exists, the flow is automatically allowed
2. ROUTE-LOOKUP - This is the inbound route lookup which includes reverse patch, if enabled.
3. Inbound ACCESS-LIST- Checks for an interface ACL
4. CONN-SETTINGS - Application layer checks (Class maps)
5. IP-OPTIONS- RFC 791
6. NAT
7. Outbound ACCESS-LIST (if an outbound access list exists on the egress interface).
9.FLOW-CREATION
10.ROUTE LOOKUP - Destination route lookup

How to immediately know if you are logged into the active or standby firewal on ASA
The prompt (introduced in 7.2(1)) command allows you to customize the hostname of the ASA to include dynamic elements.
prompt state will display the state of the firewall.
for example:
lab-dev-01# config t
lab-dev-01 (config)# prompt state
lab-dev-01/act(config)#

actFailover is enabled, and the unit is actively passing traffic.

stby Failover is enabled, and the unit is not passing traffic and is in a standby, failed, or other non-active state.
actNoFailoverFailover is not enabled, and the unit is actively passing traffic.
stbyNoFailoverFailover is not enabled, and the unit is not passing traffic. This might happen when there is an interface failure above the threshold on
the standby unit.

How to determine the path and interface of a host on SPLAT

[Expert@lab1]# ip route get 192.168.1.1
192.168.1.10 via 192.168.19.38 dev eth2 src 192.168.19.1
cache mtu 1500 advmss 1460

How to ping the inside interface of an ASA through a VPN tunnel

This is typically used for testing VPNS.

#conf t
(config)# management-access inside

Step backwards for VPN supernetting in R71

It was recently brought to my attention that Checkpoint's infamous VPN supernetting in R71 can no longer be fixed by changing the VPN Advanced Tunnel
Options to "1 VPN Per Pair of Hosts".
As with R55, you have to change the following:
$FWDIR/conf/Objects_5_0.C file. Change Support Subnets for Key Exchange to false.
It is also believed that migrated VPNS that were previously configured using the Advanced Tunnel Options retain their settings, but new VPNs will not
work. If anyone has any more information on this, I would appreciate the input.
It is believed that R75 has gone back to the use of the Advanced VPN Tunnel Options setting.

Cluster cannot be empty" or "only one interface defined" error on Checkpoint R70 and R71

Checkpoint has confirmed that this is a bug that occurs occasionally when pushing policy to clusters.
Example:
Firewall and Address Translation Policy Verification: Verifier warnings: There is only one interface defined for object . At least one more interface must be
configured for this object in order to use the Anti-Spoofing feature.

or
"Verifier warnings: A Cluster cannot be empty. It must have Cluster members"

To resolve the issue, open the cluster object for the gateway that you are pushing policy for,
and go into the Topology. Click OK, then OK on the Cluster object and Save. The error should go away.

Checkpoint R71 bug that will cause migrations to fail

Checkpoint has acknowledged that there is a bug in R71 that will cause any policy migrations from older versions to fail if there is a tilde (~) in the name
of a policy being migrated. This is true even if the policy is not used.
Therefore any policies with a tilde in the name should be renamed before migrating.

ERROR: This license does not allow configuring more than 2 interfaces with nameif and without a "no forward" command on this interface or on 1
interface(s) with nameif already configured.
This error will occur during the configuration of a ne wVLAN on an ASA 5505.
This error occurs because there is only a Base license installed on the ASA. The license will need to be upgraded to a Security Plus license.
A base license will only allow 3 VLANS to be created.

Everything you need to know about troubleshooting VRRP on Nokia Checkpoints

VRRP failover happens when one of the following events takes place:
-a monitored interface looses its link state
-VRRP hello packets from the master not seen on the secondary device
-a critical Checkpoint service or daemon fails to report its status. This requires FW Monitoring to be turned on in Voyager. If turned on, whenever the clock
is set backwards, a failover will also occur.
tcpdump -nni eth1 proto VRRP
The packets will contain the vrid and priority.
When a failure occurs, the failed device sends out a priority 0 message on all good interfaces. This tells the secondary to take over.
Example:
PrimaryHA-fw1[admin]# tcpdump -i eth-s1p1c0 proto vrrp
tcpdump: listening on eth-s4p2c0
00:46:11.379961 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver
00:46:12.399982 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver
00:46:13.479985 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver
00:46:14.560007 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver

20:
20:
20:
20:

vrid
vrid
vrid
vrid

100
100
100
100

pri
pri
pri
pri

100 [tos 0xc0]

100 [tos 0xc0]
100 [tos 0xc0]
0 [tos 0xc0]

If both firewalls are broadcasting vrrp, and the packets are not seen by the other firewall, there could be a communication problem between the firewalls.
Also ensure that the vrid matches on both firewalls.
Proper VRRP failovers usually only cause 1 or 2 packets lost .
VRRP multicast address is 224.0.0.18
To capture vrrp traffic in fw monitor:
fw monitor -e accept ip_p = 112;
Clish
show vrrp
This will show you which devices are in master and backup
Example:
PrimaryFW-A> sh vrrp

VRRP State
Flags: On
6 interface enabled
6 virtual routers configured
0 in Init state
0 in Backup state
6 in Master state
PrimaryFW-A>
PrimaryFW-A> exit
Bye.
PrimaryFW-A[admin]#
SecondaryFW-B[admin]# iclid
SecondaryFW-B> sh vrrp
VRRP State
Flags: On
6 interface enabled
6 virtual routers configured
0 in Init state
4 in Backup state
2 in Master state
SecondaryFW-B>
SecondaryFW-B> exit

show vrrp interfaces

Detailed configuration of VRRP, including priority, hello interval, and VRID
clish -c "show interfacemonitor"
Displays interface transitions
cphaprob -i list
Displays Checkpoint critical processes and their timeouts.
To log critical process failures:
ipsctl -w net:log:partner:status:debug 1
That will log to the console and to /var/log/messages. If you want to turn off:
ipsctl -w net:log:sink:console 0
To change the timeout value of a monitored process:
cphaprob -d [device] -t [timeout] -s [state] -p register

Resolving local logging issues on Checkpoint

If logs are not appearing in Smartview Tracker, they are probably logging locally.
To determine if logs are being stored locally on the gateway, go to $FWDIR/log.
Locate the fw.log file and see if it's size is incrementing. There may also be additional fw*.log files that have rolled over.
To resolve the issue, first try restarting the MLM (in a Provider environment or the Log Services in a Smartcenter Server environment).
Next, restart the firewall services on the gateway (fw kill fwd followed by fwd).
If that does not work, try restarting the firewall.
Once resolved, you can pull the stored logs from the gateway by running "fw fetchlog " from the log server. In R70, there is also an option to fetch logs in
Smartview Tracker (Tools>Remote Files Mgmt)

Configuring SNMP on SPLAT

step 1: service snmpd restart
step 2: edit /etc/snmp/snmpd.users.conf and replace public with your actual
snmp community string
step 3: service snmpd restart
step 4: netstat -an | grep 161
for checkpoint snmpd port 260:
step 1: modify the $FWDIR/conf/snmp.C file and place the actual snmp
community inside the read and write (). If you leave the write empty,
it will use "private" as the community string. This is a security risk.
step 2: run sysconfig and start the checkpoint snmpd extension
step 3: perform cpstop;cpstart
step 4: netstat -an | grep 260

Creating a Read Only SPLAT user

Creating a user
1. SSH to the firewall where account will be setup on.
2. From the command line type adduser , here we will add the user with username testuser. The command should read adduser testuser
3. Input the desired password when prompted to do so
Changing the users shell
1. Open the passwd file for editing by typing vi /etc/passwd
2. Find the line corresponding to the user you just created. If you have created a user with username testuser, the line you are looking for is
testuser:x:0:0::/home/test:/bin/cpshell
3. Change the users shell, to do this we will change /bin/cpshell to /path/to/shell.
Before the change the line should read:
testuser:x:0:0::/home/test:/bin/cpshell
After the change the line will read:
testuser:x:0:0::/home/test:/etc/scripts/myshell.sh

Checkpoint port list

TCP 256: CA and DH key exchange, net topology fetch on older SC versions, and port used to push policy to remote firewalls.

TCP 257: Logging

TCP 258: Mgt console listens for remote GUI connections

TCP 259: Client Auth via telnet

UDP 259: manages encrypted sessions

UDP 260: SNMP for the Checkpoint daemon

TCP 262: Single Sign-on daemon.

TCP 264: SC topology fetch.

UDP 500: ISAKMP

TCP 900: HTTP client auth.

TCP 4532: Session Auth agent

TCP 18181: Content Vectoring Protocol

TCP 18182: URL Filtering Protocol.

TCP 18183: Suspecious Activity Monitoring for IPS.

TCP 18186: SIC between OPSEC products and the gateway.

TCP 18190: Gateway listens for management clients.

TCP 18191: CPD process for communications such as policy installation and certificate revocation.

TCP 18192: CPD monitoring

Cisco ASA Policy Based Nat

Example:
Source address 10.1.1.1 should be translated to 192.168.1.1 when going to 172.16.1.1 and translated to 192.168.1.2 when going to 172.16.1.2

access-list policy_nat1 permit ip host 10.1.1.1 host 192.168.1.1

access-list policy_nat2 permit ip host 10.1.1.1 host 192.168.1.2
static (inside,outside) 172.16.1.1 access-list policy_nat1
static (inside,outside) 172.16.1.2 access-list policy_nat2

Cisco ASA Reverse Path Forwarding

Reverse Path Forwarding
RPF errors are typically NAT related (traffic is natted one way in one direction and another way in the other direction).
Example: --->no nat
<--- hide nat
Example of this error:
%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows; Connection for icmp src outside:192.168.25.100 dst
dmz:192.168.28.12 (type 8, code 0) denied due to NAT reverse path failure
The other reason is if RPF checking is turned on and the source host comes in on an interface where a route is not defined for the host. This type of RPF
check must be configured on a per interface basis, which will cause the firewall to examine the source IP of each packet. This also adds a little additional
overhead.
To turn on interface RPF checking run the following interface config command:
ip verify reverse-path interface outside

Great Cisco TAC podcast on Anyconnect

This podcast covers an overview of Anyconnect as well as some great troubleshooting procedures.
http://cisco-podcast.streamguys.net/cdc/security/tac/TACSecurityShow_episode_11.mp3

Troubleshooting VRRP
Check the status of the interfaces
In this example, both firewalls believe that they are in a master state.
FW-1[admin]# iclid
FW-1> sh vrrp
VRRP State
Flags: On
6 interface enabled
6 virtual routers configured
0 in Init state
0 in Backup state
6 in Master state
FW-1>
FW-1> exit
FW-2[admin]# iclid
FW-2> sh vrrp
VRRP State
Flags: On
6 interface enabled
6 virtual routers configured
0 in Init state
4 in Backup state
2 in Master state

A TCPDUMP can confirm that VRRP packets are reaching each interface.
On the Primary:
FW-1[admin]# tcpdump -i eth-s4p2c0 proto vrrp
tcpdump: listening on eth-s4p2c0
00:46:11.374424 O 192.168.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
00:46:12.344334 O 192.168.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
Secondary:
FW-1[admin]# tcpdump -i eth-s4p2c0 proto vrrp
tcpdump: listening on eth-s4p2c0
00:19:38.533454 O 192.168.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:39.544322 O 192.168.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
Now you can see that the interface on both the primary and the secondary firewalls are broadcasting vrrp multicasts. This is because the vrrp multicasts
are not reaching the firewalls interfaces. This means there is a communication breakdown which can be possibly caused by network issues.

In another example you will see that the VRIDS dont match
FW-1[admin]# tcpdump -i eth-s4p2c0 proto vrrp
00:46:11.206994 I 10.10.10.1 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 95 [tos 0xc0]
00:46:11.379961 O 192.168.1.1 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 100 [tos 0xc0]
FW-1[admin]# tcpdump -i eth-s4p2c0 proto vrrp
00:19:38.507294 O 192.168.1.2 > 224.0.0.18: VRRPv2-adver 20: vrid 102 pri 95 [tos 0xc0]
00:19:38.630075 I 10.10.10.2 > 224.0.0.18: VRRPv2-adver 20: vrid 103 pri 100 [tos 0xc0]

Usefull Nokia IPSO Commands

newimage -R -k -l ipso.tgz - install a new IPSO image
newpkg i installs software from given location (firewall software, VPN accel driver, etc)

voyager e 0 80 resets voyager after a failed ssl config attempt

dbpasswd admin -Changes the password from the command line
ipsofwd on admin -turns on ip forwarding when firewall is stopped
ipsofwd list -displays ipso properties (flowpath, etc)
ipsofwd slowpath -turns off flows (flowpath turns back on)
iclid -vrrp utility that shows status
- show vrrp -iclid command that shows # of interfaces and their respective states
- get vrrp -shows iclid stats: active interfaces/checksum/version/id
-show vrrp interface -displays interface stats for VRRP
boot s {from > prompt at boot time) boots into single-user mode
Nokia IPSO has 2 shells, IPSO and Clish.
After logging in, you are in the IPSO shell. To enter the Clish shell, type "clish"
To remove old config:
Either rm /active/config or config/active depending on version.

Usefull Checkpoint Commands

o view the active connections table: fw tab -t host_table s
To pull the latest policy from the management station: fw fetch
Display the name of the policy installed and the date it was received: fw stat

View the Checkpoint version installed: fw ver

Display cpu, memory, and disk usage: fw ctl pstat
Delete all hosts from the connections table: fw tab -t host_ip_addrs x
Display logs on the firewall for a specific IP: fw log n ft | grep
Troubleshoot source/destination access issues: fw monitor -m iIOo -e 'accept src=10.33.76.82 and dst=10.33.76.82;'
Manage VPN connections (view and delete): vpn tu
Turn on debugging for VPN's: vpndebug on and vpn debug ikeon
This will create 2 files in $FWDIR/logs. vpnd.elg (this can be viewed on the firewall using cat. It will show highlevel VPN connection information), and
ike.elg (this is the bread and butter of Checkpoint VPN troubleshooting. Click here to read my ikeview guide).
Display SIC key: cp_conf sic get
High Availabiliy: cphaprob stat -display HA status
cphaprob -i -display HA interface stats
cphastop/cphastart -stop/start HA

View license key installed: cplic print

Delete all active hosts: fw tab -t host_ip_addrs x

Common Clish commands on Nokia IPSO appliances

---setting default gateway
set static-route default nexthop gateway address 192.168.29.2 priority 1 on
---adding static routes
set static-route 172.23.124.150/32 nexthop gateway address 192.168.29.50 on
---Add proxy arp
add arpproxy address 192.168.29.56 macaddress 0:a0:8e:7d:13:d0
add arpproxy address 192.168.29.57 macaddress 0:a0:8e:7d:13:d0
---Add an interface
set interface eth1 speed 100M duplex full active on
add interface eth1c0 address 192.168.29.54/24
set interface eth1c0 enable
---VRRP
set vrrp accept-connections on
set vrrp coldstart-delay 60
set
set
set
set
set
set
set
set

vrrp
vrrp
vrrp
vrrp
vrrp
vrrp
vrrp
vrrp

interface
interface
interface
interface
interface
interface
interface
interface

eth1c0
eth1c0
eth1c0
eth1c0
eth1c0
eth1c0
eth1c0
eth1c0

monitored-circuit
monitored-circuit
monitored-circuit
monitored-circuit
monitored-circuit
monitored-circuit
monitored-circuit
monitored-circuit

vrid
vrid
vrid
vrid
vrid
vrid
vrid
vrid

---Set ntp servers

add ntp server 10.1.1.2 version 3 prefer yes
add ntp server 10.1.1.1 version 3 prefer yes
---Setting Time zone
set date timezone-city "Greenwich (GMT)"
---Add hostname
set hostname testbox

54
54
54
54
54
54
54
54

monitored-interface eth2c0 on
monitored-interface eth2c0 priority-delta 10
monitored-interface eth3c0 on
monitored-interface eth3c0 priority-delta 10
priority 100
hello-interval 1
vmac-mode default-vmac
backup-address 192.168.29.1 on

---Add Host address assignments

add host name testbox ipv4 192.168.29.54

Nokia IPSO Password reset

Boot the Nokia device into single user mode
To boot an IP440 into single user mode first restart the box.. When you see the "boot:" prompt enter "-s" and press "enter" within 10 seconds. When it
boots into single user mode it will ask for the shell, just press "enter" to accept the default "sh."
To boot an IP500 or higher into single user mode, first restart the box. When you will see the prompt "Entering autoboot mode. Type any character to
enter command mode." You have 5 seconds to press any key.
To boot at IP300 device into single user mode, first restart the box. When you see the prompt "Verifying DMI Pool Data" press the number 1. Then you
will see a "Type any character to enter command mode." You now have 5 seconds to press any key. After pressing any key type "boot -s" to enter single
user mode.
Change Password in IPSO 3.5 and Higher
Run "/etc/overpw" from the single user shell and follow the prompts to change the password. Type "reboot" to boot into multi-user mode, go into voyager
and change to a permanent password.

Cisco VPN Troubleshooting Guide

Cisco PIX 7.0 VPN Troubleshooting
Quick overview of IPSEC
It is important to understand how IPSEC works in order to understand how to troubleshoot a VPN connection. This is a quick overview of IPSEC and is by
no means a complete detailed guide.
IPSEC is a suite of protocols, defined in RFC 2401, that is used to protect information as it travels from one private network to another private network
over a public network.
IPSEC consists of Security Protocols (AH and ESP), Key Management (ISAKMP, IKE, and SKEME), and Algorithms (3DES, AES256, etc).
ISAKMP defines the procedures and packet formats used to establish, negotiate, and modify Security Associations. ISAKMP communicates over UDP 500.
Security Protocols consist of AH (Authentication Header) and ESP (Encapsulating Security Payload). AH communicates over IP 51 and provides data
authentication, integrity, and replay protection (for man in the middle attacks), but does not provide confidentiality. It is important to understand that AH
encapsulates the IP packet but does not encrypt it.
ESP communicates over IP 50 and provides the same service as AH in addition to providing data confidentiality by encrypting the original payload and
encapsulating the packet.
SAs (Security Associations):
In order to have an IPSEC conversation, you first need a security association. Each device must agree on the policies or rules of the conversation by
negotiating these policies with their potential peers. The SA represents a unidirectional instance of a security policy for a given connection.
Main mode IPSEC packet exchange:
--Initiator--- ---Responder------------pk#1Policy Proposal------>
<-------pk#2---Policy Accept/Reject-- ----------pk#3---DH Exchange-------->
<-------pk#4---DH Exchange---------- ----------pk#5---ID/Hash------------->
<------pk#6---ID/Hash--------------->
Packet handling order:
Step 1 Access lists applied to an interface and crypto map are used by Cisco IOS software to select interesting traffic to be encrypted.
Step 2 Cisco IOS software checks to see if IPSec SAs have been established.
Step 3 If the SA has already been established by manual configuration using the crypto ipsec transform-set and crypto map commands or has been
previously set up by IKE, the packet is encrypted based on the policy specified in the crypto map and is transmitted out of the interface.
Step 4 If the SA has not been established, Cisco IOS software checks to see if an IKE SA has been configured and set up.
Step 5 If the IKE SA has been set up, the IKE SA governs negotiation of the IPSec SA as specified in the IKE policy configured by the crypto isakmp
policy command, the packet is encrypted by IPSec, and it is transmitted.
Step 6 If the IKE SA has not been set up, Cisco IOS software checks to see if certification authority (CA) has been configured to establish an IKE policy.
Step 7 If CA authentication is configured with the various crypto ca commands, the router uses public and private keys previously configured, obtains the
CA's public certificate, gets a certificate for its own public key, and then uses the key to negotiate an IKE SA, which in turn is used to establish an IPSec
SA to encrypt and transmit the packet.
Configuring Phase 1:
The first 2 octets of IPs have been replaced with "y.y."
Phase I is not configured on a per connection basis. When a Phase I connection is being established, configured ISAKMP policies will be tried one at a time
until a match is found.

Example of an ISAKMP policy:

#isakmp policy 20 authentication pre-share
#isakmp policy 20 encryption 3des
#isakmp policy 20 hash md5
#isakmp policy 20 group 2
#isakmp policy 20 lifetime 43200
Troubleshooting Phase I:
Check the syslogs
Show run isakmp
This will show the isakmp policies for all VPN connections. To view a specific ISAKMP policy type show run isakmp | grep
show vpn-sessiondb detail l2l
Show crypto isakmp sa detail This command will display the state of Phase I of the IPSEC tunnel. A state of MM_Active indicates that Phase I was
successfully completed. If Phase I does not complete, refer to the table below to find out exactly what state the Phase I connection is currently in. This
will give you an indication of where the problem has occurred. More specific information can be found by running a debug(discussed later).
State Description
OAK_MM_No_STATE This is the initial state of Phase I. If you see Phase I
In this state for longer than a few seconds, this is an
indication that a failure of tunnel establishment for
Phase I has occurred.
OAK_MM_SA_SETUP The peers have agreed on parameters for the ISAKMP
SA. Phase I will be in this state after packet 1 and packet 2 exchange of the Main Mode negotiation (see above).
MM_WAIT_MSG The firewall is waiting on the remote end device to respond with DH and public key.
OAK_MM_KEY_EXCH The peers have exchanged DH public keys and have generated a shared secret.
OAK_MM_KEY_AUTH The ISAKMP SA has been authenticated.
The debug crypto isakmp 5 command will display real time information on every step of the Phase I connection. Debug level 5 should be sufficient for
most troubleshooting however level 7 provides more detailed information if necessary.
Please note that you cannot limit the debug output to a specific tunnel.
IKMP_NO_ERROR_NO_TRANS indicates a matching transform set was not found
No Proposal Chosen=isakmp policy mismatch
syslog sample of a completed connection:
Mar 10 2008 18:47:05: %PIX-3-713119: Group = y.y.41.250, IP = y.y..41.250, PHASE 1 COMPLETED
Sample Debug output:
The following shows the initiation of the first packet for an IPSEC tunnel.
58534 02/27/2004 07:42:38.600 SEV=4 IKE/41 RPT=8619 y.y.11.49
IKE Initiator: New Phase 1, Intf 2, IKE Peer
The following indicates that the IKE Phase I policy was accepted by the remote gateway.
58534 02/27/2004 07:42:38.600 IP = y.y.11.49, Oakley proposal is acceptable
This indicates Phase I has completed.
58534 02/27/2004 07:42:38.600 Group= y.y.11.49, IP=y.y.11.49, Oakley begin quick mode
The following indicates that the remote gateway has indicated that none of the policies are acceptable.
5|Oct 02 2006 09:41:41|713904: IP = y.y.138.12, Received an un-encrypted NO_PROPOSAL_CHOSEN notify message, dropping
To clear the Security Associations related to Phase 1, use the clear crypto isakmp command. This will clear ALL of the SAs currently built on this firewall.
To confirm that the IPSEC packets are reaching the firewall, a capture can be created for all UDP 500 traffic.
First create an access-list for the traffic you would like to capture.
Access-list capture1 permit udp any any eq 500
Next create a capture.
Capture cap1 access-list capture1 interface outside
Next display the results of the capture.
Show capture cap1 detail
ciscoasa#show capture cap1 detail
1: 13:04:06.284897 192.168.1.50 > 192.168.1.1: UDP:500
View capture on web
https://capture/pcap/cap1
View pre-shared keys:
more system:running-config
Configuring Phase 2:

A transform set combines encryption method and authentication method. During the IPSec security association negotiation with ISAKMP, the peers agree
to use a particular transform set to protect a particular data flow. The transform set must be the same for both peers.
You can create multiple transform sets, and then specify one or more of these transform sets in a crypto map entry.
You can view previously created transform sets by typing the show crypto ipsec transform-set command. If the desired transform set has not been
previously defined, the crypto ipsec transform-set command is used to create it.
Example:
#(config)crypto ipsec transform-set 3desmd5 esp-3des esp-md5-hmac
An access-list is used to define the interesting traffic or the traffic that should be encrypted and allowed through the VPN Tunnel. The access-list should
always be defined from local to remote. The subnet sizes need to match on the remote gateway.
Example:
#(config) access-list tunnel1 extended permit ip y.y..191.0 255.255.255.0 host y.y..155.12
If port filtering is being used, and traffic is being initiated from the remote side, the destination port of the remote host must be the source port of the
local matching acl.
A tunnel group is used to identify specific connection parameters and the definition of a group policy. The default tunnel groups are DefaultRAGroup (used
for Remote Access tunnels) and DefaultL2Lgroup(used for IPSEc Lan-to-Lan tunnels).
Example:
#(config)tunnel-group y.y.155.1 type IPsec_l2l
#(config)tunnel-group y.y.155.1 ipsec-attributes
#(config-attributes) pre-shared-key abc123
The crypto map ties together several components that define the VPN tunnel. This is where the peer defined in the tunnel-group command is tied to the
access-list and transform-set. The crypto map must be assigned a unique map id #. To view the previously used crypto map id numbers run the show ru
crypto command.
Example:
#(config)crypto map mymap 10 match address tunnel1
#(config)crypto map mymap 10 set peer y.y,155.1
#(config)crypto map mymap 10 set transform-set 3desmd5
Nat considerations:
If a local address is going to be natted outbound, the crypto acl should use the outside ip address.
Troubleshooting Phase II:
Check syslogs
Show crypto ipsec sa- This command shows the output of the IPSEC SAs. The SA will include the ip address of the local and remote endpoints, encryption
domains (interesting traffic), transform set (what encryption and hash is being used), key lifetime, and # of packet encrypt/decrypts.
debug crypto engineDisplays the traffic that is encrypted.
Example of an IPSEC SA:
This shows the crypto map used for this connection.
Crypto map tag: vpn_map, seq num: 130, local addr: x.x.160.45
The following line shows the crypto acl that includes the traffic to be protected.
access-list VPN-CIDS704976 permit ip x.x.190.0 255.255.254.0 host 10.2 5.4.80
local ident (addr/mask/prot/port): (x.x.190.0/255.255.254.0/0/0)
remote ident (addr/mask/prot/port): (10.25.4.80/255.255.255.255/0/0)
current_peer: y.y.227.136
Encrypts indicate that this side is encrypting and sending traffic. Decrypts indicates that the other side is sending traffic.
#pkts encaps: 5, #pkts encrypt: 5, #pkts digest: 5
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 5, #pkts comp failed: 0, #pkts decomp failed: 0
#send errors: 0, #recv errors: 0
This lists the local and remote endpoints.
local crypto endpt.: x.x.160.45, remote crypto endpt.: y.y.227.136
path mtu 1500, ipsec overhead 74, media mtu 1500
current outbound spi: 2AFEA5C7
There is a separate sa for inbound and outbound.
inbound esp sas:
spi: 0x9D111D2A (2635144490)
transform: esp-aes-256 esp-sha-hmac none
in use settings ={L2L, Tunnel, PFS Group 5, }
slot: 0, conn_id: 317225, crypto-map: vpn_map
sa timing: remaining key lifetime (kB/sec): (4275000/28789)
IV size: 16 bytes
replay detection support: Y
outbound esp sas:
spi: 0x2AFEA5C7 (721331655)
transform: esp-aes-256 esp-sha-hmac none
in use settings ={L2L, Tunnel, PFS Group 5, }
slot: 0, conn_id: 317225, crypto-map: vpn_map
sa timing: remaining key lifetime (kB/sec): (4274999/28789)
IV size: 16 bytes
replay detection support: Y

Clear crypto ipsec sa peer will clear the Phase 2 SAs for a given peer.
debug crypto ipsecDisplays the IPSec negotiations of phase 2.
No Valid SA/ Identity mismatch Transform set or crypto acl
Sample Debug output:
The following shows that the tunnel group configuration was found.
Oct 26 15:42:43 [IKEv1]: IP =y.y.205.92, Connection landed on tunnel_group y.y,.205.92
Sample syslog errors:
This shows interesting traffic ACL getting exchanged.
1754 11/29/2001 16:20:18.500 SEV=7 IKEDBG/0 RPT=546 y.y.205.92
Transmitting Proxy Id:
Remote host: 192.168.1.1 Protocol 0 Port 0
Local host: 10.64.10.9 Protocol 0 Port 0
Completion of Phase II.
1949 11/29/2001 16:20:18.540 SEV=4 IKE/49 RPT=3 y.y.205.92
Security negotiation complete
Responder, Inbound SPI = 0x11a56495, Outbound SPI = 0xb17718a5
Mar 10 2008 18:47:05: %PIX-5-713120: Group = y.y.41.250, IP = y.y.41.250, PHASE 2 COMPLETED (msgid=0f78e513)
Pre-shared key mismatch.
1754 11/29/2001 16:20:18.500 Group = 172.16.172.63, IP = 172.16.172.63, Received an un-encrypted PAYLOAD_MALFORMED notify message,
dropping.
Pre-shared key mismatch reported by the report peer(receiving peer):
:713903: Group = 172.1.12.1, IP = 172.1.12.1 ERROR. peer has indicated that something is wrong with our message. This could indicate a pre-shared
key mismatch.
Transform-set mismatch.
1754 11/29/2001 16:20:18.500 Group = 172.16.172.63, IP = 172.16.172.63, Received non-routine Notify message: No Proposal Chosen
Transform-set mismatch on remote peer(receiving peer):
713904 IP = 10.51.16.1, Received encrypted packet with no matching SA, dropping
713048: IP = 10.51.16.1 Error processing payload. Payload ID 1
The following indicates that the remote gateway is not finding matching interesting traffic.
1754 11/29/2001 16:20:18.500 Group = y.y.172.63, IP = y.y.172.63, Received non-routing Notify message: Invalid ID info (18)
The following indicates that the local gateway is not finding matching interesting traffic.
1754 11/29/2001 16:20:18.500 Group =y.y.172.63, IP = y.y.172.63, Static Crypto Map check, map = mymap, seq = 10, ACL does not match proxy IDs
src:192.168.1.0 dst:192.168.2.0
PFS mismatch:

713068: Group 172.1.12.1, Received non-rouing Notify message; No Proposal chosen (14)
PFS turned on on the remote peer. Local peer reports the following:
713902; Group = 10.51.16.1. QM FSM error (p2 struct &0x296fde8, mess id 0x518e80d)!
QM FSM is a generic message indicating that the phase II connection was rejected by the remote peer.
This indicated that the remote peer is natting:
%ASA-4-402116: IPSEC: Received an ESP packet (SPI= 0x72DEC2AA, sequence number= 0x41) from y.y.28.178 (user= y.y.28.178) to y.y.83.194. The
decapsulated inner packet doesn't match the negotiated policy in the SA. The packet specifies its destination as y.y.83.194, its source as y.y.28.178, and
its protocol as 1. The SA specifies its local proxy as y.y.10.16/255.255.255.240/0/0 and its remote_proxy as y.y.63.0/255.255.255.0/0/0.

When reverse route is turned on:

Jan 26 2009 18:15:07: %ASA-6-713211: Group =y.y43.160, IP = y.y.43.160, Adding static route for L2L peer coming in on a dynamic map. address:
192.168.8.5, mask: 255.255.255.255
Jan 26 2009 18:57:54: %ASA-6-713213: Group = y.y.43.160, IP =y.y43.160, Deleting static route for L2L peer that came in on a dynamic map. address:
192.168.8.5, mask: 255.255.255.255
Saturday, February 28, 2009
Checkpoint Tables and the FW Tab Command
The fw tab command displays the contents of the INSPECT tables. These tables hold all state information on the firewall, including connections, VPNS,
nats, etc.
The following options are commonly used:
-s provides a summary the tables
-t tname only displays the requested table
-x tname delete all entries in the specified table
-d debug mode
-all all tables

Useful tables:
host_table
This host_table holds the IP addresses of internal machines protected by the VPN-1/FireWall-1 NG Enforcement Module. The table only exists where the
VPN-1/FireWall-1 license is limited. The maximum number of entries in this table is the licensed number of internal machines.

arp_table
The arp_table holds the IP addresses for which the machine is willing to proxy ARP. Proxy ARP is sometimes required when using NAT. If IP addresses
that the machine is to resolve are specified in local.arp , this table can be used to check that the IP addresses were correctly specified. The automatic
ARP feature also uses this table to cause the machine to resolve translated IP addresses.

Entry format:

<!--[if !supportLists]-->1. <!--[endif]-->IP address (name resolving may occur).

<!--[if !supportLists]-->2. <!--[endif]-->MAC address of gateway machine interface that will answer ARP requests for the IP address.

<!--[if !supportLists]-->3. <!--[endif]-->Interface name (optional).

<!--[if !supportLists]-->4. <!--[endif]-->Name of the gateway interface that will proxy ARP for IP addresses.

The fwx_alloc table uses the following formats.

First entry: < 0, hiding IP address, IP protocol, first high port used; next high port to be allocated>

The first field is a space holder and is always 0. The first high port to be used is always 10000.

fwx_alloc
The fwx_alloc table holds information about the allocation of ports for the translated packets.

fwx_cntl_dyn_tab
The fwx_cntl_dyn_tab table holds information about the allocated IP addresses from the IP Pool of the Enforcement Module, and the connections using
the IP addresses.

EXAMPLE
attributes: keep, expires never, limit 25000,
hashsize 512, free function 40550248 0

<0a010104,>
00000001, 00000e10, 000000c3>
fwx_auth
The fwx_auth table holds the original information of a folded connection, so that the Security Server will know the original destination IP and port of the
connection.

EXAMPLE
attributes: expires 300, limit 25000, refresh, keep

c7cb47e3, 00000017; 286/300>

sam_blocked_ips
All IP and network addresses that were stipulated in SAM requests, are shown in sam_blocked_ips , with a requests counter for each prototype of filter
that is enforced over each certain IP and network address. Note the overshadowed requests are also accounted for, if present.

EXAMPLE
-------- sam_blocked_ips --------dynamic, id 8141, attributes: keep, limit 25000, hashsize 512 <05050505;> 00000000, 00000000,
00000000>

host_ip_addrs
The host_ip_addrs table contains the list of IP addresses in the VPN-1/FireWall-1 NG Enforcement Module.
fwx_ip_lookup_tab
The fwx_ip_lookup_tab table holds information used for IP Pool allocation queries.

EXAMPLE
attributes: keep, limit 25000, hashsize 512
fwz_crypt_pending
The fwz_crypt_pending table is used to record a possibly encrypted connection that should obtain their encryption/decryption key. This table is accessed
from the VPN kernel, and the VPN daemon. The kernel can obtain a new key from this table stored by the daemon, which negotiated the key exchange
with a trusted peer. This table also passes error messages. The fwz_crypt_pending table is dynamic.

forbidden_tab
Each embedded VPN-1/FireWall-1 system has a feature that indicates how many hosts can be located behind it. The number of hosts can be set to
unlimited. This limitation is enforced in the Inspect code using the macro COUNT_HOST .

COUNT_HOST records each packet that comes from the internal interface in a table until the limit is exceeded
IKE_SA_table
IKE SAs are stored in IKE_SA_table . The table entries have four possible formats:

<!--[if !supportLists]--> <!--[endif]-->The top two formats are only used on SecuRemote.

<!--[if !supportLists]--> <!--[endif]-->All non-expired SAs are stored using format 2.

<!--[if !supportLists]--> <!--[endif]-->Format 1 is used to store and retrieve the latest IKE SA.

<!--[if !supportLists]--> <!--[endif]-->Table entries are used to conduct IKE Quick Mode negotiation of IPSEC_SA .

<!--[if !supportLists]--> <!--[endif]-->Entries are extracted from this table when the vpn daemon is trapped for IPSEC_SA renewal. IKE daemon
tries to use previously negotiated IKE SA.

<!--[if !supportLists]--> <!--[endif]-->The IKE_SA_table is dynamic.

ATTRIBUTES
expires

3600

limit

25000

sync

keep

hashsize

512

kbuf

KEYS
PeerAddress ipaddr

IKE peer address.

Me

ipaddr

on SecuRemote it is the IP address used by SecuRemote when it negotiated this IKE SA; field is used to prevent using an old
IKE SA negotiated before SecuRemote obtained a new IP address; if initiator or responder

CookieI

u_long
[2]

initiator cookie (8 bytes). host byte order

CookieR

u_long
[2]

responder cookie (8 bytes). host byte order

VALUES
IKE_SA

kbuf A kbuf storing the fwisakmp_sa structure.

Flags

uint currently only 2 flags are defined: (PEER_MOBILE, INITIATOR); used so Enforcement Module does not need to retrieve the
whole kbuf from the kernel if we only want to know if this SA was established with a mobile user

RenegotiationTime uint renegotiation time of the SA

EXAMPLE
-------- IKE_SA_table -------dynamic, id 77, attributes: keep, sync, expires 3600,
limit 25000, hashsize 512, kbuf 1, free function
Cisco Anyconnect sample config
config t
webvpn
svc image disk0:/anyconnect-win-2.0.0343-k9.pkg 1
! this is a customerized vpn profile, if client does not needed, you can remove the following line using cisco default
! svc profiles VitalProf disk0:/vpn-vig-tdc.xml
tunnel-group-list enable
enable outside
svc enable
exit
ip local pool SSLClientPool 192.168.100.1-192.168.100.50 mask 255.255.255.0
access-list NONAT extended permit ip 192.168.5.0 255.255.255.0 192.168.100.0 255.255.255.0
access-list vpnssl-split extended permit ip 192.168.5.0 255.255.255.0 192.168.100.0 255.255.255.0
nat (inside) 0 access-list NONAT
username userA password test123
username userA attributes
service-type remote-access
exit
username userB password test12345
username userB attributes
service-type remote-access
exit
group-policy SSLCLientPolicy internal
group-policy SSLCLientPolicy attributes
dns-server value 192.168.1.51 192.168.1.61
wins-server value 192.168.1.51 192.168.1.61
address-pools value SSLClientPool
split-tunnel-policy tunnelspecified
split-tunnel-network-list value vpnssl-split
webvpn
vpn-tunnel-protocol svc
svc keep-installer installed

!svc profiles value VitalProf

exit
sysopt connection permit-vpn
tunnel-group SSLClientProfile type remote-access
tunnel-group SSLClientProfile general-attributes
default-group-policy SSLCLientPolicy
tunnel-group SSLClientProfile webvpn-attributes
group-alias SSLVPNClient enable
exit
wr mem
wr stand
debug command
sh vpn-sessiondb svc,
please be noticed, the default license for asa for web vpn or ssl vpn is only 2, you need to notify the client for this license limitation

Viewing all hidden commands on a Cisco ASA

The "show parser dump all" command will display all valid commands, including undocumented commands.
The full output of the command came out to over 300 pages in MS WORD in 11 font.
A few examples of hidden VPN commands are as follows:
clear/show ipsec sa counters
clear/show crypto ipsec sa map
clear/show crypto protocol statistics ssl
clear/show crypto protocol statistics ssh
clear/show crypto protocol statistics srtp
clear/show crypto protocol statistics other
clear/show crypto protocol statistics all
clear/show vpn-sessiondb statistics
clear/show vpn-sessiondb statistics
debug vpnlb
debug vpn-sessiondb
debug crypto isakmp timers
debug crypto ca messages
debug crypto condition peer
subnet
debug crypto condition peer
subnet
debug crypto condition peer
debug
debug
debug
debug

crypto
crypto
crypto
crypto

condition
condition
condition
condition

user
unmatched isakmp
error ipsec
error isakmp

http://www.netleets.com/search/label/tcpdump

Encryption Failure: Packet Is Dropped as There Is No Valid SA

You might see this error message when both ends of the VPN do not have the same definition for the encryption domain. First ensure that both
ends of the VPN are defined with the same encryption domain. If they are the same, you should create objects that are exactly the same size as
what is created on the remote end.
One annoying behavior FireWall-1 NG exhibits that FireWall-1 4.1 and earlier did not is the automatic simplification of subnets in IPSec SAs.
For example, if your encryption domain contains explicit objects for 192.168.0.0/24 and 192.168.1.0/24, Check Point would attempt to negotiate
an IPSec SA with 192.168.0.0/23 instead of generating SAs based on the network objects you created. To eliminate this behavior, use dbedit to
make the following changes on your management console (see FAQ 4.2 for details on editing objects_5_0.C):
dbedit> modify properties firewall_properties
ike_use_largest_possible_subnets false
dbedit> update properties firewall_properties
You must then reload the security policy for this change to take effect.

VPN Fails When Transferring Large Packets

Some applications set the Don't Fragment bit on certain packets. When the IPSec headers are added to the already large packet, the packet
requires fragmentation in order to pass through the firewall. When Check Point creates the IPSec packet, the Don't Fragment bit from the original
packet is maintained. FireWall-1 creates a fragmented packet that has the Don't Fragment bit set, so it cannot be fragmented and thus gets
dropped at the next router.
You can force FireWall-1 to clear the Don't Fragment bit by changing the ipsec_dont_fragment property in objects_5_0.C to false.
You can do this with the following commands in dbedit on the management console (craig is the firewall in this example):
dbedit> modify network_objects craig VPN:ipsec_dont_fragment
false
dbedit> update network_objects craig
Alternatively, you can use the GUIdbedit tool to change the parameter. Either way, you must then reinstall the security policy for this change to
take effect.

Debugging Interoperability Issues with IKE

Everyone has a different interpretation about how to follow standards. As a result, when third-party products talk to one another, communication
doesn't always work. One way to debug is to turn on IKE debugging.
In FireWall-1 4.1, it was necessary to stop and restart FireWall-1 in order to enable debugging. In NG, you can enable this on the firewall module
with a simple command: vpn debug ikeon. You can also disable it with vpn debug ikeoff. When you enable debugging,
$FWDIR/log/ike.elg gets created. This file contains the results of all IKE negotiations that occur. This file is a little difficult to read on its
own. Fortunately, Check Point has a tool called IKEView that allows you to view this file in a more readable form. Unfortunately, it is available
only to Check Point Certified Service Partners.

Check Point Logging Troubleshooting Guide

Are the logs being sent to the manager?
Ok, so first of all are the logs being sent to the Smart Centre Manager or the necessary Log Manager ? We can check this by
confirming whether the gateway is sending the log packets via the FW Log port tcp/257 upon the gateway and the manager. To
do this use either or both of the following commands,
netstat -an | grep 257 - This will show the state of the TCP sockets.
tcpdump -ni [interface name] port 257 - This will show a packet capture of the FW Log packets on the subsequent
interface.
If the gateway is not sending the logs then this can be down to one of the following issues,
1

SIC is not established.

The Logging configuration for the Gateway is not configured correctly.

The SmartCentre/Log Manager is not listening on port tcp/257.

There is an issue with FWD on the gateway. In some instances you may need to restart FWD via a cpstart. Though the
root cause could be down to a number of factors.

Why are the logs not being displayed within SmartView tracker ?
Ok so the manager is receiving the logs but you may still not see them within the
SmartView tracker this will be down to either the FWD (Firewall Daemon) or the log
files being corrupted.
Log Files Corrupted
If the log files are corrupted you should expect to see no logs within the SmartView
Tracker. If this is the case you will need to action the following steps :
1. Close the Log Viewer/SmartView Tracker and Policy Editor/SmartDashboard.
2. Execute the fwstop or cpstop command (depending on the version) from the
command line.
3. Remove all files starting with fw.log and fw.logptr from the $FWDIR\log
directory.
4. Execute the fwstart or cpstart (depending on the version) command.
Full details can be found at Check Points KB within Solution ID sk6432.
Only some of the logs are not being displayed
If only some of the logs are not being displayed then this could point to an issue with the trust between
the manager and the gateway.
To confirm the issue you will need to debug FWD using the following steps.
root@cp-mgnt# fw debug fwd on TDERROR_ALL_ALL=5
root@cp-mgnt# tail -f $FWDIR/log/fwd.elg
root@cp-mgnt# tail -f $FWDIR/log/fwd.elg | grep -i "Certificate is revoked"
root@cp-mgnt# fw debug fwd off
Within these steps we first enable the debug. Then we run a live tail on the log file. And then we run a
grep on the live tail for a specific error. The live tail allows us to view the end of the log file in real time.
We finally turn off the debug.
Below shows an example of an error with the SIC trust between the Gateway and Manager obtained
from the $FWDIR/log/fwd.elg,
[FWD 2177 1]@cp-mgnt[22 Jan 14:47:32] fwCert_ValCerts: Certificate is revoked. CN=cp-fw1,O=cp-mgnt..bizt7z
[FWD 2177 1]@cp-mgnt[22 Jan 14:47:41] fwCert_ValCerts: Certificate is revoked. CN=cp-fw2,O=cp-mgnt..bizt7z
In this instance resetting SIC would resolve this issue

Troubleshooting Checkpoint ClusterXL

I recently came across an issue where SmartView Monitor showed an error for ClusterXL on a freshly rebuilt Checkpoint IP565
firewall. Both Synchronization and Filter were stuck in an initilizing state, we tried the following troubleshooting steps initially to
no avail:
5

cphastop followed by cphastart

cpstop followed by cpstart

reboot of the affected firewall

On digging deeper we noticed that one of the firewall devices was configured to use multicast and one for broadcast cluster
communications, this was identified using the following command 'cphaprob -a if' which presents the following output:
eth-s1p3c0 non sync(non secured)
eth-s4p3c0 non sync(non secured)
eth-s4p4c0 non sync(non secured)
eth-s1p1c0 non sync(non secured)
eth-s1p4c0 sync(secured), multicast
eth-s1p2c0 non sync(non secured)
eth-s4p1c0 non sync(non secured)
eth-s4p2c0 non sync(non secured)
Virtual cluster interfaces: 7
eth-s1p3c0 xx.xx.xx.xx
eth-s4p3c0 xx.xx.xx.xx
eth-s4p4c0 xx.xx.xx.xx
eth-s1p1c0 xx.xx.xx.xx
eth-s1p2c0 xx.xx.xx.xx
eth-s4p1c0 xx.xx.xx.xx
eth-s4p2c0 xx.xx.xx.xx

Both firewalls must be configured to use the same method of communication, which can be
changed using the following command 'cphaconf set_ccp multicast' or 'cphaconf set_ccp
broadcast'. Providing your switching infrastructure supports multicast you should use this mode
due to the performance overhead of broadcast communication. This command failed to change
the method of communication and left us with no other option than to perform the following
steps:
1

Set Checkpoint Packages as in-active, then delete them ensuring that the Connectra package is removed first.

Re-install the Checkpoint R65 IPSO Wrapper

Re-install HFA 70

Re-establish SIC via CPConfig and SmartDashboard

Unassign and re-assign license via SmartUpdate

Push policy from the SmartDashboard

After performing thse steps the cluster CCP was back to multicast (bizare really...). We had to perform a reboot of the second
device once this was completed, at which point both nodes of the cluster reported no ClusterXL errors, 'cphaprob list' showed
the following output:
# cphaprob list
Registered Devices:
Device Name: Synchronization
Registration number: 0
Timeout: none
Current state: OK
Time since last report: 213003 sec
Device Name: Filter
Registration number: 1

Timeout: none
Current state: OK
Time since last report: 213003 sec
Device Name: cphad
Registration number: 2
Timeout: 5 sec
Current state: OK
Time since last report: 0.7 sec
Device Name: fwd
Registration number: 3
Timeout: 5 sec
Current state: OK
Time since last report: 0.5 sec
'fw ctl pstat' should also list the Synch as 'Able to Send/Receive sync packets' :
# fw ctl pstat
Machine Capacity Summary:
Memory used: 14% (90MB out of 637MB) - below low watermark
Concurrent Connections: 26% (17876 out of 67900) - below low watermark
Aggressive Aging is in monitor only
Hash kernel memory (hmem) statistics:
Total memory allocated: 200278016 bytes in 48894 4KB blocks using 2 pools
Initial memory allocated: 20971520 bytes (Hash memory extended by 179306496 bytes)
Memory allocation limit: 536870912 bytes using 10 pools
Total memory bytes used: 23487660 unused: 176790356 (88.27%) peak: 34170776
Total memory blocks used: 7126 unused: 41768 (85%) peak: 9164
Allocations: 1183931215 alloc, 0 failed alloc, 1183678473 free
System kernel memory (smem) statistics:
Total memory bytes used: 250335916 peak: 300842432
Blocking memory bytes used: 1865892 peak: 2596156
Non-Blocking memory bytes used: 248470024 peak: 298246276
Allocations: 160033475 alloc, 0 failed alloc, 160032829 free, 0 failed free
Kernel memory (kmem) statistics:
Total memory bytes used: 73389696 peak: 101169940
Allocations: 1184023246 alloc, 0 failed alloc, 1183769860 free, 0 failed free
External Allocations: 0 for packets, 0 for SXL
Kernel stacks:
0 bytes total, 0 bytes stack size, 0 stacks,
0 peak used, 0 max stack bytes used, 0 min stack bytes used,
0 failed stack calls
INSPECT:
1029526467 packets, -2128289516 operations, 373013811 lookups,
2035 record, 183665476 extract
Cookies:
-1649393933 total, 0 alloc, 0 free,
4607 dup, -1525329462 get, 138972711 put,
-1565092568 len, 217535 cached len, 0 chain alloc,
0 chain free
Connections:
54513276 total, 52537755 TCP, 1898998 UDP, 76506 ICMP,

17 other, 49485065 anticipated, 1 recovered, 17882 concurrent,

24286 peak concurrent
Fragments:
213594 fragments, 105472 packets, 389 expired, 0 short,
0 large, 0 duplicates, 0 failures
NAT:
23444077/0 forw, 29804768/0 bckw, 53234829 tcpudp,
14016 icmp, 702040-723136 alloc
Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 78286072, retransmitted : 16171, retrans reqs : 20, acks : 3
Sync packets received:
total : 17030603, were queued : 16591, dropped by net : 15
retrans reqs : 8840, received 3 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0

Check Point Firewall-1

Useful Firewall-1 command line utilities:
Unload current security policy
fw unloadlocal
VPN Tunnel command line access (e.g. delete SAs)
vpn tu
Display overlapping VPN Encryption Domains
vpn overlap_encdom [communities|traditional]
List current Firewall interfaces
fw ctl iflist
Show HA / ClusterXL state
cpstat ha
cphaprob state
cphastop / cphastart
Display State of Checkpoint HA Interfaces
cphaprob -a if
Stop/Start Checkpoint HA/ClusterXL
cphastop / cphastart
Display State of Checkpoint HA Interfaces

cphaprob -a if
Manually failover
cphaprob -d STOP -s problem -t 0 register
cphaprob list
cphaprob -d STOP unregister
Display State of ClusterXL IGMP
cphaprob stat (Notify if IGMP membership is supported)
cphaprob igmp (Display the current IGMP membership settings)

SmartCenter
Backup and Restore SmartCenter
upgrade_export
$FWDIR/bin/upgrade_tools/upgrade_import
Check whether licensed for management high availability (Management HA)
cplic check mgmtha

SecurePlatform
SecurePlatform configuration commands:
Configure Interfaces, Routes etc
sysconfig
Add static routes
config route add dest 192.168.1.0/24 via 192.168.0.1 dev eth0 metric 0 s-persistant on
apply on
Configure Network Interfaces
config conn help
config conn set name eth1 type eth onboot on iff-up on local 192.168.1.2/24 broadcast
192.168.1.255 s-persistant on s-code up mtu 1500
Configure Bonded Network Interfaces (NIC Team, 2 physical, 1 logical interface)
config conn add name bond0 type bond onboot on iff-up on mtu 1500 bond-mode activebackup bond-miimon 100 bond-downdelay 200 bond-updelay 200 bond-primary eth1 local
192.168.1.2/24
config conn add name eth1 type eth onboot on iff-up on mtu 1500 master-bond bond0
config conn add name eth4 type eth onboot on iff-up on mtu 1500 master-bond bond0

Useful SecurePlatform command line utilities:

Enter OS commands
expert
Assign interfaces to correct physical NICs
(Edit /etc/sysconfig/ethtab)
[Expert@FIREWALL]# cat ethtab
eth0 00:21:5A:27:DC:E6
eth1 00:21:5A:27:DC:E4
eth2 00:1F:29:5C:82:F5
Set Kernel parameters
(Edit $FWDIR/boot/modules/fwkern.conf)
fwha_mac_magic=011
fwha_mac_forward_magic=010
fwha_monitor_if_link_state=1
fwha_enable_igmp_snooping=1
fwha_igmp_version=2
Flag disconnected NICs
echo eth6 >> $FWDIR/conf/discntd.if
Show status of Bonded Network Interfaces
cphaconf show_bond -a
Display Versions
SPLAT: ver
Firewall: fw ver
Performance Pack: sim ver k
Linux: uname -a
Change shell to permit WinSCP connection
usermod -s /bin/bash fwadmin
Change shell timout (cpshell)
idle mm where mm = timeout in minutes (permanent change, updates /etc/cpshell/cpshell.state and is
passed on to expert shell)

Change shell timout (bash)

TMOUT = ss where ss = timeout in minutes
export TMOUT
Display the number of CPUs presented to SecurePlatform OS
grep physical id /proc/cpuinfo|wc -l
Display the CoreXL CPU Affinity
fw ctl affinity -l
Advanced Routing (gated) Commands
ps -eaf | grep gated
cpwd_admin list