VSX Troubleshooting: Quick Guide

VSX
Troubleshooting
Q u i c k g u i d e
CPUG 2010 Chur Switzerland (c) Valeri Loukine 2010

Agenda
How VSX is built (in brief)
Management scheme
Gateway architecture
Licensing
Issues to fix
Tools and methods
CPUG 2010 Chur Switzerland 2 (c) Valeri Loukine 2010

Reference Note
Pictures from Check Point publicly available

documents are used in this presentation
Information from Check Point troubleshooting

documentation used in this presentation

Management
Side
V S X A r c h i t e c t u r e

MGMT model of VSX
Three tear infrastructure
Two types
SmartCenter
Provider-1

SmartCenter Model
Nothing special

Provider-1 Model
Virtual Systems are managed by different

CMAs
Special so called “Main CMA” to manage

VSX cluster objects
“Target CMAs” to manage particular VSs

Provider-1 Model

MGMT DB objects
Two types instead of one as for regular FW
network_object
- security aspects of a Virtual Device
vs_slot_objects
- networking aspects of a Virtual Device
network_object - on the Target CMA
vs_slot objects - on Main CMA

vs_slot objects
Special DB table called vs_slot_objects
Network interfaces of VS
Routes of VS
Other VS specific attributes, such as

reference to hosting VSX object
Vital info for creation and change of VS

Network
Configuration Scripts
Configuration changes are pushed to VSX

from MGMT with NCS
NCS are generated on MGMT
On VSX GW NCS are parsed and executed

Types of NCS
local.vs - NCS file - last configuration

change

local.vsall - NCS file, full configuration,

executed on startup
local.vskeep - contains list of existing VSIDs,

used at system startup

local.vs
Interfaces lists of each vs_slot:
interfaces - interfaces’ configuration to be
interfaces_installed - existing interfaces
configuration
Each vs_slot object has 2 attributes containing routes

lists:
routes - routing table to be
routes_installed existing routing table
local.vs file is created by comparing and calculating

the difference of “interfaces” to “interfaces_installed”
and “routes” to “routes_installed”
local.vsall and
local.vskeep
Each Virtual Device has 2 NCS files kept on the
management:
VS_name.vsnew - NCS file containing interfaces
VS_name.vsrt - NCS file containing routes
These files are updated each time configuration

change is done using SmartDashboard.
local.vsall is created by concatenating all the files of

all the Virtual Devices.
local.vskeep is created by going over vs_slo_objects

table and writing all the Virtual Devices VSIDs to it.
Location of NCS
files
All .vsnew and .vsrt files - in $FWDIR/conf/
vs_repository/VSX_NAME of Main CMA
MGMT: local.vs, local.vsall and local.vskeep

files - in $FWDIR/state/VSX_NAME/VSX/
VSX GW: local.vs, local.vsall and

local.vskeep files in $FWDIR/state/__tmp/
VSX/
If scripts processed successfully, they copied
to to $FWDIR/state/local/VSX/ directory
Provider-1
forwarding

VSX private network
“Funny IPs”
For internal communication
Default cluster private network:

192.168.196.0/22
Cluster Private Network can be changed:

- In SmartDashboard, if there is no VDon
the VSX
- By using “vsx_util change_private_net”
gateway Side
V S X A r c h i t e c t u r e

Important note
VS is NOT a virtual machine
Common file space
Common kernel
VRFs and kernel contexts are different

VRFs
A Multiple routing domain (VRF) has separate:
VRF ID
Interfaces
Unicast routing table
Routing cache
Multicast forwarding cache
ARP table
Loopback interface
Sockets
VRFs enable overlapping IP addresses.

VRF: CLI changes
Mind context when working with ip route, iputils, ping,
traceroute, arp, route, ifconfig and netstat:
- traceroute –Z vrfid
- ip route vrf vrfid
- “-z vrfid” for the rest (arp –z 1, netstat –z 2 -rn)

Use “all” instead of “vrfid” to show information for
all VRFs
VRF context can be changed by “vsx set <vsid>” or

“vrfctl –s <vrf>” commands.
VRF 0 - physical machine

VSX user mode
Processes:
Multi context: fwd, cpd, cplogd and

fibmgr
Single context: vpnd and gated
System resources like CPU and HDD space

are shared

File Structure
VS folders - CTX under $FWDIR /

$CPDIR
CTX00xxx - Virtual Device ID xxx.
VSX machine (VSID 0) - $FWDIR /

$CPDIR

Creating VS object 1
License validation
Update context database with Virtual Device

information $CPDIR/conf/ctxdb.C)
Create Virtual Device registry entries (OTP for

SIC certificate)
Create Virtual Device directories and soft-links:
$CPDIR/CTX/CTX00xxx/conf
$FWDIR/CTX/CTX00xxx/log, database, …
Creating VS object 2
Create initial policy
Create the OS VRF instance
Create the VS instance in the FW kernel and

load security policy
Send a message that notifies cpd and fwd that

a new context was added. cpd adds the new
context to its db.

Troubleshooting
techniques

Useful knowledge
Management debugging (ref. P-1 lecture)
Gateway architecture and troubleshooting

techniques
ClusterXL
SecureXL

Things to Check
First
Licensing on both MGMT and GW sides
Connectivity between VSX and MGMT
All the jazz:
- local time settings

- static routes
- IP addressing - mind funny IPs
- etc...

Management
Debugging

Management Issues
Provisioning
Changes
vsx_util operations
policy installation

Important
Do not lock Main CMA while working with

VSX on Target CMAs

Debbuging fwm
TDERROR_ALL_ALL - might be too much
vsx provisioning and vsx_util:

TDERROR_ALL_VSXM
Policy installation:
TDERROR_ALL_INSTMGR

How to set debug
flags
Mind context!!!

fw debug fwm on
TDERROR_ALL_VSXM=INFO
Or
export TDERROR_ALL_VSXM=INFO
and restart fwm process

Debug output
$FWDIR/log/fwm.elg

Turning it off
fw debug fwm TDERROR_ALL_VSXM=0
fw debug fwm off

Which CMA?
Most of the cases - Main CMA
Policy installation - Target CMA

Gateway
Debugging

Common Issues
Connectivity
Policy
Interfaces
Clustering

To check first
Connectivity
Topology of VSX cluster and adjacent

networks
Local times
Licenses

Overvew
vsx stat -v
VSX Gateway Status
==================
Name: test1
Security Policy: Standard
Installed at: 25Jul2010 3:42:11
SIC Status: Trust

Number of Virtual Systems allowed by license: 25
Virtual Systems [active / configured]: 7 / 7
Virtual Routers and Switches [active / configured]: 0 / 0
Total connections [current / limit]: 4994 / 135000
Virtual Devices Status

======================
ID | Type & Name | Security Policy | Installed at | SIC Stat
-----+-------------------------+-------------------+-----------------+---------
1 | S test1_XXXXXXXXXXXX1...| Standard | 25Jul2010 3:42 | Trust

Tools
tcpdump -i <IF name>
fw monitor [–v <vsid>] -e <Your filter>

Example:
fw monitor –v 2 –e “port(80) and
ip_p=17, accept;”
Note: changing context does NOT help “fw

monitor” to limit output

Acceleration
Most of fw monitor output is accelerated, so

you will see just the first packet.
fwaccel [-vs <vsid>] [conns | templates | stat |

on | off]
Mind VS number

Cluster Issues
Get status
cphaprob [-vs vsid] stat
Interfaces status
cphaprob -a [-vs vsid] if
Problem notification list:

cphaprob [-vs vsid] list
Force member UP or DOWN, for failover tests:

clusterXL_admin up/down

Kernel Debug
One kernel for all!
Massive output, mind performance
Some express debugs:

fw ctl zdebug drop | grep <your filter> - to
see drop reason on specific traffic
Mind kernel buffer size

System Tools
arp, route, netstat, ifconfig - to have “-z X”

flag where X is VS number
-z all prints info for all VSs
ping -z...
traceroute -Z... (Capital letter)

VS Policy
To fetch the last installed policy:
fw [-vs <vsid>] fetch local
Fetching the last policy that failed to be installed

fw fetchlocal -d $FWDIR/state/__tmp/FW1/
To unload policy:
fw [-vs <vsid>] unloadlocal
Unload policy for all VSs:

fw vsx unloadall

VS configuration
To fetch configuration: fw vsx fetch
For specific VS: fw vsx fetchvs –vs 2
To see NCS script for a specific VS:

fw vsx showncs <vsid>

Other tips
Double check topology
If you cannot figure connectivity issues,

especially some traffic degradation, suspect
ClusterXL before others

Questions
And
Answers

Thank You For
Your Time!

VSX Troubleshooting: Quick Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

VSX Troubleshooting: Quick Guide

Uploaded by

Copyright:

Available Formats

VSX

CPUG 2010 Chur Switzerland (c) Valeri Loukine 2010

Tools and methods

CPUG 2010 Chur Switzerland 2 (c) Valeri Loukine 2010

Pictures from Check Point publicly available

Information from Check Point troubleshooting

CPUG 2010 Chur Switzerland 3 (c) Valeri Loukine 2010

CPUG 2010 Chur Switzerland (c) Valeri Loukine 2010

Three tear infrastructure

CPUG 2010 Chur Switzerland 5 (c) Valeri Loukine 2010

CPUG 2010 Chur Switzerland 6 (c) Valeri Loukine 2010

Virtual Systems are managed by different

Special so called “Main CMA” to manage

“Target CMAs” to manage particular VSs

CPUG 2010 Chur Switzerland 7 (c) Valeri Loukine 2010

CPUG 2010 Chur Switzerland 8 (c) Valeri Loukine 2010

network_object - on the Target CMA

vs_slot objects - on Main CMA

Other VS specific attributes, such as

Vital info for creation and change of VS

CPUG 2010 Chur Switzerland 10 (c) Valeri Loukine 2010

Configuration changes are pushed to VSX

NCS are generated on MGMT

On VSX GW NCS are parsed and executed

CPUG 2010 Chur Switzerland 11 (c) Valeri Loukine 2010

local.vs - NCS file - last configuration

local.vsall - NCS file, full configuration,

local.vskeep - contains list of existing VSIDs,

CPUG 2010 Chur Switzerland 12 (c) Valeri Loukine 2010

Each vs_slot object has 2 attributes containing routes

local.vs file is created by comparing and calculating

These files are updated each time configuration

local.vsall is created by concatenating all the files of

local.vskeep is created by going over vs_slo_objects

MGMT: local.vs, local.vsall and local.vskeep

VSX GW: local.vs, local.vsall and

CPUG 2010 Chur Switzerland 16 (c) Valeri Loukine 2010

For internal communication

Default cluster private network:

Cluster Private Network can be changed:

CPUG 2010 Chur Switzerland (c) Valeri Loukine 2010

VS is NOT a virtual machine

Common file space

VRFs and kernel contexts are different

CPUG 2010 Chur Switzerland 19 (c) Valeri Loukine 2010

VRFs enable overlapping IP addresses.

VRF context can be changed by “vsx set <vsid>” or

VRF 0 - physical machine

Multi context: fwd, cpd, cplogd and

Single context: vpnd and gated

System resources like CPU and HDD space

CPUG 2010 Chur Switzerland 22 (c) Valeri Loukine 2010

VS folders - CTX under $FWDIR /

CTX00xxx - Virtual Device ID xxx.

VSX machine (VSID 0) - $FWDIR /

CPUG 2010 Chur Switzerland 23 (c) Valeri Loukine 2010

Update context database with Virtual Device

Create Virtual Device registry entries (OTP for

Create Virtual Device directories and soft-links:

Create the OS VRF instance

Create the VS instance in the FW kernel and

Send a message that notifies cpd and fwd that

CPUG 2010 Chur Switzerland 25 (c) Valeri Loukine 2010