Best Practices - Security Gateway Performance PDF

28/10/2016 Best
Practices Security Gateway Performance
WWW.CHECKPOINT.COM BIENVENIDO: ERNESTO CABELLO | SALIR
WINNING THE SECURITY MARKET SALES TOOLS OFFERINGS / UPSELLS ASSETS / INFO SUPPORT / SERVICES
Support Center > Search Results > SecureKnowledge Details
Search Support Center
Best Practices Security Gateway Performance
Rate This My Favorites Email Print
Solution ID sk98348
Product Security Gateway, ClusterXL, Cluster 3rd party, VSX, CoreXL, SecureXL, Application Control, URL Filtering, AntiVirus, AntiBot, IPS, IPS / Web
Intelligence
Version All
Platform / All
Model
Date Created 11mar2014
Last Modified 04ago2016
Solution

Click Here to Collapse the Entire Article
Table of Contents (click on section titles to see subsections):
(1) Background
(2) Introduction and Limitations
(21) SecureXL
(22) CoreXL
(23) SMT (HyperThreading)
(24) MultiQueue
(3) Best practices
(31) Network interface cards
(32) Throughput
(33) SecureXL
(34) CoreXL
(35) SecureXL with CoreXL
(36) SMT (HyperThreading)
(37) MultiQueue
(38) Rulebase optimization
(39) IPS optimization
(310) Application Control & URL Filtering optimization
(311) AntiVirus & AntiBot optimization
(4) Initial diagnostics
(41) CPU
(42) Memory
(44) SecureXL
(45) CoreXL
(5) Advanced diagnostics
(51) CPU
(52) Memory
(54) SecureXL
(55) CoreXL
(6) Command Line syntax
(61) SecureXL
(61A) 'fwaccel' command
(61B) 'sim' command
(62) CoreXL
(62A) Gateway mode
(62B) VSX mode
(63) MultiQueue
(7) Examples
(8) Related documentation
(9) Related solutions
(10) Revision history
Organization of this article:
Chapter 1 "Background" provides a short background on performance of Security Gateway.
209.87.209.101
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk98348 1/48
28/10/2016 Best Practices Security Gateway Performance
Chapter 2 "Introduction" lists the relevant definitions, supported configurations, limitations, and commands specific to a product.
Chapter 3 "Best practices" provides the recommendations and guidelines for achieving the optimal performance.
Chapter 4 "Initial diagnostics" lists the basic commands and guidelines for checking the current utilization of machine's resources.
Chapter 5 "Advanced diagnostics" lists the advanced commands and guidelines for checking the current utilization of machine's resources.
Chapter 6 "Command Line syntax" provides the complete list of commands and their options specific to a product.
Chapter 7 "Examples" provides reallife examples.
Chapter 8 "Related documentation" lists the relevant documents.
Chapter 9 "Related solutions" lists the relevant solutions.
(1) Background
Performance of Security Gateway depends on:
CPU utilization / saturation / errors
Memory utilization / saturation / errors
Network Interfaces utilization / saturation / errors
Storage device I/O, capacity, controller utilization / saturation / errors
Throughput (packet rate * packet size)
Check Point solutions for improving the performance of Security Gateway:
SecureXL refer to sk98722 ATRG: SecureXL
CoreXL refer to sk98737 ATRG: CoreXL
SMT (HyperThreading) refer to sk93000 SMT (HyperThreading) Feature Guide
MultiQueue refer to Performance Tuning Administration Guide (R76, R77) and sk80940
ClusterXL refer to sk93306 ATRG: ClusterXL R6x and R7x
VPN refer to sk105119 Best Practices VPN Performance and to sk104760 ATRG: VPN Core
(2) Introduction and Limitations

(21) Introduction and Limitations SecureXL
Show / Hide introduction information
Definitions:
Performance Pack is a software acceleration product installed on Security Gateways. Performance Pack uses SecureXL technology and other innovative network acceleration
techniques to deliver wirespeed performance for Security Gateways. SecureXL is implemented either in software, or in hardware (SAM cards on Check Point 21000 appliances;
ADP cards on IP Series appliances).
Affinity Association of a particular network interface with a CPU core (either 'Automatic' (default), or 'Static' / 'Manual'). Interfaces are bound to CPU cores via SMP IRQ affinity
settings (refer to sk61962 SMP IRQ Affinity on Check Point Security Gateway).
Note: The default SIM Affinity setting for all interfaces is 'Automatic' the affinity for each interface is automatically reset every 60 seconds, and balanced between all available
CPU cores based on the current CPU load.
FWACCEL FireWall Accelerator (acceleration feature).
SIM SecureXL Implementation Module (acceleration device).
Connection offload Firewall kernel passes the relevant information about the connection from Firewall Connections Table to SecureXL Connections Table.
Note: In ClusterXL High Availability, the connections are not offloaded to SecureXL on Standby member.
Connection notification SecureXL passes the relevant information about the accelerated connection from SecureXL Connections Table to Firewall Connections Table.
Partial connection Connection that exists in the Firewall Connections Table, but not in the SecureXL Connections Table (versions R70 and above).
In Cluster HA partial connections are offloaded when member becomes Active
In Cluster LS partial connections are offloaded upon postsync (only for NAT / VPN connections)
Such connections must be offloaded to SecureXL, since packets in these connections must not be dropped.
If a packet matched a partial connection in the outbound, then it should be dropped.
Delayed connection Connection created from SecureXL Connection Templates without notifying the Firewall for a predefined period of time. The notified connections are deleted
by the Firewall.
Anticipated connection Connection that is anticipated by SecureXL based on policy rules to avoid dropping it by Drop Template.
In the following firewall policy:
NO. SOURCE DESTINATION VPN SERVICE ACTION
1 Any Any Any Traffic ftp Accept
2 Any Any Any Traffic Any Drop
A. Any FTP connection that will be opened will be accepted.
B. A SYN packet of a Telnet connection will be dropped and a Drop Template will be offloaded to SecureXL.
C. The FTP Data connection FTP might match this Drop Template and will be dropped.
Therefore, to prevent the drop of a legitimate connection:
A. SecureXL will offload such connections (in this example, FTP Data) and will mark them as anticipated.
B. SecureXL will try to match an anticipated connection to an existing connection or an existing Accept Template.
C. If such match was found, the packet will be forwarded to the firewall and will not be matched to a Drop Template. 209.87.209.101
Accept Template Feature that accelerates the speed, at which a connection is established by matching a new connection to a set of attributes. When a new connection matches the
Accept Template, subsequent connections are established without performing a rule match and therefore are accelerated. Accept Templates are generated from active connections
according to policy rules. Currently, Accept Template acceleration is performed only on connections with the same destination port (using wildcards for source ports).
Note: Size of Templates table (cphwd_tmpl, id 8111) is limited to 1/4 of the size of Firewall Connections Table (connections, id 8158).
Drop Template Feature that accelerates the speed, at which a connection is dropped by matching a new connection to a set of attributes. When a new connection matches the Drop
Template, subsequent connections are dropped without performing a rule match and therefore are accelerated. Currently, Drop Template acceleration is performed only on
connections with the same destination port (does not use wildcards for source ports). Drop Templates are generated from policy rules by special algorithm:
Analyze the rulebase
Produce mutually exclusive ranges
Offload the ranges to SecureXL
Once a packet is dropped, offload a Drop Template
All subsequent packets matching that range will be dropped by SecureXL
Accelerated path Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network.
Medium path (PXL) Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / HTTPS Inspection /
Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing (even when CoreXL is
disabled, the CoreXL infrastructure is used by SecureXL device to send the packet to the single FW instance that still functions).
Firewall path / Slow path (F2F) Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 SecureXL Mechanism). The packet is passed on to the
CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled.
Active Streaming (CPAS) Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed in order to understand the
application that is running (such as HTTP data). Active Streaming is Read and Writeenabled, and works as a transparent proxy. Connections that pass through Active Streaming
can not be accelerated by SecureXL.
Passive Streaming Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed in order to understand the application that
is running (such as HTTP data). Passive Streaming is Readonly and it cannot hold packets, but the connections are accelerated by SecureXL.
Passive Streaming Library (PSL) IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive
Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to sk95193 ATRG: IPS.
PXL Technology name for combination of SecureXL and PSL.
QXL Technology name for combination of SecureXL and QoS (R77.10 and above).
F2F / F2Fed Packets that can not be accelerated by SecureXL (refer to sk32578 SecureXL Mechanism) are Forwarded to Firewall.
F2P Forward to PSL/Applications. Feature that allows to perform the PSL processing on the CPU cores, which are dedicated to the Firewall.
SAM card Security Acceleration Module card (Acceleration Ready card). Connections that use SAM card, are accelerated by SecureXL and are processed by the SAM card's CPU
instead of the main CPU (refer to 21000 Appliance Security Acceleration Module Getting Started Guide)).
ADP card Accelerated Data Path card. Connections that use ADP card, are accelerated by SecureXL and are processed by network processors (NP) instead of the main CPU (refer
to sk60508 How to Configure ADP & SecureXL on IPSO).
IRQ Swizzling Traditionally, in a PCIe bus, all PCIe ports are mapped to one interrupt. Swizzling allows the PCIe slots to be balanced across four interrupts instead of one
(enabling IRQ Swizzling requires a BIOS update).
Supported operating systems:
Gaia OS
Gaia Embedded OS
SecurePlatform OS
SecurePlatform Embedded OS
IPSO OS
Crossbeam XOS
Crossbeam COS
Traffic flow:
Packet flow:
Flow logic:
209.87.209.101
Accelerated packet:
Packet matched to a template:
Nonaccelerated packet:
Limitations:
For limitations of traffic acceleration and templating, refer to sk32578 SecureXL Mechanism.
SIM Affinity is applicable only to physical interfaces (VLAN / Bond interfaces are influenced by the "physical" decision).
Note: Some systems may require BIOS update to enable IRQ Swizzling and EIRQ technologies. To determine whether a specific system supports the required technology, contact
your hardware vendor.
On VSX Gateway, WARP interfaces can not be assigned to a CPU core. 209.87.209.101
SecureXL NAT Templates are supported only in R75.40 and above (sk71200).
SecureXL Drop Templates are supported only in R76 and above (sk66402).
SecureXL Optimized Drops feature (used for DoS/DDoS protection) is supported only in R76 and above (sk90861).
SecureXL Penalty box feature (used for DoS/DDoS protection) is supported only in R75.40VS and above (on VSX Gateway, the penalty box would only be enforced on Virtual
System 0) (sk74520).
SecureXL does not support PointtoPoint interfaces (PPP, PPTP, PPPoE).
Note: In case a PPPinterface is detected, SecureXL disables itself on that interface (sk79880).
ClusterXL Sticky Decision Function (SDF) disables SecureXL.
QoS disables SecureXL (for R77.10 and above, refer to sk98229).
Delayed Synchronization in cluster:
Applies only to TCP services whose 'Protocol Type' is set to 'HTTP' or 'None'.
Delayed Synchronization is disabled if the 'Track' option in the rule is set to 'Log' or 'Account'.
Delayed Synchronization is performed only for connections matching a SecureXL Connection Template.
It is possible that a connection will exist in the Firewall Connections Table, but not in the SecureXL Connections Table (partial connection).
This situation can occur:
After policy installation
After cluster failover
If user turned off ('fwaccel off' command) and turned on ('fwaccel on' command) acceleration
Documentation:
Performance Pack Administration Guide (R75, R75.20, R75.40, R75.40VS).
Performance Tuning Administration Guide (R76, R77) Chapter 1 'Performance Pack'.
Command Line syntax:
FWACCEL (controls acceleration feature)
[Expert@HostName]# fwaccel <parameter> [‐h]
[Expert@HostName]# fwaccel6 <parameter> [‐h]
SIM (controls acceleration device)
[Expert@HostName]# sim <parameter> [‐h]
[Expert@HostName]# sim6 <parameter> [‐h]
(22) Introduction and Limitations CoreXL
Definitions:
CoreXL A performanceenhancing technology for Security Gateways on multicore processing platforms. CoreXL enhances Security Gateway performance by enabling the CPU
processing cores to concurrently perform multiple tasks.
Secure Network Distributor (SND) Traffic entering network interface cards (NICs) is directed to a processing CPU core running the SND, which is responsible for:
Processing incoming traffic from the network interfaces
Securely accelerating authorized packets (if SecureXL is enabled)
Distributing nonaccelerated packets among Firewall kernel instances (SND maintains global dispatching table which connection was assigned to which instance)
Firewall Instance / FW Instance On a Security Gateway with CoreXL enabled, the Firewall kernel is replicated multiple times. Each replicated copy, or Firewall Instance, runs on
one CPU processing core. These FW instances handle traffic concurrently, and each FW instance is a complete and independent Firewall inspection kernel. When CoreXL is enabled,
all the Firewall kernel instances on the Security Gateway process traffic through the same interfaces and apply the same security policy.
Affinity Association of a particular network interface / FW kernel instance / daemon with a CPU core (either 'Automatic' (default), or 'Manual').
Note: The default CoreXL interface affinity setting for all interfaces is 'Automatic' when SecureXL is installed and enabled.
Medium path (PXL) Packet flow when the packet is handled by the SecureXL device, except for IPS / Application Control / AntiVirus / AntiBot processing. The CoreXL layer
passes the packet to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.
Firewall path / Slow path (F2F) Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 SecureXL Mechanism). The packet is passed on to the
Supported operating systems:
Gaia OS
SecurePlatform OS
IPSO OS
Crossbeam XOS
Crossbeam COS
Architecture:
Example of default configuration for machine with 8 CPU cores:
209.87.209.101
Default number of CoreXL IPv4 FW instances:
Note: The real number of CoreXL FW instances depends on the current CoreXL license.
Number of Default number of Default number of

CPU CoreXL IPv4 Secure Network Distributors
cores FW instances (SNDs)
1 1 0
Note: CoreXL is disabled Note: CoreXL is disabled
2 2 2
4 3 1
6 20 [Number of CPU cores] 2 2
[Number of CPU cores] 4
More than 20 4
Note: However, no more than 30.
Relation between CoreXL IPv4 FW instances and CoreXL IPv6 FW instances:
(run the 'fw ctl multik stat ' command and 'fw6 ctl multik stat ' command on Security Gateway)
The number of IPv4 FW instances from a minimum of 2 to a number equal to the maximal number of CPU cores on the Security Gateway.
The number of IPv6 FW instances from a minimum of 2 to a number equal to the number of IPv4 FW instances.
The number of IPv6 FW instances cannot exceed the number of IPv4 FW instances.
The total number of IPv4 FW instances and IPv6 FW instances together cannot exceed 32.
CoreXL and ClusterXL:
Number of CoreXL FW instances must be identical on all members of the cluster because the state synchronization between members is performed per CoreXL FW instance (e.g.,
Instance #2 on Member_A can synchronize only with Instance #2 on Member_B).
Note: Member with higher number of CoreXL FW instances will enter the 'Ready' state. Refer to sk42096 Cluster member is stuck in 'Ready' state.
Limitations:
sk61701 CoreXL Known Limitations.
Documentation:
Firewall Administration Guide (R75, R75.20, R75.40, R75.40VS) Chapter 'CoreXL Administration'.
Performance Tuning Administration Guide (R76, R77) Chapter 'CoreXL Administration'.
[Expert@HostName]# fw ctl multik <parameter>
[Expert@HostName]# fw ctl affinity [‐flag]
(23) Introduction and Limitations SMT (HyperThreading)
Definitions:
SMT (Simultaneous MultiThreading Intel® HyperThreading, or Intel® HT) is a feature that is supported on Check Point appliances running Gaia OS. When enabled, SMT
doubles the number of logical CPUs on the Security Gateway, which enhances physical processor utilization. When SMT is disabled, the number of logical CPUs equals the number
of physical cores.
SMT improves performance up to 30% in NGFW software blades such as IPS, Application & URL Filtering and Threat Prevention by increasing the number of CoreXL FW
instances based on the number of logical CPUs.
Supported configurations:
SMT is supported by R77 release and later.
SMT is supported only on Security Gateways running Gaia OS with 64bit kernel.
SMT is supported only on Check Point appliances.
Limitations:
SMT is relevant only if CoreXL is enabled.
Notes:
When SMT (HyperThreading) is enabled, the number of available CPU cores is not linear.

209.87.209.101
Example:
CPU core 0‐5 ‐ socket 1
CPU core 6‐11 ‐ socket 2
CPU core 12‐15 ‐ socket 1 HyperThreading
CPU core 16‐23 ‐ socket 2 HyperThreading
Documentation:
sk93000 SMT (HyperThreading) Feature Guide
(24) Introduction and Limitations MultiQueue
Background:
Today, each network interface card has one traffic queue that is handled by one CPU at a time. Since the Secure Network Distributor (SND) SecureXL and CoreXL Distributor is running
on the CPU cores that handle the traffic queues, user cannot use more CPU cores for acceleration than the number of network interface cards passing the traffic.
Definitions:
MultiQueue is an acceleration feature that lets the user configure more than one traffic queue for each network interface card, which allows using more CPU cores for acceleration.
rx queue Receive packet queue.
tx queue Transmit packet queue.
Secure Network Distributor (SND) Traffic entering network interface cards (NICs) is directed to a processing CPU core running the SND, which is responsible for:
Processing incoming traffic from the network interfaces
Securely accelerating authorized packets (if SecureXL is enabled)
Distributing nonaccelerated packets among Firewall kernel instances
IRQ affinity Process of binding a network interface card's IRQ to one or more CPU cores.
Medium path (PXL) Packet flow when the packet is handled by the SecureXL device, except for IPS / Application Control / AntiVirus / AntiBot processing. The CoreXL layer
passes the packet to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled.
Firewall path / Slow path Packet flow when the SecureXL device is unable to process the packet (refer to sk32578 SecureXL Mechanism). The packet is passed on to the
Supported configurations:
MultiQueue is integrated into R76, R77 and above. For lower versions, a MultiQueue hotfix has to be installed (refer to sk80940).
MultiQueue is supported on Check Point Appliances (including IP Series Appliances) and on Open Servers.
MultiQueue is supported only on machines that run SecurePlatform OS or Gaia OS.
MultiQueue is supported only for network interface cards that use igb (1 GbE) and ixgbe (10 GbE) drivers.
Limitations:
MultiQueue is relevant only if SecureXL is enabled.
MultiQueue is relevant only if CoreXL is enabled.
Multiqueue is not supported on machines with a single CPU core.
MultiQueue allows to configure a maximum of 5 interfaces (due to IRQ limitations).
The number of traffic queues is limited by the number of CPU cores and the type of network interface card driver:
igb driver up to 4 RX queues
ixgbe driver up to 16 RX queues
Documentation:
Performance Tuning Administration Guide (R76, R77) Chapter 3 'Multiqueue'.
sk80940 MultiQueue hotfix for Security Gateway.
[Expert@HostName]# cpmq <parameter>
(3) Best practices

(31) Best practices Network interface cards
Show / Hide best practices information
Refer to the list of Certified Network Interfaces.
You should use the PCI Express (PCIe) cards, because they have better performance than PCIX cards.
If you are using a motherboard with multiple PCI or PCIX buses, make sure that each Network Interface Card is installed in a slot connected to a different bus.
If you are using more than two Network Interface Cards in a system with only two 64bit/66Mhz PCI buses, make sure that the leastused cards are installed in slots connected to the
same bus.
If the traffic rate is very high and causes high amount of SoftIRQ (and traffic drops on interfaces), consider increasing the sizes of receive/transmit buffer on network interface cards
as described in sk42181 How to increase sizes of buffer on SecurePlatform/Gaia for Intel NIC and Broadcom NIC.
(32) Best practices Throughput
209.87.209.101
Setting the maximal number of concurrent connections:
Versions R75.40VS, R75.45 and above
1. SmartDashboard open Security Gateway object.
2. Go to 'Optimizations' pane.
3. The 'Calculate the maximum limit for concurrent connections ' should be set to 'Automatically'.
Note: Manual limit should be set only for security reasons.
4. Enter the desired value.
5. The 'Calculate connections hash table size and memory pool ' should be set to 'Automatically'.
6. Click on 'OK'.
7. Install the policy.
Versions R75.40 and lower
1. SmartDashboard open Security Gateway object.
2. Go to 'Optimizations' pane.
3. In the 'Calculate the maximum limit for concurrent connections
', select the 'Manually' (recommended setting is 'Automatically').
4. Enter the desired value.
5. The 'Calculate connections hash table size and memory pool
' should be set to 'Automatically'.
6. Click on 'OK'.
Notes:
You should ensure that the total number of concurrent connections is appropriate to the TCP end timeout.
Too many concurrent connections will adversely affect the Security Gateway's performance.
You can calculate the maximum number of concurrent connections by multiplying the session establishment rate by the TCP session timeout:
[MAXIMAL number of concurrent connections] = [MAXIMAL session establishment rate] x [TCP entire session timeout]
Notes:
This formula is used to understand what number of concurrent connections a session rate test will generate.
It is very hard to predict the maximal connections capacity of a Security Gateway because of multiple varying factors.
Administrator should monitor the memory utilization on Security Gateway after changing these settings.
Only the maximal (possible) session rate should be considered.
This formula will not predict connections capacity, which is stated in Check Point datasheet documents.
Average session rate can be obtained using the sk101878 CPView Utility from 'Traffic' tab.
Additional information can be obtained based on sk67560 How to export History Report from SmartView Monitor from 'Traffic' view.
TCP timeout varies highly between applications and protocols (e.g., SSH is usually long, HTTP is usually short).
By reducing the following timeouts, you increase the capacity of actual TCP and UDP connections (SmartDashboard 'Policy' menu 'Global Properties' 'Stateful
Inspection'):
TCP end timeout determines the amount of time a TCP connection will stay in the FireWall Connections Table (id 8158) after a TCP session has ended.
UDP virtual session timeout determines the amount of time a UDP connection will stay in the FireWall Connections Table (id 8158) after the last UDP packet was seen by
the Security Gateway.
Optimal connection/sec rate:
1. SmartDashboard go to 'Policy' menu click on 'Global Properties'.
2. Go to 'FireWall' pane.
3. Uncheck the following boxes:
Accept RIP
Accept Domain Name over UDP (Queries)
Accept Domain Name over TCP (Zone Transfer)
Accept ICMP requests
4. Click on 'OK'.
ARP cache table:
Increase the number of entries in the ARP cache table:
If you are testing large subnets that are directly connected to the Security Gateway without a router.
If 'kernel: neighbour table overflow ' appears repeatedly in /var/log/messages files and in the output of the 'dmesg' command.
Related solutions:
sk43772 'kernel: neighbour table overflow' appears repeatedly in /var/log/messages files
sk92372 How to change the size of IPv6 Neighbors cache table
NAT session rate:
For Security Gateway versions R75.40 and above:
Enable SecureXL NAT Templates per sk71200 SecureXL NAT Templates.
For Security Gateway versions R75.30 and lower:
1. Disable SecureXL.
2. And change one of the following:
Either decrease the TCP end timeout to 2 seconds (SmartDashboard 'Policy' menu 'Global Properties' 'Stateful Inspection
' ).
Or increase the hash size of SND's Connection Table by increasing (permanently per sk26202) the value of the kernel
parameter fwmultik_gconn_tab_hsize from the default 524288 to 8388608 and rebooting the machine.
Important Note: This change reduces the capacity for the maximum number of concurrent connections.
(33) Best practices SecureXL
Note: This section does not take the CoreXL into consideration.
BIOS settings:
209.87.209.101
If your BIOS supports CPU clock setting, then make sure that the BIOS is set to the actual CPU speed (no overclocking).
If your BIOS supports HyperThreading, then refer to "SMT (HyperThreading)" section.
Network Interface Cards:
Refer to the list of Certified Network Interfaces.
You should use the PCI Express (PCIe) cards, because they have better performance than PCIX cards.
If you are using a motherboard with multiple PCI or PCIX buses, make sure that each Network Interface Card is installed in a slot connected to a different bus.
If you are using more than two Network Interface Cards in a system with only two 64bit/66Mhz PCI buses, make sure that the leastused cards are installed in slots connected to the
same bus.
Network interface affinity:
Each NIC should be bound (affined) to a separate CPU core using the 'Static' affinity mode (run the 'sim affinity s' command).
Pairs of interfaces carrying significant data flows (based on network topology) should be assigned to pairs of CPU cores on the same physical CPU processor.
Pairs of interfaces that serve the same connections (based on network topology) should be assigned to pairs of CPU cores on the same physical CPU core.
For systems with 4 CPU cores and Dual Port NICs, the IRQ Swizzling technology should be enabled to properly distribute IRQs among 4 CPU cores.
Note: Applies only to Dual Port NICs. IRQ Swizzling is not required with Quad Port NICs.
For systems with 8 CPU cores and Quad Port NICs, the EIRQ technology should be enabled to properly distribute IRQs among all 8 CPU cores.
Note: At the time of this writing, there is no certified platform with this ability.
NAT session rate:
For Security Gateway versions R75.40 and above:
Enable SecureXL NAT Templates per sk71200 SecureXL NAT Templates.
For Security Gateway versions R75.30 and lower:
1. Disable SecureXL.
2. And change one of the following:
Either decrease the TCP end timeout to 2 seconds (SmartDashboard 'Policy' menu 'Global Properties' 'Stateful Inspection
' ).
Or increase the hash size of SND's Connection Table by increasing (permanently per sk26202) the value of the kernel
parameter fwmultik_gconn_tab_hsize from the default 524288 to 8388608 and rebooting the machine.
Important Note: This change reduces the capacity for the maximum number of concurrent connections.
Accelerated Drop Rules:
Available in R75.40 and above.
Accelerated Drop Rules protect the Security Gateway and site from Denial of Service attacks by dropping packets at the acceleration layer (SecureXL).
The drop rules are configured in a file on the Security Gateway (using the 'sim dropcfg <options> ' command), which is then offloaded to the SecureXL device for
enforcement.
Note: There is no relation between the rules being configured in the file and the rules configured in SmartDashboard.
Refer to sk67861 Accelerated Drop Rules Feature in R75.40 and above.
Optimized Drops:
Available in R76 and above.
Allows SecureXL to accelerate dropped traffic.
Once the feature is enabled, dropped traffic is accelerated depending on traffic drop rate per second. Once drop rate matches the thresholds, feature is dynamically
activated/deactivated.
Refer to sk90861 Optimized Drops feature in R76 and above.
Delayed Synchronization in cluster:
To decrease the load on CPU in cluster environment:
Either disable the synchronization of noncritical connections (e.g., UDP DNS, ICMP).
Or (if connection must be synchronized) start synchronizing the connection only some time after its initiation (rightclick on the service 'Edit...' 'Advanced...'
check the box 'Start synchronizing [X] seconds after connection initiation ' install policy).
Notes:
Applies only to TCP services whose 'Protocol Type' is set to 'HTTP' or 'None'.
Delayed Synchronization is disabled if the 'Track' option in the rule is set to 'Log' or 'Account'.
Delayed Synchronization is performed only for connections matching a SecureXL Connection Template.
(34) Best practices CoreXL
Note: This section does not take the SecureXL into consideration.
Notes:
CoreXL improves performance with almost linear scalability in the following scenarios:
Much of the traffic can not be accelerated by SecureXL
Many IPS features enabled, which disable SecureXL functionality
Large rulebase
NAT rules
CoreXL will not improve performance in the following scenarios:
SecureXL accelerates much of the traffic
Traffic mostly consists of VPN
Traffic mostly consists of VoIP
It is very important to verify that the CPU cores are equally utilized (run the 'top' command). If this is not the case, you should consider changing the distribution of the Secure
Network Distributors (SNDs) and CoreXL FW instances.
209.87.209.101
Under normal circumstances, it is not recommended for the SND and a CoreXL FW instance to share the same CPU core.
However, it is necessary in the following cases:
When using a machine with exactly two cores. It is better for both SNDs and CoreXL FW instances to share CPU cores, instead of allocating only one CPU core to each.
When you know that almost all of the packets are being processed in the Accelerated path, and you want to assign all CPU cores to this path. If the CoreXL FW instances do
not process significant amount of traffic, then it is appropriate to share the CPU cores.
The interfaces affinity should be configured only to to CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances), with these exceptions:
On machines with exactly two CPU cores (both SNDs and CoreXL FW instances use the same CPU cores).
For tests, in which traffic is accelerated by SecureXL (if it is enabled).
If you replaced the CPU on the machine to a CPU with more/less CPU cores than the previous CPU, then you must reconfigure the number of CoreXL FW instances in the
'cpconfig' menu.
In a cluster environment, changing the number of CoreXL FW instances should be treated as a version upgrade member with higher number of CoreXL FW instances will enter
the 'Ready' state. Refer to sk42096 Cluster member is stuck in 'Ready' state.
To change the number of CoreXL FW instances, schedule a maintenance window and follow either a Minimal Effort Upgrade procedure, or a Zero Downtime Upgradeprocedure
from the Installation and Upgrade Guide (R75, R75.20, R75.40, R75.40VS, R76, R77 Gaia, R77 NonGaia).
Changing the distribution of the SND, CoreXL FW instances, and daemons among the CPU cores:
Note: Run the 'fw ctl affinity l r a v
' command to see the current distribution.
To change the distribution of the SND, CoreXL FW instances, and daemons, change the current affinities of interfaces and/or of daemons.
Notes:
To make the affinity settings persistent, edit the '$FWDIR/conf/fwaffinity.conf ' file on Security Gateway (see the contents of the file for the correct syntax).
To apply the configuration from the '$FWDIR/conf/fwaffinity.conf ' file onthefly (without reboot), execute the '$FWDIR/scripts/fwaffinity_apply ' shell
script on Security Gateway (see the contents of the script for available flags).
To ensure CoreXL's efficiency, all traffic must be directed to CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances).
Therefore, if you change affinities of interfaces and/or daemons, you will need to accordingly set the number of CoreXL FW instances and ensure that the CoreXL FW instances run
on other CPU cores.
It is recommended to allocate an additional CPU core to the SND only if all of the following conditions are met:
Your platform has at least 8 CPU cores.
The 'idle' value (run 'top' command and press 1 to display all CPU cores) for the CPU core currently running the SND is in the 0%5% range.
The sum of the 'idle' values (run the 'top' command and press 1 to display all CPU cores) for the CPU cores running CoreXL FW instances is significantly higher than
100%.
If any of the above conditions are not met, the default configuration of one processing core allocated to the SND is sufficient, and no further configuration is necessary.
It is recommended to allocate an additional CPU core to the FWD daemon if the Security Gateway is performing heavy logging.
Note: Avoiding the CPU cores that are running the SND is important only if these CPU cores are explicitly defined as affinities of interfaces. If interface affinities are set to
'Automatic', any CPU core that is not running a CoreXL FW instance can be used for the FWD daemon, and interface traffic will be automatically diverted to other CPU cores.
Example of configuration for machine with 8 CPU cores:
(35) Best practices SecureXL with CoreXL
Background:
When most of the traffic is accelerated by SecureXL (run 'fwaccel stats s' command), the load on CPU cores that run as Secure Network Distributor (SND) can be very high, while
the load on CPU cores that run CoreXL FW instances can be very low. This is an inefficient utilization of CPU capacity.
Notes:
Traffic is processed by the CoreXL FW instances only when the traffic is not accelerated by SecureXL (if SecureXL is installed and enabled).
With CoreXL, there are cases when performance without SecureXL is better than with it, even when SecureXL does manage to accelerate part of the traffic.
Packet flow:
When SecureXL is enabled, a packet enters the Security Gateway and first reaches the SecureXL device. The packet will be handled in one of three ways:
1. Accelerated path The packet is completely handled by the SecureXL device. It is processed and forwarded to the network.
209.87.209.101
2. Medium path (PXL) The packet is handled by the SecureXL device, except for IPS / Application Control / AntiVirus / AntiBot processing. The CoreXL layer passes the packet
to one of the CoreXL FW instances to perform the processing.
This path is available only when CoreXL is enabled.
3. Firewall path / Slow path The SecureXL device is unable to process the packet (refer to sk32578 SecureXL Mechanism). The packet is passed on to the CoreXL layer and then to
one of the CoreXL FW instances for full processing.
This path also processes all packets when SecureXL is disabled.
Default affinity settings for interfaces:
If SecureXL is enabled the default affinities of all interfaces are 'Automatic' the affinity for each interface is automatically reset every 60 seconds, and balanced between
available CPU cores based on the current load.
If SecureXL is disabled the default affinities of all interfaces are with available CPU cores those CPU cores that are not running a CoreXL FW instance or not defined as the
affinity for a daemon.
Setting interface affinities:
If SecureXL is enabled the affinities of all interfaces are handled by the 'sim affinity' command. Interface affinities are automatically distributed among CPU cores that
are not running CoreXL FW instances and that are not set as the affinity for any daemon.
If SecureXL is disabled interface affinities are loaded at boot based on the $FWDIR/conf/fwaffinity.conf
configuration file (if SecureXL is enabled, then lines beginning
with "i" in this file are ignored).
To balance the load on the CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances):
A. Check the amount of interrupts from each interface run:
[Expert@HostName]# cat /proc/interrupts
B. Check, which CPU cores (that run as SNDs) are most utilized run:
[Expert@HostName]# top
Note: Press digit 1 (above the letter Q) to display all CPU cores and press Shift+W to save this configuration.
C. Assign the interfaces accordingly across the CPU cores that run as SNDs run:
[Expert@HostName]# sim affinity s
(36) Best practices SMT (HyperThreading)
Notes:
SMT is not recommended if only FireWall/VPN blades are used, because performance improvement by SMT is achieved on NGFW software blades.
SMT is not recommended for environments that have high memory utilization.
Reason: Firewall consumes memory for traffic inspection. If the memory utilization is already very high before enabling the SMT, the performance will decrease
noticeably because SMT adds more CoreXL FW instances.
To check the extent of memory utilization on the Security Gateway, refer to:
Initial diagnostics Memory
Advanced diagnostics Memory
SMT is not recommended if these blades/features are enabled:
Data Loss Prevention blade 209.87.209.101
AntiVirus in Traditional Mode
Using Services with Resources in Firewall policy
Reason: Each of these blades might have high memory consumption. These blades run Security Servers that are executed per CoreXL FW instance. Since SMT adds
more CoreXL FW instances, overall memory consumption on Security Gateway might increase considerably.
To check the extent of memory utilization on the Security Gateway, refer to:
Initial diagnostics Memory
Advanced diagnostics Memory
SMT is not recommended for environments that use Hide NAT extensively.
Reason: An entire range of ports for Hide NAT is divided between the current CoreXL FW instances. The more CoreXL FW instances are running, the less ports for
Hide NAT will be available for each CoreXL FW instance. As a result, if one CoreXL FW instance is handling a high number of NATed connections, its port range
may get exhausted, while at the same time, other CoreXL FW instances may have enough available ports for Hide NAT.
The port distribution is based on the following factors:
Number of CoreXL FW instances
Whether cluster is enabled
Whether SecureXL is enabled
Whether VPN blade is enabled
To check the extent of NAT on the Security Gateway, use the cpsizeme tool (refer to sk88160):
1. Run the tool for at least 24 hours.
2. Run the 'cpsizeme S' command.
3. Select "Show summary of last successful session
" (to see the summary of the collected statistics).
4. Check the "Estimated average of NAT connections:
" counter.
If you have two ports on 12600/12700/12800, 13500/13800 and 21000 appliances to handle most of the traffic, it is also recommended to enable MultiQueue feature to increase
SMT performance (refer to Performance Tuning Administration Guide (R76, R77) Chapter 3 'Multiqueue').
Documentation:
(37) Best practices MultiQueue
When MultiQueue will not help:
When most of the processing is done in CoreXL either in the Medium path, or in the Firewall path (Slow path).
All current CoreXL FW instances are highly loaded, so there are no CPU cores that can be reassigned to SecureXL.
When IPS, or other deep inspection Software Blades are heavily used.
When all network interface cards are processing the same amount of traffic.
When all CPU cores that are currently used by SecureXL are congested.
When trying to increase traffic session rate.
When there is not enough diversity of traffic flows. In the extreme case of a single flow, for example, traffic will be handled only by a single CPU core. (Clarification: The more
traffic is passing to/from different ports/IP addresses, the more you benefit from MultiQueue. If there is a single traffic flow from a single Client to a single Server, then Multi
Queue will not help.)
Guidelines for configuring MultiQueue:
Network interface cards must support MultiQueue.
MultiQueue is relevant only if SecureXL is enabled.
MultiQueue is relevant only if CoreXL is enabled.
Examine the CPU affinity (for interfaces (SND) and for CoreXL FW instances).
Examine the CPU utilization.
Decide if more CPU cores can be allocated to the Secure Network Distributor (SND).
MultiQueue is recommended when all these conditions occur:
Load on CPU cores that run as SND is high (idle < 20%).
Load on CPU cores that run CoreXL FW instances is low (idle > 50%).
There are no CPU cores left to be assigned to the SND by changing interface affinity.
Number of active RX queues:
By default on a Security Gateway, the number of active RX queues is calculated by:

Number of Active RX queues = [Total Number of CPU cores] [Number of CoreXL FW instances]
By default on a VSX gateway, the number of active RX queues is calculated by:
Number of Active RX queues = The lowest CPU ID, to which the fwk process on VSX is assigned
Notes:
For best performance, it is not recommended to assign both SND and a CoreXL FW instance to the same CPU core.
Do not change the IRQ affinity of queues manually. Changing the IRQ affinity of the queues manually can adversely affect performance.
Documentation:
sk80940 MultiQueue hotfix for Security Gateway.
(38) Best practices Rulebase optimization
209.87.209.101
1. Refer to sk32578 SecureXL Mechanism to allow more connections to be accelerated by SecureXL.
2. Place most used rules at the top use the Hit Count in the SmartDashboard (R75.40 and above) and SmartView Monitor ('Top' view).
Related solution: sk72860 How to reset the 'Hit Count' in SmartDashboard.
3. In addition, you can provide a full Connections Table from your Security Gateway to Check Point Support for thorough analysis (run the command 'fw tab t connections
u > /var/log/Connections_Table.txt ').
(39) Best practices IPS optimization
1. Create a dedicated IPS profile for each Security Gateway (to finetune the performance of each individual Gateway).
2. Note that protections are categorized by performance impact. As such, enabling protections with "Critical" performance impact will highly increase CPU consumption.
3. Enable / Disable protections depending on your network devices and your needs (do not enable all the available protections).
4. Avoid setting protections to run in "Detect" mode it might increase CPU consumption (without increasing the security).
5. Identify the protections that consume most of the CPU resources follow sk43733 How to measure CPU time consumed by IPS protections.
6. Disable protections that are not needed in your environment. To check, which protections are needed, you may use vulnerability tools (such as Nessus).
7. Set Protection Scope to "Protect internal hosts only" (SmartDashboard Security Gateway properties IPS pane).
Related solutions:
sk92527 Traffic rate through Security Gateway is decreased significantly when assigning any IPS profile other than 'Default_Protection'.
(310) Best practices Application Control & URL Filtering optimization
Application Control & URL Filtering policy:
1. Create as specific rules as possible, to prevent the unwanted traffic from hitting the wrong rule.
Limit the scope of the rule by selecting only the relevant services (instead of the default 'Any') rightclick on the column titles click on the 'Service' column:
2. Avoid using "Any" in "Source" and "Destination" columns.
3. In the "Destination" column, use the default "Internet", unless a specific destination has to be explicitly selected (e.g., in case of internal servers).
4. There is no need for "Clean Up" rule at the bottom (because the behavior of Application Control & URL Filtering Blades is different from that of the Firewall Blade).
Such rule should only be used for short period of time and only for logging purposes.
5. Remove the "Any Any Allow" rule, if such rule exists (because the behavior of Application Control & URL Filtering Blades is different from that of the Firewall Blade).
Advanced Engine Settings:
1. Go to "Check Point Online Web Service" section.
2. In the "Website categorization mode" section, select "Background" (to prevent latency due to packet holding until categorization is completed).
Related solutions:
(311) Best practices AntiVirus & AntiBot optimization
AntiVirus & AntiBot policy:
1. Create as specific rules as possible, to prevent the unwanted traffic from hitting the wrong rule.
2. Avoid using "Any" in "Source" and "Destination" columns.
3. In the "Destination" column, use the default "Internet", unless a specific destination has to be explicitly selected (e.g., in case of internal servers).
4. There is no need for "Clean Up" rule at the bottom (because the behavior of AntiVirus & AntiBot Blades is different from that of the Firewall Blade).
209.87.209.101
5. Remove the "Any Any Allow" rule, if such rule exists (because the behavior of AntiVirus & AntiBot Blades is different from that of the Firewall Blade).
Advanced Engine Settings:
1. Go to "Check Point Online Web Service" section.
2. In the "Website categorization mode" section, select "Background" (to prevent latency due to packet holding until categorization is completed).
Decreasing performance impact:
1. Edit the AntiVirus & AntiBot profile:
A. Go to 'Profiles' pane.
B. Select the relevant profile click on 'Edit...' button.
C. Go to 'General Properties
' pane.
D. In the 'Performance Impact
' field, select 'Low'.
E. Click on 'OK'.
F. Save the changes: go to 'File' menu click on 'Save'.
G. Install the policy onto the relevant Security Gateway / Cluster object.
2. Exclude networks:
Consider excluding networks, whose traffic does not have to be inspected follow sk92515 How to configure AntiVirus Exceptions.
Note: Standard exceptions are still being inspected (i.e., CPU is consumed), however the traffic will be allowed. The exceptions per sk92515 are completely excluded from
the AntiVirus & AntiBot engine inspection (i.e., decrease the load on CPU).
3. Decrease the CPU consumption by RAD process and DLPU process:
Consider disabling MD5 check and GZIP inspection follow sk94056 When AntiVirus and AntiBot blades are enabled, RAD process and DLPU process consume CPU at
high level.
4. Disable scanning of archive file:
Consider disabling 'Archive Scanning' because it requires high amount of CPU resources:
A. Go to 'Profiles' pane.
B. Select the relevant profile click on 'Edit...' button.
C. Go to 'AntiVirus Settings
' pane.
D. In the 'Archives' section, uncheck the box 'Enable Archive scanning (impacts performance).
'.
E. Click on 'OK'.
F. Save the changes: go to 'File' menu click on 'Save'.
G. Install the policy onto the relevant Security Gateway / Cluster object.
Related solutions:
(4) Initial diagnostics

(41) Initial diagnostics CPU
Show / Hide initial diagnostics information
1. cpview
Background:
Displays the CPU utilization (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the CPU utilization during the problem
Analysis:
On the 'Overview' tab, refer to 'CPU:' section look at counter 'Used'
On the 'I/S' tab, go to 'CPU' menu go to 'Overview' menu refer to section 'CPU'
2. cpstat f multi_cpu os 209.87.209.101
Background:
Displays internal statistics for OS about all CPU cores as collected by Check Point
Diagnostics:
Collect this output continuously during the problem
Analysis:
Look at all the columns
Example from machine with 8 CPU cores:
Processors load
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
| 1| 1| 2| 96| 4| ?| 0|
| 2| 2| 4| 94| 6| ?| 0|
| 3| 1| 1| 98| 2| ?| 0|
| 4| 2| 3| 94| 6| ?| 0|
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
3. top
Background:
Manual page http://linux.die.net/man/1/top
Displays dynamic realtime view of a running system on Linux
Usage:
When running the command for the first time:
1. Press '1' to display all CPU cores
2. Press 'd' to set update interval type 2 and press Enter
3. Press 'n' to set maximum tasks displayed type 15 and press Enter
4. Press Shift+W to save the top's configuration
When running the command for diagnostics:
Press Shift+P to sort the output by CPU utilization
Press Shift+M to sort the output by Memory utilization
Diagnostics:
Analysis:
Look at the load in "User Space" counter us
High CPU consumption in "User Space" can be caused by processes that perform heavy tasks (e.g., too much logging by fwd, reloading the configuration during policy
installation, etc.)
Look at the load in "System (kernel) Space" counter sy
High CPU consumption in "System (kernel) Space" can be caused by heavy tasks (e.g., deep inspection of packets, enabling of all blades, enabling of all IPS protections
in Prevent mode, etc.)
Look at the amount of "Idle" counter id
The more CPU is idle, the better the machine's performance is
Look at the amount of "I/O waiting" counter wa
High amount of "I/O waiting" is caused by heavy reading from/writing to hard disk (e.g., during policy installation, heavy logging, insufficient RAM, etc.)
Look at the amount of "SoftIRQ" counter si
High amount of "SoftIRQ" is usually caused by high amount of traffic
Look at the amount of "Hardware IRQs" counter hi
Look at the CPU consumption by the processes column %CPU
Constant high CPU consumption by a process can be caused by numerous factors function stack should be collected from the process using a special Check Point shell
script ('pstack') refer to "(51) Advanced diagnostics CPU" section
Look at the Memory consumption by the processes column %MEM
Constant increase in memory consumption by a process might suggest some memory leak valgrind tool should be used to collect the necessary information from the process
refer to "(52) Advanced diagnostics Memory" section
Example from machine with 2 CPU cores:
top ‐ 15:18:47 up 14:06, 1 user, load average: 0.55, 0.37, 0.35
Tasks: 213 total, 1 running, 212 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.7%us, 0.7%sy, 0.0%ni, 97.3%id, 0.0%wa, 0.0%hi, 1.4%si, 0.0%st
Mem: 2005908k total, 1907412k used, 98496k free, 103952k buffers
Swap: 4225084k total, 323268k used, 3901816k free, 198288k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16948 admin 24 0 71692 10m 4744 S 2 0.6 15:14.00 DAService
4103 admin 15 0 0 0 0 S 1 0.0 8:38.37 fw_worker_0
4105 admin 15 0 0 0 0 S 1 0.0 10:46.19 fw_worker_2
4104 admin 15 0 0 0 0 S 1 0.0 4:05.49 fw_worker_1
1 admin 15 0 2040 720 624 S 0 0.0 0:01.44 init
2 admin RT ‐5 0 0 0 S 0 0.0 0:01.82 migration/0
3 admin 15 0 0 0 0 S 0 0.0 0:00.02 ksoftirqd/0
... ... ...
4. ps auxwwwf
Background:
Manual page http://linux.die.net/man/1/ps
Displays information about the current processes (daemons)
Usage:
209.87.209.101
Refer to the manual page
By default, output is sorted by PID column
Simplest sorting can be done by pipelining the output of ps command to sort command for example, to sort the CPU usage, run:
[Expert@HostName]# ps auxww | sort nrk 3
Diagnostics:
Analysis:
Look at the amount of "CPU", "MEM", "VSZ", "RSS", "TIME" consumed by the daemons
Constant increase in memory consumption might suggest some memory leak valgrind tool should be used to collect the necessary information from the process refer to "
(52) Advanced diagnostics Memory" section
Constant high CPU consumption can be caused by numerous factors function stack should be collected from the process using a special Check Point shell script ('pstack
refer to "(51) Advanced diagnostics CPU" section
Example (excerpt):
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
admin 1 0.0 0.0 2040 720 ? Ss 01:12 0:01 init [3]
admin 2 0.0 0.0 0 0 ? S< 01:12 0:01 [migration/0]
admin 3 0.0 0.0 0 0 ? S 01:12 0:00 [ksoftirqd/0]
admin 4 0.0 0.0 0 0 ? S< 01:12 0:00 [watchdog/0]
..................
admin 4103 1.0 0.0 0 0 ? S 01:13 8:41 [fw_worker_0]
admin 4104 0.4 0.0 0 0 ? S 01:13 4:07 [fw_worker_1]
admin 4105 1.2 0.0 0 0 ? S 01:13 10:50 [fw_worker_2]
admin 4115 0.0 0.2 13976 4184 ? Ss 01:13 0:02 /opt/CPshrd‐R77/bin/cpwd
admin 5098 0.2 2.4 342988 49860 ? Ssl 01:13 2:07 \_ cpd
admin 5136 0.0 0.3 153612 6216 ? Ss 01:13 0:00 \_ mpdaemon /opt/CPshrd‐R77/log/
..................
admin 5374 0.1 5.6 610672 113844 ? Ssl 01:13 1:32 \_ fwd
admin 5449 0.0 0.4 27856 9232 ? S 01:13 0:00 | \_ cpca
admin 5720 0.0 2.0 207256 41100 ? S 01:13 0:01 | \_ syslog 514 all
admin 6030 0.0 2.7 295536 54336 ? Sl 01:14 0:06 | \_ vpnd 0
admin 6033 0.0 2.5 239748 52012 ? Sl 01:14 0:15 | \_ pdpd 0 ‐t
admin 6034 0.0 2.5 211336 50924 ? S 01:14 0:11 | \_ pepd 0 ‐t
admin 6036 0.0 2.1 233164 43804 ? Sl 01:14 0:02 | \_ fwpushd 0
admin 6038 0.0 0.6 35016 13232 ? S 01:14 0:00 | \_ wstlsd 0 0
..................
admin 5382 0.5 7.5 453272 151396 ? Ssl 01:13 4:18 \_ fwm
admin 5390 0.0 1.0 46240 21504 ? Ss 01:13 0:00 \_ status_proxy
admin 5750 0.0 0.8 35228 17264 ? Ss 01:13 0:01 \_ rad
admin 5827 0.0 1.1 42500 22748 ? Ss 01:13 0:00 \_ cpstat_monitor
admin 5996 0.0 0.5 57176 11516 ? Ssl 01:14 0:02 \_ fwucd
admin 6022 0.0 1.0 132424 21652 ? Ssl 01:14 0:03 \_ dlpu ‐i 0 0
..................
admin 4636 0.0 0.3 25020 7784 ? Ss 01:13 0:00 /bin/pm
admin 4653 0.1 0.5 31620 11908 ? Ss 01:13 1:02 \_ /bin/confd
admin 4654 0.1 0.7 38040 14312 ? SNsl 01:13 0:57 \_ /bin/searchd ‐niceboost 10
admin 4656 0.0 0.4 99740 9880 ? Ssl 01:13 0:01 \_ /bin/rconfd /etc/actions_mapping.xml
admin 4657 0.0 0.1 6472 3824 ? Ss 01:13 0:18 \_ /bin/monitord
admin 4683 0.0 0.3 25252 7168 ? Ss 01:13 0:00 \_ /bin/cloningd
admin 4685 0.0 0.1 10048 2348 pts/0 Ss+ 01:13 0:00 \_ /bin/clishd default server
admin 4687 0.0 0.1 19032 2796 ? Ssl 01:13 0:00 \_ /bin/clish ‐p
admin 4688 0.0 0.0 2948 868 ? Ss 01:13 0:00 \_ /usr/bin/tclsh /usr/libexec/netflowd
admin 4689 0.0 0.3 30360 7368 ? Ss 01:13 0:07 \_ /usr/sbin/snmpd ‐f ‐c /etc/snmp/userDefinedSettings.conf
admin 5041 0.0 0.3 32768 6912 ? Ss 01:13 0:00 \_ /bin/routed ‐N
admin 5047 0.0 0.3 33128 7820 ? S 01:13 0:00 | \_ /bin/routed ‐i default ‐f /etc/routed0.conf ‐h 0
..................
admin 9438 0.0 0.0 1652 496 tty1 Ss+ 01:14 0:00 /sbin/agetty 9600 tty1
admin 9441 0.0 0.0 2416 1052 ? Ss 01:14 0:00 /bin/bash /bin/console_agetty
admin 9463 0.0 0.0 1652 508 tty4 Ss+ 01:14 0:00 \_ /sbin/agetty 9600 tty4 vt100
admin 16948 2.1 0.5 71692 11072 ? Sl 03:15 15:21 /opt/CPda/bin/DAService
5. cat /proc/interrupts
Background:
Manual page http://linux.die.net/man/5/proc
Displays the number of interrupts on each CPU core from each IRQ
Usage:
By default, output is sorted by IRQ number
The only relevant devices are network interfaces
To shorten the output, use the 'grep' command for example, run:
[Expert@HostName]# cat /proc/interrupts | grep E "CPU|eth"
To collect this output continuously, use the 'watch' command for example, run:
[Expert@HostName]# watch d n 1 'cat /proc/interrupts | grep E "CPU|eth"'
Diagnostics:
Analysis:
Look at the general trend which CPU receives more interrupts and from which interfaces
If some CPU cores receive more interrupts than others, then affinity of interfaces to CPU cores should be optimized interface should be redistributed better
Example from machine with 4 CPU cores and 4 interfaces:
CPU0 CPU1 CPU2 CPU3
0: 51254849 0 0 0 IO‐APIC‐edge timer
1: 4 0 0 5 IO‐APIC‐edge i8042
6: 2 3 0 0 IO‐APIC‐edge floppy
7: 0 0 0 0 IO‐APIC‐edge parport0
8: 3 0 0 0 IO‐APIC‐edge rtc
9: 0 0 0 0 IO‐APIC‐level acpi
12: 0 0 0 115 IO‐APIC‐edge i8042
15: 302 1043 168 1127 IO‐APIC‐edge ide1
59: 320145 6732 5944 6189 IO‐APIC‐level ioc0, eth3
67: 15765753 0 0 0 IO‐APIC‐level eth0
75: 24271 1285 36 0 IO‐APIC‐level eth1
209.87.209.101
83: 24272 0 1322 0 IO‐APIC‐level eth2
NMI: 0 0 0 0
LOC: 50950084 50945459 50949418 50945128
ERR: 0
MIS: 0
6. cat /proc/cpuinfo
Background:
Displays a collection of CPU and system architecture dependent items about CPU (on multiCPU (SMP) machines will show information for each CPU)
Diagnostics:
Collect this output to see the information about CPU (architecture, vendor, number)
Analysis:
Look at the processor number
Look at the processor model
Look at the CPU clock frequency
Look at the supported flags (e.g., pae)
Example (excerpt):
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
stepping : 2
cpu MHz : 2399.483
cache size : 12288 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce ...
bogomips : 4806.12
7. dmesg
Background:
Manual page http://linux.die.net/man/8/dmesg
Displays the Linux kernel ring buffer bootup messages from various kernel modules and hardware components
Diagnostics:
Collect this output to search for errors from various kernel modules and hardware components
Analysis:
Check each line for kernel boot parameters (e.g., vmalloc), errors, failures, components/features being disabled
(42) Initial diagnostics Memory
1. cpview
Background:
Displays the Memory utilization (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the Memory utilization during the problem
Analysis:
On the 'Overview' tab, refer to 'Memory:' section look at counters 'Free MB'
On the 'I/S' tab, go to 'Memory' menu go to 'FWKernel' menu refer to section 'Firewall kernel memory usage summary:
' look at counter '%Usage'
2. fw ctl pstat
Background:
Displays FireWall internal statistics about memory and traffic
Diagnostics:
Collect the output before and after the suspected problem
Use different flags to get more data:
for HMEM: fw ctl pstat h
for SMEM: fw ctl pstat s
for KMEM: fw ctl pstat k
for Handles (kbufs): fw ctl pstat l
209.87.209.101
Counters are reset when Check Point Services are stopped (with 'cpstop' command)
Under memory, 'allocations' counter always grows, may wrap around
Analysis:
No single field that indicates a problem need to interpret all counters together
HMEM
failures under HMEM no real memory problem, just mean HMEM is full ; HMEM should have been configured larger
"failed allocations" under HMEM (only) do not indicate any problem
SMEM
failures under SMEM reached Check Point memory limit , exhausted OS memory, large nonsleep allocation , indicate some shortage
"failed allocations" under SMEM may not mean that a user's allocation failed, maybe HMEM extension failed
"failed free" under SMEM means an overrun or freeing an invalid pointer indicates a bug
KMEM
failures under KMEM application asked for memory and could not get it , usually, it is a memory problem
"failed allocations" under KMEM means that the application didn't get memory
Example:
System Capacity Summary:
Memory used: 12% (183 MB out of 1515 MB) ‐ below watermark
Concurrent Connections: 79 (Unlimited)
Aggressive Aging is not active
Hash kernel memory (hmem) statistics:
Total memory allocated: 155189248 bytes in 37888 (4096 bytes) blocks using 37 pools
Total memory bytes used: 33563132 unused: 121626116 (78.37%) peak: 53675664
Total memory blocks used: 9082 unused: 28806 (76%) peak: 13407
Allocations: 96170689 alloc, 0 failed alloc, 95799050 free
System kernel memory (smem) statistics:
Total memory bytes used: 280891432 peak: 297288060
Total memory bytes wasted: 12156438
Blocking memory bytes used: 902804 peak: 933368
Non‐Blocking memory bytes used: 279988628 peak: 296354692
Allocations: 10526 alloc, 1 failed alloc, 8437 free, 0 failed free
vmalloc bytes used: 7340032 expensive: yes
Kernel memory (kmem) statistics:
Total memory bytes used: 159039668 peak: 185833572
Allocations: 96179738 alloc, 1 failed alloc
95807459 free, 0 failed free
External Allocations: 0 for packets, 63652 for SXL
Cookies:
17293941 total, 0 alloc, 0 free,
19087 dup, 17386724 get, 117090 put,
32037742 len, 0 cached len, 0 chain alloc,
0 chain free
Connections:
82980 total, 31766 TCP, 50795 UDP, 7 ICMP,
412 other, 0 anticipated, 0 recovered, 79 concurrent,
568 peak concurrent
Fragments:
0 fragments, 0 packets, 0 expired, 0 short,
0 large, 0 duplicates, 0 failures
NAT:
0/0 forw, 0/0 bckw, 0 tcpudp,
0 icmp, 0‐0 alloc
Sync: off
3. cpstat f memory os
Background:
Displays internal statistics for OS about memory as collected by Check Point
Diagnostics:
Collect this output continuously during the problem (usually, over some time period)
Analysis:
High memory consumption can be caused by high amount of traffic
Constant increase in memory consumption might suggest some memory leak follow sk35496 How to detect a memory leak on Security Gateway with SecurePlatform OS /
Gaia OS
Example:
Total Virtual Memory (Bytes): 6380535808
Active Virtual Memory (Bytes): 1882390528
Total Real Memory (Bytes): 2054049792
Active Real Memory (Bytes): 1643986944
Free Real Memory (Bytes): 410062848
Memory Swaps/Sec: ‐
Memory To Disk Transfers/Sec: ‐
4. cat /proc/meminfo
Background:
Displays the amount of free and used memory (both physical and swap) on the system as well as the shared memory and buffers used by the kernel
Diagnostics:
Analysis:
Sum up these counters: MemFree + Buffers + Cached, and compare the result with counter MemTotal follow sk42717 How to read the output of 'cat /proc/meminfo'
on Linuxbased system
209.87.209.101
Constant increase in memory consumption might suggest some memory leak follow sk35496 How to detect a memory leak on Security Gateway with SecurePlatform OS /
Gaia OS
Example:
MemTotal: 2005908 kB
RawMemTotal: 2097152 kB
MemFree: 95092 kB
Buffers: 107136 kB
Cached: 198520 kB
SwapCached: 90452 kB
Active: 1335364 kB
Inactive: 121220 kB
HighTotal: 262016 kB
HighFree: 808 kB
LowTotal: 1743892 kB
LowFree: 94284 kB
SwapTotal: 4225084 kB
SwapFree: 3901816 kB
Dirty: 864 kB
Writeback: 0 kB
AnonPages: 1137312 kB
Mapped: 108660 kB
Slab: 82164 kB
PageTables: 19132 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 5228036 kB
Committed_AS: 4025512 kB
VmallocTotal: 247800 kB
VmallocUsed: 114732 kB
VmallocChunk: 117952 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
5. dmesg
Background:
Diagnostics:
Analysis:
(43) Initial diagnostics Network interface cards
1. cpview
Background:
Displays the traffic and interfaces statistics (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the traffic and interfaces during the problem
Analysis:
On the 'Overview' tab, refer to 'Traffic counters:' section
On the 'SysInfo' tab, refer to 'Hardware Information: ' section
On the 'Traffic' tab, go to 'Overview' menu refer to 'Traffic Rate:' section
On the 'Traffic' tab, go to 'Overview' menu refer to 'Concurrent Connections: ' section
On the 'Traffic' tab, go to 'Overview' menu refer to 'Interface drops:' section
On the 'Traffic' tab, go to 'Interfaces' menu refer to 'Errors and Drops:' section
2. netstat ni
Background:
Manual page http://linux.die.net/man/8/netstat
Displays a table of all network interfaces
Diagnostics:
Analysis:
RXOK number of packets that have been received errorfree
RXERR number of packets that have been received damaged
RXDRP number of received packets that have been dropped
RXOVR number of received packets that have been lost because of an overrun (number of times the receiver hardware was unable to hand received data to a hardware
buffer the internal FIFO buffer of the chip is full, but is still tries to handle incoming traffic ; most likely, the input rate of traffic exceeded the ability of the receiver to
handle the data)
TXOK number of packets that have been transmitted errorfree
TXERR number of packets that have been transmitted damaged
TXDRP number of transmitted packets that have been dropped
TXOVR number of transmitted packets that have been lost because of an overrun 209.87.209.101
Flg shows the flags that have been set for this interface these characters are onecharacter versions of the long flag names that are displayed in the output of 'ifconfig'
command:
A = this interface will receive all Multicast addresses
B = a Broadcast address has been set
D = debugging is turned on
L = this interface is a loopback device
M = all packets are received (promiscuous mode)
m = master
N = trailers are avoided
O = ARP is turned off for this interface
P = this is a PointtoPoint connection
R = interface is running
s = slave
U = interface is up
Example:
Kernel Interface table
Iface MTU Met RX‐OK RX‐ERR RX‐DRP RX‐OVR TX‐OK TX‐ERR TX‐DRP TX‐OVR Flg
br0 1500 0 4384062 0 0 0 28 0 0 0 BMRU
eth0 1500 0 12505484 0 0 0 20359171 0 0 0 BMRU
eth1 1500 0 11882710 0 62 0 449677832 0 0 0 BMRU
eth2 1500 0 2592575900 362803 3626586 0 1158425513 0 0 0 BMRU
eth3 1500 0 1189952933 30 9382997 0 2575575506 0 0 0 BMRU
lo 16436 0 174524 0 0 0 174524 0 0 0 LRU
3. ifconfig name_of_interface
Background:
Manual page http://linux.die.net/man/8/ifconfig
Displays the status and traffic statistics for the currently active interfaces
Diagnostics:
Analysis:
UP indicates that the kernel modules related to the Ethernet interface has been loaded
BROADCAST indicates that interface supports broadcasting a necessary characteristic to obtain IP address via DHCP
NOTRAILERS indicates that trailer encapsulation is disabled (Linux usually ignore trailer encapsulation so this value has no effect at all)
RUNNING indicates that interface is ready to accept data
MULTICAST indicates that the interface supports multicasting
MTU Maximum Transmission Unit is the size of each packet received by the interface (default value is 1500; setting MTU to a higher value could hazard packet
fragmentation or buffer overflows)
Metric used to compute the cost of a route it tells the OS, to which interface a packet should be forwarded, when multiple interfaces could be used to reach the packet's
destination
Takes values of 0,1,2,3,... The lower the value, the more leverage it has. This parameter has significance only while routing packets. For example, if you have two
Ethernet cards and you want to forcibly make your machine use one card over the other in sending the data. Then you can set the Metric value of the Ethernet card
which you favor lower than that of the other Ethernet card.
RX packets total number of packets received via the interface
RX errors number of damaged packets received
RX dropped number of dropped packets due to reception errors
RX overruns number of received packets that experienced data overruns (number of times the receiver hardware was unable to hand received data to a hardware buffer)
RX frame number received packets that experienced frame errors
TX packets total number of packets transmitted via the interface
TX errors number of packets that experienced transmission error
TX dropped number of dropped transmitted packets due to transmission errors
TX overruns number of transmitted packets that experienced data overruns (number of times the transmitter hardware was unable to hand received data to a hardware
buffer)
TX carriers number received packets that experienced loss of carriers
TX collisions number of transmitted packets that experienced Ethernet collisions (a nonzero value of this field indicates possibility of network congestion)
TX txqueuelen configured length of transmission queue
RX bytes total bytes received over this interface
TX bytes total total bytes transmitted over this interface
Example:
eth0 Link encap:Ethernet HWaddr 00:50:56:AA:69:33
inet addr:172.30.41.90 Bcast:172.30.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:21363169 errors:0 dropped:0 overruns:0 frame:0
TX packets:128400 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1608073037 (1.4 GiB) TX bytes:33703584 (32.1 MiB)
4. netstat anp
Background:
Displays both listening and nonlistening (for TCP this means established connections) sockets
Diagnostics:
Analysis:
Under "Active Internet connections" look at "RecvQ" and at "SendQ"
RecvQ data (in bytes), which has not yet been pulled from the socket buffer by the application (value should be as close to 0 as possible)
SendQ data (in bytes), which the sending application has given to the transport, but has yet to be ACKnowledged by the receiving TCP (value should be as close to 0 as
possible a large number may indicate a network bottleneck)
Example:
209.87.209.101
[Expert@FW]# netstat ‐anp
Active Internet connections (servers and established)
Proto Recv‐Q Send‐Q Local Address Foreign Address State PID/Program name
tcp 0 2368 172.30.20.69:22 172.30.20.228:3676 ESTABLISHED 16179/0
udp 256956 0 0.0.0.0:9282 0.0.0.0:*
[Expert@FW]#
5. ethtool name_of_interface
Background:
Manual page http://linux.die.net/man/8/ethtool
Displays Ethernet card settings
Diagnostics:
Collect this output to check the current status
Analysis:
Check every line of the output
Configuration on both ends of the cable should be identical (speed /duplex / autonegotiation)
Example:
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto‐negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto‐negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto‐negotiation: on
Supports Wake‐on: d
Wake‐on: d
Current message level: 0x00000007 (7)
Link detected: yes
6. ethtool i name_ofinterface
Background:
Displays information about associated driver
Diagnostics:
Analysis:
Use driver with NAPI
Use the latest version of the driver
Example:
driver: e1000
version: 7.6.15.5‐NAPI
firmware‐version: N/A
bus‐info: 0000:02:00.0
7. ethtool g name_ofinterface
Background:
Displays information about RX/TX ring parameters
Diagnostics:
Analysis:
Check the current size of the buffers versus the maximum allowed follow sk42181 How to increase sizes of buffer on SecurePlatform/Gaia for Intel NIC and Broadcom
NIC
Example:
Ring parameters for eth0:
Pre‐set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 1024
8. arp an | wc l
Background:
Manual page http://linux.die.net/man/8/arp
Displays the system's ARP cache table
209.87.209.101
Diagnostics:
Analysis:
Check the number of entries in the ARP table
Refer to sk43772 'kernel: neighbour table overflow' appears repeatedly in /var/log/messages files
Refer to sk92372 How to change the size of IPv6 Neighbors cache table
Example:
[Expert@HostName]# arp ‐an | wc ‐l
1
[Expert@HostName]# arp ‐an
? (172.30.1.1) at 00:1C:EB:1C:B4:43 [ether] on eth0
[Expert@HostName]# arp ‐en
Address HWtype HWaddress Flags Mask Iface
172.30.1.1 ether 00:1C:EB:1C:B4:43 C eth0
[Expert@HostName]#
9. dmesg
Background:
Diagnostics:
Analysis:
(44) Initial diagnostics SecureXL
1. cpview
Background:
Displays the SecureXL status and statistics (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the SecureXL performance during the problem
Analysis:
On the 'SysInfo' tab, refer to 'Configuration Information: ' section look at 'PPack Status'
On the 'Traffic' tab, go to 'Overview' menu refer to section 'Templates:'
On the 'I/S' tab, go to 'SXL' menu go to 'Overview' menu
2. fwaccel stat
Background:
Displays the status of SecureXL and SecureXL Templates
Diagnostics:
Analysis:
"Accelerator Status" should be "on"
"Accept Templates" should be "enabled"
If "Accept Templates" are "disabled" from a specific rule, then optimize the rulebase follow sk32578 SecureXL Mechanism
Status of "Drop Templates" is determined by sk66402 SecureXL Drop Templates are not supported in versions lower than R76
Status of "NAT Templates" is determined by sk71200 SecureXL NAT Templates
"Cryptography Features" section appears only if SecureXL device supports cryptography, or if a VPN Accelerator card is installed on the machine.
Example:
Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #31
Drop Templates : disabled
NAT Templates : disabled by user
Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, WireMode,
DropTemplates, NatTemplates, Streaming,
MultiFW, AntiSpoofing, ViolationStats,
Nac, AsychronicNotif, ERDOS
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST‐40, AES‐128, AES‐256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES‐XCBC, SHA256
3. fwaccel stats s
209.87.209.101
Background:
Displays summary of SecureXL acceleration statistics
Diagnostics:
Analysis:
If the ratio of "Accelerated/Total" is less than 80%, it means that acceleration is poor, and the optimization is required
The higher the ratio of "F2Fed pkts/Total pkts" is, the more traffic is Forwarded to Firewall for inspection (the poorer the performance is) see the fwaccel
conns command below
The higher the ratio of "PXL pkts/Total pkts" is, the more packets passed in Medium path. Meaning, packets were handled by the SecureXL device, except for IPS /
Application Control / AntiVirus / AntiBot processing. The CoreXL layer passed the packets to one of the CoreXL FW instances to perform the processing. This path is
available only when CoreXL is enabled.
The higher the ratio of "QXL pkts/Total pkts" is, the more packets were processed by QoS.
Example:
Accelerated conns/Total conns : 364/13215 (2%)
Delayed conns/(Accelerated conns + PXL conns) : 48/12023 (0%)
Accelerated pkts/Total pkts : 18252/564927 (3%)
F2Fed pkts/Total pkts : 36776/564927 (6%)
PXL pkts/Total pkts : 509899/564927 (90%)
QXL pkts/Total pkts : 0/564927 (0%)
4. fwaccel conns
Background:
Displays entries from SecureXL connections table
Diagnostics:
Collect this output continuously during the problem (Important Note: This command consumes high amount of memory)
Analysis:
Look at the Flags column:
F = Forward to Firewall the connection is not accelerated
To quickly determine the amount of F2F connections:
1. Check the total number of connections in SecureXL connections table:
[Expert@HostName]# fwaccel conns s
2. Check the total number of F2F connections:
[Expert@HostName]# fwaccel conns | grep ' F' | grep v 'Flags' | wc l
To see why specific connections are Forwarded to Firewall, run kernel debug. Refer to "(54) Advanced diagnostics SecureXL" section
U = Unidirectional the connection can pass data on either C2S or S2C data packets from the opposite direction will be F2F'ed
N = NAT is being performed on the connection by the device
A = Accounting is performed on the connection (the connection is viewed by either rulebase accounting or SmartView Monitor)
C = Encryption is done on the connection by the device
W = the connection is in wire mode
P = Partial (versions R70 and higher)
S = Streaming PXL (versions R70 and higher)
Example (all IP addresses were replaced by "X"):
Source SPort Destination DPort PR Flags C2S i/f S2C i/f
‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐ ‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐
X.X.X.X 61242 X.X.X.X 80 6 ..N.... eth5/eth0 eth0/eth5
X.X.X.X 6000 X.X.X.X 3842 6 ....... eth0/eth4 eth4/eth0
X.X.X.X 4285 X.X.X.X 80 6 F.N.... eth5/eth0 eth0/eth5
X.X.X.X 161 X.X.X.X 5002 17 F...... eth2/eth2 ‐/‐
X.X.X.X 21891 X.X.X.X 0 1 F...... eth2/eth4 eth4/eth2
X.X.X.X 34303 X.X.X.X 389 6 F.N.... eth2/eth2 eth2/‐
5. fwaccel templates
Background:
Displays SecureXL Connection Templates
Note: Size of Templates table (cphwd_tmpl, id 8111) is limited to 1/4 of the size of Firewall Connections Table (connections, id 8158).
Diagnostics:
Collect this output continuously during the problem (Important Note: This command consumes high amount of memory)
Analysis:
Check whether templates are created:
[Expert@HostName]# fwaccel templates s
Check whether templates were created for relevant connections (search for relevant IP addresses using the grep command)
Look at the Flags column:
You should not see connections with F, N, or C flags
F = Forward to Firewall the connection is not accelerated
U = Unidirectional the connection can pass data on either C2S or S2C data packets from the opposite direction will be F2F'ed
N = NAT is being performed on the connection by the device
A = Accounting is performed on the connection (the connection is viewed by either rulebase accounting or SmartView Monitor)
C = Encryption is done on the connection by the device
W = the connection is in wire mode
P = Partial (versions R70 and higher)
S = Streaming PXL (versions R70 and higher)
D = Drop Template
L = Log drop action 209.87.209.101
The LCT column shows how many seconds ago a connection was created by the template
The DLY column shows Delayed Synchronization value for the template
Example (all IP addresses were replaced by "X"):
Source SPort Destination DPort PR Flags LCT DLY C2S i/f S2C i/f Inst Identity
‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐ ‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐ ‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐ ‐‐‐‐‐‐‐‐
X.X.X.X * X.X.X.X 161 17 ...A...S. 65 0 36/21 21/36 0 0
X.X.X.X * X.X.X.X 161 17 ...A...S. 5 0 36/8 8/36 1 0
X.X.X.X * X.X.X.X 1437 17 ...A...S. 27 0 36/8 8/36 1 0
X.X.X.X * X.X.X.X 161 17 ...A...S. 23 0 36/21 21/36 2 0
X.X.X.X * X.X.X.X 88 17 ...A...S. 11 0 8/36 36/8 4 0
6. sim if
Background:
Displays the list of interfaces used and seen by the SecureXL
Diagnostics:
Collect this output to see the current status
Analysis:
Look at the Name column and Address column
Look at the F column (refer to "(61B) 'sim' command" section 'sim if' command)
Example:
Name | Address | MTU | F | SIM F | IRQ | Dev | Output
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
eth0 | 172.30.168.38 | 1500 | 039 | 00000 | 67 | 0x85b1b000 | 0xffffffff
eth1 | 10.20.30.38 | 1500 | 029 | 00008 | 75 | 0xbc5d3000 | 0xffffffff
eth2 | 10.5.0.1 | 1500 | 029 | 00000 | 83 | 0xbcbbb000 | 0xffffffff
7. sim affinity l
Background:
Displays the current affinity of network interfaces to CPU cores
Diagnostics:
Check the SIM Affinity mode:
If SIM Affinity was configured in Static mode, then the following configuration file should exist and should not be empty:
[Expert@HostName]# cat $PPKDIR/boot/modules/sim_aff.conf
If SIM Affinity was configured in Automatic mode, then the following Task should exist in CPD Scheduler:
[Expert@HostName]# cpd_sched_config print | grep A 5 "Sim_Affinity"
Task: "Sim_Affinity"
Command: sim
Arguments: affinity ‐c
Interval: 60
Active: true
RunAtStart: false
Example:
eth0 : 0
eth1 : 0
eth2 : 1
eth3 : 1
(45) Initial diagnostics CoreXL
1. cpview
Background:
Displays the CoreXL status and statistics (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the CoreXL performance during the problem
Analysis:
On the 'SysInfo' tab, refer to 'Configuration Information: ' section look at 'CoreXL Status'
On the 'SysInfo' tab, refer to 'Configuration Information: ' section look at 'CoreXL instances'
On the 'I/S' tab, go to 'CoreXL' menu go to 'General' menu refer to 'Queues:' section look at counter 'Enqueue fail'
On the 'I/S' tab, go to 'CoreXL' menu go to 'Instances' menu activate the statistics (!may affect performance) go to each CoreXL FW instance ('FW
Instance<N>'):
refer to 'FW Stats' section
refer to 'Top FWLock consumers: ' section
2. fw ctl multik stat 209.87.209.101
Background:
Displays status of CoreXL instances and summary for traffic that passes through each CoreXL FW instance (current number and peak number of concurrent connections)
Diagnostics:
Analysis:
Check the number and the allocation of FW instances to CPU cores (in cluster, the output must be identical on all members)
Check the peak number connections on FW instances (instances should be loaded as equally as possible)
Example from machine with 4 CPU cores (1 SND + 3 CoreXL FW instances):
ID | Active | CPU | Connections | Peak
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
0 | Yes | 3 | 26 | 66
1 | Yes | 2 | 36 | 64
2 | Yes | 1 | 39 | 53
3. fw ctl affinity l r v a
Background:
Displays affinity of interfaces, processes and CoreXL FW instances to CPU cores
Diagnostics:
Analysis:
Check the affinity settings (of interfaces, processes and FW instances) to CPU cores (in cluster, the output must be identical on all members)
The interfaces affinity should be configured only to to CPU cores that are running as SNDs (CPU cores that are not running CoreXL FW instances)
Under normal circumstances, it is not recommended for the SND and a CoreXL FW instance to share the same CPU core
Example from machine with 4 CPU cores and 4 interfaces (1 SND + 3 CoreXL FW instances):
CPU 0: eth0 (irq 67) eth3 (irq 59)
CPU 1: eth1 (irq 75)
fw_2
CPU 2: eth2 (irq 83)
fw_1
CPU 3: fw_0
All: mpdaemon fwucd cpca vpnd fwm ... cpd cprid
4. cat /proc/interrupts
Background:
Displays the number of interrupts on each CPU core from each IRQ
Usage:
By default, output is sorted by IRQ number
The only relevant devices are network interfaces
To shorten the output, use the 'grep' command for example, run:
[Expert@HostName]# cat /proc/interrupts | grep E "CPU|eth"
To collect this output continuously, use the 'watch' command for example, run:
[Expert@HostName]# watch d n 1 'cat /proc/interrupts | grep E "CPU|eth"'
Diagnostics:
Analysis:
Look at the general trend which CPU receives more interrupts and from which interfaces
If some CPU cores receive more interrupts than others, then affinity of interfaces to CPU cores should be optimized interface should be redistributed better
Example from machine with 8 CPU cores and 5 interfaces:
CPU0 CPU1 CPU2 CPU3
0: 51254849 0 0 0 IO‐APIC‐edge timer
1: 4 0 0 5 IO‐APIC‐edge i8042
6: 2 3 0 0 IO‐APIC‐edge floppy
7: 0 0 0 0 IO‐APIC‐edge parport0
8: 3 0 0 0 IO‐APIC‐edge rtc
9: 0 0 0 0 IO‐APIC‐level acpi
12: 0 0 0 115 IO‐APIC‐edge i8042
15: 302 1043 168 1127 IO‐APIC‐edge ide1
59: 320145 6732 5944 6189 IO‐APIC‐level ioc0, eth3
67: 15765753 0 0 0 IO‐APIC‐level eth0
75: 24271 1285 36 0 IO‐APIC‐level eth1
83: 24272 0 1322 0 IO‐APIC‐level eth2
NMI: 0 0 0 0
LOC: 50950084 50945459 50949418 50945128
ERR: 0
MIS: 0
(5) Advanced diagnostics 209.87.209.101

(51) Advanced diagnostics CPU
Show / Hide advanced diagnostics information
1. cpview
Background:
Displays the CPU utilization (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the CPU utilization during the problem
Analysis:
On the 'I/S' tab, go to 'CPU' menu go to 'Contexts' menu refer to section 'Contexts'
2. vmstat [n] [delay_in_sec [number_of_samples]]
Background:
Manual page http://linux.die.net/man/8/vmstat
Displays information about processes, memory, paging, block IO, and CPU activity
Usage:
By default, the output's header (with names of columns) is displayed every 20 samples. If output is redirected to a file, then it will be very difficult to analyze it using such
commands as 'grep' / 'awk'. Therefore, if output is redirected to a file, use the 'n' flag to display the header only once (at the very top) run:
[Expert@HostName]# vmstat n [delay_in_sec [number_of_samples]]
By default, only 1 sample is printed (with the average values since boot). Therefore, to collect this output continuously, specify the delay between the samples run:
[Expert@HostName]# vmstat [n] delay_in_sec [number_of_samples]
If you specified the delay between the samples, then the output will be collected indefinitely (until the command is stopped or killed). If you want to limit the output, then
specify the total number of samples run:
[Expert@HostName]# vmstat [n] delay_in_sec number_of_samples
Diagnostics:
Analysis:
Look at the "procs" section number of processes waiting for CPU (counter r)
Look at the "swap" section reading from swap file (si) and writing to swap file (so)
Look at the "io" section reading from hard disk (bi) and writing to hard disk (bo)
Look at the "system" section number of Context Switches (cs)
Look at the "cpu" section at all counters:
"User Space" counter us
"System (kernel) Space" counter sy
"Idle" counter id
"I/O waiting" counter wa
Example:
procs ‐‐‐‐‐‐‐‐‐‐‐memory‐‐‐‐‐‐‐‐‐‐ ‐‐‐swap‐‐ ‐‐‐‐‐io‐‐‐‐ ‐‐system‐‐ ‐‐‐‐‐cpu‐‐‐‐‐‐
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 323280 100128 103192 198132 1 2 8 35 117 130 2 3 96 0 0
2 0 323280 100400 103196 198132 0 0 0 260 1297 6904 1 4 95 0 0
1 0 323280 100404 103196 198132 0 0 0 0 1289 7100 1 4 95 0 0
7 0 323280 100252 103200 198132 0 0 0 32 1308 6971 0 2 97 0 0
1 0 323280 100428 103200 198132 0 0 0 0 1265 7111 1 3 97 0 0
3. sar [u] [P { <cpu> | ALL }] [interval_in_sec [number_of_samples]]
Background:
Manual page http://linux.die.net/man/1/sar
Displays information about CPU activity, network devices, memory, paging, block IO, etc.
Usage:
Refer to sk112734 How to collect System Activity Report using the "sar" command
Diagnostics:
Collect this output during the problem
Analysis:
Look at the load in "User Space" counter user
High CPU consumption in "User Space" can be caused by processes that perform heavy tasks (e.g., too much logging by fwd, reloading the configuration during policy
installation, etc.)
Look at the load in "System (kernel) Space" counter system
High CPU consumption in "System (kernel) Space" can be caused by heavy tasks (e.g., deep inspection of packets, enabling of all blades, enabling of all IPS protections
in Prevent mode, etc.)
Look at the amount of "Idle" counter idle
The more CPU is idle, the better the machine's performance is
Look at the amount of "I/O waiting" counter iowait
High amount of "I/O waiting" is caused by heavy reading from/writing to hard disk (e.g., during policy installation, heavy logging, insufficient RAM, etc.)
Look at the counter steal Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
209.87.209.101
Example:
[Expert@R77‐GW:0]# sar ‐u
Linux 2.6.18‐92cp (R77‐GW) 08/04/16
16:10:01 CPU %user %nice %system %iowait %steal %idle
16:20:01 all 0.50 0.00 1.49 0.43 0.00 97.58
16:30:01 all 0.53 0.00 1.78 0.36 0.00 97.32
16:40:01 all 0.53 0.00 1.55 0.24 0.00 97.68
16:50:01 all 0.51 0.00 1.76 0.26 0.00 97.46
17:00:01 all 0.51 0.21 1.71 0.33 0.00 97.25
17:10:01 all 0.48 0.00 1.57 0.35 0.00 97.59
17:20:01 all 0.49 0.00 1.64 0.31 0.00 97.57
17:30:01 all 0.48 0.00 1.60 0.48 0.00 97.44
17:40:01 all 0.48 0.00 1.61 0.31 0.00 97.60
17:50:01 all 0.50 0.00 1.68 0.32 0.00 97.51
18:00:01 all 0.49 0.18 1.44 0.29 0.00 97.60
18:10:01 all 0.48 0.00 1.76 0.45 0.00 97.31
18:20:01 all 0.49 0.00 1.71 0.32 0.00 97.48
18:30:01 all 0.49 0.00 1.53 0.41 0.00 97.56
18:40:01 all 0.53 0.00 1.88 0.32 0.00 97.26
18:50:01 all 0.49 0.00 1.56 0.29 0.00 97.66
Average: all 0.50 0.02 1.64 0.34 0.00 97.49
[Expert@R77‐GW:0]#
[Expert@R77‐GW:0]# sar ‐P 1
Linux 2.6.18‐92cp (R77‐GW) 08/04/16
16:10:01 CPU %user %nice %system %iowait %steal %idle
16:20:01 1 0.51 0.00 1.28 0.20 0.00 98.01
16:30:01 1 0.52 0.00 1.41 0.03 0.00 98.04
16:40:01 1 0.51 0.00 1.35 0.04 0.00 98.10
16:50:01 1 0.47 0.00 1.38 0.01 0.00 98.14
17:00:01 1 0.52 0.20 1.37 0.05 0.00 97.86
17:10:01 1 0.47 0.00 1.39 0.08 0.00 98.06
17:20:01 1 0.51 0.00 1.38 0.11 0.00 98.01
17:30:01 1 0.50 0.00 1.36 0.20 0.00 97.94
17:40:01 1 0.49 0.00 1.30 0.08 0.00 98.14
17:50:01 1 0.49 0.00 1.37 0.11 0.00 98.03
Average: 1 0.50 0.02 1.36 0.09 0.00 98.03
4. sh pstack_sym.sh <PID_of_problematic_daemon>
Background:
Manual page http://linux.die.net/man/1/pstack
Displays a stack trace of a running process
Check Point R&D have created a special shell script to collect the necessary information based on the 'pstack' utility
Usage:
A. Download the package that contains the 'pstack' utility and special Check Point shell script that collects the necessary information.
B. Transfer the package to the problematic machine (into some directory, e.g., /some_path_to_pstack/
).
C. Navigate to the directory with the package:
[Expert@HostName]# cd /some_path_to_pstack/
D. Unpack the package:
[Expert@HostName]# tar xvf pstack_sym.tar
E. Install the 'pstack' utility:
[Expert@HostName]# rpm ihv force nodeps pstack1.17.i386.rpm
Note: there is no need to restart any service or to reboot the machine
F. Determine the PID of the problematic daemon (that consumes the CPU at high level) run the top command and look at the leftmost column 'PID'.
Alternatively, if you already know the name of the daemon, run this command:
[Expert@HostName]# ps auxw | grep v grep | grep i NAME_OF_PROBLEMATIC_DAEMON_THAT_CONSUMES_CPU | awk '{print $2}'
G. During the problem, run the special shell script at least 57 times:
[Expert@HostName]# sh pstack_sym.sh <PID_of_problematic_daemon>
Important Note: It might take 1015 minutes for the script to complete the analysis please wait patiently.
i. run the script
ii. wait until the script finishes
iii. follow the instructions on the screen
iv. run the script
v. wait until the script finishes
vi. follow the instructions on the screen
vii. etc., etc., etc.
H. Send the following to Check Point Support for analysis:
Outputs collected by the script (location is shown on the screen)
Log files from the problematic daemon (should be located either in $CPDIR/log/, or in $FWDIR/log/)
CPinfo file from the problematic machine (just to see the versions and the configuration)
Analysis:
All the outputs will be analyzed by Check Point Support and R&D
209.87.209.101
Example:
[Expert@FW]# sh pstack_sym.sh 30477

**************************************************
Starting the analysis ‐ please wait patiently
**************************************************

30477: cpd
(No symbols found)
0x77751db7 : select + 57 (libc‐2.3.2.so) (112, 7f9b7de0, 7f9b85e0, 7f9b8de0, 779c3244, 0) + 1850
0x779acaac : T_event_mainloop_iter_select + 1ec (libComUtils.so) (87f8b90, 779c3244, 96, 779a6259, 779a4560, 779c3738) + 20
0x779ad23f : T_event_mainloop_iter + 16f (libComUtils.so) (87f8b90, 77f95218, 0, 885dad8, 7f9b9678, 77fb5250) + 10
0x779ad298 : T_event_mainloop_e + 28 (libComUtils.so) (87f8b90, 805359c)
0x77f12402 : opsec_mainloop + 22 (libopsec.so) (885dc88, 0, 80516f2, 770d, 0, 0) + 9a0
0x0804cfd8 : /opt/CPshrd‐R71/bin/cpd + 4fd8 (1, 7f9ba0c4, 7f9ba0cc, 0, 777b1898, 77fbe020) + 40
0x776917fd : __libc_start_main + ed (libc‐2.3.2.so) (804cb10, 1, 7f9ba0c4, 80509e0, 8050a28, 77fb5be0) + 80645f48

**************************************************
This script has completed the analysis
**************************************************

Going to TAR the output files :
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
tar: Removing leading `/' from member names
tmp/pstack_30477.out
tmp/pstack_30477_FINAL_OUTPUT.txt
tmp/pstack_30477_maps.out
tmp/pstack_30477_objdump.out
tmp/pstack_30477_proc‐stat.txt
tmp/pstack_30477_ps_auxxxwwf.txt
tmp/pstack_30477_top.txt
opt/CPshrd‐R71/log/cpwd.elg
opt/CPshrd‐R71/log/cpwd.elg.1
opt/CPshrd‐R71/log/cpwd.elg.2
var/log/messages
var/log/messages.1
var/log/messages.2
var/log/messages.3
var/log/messages.4

Please send the following file to Check Point Support...
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
/tmp/pstack_30477.tar
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐

[Expert@FW]#
(52) Advanced diagnostics Memory
1. cpview
Background:
Displays the Memory utilization (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the Memory utilization during the problem
Analysis:
On the 'I/S' tab, go to 'Memory' menu go to 'FWKernel' menu refer to section 'KMEM:' look at counter 'Failed'
On the 'I/S' tab, go to 'Memory' menu go to 'FWKernel' menu refer to section 'HMEM:' look at counter 'Failed'
On the 'I/S' tab, go to 'Memory' menu go to 'FWKernel' menu refer to section 'SMEM:' look at counter 'Failed'
On the 'I/S' tab, go to 'Memory' menu go to 'SMEMFailures' menu
On the 'I/S' tab, go to 'Memory' menu go to 'Contexts' menu
2. vmstat [n] [delay_in_sec [number_of_samples]]
Background:
Manual page http://linux.die.net/man/8/vmstat
Displays information about processes, memory, paging, block IO, and CPU activity
Usage:
By default, the output's header (with names of columns) is displayed every 20 samples. If output is redirected to a file, then it will be very difficult to analyze it using such
commands as 'grep' / 'awk'. Therefore, if output is redirected to a file, use the 'n' flag to display the header only once (at the very top) run:
[Expert@HostName]# vmstat n [delay_in_sec [number_of_samples]]
By default, only 1 sample is printed (with the average values since boot). Therefore, to collect this output continuously, specify the delay between the samples run:
[Expert@HostName]# vmstat [n] delay_in_sec [number_of_samples]
If you specified the delay between the samples, then the output will be collected indefinitely (until the command is stopped or killed). If you want to limit the output, then
specify the total number of samples run:
[Expert@HostName]# vmstat [n] delay_in_sec number_of_samples
Diagnostics:
Analysis:
Look at the "procs" section number of processes waiting for CPU (r)
Look at the "memory" section sum of these counters should be compared to the total amount of RAM
Look at the "swap" section reading from swap file (si) and writing to swap file (so) 209.87.209.101
Example:
2 0 323280 100128 103192 198132 1 2 8 35 117 130 2 3 96 0 0
2 0 323280 100400 103196 198132 0 0 0 260 1297 6904 1 4 95 0 0
1 0 323280 100404 103196 198132 0 0 0 0 1289 7100 1 4 95 0 0
7 0 323280 100252 103200 198132 0 0 0 32 1308 6971 0 2 97 0 0
1 0 323280 100428 103200 198132 0 0 0 0 1265 7111 1 3 97 0 0
3. sar [q][r][B][W] [interval_in_sec [number_of_samples]]
Background:
Usage:
Diagnostics:
Analysis:
Look at the number of tasks waiting for run time counter runqsz
Look at the system load average counters ldavg1, ldavg5, ldavg15
Look at the percentage of used memory counter memused
Look at the percentage of used swap memory counter swpused
Look at the total number of kB the system paged in from disk per second counter pgpgin/s
Look at the total number of kB the system paged out to disk per second counter pgpgout/s
Look at the number of page faults (major + minor) made by the system per second counter fault/s
Look at the number of major faults the system has made per second counter majflt/s
Look at the total number of swap pages counter pswpin/s and counter pswpout/s
Example:
[Expert@R77‐GW:0]# sar ‐q
Linux 2.6.18‐92cp (R77‐GW) 08/04/16
16:10:01 runq‐sz plist‐sz ldavg‐1 ldavg‐5 ldavg‐15
16:20:01 1 114 1.11 1.04 1.06
16:30:01 0 115 1.07 1.13 1.09
16:40:01 0 115 1.15 1.11 1.09
16:50:01 1 114 1.03 1.04 1.06
17:00:01 0 114 1.10 1.09 1.08
17:10:01 1 114 1.08 1.08 1.08
17:20:01 1 112 1.01 1.10 1.09
17:30:01 0 112 1.23 1.11 1.09
17:40:01 1 112 1.15 1.12 1.09
17:50:01 0 114 1.14 1.10 1.09
Average: 0 114 1.11 1.09 1.08
[Expert@R77‐GW:0]# sar ‐r
Linux 2.6.18‐92cp (R77‐GW) 08/04/16
16:10:01 kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
16:20:01 1221768 784220 39.09 43328 381364 2128604 0 0.00 0
16:30:01 1220728 785260 39.15 44648 381408 2128604 0 0.00 0
16:40:01 1219280 786708 39.22 45880 381432 2128604 0 0.00 0
16:50:01 1218016 787972 39.28 47300 381456 2128604 0 0.00 0
17:00:01 1216792 789196 39.34 48628 381420 2128604 0 0.00 0
17:10:01 1215504 790484 39.41 50076 381444 2128604 0 0.00 0
17:20:01 1214964 791024 39.43 51272 381472 2128604 0 0.00 0
17:30:01 1213444 792544 39.51 52524 381492 2128604 0 0.00 0
17:40:01 1212624 793364 39.55 53568 381436 2128604 0 0.00 0
17:50:01 1210840 795148 39.64 54940 381488 2128604 0 0.00 0
Average: 1216396 789592 39.36 49216 381441 2128604 0 0.00 0
[Expert@R77‐GW:0]# sar ‐B
Linux 2.6.18‐92cp (R77‐GW) 08/04/16
16:10:01 pgpgin/s pgpgout/s fault/s majflt/s
16:20:01 0.00 34.64 4417.09 0.00
16:30:01 0.03 37.03 4506.64 0.00
16:40:01 0.01 35.28 4402.80 0.00
16:50:01 0.00 36.98 4462.80 0.00
17:00:01 0.00 36.63 4519.38 0.00
17:10:01 0.00 37.15 4489.80 0.00
17:20:01 0.00 35.27 4440.20 0.00
17:30:01 0.01 35.86 4487.54 0.00
17:40:01 0.01 35.87 4460.36 0.00
Average: 0.01 36.08 4464.99 0.00
[Expert@R77‐GW:0]# sar ‐W
Linux 2.6.18‐92cp (R77‐GW) 08/04/16
16:10:01 pswpin/s pswpout/s
16:20:01 0.00 0.00
16:30:01 0.00 0.00
16:40:01 0.00 0.00
16:50:01 0.00 0.00
17:00:01 0.00 0.00
17:10:01 0.00 0.00
17:20:01 0.00 0.00
17:30:01 0.00 0.00
17:40:01 0.00 0.00
Average: 0.00 0.00
4. valgrind
Background:
Manual pages:
209.87.209.101
http://linux.die.net/man/1/valgrind
http://linux.die.net/HOWTO/ValgrindHOWTO/index.html
Suite of tools for debugging and profiling programs
Check Point uses this tool mostly to detect memory leaks in User Space processes
Installation:
Note: This tool is for 32bit OS if running Gaia OS in 64bit, then first you have to switch to 32bit kernel and reboot.
A. Download the package that contains the 'valgrind' utility and relevant instructions.
B. Transfer the package to the problematic machine (into some directory, e.g., /some_path_to_valgrind/
).
C. Navigate to the directory with the package:
[Expert@HostName]# cd /some_path_to_valgrind/
D. Unpack the package:
[Expert@HostName]# tar xvf VALGRIND_v3.7.0.tar
E. Unpack the 'valgrind' tool:
[Expert@HostName]# tar xvzf valgrind.tar.gz
F. Copy the 'valgrind' files to relevant directories:
[Expert@HostName]# cd valgrind_install_files
[Expert@HostName]# cp r bin/* /usr/bin
[Expert@HostName]# cp r lib/* /usr/lib
G. Test the 'valgrind' tool:
[Expert@HostName]# valgrind version
Important Notes:
A. When a command / daemon is started under 'valgrind', 'valgrind' loads all the relevant library files to read the relevant information from them (to know how to collect the
memory data). This process (loading of library files) takes time and might cause the CPU load to increase up to 100%.
Therefore, schedule a maintenance window to start the 'valgrind'.
After 'valgrind' loads all the library files, it does not cause the additional load on CPU.
B. Do NOT KILL the 'valgrind' it will NOT be able to save the collected data.
You have to TERMINATE the 'valgrind':
[Expert@HostName]# kill TERM $(pidof valgrind)
C. If VPND daemon has to be started under 'valgrind', then first you need to run this command (to prevent the VPND daemon from breaking the pipe to FWD daemon used to
send the PID of the VPND process back to FWD daemon for monitoring purposes):
[Expert@HostName]# export FWD_NO_CHILD_UP_DBG=1
Usage:
Refer to 'Collect_Relevant_Information.txt ' file inside the package.
To correctly stop and start the daemons using the WatchDog, refer to sk97638 Check Point Processes and Daemons.
Analysis:
All the outputs will be analyzed by Check Point Support and R&D
Example for FWM daemon:
i. Stop the FWM daemon:
[Expert@HostName]# cpwd_admin list
[Expert@HostName]# cpwd_admin stop name FWM path "$FWDIR/bin/fw" command "fw kill fwm"

ii. Note: If VPND daemon has to be started under 'valgrind', then first you need to run this command (to prevent the VPND daemon from breaking the pipe to
FWD daemon used to send the PID of the VPND process back to FWD daemon for monitoring purposes):

[Expert@HostName]# export FWD_NO_CHILD_UP_DBG=1
iii. Start FWM daemon under 'valgrind':
[Expert@HostName]# valgrind logfile=/var/log/valgrind_output.txt verbose tracechildren=yes track
fds=yes timestamp=yes leakcheck=full leakresolution=high showreachable=yes errorlimit=no smc

check=stack demangle=yes $FWDIR/bin/fwm
iv. Replicate the issue.
v. Terminate (do NOT kill) the valgrind:
[Expert@HostName]# kill TERM $(pidof valgrind) 209.87.209.101
Note: Wait for several minutes for 'valgrind' to write its summary report.
vi. Start the FWM daemon:
[Expert@HostName]# cpwd_admin start name FWM path "$FWDIR/bin/fwm" command "fwm"

vii. Sent for analysis:
/var/log/valgrind_output*
$FWDIR/log/fwm.el*
$CPDIR/log/cpwd.el*
5. slabtop [d refresh_in_sec] [sSort_Criteria]
Background:
Manual pages:
http://linux.die.net/man/1/slabtop
http://linux.die.net/man/5/slabinfo
http://linux.die.net/man/5/proc
and http://en.wikipedia.org/wiki/Slab_allocation
Displays kernel slab cache information in real time (caches of frequently used objects in the Linux kernel buffer heads, inodes, dentries, etc.)
Instead of manually parsing the highly verbose /proc/slabinfo file manually, it is recommended to use the /usr/bin/slabtop program.
Usage:
By default, output is refreshed every 3 seconds, unless you use the "d refresh_in_sec" option
By default, output is sorted by the number of objects (column "OBJS", which is equal to using the "so" option).
Recommended output sort is by the cache size of the slab (column "CACHE SIZE", use the "sc" option).
Diagnostics:
Analysis:
Provide the output(s) to Check Point Support (involvement of R&D is required)
The most interesting information is the amount of resources a certain kernel module is using (if this amount seems too high, then there may be something wrong with this
module)
Some of the more commonly used statistics from /proc/slabinfo file that are included in the output of /usr/bin/slabtop:
OBJS The total number of objects (memory blocks), including those in use (allocated), and some spares not in use
ACTIVE The number of objects (memory blocks) that are in use (allocated)
USE Percentage of total objects that are active [(ACTIVE/OBJS)*100% ]
OBJ SIZE The size of the objects
SLABS The total number of slabs
OBJ/SLAB The number of objects that fit into a slab
CACHE SIZE The cache size of the slab
NAME The name of the slab
Example of 'slabtop' output (excerpt):
Active / Total Objects (% used) : 160565 / 174730 (91.9%)
Active / Total Slabs (% used) : 5005 / 5005 (100.0%)
Active / Total Caches (% used) : 94 / 141 (66.7%)
Active / Total Size (% used) : 19571.55K / 20754.12K (94.3%)
Minimum / Average / Maximum Object : 0.01K / 0.12K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
85536 82456 96% 0.05K 1188 72 4752K buffer_head
12876 10385 80% 0.13K 444 29 1776K dentry_cache
9516 9498 99% 0.05K 122 78 488K selinux_inode_security
7571 6711 88% 0.03K 67 113 268K size‐32
7524 6869 91% 0.09K 171 44 684K vm_area_struct
7056 7056 100% 0.48K 882 8 3528K ext3_inode_cache
5936 5703 96% 0.27K 424 14 1696K radix_tree_node
4524 4453 98% 0.05K 58 78 232K sysfs_dir_cache
Example of 'cat /proc/slabinfo' output (excerpt):
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables ...
bridge_fdb_cache 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0
jbd_1k 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0
jbd_4k 1 1 4096 1 1 : tunables 24 12 8 : slabdata 1 1 0
......
scsi_io_context 0 0 104 37 1 : tunables 120 60 8 : slabdata 0 0 0
ext3_inode_cache 4596 4888 492 8 1 : tunables 54 27 8 : slabdata 611 611 0
ext3_xattr 0 0 48 78 1 : tunables 120 60 8 : slabdata 0 0 0
journal_handle 89 169 20 169 1 : tunables 120 60 8 : slabdata 1 1 0
journal_head 205 432 52 72 1 : tunables 120 60 8 : slabdata 6 6 0
......
size‐32(DMA) 0 0 32 113 1 : tunables 120 60 8 : slabdata 0 0 0
size‐64 3724 4071 64 59 1 : tunables 120 60 8 : slabdata 69 69 0
size‐32 17449 18306 32 113 1 : tunables 120 60 8 : slabdata 162 162 60
kmem_cache 141 165 256 15 1 : tunables 120 60 8 : slabdata 11 11 0
6. fw ctl failmem PERCENTAGE_of_ALLOCATIONS_to_FAIL
Background:
Administrator can test the behaviour of the Security Gateway during a memory shortage by failing a certain percentage of all memory allocations
Usage:
Refer to sk100766 Simulation of low memory resources on Security Gateway random failing of memory allocations
Diagnostics: 209.87.209.101
Run this test to check how the Security Gateway is going to behave during a memory shortage
Analysis:
Monitor Security Gateway's resources utilization during this test Memory, CPU, Interfaces, how traffic is passing, etc.
(53) Advanced diagnostics Network interface cards
1. netstat s
Background:
Displays summary statistics for each protocol
Diagnostics:
Analysis:
In the "Ip" section, look at "incoming packets discarded"
In the "Icmp" section, look at "ICMP messages failed"
In the "Tcp" section, look at "bad segments received"
In the "Udp" section, look at "packet receive errors"
Search for lines with "error", "fail", "timeout", "loss", "lost"
Example:
Ip:
17862464 total packets received
0 forwarded
0 incoming packets discarded
948002 incoming packets delivered
788408 requests sent out
287 outgoing packets dropped
1 dropped because of missing route
Icmp:
297 ICMP messages received
0 input ICMP message failed.
......
IcmpMsg:
InType0: 7
InType3: 290
OutType3: 290
OutType8: 11
Tcp:
66034 active connections openings
11704 passive connection openings
23572 failed connection attempts
......
Udp:
15488 packets received
287 packets to unknown port received.
......
TcpExt:
1 packets pruned from receive queue because of socket buffer overrun
1142 TCP sockets finished time wait in fast timer
......
IpExt:
InMcastPkts: 420
InBcastPkts: 277064
2. ethtool S name_of_interface
Background:
Displays NICspecific and driverspecific statistics
Diagnostics:
Analysis:
Check every line that contains "error", "drop", "buffer", "fail"
Example:
NIC statistics:
rx_packets: 21610536
tx_packets: 130262
rx_bytes: 1712923726
tx_bytes: 34001412
......
rx_errors: 0
tx_errors: 0
tx_dropped: 0
......
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
......
3. sar [n { DEV | EDEV }] [interval_in_sec [number_of_samples]]
209.87.209.101
Background:
Usage:
Diagnostics:
Analysis:
Look at all the counters
Example:
[Expert@R77‐GW:0]# sar ‐n DEV 1 2
Linux 2.6.18‐92cp (R77‐GW) 08/04/16
19:12:02 IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
19:12:03 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19:12:03 eth0 1.01 2.02 107.07 193.94 0.00 0.00 0.00
19:12:03 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19:12:03 IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
19:12:04 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19:12:04 eth0 6.06 9.09 595.96 1076.77 0.00 0.00 0.00
19:12:04 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth0 3.54 5.56 351.52 635.35 0.00 0.00 0.00
Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[Expert@R77‐GW:0]# sar ‐n EDEV 1 2
Linux 2.6.18‐92cp (R77‐GW) 08/04/16
19:12:36 IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
19:12:37 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19:12:37 eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19:12:37 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19:12:37 IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
19:12:38 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19:12:38 eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
19:12:38 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
(54) Advanced diagnostics SecureXL
Refer to sk98722 ATRG: SecureXL.
1. cpview
Background:
Displays the SecureXL status and statistics (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the SecureXL performance during the problem
Analysis:
On the 'Traffic' tab, go to 'Overview' menu refer to section 'Templates:'
On the 'Traffic' tab, go to 'Overview' menu refer to section 'Logged drops:' look at counter 'SXL'
On the 'I/S' tab, go to 'SXL' menu go to 'F2FReasons' menu
On the 'I/S' tab, go to 'SXL' menu go to 'DropReasons' menu
2. Forwarded to Firewall (F2F) traffic
Background:
Some connections cannot be accelerated by design refer to sk32578 SecureXL Mechanism
Example output of 'fwaccel conns' command (all IP addresses were replaced by "X"):
Source SPort Destination DPort PR Flags C2S i/f S2C i/f
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐ ‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐
X.X.X.X 161 X.X.X.X 5002 17 F...... eth2/eth2 ‐/‐
X.X.X.X 21891 X.X.X.X 0 1 F...... eth2/eth4 eth4/eth2
X.X.X.X 34303 X.X.X.X 389 6 F.N.... eth2/eth2 eth2/‐
209.87.209.101
Diagnostics:
Collect this information during the problem (Important Note: These commands might increase CPU and memory consumption)
Analysis:
To quickly determine the amount of F2F connections:
1. Check the total number of connections in SecureXL connections table:
[Expert@HostName]# fwaccel conns s
2. Check the total number of F2F connections:
[Expert@HostName]# fwaccel conns | grep ' F' | grep v 'Flags' | wc l
To see why specific connections are Forwarded to Firewall, run the following kernel debug (debug will apply only to new connections):
[Expert@HostName]# fw ctl debug 0
[Expert@HostName]# fwaccel dbg resetall
[Expert@HostName]# sim dbg resetall
[Expert@HostName]# fw ctl debug ‐buf 32000
[Expert@HostName]# fwaccel dbg + offload
[Expert@HostName]# sim dbg + f2f
[Expert@HostName]# fwaccel dbg list
[Expert@HostName]# sim dbg list
[Expert@HostName]# fw ctl kdebug ‐T ‐f > /var/log/debug.txt
3. fwaccel stats
Background:
Displays SecureXL acceleration statistics
Diagnostics:
Analysis:
Refer to Performance Pack Administration Guide (R70, R71, R75, R75.20, R75.40, R75.40VS) Chapter 'Command Line' fwaccel stats and fwaccel6 stats.
Refer to Performance Tuning Administration Guide (R76, R77) Chapter 'Performance Pack' Command Line fwaccel stats and fwaccel6 stats.
Example:
Name Value Name Value
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
Accelerated Path
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
accel packets 0 accel bytes 0
conns created 60665 conns deleted 60606
... ...
Accelerated VPN Path
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
C crypt conns 0 enc bytes 0
dec bytes 0 ESP enc pkts 0
... ...
Medium Path
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
PXL packets 0 PXL async packets 0
PXL bytes 0 C PXL conns 0
C PXL templates 0
Accelerated QoS Path
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
QXL packets 0 QXL async packets 0
QXL bytes 0 C QXL conns 0
C QXL templates 0
Firewall Path
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
F2F packets 14870123 F2F bytes 966432478
C F2F conns 59 TCP violations 0
C partial conns 0 C anticipated conns 0
port alloc f2f 0
General
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
memory used 0 free memory 0
C used templates 0 pxl tmpl conns 0
... ...
(*) Statistics marked with C refer to current value, others refer to total value
4. cat /proc/ppk/stats
Background:
Displays total number of packets that passed through interfaces
Diagnostics:
Example:
IRQ | Stats | Interface
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
67 793651087 eth0
75 0 eth1
83 0 eth2
59 0 eth3
Ifn1 | Ifn2 | Stats
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
5. cat /proc/ppk/conf
209.87.209.101
Background:
Displays SecureXL configuration and basic statistics
Diagnostics:
Analysis:
For Flags line, refer to "(61B) 'sim' command" section 'sim if' command
For TmplQuota lines, refer to "(61B) 'sim' command" section 'sim tmplquota' command
For Debug flags section, refer to "(61B) 'sim' command" section 'sim dbg' command
(Note: the numbers in the left column correspond to order of debug modules in the output of 'sim dbg list' command)
Example:
Flags : 0x00009a16
Accounting Update Interval : 60
Conn Refresh Interval : 512
SA Sync Notification Interval : 0
UDP Encapsulation Port : 0
Min TCP MSS : 0
TCP Auto‐Expire Timeout : 20
Connection Limit : 18446744073709551615
TmplQuota Enabled : 0
TmplQuota Quota (rate) : 512
TmplQuota Drop Dduration : 300
TmplQuota Monitor only : 0
TmplQuota Dropped pkts : 0
Total Number of conns : 54
Number of F2F conns : 54
Number of Crypt conns : 0
Number of TCP conns : 18
Number of Non‐TCP conns : 36
Number of Delayed TCP conns : 0
Number of Delayed Non‐TCP conns: 0
Debug flags :
0 : 0x0
1 : 0x0
2 : 0x0
3 : 0x0
4 : 0x0
5 : 0x0
6 : 0x0
7 : 0x0
8 : 0x0
9 : 0x0
10 : 0x0
11 : 0x0
12 : 0x0
13 : 0x0
14 : 0x0
15 : 0x0
6. cat /proc/ppk/ifs
Background:
Displays the list of interfaces used and seen by the SecureXL
Diagnostics:
Analysis:
Look at the Name column and Address column
Look at the F column (refer to "(61B) 'sim' command" section 'sim if' command)
Example:
No | Interface | Address | IRQ | F | SIM F | Dev | Output Func | Features
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
2 | eth0 | 172.30.41.90 | 67 | 39 | 0 | 0xeffc2000 | 0xf0a862e0 | 0x000013a0
3 | eth1 | 10.10.10.90 | 75 | 29 | 0 | 0xefc33000 | 0xf0a862e0 | 0x000013a0
4 | eth2 | 20.20.20.90 | 83 | 29 | 0 | 0xef7fd000 | 0xf0a862e0 | 0x000013a0
5 | eth3 | 30.30.30.90 | 59 | 29 | 0 | 0xef496000 | 0xf0a862e0 | 0x000013a0
7. cat /proc/ppk/affinity
Background:
Displays the status and the thresholds for New Affinity (feature will be activated only if there is no massive VPN traffic and the packetspersecond rate (cutthrough) is high
enough to benefit from the new affinity; feature will be activated only if CPU strength is greater than 3 GHz)
Diagnostics:
Example:
New affinity enabled : no
Min CPU speed to act (MHz) : 3000
Min accelerated PPS to act : 250000
Max enc. bytes rate to act : 10000
Current accelerated PPS : 0
Current enc. bytes rate : 0
Currently new sim affinity set : no
8. cat /proc/ppk/cpls
Background:
Displays the SecureXL configuration for ClusterXL Load Sharing support
209.87.209.101
Diagnostics:
Analysis:
The most important data in this output is the value of fwha_conf_flags it should include the value of 0x1 (i.e., ACTIVE).
Example:
fwha_conf_flags: 0
fwha_df_type: 0
fwha_member_id: 255
fwha_port: 0
fwha_src_mac: 00 00 00 00 00 00
fwha_src_mac_mask: 00 00 00 00 00 00
udp_enc_port: 0
selection table size: 0
selection table:
00 00 00 00 00 00 00 00 00 00
9. cat /proc/ppk/statistics
Background:
Displays SecureXL statistics the same output as from 'fwaccel stats l' command
Diagnostics:
Example:
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
conns created 67518 conns deleted 67475
temporary conns 0 templates 0
nat conns 0 accel packets 0
accel bytes 0 F2F packets 16599213
ESP enc pkts 0 ESP enc err 0
ESP dec pkts 0 ESP dec err 0
ESP other err 0 espudp enc pkts 0
espudp enc err 0 espudp dec pkts 0
espudp dec err 0 espudp other err 0
AH enc pkts 0 AH enc err 0
AH dec pkts 0 AH dec err 0
AH other err 0 memory used 0
free memory 0 acct update interval 60
current total conns 43 TCP violations 0
conns from templates 0 TCP conns 15
delayed TCP conns 0 non TCP conns 28
delayed nonTCP conns 0 F2F conns 43
F2F bytes 1076600709 crypt conns 0
enc bytes 0 dec bytes 0
partial conns 0 anticipated conns 0
dropped packets 0 dropped bytes 0
nat templates 0 port alloc templates 0
conns from nat templ 0 port alloc conns (tm 0
port alloc f2f 0 PXL templates 0
PXL conns 0 PXL packets 0
PXL bytes 0 PXL async packets 0
conns auto expired 0 C used templates 0
pxl tmpl conns 0 C conns from tmpl 0
C non TCP F2F conns 28 C tcp handshake conn 14
C tcp established co 1 C tcp closed conns 0
C tcp f2f handshake 14 C tcp f2f establishe 1
C tcp f2f closed con 0 C tcp pxl handshake 0
C tcp pxl establishe 0 C tcp pxl closed con 0
QXL templates 0 QXL conns 0
QXL packets 0 QXL bytes 0
QXL async packets 0 outbound packets 0
outbound pxl packets 0 outbound f2f packets 124862
outbound bytes 0 outbound pxl bytes 0
outbound f2f bytes 33552839 trimmed pkts 0
10. cat /proc/ppk/drop_statistics
Background:
Displays SecureXL drop statistics
Diagnostics:
Example:
Reason Packets Reason Packets
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
general reason 0 PXL decision 0
fragment error 0 hl ‐ spoof viol 0
F2F not allowed 0 hl ‐ TCP viol 0
corrupted packet 0 hl ‐ new conn 0
clr pkt on vpn 0 partial conn 0
encrypt failed 0 drop template 0
decrypt failed 0 outb ‐ no conn 0
interface down 0 cluster error 0
XMT error 0 template quota 0
anti spoofing 0 Attack mitigation 0
local spoofing 0 sanity error 0
monitored spoofed 0 QXL decision 0
11. cat /proc/ppk/viol_statistics
Background:
Displays violations statistics
Diagnostics:
209.87.209.101
Example:
Violation Packets Violation Packets
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
pkt is a fragment 0 pkt has IP options 406
ICMP miss conn 932 TCP‐SYN miss conn 169
TCP‐other miss conn 17 UDP miss conn 10305933
other miss conn 0 VPN returned F2F 0
ICMP conn is F2Fed 7 TCP conn is F2Fed 23665
UDP conn is F2Fed 6275273 other conn is F2Fed 0
uni‐directional viol 0 possible spoof viol 0
TCP state viol 0 out if not def/accl 0
bridge, src=dst 0 routing decision err 0
sanity checks failed 0 temp conn expired 0
fwd to non‐pivot 0 broadcast/multicast 0
cluster message 0 partial conn 0
PXL returned F2F 0 cluster forward 0
chain forwarding 0 general reason 0
12. cat /proc/ppk/mcast_statistics
Background:
Displays SecureXL multicast statistics
Diagnostics:
Example:
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
in packets 406 out packets 0
if restricted 0 conns with down if 0
f2f packets 0 f2f bytes 0
dropped packets 0 dropped bytes 0
accel packets 0 accel bytes 0
mcast conns 1
(55) Advanced diagnostics CoreXL
Refer to sk98737 ATRG: CoreXL.
1. fw i FW_INSTANCE_ID tab t connections [flags]
Background:
Displays the Connections Table for specified CoreXL FW instance
Diagnostics:
Example:
localhost:
connections
dynamic, id 8158, attributes: keep, sync, aggressive aging, kbufs 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34, expires 25, refresh, , hashsize
1048576, unlimited
<00000001, ac1e295a, 0000b47d, 54279820, 00000050, 00000006; 00020001, 00046000, 49000000, 00000164, 00000000, 531e03e8, 00000000, 5a291eac, c0000000,
ffffffff, ffffffff, ffffffff, 00000001, 00000000, 00000000, 00008000, 49196800, 00000000, 00000000, 00000000, 00000000, 00000000, fb13c000, 00000000,
0f13d800, 00000000, 00000000, 00000000, df175800, 00000000, 00000000; 25/25>
<00000000, ac1e7560, 0000008a, ac1effff, 0000008a, 00000011; 00010001, 00004000, 00000001, 0000018b, 00000106, 531e03da, 00000000, 5a291eac, c0000000,
00000001, ffffffff, ffffffff, ffffffff, 02000000, 00000000, 00000000, 99163800, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000,
7a179800, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000; 23/40>
2. cat $FWDIR/conf/fwaffinity.conf
Background:
Displays the CoreXL affinity configuration
Diagnostics:
Analysis:
Check the affinity settings
Example:
i eth0 1
n fwd 3
3. cat /etc/fw.boot/boot.conf
Background:
Displays the CoreXL FW instances configuration
Diagnostics:
Collect this output to see the current status if CoreXL configuration is not saved correctly / does not survive reboot
Analysis:
209.87.209.101
Check each line under the 'DEFAULT_FILTER_PATH
'
Example:
CTL_IPFORWARDING 1
DEFAULT_FILTER_PATH /etc/fw.boot/default.bin
KERN_INSTANCE_NUM 3
COREXL_INSTALLED 1
KERN6_INSTANCE_NUM 2
IPV6_INSTALLED 0
4. cpview
Background:
Displays the CoreXL status and statistics (and many other counters). Refer to sk101878 CPView Utility.
Diagnostics:
Monitor the CoreXL performance during the problem
Analysis:
On the 'I/S' tab, go to 'CoreXL' menu go to 'General' menu
On the 'I/S' tab, go to 'CoreXL' menu go to 'Global' menu
On the 'I/S' tab, go to 'CoreXL' menu go to 'Instances' menu activate the statistics (!may affect performance)
(6) Command Line syntax

(61) Command Line syntax SecureXL
(61A) FWACCEL (controls acceleration feature)
Show / Hide SecureXL 'fwaccel' syntax
[Expert@HostName]# fwaccel <parameter> [h]
[Expert@HostName]# fwaccel6 <parameter> [h]
Command Description
fwaccel help Prints the general help message with available parameters
fwaccel ehelp Prints the extended help message with available parameters
fwaccel <parameter> h Prints the specific help message for given <parameter>
Starts acceleration onthefly (if Performance Pack is installed)
"q" flag suppresses the output
"a" flag means to start acceleration on all Virtual Systems
Returned strings:
SecureXL device is enabled.
Failed to start SecureXL.
No license for SecureXL.
SecureXL is disabled by the firewall. Please try again later.
The installed SecureXL device is not compatible with the installed firewall
(version mismatch).
The SecureXL device is in the process of being stopped. Please try again later.
SecureXL cannot be started while "flows" are active.
SecureXL is already started.
fwaccel on [a] [q] SecureXL will be started after a policy is loaded.
fwaccel: Failed to check FloodGate1 status. Acceleration will not be started.
FW1: SecureXL acceleration cannot be started while QoS is running in express
mode.
Please disable FloodGate1 express mode or SecureXL.
FW1: SecureXL acceleration cannot be started while QoS is running with citrix
printing rule.
Please remove the citrix printing rule to enable SecureXL.
FW1: SecureXL acceleration cannot be started while QoS is running with UAS
rule.
Please remove the UAS rule to enable SecureXL.
FW1: SecureXL acceleration cannot be started while QoS is running.
Please remove the QoS blade to enable SecureXL.
Failed to enable SecureXL device
fwaccel_on: failed to set process context <VSID>
Stops acceleration onthefly
"q" flag suppresses the output
"a" flag means to stop acceleration on all Virtual Systems
Returned strings:
fwaccel off [a] [q]
SecureXL device disabled
SecureXL device is not active
Failed to disable SecureXL device
fwaccel_on: failed to set process context <VSID>
Shows SecureXL FWAccel and FireWall version
Example:
fwaccel ver Firewall version: R77.10 ‐ Build 243
Acceleration Device: Performance Pack
Accelerator Version 2.1
Firewall API version: 2.90NG (13/6/2013)
Accelerator API version: 2.90NG (13/6/2013) 209.87.209.101
Shows SecureXL status
"a" flag means to show SecureXL status for all Virtual Systems
Example:
Accept Templates : enabled
Returned strings:
Accelerator Status:
Accelerator Status : off
Accelerator Status : disabled by Firewall
Accelerator Status : waiting for policy load
Accelerator Status : no license for SecureXL
Accept Templates:
fwaccel stat [a]
Accept Templates : enabled
Accept Templates : not supported by the SecureXL device
Accept Templates : disabled by user
Drop Templates (refer to sk66402):
Drop Templates : enabled
Drop Templates : not supported by the SecureXL device
Drop Templates : disabled by Firewall
Drop Templates : disabled by user
NAT Templates (refer to sk71200):
NAT Templates : enabled
NAT Templates : not supported by the SecureXL device
NAT Templates : disabled by Firewall
fwaccel stats Prints all acceleration statistics (output is divided into relevant sections)
fwaccel stats h Prints the help message with available flags for 'stats' parameter
fwaccel stats s Prints the summary of all acceleration statistics
fwaccel stats d Prints the acceleration statistics for dropped packets
Prints the acceleration statistics for Network Access Control (NAC)
fwaccel stats n
Note: Refer to Identity Awareness Administration Guide (R75, R75.20, R75.40, R75.40VS, R76, R77)
fwaccel stats p Prints the acceleration statistics for SecureXL violations (F2F packets)
fwaccel stats l Prints all acceleration statistics in Legacy mode (output is not divided into sections)
fwaccel stats m Prints the acceleration statistics for multicast traffic
fwaccel stats r Resets all acceleration statistics
Prints the SecureXL Connections Table ('cphwd_db')
Note: Depending on the number of concurrent connections, might consume memory at very high level
Returned strings:
fwaccel conns The SecureXL connections table is empty
Failed to read the SecureXL connections table
There are VALUE connections in SecureXL connections table
...
Total number of connections: VALUE
fwaccel conns h Prints the help message with available flags for 'conns' parameter
Prints the summary of SecureXL Connections Table (number of connections)
fwaccel conns s
Note: Depending on the number of concurrent connections, might consume memory at very high level
fwaccel conns m max_entries Prints the SecureXL Connections Table limited to the number of max_entries
fwaccel conns f <filter> Prints the SecureXL Connections Table entries based on <filter> (run 'fwaccel conns h');
Filtering flag is a single letter (either capital, or small)
Prints the SecureXL Connections Templates ('cphwd_tmpl')
Note: Depending on the number of current templates, might consume memory at very high level
Returned strings:
fwaccel templates The SecureXL templates table is empty
Failed to read the SecureXL templates table
There are VALUE templates in SecureXL templates table
...
Total number of templates: VALUE
fwaccel templates h Prints the help message with available flags for 'templates' parameter
Prints the summary of SecureXL Connections Templates (number of templates)
fwaccel templates s
Note: Depending on the number of current templates, might consume memory at very high level
fwaccel templates d Prints the summary of SecureXL Connections Templates for dropped packets
fwaccel templates m max_entries Prints the SecureXL Connections Templates limited to the number of max_entries
Prints the SecureXL Identities Table ('cphwd_dev_identity_table')
Returned strings:
The SecureXL identity table is empty
Failed to read the SecureXL identity table no memory
There are VALUE identities in SecureXL identities table
fwaccel identities ...
Total number of identities: VALUE
Notes:
Refer to Identity Awareness Administration Guide (R75, R75.20, R75.40, R75.40VS, R76, R77)

Command is not available for IPv6
fwaccel identities h Prints the help message with available flags for 'identities' parameter
fwaccel identities s Prints the summary of SecureXL Identities Table (number of entries) 209.87.209.101
fwaccel identities m max_entries Prints the SecureXL Identities Table limited to the number of max_entries
Prints the SecureXL Revoked IPs Table ('cphwd_dev_revoked_ips_table')
Returned strings:
The SecureXL templates' revoked IPs table is empty
Failed to read the SecureXL templates' revoked IPs table no memory
There are VALUE revoked IPs in SecureXL templates' revoked IPs table
fwaccel revoked_ips ...
Total number of templates' revoked IPs:: VALUE
Notes:
Refer to Identity Awareness Administration Guide (R75, R75.20, R75.40, R75.40VS, R76, R77)

Command is not available for IPv6
fwaccel revoked_ips h Prints the help message with available flags for 'revoked_ips' parameter
fwaccel revoked_ips s Prints the summary of SecureXL Revoked IPs Table (number of entries)
fwaccel revoked_ips m max_entries Prints the SecureXL Revoked IPs Table limited to the number of max_entries
Controls SecureXL debugging (run 'fwaccel dbg h')
By default, debug messages will be printed to /var/log/messages file, therefore you must set the usual kernel debugging
options with:
fwaccel dbg [Expert@HostName]# fwaccel dbg resetall
[Expert@HostName]# fw ctl debug buf 32000
[Expert@HostName]# fwaccel dbg m MODULE + FLAG1 FLAG2 ... FLAGn
[Expert@HostName]# fw ctl kdebug T f > /var/log/debug.txt
fwaccel dbg h Prints the help message with available options, list of debug modules and their flags
fwaccel dbg list Prints all currently enabled debug flags for all modules
fwaccel dbg resetall Resets all debug flags for all modules to their default (none)
fwaccel dbg m MODULE reset Resets all debug flags for specified module to their default (none)
fwaccel dbg m MODULE all Enables all supported debug flags for specified module
fwaccel dbg m MODULE + FLAG1 FLAG2 ... FLAGn Enables specified debug flags for specified module
fwaccel dbg m MODULE FLAG1 FLAG2 ... FLAGn Disables specified debug flags for specified module
Sets debugging filter only the specified connection will be printed in the debug output
Notes:
Only 1 filter can be set at one time
fwaccel dbg f Source_IP,Source_Port,Dest_IP,Dest_Port,Proto
You can use the asterisk "*" as a wildcard for IP/port/proto
Example for SSH connection:
[Expert@HostName]# fwaccel dbg f 172.30.1.1,*,172.30.41.90,22,6
fwaccel dbg f reset Resets the debugging filter
(61B) SIM (controls acceleration device)
Show / Hide SecureXL 'sim' syntax
[Expert@HostName]# sim <parameter> [h]
[Expert@HostName]# sim6 <parameter> [h]
Command Description
sim help Prints the general help message with available parameters
sim ver [k] Shows SecureXL SIM version
Prints the list of interfaces used and seen by the SecureXL implementation (Performance Pack)
Configuration flags
(sum of the following values in the "F" column):
0x0001 If set, the packet should be dropped at the end of the inbound processing, if the packet is "cutthrough"
packet. In outbound, all the packets should ne forwarded to the network.
0x0002 If set, and the SIM "tcp" feature is enabled (on), then an appropriate notification should be sent whenever a
TCP state change occurs (connection established/teared down).
0x0004 If set, then when encapsulating an encrypted packet (UDP encapsulation), the UDP header's checksum field
should be set correctly. If flag is not set, then the UDP header's checksum field should be set to zero. It is safe to
ignore this flag if it is set to 0 (i.e., still calculate the checksum).
0x0008 If set, then when the Connections Table has reached the specified limit, new connections that match a
template should not be created and the packet matching the template should be dropped. If flag is not set, the packet
should be forwarded to the firewall.
0x0010 If set, then fragments should be forwarded to the firewall.
0x0020 If set, then connection creation from templates should be disabled. Connection can still be offloaded to the
device. This flag disables only the creation of TCP templates.
0x0040 Accelerated connections should periodically be refreshed in the firewall through the notification. The
refreshing should be done only if this global flag is set.
0x0080 If set, then connection creation from templates should be disabled. Connection can still be offloaded to the
device. This flag disables only the creation of nonTCP templates.
0x0100 If set, then sequence verification violations should be allowed for connections that did not complete the 3way
handshake process (instead of F2Fing violating packets).
00x200 If set, then sequence verification violations should be allowed for connections that completed the 3way
sim if handshake process (instead of F2Fing violating packets).
0x0400 If set, then RST TCP packets should be forwarded to firewall.
0x0800 If set, then Path MTU Discovery should not be enforced for IP multicast packets.
0x1000 If set, then SIM "drop_templates" feature should be disabled.
0x2000 Indicates whether Link Selection Load Sharing feature was enabled by the user.
0x4000 If set, then SecureXL asynchronic notification feature should be disabled.
0x8000 Indicates that Firewall Connections Table capacity is unlimited.
Examples:
F=039 means the sum of the following flags:
0x0001
0x0008
0x0010
0x0020
F=0x00009a16 means the sum of the following flags:
0x0002 209.87.209.101
0x0004
0x0010
00x200
0x0800
0x1000
0x8000
Enables/Disables acceleration of VPN traffic:
sim vpn <on | off> sim vpn on = enable acceleration of VPN traffic
sim vpn off = disable acceleration of VPN traffic
sim ranges Prints the loaded ranges
sim ranges l Prints the list of loaded ranges
sim ranges a Prints all loaded ranges
sim ranges range_id Prints specified loaded range
sim ranges s range_id Prints summary for specified loaded range
Controls network interfaces' affinity settings
sim affinity
Note: Command is available only on Linuxbased OS
sim affinity h Prints the help message with available options for 'affinity' parameter
sim affinity l Prints the current network interfaces' affinity
sim affinity a Sets the network interfaces' affinity in 'Automatic' mode
sim affinity s Sets the network interfaces' affinity in 'Static' ('Manual') mode
Controls SIM module's template quota
sim tmplquota
sim tmplquota h Prints the help message with available options for 'tmplquota' parameter
Enables/Disables template quota feature:
sim tmplquota e <1 | 0> sim tmplquota e 1 = enable the template quota feature
sim tmplquota e 0 = disable the template quota feature
sim tmplquota q <quota> Sets maximum of connections per second per template (quota is allowed)
sim tmplquota d <drop_duration> Sets drop duration (in seconds) for drop state
Controls monitor only mode:
sim tmplquota m <0 | 1> sim tmplquota m 0 = disable monitor only mode (default)
sim tmplquota m 1 = enable monitor only mode
sim tmplquota r Resets SIM module template quota values to their defaults
sim tmplquota d <file_name> Loads exclusion list of Source IP addresses / Source Subnets from the file
Prints only templates in drop state (output is printed into /var/log/messages files and into Linux kernel ring buffer
sim tab d templates
(output of 'dmesg' command))
Applies SIM Affinity in 'Automatic' mode
sim affinityload
Configures drop parameters (run 'sim dropcfg')
Notes:
sim dropcfg Command is available only on Linuxbased OS.
Feature is available in R75.40 and above.
Refer to sk67861 Accelerated Drop Rules Feature in R75.40 and above.
sim dropcfg h Prints the help message with available options for 'dropcfg' parameter
sim dropcfg l Prints current drop configuration
Sets drop configuration file
Note: The drop rules configuration does not survive the reboot. Therefore, in order to apply the configured drop rules after the
sim dropcfg f </path_to/file_name>
reboot, use a startup script (e.g., /etc/rc.d/rc.local) to run the 'sim dropcfg f </path_to/file_name>'
command automatically during each boot)
sim dropcfg e Enforces drop configuration on the external interface only
sim dropcfg y Avoids confirmation
sim dropcfg r Resets drop rules
Controls SecureXL SIM debugging (run 'sim dbg h')
By default, debug messages will be printed to /var/log/messages file, therefore you must set the usual kernel debugging
options with:
sim dbg [Expert@HostName]# fwaccel dbg resetall
[Expert@HostName]# fw ctl debug buf 32000
[Expert@HostName]# sim dbg m MODULE + FLAG1 FLAG2 ... FLAGn
[Expert@HostName]# fw ctl kdebug T f > /var/log/debug.txt
sim dbg h Prints the help message with available options, list of debug modules and their flags
sim dbg list Prints all currently enabled debug flags for all modules
sim dbg resetall Resets all debug flags for all modules to their default (none)
sim dbg m MODULE reset Resets all debug flags for specified module to their default (none)
sim dbg m MODULE all Enables all supported debug flags for specified module
sim dbg m MODULE + FLAG1 FLAG2 ... FLAGn Enables specified debug flags for specified module
sim dbg m MODULE FLAG1 FLAG2 ... FLAGn Disables specified debug flags for specified module
Sets debugging filter only the specified connection will be printed in the debug output
Notes:
sim dbg f Source_IP,Source_Port,Dest_IP,Dest_Port,Proto Only 1 filter can be set at one time
You can use the asterisk "*" as a wildcard for IP/port/proto
Example for SSH connection:
[Expert@HostName]# sim dbg f 172.30.1.1,*,172.30.41.90,22,6
sim dbg f reset Resets the debugging filter
sim feature Controls SIM module features
sim feature h Prints the help message with available options for 'feature' parameter
209.87.209.101
Enables/Disables the specified SIM module feature
Available features:
anti_spoofing
delayed
drop_templates
dynamic_vpn
linksel
sim feature <feature_name> <on | off> linksel_ls
mcast_route
mcast_route_v2
nac
qos
routing
streaming
tcp
vpn
wire
(62) Command Line syntax CoreXL
(62A) Show / Hide CoreXL syntax in Gateway mode
[Expert@HostName]# fw ctl affinity [flag1] [flag2] ...
[Expert@HostName]# fw6 ctl multik <parameter>
[Expert@HostName]# fw6 ctl affinity [flag1] [flag2] ...
Command Description
fw ctl multik Controls CoreXL FW instances
fw ctl multik Prints the general help message with available parameters
fw ctl multik stat Prints the summary table for CPU cores and CoreXL FW instances
fw ctl multik start Starts CoreXL
fw i Instance_ID ctl multik start Starts specific CoreXL FW instance
fw ctl multik stop Stops CoreXL
fw i Instance_ID ctl multik stop Stops specific CoreXL FW instance
fw ctl affinity <options> Controls CoreXL affinities of interfaces/processes/CoreXL FW instances to CPU cores
fw ctl affinity Prints the help message with available options
fw d ctl affinity corelicnum Prints the number of system CPU cores allowed by CoreXL license
fw ctl affinity l Prints the current CoreXL affinities output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores
Prints the current CoreXL affinities in reverse order output shows CPU cores and which interface/process/CoreXL FW instance
fw ctl affinity l r
is affined to each CPU core
Prints all current CoreXL affinities output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores, and also
fw ctl affinity l a
shows targets without specific affinity
Prints the current CoreXL affinities verbose output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores
fw ctl affinity l v
(targets are shown as 'Interface' (with IRQ), 'Kernel', 'Process'
Prints the current CoreXL affinities output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores, and
fw ctl affinity l q
suppresses errors
fw ctl affinity l r a v Prints the current CoreXL affinities verbose output that combines all possible outputs (shows all targets in reverse order)
fw ctl affinity l p PID [r] [a] [v] Prints the current CoreXL affinity of the specified process (by PID) to CPU cores
fw ctl affinity l n Daemon_Name [r] [a] [v] Prints the current CoreXL affinity of the specified process (by name [maximal length = 255 characters]) to CPU cores
fw ctl affinity l k Instance_ID [r] [a] [v] Prints the current CoreXL affinity of the specified CoreXL FW instance to CPU cores
fw ctl affinity l i Interface_Name [r] [a] [v] Prints the current CoreXL affinity of the specified interface to CPU cores
fw ctl affinity s <target> { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL Affinity
fw ctl affinity s p PID { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL affinity of the specified process (by PID) to CPU cores
fw ctl affinity s n Daemon_Name { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL affinity of the specified process (by name [maximal length = 255 characters]) to CPU cores
fw ctl affinity s k Instance_ID { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL affinity of the specified CoreXL FW instance to CPU cores
fw ctl affinity s i Interface_Name { CPU_ID [ CPU_ID ... ] | all } Sets CoreXL affinity of the specified interface to CPU cores
(62B) Show / Hide CoreXL syntax in VSX mode
[Expert@HostName]# fw ctl affinity [flag1] [flag2] ...
[Expert@HostName]# fw6 ctl multik <parameter>
[Expert@HostName]# fw6 ctl affinity [flag1] [flag2] ...
Command Description
fw ctl multik Controls CoreXL FW instances
fw ctl multik stat Prints the summary table for CPU cores and CoreXL FW instances
fw ctl multik stat Prints the general help message with available parameters
fw ctl multik start Starts CoreXL
fw i Instance_ID ctl multik start Starts specific CoreXL FW instance
fw ctl multik stop Stops CoreXL
fw i Instance_ID ctl multik stop Stops specific CoreXL FW instance
fw ctl affinity <options> Controls CoreXL affinities of Virtual Devices/interfaces/processes/CoreXL FW instances to CPU cores209.87.209.101
fw ctl affinity Prints the help message with available options
fw d ctl affinity corelicnum Prints the number of system CPU cores allowed by CoreXL license
fw ctl affinity l Prints the current CoreXL affinities
fw ctl affinity l r Prints the current CoreXL affinities in reverse order
fw ctl affinity l a Prints all current CoreXL affinities
fw ctl affinity l v Prints the current CoreXL affinities verbose output
fw ctl affinity l q Prints the current CoreXL affinities and suppresses errors
fw ctl affinity l r a v Prints the current CoreXL affinities verbose output that combines all possible outputs (shows all targets in reverse order)
Prints the current CoreXL affinities extended output
Notes:
If "vsid" flag is omitted, the current context will be used (in which the command was issued)
The "vsid" flag accepts either a single VSID (e.g., 'vsid 7'), or a range of VSID numbers (e.g., 'vsid 05
or a combination (e.g., 'vsid 02 4')
The "cpu" flag accepts either a single CPU ID (e.g., 'cpu 7'), or a range of CPU ID numbers (e.g., 'cpu 05
fw ctl affinity l x [vsid VSID_ranges] [cpu CPU_ID_ranges] [flags e|k|t|n|h] [ or a combination (e.g., 'cpu 02 4')
r] [a] [v] The "flags" requires at least one of the following arguments (multiple arguments must be specified together):
e do not print exception processes
k do not print kernel threads
t print all process threads
n print process name instead of /proc/<PID>/cmdline
h print CPU mask in Hex format
o print output into the '/tmp/affinity_list_output' file (VSX R77.10 and above)
fw ctl affinity s Sets CoreXL Affinity
fw ctl affinity s i Interface_Name CPU_IDs | all Sets affinities of the specified interface to CPU cores
fw ctl affinity s p PID CPU_IDs | all Sets affinities of the specified process (by PID) to CPU cores
Sets affinity of the specified process (by name) to CPU cores
Notes:
fw ctl affinity s d pname Daemon_Name [vsid VSID_ranges] cpu CPU_ID_ranges The "vsid" flag accepts either a single VSID (e.g., 'vsid 7'), or a range of VSID numbers (e.g., 'vsid 05
or a combination (e.g., 'cpu 02 4')
Sets affinities of Virtual Devices (VS, VR, VSW) to CPU cores
Notes:
fw ctl affinity s d [vsid VSID_ranges] cpu CPU_ID_ranges The "vsid" flag accepts either a single VSID (e.g., 'vsid 7'), or a range of VSID numbers (e.g., 'vsid 05
Sets affinities of the specified FWK instances to CPU cores
Notes:
The "inst" flag accepts either a single FWK_ID (e.g., 'inst 7'), or a list of FWK_ID numbers (e.g., 'inst 0
fw ctl affinity s d inst Instances_ranges cpu CPU_ID_ranges 2 4')
fw ctl affinity s d fwkall Number_of_CPUs Sets affinities of all FWK instances to CPU cores (where Number_of_CPUs is an integer number)
fw ctl affinity vsx_factory_defaults Resets all VSX affinity settings (prompts the user) (VSX R77 and above)
fw ctl affinity vsx_factory_defaults_no_prompt Resets all VSX affinity settings (does not prompt the user) (VSX R77 and above)
(63) Command Line syntax MultiQueue
Show / Hide MultiQueue 'cpmq' syntax
[Expert@HostName]# cpmq <parameter>
Command Description
cpmq get Shows MultiQueue status of active supported interfaces
cpmq get a Shows MultiQueue status of all supported interfaces (those with enabled MultiQueue and those with disabled MultiQueue)
cpmq get v Shows MultiQueue status of active supported interfaces with IRQ affinity information
cpmq get rx_num igb Shows the number of active RX queues for interfaces that use igb driver
cpmq get rx_num ixgbe Shows the number of active RX queues for interfaces that use ixgbe driver
cpmq set [f] Enables/Disables MultiQueue per interface (the "f" flag forces the operation)
Sets the number of active RX queues to the number of CPU cores that run as CoreXL SND (CPU cores that are not used by
cpmq set rx_num all default [f] CoreXL FW instances) for all interfaces (those that use igb driver and those that use ixgbe driver)
Note: This is recommended configuration
Sets the number of active RX queues to the number of CPU cores, which are not used by CoreXL FW instances (recommended)
cpmq set rx_num igb default [f]
for interfaces that use igb driver
Sets the number of active RX queues to the number of CPU cores, which are not used by CoreXL FW instances (recommended)
cpmq set rx_num ixgbe default [f]
for interfaces that use ixgbe driver
cpmq set rx_num all <number> [f] Sets the number of active RX queues to the number between 2 and the number of CPU cores for all interfaces (those that
use igb driver and those that use ixgbe driver)
cpmq set rx_num igb <number> [f] Sets the number of active RX queues to the number between 2 and the number of CPU cores for interfaces that use igb driver
cpmq set rx_num ixgbe <number> [f] Sets the number of active RX queues to the number between 2 and the number of CPU cores for interfaces that use ixgbe driver
Sets the IRQ affinity for MultiQueue interfaces after the following occurs (in the given order):
A. MultiQueue is enabled on an interface
B. The interface status is changed to 'down'
cpmq set affinity
C. The machine is rebooted
D. The interface status is changed back to 'up'
Run this command after the interface status is changed back to 'up' to reset the IRQ affinity for this interface.
Reconfigures the MultiQueue (the "q" flag suppresses the output):
After changing the number of CoreXL FW instances
cpmq reconfigure [q]
After changing the physical interfaces on the machine
After changing the number of CPU cores, to which the fwk processes on VSX are assigned 209.87.209.101
(7) Examples
Show / Hide example #1
1. Output of 'top' command shows that 'fw_worker_X' processes constantly consume the CPU at 100%:
4083 admin 15 0 0 0 0 S 2 100 101:01.47 fw_worker_1
4019 admin 15 0 0 0 0 S 2 100 101:01.47 fw_worker_2
4023 admin 15 0 0 0 0 S 2 100 101:01.47 fw_worker_5
2. Output of 'fwaccel stat' command shows very poor acceleration ratios and high percentage of packets that are passing through Medium path (PXL):
Accelerated conns/Total conns : 364/13215 (2%)
Delayed conns/(Accelerated conns + PXL conns) : 48/12023 (0%)
F2Fed pkts/Total pkts : 36776/564927 (6%)
QXL pkts/Total pkts : 0/564927 (0%)
3. Output of 'fwaccel stats s' command shows that 'Accept Templates' are disabled after specific rule:
4. Thorough analysis of Firewall's Connections Table (using 'connstat' utility from sk85780) during the issue (~99800 concurrent connections) showed that most matched rules are
lower than Rule #251 (after which SecureXL Accept Templates are disabled):
Top 10 Rules:
=============
Rule: 342 Hits: 71463
Rule: 343 Hits: 16732
Rule: 356 Hits: 4488
Rule: 51 Hits: 722
Rule: 25 Hits: 666
Rule: 362 Hits: 426
5. Rulebase was optimized per:
sk32578 SecureXL Mechanism
sk42401 Factors that adversely affect performance in SecureXL
sk63400 SecureXL Traffic Limitations
6. Performance has improved greatly after this change.
7. Output of 'fwaccel stats s' command showed that 'Accept Templates' are now disabled after much lower rule:
8. The following steps were taken to decrease the percentage of packets that are passing through Medium path (PXL):
A. Current IPS profile:
2400 protections in 'Detect' mode
870 protections in 'Prevent' mode
B. Created new IPS profile according to the environment:
15 protections in 'Detect' mode
570 protections in 'Prevent' mode
All other protections were set to 'Inactive'
C. In the Application Control & URL Filtering policy, disabled the "Any Any Allow" rule.
D. In the AntiVirus & AntiBot configuration, excluded networks, whose traffic does not have to be inspected per sk92515 How to configure AntiVirus Exceptions.
9. Performance has improved greatly after the overall optimization:
Before the optimization:
After the optimization:
209.87.209.101
Show / Hide example #2
Relevant outputs were collected during a period of time from Security Gateway with the help of the shell script that runs relevant commands every 1 second. All outputs were analyzed and
correlated to each other.
1. Output of 'vmstat' command showed that CPU time is consumed by:
IOWait : min 0% , average 36.6% , max 82%
Kernel Space : min 11% , average 16.2% , max 45%
User Space : min 0% , average 1.8% , max 36%
This leaves IDLE at min 0% , average 45.3% , max 89%
Example:
......
1 7 334816 708020 820 37388 2152 0 3280 4 18012 17602 0 13 7 80 0
......
0 9 334348 652800 988 29420 1664 0 3944 4 21575 19725 1 13 5 80 0
......
1 12 291604 723400 884 29696 196 0 4272 12 20799 18273 5 18 0 77 0
2. Output of 'cat /proc/stat' command showed that CPU time is consumed by:
User Space : min 0% , average 1% , max 7%
Kernel Space : min 0% , average 1% , max 3%
Idle : min 16% , average 34% , max 54%
IOWait : min 32% , average 48% , max 69%
IRQ : min 0% , average 1% , max 1%
SoftIRQ : min 11% , average 14% , max 19
3. Output of 'top' command showed:
fw_worker_0 / fw_worker_1 / fw_worker_2 constantly consume CPU at ~1520%
fw_worker_X spike to 2025%
'fw' spikes up to 50, 60, 80%
fwssd , in.asessiond , in.aufpd , in.ufclnt , mdq , igwd , sds , stormd , dtps
Example:
6283 root 19 0 54688 16m 12m D 87 0.8 0:00.56 fw
2283 root 15 0 0 0 0 R 30 0.0 4844:09 fw_worker_2
2209 root 15 0 0 0 0 R 16 0.0 5254:57 fw_worker_1
2144 root 15 0 0 0 0 S 14 0.0 5227:59 fw_worker_0
4. Output of 'ps auxwf' command correlated to the output of 'top' command showed that during the spikes of CPU load by 'fw', the CPD daemon calls the 'sim affinity c'
command, which calls a series of Check Point shell script that calculate the current load by running various 'fw ctl' commands. These commands process high amount of data
and this causes the spike in CPU load.
The CPD daemon calling the 'sim affinity c' command shows that SIM Affinity is configured in Automatic mode (refer to sk63330 Explanation about 'sim affinity c' ,
'fwaffinity_used_cpus' , 'fw ctl affinity l v'):
root 15829 0.1 1.2 231704 25748 ? Ssl Jul23 49:45 \_ cpd
root 5153 0.0 0.0 2832 1088 ? S 12:19 0:00 | \_ sim affinity ‐c
root 5212 0.0 0.0 1236 564 ? S 12:19 0:00 | \_ sh ‐c $FWDIR/scripts/fwaffinity_used_cpus > /tmp/sim_fw_cpus.tmp
root 5213 0.0 0.0 1236 564 ? S 12:19 0:00 | \_ /bin/sh /opt/CPsuite‐R75.20/fw1/scripts/fwaffinity_used_cpus
root 5216 0.0 0.8 55636 18080 ? S 12:19 0:00 | \_ fw ctl affinity ‐l ‐v
root 5226 0.0 0.0 1236 552 ? S 12:19 0:00 | | \_ sh ‐c /bin/netstat ‐n 2>&1
root 5227 0.0 0.0 1668 560 ? R 12:19 0:00 | | \_ /bin/netstat ‐n
root 5217 0.0 0.0 1644 476 ? S 12:19 0:00 | \_ grep ‐E ‐v ^Interface.*:
root 5218 0.0 0.0 2028 644 ? S 12:19 0:00 | \_ awk ‐F: {print $2}
root 5219 0.0 0.0 1628 464 ? S 12:19 0:00 | \_ sed ‐e s/CPU//
root 5220 0.0 0.0 1604 372 ? S 12:19 0:00 | \_ tr [:space:] \n
root 5221 0.0 0.0 1644 468 ? S 12:19 0:00 | \_ grep ‐v ^[:space:]*$
root 5222 0.0 0.0 35424 508 ? S 12:19 0:00 | \_ sort
root 5223 0.0 0.0 1592 384 ? S 12:19 0:00 | \_ uniq
5. SIM Affinity was configured from Auto Mode to Static Mode:
[Expert@HostName]# sim affinity s
6. To minimize the load caused by IO Waiting, Linux kernel was finetuned per sk60703 IOWait consumes 100% CPU after Security policy installation.
(8) Related documentation
Show / Hide related documentation
SecureXL (Performance Pack):
Performance Pack Administration Guide (R75, R75.20, R75.40, R75.40VS).
Performance Tuning Administration Guide (R76, R77) Chapter 1 'Performance Pack'.
CoreXL:
Firewall Administration Guide (R75, R75.20, R75.40, R75.40VS) Chapter 'CoreXL Administration'. 209.87.209.101
Performance Tuning Administration Guide (R76, R77) Chapter 'CoreXL Administration'.
sk61701 CoreXL Known Limitations
MultiQueue:
Gaia Administration Guide (R75.40, R75.40VS, R76, R77).
SecurePlatform Administration Guide (R75, R75.40, R75.40VS, R76, R77).
Command Line Interface Reference Guide (R75, R75.20, R75.40, R75.40VS, R76, R77).
Hardware Compatibility List.
(9) Related solutions
Show / Hide related solutions
Performance:
sk105119 Best Practices VPN Performance
sk101878 CPView Utility
sk103212 Traffic analysis using 'cpmonitor' tool
sk85780 How to use the 'connstat' utility
sk106410 How to check Top Rule Hits on Security Gateway
sk33781 Performance analysis for Security Gateway NGX R65 / R7x
sk43733 How to measure CPU time consumed by IPS protections
sk67560 How to export History Report from SmartView Monitor
sk35496 How to detect a memory leak on Security Gateway with SecurePlatform OS / Gaia OS
sk100766 Simulation of low memory resources on Security Gateway random failing of memory allocations
sk36846 Interpreting high SoftIRQ values
sk42181 How to increase sizes of buffer on SecurePlatform Gaia for Intel NIC and Broadcom NIC
sk43772 'kernel: neighbour table overflow' appears repeatedly in /var/log/messages files
sk92372 How to change the size of IPv6 Neighbors cache table
sk71001 High Connection Capacity (64bit) on Gaia
sk22343 What is the maximum memory supported by SecurePlatform
sk98711 Virtual Memory consumption on Security Gateway increases when CPU is under high load
sk72860 How to reset the 'Hit Count' in SmartDashboard
sk79300 How to correlate a rule from SmartDashboard to its corresponding entry in kernel table to see the rule's Hit Count
sk60703 IOWait consumes 100% CPU after Security policy installation
SecureXL:
sk98722 ATRG: SecureXL
sk42401 Factors that adversely affect performance in SecureXL
sk63400 SecureXL Traffic Limitations
sk71200 SecureXL NAT Templates
sk66402 SecureXL Drop Templates are not supported in versions lower than R76
sk90861 Optimized Drops feature in R76 and above
sk74520 What is the SecureXL penalty box mechanism for offending IP addresses?
209.87.209.101
sk98229 Enabling QoS support for acceleration technologies (SecureXL and CoreXL)
sk104679 SecureXL does not accelerate traffic when ISP Redundancy is enabled
sk33250 Automatic SIM Affinity on MultiCore CPU Systems
sk61962 SMP IRQ Affinity on Check Point Security Gateway
sk63330 Explanation about 'sim affinity c' , 'fwaffinity_used_cpus' , 'fw ctl affinity l v'
CoreXL:
sk98737 ATRG: CoreXL
sk61701 CoreXL Known Limitations
sk98229 Enabling QoS support for acceleration technologies (SecureXL and CoreXL)
sk62620 What is the fw_worker_X process?
sk61962 SMP IRQ Affinity on Check Point Security Gateway
sk35990 How Connections Table limit capacity behaves in CoreXL
sk43443 How to debug CoreXL
sk42096 Cluster member is stuck in 'Ready' state
sk63330 Explanation about 'sim affinity c' , 'fwaffinity_used_cpus' , 'fw ctl affinity l v'
sk65463 'Peak' number of connections discrepancy between the output of 'fw tab t connections s' command and the output of 'fw ctl pstat' command when CoreXL is
enabled
SMT (HyperThreading):
MultiQueue:
sk80940 MultiQueue hotfix for Security Gateway
Check Point Infrastructure:
sk97638 Check Point Processes and Daemons
sk52421 Ports used by Check Point software
sk97443 Support Debug Tools
sk65133 Connections Table Format
How To:
sk65385 "How To" Solutions and Documents
(10) Revision history
Show / Hide revision history
Date Description
04 Aug 2016 Added instructions for Sar command
11 Jan 2016 Corrected the syntax for using pstack script
11 Oct 2015 Updated definition of "Medium path (PXL)"
04 July 2015 Added additional related solution
11 June 2015 Added additional related solution
13 May 2015 Added additional related solution
25 Apr 2015 Updated instructions for Valgrind tool
08 Apr 2015 Added additional related solution
02 Mar 2015 Updated the best practices for Application Control & URL Filtering optimization
209.87.209.101
10 Feb 2015 Added additional related solutions
14 Jan 2015 Corrected the the formula for calculating the maximum number of concurrent connections
20 Nov 2014 Corrected the best practices for Application Control & URL Filtering optimization
05 Nov 2014 Added additional related solutions
08 Sep 2014 Added CPView Utility
02 July 2014 Added additional related solutions
19 May 2014 Added additional related solutions
13 May 2014 Added additional related solutions
12 Apr 2014 Added additional related solutions
Added additional related solutions
16 Mar 2014
Added SecureXL Penalty box feature to SecureXL Limitations
Added additional related solutions
12 Mar 2014 Added instructions for 'valgrind' tool
Added instructions for 'pstack' shell script
11 Mar 2014 First release of this document
Please rate this document [1=Worst,5=Best]
Give us Feedback
Comment Enter your comment here Submit

©2016 Check Point Software Technologies Ltd. All rights reserved. Check Point Software Technologies, Inc. is a wholly owned subsidiary of Check Point Software
Technologies Ltd.
Copyright | Privacy Policy | Site Map
209.87.209.101

Best Practices - Security Gateway Performance PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Best Practices - Security Gateway Performance PDF

Uploaded by

Copyright:

Available Formats

28/10/2016 Best

WINNING THE SECURITY MARKET SALES TOOLS OFFERINGS / UPSELLS ASSETS / INFO SUPPORT / SERVICES

1 Any Any Any Traffic ftp Accept

2 Any Any Any Traffic Any Drop

Number of Default number of Default number of

Refer to Identity Awareness Administration Guide (R75, R75.20, R75.40, R75.40VS, R76, R77)

fwaccel identities s Prints the summary of SecureXL Identities Table (number of entries) 209.87.209.101

Refer to Identity Awareness Administration Guide (R75, R75.20, R75.40, R75.40VS, R76, R77)

Comment Enter your comment here Submit

You might also like

Best Practices - Security Gateway Performance PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Best Practices - Security Gateway Performance PDF

Uploaded by

Copyright:

Available Formats

28/10/2016 Best

WINNING THE SECURITY MARKET SALES TOOLS OFFERINGS / UPSELLS ASSETS / INFO SUPPORT / SERVICES

1 Any Any Any Traffic ftp Accept

2 Any Any Any Traffic Any Drop

Number of Default number of Default number of

Refer to Identity Awareness Administration Guide (R75, R75.20, R75.40, R75.40VS, R76, R77)

fwaccel identities ­s Prints the summary of SecureXL Identities Table (number of entries) 209.87.209.101

Refer to Identity Awareness Administration Guide (R75, R75.20, R75.40, R75.40VS, R76, R77)

Comment Enter your comment here Submit

You might also like

fwaccel identities s Prints the summary of SecureXL Identities Table (number of entries) 209.87.209.101