Professional Documents
Culture Documents
Fortigate Troubleshooting 40 Mr3
Fortigate Troubleshooting 40 Mr3
FortiOS Handbook Troubleshooting v3 24 January 2012 01-431-0129304-20120124 Copyright 2012 Fortinet, Inc. All rights reserved. Contents and terms are subject to change by Fortinet without prior notice. Reproduction or transmission of this publication is encouraged.
Trademarks
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
Visit these links for more information and documentation for your Fortinet products: Fortinet Knowledge Base - http://kb.fortinet.com Technical Documentation - http://docs.fortinet.com Training Services - http://campus.training.fortinet.com Technical Support - http://support.fortinet.com You can report errors or omissions in this or any Fortinet technical document to techdoc@fortinet.com.
FortiOS Handbook
Contents
Introduction
Before you begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How this guide is organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11
11 11
Life of a Packet
Stateful inspection . . . . . . . . . . . . . . . . . . Connections over connectionless . . . . . . . . What is a session? . . . . . . . . . . . . . . . . Differences between connections and sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
13 14 14 14 15 16 16 17 17 18 18 19 19 19 19 19 19 20 20 20 20 20 20 21 21 21 21 21 21 23 24
Flow inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Proxy inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Comparison of inspection layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . FortiOS functions and security layers . . . . . . . . . . . . . . . . . . . . . . . . . Packet flow . . . . . . . . . . . . . Packet inspection (Ingress) . . Interface . . . . . . . . . . . . DoS sensor . . . . . . . . . . . IP integrity header checking . . IPsec . . . . . . . . . . . . . . Destination NAT (DNAT) . . . . Routing . . . . . . . . . . . . . Policy lookup . . . . . . . . . . Session tracking . . . . . . . . User authentication . . . . . . Management traffic. . . . . . . SSL VPN traffic. . . . . . . . . Session helpers . . . . . . . . Flow-based inspection engine . Proxy-based inspection engine IPsec . . . . . . . . . . . . . . Source NAT (SNAT) . . . . . . Routing . . . . . . . . . . . . . Egress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Example 1: client/server connection . . . . . . . . . . . . . . . . . . . . . . . . . . Example 2: Routing table update. . . . . . . . . . . . . . . . . . . . . . . . . . . . Example 3: Dialup IPsec VPN with application control. . . . . . . . . . . . . . . . .
Contents
Troubleshooting process
Establish a baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Define the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gathering Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Search for a solution . . . . . . . . . . . . . . Technical Documentation . . . . . . . . . Release Notes . . . . . . . . . . . . . . . Knowledge Base . . . . . . . . . . . . . . Fortinet Technical Discussion Forums . . . Fortinet Training Services Online Campus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
27 28 29 29 29 29 29 29 30 30 30 30 31 31
Create a troubleshooting plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Providing Supporting Elements . . . . . . . . . . . . . . . . . . . . . . . . . . Obtain any required additional equipment . . . . . . . . . . . . . . . . . . . . . . . Ensure you have administrator level access to required equipment . . . . . . . . . . Contact Fortinet customer support for assistance . . . . . . . . . . . . . . . . . . .
Troubleshooting tools
FortiOS diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . Check date and time . . . . . . . . . . . . . . . . . . . . . . . . . Resource usage . . . . . . . . . . . . . . . . . . . . . . . . . . . How to troubleshoot high memory usage . . . . . . . . . . . . How to troubleshoot high CPU usage . . . . . . . . . . . . . . Proxy operation . . . . . . . . . . . . . . . . . . . . . . . . . . . Hardware NIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hardware troubleshooting . . . . . . . . . . . . . . . . . . . . Conserve mode . . . . . . . . . . . . . . . . . . . . . . . . . . . Antivirus failopen. . . . . . . . . . . . . . . . . . . . . . . . . Traffic trace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Session table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Web-based manager session information . . . . . . . . . . . . How to find which security policy a specific connection is using CLI session information . . . . . . . . . . . . . . . . . . . . . Firewall session setup rate . . . . . . . . . . . . . . . . . . . . . . Finding object dependencies . . . . . . . . . . . . . . . . . . . . CLI method . . . . . . . . . . . . . . . . . . . . . . . . . . . Web-based manager method . . . . . . . . . . . . . . . . . . Flow trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Flow trace output example - HTTP . . . . . . . . . . . . . . . Flow trace output example - IPsec (policy-based) . . . . . . . Packet sniffing and packet capture . . . . . . . . . . . . . . . . . Packet sniffing . . . . . . . . . . . . . . . . . . . . . . . . . . Packet capture . . . . . . . . . . . . . . . . . . . . . . . . . . FA2 and NP2 based interfaces. . . . . . . . . . . . . . . . . . . . Debug command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
33 34 34 35 35 38 39 39 41 41 42 42 43 44 44 46 47 47 47 48 49 51 51 52 53 54 55
Contents
Debug output example Other commands. . . . . . ARP table . . . . . . . Time and date settings. IP address . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
56 57 57 58 58 59 60 60 60 60 61 62 62 63 63
FortiGate ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Diagnostic commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FortiAnalyzer/FortiManager ports . . . . . . . . . . . . . . . . . . . . . . . . . FortiGuard troubleshooting. . . . . . . . . . . . . . Troubleshooting process for FortiGuard updates FortiGuard server settings . . . . . . . . . . . . Displaying the server list . . . . . . . . . . . Sorting the server list . . . . . . . . . . . . Calculating weight . . . . . . . . . . . . . . FortiGuard URL rating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
65
65 66 67 69 69 69 69 71 71 71 72 72 72 72 72 72 73 73 73
Assisting technical support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Support priority levels Priority 1 . . . . . Priority 2 . . . . . Priority 3 . . . . . Priority 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
77
77 78 80 81
Contents
Common issues and questions. . . . . . . . . . . . . . . . Check hardware connections . . . . . . . . . . . . . . Check FortiOS network settings . . . . . . . . . . . . . Interface settings. . . . . . . . . . . . . . . . . . . DNS settings . . . . . . . . . . . . . . . . . . . . . DHCP Server settings . . . . . . . . . . . . . . . . Check CPU and memory resources . . . . . . . . . . . Check modem status . . . . . . . . . . . . . . . . . . Run ping and traceroute . . . . . . . . . . . . . . . . . Ping . . . . . . . . . . . . . . . . . . . . . . . . . Traceroute . . . . . . . . . . . . . . . . . . . . . . Check the logs . . . . . . . . . . . . . . . . . . . . . . Verify the contents of the routing table (in NAT mode) . Check the bridging information in Transparent mode . . What checking the bridging information can tell you How to check the bridging information . . . . . . . How to display forwarding domain information . . . Perform a sniffer trace . . . . . . . . . . . . . . . . . . What can sniffing packets tell you . . . . . . . . . . How do you sniff packets . . . . . . . . . . . . . . Debug the packet flow . . . . . . . . . . . . . . . . . . Check number of sessions used by UTM proxy . . . . . Conserve or failopen mode . . . . . . . . . . . . . Checking sessions in use . . . . . . . . . . . . . . Related commands . . . . . . . . . . . . . . . . . Examine the firewall session list . . . . . . . . . . . . . Check source NAT information . . . . . . . . . . . Checking wireless information . . . . . . . . . . . . . . Troubleshooting station connection issue . . . . . . Enable diagnostic for particular station . . . . . . . Other diagnose commands . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Troubleshooting advanced
Traffic shaping issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . Use traffic shapers to limit traffic in testing and network simulations . . Monitoring traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Displaying configured traffic shaping . . . . . . . . . . . . . . . . . . Troubleshooting protocols and users using traffic shaping . . . . . . . Displaying current bandwidth and dropped packets for a traffic shaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
103
103 104 104 104 105 106 107 107 108 108 108 109 109
User and administrator logon issues . . . . . . . . . . . . . . . . . . . . . . . . . User logon issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Use correct username and password combination for user. . . . . . . . . Check authentication security policies . . . . . . . . . . . . . . . . . . . Use proper two-factor authentication code (FortiToken or delivered code). User credentials must exist on the remote server (remote authentication) . Administrator logon issues . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents
Allow access for interface is not enabled . . . . . . . . . . . . . . . . . . . 109 Trusted hosts for admin account will not allow current IP. . . . . . . . . . . 110 FortiGate asking for password when creating a remotely authenticated administrator account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 IPsec VPN issues . . . . . . . . . . . . . . . . VPN negotiations appear to be slow . . . . Many VPN negotiations take time . . . Keep VPN information up to date . . . Check for routing problems . . . . . . Limit number of P1 proposals . . . . . VPN tunnel proposal will not connect . . . VPN Tunnel up but no traffic going over it . Other useful VPN IKE related commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 112 112 112 112 112 112 115 115 116 116 116 116
Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cannot log to a supported log device . . . . . . . . . . . . The alert email did not send an email to the email address . The FortiGate unit stopped logging: what happened? . . .
119
exec tac report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 get firewall iprope appctrl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 get firewall iprope list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 get firewall proute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 get firewall shaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 get hardware cpu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 get hardware nic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 get hardware memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 get hardware npu list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 get hardware npu performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 get hardware npu status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 get hardware status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 get ips session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 get router info kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 get router info routing-table all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 get system arp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 get system auto-update status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 get system auto-update versions . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 get system ha status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 get system performance firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Troubleshooting for FortiOS 4.0 MR3 01-431-0129304-20120124 http://docs.fortinet.com/
Contents
get system performance status . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 get system performance top . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 get system session-helper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 get system session-info full-stat . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 get system session-info list. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 get system session-info ttl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 get system startup-error-log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 get system status. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 get test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 get test urlfilter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 get vpn ipsec stats crypto . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 get vpn ipsec stats tunnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 get vpn ipsec tunnel details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 get vpn ipsec tunnel summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 get vpn status ssl hw-acceleration-status . . . . . . . . . . . . . . . . . . . . . . . 177 get vpn status ssl list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 get webfilter ftgd-statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 get webfilter status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
183
183 183 183 184 184 184 185 185 185 186 186 186 186 187 187
Contents
Appendix
Document conventions . . . . . . . IP addresses . . . . . . . . . . Example Network configuration Information highlights . . . . . Typographical conventions . . Entering FortiOS configuration data Entering text strings (names) . . Entering numeric values . . . . Selecting options from a list . . Enabling or disabling options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
189
189 189 191 192 192 193 193 193 194 194
Registering your Fortinet product . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Technical Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Comments on Fortinet technical documentation . . . . . . . . . . . . . . . . . 194 Customer service and support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 Fortinet products End User License Agreement . . . . . . . . . . . . . . . . . . . . 195
Index
197
Contents
10
FortiOS Handbook
Introduction
Welcome and thank you for selecting Fortinet products for your network protection. This guide is intended for administrators who need guidance on different network needs and information on basic and advanced troubleshooting. This chapter contains the following topics: Before you begin How this guide is organized
11
Introduction
Troubleshooting bootup and FSSO addresses potential problems your unit may have when booting up. Also covered is troubleshooting an FSSO installation. The format is an easy to follow step by step question and answer format.
12
FortiOS Handbook
Life of a Packet
Directed by security policies, a FortiGate unit screens network traffic from the IP layer up through the application layer of the TCP/IP stack. This chapter provides a general, high-level description of what happens to a packet as it travels through a FortiGate security system. The FortiGate unit performs three types of security inspection: stateful inspection, that provides individual packet-based security within a basic session state flow-based inspection, that buffers packets and uses pattern matching to identify security threats proxy-based inspection, that reconstructs content passing through the FortiGate unit and inspects the content for security threats. Each inspection component plays a role in the processing of a packet as it traverses the FortiGate unit en route to its destination. To understand these inspections is the first step to understanding the flow of the packet. This section contains the following topics: Stateful inspection Flow inspection Proxy inspection Comparison of inspection layers FortiOS functions and security layers Packet flow Example 1: client/server connection Example 2: Routing table update Example 3: Dialup IPsec VPN with application control
Stateful inspection
With stateful inspection, the FortiGate unit looks at the first packet of a session to make a security decision. Common fields inspected include TCP SYN and FIN flags to identity the start and end of a session, the source/destination IP, source/destination port and protocol. Other checks are also performed on the packed payload and sequence numbers to verify it as a valid communication and that the data is not corrupted or poorly formed. What makes it stateful is that one or both ends must save information about the session history in order to communicate. In stateless communication, only independent requests and responses are used, that do not depend on previous data. For example, UDP is stateless by nature because it has no provision for reliability, ordering, or data integrity.
13
Stateful inspection
Life of a Packet
The FortiGate unit makes the decision to drop, pass or log a session based on what is found in the first packet of the session. If the FortiGate unit decides to drop or block the first packet of a session, then all subsequent packets in the same session are also dropped or blocked without being inspected. If the FortiGate unit accepts the first packet of a session, then all subsequent packets in the same session are also accepted without being inspected.
What is a session?
A session is established on an existing connection, for a defined period of time, using a determined type of communication or protocol. Sessions can have specific bandwidth , and time to live (TTL) parameters. You can compare a session to a conversation. A session is established when one end point initiates a request by establishing a TCP connection on a particular port, the receiving end is listening on that port, and replies. You could telent to port 80 even though telnet normally uses port 23, because at this level, the application being used cannot be determined. However, the strong points of sessions and stateful protocols can also be their weak points. Denial of service (DoS) attacks involve creating so many sessions that the connection state information tables are full and the unit will not accept additional sessions.
14
Life of a Packet
Flow inspection
1 3 2
SY N, IP, TC 1 P 2
3
nt Se et ck Pa
1 3 2
d ive ce et e R ck Pa
Flow inspection
With flow inspection, the FortiGate unit samples multiple packets in a session and multiple sessions, and uses a pattern matching engine to determine the kind of activity that the session is performing and to identify possible attacks or viruses. For example, if application control is operating, flow inspection can sample network traffic and identify the application that is generating the activity. Flow-based antivirus can sample network traffic and determine if the content of the traffic contains a virus, IPS can sample network traffic and determine if the traffic constitutes an attack. The security inspection occurs as the data is passing from its source to its destination. Flow inspection identifies and blocks security threats in real time as they are identified. Figure 2: Flow inspection of packets through the FortiGate unit
3 2
nt Se et ck Pa
1 2
ed eiv t c Re cke Pa
Flow-based inspection typically requires less processing than proxy-based inspection, and therefore flow-based antivirus performance can be better than proxy-based antivirus performance. However, some threats can only be detected when a complete copy of the payload is obtained so, proxy-based inspection tends to be more accurate and complete than flow-based inspection.
15
Proxy inspection
Life of a Packet
Proxy inspection
With flow inspection, the FortiGate unit will pass all the packets between the source and destination, and keeps a copy of the packets in its memory. It then uses a reconstruction engine to build the content of the original traffic. The security inspection occurs after the data has passed from its source to its destination. Proxy inspection examines the content contained a content protocol session for security threats. Content protocols include the HTTP, FTP, and email protocols. Security threats can be found in files and other content downloaded using these protocols. With proxy inspection, the FortiGate unit downloads the entire payload of a content protocol session and re-constructs it. For example, proxy inspection can reconstruct an email message and its attachments. After a satisfactory inspection the FortiGate unit passes the content on to the client. If the proxy inspection detects a security threat in the content, the content is removed from the communication stream before the it reaches its destination. For example, if proxy inspection detects a virus in an email attachment, the attachment is removed from the email message before its sent to the client. Proxy inspection is the most thorough inspection of all, although it requires more processing power, and this may result in lower performance. If you enable ICAP in a security policy, HTTP traffic intercepted by the policy is transferred to the ICAP servers in the ICAP profile added to the policy. The FortiGate unit is the surrogate, or middle-man, and carries the ICAP responses from the ICAP server to the ICAP client; the ICAP client then responds back, and the FortiGate unit determines the action that should be taken with these ICAP responses and requests. Figure 3: Proxy inspection of packets through the FortiGate unit
1 3 2
nt Se et ck Pa
1 3 2
ed eiv t c Re cke Pa
16
Life of a Packet
Feature Memory, CPU required Level of threat protection Authentication IPsec and SSL VPN Antivirus protection Application control Delay in traffic Reconstruct entire content
Security Function Firewall IPsec VPN Traffic Shaping User Authentication Management Traffic SSL VPN Intrusion Prevention Flow-based Antivirus Application Control VoIP inspection Proxy Antivirus Email Filtering Web Filtering (Antispam) Data Leak Prevention
Stateful
Flow
Proxy
Packet flow
After the FortiGate units external interface receives a packet, the packet proceeds through a number of steps on its way to the internal interface, traversing each of the inspection types, depending on the security policy and UTM profile configuration. The diagram in Figure 4 on page 18 is a high level view of the packets journey. The description following is a high-level description of these steps as a packet enters the FortiGate unit towards its destination on the internal network. Similar steps occur for outbound traffic.
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
17
Packet flow
Life of a Packet
Packet
Session Helpers
Management Traffic
SSL VPN
User Authentication
Traffic Shaping
Session Tracking
Policy Lookup
No (Fast Path)
UTM
Yes
No
Flow-based Antivirus
Application Control
IPS
Yes
VoIP Inspection Data Leak Prevention
Email Filter
Web Filter
Antivirus
ICAP
IPsec
NAT (SNAT)
3
Routing Interface
1 2
Packet
Interface
Ingress packets are received by a FortiGate interface.The packet enters the system, and the interface network device driver passes the packet to the Denial of Service (DoS) sensors, if enabled, to determine whether this is a valid information request or not.
18
Life of a Packet
Packet flow
DoS sensor
DoS scans are handled very early in the life of the packet to determine whether the traffic is valid or is part of a DoS attack. Unlike signature-based IPS which inspects all the packets within a certain traffic flow, the DoS module inspects all traffic flows but only tracks packets that can be used for DoS attacks (for example TCP SYN packets), to ensure they are within the permitted parameters. Suspected DoS attacks are blocked, other packets are allowed.
IPsec
If the packet is an IPsec packet, the IPsec engine attempts to decrypt it. The IPsec engine applies the correct encryption keys to the IPsec packet and sends the unencrypted packet to the next step. IPsec is bypassed when for non-IPsec traffic and for IPsec traffic that cannot be decrypted by the FortiGate unit.
Routing
The routing step determines the outgoing interface to be used by the packet as it leaves the FortiGate unit. In the previous step, the FortiGate unit determined the real destination address, so it can now refer to its routing table and decide where the packet must go next. Routing also distinguishes between local traffic and forwarded traffic and selects the source and destination interfaces used by the security policy engine to accept or deny the packet.
Policy lookup
The policy look up is where the FortiGate unit reviews the list of security policies which govern the flow of network traffic, from the first entry to the last, to find a match for the source and destination IP addresses and port numbers. The decision to accept or deny a packet, after being verified as a valid request within the stateful inspection, occurs here. A denied packet is discarded. An accepted packet will have further actions taken. If IPS is enabled, the packet will go to Flow-based inspection engine, otherwise it will go to the Proxy-based inspection engine.
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
19
Packet flow
Life of a Packet
If no other UTM options are enabled, then the session was only subject to stateful inspection. If the action is accept, the packet will go to Source NAT to be ready to leave the FortiGate unit.
Session tracking
Part of the stateful inspection engine, session tracking maintains session tables that maintain information about sessions that the stateful inspection module uses for maintaining sessions, NAT, and other session related functions.
User authentication
User authentication added to security policies is handled by the stateful inspection engine, which is why Firewall authentication is based on IP address. Authentication takes place after policy lookup selects a security policy that includes authentication. This is also known as identify-based policies. Authentication also takes place before UTM features are applied to the packet.
Management traffic
This local traffic is delivered to the FortiGate unit TCP/IP stack and includes communication with the web-based manager, the CLI, the FortiGuard network, log messages sent to FortiAnalyzer or a remote syslog server, and so on. Management traffic is processed by applications such as the web server which displays the FortiOS web-based manager, the SSH server for the CLI or the FortiGuard server to handle local FortiGuard database updates or FortiGuard Web Filtering URL lookups.
Session helpers
Some protocols include information in the packet body (or payload) that must be analyzed to successfully process sessions for this protocol. For example, the SIP VoIP protocol uses TCP control packets with a standard destination port to set up SIP calls. To successfully process SIP VoIP calls, FortiOS must be able to extract information from the body of the SIP packet and use this information to allow the voice-carrying packets through the firewall. FortiOS uses session helpers to analyze the data in the packet bodies of some protocols and adjust the firewall to allow those protocols to send packets through the firewall.
Once the packet has passed the flow-based engine, it can be sent to the proxy inspection engine or egress.
20
Life of a Packet
IPsec
If the packet is transmitted through an IPsec tunnel, it is at this stage the encryption and required encapsulation is performed. For non-IPsec traffic (TCP/UDP) this step is bypassed.
Routing
The final routing step determines the outgoing interface to be used by the packet as it leaves the FortiGate unit.
Egress
Upon completion of the scanning at the IP level, the packet exits the FortiGate unit.
21
Life of a Packet
10 Routing 11 Interface transmission to network 12 Packet forwarded to web server Response from web server 1 Web Server sends response packet to client. 2 Packet intercepted by FortiGate unit interface 2.1 Link level CRC and packet size checking. 3 IP integrity header checking. 4 DoS sensor. 5 Proxy inspection 5.1 Antivirus scanning. 6 Source NAT. 7 Stateful Policy Engine 7.1 Session Tracking 8 Next hop route 9 Interface transmission to network 10 Packet returns to client
22
Life of a Packet
FortiGate Unit
IP Integrity Header checking NAT (DNAT)
DoS Sensor
Session Tracking
User Authentication
Policy Lookup
Routing
Antivirus
Web Filter
FortiGuard
NAT (SNAT)
Routing
Packet Exits
Internet
Web Server
Packet Enters
NAT (SNAT)
Session Tracking
Routing
3 2
23
Life of a Packet
6 Routing module 6.1 Update routing table Figure 6: Routing table update
3 2 1
Packet
FortiGate Unit
Interface (Link layer) DoS Sensor IP Integrity Header checking Management Traffic
Routing Table
Routing Module
24
Life of a Packet
3 IP integrity header checking. 4 DoS sensor 5 Flow inspection engine 5.1 IPS 5.2 Application control 6 Stateful policy engine 6.1 Session tracking 7 Next hop route 8 IPsec 8.1 Encrypts packet 9 Routing 10 Interface transmission to network 11 Encrypted Packet returns to internet Figure 7: Dialup IPsec with application control
3 2 1
FortiGate Unit
Interface (Link layer) DoS Sensor IP Integrity Header checking IPsec NAT
Packet decryption
Application Control
IPS
Session Tracking
Packet Exits
Source NAT Routing Interface (Link layer)
3 2 1
Internal Server
Destintion NAT DoS Sensor IP Integrity Header checking Interface (Link layer)
3 2 1
Response Packe
Packet Enters
Application Control IPS Session Tracking Next Hop Route
Packet encryption
3 2
25
Life of a Packet
26
FortiOS Handbook
Troubleshooting process
Before you begin troubleshooting anything but the most minor issues, you need to prepare. Doing so will shorten the time to solve your issue. This section helps to explain how you prepare before troubleshooting, as well as creating a troubleshooting plan and contacting support. This section contains the following topics: Establish a baseline Search for a solution Create a troubleshooting plan Obtain any required additional equipment Ensure you have administrator level access to required equipment Contact Fortinet customer support for assistance
Establish a baseline
FortiGate units operate at all layers of the OSI model. For this reason troubleshooting problems can become complex. If you establish a normal operation parameters, or baseline, for your system before the problem occurs it will help reduce the complexity when you are troubleshooting. Many of the guiding questions in the following sections are some form of comparing the current problem situation to normal operation on your FortiGate unit. For this reason it is a best practice that you know what your normal operating status is, and have a record of it you can refer to. This can easily be accomplished by monitoring the system performance with logs, SNMP tools, or regularly running information gathering commands and saving the output. This regular operation data will show trends, and enable you to see when changes happen and there may be a problem. Back up your FortiOS configuration on a regular basis. This is a good practice for everyday as well as when troubleshooting. You can restore the backed up configuration when needed and save the time and effort of re-creating it from the factory default settings. Some basic commands you can use to obtain normal operating data for your system: get system status diagnose firewall statistic show get router info routing-table all get ips session
Displays versions of firmware and FortiGuard engines, and other system information. Displays the amount of network traffic broken down into categories such as email, VoIP, TCP, UDP, IM, Gaming, P2P, and Streaming. Displays all the routes in the routing table including their type, source, and other useful data. Displays memory used and max available to IPS as well and counts. Displays list of FortiGuard related counts of status,
27
Troubleshooting process
These commands are just a sample. Feel free to include any extra information gathering commands that apply to your system. For example if you have active VPN connections, record information about them using the get vpn * series of commands. See Troubleshooting get commands on page 119. For an extensive snapshot of your system, run the CLI command used by TAC to gather extensive information about a system exec tac report. It runs many diagnose commands that are for specific configurations. This means no matter what features you are using, this command will record their current state. Then if you need to perform troubleshooting at a later date, you can run the same command again and compare the differences to quickly locate suspicious output you can investigate. See exec tac report on page 120.
28
Troubleshooting process
These questions will help you define the problem. Once the problem is defined, you can search for a solution and then create a plan on how to solve it.
Gathering Facts
Fact gathering is an important part of defining the problem. Record the following information as it affects your problem: Where did the problem occur? When did the problem occur and to whom? What components are involved? What is the affected application? Can the problem be traced using a packet sniffer? Can the problem be traced in the session table? Can log files be obtained that indicate a failure has occurred? Answers to these questions will help you narrow down the problem, and what you have to check during your troubleshooting. The more things you can eliminate, the fewer things you need to check during troubleshooting. For this reason, be as specific and accurate as you can while gathering facts.
Technical Documentation
Installation Guides, Administration Guides, Quick Start Guides, and other technical documents are available online at the following URL: http://docs.fortinet.com
Release Notes
Issues that are uncovered after the technical documentation has been published will often be listed in the Release Notes that accompany the device.
Knowledge Base
The Fortinet Knowledge Base provides access to a variety of articles, white papers, and other documentation providing technical insight into a range of Fortinet products. The Knowledge Base is available online at the following URL: http://kb.fortinet.com
29
Troubleshooting process
30
Troubleshooting process
Normally network administrators have additional networking equipment available either to loan you, or a lab where you can bring the FortiGate unit to test. If you do not have access to equipment, check for shareware applications that can perform the same task. Often there are software solutions when hardware is too expensive.
31
Troubleshooting process
32
FortiOS Handbook
Troubleshooting tools
FortiOS provides a number of tools that help with troubleshooting both hardware and software issues. These tools include diagnostics and ports; ports are used when you need to understand the traffic coming in or going out on a specific port, for example, UDP 53, which is used by the FortiGate unit for DNS lookup and RBL lookup. This section also contains information about troubleshooting FortiGuard issues. This section contains the following topics: FortiOS diagnostics FortiGate ports FortiAnalyzer/FortiManager ports FortiGuard troubleshooting
FortiOS diagnostics
A collection of diagnostic commands are available in FortiOS for troubleshooting and performance monitoring. While some of these areas have web-based manager areas, all have relevant CLI commands with the main commands listed in this section. Within the CLI commands, the two main groups of diagnostic commands are get and diagnose commands. Both commands display information about system resources, connections, and settings that enable you to locate and fix problems, or to monitor system performance. The one exception to these two main groups it the command exec tac report. This is an execute command that runs an exhaustive series of diagnostic commands. It runs commands that are only needed if you are using certain features like HA, VPN tunnels, or a modem. The report takes a few minutes to complete due to the amount of output generated. If you have your CLI output logged to a file, you can run this command to familiarize yourself with the CLI commands involved. Do not include the output from this command in FortiCare tickets unless it is specifically requested. See exec tac report on page 120. When you call Fortinet Customer Support, you will be asked to provide information about your unit and its current state using the output from these CLI commands. This topic includes diagnostics commands to help with: Check date and time Resource usage Proxy operation Hardware NIC Conserve mode Traffic trace Session table Firewall session setup rate Finding object dependencies
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
33
FortiOS diagnostics
Troubleshooting tools
Flow trace Packet sniffing and packet capture Debug command Other commands Additional diagnostic commands are covered in Troubleshooting get commands on page 119, and commands related to specific features are covered in the chapter for that specific feature. For example in-depth diagnostics for dynamic routing are covered in the dynamic routing chapter.
Resource usage
Each program running on a computer has one or more processes associated with it. For example if you open a Telnet program, it will have an associated telnet process. The same is true in FortiOS. All the processes have to share the system resources in FortiOS including memory and CPU.
34
Troubleshooting tools
FortiOS diagnostics
Monitor the CPU/memory usage of internal processes using the following command: get system performance top <delay> <max_lines> The data listed includes the name of the daemon, the process ID, whether the process is sleeping or running, the CPU percentage being used, and the memory percentage being used. See Check CPU and memory resources on page 86.
35
FortiOS diagnostics
Troubleshooting tools
sshd
901
0.5
4.0
Where the codes displayed on the second output line mean the following: U is % of user space applications using CPU. In the example, 0U means 0% of the user space applications are using CPU. S is % of system processes (or kernel processes) using CPU. In the example, 0S means 0% of the system processes are using the CPU. I is % of idle CPU. In the example, 98I means the CPU is 98% idle. T is the total FortiOS system memory in Mb. In the example, 123T means there are 123 Mb of system memory. F is free memory in Mb. In the example, 25F means there is 25 Mb of free memory. KF is the total shared memory pages used. In the example, 32KF means the system is using 32 shared memory pages. Each additional line of the command output displays information for each of the processes running on the FortiGate unit. For example, the third line of the output is: newcli Where: newcli is the process name. Other process names can include ipsengine, sshd, cmdbsrv, httpsd, scanunitd, and miglogd. 903 is the process ID. The process ID can be any number. R is the current state of the process. The process state can be: R running S sleep Z zombie D disk sleep. 0.5 is the amount of CPU that the process is using. CPU usage can range from 0.0 for a process that is sleeping to higher values for a process that is taking a lot of CPU time. 5.5 is the amount of memory that the process is using. Memory usage can range from 0.1 to 5.5 and higher. Enter the following single-key commands when diagnose sys top is running: Press q to quit and return to the normal CLI prompt. Press p to sort the processes by the amount of CPU that the processes are using. Press m to sort the processes by the amount of memory that the processes are using. 903 R 0.5 5.5
36
Troubleshooting tools
FortiOS diagnostics
2 Determine what features are using most of the CPU resources. There is a command in the CLI to let you see the top few processes currently running that use the most CPU resources. The CLI command get system performance top outputs a table of information. You are interested in the second most right column CPU usage by percentage. If the top few entries are using most of the CPU, note which processes they are and investigate those features to try and reduce their CPU load. Some examples of processes you will see are ipsengine the IPS engine that scans traffic for instrusions scanunitd antivirus scanner httpsd secure HTTP iked internet key exchange (IKE) in use with IPsec VPN tunnels newcli active whenever you are accessing the CLI sshd there are active secure socket connections cmdbsrv the command database server application Go to the features that are at the top of the list and look for evidence of them overusing the CPU. Generally the monitor for a feature is a good place to start. 3 Check for unnecessary CPU wasters. These are some best practises that will reduce your CPU usage, even if you are not experiencing high CPU usage. Note that if you require a feature this section tell you to turn off, ignore it. Use hardware acceleration wherever possible to offload tasks from the CPU. Offloading tasks such as encryption frees up the CPU for other tasks. Avoid the use of GUI widgets that require computing cycles, such as the Top Sessions widget. These widgets are constantly polling the system for their information which uses CPU and other resources. Schedule antivirus, IPS, and firmware updates during off peak hours. Usually these dont consume CPU resources but they can disrupt normal operation. Check the log levels and which events are being logged. This is the severity of the messages that are recorded. Consider going up one level to reduce the amount of logging. Also if there are events you do not need to monitor, remove them from the list. Log to disk instead of memory. Logging to memory quickly uses up resources. Logging to local disk is fast and doesnt take much CPU. If the disk is almost full, transfer logs or data off the disk to free up space. When a disk is almost full it consumes a lot of resources to find the free space and organize the files. If you have packet logging enabled, consider disabling it. When its enabled it records every packet that comes through that policy. Halt all sniffers and traces. Ensure you are not scanning traffic twice. If traffic enters the FortiGate unit on one interface, goes out another, and then comes back in again that traffic does not need to be rescanned. Doing so is a waste of resources. However, ensure that traffic truly is being scanned once. Reduce the session timers to close unused sessions faster. To do this in the CLI enter the following commands and values. These values reduce the values from defaults. Note that tcp-timewait has 10 seconds added by the system by default.
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
37
FortiOS diagnostics
Troubleshooting tools
config system global set tcp-halfclose-timer 30 set tcp-halfopen-timer 30 set tcp-timewait-timer 0 set udp-idle-timer 60 end Remove dns-udp firewall session helper (number 14) if not used. Do not enable nice to have features. 4 When CPU usage is under control, use SNMP to monitor CPU usage. Alternately, use logging to record CPU and memory usage every 5 minutes. Once things are back to normal, you should set up a warning system to alert you of future CPU overusage. A common method to do this is with SNMP. SNMP monitors many values on the FortiOS and allows you to set high water marks that will generate events. You run an application on your computer to watch for and record these events. Go to System > Config > SNMP to enable and configure an SNMP community. If this method is too complicated, you can use logging to record CPU usage every 5 minutes. However this method will not alert you to problems; it will just record them as they happen.
Proxy operation
Monitor proxy operations using the following command: diag test application <application> <option> The <application> value can include the following: ftpd ftp proxy http http proxy im im proxy imapi map proxy ipsengine ips sensor ipsmonitor ips monitor pop3 pop3 proxy smtp smtp proxy urlfilter urlfilter daemon The <option> value for use with this command can include: 1 Dump Memory Usage 2 Drop all connections 4 Display connection state * 44 Display info per connection * 444 Display connections per state * 5 Toggle AV Bypass mode 6 Toggle Print Stat mode every ~40 seconds 7 Toggle Backlog Drop 8 Clear stats 88 Toggle statistic recording - stats cleared
Troubleshooting for FortiOS 4.0 MR3 01-431-0129304-20120124 http://docs.fortinet.com/
38
Troubleshooting tools
FortiOS diagnostics
9 Toggle Accounting info for display 99 Restart proxy These commands, except for the ones identified with an *, should only be used under the guidance of Fortinet Support.
Hardware NIC
Monitor hardware network operations using the following command: diag hardware deviceinfo nic <interface> The information displayed by this command is important as errors at the interface are indicative of data link or physical layer issues which may impact the performance of the FortiGate unit. The following is sample output when <interface> = internal: System_Device_Name Current_HWaddr Permanent_HWaddr Link Speed Duplex [] Rx_Packets=5685708 Tx_Packets=4107073 Rx_Bytes=617908014 Tx_Bytes=1269751248 Rx_Errors=0 Tx_Errors=0 Rx_Dropped=0 Tx_Dropped=0 [..] port5 00:09:0f:68:35:60 00:09:0f:68:35:60 up 100 full
Hardware troubleshooting
The diag hardware deviceinfo nic command displays a list of hardware related error names and values. The following table explains the items in the list and their meanings. Table 3: Possible hardware errors and meanings Field Rx_Errors = rx error count Rx_CRC_Errors + Rx_Length_Errors Rx_Align_Errors Rx_Dropped or Rx_No_Buffer_Count Definition Bad frame was marked as error by PHY. This error is only valid in 10/100M mode.
39
FortiOS diagnostics
Troubleshooting tools
Table 3: Possible hardware errors and meanings Field Rx_Missed_Errors Definition Equals Rx_FIFO_Errors + CEXTERR (Carrier Extension Error Count). Only valid in 1000M mode, whichis marked by PHY. Tx_Errors = ECOL (Excessive Collisions Count). Only valid in half-duplex Tx_Aborted_Errors mode. Tx_Window_Errors LATECOL (Late Collisions Count). Late collisions are collisions that occur after 64-byte time into the transmission of the packet while working in 10 to100Mb/s data rate and 512-byte timeinto the transmission of the packet while working in the 1000Mb/s data rate. This register only increments if transmits are enabled and the device is in halfduplex mode. Rx_Dropped See Rx_Errors. Tx_Dropped Not defined. Collisions Total number of collisions experienced by the transmitter. Valid in half-duplex mode. Rx_Length_Errors Transmission length error. Rx_Over_Errors Not defined. Rx_CRC_Errors Frame CRC error. Rx_Frame_Errors Same as Rx_Align_Errors. This error is only valid in 10/100M mode. Rx_FIFO_Errors Same as Rx_Missed_Errors - a missed packet count. Tx_Aborted_Errors See Tx_Errors. Tx_Carrier_Errors The PHY should assert the internal carrier sense signal during every transmission. Failure to do so may indicate that the link has failed or the PHY has an incorrect link configuration. This register only increments if transmits are enabled. This register is not valid in internal SerDes 1 mode (TBI mode for the 82544GC/EI) and is only valid when the Ethernet controller is operating at full duplex. Tx_FIFO_Errors Not defined. Tx_Heartbeat_Errors Not defined. Tx_Window_Errors See LATECOL. Tx_Single_Collision_Fram Counts the number of times that a successfully transmitted es packed encountered a single collision. The value only increments if transmits are enabled and the Ethernet controller is in half-duplex mode. Tx_Multiple_Collision_Fra A Multiple Collision Count which counts the number of times mes that a transmit encountered more than one collision but less than 16. The value only increments if transmits are enabled and the Ethernet controller is in half-duplex mode. Tx_Deferred Counts defer events. A defer event occurs when the transmitter cannot immediately send a packet due to the medium being busy because another device is transmitting, the IPG timer has not expired, half-duplex deferral events are occurring, XOFF frames are being received, or the link is not up. This register only increments if transmits are enabled. This counter does not increment for streaming transmits that are deferred due to TX IPG. Rx_Frame_Too_Longs The Rx frame is over size.
40
Troubleshooting tools
FortiOS diagnostics
Table 3: Possible hardware errors and meanings Field Rx_Frame_Too_Shorts Rx_Align_Errors Symbol Error Count Definition The Rx frame is too short. This error is only valid in 10/100M mode. Counts the number of symbol errors between reads SYMERRS. The count increases for every bad symbol received, whether or not a packet is currently being received and whether or not the link is up. This register only increments in internal SerDes mode.
The counters displayed depend on the type of the NIC interface. Please see the following website for more information: http://kc.forticare.com/default.asp?id=1979&Lang=1&SID=
Conserve mode
The FortiOS antivirus and IPS systems operate in one of two modes, depending on the available shared memory. If the shared memory utilization is below a defined upper threshold, the system is in non-conserve mode. If shared memory usage goes above this threshold, the system enters conserve mode and remains in this state until the shared memory usage drops below a second threshold, slightly lower than the original. These thresholds are non-configurable; the threshold above which the system enters conserve mode is 80 percent, the system will not go back to non-conserve mode until the shared memory usage goes below 70 percent.
Antivirus failopen
Antivirus failopen is a safeguard feature that determines the behavior of the FortiGate unit antivirus system if it becomes overloaded in high traffic.
41
FortiOS diagnostics
Troubleshooting tools
The feature is configurable only though the CLI. config system global set av_failopen {off|one-shot|pass |idledrop} end av-failopen-session controls the behavior when the proxy connection pool is exhausted. Again in this case, the FortiGate unit does not send the SYN-ACK. Failopen is only available on FortiGate models 300A and higher. On other lower FortiGate models, the failopen action is configured to pass. The set av-failopen command has the following four options: off If the FortiGate unit enters conserve mode, the antivirus system will stop accepting new AV sessions but will continue to process current active sessions. one-shot If the FortiGate unit enters conserve mode, all subsequent connections bypass the antivirus system but current active sessions will continue to be processed. One-shot is similar to pass but will not automatically turn off once the condition causing avfailopen has stopped. idledrop When configured in this mode, the antivirus failopen mechanism will drop connections based on the clients that have the most open connections. pass Pass becomes the default setting when the av-failopen-session command has been run. If the system enters conserve mode, connections bypass the antivirus system until the system enters non-conserve mode again. Current active sessions will continue to be processed. The one-shot and pass options do not content filter traffic. Therefore, the data stream could contain malicious content.
Traffic trace
Traffic tracing allows a specific packet stream to be followed. This is useful to confirm packets are taking the route you expected on your network. View the characteristics of a traffic session though specific security policies using: diag sys session Trace per-packet operations for flow tracing using: diag debug flow Trace per-Ethernet frame using: diag sniffer packet
Session table
A session is a communication channel between two devices or applications across the network. Sessions enable FortiOS to inspect and act on a sequential group of packets in a session all together instead of inspecting each packet individually. Each of these sessions has an entry in the session table that includes important information about the session.
42
Troubleshooting tools
FortiOS diagnostics
Use as a tool
Session tables are useful troubleshooting tools because they allow you to verify connections that you expect to see open. For example, if you have a web browser open to browse the Fortinet website, you would expect a session entry from your computer, on port 80, to the IP for the Fortinet website. Another troubleshooting method is if there are too many sessions for FortiOS to process, you can examine the session table for evidence why this is happening. The FortiGate session table can be viewed from either the CLI or the web-based manager. The most useful troubleshooting data comes from the CLI. The session table in web-based manager also provides some useful summary information, particularly the current policy number that the session is using.
Session monitor
The session monitor is the session table. It lists the protocol used, source and destination addresses, source and destination ports, what policy ID was matched (if any), how long until the session expires, and how long it has been established. If there is no policy ID listed in the session entry, the traffic originated from the FortiGate unit. Otherwise all sessions must match a security policy to pass through the FortiGate unit. As there are potentially many sessions active at one time, there are different methods you can use to filter unimportant sessions out of your search. The easiest filter is to display only IPv4 or IPv6 sessions. By default both are displayed. For this option, you must have IPv6 displayed (to enable go to System > Admin > Settings).
43
FortiOS diagnostics
Troubleshooting tools
44
Troubleshooting tools
FortiOS diagnostics
session info: proto=6 proto_state=05 expire=89 timeout=3600 flags=00000000 av_idx=0 use=3 bandwidth=204800/sec guaranteed_bandwidth=102400/sec traffic=332/sec prio=0 logtype=session ha_id=0 hakey=4450 tunnel=/ state=log shape may_dirty statistic(bytes/packets/err): org=3408/38/0 reply=3888/31/0 tuples=2 orgin->sink: org pre->post, reply pre->post oif=3/5 gwy=192.168.11.254/10.0.5.100 hook=post dir=org act=snat 10.0.5.100:1251>192.168.11.254:22(192.168.11.105:1251) hook=pre dir=reply act=dnat 192.168.11.254:22>192.168.11.105:1251(10.0.5.100:1251) pos/(before,after) 0/(0,0), 0/(0,0) misc=0 domain_info=0 auth_info=0 ftgd_info=0 ids=0x0 vd=0 serial=00007c33 tos=ff/ff Since output can be verbose, the filter option allows specific information to be displayed, for example: diag sys session filter <option> The <option> values available include the following: clear dport dst negate policy proto sport src vd clear session filter destination port destination IP address inverse filter policy ID protocol number source port source IP address index of virtual domain. -1 matches all
Even though UDP is a sessionless protocol, the FortiGate unit still keeps track of the following two different states: UDP reply not seen with a value of 0 UDP reply seen with a value of 1 The following illustrates FW session states from the session table: State log local ext Meaning Session is being logged. Session is originated from or destined for local stack. Session is created by a firewall session helper.
45
FortiOS diagnostics
Troubleshooting tools
State may_dirty
Meaning Session is created by a policy. For example, the session for ftp control channel will have this state but ftp data channel will not. This is also seen when NAT is enabled. Session will be checked by IPS signature. Session will be checked by IPS anomaly. Session is being bridged (TP) mode.
ndr nds br
46
Troubleshooting tools
FortiOS diagnostics
CLI method
When running multiple VDOMs, this command is run in the Global configuration only and it searches for the named object both in the Global and VDOM configuration most recently used: diag sys checkused <path.object.mkey> For example, to verify which objects are referred to in a security policy with an ID of 1, enter the command as follows: diag sys checkused firewall.policy.policyid 1 To check what is referred to by interface port1, enter the following command: diag sys checkused system.interface.name port1 To show all the dependencies for an interface, enter the command as follows: diag sys checkused system.interface.name <interface name> Sample Output:
entry used by table firewall.address:name '10.98.23.23_host entry used by table firewall.address:name 'NAS' entry used by table firewall.address:name 'all' entry used by table firewall.address:name 'fortinet.com' entry used by table firewall.vip:name 'TORRENT_10.0.0.70:6883' entry used by table firewall.policy:policyid '21' entry used by table firewall.policy:policyid '14' entry used by table firewall.policy:policyid '19'
In this example, the interface has dependent objects, including four address objects, one VIP, and three security policies.
47
FortiOS diagnostics
Troubleshooting tools
2 Select the number in the Ref. column for the desired interface. A Window listing the dependencies will appear. 3 Use these detailed entries to locate and remove object references to this interface. The trash can icon will change from gray when all object dependencies have been removed. 4 Remove the interface by selecting the check box for the interface, and select Delete.
Flow trace
To trace the flow of packets through the FortiGate unit, use the following command: diag debug flow trace start Follow packet flow by setting a flow filter using this command: diag debug flow filter <option> Filtering options include the following: addr clear daddr dport negate port proto saddr sport vd IP address clear filter destination IP address destination port inverse filter port protocol number source IP address source port index of virtual domain, -1 matches all
Enable the output to be displayed to the CLI console using the following command: diag debug flow show console diag debug flow output is recorded as event log messages and are sent to a FortiAnalyzer unit if connected. Do not let this command run longer than necessary since it generates significant amounts of data. Start flow monitoring with a specific number of packets using this command: diag debug flow trace start <N> Stop flow tracing at any time using: diag debug flow trace stop The following is an example of the flow trace for the device at the following IP address: 203.160.224.97
48
Troubleshooting tools
FortiOS diagnostics
diag debug enable diag debug flow filter addr 203.160.224.97 diag debug flow show console enable diag debug flow show function-name enable diag debug flow trace start 100
49
FortiOS diagnostics
Troubleshooting tools
direction" Apply destination NAT to inverse source NAT action: id=20085 trace_id=210 func=__ip_session_run_tuple line=1516 msg="DNAT 192.168.11.59:31925>192.168.3.221:1487" Lookup for next-hop gateway address for reply traffic: id=20085 trace_id=210 func=vf_ip4_route_input line=1543 msg="find a route: gw-192.168.3.221 via port5" ACK received: id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2700 msg="vd-root received a packet(proto=6, 192.168.3.221:1487->203.160.224.97:80) from port5." Match existing session in the original direction: id=20085 trace_id=211 func=resolve_ip_tuple_fast line=2727 msg="Find an existing session, id-00000e90, original direction" Apply source NAT: id=20085 trace_id=211 func=__ip_session_run_tuple line=1502 msg="SNAT 192.168.3.221->192.168.11.59:31925" Receive data from client: id=20085 trace_id=212 func=resolve_ip_tuple_fast line=2700 msg="vd-root received a packet(proto=6, 192.168.3.221:1487->203.160.224.97:80) from port5." Match existing session in the original direction: id=20085 trace_id=212 func=resolve_ip_tuple_fast line=2727 msg="Find an existing session, id-00000e90, original direction" Apply source NAT: id=20085 trace_id=212 func=__ip_session_run_tuple line=1502 msg="SNAT 192.168.3.221->192.168.11.59:31925" Receive data from server: id=20085 trace_id=213 func=resolve_ip_tuple_fast line=2700 msg="vd-root received a packet(proto=6, 203.160.224.97:80->192.168.11.59:31925) from port6." Match existing session in reply direction: id=20085 trace_id=213 func=resolve_ip_tuple_fast line=2727 msg="Find an existing session, id-00000e90, reply direction"
50
Troubleshooting tools
FortiOS diagnostics
Apply destination NAT to inverse source NAT action: id=20085 trace_id=213 func=__ip_session_run_tuple line=1516 msg="DNAT 192.168.11.59:31925>192.168.3.221:1487"
51
FortiOS diagnostics
Troubleshooting tools
Features Record packet interface Configure complex sniffer filters on multiple interface sniff IPv6 sniff non-IP packets Filter packets by protocol and/or port Filter packets by source and/or destination address
Packet sniffing
Before you start sniffing packets on the CLI, you should be prepared to capture the output to a file there can be huge amounts of data that you will not be able to see without saving it to a file. One method is to use a terminal program like puTTY to connect to the FortiGate units CLI. Then once the packet sniffing count is reached you can end the session and analyze the output in the file. Details within packets passing through particular interfaces can be displayed using the packet sniffer with the following command: diag sniffer packet <interface> <filter> <verbose> <count> The <interface> value is required, with the rest being optional. If not included the default values will be none 4 0 . For example the simplest valid sniffer command would be: diag sniffer packet any The <interface> value can be any physical or virtual interface name. Use any to sniff packets on all interfaces. The <filter> value limits the display of packets using filters, including Berkeley Packet Filtering (BPF) syntax. The <filter> value must be enclosed in quotes. '[[src|dst] host <host_name_or_IP1>] [[src|dst] host <host_name_or_IP2>] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]] [[arp|ip|ip6|gre|esp|udp|tcp] [port_no]] If a second host is specified in the filter, only the traffic between the two hosts will be displayed. Optionally, you can use logical OR to match only one of the hosts, or match one of multiple protocols or ports. When defining a port, there are up to two parts protocol and port number. For example, to display UDP 1812 traffic or TCP 8080 traffic, use the following: 'udp port 1812 or tcp port 8080 To display all IP traffic that has a source of 192.168.1.2 and a destination of 192.168.2.3: 'ip src host 192.168.1.2 and dst host 192.168.2.3 The <verbose> option allows different levels of information to be displayed. The verbose levels include: 1 Print header of packets
52
Troubleshooting tools
FortiOS diagnostics
2 Print header and data from the IP header of the packets 3 Print header and data from the Ethernet header of the packets 4 Print header of packets with interface name 5 Print header and data from ip of packets with interface name 6 Print header and data from ethernet of packets with interface name The <count> value indicates the number of packets to sniff before stopping. If this variable is not included, or is set to zero, the sniffer will run until you manually halt it with Ctrl-C.
Packet capture
FortiOS 4.0 MR3 Patch 2 introduced packet capture to the web-based manager. To configure packet capture filters, go to System > Config > Advanced. When you add a packet capture filter, enter the following information and select OK. Select the interface to sniff from the dropdown menu. Interface You must select one interface. You cannot change the interface without deleting the filter and creating a new one, unlike the other fields. Enter the number of packets to capture before the filter stops. This number cannot be zero. You can halt the capturing before this number is reached. Enter the source address as an IP address and netmask. Source Address This field cannot be empty and it cannot be a hostname. To capture traffic from all source addresses enter 0.0.0.0/0.0.0.0. Enter one or more ports to capture on the source interface. Separate multiple ports with commas. Enter a range using a dash without spaces, for example 88-90. Enter the destination address as an IP address and netmask. Destination Address This field cannot be empty and it cannot be a hostname. To capture traffic from all destination addresses enter 0.0.0.0/0.0.0.0. Enter one or more ports to capture on the source interface. Separate multiple ports with commas. Enter a range using a dash without spaces, for example 88-90. Select a protocol to capture from the drop down list. Or select ALL from the list to capture all protcols.
Source Port(s)
Destination Port(s)
Protocol
53
FortiOS diagnostics
Troubleshooting tools
Select the interface to sniff from the dropdown menu. Interface You must select one interface. You cannot change the interface without deleting the filter and creating a new one, unlike the other fields. Select this option if you are troubleshooting IPv6 networking, or if your network uses IPv6. Otherwise, leave it disabled. The protocols available in the list are all IP based except for ICMP (ping). To capture non-IP based packets select this feature. Some examples of non-IP packets include IPsec, IGMP, ARP, and as mentioned ICMP.
If you select a filter and go back to edit it, you have the added option of starting and stopping packet capture in the edit window, or downloading the captured packets. You can also see the filter status and the number of packets captured. You can also select the filter and select Start to start capturing packets. While the filter is running, you will see the number of captured packets increasing until it reaches the max packet count or you select Stop. While the filter is running you cannot download the output file. When the packet capture is complete, you can select Download to send the packet capture filter captured packets to your local computer as a *.pcap file. To read this file format, you will need to use Wireshark or a similar third party application. Using this tool you will have extensive analytics available to you and the full contents of the packets that were captured.
54
Troubleshooting tools
FortiOS diagnostics
1 port7 1 port8 ID PORTS -- ----2 port9 2 port10 2 port11 2 port12 ID PORTS -- ----3 port13 3 port14 3 port15 3 port16
Run the following commands: diag npu np2 fastpath disable 0 (where 0 is the NP2 number) Then, run this command: diag npu np2 fastpath-sniffer enable port1 Sample output: NP2 Fast Path Sniffer on port1 enabled This will cause all traffic on port1 of NP2 0 to be sent to the CPU meaning a standard sniffer trace can be taken and other diag commands should work if it was a standard CPU driven port. These commands are only for the newer NP2 interfaces. FA2 interfaces are more limited as the sniffer will only capture the initial packets before the session is offloaded into HW (FA2). The same holds true for the diag debug flow command as only the session setup will be shown, however, this is usually enough for this command to be useful.
Debug command
Debug output provides continuous, real-time event information. Debugging output continues until it is explicitly stopped or until the unit is rebooted. Debugging output can affect system performance and will be continually generated even though output might not be displayed in the CLI console. Debug information displayed in the console will scroll in the console display and may prevent CLI commands from being entered, for example, the command to disable the debug display. To turn off debugging output as the display is scrolling by, press the key to recall the recent diag debug command, press backspace, and type 0, followed by Enter. Debug output display is enabled using the following command: diag debug enable When finished examining the debug output, disable it using: diag debug disable Once enabled, indicate the debug information that is required using this command:
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
55
FortiOS diagnostics
Troubleshooting tools
diag debug <option> <level> Debug command options include the following: application authd cli console crashlog disable enable flow info kernel rating report reset method of debugging output from many FortiGate daemons configure FSSO or clear authentication daemon configure cli debug level configure console settings for debugging get or clear the crash log info halt debug output start outputting filtered debug output trace packet flow in kernel show active debug level settings configure kernel and ha debug levels display website rating server list and information report for tech support reset all debug level to default
The debug level can be set at the end of the command. Typical values are 2 and 3, for example: diag debug application DHCPS 2 diag debug application spamfilter 2 Fortinet support will advise which debugging level to use. Timestamps can be enabled to the debug output using the following command: diag debug console timestamp enable
56
Troubleshooting tools
FortiOS diagnostics
FGh_FtiLog1: initiate an SA with selectors: 192.168.11.2/0.0.0.0->192.168.10.201, ports=0/0, protocol=0/0 Send IKE Packet(quick_outI1):192.168.11.2:500(if0) -> 192.168.10.201:500, len=348 Initiator: sent 192.168.10.201 quick mode message #1 (OK) FGh_FtiLog1: set retransmit: st=168, timeout=6. In this example: 192.168.11.2->192.168.10.201:500 dpd_fail=0 pfs=1536... Source and Destination gateway IP address Found existing Phase 1 Create new Phase 2 tunnel
57
FortiOS diagnostics
Troubleshooting tools
IP address
There may be times when you want to verify the IP addresses assigned to the FortiGate unit interfaces are what you expect them to be. This is easily accomplished from the CLI using the following command. diag ip address list The output from this command lists the IP address and mask if available, the index of the interface (a sort of ID number) and the devname is the name of the interface. While physical interface names are set, virtual interface names can vary. Listing all the virtual interface names is a good use of this command. For vsys_ha and vsys_fgfm, the IP addresses are the local host these are internally used virtual interfaces. # diag ip address list IP=10.31.101.100->10.31.101.100/255.255.255.0 index=3 devname=internal IP=172.20.120.122->172.20.120.122/255.255.255.0 index=5 devname=wan1 IP=127.0.0.1->127.0.0.1/255.0.0.0 index=8 devname=root IP=127.0.0.1->127.0.0.1/255.0.0.0 index=11 devname=vsys_ha IP=127.0.0.1->127.0.0.1/255.0.0.0 index=13 devname=vsys_fgfm Other related commands include flushing the IP addresses (diag ip address flush), which will force a reload of the IP addresses. This can be useful if you think an IP address is wrong and dont want to reboot the unit. You can add or delete a single IP address (diag ip address add <ipv4_addr> or diag ip address delete <ipv4_addr>). For more information on useful diagnostic commands, see Troubleshooting get commands on page 119and Troubleshooting common issues on page 77.
58
Troubleshooting tools
FortiGate ports
FortiGate ports
In the TCP and UDP stacks, there are 65 535 ports available for applications to use when communicating with each other. Many of these ports are commonly known to be associated with specific applications or protocols. These known ports can be useful when troubleshooting your network. Use the following ports while troubleshooting the FortiGate device: Port(s) UDP 53 UDP 53 or UDP 8888 UDP 53 (default) or UDP 8888 and UDP 1027 or UDP 1031 UDP 123 UDP 162 UDP 514 Functionality DNS lookup, RBL lookup FortiGuard Antispam or Web Filtering rating lookup FDN Server List - source and destination port numbers vary by originating or reply traffic. See the article How do I troubleshoot performance issues when FortiGuard Web Filtering is enabled? in the Knowledge Base.
NTP Synchronization SNMP Traps SYSLOG - All FortiOS versions can use syslog to send log messages to remote syslog servers. FortiOS v2.80 and v3.0 can also view logs stored remotely on a FortiAnalyzer unit. Configuration backup to FortiManager unit or FortiGuard Analysis and Management Service. SMTP alert email, encrypted virus sample auto-submit LDAP or PKI authentication FortiGuard Antivirus or IPS update - When requesting updates from a FortiManager unit instead of directly from the FDN, this port must be reconfigured as TCP 8890. FortiGuard Analysis and Management Service FortiGuard Analysis and Management Service log transmission (OFTP) SSL Management Tunnel to FortiGuard Analysis and Management Service (FortiOS v3.0 MR6 or later) Quarantine, remote access to logs and reports on a FortiAnalyzer unit, device registration with FortiAnalyzer units (OFTP) RADIUS authentication
TCP 1812
TCP 8000 and FSSO TCP 8002 TCP 10151 FortiGuard Analysis and Management Service contract validation
59
Diagnostic commands
Troubleshooting tools
Diagnostic commands
FortiAnalyzer/FortiManager ports
If you have a FortiAnalyzer unit or FortiManager unit on your network you may need to use the following ports for troubleshooting network traffic. Functionality DNS lookup NTP synchronization Windows share SNMP traps Syslog, log forwarding Log and report upload SMTP alert email User name LDAP queries for reports RVS update RADIUS authentication Log aggregation client Port(s) UDP 53 UDP 123 UDP 137-138 UDP 162 UDP 514 TCP 21 or TCP 22 TCP 25 TCP 389 or TCP 636 TCP 443 TCP 1812 TCP 3000
For more information about FortiAnalyzer/FortiManager ports, see the Fortinet Knowledge Base at the following address: http://kc.forticare.com/default.asp?SID=&Lang=1&id=773.
FortiGuard troubleshooting
The FortiGuard service provides updates to Antivirus, IPSec, Webfiltering, and more. The FortiGuard Distribution System (FDS) involves a number of servers across the world that provide updates to your FortiGate unit. Problems can occur both with connection to FDS, and its configuration on your local FortiGate unit. Some of the more common troubleshooting methods are listed here including Troubleshooting process for FortiGuard updates FortiGuard server settings FortiGuard URL rating
60
Troubleshooting tools
FortiGuard troubleshooting
2 If the device is part of an HA cluster, do all members of the cluster have the same level of support? As with the previous step, you can verify the support contract status for all the devices in your HA cluster at the Fortinet Support website. 3 Have services been enabled on the device? To see the FortiGuard information and status for a device, in the web-based manager go to System > Config > FortiGuard. On that page you can verify the status of each component, and if required enable each service. If there are problems, see the FortiGuard section of the FortiOS Handbook. 4 Is the device able to communicate with FortiGuard servers? At System > Config > FortiGuard you can also attempt to update AV and IPS, or test the availability of WF and AS default and alternate ports. If there are problems, see the FortiGuard section of the FortiOS Handbook. 5 Is there proper routing to reach the FortiGuard servers? Ensure there is a static or dynamic route that enables your ForitGate unit to reach the FortiGuard servers. Usually a generic default route to the internet is enough, but you may need to verify this if your network is complex. 6 Are there issues with DNS? An easy way to test this is to attempt a traceroute from behind the FortiGate unit to an external network using the FQDN for a location. If the traceroute FQDN name does not resolve, you have general DNS problems. See DNS settings on page 85. 7 Is there anything upstream that might be blocking FortiGuard traffic, either on the network or ISP side? Many firewalls block all ports by default, and often ISPs block ports that are low. There may be a firewall between the FortiGate unit and the FortiGuard servers that is blocking the traffic. FortiGuard uses port 53 by default, so if it is being blocked you need to either open a hole for it, or change the port it is using. 8 Is there an issue with source ports? It is possible that ports used to contact FortiGuard are being changed before reaching FortiGuard or on the return trip before reaching your FortiGate unit. A possible solution for this is to use a fixed-port at NATd firewalls to ensure the port remains the same. Packet sniffing can be used to find more information on what is happening with ports. See Perform a sniffer trace on page 94. 9 Are there security policies that include antivirus? If no security policies include antivirus, the antivirus databse will not be updated. If antivirus is included, only the database type used will be updated.
61
FortiGuard troubleshooting
Troubleshooting tools
-=- Server List (Mon Feb 18 12:55:48 2008) -=IP a.b.c.d 10.1.101.1 10.2.102.2 10.3.103.3 10.4.104.4 10.5.105.5 Weight 0 10 20 20 20 25 RTT Flags DI TZ 2 1 0 0 0 0 Packets 1926879 10263 16105 6741 5249 12072 CurrLost TotalLost 0 11176 0 633 0 80 0 776 0 987 0 178
Output Details Hostname is the name of the FortiGuard server the FortiGate unit will attempt to contact. The Server List includes the IP addresses of alternate servers if the first entry cannot be reached. In this example the IP addresses are not public addresses The following flags in get webfilter status indicate the server status: D - the server was found through the DNS lookup of the hostname. If the hostname returns more than one IP address, all of them will be flagged with D and will be used first for INIT requests before falling back to the other servers. I - the server to which the last INIT request was sent. F - the server has not responded to requests and is considered to have failed. T - the server is currently being timed.
62
Troubleshooting tools
FortiGuard troubleshooting
Calculating weight
The weight for each server increases with failed packets and decreases with successful packets. To lower the possibility of using a remote server, the weight is not allowed to dip below a base weight, calculated as the difference in hours between the FortiGate unit and the server times 10. The further away the server is, the higher its base weight and the lower in the list it will appear. The output for the diag debug rating command will vary based on the state of the FortiGate device. The following output is from a FortiGate device that has no DNS resolution for service.fortiguard.net.
If only three IP addresses appear with the D flag, it means that DNS is good but probably the FortiGuard ports 53 and 8888 are blocked. When the license is expired, an INIT request will be sent every 10 minutes for up to six attempts. If a license is not found after this limit is reached, the INIT requests will be sent every day. A low source port number may appear which means that ports 1024 and 1025 could be blocked on the path to the FDS. Increase the source port on the FortiGate device with the following commands: config sys global set ip-src-port-range <start-end> (Default 1024-25000) Be careful moving ports like this as it may cause some services to stop working if they cant access their original ports. If you make this change, ensure all services that use ports are checked and updated to new port numbers if needed.
63
FortiGuard troubleshooting
Troubleshooting tools
id=93000 msg="pid=57 urlfilter_main-723 in main.c received pkt:count=91, a=/tmp/.thttp.socket/21" id=22009 msg="received a request /tmp/.thttp.socket, addr_len=21: d= ="www.goodorg.org:80, id=12853, vfid=0, type=0, client=192.168.3.90, url=/" id=99501 user="N/A" src=192.168.3.90 sport=1321 dst=<dest_ip> dport=80 service="http" cat=43 cat_desc=Organisation" hostname="www.goodorg.org" url="/" status=blocked msg="URL belongs to a denied category in policy" Sample output: id=22009 msg="received a request /tmp/.thttp.socket, addr_len=21: d=pt.dnstest.google.com:80, id=300, vfid=0, type=0, client=192.168.3.12, url=/gen_204" id=93003 user="N/A" src=192.168.3.12 sport=21715 dst=<dest_ip> dport=80 service="http" cat=41 cat_desc="Search Engines" hostname="pt.dnstest.google.com" url="/gen_204" status=passthrough msg="URL belongs to an allowed category in the policy"
Table 4: Breakdown of sample output parts from URL rating command
id=93000 msg="pid=57 urlfilter_main-723 in main.c received pkt:count=91, a=/tmp/.thttp.socket/21" id=22009 msg="received a request /tmp/.thttp.socket, addr_len=21: d= ="www.goodorg.org:80, id=12853, vfid=0, type=0, client=192.168.3.90, url=/" id=99501 user="N/A" src=192.168.3.90 sport=1321 dst=<dest_ip> dport=80 service="http" cat=43 cat_desc=Organisation" hostname="www.goodorg.org" url="/" status=blocked msg="URL belongs to a denied category in policy"
The process ID (PID) is listed along with the function in the file running (main.c). Then it lists the number of packets received and the associated socket where the packets came from. Received a request on a particular socket (/tmp/.thttp.socket). The website to be rated is www.goodorg.org:80 and the client browser that wants the verification is 192.169.3.90.
No user associated with this source address (192.168.3.90) and port (1321). The destination IP is unknown and the port is the standard HTTP port 80, which is confirmed by service=http. The cat keyword gives the category of the URL being checked, which turns out to be an organization. This is confirmed by the hostname of goodorg.org. The status is stated as blocked with the reason stated as URL belongs to a denied category in policy.
64
FortiOS Handbook
65
Creating an account
AMER
Regional TAC Focused Teams Technical Support RMA Customer Services Remote Access Labs
Creating an account
To receive technical support and service updates, Fortinet products in the organization must be registered. The Product Registration Form on the support website will allow the registration to be completed online. Creating an account on the support website is the first step in registering products
66
Registering a device
Once the account has been created, the Product Registration Form will be displayed and the product details can be provided. Alternately, the product registration can be completed at a later time.
Registering a device
Complete the following steps when registering a device for support purposes: 1 Log in using the Username and Password defined when the account was created
67
Registering a device
2 Select Add Registration on the left-hand side. 3 Select New Fortinet Product/License Registration. Figure 11: Adding a product to a support account
68
Reporting problems
6 Enter the Support Contract No. provided by Fortinet when the support contract was purchased. 7 In the Product Description field, explain where this unit is physically located. 8 Click Next and accept the End User License Agreement (EULA) to complete the registration.
Reporting problems
Problems can be reported to a Fortinet Technical Assistance Center in the following ways: By logging an online ticket By phoning a technical support center
Fortinet partners
Fortinet Partners are entitled to priority web-based technical support. This service is designed for partners who provide initial support to their customers and who need to open a support ticket with Fortinet on their behalf. We strongly encourage submission and follow up of support tickets using this service. The support ticket can be submitted after logging into the partner website using one of the following links using FortiPartner account details: http://partners.fortinet.com This link will redirect to the general Partner Extranet website. Click Support > Online Support Ticket. https://www.forticare.com:1443/customersupport/login/partnerlog in.aspx This link redirects to the Partner Online Support Ticket section also known as FortiCare. The Partner Online Support Ticket section is accessed through HTTPS on port 1443. Ensure that the firewall allows external access this port. Also, a customers Fortinet device must have a valid support contract to be able submit the support request as a Partner.
Fortinet customers
Fortinet customers should complete the following steps to report a technical problem online: 1 Log in to the support web site at the following address with the account credentials used when the account was created: https://support.fortinet.com 2 Click View Products. 3 In the Products List, select the product that is causing the problem. 4 Complete the Create Support Ticket fields.
69
Reporting problems
70
Reporting problems
3 Select the appropriate ticket number. Closed tickets cannot be updated. A new ticket must be submitted if it concerns the same problem. 4 Add a New Comment or Attachment. 5 Click Submit when complete. Every web ticket update triggers a notification to the ticket owner, or ticket queue supervisor.
Americas
Telephone: 1-866-648-4638 Hours: Monday to Friday 6:00 AM to 6:00 PM (Pacific Daylight Time)
71
EMEA
Telephone: +33-4-898-0555 If a support call is placed outside of EMEA business hours (Monday to Friday 9:00 AM to 6:00 PM (Central European Daylight Time)), priority 1 issues will be transferred to another Fortinet Technical Solutions center according to the Follow the Sun policy, meaning wherever its daylight, that TAC will be taking all support calls. Hours: Monday to Friday 9:00 AM to 6:00 PM (Central European Daylight Time)
APAC
Telephone: +603-2711-7391 Hours: Monday to Friday 9:00 AM to 6:00 PM (Malaysia Time)
Priority 1
This Critical priority is assigned to support cases in which: The network or system is down causing customers to experience a total loss of service. There are continuous or frequent instabilities affecting traffic-handling capability on a significant portion of the network. There is a loss of connectivity or isolation to a significant portion of the network. This issue has created a hazard or an emergency.
Priority 2
This Major priority is assigned to support cases in which:
Troubleshooting for FortiOS 4.0 MR3 01-431-0129304-20120124 http://docs.fortinet.com/
72
The network or system event is causing intermittent impact to end customers. There is a loss of redundancy. There is a loss of routine administrative or diagnostic capability. There is an inability to deploy a key feature or function. There is a partial loss of service due to a failed hardware component.
Priority 3
This Medium priority is assigned to support cases in which: The network event is causing only limited impact to end customers. Issues seen in a test or pre-production environment exist that would normally cause adverse impact to a production network. The customer is making time sensitive information requests. There is a successful workaround in place for a higher priority issue.
Priority 4
This Minor priority is assigned to support cases in which: The customer is making information requests and asking standard questions about the configuration or functionality of equipment. Customers must report Priority 1 and 2 issues by phone directly to the Fortinet EMEA Support Center. For lower priority issues, you may submit an assistance request (ticket) via the web system. The web ticket system also provides a global overview of all ongoing support requests.
73
The Product Support Details for the selected device will be displayed.
74
75
3 In the RMA Replacement section of the Product Support Details page, enter the serial number of the replacement device and click RMA Replace. Figure 16: RMA Replacement section of Product Support Details
This will transfer the support contract from the defective unit to the new unit with the serial number provided.
76
FortiOS Handbook
77
4 Ensure you are using the proper cable type. There are different types of ethernet network cable with the main difference being the wiring is it a crossover or patch cable? However there are even different accepted cross-over wiring orders, so its possible that with two cross over cables, one may not work. Also if the cable is hand-made instead of purchased, there are multiple places for problems cable rating (Cat5, Cat5e, Cat6), bad connections in the ends, cable too long, or even breaks in the cable wiring. 5 Try using a different cable. When all else fails, try a different cable. It is possible there are breaks in the wires, or the cable was wired differently than you expect. Changing to a new cable will quickly let you know if the cable was bad. 6 Check the interfaces are connected. Use the CLI command get hardware nic <interface_name> to check the interface with the connectivity problems. For example checking wan1 would be get hardware nic wan1. In the output of this command look for a line that states Link: up. If the Link is down, it is a physical connection problem. If the Link is up, it is an internal problem. 7 Try connecting to a different network device. It is possible there is a problem with the configuration of the device you are trying to connect to. If it has a dead interface for example, even if everything else is working you will not see the LED connection lights on the FortiGate unit. If you have tried everything else, try connecting to a different network device.
78
3 Check the default route. For network traffic to find its way across the network, it must have a route. Without a route, the traffic doesnt know where to go. Ensure there is a valid static route on your FortiGate unit that will direct outbound traffic to the Internet. Ensure the proper interface is used, and that the gateway used does exist. Also ensure the default route admin distance is lower than other routes you may have configured. 4 Check DHCP settings. If you are using DHCP for any computers trying to connect to the Internet, the settings may have problems. If your computers are having IP address or DNS problems, its possibly DHCP server related. Things to check include the range of addresses, the DHCP server is assigned to the correct interface, and any overrides are assigned to the proper IP and MAC address. It is also possible there is an interface waiting for a DHCP server that isnt there. 5 Check the interface settings. Typically the FortiGate unit interface connected to the Internet is Wan1 or Wan2. Ensure that interface has an IP address and subnet mask, that the IP address exists on the subnet the interface is connected to, and check that both the admin and link status are up. If everything else is working but the admin status is down, the interface will not work. If your FortiGate unit interfaces are labelled port1, port2, and so on consider setting alias names for the interfaces based on their function. This will save you time when troubleshooting. 6 Check the firewall settings for the interfaces. For traffic to flow between two FortiGate unit interfaces, there must be a security policy to allow that traffic. Check there is a policy to allow traffic between your two interfaces. Also check that the policy is not restricting traffic to a specific time of day, specific users, or such. If those restrictions exist, and you remove them for testing remember to put them back in place. 7 Check the firewall settings for basic networking protocols. It is common practice to have a DENY security policy at the top of the list that will block unwanted protocols. If this type of policy exists on your ForitGate unit, consider disabling it to check if it is preventing access to the Internet. If it is the problem, go through the list of blocked protocols and remove any common internet protocols needed for common traffic such as FTP, HTTP, HTTPS, PING, DNS, and so on. 8 Check the logs. If you cannot find the source of a problem, before you contact support you should always check your logs. The easiest log to start with is the event log. This log records major events that happen on the FortiGate unit such as configuration changes, failed logins, FortiGuard and firmware updates, and critical problems. These log messages may help you identify the problem, and give a place to start trying to fix it. Other logs such as UTM, traffic, and vulnerability logs may prove useful but they are more specific and complex. If you do not have logging configured, go to Log&Report > Log Config > Log Setting. Select Memory and a minimum log level of Information. Select Event Logging, and Enable All. This will start logging events on your FortiGate unit and give you a valuable troubleshooting tool.
79
9 Perform a packet capture on the traffic. If everything else looks properly configured, try capturing packets from the traffic to get a better idea what is happening at the low level. To enable packet capturing on the web-based interface go to System > Config > Advanced. Select Create New to create a packet capture filter by selecting the interface, protocol, and source and destination address and port. When you run the capture filter it will capture packets that match, and save them to a *.pcap file on your local computer, named for the interface. To read this file, you must use a third-party application such as Wireshark. This will show you the packet source, destination, protocol, and other information that may help you solve your connection issues. 10 Contact support. If you have gone through all these steps, and you have not determined why you have no Internet connection it is time to contact support. Remember to tell support everything you have tried up to this point to ensure they do not duplicate work you have already done.
80
81
4 Three things to check if you are running HA. Ensure Bridge Protocol Data Units (BPDUs) are forwarded between HA units. BPDU messages are exchanged across bridges to detect loops in a network topology, but HA units sharing the same MAC address can cause problems with BPDUs. This is not an issue if HA units are directly connected. Configure multiple redundant interfaces to the switch when operating in activepassive HA mode. In an HA cluster, virtual MAC addresses are used so if there is a failover, the new primary unit will have the same virtual MAC and IP address as the primary unit that failed. The problem is that switches usually see this as the same MAC on multiple interfaces on the same subnet, which is usually associated with a hacker masquerading on the network. Form the HA cluster carefully. As the cluster is being configured, the network interfaces will momentarily lose connectivity as the FGCP assigns virtual MACs to the interfaces. If this is the case, try deleting the arp table on your management computer using the command arp -d. If the priorities of the subordinate units are not lower than the primary unit, the cluster may take longer to determine the primary unit and lost connectivity for a longer period as a result. Also if you power on a secondary unit first, it will become the primary, and then when you power on the intended primary there will be a switchover. This should be avoided. 5 Capture packets for in-depth information. If you have checked the bridge and session tables without finding the problem, the next step is to capture some packets. Captured packets to and from an IP address can tell you exactly what is going on at a lower level than most other troubleshooting methods. To enable packet capturing on the FortiGate units web-based interface, go to System > Config > Advanced. Select Create New to create a packet capture filter by selecting the interface, protocol, and source and destination address and port. When you run the capture filter it will capture packets that match the filter, and save them to a *.pcap file on your local computer, named for the interface. To read this file, you must use a third-party application such as Wireshark. This will show you the packet source, destination, protocol, and other information that may help you solve your connection issues.
82
4 Check modem status on page 87 Is the modem connected? Are there PPP issues? 5 6 Run ping and traceroute on page 87 Are you experiencing complete packet loss? 7 Check the logs on page 91 8 Verify the contents of the routing table (in NAT mode) on page 92 Are there routes in the routing table for default and static routes? Do all connected subnets have a route in the routing table? Does a route wrongly have a higher priority than it should? 9 Check the bridging information in Transparent mode on page 92 Are you having problems in transparent mode? 10 Perform a sniffer trace on page 94 Is traffic entering the FortiGate unit and does it arrive on the expected interface? Is the ARP resolution correct for the next-hop destination? Is the traffic exiting the FortiGate unit to the destination as expected? Is the traffic being sent back to the originator? 11 Debug the packet flow on page 96 Is the traffic entering or leaving the FortiGate unit as expected? 12 Check number of sessions used by UTM proxy on page 97 Have you reached the maximum number of sessions for a protocol? Are new sessions not starting for a certain protocol? 13 Examine the firewall session list on page 101 Are there active firewall sessions? 14 Checking wireless information on page 102 For troubleshooting tips for specific non-connectivity areas, see Troubleshooting advanced on page 103. In addition to these steps, you may find other diagnose commands useful. See Other diagnose commands on page 102.
83
Interface settings
If you can access the FortiGate unit with the management cable only, the first step is to display the interface settings. To display the settings for the internal interface, use the following CLI command: FGT# show system interface internal or for a complete listing of all the possible interface settings, use the following CLI command: config system interface edit internal get
84
end Check the interface settings to ensure they are not preventing traffic. Specific things to check include (web-based manager names are shown, CLI names may vary slightly): Link Status Down until a valid cable is plugged into this interface, after which it will be Up. The Link Status is shown physically by the connection LED for the interface if it lights up green, it is a good connection. If Link Status is down, the interface does not work. Link Status is also displayed on the System > Network > Interface screen by default. Addressing mode do not use DHCP if you dont have a DHCP server you will not be able to logon to an interface in DHCP mode as it will not have an IP address. IP/Netmask an interface needs an IP address to be able to connect to other devices. Ensure there is a valid IP address in this field. The one exception is if DHCP is enabled for this interface to get its IP address from an external DHCP server. IPv6 address unless specifically stated all IP addresses are IPv4. The same protocol must be used by both ends to complete the connection. Ensure both this interface and the remote connection are both using IPv4 or both using IPv6 addressing. Administrative access If no protocols are selected, you will have to use the local management cable to connect to the unit. If you are using IPv6, configure the IPv6 administrative access protocols. Administrative status set to Up or interface will not work.
DNS settings
While this section is not complicated, many networking problems can be traced back to DNS problems. Things to check in this area include: Are there values for both primary and secondary entries? Is the local domain name correct? Are you using IPv6 addressing? If so, are the IPv6 DNS settings correct? Are you using Dynamic DNS (DDNS)? If so, it is using the correct server, credentials, and interface? Can you contact both DNS servers to verify the servers are operational? If an interface addressing mode is set to DHCP and is set to override the internal DNS, is that interface receiving a valid DNS entry from the DHCP server is it a reasonable address and can it be contacted to verify its operational? Are there any DENY security policies that need to allow DNS? Can any internal device perform a successful traceroute to a location using the FQDN? See Traceroute on page 89.
85
Is the range of IP addresses this DHCP server uses valid? Are those addresses in use by other devices? If one or more devices are using IP addresses in this range, you can use the IP reservation feature to ensure the DHCP server does not use these addresses. Is there a gateway entry? Include a gateway entry to ensure clients of this server have a default route. Is the system DNS setting being used? The best practice is to whenever possible use the system DNS to avoid confusion. However, the option to specify up to three custom DNS servers is available, and all three entries should be used for redundancy. There are some situations, such as a new wireless interface, or during the initial FortiGate unit configuration, where interfaces override the system DNS entries. When this happens, it often shows up as intermittent Internet connectivity. To fix the problem, go to the problem interface and ensure its set to use the system DNS entries.
86
The other lines of output such as network usage, average session setup rate, and viruses caught, IPS attacks blocked can also help you determine why system resource usage it high. For example, if network usage is high it will result in high traffic processing on the FortiGate, or if the session setup rate is very low or zero the proxy may be overloaded and not able to do its job.
87
Ping
The ping command sends a very small packet to the destination, and waits for a response. The response has a timer that may expire, indicating the destination is unreachable. The behavior of ping is very much like a sonar ping from a submarine, where the command gets its name. Ping is part of Layer-3 on the OSI Networking Model. Ping sends Internet Control Message Protocol (ICMP) echo request packets to the destination, and listens for echo response packets in reply. However, many public networks block ICMP packets because ping can be used in a denial of service (DoS) attack (such as Ping of Death or a smurf attack), or by an attacker to find active locations on the network. By default, FortiGate units have ping enabled and broadcast-forward is disabled on the external interface.
ms ms ms ms ms
88
round-trip min/avg/max = 0.2/0.2/0.3 ms To ping from an MS Windows PC 1 Open a command window. In Windows XP, select Start > Run, enter cmd, and select OK. In Windows 7, select the Start icon, enter cmd in the search box,and select cmd.exe from the list. 2 Enter ping 10.11.101.100 to ping the default internal interface of the FortiGate unit with four packets. Other options include: -t to send packets until you press Control-C -a to resolve addresses to domain names where possible -n X to send X ping packets and stop Output appears as: C:\>ping 10.11.101.101 Pinging 10.11.101.101 with 32 bytes of data: Reply from 10.11.101.101: bytes=32 time=10ms TTL=255 Reply from 10.11.101.101: bytes=32 time<1ms TTL=255 Reply from 10.11.101.101: bytes=32 time=1ms TTL=255 Reply from 10.11.101.101: bytes=32 time=1ms TTL=255 Ping statistics for 10.11.101.101: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 10ms, Average = 3ms To ping from a Linux PC 1 Go to a shell prompt. 2 Enter /bin/etc/ping 10.11.101.101.
Traceroute
Where ping will only tell you if it reached its destination and came back successfully, traceroute will show each step of its journey to its destination and how long each step takes. If ping finds an outage between two points, traceroute can be used to locate exactly where the problem is.
What is traceroute
Traceroute works by sending ICMP packets to test each hop along the route. It will send out three packets, and then increase the time to live (TTL) setting by one each time. This effectively allows the packets to go one hop farther along the route. This is the reason why most traceroute commands display their maximum hop count before they start tracing the route that is the maximum number of steps it will take before declaring the destination unreachable. Also the TTL setting may result in steps along the route timing out due to slow responses. There are many possible reasons for this to occur.
89
Traceroute by default uses UDP datagrams with destination ports numbered from 33434 to 33534. The traceroute utility usually has an option to specify use of ICMP echo request (type 8) instead, as used by the Windows tracert utility. If you have a firewall and if you want traceroute to work from both machines (Unix-like systems and Windows) you will need to allow both protocols inbound through your FortiGate security policies (UDP with ports from 33434 to 33534 and ICMP type 8). You can also use the packet count column of the Policy > Policy > Policy page to track traceroute packets. This allows you to verify the connection, but also confirm which security policy the traceroute packets are using.
90
9 53 ms 58 ms [144.232.19.181] 10 82 ms 90 ms [144.232.20.61] 11 122 ms 123 ms [144.232.18.150] 12 129 ms 119 ms 13 172 ms 164 ms [144.223.243.58] 14 99 ms 94 ms 15 108 ms 102 ms 16 98 ms 95 ms Trace complete.
The first, or the left column, is the hop count, which cannot go over 30 hops. When that number is reached, the traceroute ends. The second, third, and fourth columns display how much time each of the three packets takes to reach this stage of the route. These values are in milliseconds and normally vary quite a bit. Typically a value of <1ms indicates a local connection. The fifth, or the column farthest to the right, is the domain name of that device and its IP address or possibly just the IP address. To perform a traceroute on a Linux PC 1 Go to a command line prompt. 2 Enter /bin/etc/traceroute fortinet.com. The Linux traceroute output is very similar to the MS Windows tracert output.
91
filter or order log entries based on different fields (such as level, service, or IP address) to look for patterns that may indicate a specific problem (such as frequent blocked connections on a specific port for all IP addresses) use log reporting to help you visualize large amounts of log data go to Log&Report > Report Access > Cover Page, select Option, and configure the date, time, and contents of the log report. You also have the option to email generated log reports to an administrator. Logs will help identify and locate any problems, but they will not solve the problems. The job of logs is to speed up your problem solving and save you time and effort. For more information on Logging and Log Reports, see the Logging and Reporting guide.
92
Sample Output:
#diagnose netlink brctl list list bridge information 1. root.b fdb: size=256 Total 1 bridges used=6 num=7 depth=2 simple=no
Sample Output
diagnose netlink brctl domain ione 101 show bridge root.b ione forward domain. id=101 dev=trunk_1 6 To list the existing bridge MAC table, use the following command: diagnose netlink brctl name host <name>
Sample Output
show bridge control interface root.b host. fdb: size=256, used=6, num=7, depth=2, simple=no Bridge root.b host table
port no 2 5 3 4 3 device 7 6 8 9 8 devname wan2 vlan_1 dmz internal dmz mac addr 02:09:0f:78:69:00 02:09:0f:78:69:01 02:09:0f:78:69:01 02:09:0f:78:69:02 00:80:c8:39:87:5a ttl 0 0 0 0 194 attributes Local Static Local Static Local Static Local Static
93
4 1
9 3
internal wan1
02:09:0f:78:67:68 00:09:0f:78:69:fe
8 0
Local Static
To list the existing bridge port list, use this command: diagnose netlink brctl name port <name> Sample Output: show bridge root.b data port. trunk_1 peer_dev=0 internal peer_dev=0 dmz peer_dev=0 wan2 peer_dev=0 wan1 peer_dev=0
94
<interface_name>
The name of the interface to sniff, such as port1 or internal. This can also be any to sniff all interfaces. What to look for in the information the sniffer reads. none indicates no filtering, and all packets will be displayed as the other arguments indicate. The filter must be inside single quotes (). The level of verbosity as one of:
1 - print header of packets 2 - print header and data from IP of packets 3 - print header and data from Ethernet of packets
<filter>
<verbose>
<count>
The number of packets the sniffer reads before stopping. If you do not put a number here, the sniffer will run forever unit you stop it with <CTRL C>.
For a simple sniffing example, enter the CLI command diag sniffer packet port1 none 1 3. This will display the next three packets on the port1 interface using no filtering, and using verbose level 1. At this verbosity level you can see the source IP and port, the destination IP and port, action (such as ack), and sequence numbers. In the output below, port 443 indicates these are HTTPS packets, and 172.20.120.17 is both sending and receiving traffic. Head_Office_620b # diag sniffer packet port1 none 1 3 interfaces=[port1] filters=[none] 0.545306 172.20.120.17.52989 -> 172.20.120.141.443: psh 3177924955 ack 1854307757 0.545963 172.20.120.141.443 -> 172.20.120.17.52989: psh 1854307757 ack 3177925808 0.562409 172.20.120.17.52988 -> 172.20.120.141.443: psh 4225311614 ack 3314279933 For a more advanced example of packet sniffing, the following commands will report packets on any interface travelling between a computer with the host name of PC1 and the computer with the host name of PC2. With verbosity 4 and above, the sniffer trace will display the interface names where traffic enters or leaves the FortiGate unit. Remember to stop the sniffer, type CTRL+C. FGT# diagnose sniffer packet any "host <PC1> or host <PC2>" 4 or FGT# diagnose sniffer packet any "(host <PC1> or host <PC2>) and icmp" 4 The following sniffer CLI command includes the ARP protocol in the filter which may be useful to troubleshoot a failure in the ARP resolution (for instance PC2 may be down and not responding to the FortiGate ARP requests).
95
FGT# diagnose sniffer packet any "host <PC1> or host <PC2> or arp" 4
96
97
If the proxy session table is full for one or more protocols and your FortiGate unit enters into conserve or failopen mode, it will appear as if you have lost connections, network services are intermittent or non-existent, and yet other services work normally for a while until their sessions end and they join the queue of session starved applications.
8/8032 76
98
Error Count (poll) Error Count (scan reset) Error Count (urlfilter wait) Last Error Web responses clean Web responses scan errors Web responses detected Web responses infected with worms Web responses infected with viruses Web responses infected with susp Web responses file blocked Web responses file exempt Web responses bannedword detected Web requests oversize pass Web requests oversize block Last Server Scan errors URL requests exempt URL requests blocked URL requests passed URL requests submit error URL requests rating error URL requests rating block URL requests rating allow URL requests infected with worms Web requests detected Web requests file blocked Web requests file exempt POST requests clean POST requests scan errors POST requests infected with viruses POST requests infected with susp POST requests file blocked POST requests bannedword detected POST requests oversize pass POST requests oversize block Web request backlog drop Web response backlog drop
Worker Accounting poll=721392/649809/42 pollfail=0 cmdb=85 scan=19266 acceptor=25975 HTTP Accounting setup_ok=8316 setup_fail=0 conn_ok=0 conn_inp=8316 urlfilter=16553/21491/20 uf_lookupf=0 scan=23786 clt=278876 srv=368557 SMTP Accounting setup_ok=12 setup_fail=0 conn_ok=0 conn_inp=12 scan=12 suspend=0 resume=0 reject=0 spamadd=0 spamdel=0 clt=275 srv=279 POP3 Accounting
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
99
setup_ok=30 setup_fail=0 conn_ok=0 conn_inp=30 scan=3 clt=5690 srv=5836 IMAP Accounting setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0 scan=0 clt=0 srv=0 FTP Accounting setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0 scan=0 clt=0 srv=0 datalisten=0 dataclt=0 datasrv=0 NNTP Accounting setup_ok=0 setup_fail=0 conn_ok=0 conn_inp=0 scan=0 clt=0 srv=0 The output from this command falls into the following sections: HTTP Common current connections There is an entry for each protocol that displays the connections currently used, and the maximum connections allowed. This maximum is for the UTM proxy which means all the protocols connections combined cannot be larger than this number. To support this, note that the maximum session count for each protocol is the same. You may also see a line titled Max Concurrent Connections for each protocol. This number is the maximum connections of this type allowed at one time. If VDOMs are enabled, this value is defined either on the global or per-VDOM level at VDOM > Global Resources or in the CLI at config system resource-limits. Worker Stat This is statistics about the UTM proxy including how long it has been running, and how many errors it has found. HTTP Stat This section includes statistics about the HTTP protocol proxy. This is a very extensive list covering errors, web responses, and any UTM positive matches. There are similar sections for each protocol, but the specific entries in each vary based on what UTM scanning is looking for in each spam control for email, file transfer blocking for FTP, and so on. Worker Accounting Lists accounting information about the UTM proxy such as polling statistics, how many sessions were scanned, and how many were just accepted. This information can tell you if expect AV scanning is taking place or not. Under normal operation there should be no errors or fails. HTTP Accounting The accounting sections for each protocol provide information about successful session creation, failures, how many sessions are being scanned or filtered, and how many are client or server originated. If setup_fail is larger than zero, run the command again to see if it is increasing quickly. If it is, your FortiGate unit may be in conserve mode.
Related commands
To clear the UTM proxy statistics # get test app proxyworker 8 To drop all idle connections # get test app proxyworker 222 To display statistics per VDOM # get test app proxyworker 4444 For additional related commands, see get test on page 166.
Troubleshooting for FortiOS 4.0 MR3 01-431-0129304-20120124 http://docs.fortinet.com/
100
101
102
FortiOS Handbook
Troubleshooting advanced
There are issues other than connectivity that require troubleshooting. These are more advanced issues that may require investigating a number of different issues before the problem is solved. The advanced troubleshooting sections include, and can help answer the following questions: Traffic shaping issues on page 103 Is an application or user consuming all your network bandwidth? Do some applications need more bandwidth than they are getting? Is less bandwidth allowed than expected on a connection? Are sessions or packets being dropped? User and administrator logon issues on page 107 Are one or more of your users unable to log on to the network? Are one or more of your users unable to initiate VPN connections? Are you having problems logging in to administrator accounts? IPsec VPN issues on page 111 Are VPN negotiations slow? Are VPN connections not completing the initial Phase 1 handshake? Is the IPsec VPN tunnel down? For connectivity troubleshooting tips, see Troubleshooting common issues on page 77. In addition to these steps, you may find diagnose commands useful. See Other diagnose commands on page 117.
103
Troubleshooting advanced
Monitoring traffic
Traffic shaping is best used with a traffic monitor, such as the Traffic History widget that is available on the Dashboard. Configure two one to watch the internal interface and another for the external interface. Run these monitors for a while before trying the traffic shaping procedures to get a feel for your bandwidth usage. To configure two traffic history monitors - web-based management 1 Go to System > Dashboard > Status. 2 Select Widget > Traffic History. 3 Select the edit icon in the widgets menu bar. 4 Name the widget Internal, and select the Internal interface to monitor. 5 Enable Refresh, and select OK. 6 Repeat steps 2 through 5 selecting and naming this second monitor external. Let the monitors run for a while to display network traffic before starting troubleshooting. The times you start and stop traffic on the FortiGate unit is displayed in the monitors. It may be advisable to take screen captures of the monitors at each stage for easier comparison later.
104
Troubleshooting advanced
# diag firewall shaper traffic-shaper list policy 4 ipv4 name traffic_test_app maximum-bandwidth 12 KB/sec guaranteed-bandwidth 0 KB/sec current-bandwidth 0 B/sec priority 3 policy 4 packets dropped 0 stats Display statistics for the traffic shaper. Add list to the command to view the stats. The stats are very brief number of shapers, number of IPv4 shapers, number of IPv6 shapers, and number of packets dropped. Or, you can wipe the stats by adding the word clear on the end of the command. This will remove all store statistics and start fresh. state Displays the total number of global traffic shapers. This is useful if VDOMs are enabled on your FortiGate unit as it will list all traffic shapers on your FortiGate unit. This command has no additional arguments.
4 Go to Firewall > Policy > Policy. The following steps will override any existing security policies. Do not perform these steps on a live network without notifying users first. Optionally, use the VDOM method mentioned earlier to limit disruption of users.
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
105
Troubleshooting advanced
5 Select Create New. 6 Select any and all for source and destination interfaces and addresses. 7 Select always for schedule. 8 For service, select Multiple and enter all the FTP related entries. 9 Select ACCEPT for Action. 10 Enable NAT. 11 Enable Traffic Shaping, and select traffic_test_app. 12 Select OK. 13 Select the new security policy in the list. 14 Select Move, and insert it before Seq. No. 1. 15 Repeat steps 5 through 10, but for service select all services. This works because the first policy will catch all FTP traffic first before it gets to this policy. 16 Select OK. 17 Select the new security policy in the list. 18 Select Move, and insert it below Seq. No. 1. This procedure creates two security policies that will catch all the traffic the first one catches all FTP traffic and limits its bandwidth, and the second one matches everything else. If FTP is found to not be the problem, it is easy to remove FTP from the first policy and add a different protocol or group of protocols to see if they are responsible. Once the bandwidth consuming protocol is discovered, it can be traffic shaped using security policies (clone existing policies and change the first one to only apply to that protocol with the traffic shaping applied), users can be informed to limit their use of that protocol or application, or that protocol could be blocked completely.
106
Troubleshooting advanced
myTrafficShaper is a shared traffic shaper. Some of the output displayed has different meanings depending how this shaper is applied. If this shaper is applied to only one security policy, then the numbers displayed are for the traffic flowing through that one security policy. But if this shaper is applied to multiple policies, the currentbandwidth and packets dropped will be combined for all instances, and will be less useful for troubleshooting. For this reason, if possible, either create multiple copies of the traffic shaper with different names to allow for easier troubleshooting, or if possible disable all security policies but one until you get the information you need.
107
Troubleshooting advanced
108
Troubleshooting advanced
To verify there is a certificate to authenticate 1 Go to User > User Group > User Group. 2 Expand Firewall if required. You should pay attention to the user groups that include the affected user or users. Take special attention if multiple affected users are all part of one group. 3 Go to Policy +> Policy > Policy. Ensure the Authentication column is visible. If not, do the following to show the column: Select Column Settings. Move Authentication to the list on the right. Move the Authentication entry to your preferred place in the list using the Move up and Move down buttons. Select OK. 4 Locate one or more authentication policies that authenticate the user group or groups this user belongs to as noted earlier. The user group or groups that a security policy authenticates will appear in the Authentication column. 5 Edit these security policies to look for any that use certificate authentication. Identify which certificate is required, and ensure the user has that certificate installed on their system.
109
Troubleshooting advanced
When a FortiGate unit is shipped, all access on external interfaces is disabled by default except for PING for troubleshooting purposes. This means any attempt by an administrator to logon over a dmz interface will be blocked by default. The following procedures demonstrate enabling full access on dmz interface. For security purposes, you should only enable access that is required. If you open access for troubleshooting, remember to disable it afterwards. Failure to do so will leave a gap in your security that hackers may be able to exploit. To enable administrator access on the dmz interface - web-based manager 1 Log on as administrator. 2 Go to System > Network > Interface, select the dmz interface, and select Edit. 3 Under Administrative Access, select HTTPS, PING, HTTP, SSH, SNMP, and TELNET. 4 If you use IPv6, select the above protocols to be used for IPv6 administrative access. 5 Select OK. 6 Repeat for each interface where administrative access is required. To enable administrator access on the dmz interface - CLI config system interface edit dmz set allowaccess HTTPS PING HTTP SSH SNMP TELNET set allowaccess6 HTTPS PING HTTP SSH SNMP TELNET next end
110
Troubleshooting advanced
To verify trusted host logon issues - CLI 1 Record the IP address where the administrator is attempting to logon to the FortiGate unit. 2 Log on to the FortiGate unit using the console connection for your FortiGate unit. 3 Enter the following CLI commands: config system admin edit <admin_account> get This will display all the settings for this admin account, including trusted hosts 4 Compare the listed trusted hosts to the IP address the attempted logon is coming from. If there is a match, the problem is not due to trusted hosts. 5 If there is no match and the new address is valid (secure), add it to the list of trusted hosts using the following CLI commands: set trusthost3 <ipv4_addr> <ipv4_netmask> next end Use the numbered trusthost that follows the last configured trusthost. For example if trusthost, trusthost1, and trusthost2 are in use you will use trusthost3. If the problem was due to trusted hosts, the admin account will be able to log on now.
FortiGate asking for password when creating a remotely authenticated administrator account
Local administrator accounts require you to enter a password so the administrator can be locally authenticated when logging on. A remote administrator account uses a remote authentication server to authenticate the administrator when logging on instead of using a local password. When you are creating a remote administrator account, if you leave out the password you will not be able to create the administrator account. This password field is a required field. For remote administrator accounts, the password field is backup password. During normal operations this account will authenticate remotely. However, if the authentication server is not reachable, authentication will occur at the local level instead to ensure administrator access is available.
111
Troubleshooting advanced
VPN tunnel proposal will not connect VPN Tunnel up but no traffic going over it Other useful VPN IKE related commands
112
Troubleshooting advanced
Each end of the attempted connection may have multiple protocols configured. While this gives a broader range of settings for a better chance of a match, it can result in long rambling attempts to connect that are difficult to troubleshoot. Best practices are to set the proposal information on each end to exactly the same and it will match with only one proposal definition required. However, frequently you do not have access to both ends of the proposed tunnel and instead have to match the other end. If you are trying to offload VPN processing to a network processing unit (NPU), remember that only SHA1 authentication is supported. For high levels of authentication such as SHA256, SHA384, and SHA512 hardware offloading is not an option all VPN processing must be done in software. To determine what the other end of the VPN tunnel is proposing 1 Start a terminal program such as puTTY and set it to log all output. When necessary refer to the logs to locate information when output is verbose. 2 Log on to the FortiGate unit using a super_admin account. 3 Enter the following CLI commands. 4 Display all the possible IKE error types and the number of times they have occurred: diag vpn ike errors 5 Check for existing debug sessions: diag debug info If a debug session is running, to halt it enter: diag debug disable 6 Confirm your proposal settings: diag vpn ike config list 7 If your proposal settings do not match what you expect, make a change to it and save it to force an update in memory. If that fixes the problem, stop here. 8 List the current vpn filter: diag vpn ike filter 9 If all fields are set to any, there are no filters set and all VPN ike packets will be displayed in the debug output. If your system has only a few VPNs, skip setting the filter. If your system has many VPN connections this will result in very verbose output and make it very difficult to locate the correct connection attempt. 10 Set the VPN filter to display only information from the destination IP address for example 10.10.10.10 : diag vpn ike log-filter dst-addr4 10.10.10.10 To add more filter options, enter them one per line as above. Other filter options are displayed in Table 5.
Table 5: Filter options for diag vpn ike filter
Erases the current filter. The IPv6 destination address range to filter by. The destination port range to filter by. Interface that IKE connection is negotiated over.
113
Troubleshooting advanced
Displays the current filter. The phase1 name to filter by. Negate the specified filter parameter. The IPv4 source address range to filter by. The IPv6 source address range to filter by. The source port range to filter by. Index of virtual domain. 0 matches all.
11 Start debugging: diag debug app ike 255 diag debug enable 12 Have the remote end attempt a VPN connection. If the remote end attempts the connection, they become the initiator. This situation makes it easier to debug VPN tunnels because then you have the remote information and all of your local information. by initiate the connection, you will not see the other ends information. 13 If possible go to the web-based manager on your FortiGate unit, go to the VPN monitor and try to bring the tunnel up. 14 Stop the debug output: diag debug disable 15 Go back through the output to determine what proposal information the initiator is using, and how it is different from your VPN P1 proposal settings. Things to look for in the debug output of attempted VPN connections are shown in Table 6.
Table 6: Important terms to look for in VPN debug output
initiator responder local ID error no SA proposal chosen R U THERE and R U THERE ack
Starts the VPN attempt, in the above procedure that is the remote end. Answers the initiators request. In aggressive mode, this is not encrypted. There was no proposal match there was no encryptionauthentication pair in common, usually occurs after a long list of proposal attempts. dead peer detection (dpd), also known as dead gateway detection after three failed attempts to contact the remote end it will be declared dead, no farther attempts will be made to contact it. Lists the proposal settings that were agreed on. Negotiating a new security association (SA) key, and the key life.
114
Troubleshooting advanced
R U THERE
This is the keep alive message. The reply is R U THERE ack. If you see this, it means Phase 1 was successful. The negotiation was successful, the VPN tunnel is operational.
tunnel up
config list Lists all the IKE configurations. counts Displays list of IKE objects and their current, maximum, and total counts. For example: crypto.md5: now 0 max 1 total 24 crypto Either sets or displays hardware or software crypto settings. crypto hardware - use hardware crypto (where possible) crypto software - use software crypto status - show number of hardware and software crypto objects filter log filter log-filter gateway Sets the IKE filter. See Table 5 on page 113.
Displays VPN IKE gateways. Optionally clear or flush (same result) the gateways in memory.
115
Logging
Troubleshooting advanced
restart
Restarts the IKE daemon. This is useful if during troubleshooting you discover that settings in memory do not match what you set. All VPN connections will be lost when restarting. If you have active VPN connections, give those connections sufficient notice to close gracefully.
routes list Displays all routes in memory for IKE VPN tunnels. status Displays status of IKE objects. detailed - lists vdom, name, version, IKE SA, and IPsec SA summary - lists number of IKE SA and IPsec SA objects
Logging
The following contains information regarding troubleshooting logging issues on the unit. This topic contains the following: Cannot log to a supported log device The alert email did not send an email to the email address The FortiGate unit stopped logging: what happened?
The alert email did not send an email to the email address
Verify that the alert email configuration settings are correct; an alert email will not be sent if the email address requires authentication and Authentication and its settings are not configured. Use Test Connectivity after verifying the alert email configuration settings see if a test alert email is sent to the address or addresses.
116
Troubleshooting advanced
117
Troubleshooting advanced
118
FortiOS Handbook
119
Syntax
exec tac report
Parameters
None.
120
Usage/Remarks
This command provides extremely detailed information about your FortiGate unit. Fortinet support may request that you run this command and send them the output when they are troubleshooting a FortiGate problem with you. When you run this command, it will take a few minutes to collect and display all the information. Your console program (telnet, ssh, or other such program) should log the output to file. Otherwise, the output will scroll by too fast to see. No output is listed for this command because the output is extensive, and many of the commands from this report are described elsewhere in this section. Do not include the output from this command in FortiCare tickets unless it is specifically requested by support personnel. Scope: Global
121
Syntax
get firewall iprope appctrl list get firewall iprope appctrl status
Parameters
None.
Usage/Remarks
Use this command to view the rules defined for peer to peer, or the status of those rules. Scope: Vdom
Output Example
FGT # get firewall iprope appctrl app-id=17953 list-id=2004 app-id=17954 list-id=2004 app-id=17956 list-id=2004 app-id=17957 list-id=2004 app-id=107347980 list-id=2004 app-id=108855300 list-id=2004 app-id=109051910 list-id=2004 app-id=109051912 list-id=2004 list action=Pass action=Pass action=Pass action=Pass action=Pass action=Pass action=Pass action=Pass
FGT # get firewall iprope appctrl status appctrl table 3 list 1 app 1083 shaper 0 Keyword/Variable list status Description Display all the application IDs, which list they are in by ID, and the action taken for each application. Displays the number of ipropes for application control tables, lists, applications, and traffic shapers.
122
Syntax
get firewall iprope list <policy group number>
Parameters
<policy group > The number of the policy group as defined in the webbased manager, in Policy > Policy > Policy.
Usage/Remarks
Use this command to view the view the rules defined for a policy group. This is useful to understand the behavior of the policy group per protocol. The policy group number is easiest to find by just running this command without the number (get firewall iprope list) and finding the group that includes your policy. For example if you have a traffic shaper for your security policy called shaper_test, look for that information in one of the policy group entries to ensure it is the correct one. After that you can use the policy group number to limit the output to only the group you want. Scope: Vdom
Output Example
FGT # get firewall iprope list 0020005 policy flag (10809): log redir d_rm master flag2 (0): shapers: shaper_test(3/12800/64000)/shaper_test(3/12800/64000) per_ip= imflag: sockport: 0 action: accept index: 1 schedule(always) group=00200005 av=00004e20 au=00000000 host=0 split=00000000 chk_client_info=0x0 app_list=0 ips_view=0 misc=0 grp_info=0 seq=0 hash=0 tunnel= zone(0): ->zone(0): source(0): dest(0): source wildcard(0): destination wildcard(0): vip(2): 2 1 service(4): [1:0x0:0/(0,65535)->(8,8)] [6:0x0:0/(0,65535)->(443,443)] [6:0x0:0/(0,65535)->(22,22)] [6:0x0:0/(0,65535)->(80,80)] nat(0): mms: 0 0
123
A series of options that are either on or off, including log, redir, d_rm, master, pol_stats, auth, a_i, nlb, and nat. The number (10809) represents what flags are used. Using 10809 as an example, log redir, d_rm, and master flags are set. Flags for various things including fc_chk and disclaim. Shapers lists the configured traffic shapers by name, separated for shared and per ip shapers. action This is the action the firewall policy will take, defined as one of ACCEPT, DENY, IPSEC, or SSL-VPN. The source or destination IP address or IP address range. For example an entry of 0.0.0.0255.255.255.255 would be an address of all. This is a list of the services for this policy. The number is the number of services with entries that immediately follow. The first service entry, in this case from a list of four. It covers all ports on the source that are going to port 8 on the destination. If there was a range of ports, for example port 8000-8888, the entry would be >(8000,8888). These values are only used for MMS related firewall entries. Only FortiOS Carrier support MMS traffic.
flag2 (0): shapers: shaper_test(3/12800/64000)/sh aper_test(3/12800/64000) per_ip= imflag: sockport: 0 action: accept index: 1 tunnel= source(0): dest(0):
service(4):
[1:0x0:0/(0,65535)->(8,8)]
nat(0): mms: 0 0
124
Syntax
get firewall proute
Parameters
None.
Usage/Remarks
Use this command to view the policy routes configured on this FortiGate units current VDOM. Scope: Vdom
Output Example
FGT # get firewall proute list route policy info(vf=root): iff=12 src=10.10.10.0/255.255.255.0 tos=0x00 tos_mask=0x00 dst=10.10.11.0/255.255.255.0 protocol=11 port=1:65535 oif=13 gwy=0.0.0.0 Keyword/Variable vf iff src tos tos_mask dst protocol port oif gwy Description The current virtual domain (VDOM). All policy routes for this VDOM are displayed. Incoming interface The source IP and netmask for incoming traffic to be matched to the policy Type of service bit pattern in hexadecimal. This is matched and is good for a specific TOS. Type of service bit mask in hexadecimal. Masks out unwanted bits. This is good for matching multiple TOS values. The destination IP and netmask for incoming traffic to be matched to the policy. The protocol number to be matched. The range of ports to match for incoming traffic. Outgoing interface where traffic is being directed to. Outgoing gateway where traffic is being directed to
125
Syntax
get firewall shaper [ per-ip| per-ip-shaper| traffic| trafficshaper ]
Parameters
Keyword/Variable per-ip per-ip-shaper traffic traffic-shaper Description Displays all configured per-ip traffic shapers with their configurations. Lists all configured per-ip traffic shapers. Displays all configured shared traffic shapers with their configurations including current bandwidth usage. Lists all configured shared traffic shapers.
Usage/Remarks
Use this command to view traffic shaper bandwidths with traffic levels, and list configured traffic shapers for either shared (traffic) or per-ip traffic shapers. For per-ip and traffic commands, if there is no shaper configured none will be displayed. Scope: Vdom
Output Example
FGT # get firewall shaper per-ip [ip_test] maximum-bandwidth: 12 KB/sec maximum-concurrent-session: 100 packets dropped: 0 Keyword/Variable ip_test 12 KB/sec Description The name of the configured per-ip traffic shaper. The maximum allowed bandwidth for this traffic shaper. The web-based manager units for this field are kbits/sec where this value is in kBytes/sec. Any traffic for this IP address that is over this limit will be dropped. For example entering 100 kbits/sec results in a value of 12 kBytes/sec being displayed here. 100 The maximum allowed number of concurrent sessions for this traffic shaper. If the maximum is reached, no new sessions will be started on this IP address until another session ends. The number of packets that have been dropped from traffic using this shaper. Packets at rates greater than the maximum bandwidth limit are dropped.
126
FGT # get firewall shaper traffic [guarantee-100kbps] maximum-bandwidth: 131072 KB/sec guaranteed-bandwidth: 12 KB/sec current-bandwidth: 0 B/sec priority: high tos ff packets dropped: 0 Keyword/Variable guarantee-100kbps Description The name of the configured shared traffic shaper. There are a number of default configured traffic shapers that include guarantee-100kbps, high-priority, low-priority, mediumpriority, and shared-1M-pipe. The maximum allowed bandwidth for this traffic shaper. Any traffic for this shaper that is over this limit will be dropped. The web-based manager units for this field are kbits/sec where this value is in kBytes/sec. For example entering 1 048 576 kbits/sec results in a value of 131 072 kBytes/sec being displayed here. 12 KB/sec 0 B/sec The guaranteed minimum bandwidth ensures there will never be less than this amount of bandwidth available. The current amount of traffic using this traffic shaper. If this shaper is not assigned to any security policies this value will be zero. Note that the units here are B/sec and not the KB/sec of the other fields for this command. high The quality of service (QoS) level for traffic using this shaper. Values can be low, medium, or high. Higher priority traffic will be queued ahead of lower priority traffic to ensure its guaranteed minimum bandwidth is met when necessary. The type of service (TOS) used in the TCP packet headers. It is used to enact the selected priority. The number of packets that have been dropped from traffic using this shaper. Packets at rates greater than the maximum bandwidth limit are dropped.
131072 KB/sec
tos ff 0
Syntax
get hardware cpu
Parameters
None.
127
Usage/Remarks
Use this command to view the specifications about CPUs on the FortiGate unit. Multiple CPUs will each have an entry. Apart from the CPU model information (vendor, cpu family, model name, MHz, and cache size) you can use the flags to determine what features are supported. For example mmx, and 3dnow indicate Single instruction, multiple data (SIMD) support. Scope: Global
Output Example
FGT # get hardware cpu processor: 0 vendor_id: GenuineIntel cpu family: 6 model: 5 model name: Celeron (Covington) stepping: 0 cpu MHz: 1200.078 cache size: 64 KB fdiv_bug: no hlt_bug: no f00f_bug: no coma_bug: no fpu: yes fpu_exception: yes cpuid level: 2 wp: yes flags: fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss tm pbe bogomips: 2392.06
128
Syntax
get hardware nic <interface>
Parameters
<interface> Enter the interface name. For example, internal, wan1, wan2.
Usage/Remarks
Use this command to view the interface information and statistics. This is useful when debugging network problems on a particular interface, or providing Customer Support with hardware information about your FortiGate unit. This command is the only way for you to see your MAC both its original value (Permanent_HWaddr ) and if it has changed as its current value (Current_HWaddr)as well. The output will be different for different NICs and may include more fields or different information fields. Scope: Global
Output Example
FGT # get hardware nic wan1 Description sundance Ethernet driver1.01+LK1.21 chip_id 6 IRQ 5 System_Device_Name wan1 Current_HWaddr 00:09:0f:78:71:32 Permanent_HWaddr 00:09:0f:78:71:32 State up Link Speed Duplex FlowControl MTU_Size Rx_Packets Tx_Packets Rx_Packets Tx_Bytes Collisions Rx_Missed_Errors Tx_Carrier_Errors Keyword/Variable Description chip_id
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
up 100 full Tx off, Rx off 1500 52377 53098 47029877 7199983 0 0 0 Description Name of the network driver. Id of the chipset
129
Keyword/Variable IRQ Current_HWaddr Permanent_HWaddr State Link Speed Duplex Rx_Packets Tx_Packets Rx_Packets Tx_Bytes Collisions Rx_Missed_Errors Tx_Carrier_Errors
Description IRQ Current MAC address. Permanent MAC address. Administrative status (up/down). Link status (up/down). Negotiated or configured network speed. Negotiated or configured duplex mode. Packets' number received by the network device. Packets number transmitted by the network device. Amount of bytes received on the interface. Amount of bytes sent from this interface. Number of collisions usually due mostly to incorrect incorrect speed or duplex settings. Equals Rx_FIFO_Errors + CEXTERR (Carrier Extension Error Count). Only valid in 1000M mode, which is marked by PHY. The PHY should assert the internal carrier sense signal during every transmission. Failure to do so may indicate that the link has failed, or the PHY has an incorrect link configuration. This register only increments if transmits are enabled. This register is not valid in internal SerDes1 mode (TBI mode for the 82544GC/EI), and is only valid when the Ethernet controller is operating at full duplex.
130
Syntax
get hardware memory
Parameters
None.
Usage/Remarks
Use this command to view the status of the memory resources. This is useful to confirm and identify any potential memory leaks or even to simply confirm that reduced FortiOS performance is due to a shortage of free memory. Scope: Global
Output Example
FGT # get hardware memory total: Mem: 261955584 Swap: 0 MemTotal: MemFree: MemShared: Buffers: Cached: SwapCached: Active: Inactive: HighTotal: HighFree: LowTotal: LowFree: SwapTotal: SwapFree: Keyword/Variable Mem Swap MemTotal MemFree SwapCached Active Inactive used: free: shared: buffers: cached: shm: 97312768 164642816 0 217088 55382016 51322880 0 0 255816 kB 160784 kB 0 kB 212 kB 54084 kB 0 kB 22832 kB 31524 kB 0 kB 0 kB 255816 kB 160784 kB 0 kB 0 kB Description Memory size. Amount of memory in the swap. HighTotal + LowTotal = amount of memory available on the unit. Free memory on the unit. Amount of memory out of the Swap, but remaining in the cache. Amount of memory recently used. Amount of memory which has not been used for a while.
131
Description Amount of memory which belongs to the zone ZONE_HIGHMEM. Amount of Free memory available in the zone ZONE_NORMAL.
132
Syntax
get hardware npu { legacy | np1 | np2 | np4 } list
Parameters
{ legacy | np1 | np2 | np4 } Specify which level of NPU interfaces are to be displayed.
Usage/Remarks
Use this command to view the NPU devices and port numbers. This is useful because some FortiOS products contain network processors. Network processor features, and offloading requirements vary by network processor model. The output is grouped by the ID number. Each ID number has four interfaces associated with it. These are the interfaces that are available to create paired interfaces. This is because each such group is assigned to one NPU. Mixing interfaces from different NPUs is not allowed. For example, from the output example, you would not be able to pair port1 with port14. Use this information when configuring your paired interfaces for accelerated offloading. If the specified version of NPU is not present on this FortiGate unit, a message indicating that will be displayed. Scope: Global
Output Example
FGT # get hardware npu np1 list ID Interface 0 port9 port10 FGT # get hardware npu np2 list ID PORTS -- ----0 port1 0 port2 0 port3 0 port4 ID PORTS -- ----1 port5 1 port6 1 port7 1 port8 ID PORTS -- ----FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
133
2 2 2 2
ID PORTS -- ----3 port13 3 port14 3 port15 3 port16 Keyword/Variable ID Interface Description The ID of the NPU assigned to these ports. The names of the interfaces in the NPUs paired group. An NPU can handle either two or four ports. Any incoming traffic on this group of ports that leaves the FortiGate unit on the same group of ports can be handled by the NPU. Otherwise, the FortiGate units CPU will be involved. PORTS The names of the interfaces in the NPUs group.
134
Syntax
get hardware npu { legacy | np1 | np2 | np4 }performance <dev_id>
Parameters
{ legacy | np1 | np2 | np4 } <dev_id> Specify which level of NPU interfaces are to be displayed. NPU ID number. If you are unsure you can enter a ? to get a list of available valid NPU ID numbers.
Usage/Remarks
Use this command to view the performance numbers for NPU devices. This can be useful when you are checking NPU trafficeither for specific problems or for network optimization. Network processor features, and therefore offloading requirements, vary by network processor model. The values displayed are divided into two sections. The first is the ISCP2 performance numbers. In this section, the values are mostly counts for the stated values. For example BADCSUM is the number of bad checksums encountered. The last section starts with IRQ. This is a list of 18 different registers. Scope: Global
Output Example
FGT # get hardware npu np2 performance 1 ISCP2 Performance: Nr_int : 0x00000005 INTwoInd : 0x00000000 RXwoDone : 0x00000000 PKTwoEnd : 0x00000000 PKTCSErr : 0x00000000 PKTidErr : 0x00000000 PHYInt : 0x0/0x0/0x0/0x0 CSUMOFF : 0x00000000 BADCSUM : 0x00000000 MSGINT : 0x00000000 IPSEC : 0x00000000 IPSVLAN : 0x00000000 SESMISS : 0x00000000 TOTUP : 0x00000000 TCPPLTOT : 0x00000000 TCPPLBADCT: 0x00000000 TCPPLBADL: 0x00000000 TCPPLSESI : 0x00000000 TCPPLSESR : 0x00000000 TCPPLBD : 0x00000000 TXInt : 0x0/0x0/0x0/0x0 RxI : 0x0 BDEmpty : 0x0/0x0/0x0/0x0 Congest: 0x0/0x0/0x0/0x0 TMM_Busy : 0x0 Poll List: 0 0 0 0 MSG Performance: TOTMSG : 0x00000000 BADMSG : 0x00000000 TOUTMSG : 0x00000000 MSGLostEvent : 0x00000000 QUERY : 0x00000000 TAE : 0x00000000
135
SAEXP-SN : 0x00000000 SAEXP-TRF : 0x00000000 OUTUPD 0x00000000 INUPD : 0x00000000 NULLTK: 0x00000000 NAT Performance: BYPASS (Enable) BLOCK (Disable)
IRQ :00000005 QFTL :00000000 DELF :00000000 FFTL :00000000 OVTH :00000005 QRYF :00000000 INSF :00000000 INVC :00000000 ALLO :00000005 FREE :00000005 ALLOF :00000000 BPENTR:00000000 B KENTR:00000000 PBPENTR:00000000 PBKENTR:00000000 NOOP :00000000 THROT :0000000 0(0x002625a0) SWITOT:00000000 SWDTOT:00000000 ITDB:00000000 OTDB:000000000 SPISES:00000000 FLUSH:00000021 Keyword/Variable Description
ISCP2 Performance This section displays information about the ISCP2 performance that includes packet, TCP, and IPsec counts and errors. MSG Performance NAT Performance This section displays information about message traffic performance. This section displays information about NAT traffic performance. If BYPASS is enabled, many of the counters will be zero.
136
Syntax
get hardware npu { legacy | np1 | np2 | np4 }status <dev_id>
Parameters
{ legacy | np1 | np2 | np4 } <dev_id> Specify which level of NPU interfaces are to be displayed. NPU ID number. If you are unsure you can enter a ? to get a list of available valid NPU ID numbers.
Usage/Remarks
Use this command to display the status of the interfaces attached to the NPU. This can be useful for advanced users to debug traffic on the NPU interfaces. If you enter an NPU or device ID that doesnt exist, an error message stating that it was an invalid NPU ID will be displayed. Scope: Global
Output Example
FGT # get hardware npu np2 status 1 NP2 Status ISCP2 c2160000 (Neighbor f7940000) 1a29:0703 256MB Base f8a5a000 DBG 0x00000000 RX SW Done 0 MTP 0x0 desc_alloc = c2152000 desc_size = 0x2000 count = 0x100 nxt_to_u = 0x0 nxt_to_f = 0x0 Total Interfaces: 4 Total Ports: 4 Number of Interface In-Use: 4 Interface c2160100 netdev c22bb000 0 Name port5 PHY: Attached LB Mode 0 LB IDX 0/1 LB Ports: c2160644, 00000000, 00000000, 00000000 Port c2160644 Id 0 Status Down ictr 4 desc = c2232000 desc_size = 0x00001000 count = 0x00000100 nxt_to_u = 0x00000000 nxt_to_f = 0x00000000 Intf c2160100 Interface c2160250 netdev c2168c00 1 Name port6 PHY: Attached LB Mode 0 LB IDX 0/1 LB Ports: c21606f8, 00000000, 00000000, 00000000 Port c21606f8 Id 1 Status Down ictr 0 desc = c2126000 desc_size = 0x00001000 count = 0x00000100
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
137
nxt_to_u = 0x00000000 nxt_to_f = 0x00000000 Intf c2160250 Interface c21603a0 netdev c2168800 2 Name port7 PHY: Attached LB Mode 0 LB IDX 0/1 LB Ports: c21607ac, 00000000, 00000000, 00000000 Port c21607ac Id 2 Status Down ictr 0 desc = c2123000 desc_size = 0x00001000 count = 0x00000100 nxt_to_u = 0x00000000 nxt_to_f = 0x00000000 Intf c21603a0 Interface c21604f0 netdev c2168400 3 Name port8 PHY: Attached LB Mode 0 LB IDX 0/1 LB Ports: c2160860, 00000000, 00000000, 00000000 Port c2160860 Id 3 Status Down ictr 0 desc = c2122000 desc_size = 0x00001000 count = 0x00000100 nxt_to_u = 0x00000000 nxt_to_f = 0x00000000 Intf c21604f0 NAT Information: cmdq_qw = 0x2000 cmdq = c2140000 head = 0x5 tail = 0x5 APS (Enabled) information: Session Install when TMM TSE OOE: Disable Session Install when TMM TAE OOE: Disable IPS anomaly check policy: Follow config MSG Base = c2130000 QL = 0x1000 H = 0x0
138
Syntax
get hardware status
Parameters
None.
Usage/Remarks
Use this command to get basic information about your FortiGate units hardware. This is a short list that can be used to help guide troubleshooting efforts, especially by customer service. For example, if there is supposed to be compact flash memory available but this command shows 0MB available, then you know there is a hardware problem with the memory. Also, if there are known issues with particular chipsets, this command will display chipset information for hardware such as network cards. The CPU information can vary from simply the number of cores, to the more in-depth information listed here. Scope: Global
Output Example
FGT # get hardware status Model name: Fortigate-3810A ASIC version: CP6 ASIC SRAM: 64M CPU: Intel(R) Core(TM)2 Duo CPU E4300 @ 1.80GHz RAM: 3532 MB Compact Flash: 122 MB /dev/hda USB Flash: not available Network Card chipset: Broadcom 570x Tigon3 Ethernet Adapter (rev.0x8003) Network Card chipset: Intel(R) PRO/1000 Network Connection (rev.0006)Related Commands
139
Syntax
get ips session
Parameters
None.
Usage/Remarks
Viewing the IPS session status allows you to see if traffic is checked by the IPS engine and if the IPS engine is working as expected. You can see the type of traffic and how many sessions are being processed by the IPS engine. You can also see how much memory the IPS engine is using. The displayed information is repeated with short delays to provide a time-lapse view of the IPS sessions status. Scope: Vdom
Output Example
FGT # get ips session SYSTEM: memory capacity 73400320 memory used 3982780 recent pps\bps 0\0K session in-use 0 TCP: in-use\active\total 0\0\0 UDP: in-use\active\total 0\0\0 ICMP: in-use\active\total 0\0\0 Keyword/Variable memory capacity memory used Description Total memory capacity in system available to IPS. Memory used by IPS sessions. If this value is high, close to the memory capacity, it will likely result in dropped packets and other issues. Recent IPS traffic in packets per second (pps) and bits per second (bps). Use these numbers to determine the level of IPS traffic. If they are very high, there will likely be problems with the FortiGate units performance. Current number of IPS sessions in use. The number of in use, active, and total Transmission Control Protocol (TCP) packets sent over this session. The number of in use, active, and total User Datagram Protocol (UDP) packets sent over this session. The number of in use, active and total Internet Control Message Protocol (ICMP) packets sent over this session.
recent pps\bps
140
Syntax
get router info kernel
Parameters
None.
Usage/Remarks
Use this command to view the kernel routing table. Computers and network devices connected to the Internet use routing tables to compute the next hop destination for a packet. The routing kernel table is an table of routes to particular network destinations that is stored in a router or a networked computer. This information contains the topology of the network immediately around it. Building the routing table is the primary goal of routing protocols and static routes. In the routing table, the information that displays allows you to see the type of route, the protocol used, and the priority of each entry. You can use get router info routing-table all on page 142 for more information on each routing entry. Scope: Vdom
Output Example
FGT # get router info kernel tab=254 vf=2 scope=0 type=1 proto=11 prio=0 0.0.0.0/0.0.0.0/0>192.168.0.0/24 pref=0.0.0.0 gwy=9.1.1.1 dev=5(port1) Keyword/Variable 192.168.0.0/24 gwy=9.1.1.1 dev=5(port1) Description The destination network or destination host. The gateway address for the next hop. Interface to which packets for this route is sent.
141
Syntax
get router info routing-table all
Parameters
None.
Usage/Remarks
Use this command to view the routing table. This routing table may be larger and have more entries than the kernel routing table when using the get router info kernel command. Computers and network devices connected to Internet use routing tables to compute the next hop destination for a packet. It is an table of routes to particular network destinations that is stored in a router or a networked computer. This information contains the topology of the network immediately around it. Scope: Vdom
Output Example
FGT# get router info routing-table all Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - ISIS inter area * - candidate default S* C C 0.0.0.0/0 [10/0] via 172.20.120.2, wan1 10.31.101.0/24 is directly connected, internal 172.20.120.0/24 is directly connected, wan1
The output of this command is fairly self-explanatory; however, if you select a specific routing protocol (static, rip, etc) when entering the command, only that part of this larger list of entries will be displayed. The default route is indicated by the asterisk (*). How a route is connected to the FortiGate unit is important in determining priority. The two C entries state is directly connected, which indicates the lowest possible hop count to get there.
142
Syntax
get system arp
Parameters
None.
Usage/Remarks
This command is useful to view or alter the contents of the kernel's ARP tables. For example. when you suspect a duplicate Internet address is the cause for some intermittent network problem. This command is not available in multiple VDOM mode. Scope: Vdom
Output Example
FGT # get system arp Address Age(min) Hardware Addr Interface internal 172.20.120.16 0 172.20.120.138 Keyword/Variable Address Age Hardware Addr Interface 00:0d:87:5c:ab:65 internal 0 00:08:9b:09:bb:01 Description The IP address that is linked to the MAC address. The default is 0.0.0.0 Current duration of the ARP entry in minutes. The default is 0. The hardware, or MAC address, to link with this IP address. The default is 00:00:00:00:00:00: The physical interface of the FortiGate unit where the address is connected.
143
Syntax
get system auto-update status
Parameters
None.
Usage/Remarks
Use this command when you need to view the current antivirus and IPS status from the FortiGate unit. The status command is used to test the current connectivity status, and can retrieve configuration information, such as push updates, update schedules, or other parameters related to the FortiGuard service operation. The versions command provides extended information about each FortiGuard component and its version, build number, contract expiry date, last update attempts and results. Result: Connectivity failure indicates the connections to the FortiGuard servers is not possible. This may be a serious problem that needs to be addressed. Until it is solved, you may not receive any FortiGuard updates leaving your network vulnerable. These commands also allow the user to check whether the FortiGate unit is running the latest AV and IPS packages. Scope: Global
Output Example
FGT # get system auto-update status FDN availability: available at Mon May 26 20:16:43 2008 Push update: disable Scheduled update: enable Update every: 1 hours at 16 minutes after the hour Virus definitions update: enable IPS definitions updates: enable Server override: disable Push address override: disable Web proxy tunneling: disable
144
Description Specify availability status and last access time (access time corresponds to the scheduled update settings). Possible values are: available/unavailable. Specify wether push update method is enabled or disabled. Possible values are: enable/disable Specify wether scheduled update is enabled or disabled. Possible values are: enable/disable. If scheduled update is enabled, specify the time defined to launch the update. Specify wether the IPS definitions update is enabled or disabled. Possible values are: enable/disable. Specify wether the use of another FDS server is enabled or disabled. Possible values are: enable/disable. If enabled a new line is displayed showing the FDS IP address defined in the configuration. For example: Server override: enable Server: 10.0.0.1
Push update
Push address override If push update is enabled, specify wether the Fortigate override address feature is enabled or disabled. Possible values are: enable/disable. If enabled, a new line is displayed showing the FDS IP address and the TCP port (a.b.c.d:port) defined in the configuration. Example: Push address override: enable Address: 10.0.0.2:9443 Web proxy tunneling Specify wether FortiGate device is using a proxy to retrieve AV and IPS definitions updates. Possible values are: enable/disable. If enabled, additional lines are displayed showing the proxy settings. Example: Web proxy tunneling: enable Proxy address: 10.0.0.3 Proxy port: 8890 Username: foo Password: foo
145
Syntax
get system auto-update versions
Parameters
None.
Usage/Remarks
Use this command to view the antivirus and IPS engines and definitions and license information. This is useful in determining when the antivirus engine, virus definitions, attack definitions, IPS attack definitions, AS Rule set, AS engine, and FDS address were last updated, as well as when their contracts expire, which version they are using, and the result of the last update. Scope: Global
Output Example
FGT # get system auto-update versions AV Engine --------Version: 3.003 Contract Expiry Date: n/a Last Updated using manual update on Wed Jan Last Update Attempt: n/a Result: Updates Installed
9 18:26:00 2008
Virus Definitions --------Version: 8.631 Contract Expiry Date: n/a Last Updated using manual update on Tue Jan 15 14:27:00 2008 Last Update Attempt: n/a Result: Updates Installed Attack Definitions --------Version: 2.461 Contract Expiry Date: n/a Last Updated using manual update on Fri Jan 18 11:23:00 2008 Last Update Attempt: n/a Result: Updates Installed IPS Attack Engine --------Version: 1.091 Contract Expiry Date: n/a Last Updated using manual update on Wed Jan Last Update Attempt: n/a
9 18:22:00 2008
146
Keyword/Variable Version Last Updated using manual update on Last Update Attempt Result
Description Version number of the engine or the definitions. Date of the last manual update. The date when the last update was attempted. The status of the last update.
147
Syntax
get system ha status
Parameters
Scope: Global.
Usage/Remarks
Usually you would log in to the primary unit CLI using SSH or telnet. In this case the diagnose system ha status command displays information about the primary unit first, and also displays the HA state of the primary unit (the primary unit operates in the work state). For a virtual cluster configuration, the diagnose system ha status command displays information about how the cluster unit that you have logged into is operating in virtual cluster 1 and virtual cluster 2. For example, if you connect to the cluster unit that is the primary unit for virtual cluster 1 and the subordinate unit for virtual cluster 2, the output of the diagnose system ha status command shows virtual cluster 1 in the work state and virtual cluster 2 in the standby state. The diagnose system ha status command also displays additional information about virtual cluster 1 and virtual cluster 2. See the FortiGate CLI Reference Guide or the High Availability chapter for more information. Scope: Global
Output Example
FGT # get system ha status Model: 5000 Mode: a-a Group: 0 Debug: 0 ses_pickup: disable load_balance: disable schedule: round robin Master:128 5001_Slot_4 FG50012204400045 1 Slave :100 5001_Slot_3 FG50012205400050 0 number of vcluster: 1 vcluster 1: work 10.0.0.2 Master:0 FG50012204400045 Slave :1 FG50012205400050 Keyword/Variable Model Mode Description The FortiGate model number. The HA mode of the cluster: a-a or a-p. If the unit is not in HA mode, standalone will be displayed.
148
Description The group ID of the cluster. The debug status of the cluster. The status of the session pick-up feature, also known as session failover. Session failover means that a cluster maintains active network TCP and IPsec VPN sessions after a device or link failover. Session failover does not failover UDP, multicast, ICMP, or SSL VPN sessions. In some cases UDP sessions may be maintained after a failover.
The status of the load-balance-all keyword: enable or disable. Relevant to active-active clusters only. The active-active load balancing schedule. Relevant to activeactive clusters only. Master displays the device priority, host name, serial number, and cluster index of the primary (or master) unit. Slave displays the device priority, host name, serial number, and cluster index of the subordinate (or slave, or backup) unit or units. The list of cluster units changes depending on how you log into the CLI. Usually you would use ssh or telnet to log in to the primary unit CLI. In this case, the primary unit would be at the top the list followed by the other cluster units. If you use the execute ha manage command or a console connection to log in to a subordinate units CLI, and then enter diagnose system ha status, the subordinate unit that you have logged into appears at the top of the list of cluster units.
number of vcluster
The number of virtual clusters. If virtual domains are not enabled, the cluster has one virtual cluster. If virtual domains are enabled, the cluster has two virtual clusters.
149
Syntax
get system performance firewall packet-distribution get system performance firewall statistics
Parameters
None
Usage/Remarks
Use this command to quickly see information about traffic through the firewall. The packet-distribution command divides network traffic into ten different packet sizes and lists the number of packets received in each category. This can help you spot hacking attempts, or optimize your network. The statistics command provides packet and byte counts for different major protocols and packet types through the firewall. Packet types include TCP, UDP, ICMP, and IP. Scope: All (Global and Vdom)
Output Example
FGT# get sys per firewall packet-distribution getting packet distribution statistics... 0 bytes 63 bytes: 3624747 packets 64 bytes 127 bytes: 802612 packets 128 bytes 255 bytes: 371774 packets 256 bytes 383 bytes: 712180 packets 384 bytes 511 bytes: 30691 packets 512 bytes 767 bytes: 57220 packets 768 bytes 1023 bytes: 23460 packets 1024 bytes 1279 bytes: 3055 packets 1280 bytes 1500 bytes: 64 packets > 1500 bytes: 0 packets FGT# get sys per firewall statistics getting traffic statistics... Browsing: 708624 packets, 408018318 bytes DNS: 411789906583486464 packets, 0 bytes E-Mail: 0 packets, 0 bytes FTP: 0 packets, 0 bytes Gaming: 0 packets, 0 bytes IM: 0 packets, 0 bytes Newsgroups: 0 packets, 0 bytes P2P: 0 packets, 0 bytes Streaming: 0 packets, 0 bytes TFTP: 201239863725391872 packets, 56530359549952 bytes VoIP: 3543070 packets, 25 bytes Generic TCP: 55834574848 packets, 1786706395136 bytes Generic UDP: 0 packets, 0 bytes
Troubleshooting for FortiOS 4.0 MR3 01-431-0129304-20120124 http://docs.fortinet.com/
150
151
Syntax
get system performance status
Parameters
None.
Usage/Remarks
Use this command to quickly see important information about your FortiGate units state. Information includes CPU usage, memory usage, network usage, number of sessions, viruses caught, IPS attacks blocked, and FortiGate unit uptime. These numbers provide a quick look at how the FortiGate unit is doing. If any one number needs attention, you can use other commands to get more information on that area. For example, if the CPU states show 98 percent system, you know that your FortiGate unit is running at full capacity and you need to check the processes to see if any one process is using all the resources and if there are valid reasons for it. Another example is that if memory usage is near 100 percent, AV failover may occur which, if enabled, may pass traffic without being scanned or refuse new connections. Scope: All (Global and Vdom)
Output Example
FGT# get sys performance status CPU states: 0% user 0% system 0% nice 100% idle Memory states: 10% used Average network usage: 0 kbps in 1 minute, 0 kbps in 10 minutes, 13 kbps in 30 minutes Average sessions: 31 sessions in 1 minute, 30 sessions in 10 minutes, 31 sessions in 30 minutes Virus caught: 0 total in 1 minute IPS attacks blocked: 0 total in 1 minute Uptime: 44 days, 18 hours, 42 minutes
152
Syntax
get system performance top <delay> <max_lines>
Parameters
delay max_lines The amount of time, in seconds, in which the process information is polled. The default is 5 seconds. The maximum number of processes displayed in the output. The default is 20 lines.
Usage/Remarks
Use this command when you need to view the processes running and the information about each process. It displays as a static set of columns where the information changes in place. To exit this display, press <Ctrl-C>. Scope: All (Global and Vdom)
Output Example
FGT # get system performance top 5 20 Run Time: 0 days, 1 hours and 53 minutes 1U, 1S, 97I; 248T, 99F, 73KF newcli 113 S 1.0 sshd 107 S 0.7 newcli 114 R < 0.3 thttp 48 S 0.0 ipsengine 55 S < 0.0 ipsengine 50 S < 0.0 cmdbsvr 18 S 0.0 httpsd 71 S 0.0 httpsd 92 S 0.0 httpsd 37 S 0.0 scanunitd 90 S < 0.0 scanunitd 91 S < 0.0 merged_daemons 45 S 0.0 newcli 108 S 0.0 updated 59 S 0.0 newcli 112 S < 0.0 miglogd 35 S 0.0 nsm 28 S 0.0 imd 53 S 0.0 authd 52 S 0.0 Keyword/Variable Run Time U S I
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
2.1 1.9 2.1 5.4 5.2 5.2 3.7 3.3 3.3 2.8 2.2 2.1 2.1 2.1 2.0 2.0 1.9 1.9 1.8 1.7
Description Displays how long the FortiOS has been running as a string User CPU usage (%) System CPU usage (%) Idle CPU usage (%)
153
Description Total memory Free memory Kernel-free memory Process name Process identification (PID) One letter process status. S: sleeping process R: running process <: high priority
Column 4 Column 5
154
Syntax
get system session-helper
Parameters
None.
Usage/Remarks
Use this command to display if any of the pre-defined session-helpers are in use. Scope: Global
Output Example
FGT # get system session-helper == 1 == == 2 == == 3 == == 4 == == 5 == == 6 == == 7 == == 8 == == 9 == == 10 == == 11 == == 12 == == 13 == == 14 == == 15 == == 16 == == 17 == == 18 == == 19 == == 20 == Keyword/Variable 1...20 Description If one of the pre-defined session helpers is in use, it will be listed here.
155
Syntax
get system session-info full-stat
Parameters
None.
Usage/Remarks
Use this command to display in-depth information session info about the systems state. This includes session table size, expected session table size, session count, firewall error details, and more. This data can provide an important picture of what is happening on your FortiGate unit, for example: if the ephemeral buffer is full, you have a very busy device which may indicate a DoS attack is under way. See ephemeral=2/32752. if there are many sessions in a SYN_SENT state to the point of causing other problems, you may be experiencing a SYN flood type DoS attack. See 10 in SYN_SENT state. if the HTTP proxy is too busy to handle new connections, then the ACCEPT queue fills up, and resets if it completely fills up, resetting the users connection. See acceptqf=0. Scope: Global
Output Example
FGT # get system session-info full-stat session table: table_size=131072 max_depth=1 used=50 expect session table: table_size=2048 max_depth=1 used=2 misc info: session_count=167 setup_rate=0 exp_count=0 clash=0 memory_tension_drop=0 ephemeral=2/32752 removeable=134 ha_scan=0 delete=0 flush=0 dev_down=0/0 TCP sessions 6 in ESTABLISHED state 10 in SYN_SENT state 25 in TIME_WAIT state 60 in CLOSE state 6 in CLOSE_WAIT state firewall error stat: error1=00000000 error2=00000000 error3=00000000 error4=00000000 tt=00000000 cont=0000630f ids_recv=0000dd8e url_recv=00000000 av_recv=00000000 fqdn_count=00000000
Troubleshooting for FortiOS 4.0 MR3 01-431-0129304-20120124 http://docs.fortinet.com/
156
tcp reset stat: syncqf=0 acceptqf=0 no-listener=0 data=0 ses=0 ips=0 global: ses_limit=0 ses6_limit=0 rt_limit=0 rt6_limit=0 The session setup rate is the second item on the first line of output. The information provided by this command includes: Keyword/Variable session table table_size=131072 max_depth=1 used=50 expect session table table_size=2048 max_depth=1 used=2 The number of expect session entries in the expect session table. Description The maximum number of entries possible in the expect session table. The number of session entries in the session table The maximum number of entries possible in the session table. Description
session_count=167 The number of sessions in the kernel. setup_rate=0 exp_count=0 clash=0 How fast sessions have been created. The number of expect sessions in the kernel. Count of the collisions that occurred during the creation of a new session.
memory_tension_dr Number of dropped sessions due to the system running out op=0 of memory. ephemeral=2/3275 2 The ephemeral buffer is used to protect against DoS attacks. It is a type of input buffer so the real session table does not get overloaded if a DoS attack does happen. The first number is how many sessions are in use, and the second number is the maximum number allowed. If the two numbers are close, it is a good chance there is a DoS attack underway, such as a DDoS using UDP packets.
157
Description During the lifetime of a TCP session, it changes state to reflect what stage it is at starting a connection, established, or closing. The connection has been established and is ready for data transfer. The FortiGate unit is waiting for the remote end of the session to acknowledge the SYN that was sent. If there are many sessions in this state, a SYN flood DoS attack may be in progress. To reduce the number of sessions in the SYN_SENT state, or half open state, you can reduce the wait period using the CLI command: config system global set tcp-halfopen-timer XX end where XX is the number of seconds to wait for the peer to respond before closing the connection. The valid range is from 1 to 86 400, and the default is 60 seconds. Reducing this value will free up half-open sessions before the session table completely fills up.
25 in TIME_WAIT state
The state when a connection termination request is sent to wait for enough time to ensure the remote end received acknowledgement of the termination request. If there are too many sessions in TIME_WAIT state, you can reduce the wait period using the CLI command: config system global set tcp-timewait-timer XX end where XX is the number of seconds to wait before closing the session. The valid range is 0 to 300 seconds, with the default being 120 seconds. A value of zero indicates no wait.
The session is closed. The session is being closed by the FortiGate unit there will be no more data from the sender. This waits for the last ACK before going to the CLOSE state. If there are a lot of sessions in this state, also called half closed state, you can reduce the timer using the CLI command: config system global set tcp-halfclose timer XX end where XX is the number of seconds to wait after one peer has sent a FIN packet for this session. The valid range is 0 to 86 400 seconds, with the default being 120 seconds.
158
Description This indicates there was a mismatch between the encryption state of the packet received and the session the packet belongs to. Either the packet was plain text when the session says it should have been decrypted (such as during a spoof), or the packet was decrypted properly but the session has no associated IPsec tunnel. Description SYN queue full Reset happens when the syn queue is full. FortiGate sends a reset (RST) when any of the following occur: syn queue full accept queue full FortiGate unit goes into conserve mode CPU use is too high (queues arent served correctly, and backlog increases) When this happens, the remote end will TCP timeout and close the TCP session, while on the FortiGate side the TCP session will go to a CLOSE_WAIT state.
acceptqf=0
ACCEPT queue full When a new packets comes in for a new session before being transmitted to the application layer of the device they are stored in a kernel queue. When this queue gets full, it means that the application layer was not able to process the packet. This is a symptom of an overloaded FortiGate unit. This happens when the new session rate of traffic being handled by proxy is too high.
no-listener=0
159
Syntax
get system session-info list
Parameters
None.
Usage/Remarks
Use this command to tell you the default time to live setting for sessions. All sessions created will have this length of time before they expire and need to create a new session table entry. Scope: Global
Output Example
FGT # get system session-info list session info: proto=17 proto_state=00 duration=4685 expire=174 timeout=0 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=4 origin-shaper= reply-shaper= per_ip_shaper= ha_id=0 hakey=30944 policy_dir=0 tunnel=/ state=local may_dirty rem statistic(bytes/packets/allow_err): org=39032/119/1 reply=0/0/0 tuples=2 orgin->sink: org pre->in, reply out->post dev=2->9/9->2 gwy=255.255.255.255/0.0.0.0 hook=pre dir=org act=noop 172.20.120.225:68>255.255.255.255:67(0.0.0.0:0) hook=post dir=reply act=noop 255.255.255.255:67>172.20.120.225:68(0.0.0.0:0) misc=0 policy_id=0 id_policy_id=0 auth_info=0 chk_client_info=0 vd=0 serial=0002dcfe tos=ff/ff ips_view=0 app_list=0 app=0 dd_type=0 dd_rule_id=0 per_ip_bandwidth meter: addr=172.20.120.225, bps=112
Description The protocol this packet contains. Common protocol numbers include 17 (UDP), 6 (TCP), 47(GRE), 89(OSPF), and 1 (ICMP). See Assigned Internet Protocol Numbers.
160
Keyword/Variable proto_state=00
Description proto_state has two digits to keep track of the original direction and the reply direction. There are different states for each of TCP, SCTP, and UDP. ICMP traffic has no state and will always show as proto_state=00. For example if a TCP session has proto_state=67, the originator has closed the session, and the responder is waiting for the last ACK or halfclose timer to expire. TCP packet states 0 - none 1 - ESTABLISHED 2 - SYN_SENT 3 - SYN & SYN/ACK 4 - FIN_WAIT 5 - TIME_WAIT 6 - CLOSE 7 - CLOSE_WAIT 8 - LAST_ACK 9 - LISTEN UDP packet states 0 - Reply Not Seen 1 - Reply Seen SCTP packet states 0 - SCTP_S_NONE 1 - SCTP_S_ESTABLISHED 2 - SCTP_S_CLOSED 3 - SCTP_S_COOKIE_WAIT 4SCTP_S_COOKIE_ECHOED 5SCTP_S_SHUTDOWN_SENT 6SCTP_S_SHUTDOWN_RECD 7 - SCTP_S_ACK_SENT 8 - SCTP_S_MAX
duration=4685 expire=174
The number of seconds since this session was created. The number of seconds until this session expires. GRE sessions have a very large expire value. This is to prevent them from timing out, and dropping the GRE session.
timeout=0
If there is a timer that will expire, this is the number of seconds until that timer expires. One example is the standard session ttl timer is 3600 seconds. Description The traffic shapers applied to traffic from the origin, from the responder (reply), or any per IP traffic shapers. The value is the name of the shaper that has been applied. The cluster ID. in an HA cluster. If this session is over a tunnel, this displays information about the tunnel.
161
Description The status of the session. Possible states include: br ext log local may_dirty Session is being bridged, that is, in transparent mode. Session is created by a firewall session helper. Session is being logged Session is originating from, or destined for, a local stack. Session is created by a policy. For example, the session for FTP channel control will have this state but the FTP data channel will not. Session will be checked by an IPS signature. Session will be checked by an IPS anomaly. Session will possibly be offloaded to NPU. Session is handled by WCCP.
Description misc displays the security policys ID. Each security policy has an identifier, such as 20004. If this session has a UTM profile applied to it, misc indicates which one it is. vd is the VDOM this security policy applies to. The ID number of the security policy that matches this session. If an authentication policy is applied, this displays the ID of that policy. For example an IBP would be listed here.
policy_id=0 auth_info=0
statistic(bytes/packets/allow_err): org=39032/119/1 reply=0/0/0 The bytes, packets, and errors sent from the original direction. The bytes, packets, and errors sent from the reply direction.
A counter that rolls over. This value is the same for parent and child sessions. Type of Service (TOS) from the TOS field in the packet header. The values apply to origin and reply directions accordingly. Data re-duplication (dd) values. These are associated with WAN Opt. Rule ID matches the WAN Opt rule number.
per_ip_bandwidth meter: addr=172.20.120.225, bps=112 The IP address this meter applies to. The bytes per second bandwidth for this IP address.
162
Syntax
get system session-info ttl
Parameters
None.
Usage/Remarks
Use this command to tell you the default time to live setting for sessions. All sessions created will have this length of time before they expire and need to create a new session table entry. A similar command is get system session-ttl . It provides similar output. Scope: Vdom
Output Example
FGT # get system session-info ttl default : 3600 port: Keyword/Variable default port Description The default is one hour or 3600 seconds. The session port.
163
Syntax
get system startup-error-log
Parameters
None.
Usage/Remarks
Use this command to view the start-up configuration errors on the console. If there are no errors, this command displays the http-err webproxy replacement message settings. Scope: Global and Vdom
Output Example
FGT # get system startup-error-log >>> config system replacemsg webproxy "http-err" >>> set buffer "<html><head><title>%%HTTP_ERR_CODE%% %%HTTP_ERR_DESC%%</title></head><body><font size=2><table width=\"100%\"><tr><td bgcolor=#3300cc align=\"center\" colspan=2><font color=#ffffff><b>%%HTTP_ERR_CODE%% %%HTTP_ERR_DESC%%</b></font></td></tr></table><br><br>The webserver for %%PROTOCOL%%%%URL%% reported that an error occurred while trying to access the website. Please click <u><a href=\"javascript:history.back()\">here</a></u> to return to the previous page.<br><br><hr></font></body></html>" >>> set header http >>> set format html
164
Syntax
get system status
Parameters
None.
Usage/Remarks
Use this command to display basic information about your FortiGate unit such as firmware version, serial number, hostname, number of VDOMs, HA mode, and system time. Scope: Global and Vdom
Output Example
FGT # get system status Version: Fortigate-3810A v4.0,build0418,110209 (Interim) Virus-DB: 11.00782(2010-05-07 00:42) Extended DB: 1.00001(2010-05-21 13:37) IPS-DB: 2.00910(2010-12-02 17:49) FortiClient application signature package: 1.393(2011-06-06 17:13) Serial-Number: FG3K8A3407600241 BIOS version: 04000009 Log hard disk: Not available Hostname: FG3K8A3407600241 Operation Mode: NAT Current virtual domain: root Max number of virtual domains: 10 Virtual domains status: 1 in NAT mode, 0 in TP mode Virtual domain configuration: disable FIPS-CC mode: disable Current HA mode: standalone Distribution: International Branch point: 418 Release Version Information: Interim System time: Tue Jun 7 14:40:16 2011
165
get test
get test
Displays information statistics, control proxies, and status.
Syntax
get test <application> <level>
Parameters
application acd Aggregate Controller ddnscd ddnscd daemon dhcprelay dhcprelay dnsproxy dns proxy ftpd ftp proxy http http proxy im im proxy imap imap proxy ipldbd ipldbd daemon ipsengine ips sensor ipsmonitor ips monitor l2tpcd l2tpcd nntp nntp proxy pop3 pop3 proxy proxyacceptor proxy acceptor proxyworker proxy worker scanunit scanning unit sflowd sflowd smtp smtp proxy snmpd snmpd daemon urlfilter urlfilter daemon vs virtual-server wad wan optimization proxy wccpd -wccp daemon <level> 1: Dump Memory Usage 2: Drop all connections 22: Drop max idle connections 222: Drop all idle connections 4: Display connection stat 44: Display info per connection
166
get test
444: Display connections per state 4444: Display per-VDOM statistics 44444: Display information about idle connections 5: Toggle AV Bypass mode. Toggle AV bypass mode. You can use this level to diagnose AV scanning. When bypass mode is activated, no AV scanning is done on traffic handled by the proxy. Note: Antivirus scanning is disabled. 6: Toggle Print Stat mode every ~40 seconds 7: Toggle Backlog Drop 8: Clear stats 88: Toggle statistic recording stats cleared 9: Toggle Accounting info for display 99: Restart proxy. When you suspect abnormal behavior of the proxy, you can use this level value to restore it to its normal state. Note: You will have a disruption in services. 11: Display the SSL session ID cache statistics 12: Clear the SSL session ID cache statistics 13: Display the SSL session ID cache 14: Clear the SSL session ID cache Note: Not all level numbers may be applicable to all applications. Use the command diagnose test application <application> 0 to see a list of valid commands.
Usage/Remarks
Use this command to display information about advanced FortiOS applications such as SSL, and VoIP. Scope: Global
Output Examples
FGT # get test ftpd 1 HTTP Proxy Test Usage Fortigate-1000# malloc memory usage ================== pool buffer size: 2048 count: 2143 available: 2139 total memory used by buffer pools: 4286 (kB) total allocated (malloc/realloc) memory: 0 (kB) shm memory usage ================== buffer pool not initialized
FortiOS Handbook v3: Troubleshooting 01-431-0129304-20120124 http://docs.fortinet.com/
167
get test
total allocated memory 0 FGT # get test http 44 diagnose test application http 44 id=17 clt=914(r=1, w=0) srv=915(r=1, w=0) c:192.168.50.2:3608 -> s:192.168.135.21:80 c2s/s2c=0/0 state=RESPONSE_PASS_STATE duration=0 expire=3590 Current connections = 1/2368 Keyword/Variable 44 Description The first number is the counter of current sessions in the connection pool of the proxy. The second number is the maximum size of the proxy connection pool table.
FG # get test imap 4 Current connections = 5/2670 Fortigate-1000# Running time (HH:MM:SS:usec) = 22:45:40:124613 Bytes sent = 65 (kb) Bytes received = 2895 (kb) Error Count (alloc) = 0 Error Count (accept) = 0 Error Count (bind) = 0 Error Count (connect) = 0 Error Count (read) = 0 Error Count (write) = 0 Error Count (poll) = 0 Last Error = 0 Scan Backlog drop = 0 Emails clean = 3 Emails detected = 2 Emails with scan errors = 0 Worms = 0 Blocked = 0 Virus = 2 Suspicious = 0 Fragmented emails = 0 Spam Detected = 0 Content Filtered emails = 0 Oversize Email Pass = 0 Oversize Email Blocked = 0 AV Bypass is off Print is off Drop on backlog is off Account is on setup_ok=7 setup_fail=0 poll_ok=2576/2576/1 sel_fail=0 conn_ok=0 conn_inp=7 step1=0 step2=0 scan=5 listen=7 cmdb=2 clt=2304 srv=258 FG # get test ipsengine 7 FG # PACKET STATISTICS:
Troubleshooting for FortiOS 4.0 MR3 01-431-0129304-20120124 http://docs.fortinet.com/
168
get test
total packets 3487023 tcp packets 87 udp packets 0 icmp packets 2476747 discard packets 0 alert packets 45 log packets 0 pass packets 0 fragment packets 0 frag_trackers 0 rebuilt_frags 0 frag_incomplete 0 frag_timeout 0 rebuild_element 0 frag_mem_faults 0 tcp_stream_pkts 87 rebuilt_tcp 5 tcp_streams 4 rebuilt_segs 0 str_mem_faults 0 FG FG FG FG # # # # get test ipsmonitor 2 enable ipsengine? no diagnose test ipsmonitor 2 enable ipsengine? Yes
FG # get test pop3 444 FG # [OVERSIZE_STATE ] 1/1 FG # get test smtp 44 FG # id=0 clt=10(r=1, w=0) srv=11(r=1, w=0) c:192.168.200.2:60811 -> s:192.168.50.2:25 c2s/s2c=0/0 state=CONNECTED_STATE duration=0 expire=3581 Current connections = 1/2669
169
Syntax
get test application urlfilter <number>
Parameters
<number> 1 This menu 2 Clear cache 3 Display WF cache contents 4 Display WF cache TTL list 5 Display WF cache LRU list 6 Display WF cache in tree format 7 Toggle switch for dumping unrated packet 8 Increase timeout for polling 9 Decrease timeout for polling 10 Print debug values 11 Clear Spam Filter cache 12 Clear AV Query cache 13 Toggle switch for dumping expired license packets 14 Show running timers (except request timers) 144 Show running timers (including request timers) 15 Send INIT requests 16 Display WF cache contents of prefix type 99 Restart the urlfilter daemon
Usage/Remarks
Use this command to troubleshoot and work with the URL filter engine. If you do not include one of the numbers listed above, the list of numbers will be displayed for you. Scope: Global
Output Example
FGT # get test urlfilter 3 utree_stat: keylen=223 nodes=15 leaf=11 com.cisco.www cate = 52 len=13 ch=1 dhtml_pulldown/dropdownlib-100.js cate = 52 len=33 ch=0 niffer/snifflib-100.js cate = 52 len=22 ch=0 potlight/spotlightlib-120.js cate = 52 len=28 ch=0 offer/sp/cookie.js cate = 52 len=18 ch=0 cisco_detect.js cate = 52 len=15 ch=0 flyouts.js cate = 52 len=10 ch=0 global.js cate = 52 len=9 ch=0 hbx.js cate = 52 len=6 ch=0
Troubleshooting for FortiOS 4.0 MR3 01-431-0129304-20120124 http://docs.fortinet.com/
170
171
Syntax
get vpn ipsec stats crypto
Parameters
None.
Usage/Remarks
Use this command to display which crypto devices are used with VPN connections such as 3DES, SHA1, and so on. A zero indicates that type of crypto is not used with any VPN connections. This command can be useful to help enforce the use of more secure crypto. In the output, you should be aware that there is hardware (CP6) and software crypto categories. There are also separate columns for encrypted (outbound) and decrypted (inbound), or generated (outbound) and validated (inbound). Scope: Vdom
Output Example
FG10CH3G09603750 # get vpn ipsec stats crypto IPsec crypto devices in use: CP6 (encrypted/decrypted): null: 0 0 des: 0 0 3des: 0 0 aes: 0 0 CP6 (generated/validated): null: 0 0 md5: 0 0 sha1: 0 0 sha256: 0 0 SOFTWARE (encrypted/decrypted): null: 0 0 des: 0 0 3des: 0 0 aes: 0 0 SOFTWARE (generated/validated): null: 0 0 md5: 0 0 sha1: 0 0 sha256: 0 0
172
Syntax
get vpn ipsec stats tunnel
Parameters
None.
Usage/Remarks
Use this command to display information about IPsec VPN tunnels including how many of each type (static, dynamic, or manual addressing), how many errors there have been, and how many selectors there are. Scope: Vdom
Output Example
FG10CH3G09603750 # get vpn ipsec stats tunnel tunnels total: 1 static/ddns: 1 dynamic: 0 manual: 0 errors: 0 selectors total: 1 up: 1
173
Syntax
get vpn ipsec tunnel details
Parameters
None.
Usage/Remarks
Use this command to display in depth information about IPsec VPN tunnels. This command is useful when troubleshooting VPN tunnels due to the amount of information displayed. Scope: Vdom
Output Example
FG10CH3G09603750 # get vpn ipsec tunnel details gateway name: 'ph1' type: policy-based local-gateway: 172.16.68.34:0 (dynamic) remote-gateway: 172.16.68.35:0 (static) mode: ike-v1 interface: 'port4' (12) rx packets: 0 bytes: 0 errors: 0 tx packets: 0 bytes: 0 errors: 0 dpd: enabled/negotiated idle: 5000ms retry: 3 count: 0 selectors name: 'ph2' auto-negotiate: enable mode: tunnel src: 0:0.0.0.0/0.0.0.0:0 dst: 0:0.0.0.0/0.0.0.0:0 SA lifetime/rekey: 180/123 mtu: 1436 tx-esp-seq: 1 replay: enabled inbound spi: 585e89b2 enc: 3des 40a50330e01e36fbb58cbf26f2d77e4e57fc5c1fc41d20ec auth: sha1 1f1292da090bde71f955f4e8cb23d4e0c7768a8e outbound spi: efd4667a enc: 3des 66bfeaf3fcd635d1e309be18a8eea41d53630371590ff8e0 auth: sha1 c7ec4ac4632a22785b713644a4d1581269b29788 SA
174
lifetime/rekey: 180/120 mtu: 1436 tx-esp-seq: 1 replay: enabled inbound spi: 585e89b1 enc: 3des 69e72f1bf0a0cd76aded631c0010f17e5d1acf558dad0329 auth: sha1 ed96712db7e633bfa9720b2bf7a276d198b7661a outbound spi: efd4667b enc: 3des 28d49a7e392487e6078907bac49560e00b21428a5bfcf8be auth: sha1 84eb5525df4511a36c3ac291ff5526bbf0be9ef2
175
Syntax
get vpn ipsec tunnel summary
Parameters
None.
Usage/Remarks
Use this command to display brief information about IPsec VPN tunnels including their name, IP address and port, how many selectors that tunnel has, and packets received, sent, and in error. This information is displayed for each tunnel defined. Scope: Vdom
Output Example
FG10CH3G09603750 # get vpn ipsec tunnel summary 'ph1' 172.16.68.35:0 selectors(total,up): 1/1 rx(pkt,err): 0/0 tx(pkt,err): 0/0
176
Syntax
get vpn status ssl hw-acceleration-status
Parameters
None.
Usage/Remarks
Use this command to display the status of SSL hardware accelerated VPN connections. This is useful if you need to troubleshoot a VPN connection that should be accelerated but is not. Scope: Vdom
Output Example
FGT # get vpn status ssl hw-acceleration-status Acceleration hardware detected: kxp=on cipher=on
177
Syntax
get vpn status ssl list
Parameters
None.
Usage/Remarks
Use this command to display the status of SSL list of VPN connections. Scope: Vdom
Output Example
FGT # get vpn status ssl list SSL VPN is disabled.
178
Syntax
get webfilter ftgd-statistics
Parameters
None.
Usage/Remarks
Use this command to display many details about your FortiGuard statistics. These numbers can be useful in determining the source of a problem. For example DNS failures can reveal the source of problems that may be widespread and hard to track back to the source. Scope: Global
Output Example
FGT # get webfilter ftgd-statistics Rating Statistics: ===================== DNS failures : DNS lookups : Data send failures : Data read failures : Wrong package type : Hash table miss : Unknown server : Incorrect CRC : Proxy request failures : Request timeout : Total requests : Requests to FortiGuard servers : Server errored responses : Relayed rating : Invalid profile : Allowed Blocked Logged Errors Cache Statistics: ===================== Maximum memory Memory usage Nodes Leaves Prefix nodes Exact nodes : : : :
3 16 876 0 0 1 0 0 0 292 0 0 11 0 0 0 0 0 0
: : : : : :
0 0 0 0 0 0
179
Requests Misses Hits Prefix hits Exact hits No cache directives Add after prefix Invalid DB put DB updates Percent full Branches Leaves Prefix nodes Exact nodes Miss rate Hit rate Prefix hits Exact hits Keyword/Variable DNS lookups Data send failures Request timeout Total requests Requests to FortiGuard servers Relayed rating Description
: : : : : : : : : : : : : : : : : : 0 0
0 0 0
0 0 0 0 0% 0% 0% 0% 0% 0% 0% 0% 0%
Number of DNS look-ups for the domain name of the requested URL. Number of non-responsive servers. Number of seconds for the request time-out. A FortiGate unit sends the URL rating request every 2 seconds. Total number of URL rating requests to cache and FortiGuard servers. Total number of URL rating requests to FortiGuard servers.
The number of times the master communicates with FortiGuard servers and relays all URL rating requests from the slaves in a HA cluster. Amount of memory assigned to FortiGuard cache. The default is 2 percent. The amount of memory used to store the URL tree. The number of prefixes used for a URL to increase the cache hit rate. The number of exact matches used for a URL.
180
Syntax
get webfilter status <refresh rate>
Parameters
<refresh rate> How often to refresh the server list(s).
Usage/Remarks
Use this command to display webfilter statistics and server information if the service is enabled. If the service is not enabled, this command displays the language of the locale and states the service is not enabled. This command was previously called diag debug rating. Scope: Global and Vdom
Output Example
FGT # get webfilter status 4 Locale : english License : Contract Expiration : Wed Feb 11 02:00:00 2009 Hostname : service.fortiguard.net -=- Server List (Mon May 26 22:36:34 2008) -=IP Weight RTT Flags 212.95.252.121 10 77 62.209.40.73 0 86 62.209.40.72 0 92 DI 82.71.226.65 10 98 D 212.95.252.120 10 95 69.20.236.179 60 198 66.117.56.42 60 189 66.117.56.37 60 192 69.20.236.180 60 213 209.52.128.90 60 268 121.111.236.179 80 383 121.111.236.180 80 404 72.52.72.243 90 289 218.106.244.81 90 455 69.90.198.55 90 297 D TZ Packets Curr Lost Total Lost 0 83 0 8 1 83 0 8 1 85 0 9 0 84 0 8 0 76 0 2 -5 84 0 9 -5 83 0 8 -5 83 0 8 -5 83 0 8 -5 84 0 9 9 83 0 8 9 83 0 8 -8 83 0 8 -8 83 0 8 -8 85 0 9
181
Description Local environment language. The license status: Contract Expired Trial
Expiration Hostname
The date and time the license expires. The FortiGuard server the FortiGate unit connects to obtain the service. The FortiGuard server will return the information to the FortiGate unit. The default is service.fortiguard.net The IP address of other FortiGuard servers. The priority value which the FortiGate unit uses to send the URL rating request. The lower weight value takes higher preference. The weight is calculated by time zone, packet round-trip time, and success rate. The round-trip time between the URL rating request and the response time from the FortiGuard server. Time zone of the FortiGuard server (Greenwich Mean Time +/the number). Total packets sent to the FortiGuard server. The number of times the request is retried in a timeout period. The default is 15 seconds. Total number of unresponsive requests.
IP Weight
182
FortiOS Handbook
183
C. You have problems with the console text D. You have visible power problems E. You have a suspected defective FortiGate unit
184
FSSO issues
6 If No, ensure you serial communication parameters are set to no flow control, and the proper baud rate and reboot the FortiGate unit by powering off and on. FortiGate units ship with a baud rate of 9600 by default. If you have access, parse an archived configuration file for the term baudrate or verify this setting with the CLI command:
config system console get
7 Did the reboot fix the problem? If that fixes your problem, you are done. If that does not fix your problem, go to E. You have a suspected defective FortiGate unit
FSSO issues
The following is a flow chart for troubleshooting FSSO issues. A. Initial information gathering B. The CA is not running and not connected C. The CA is running but not connected D. The CA is connected E. There are at least some users logged on F. Test user does not appear on the FSSO list
185
FSSO issues
D. The CA is connected
1 Are you seeing groups on the FortiGate? If not, check the group filter on the CA. 2 Are the FortiGate and CA groups using the same mode? If not, change the modes to match. 3 Are you seeing logon events on the FortiGate unit? You can check this with the following CLI commands. diagnose debug enable diagnose debug authd fsso list If there are any users logged in, go to step . Otherwise, continue on. 4 Are DC agents installed on all Domain Controllers? If not install the DC Agents.
186
FSSO issues
5 Are you using an LDAP server on the FSSO connector? To check go to User > Directory Service > Edit FSSO connector > LDAP. If an LDAP server is configured, disable it and go to step 3. If there is no LDAP server configured, contact support and open a support ticket.
187
FSSO issues
Check which domain controller authenticated the host (run echo %logonserver% on the host) and troubleshoot that domain controller.
Does the logon server have If not, install the DC agent. the DC agent installed? If it is installed, enable logging on the DC agent on the logon server. Use the logs produced for farther troubleshooting.
188
FortiOS Handbook
Appendix
Document conventions
Fortinet technical documentation uses the conventions described below.
IP addresses
To avoid publication of public IP addresses that belong to Fortinet or any other organization, the IP addresses used in Fortinet technical documentation are fictional and follow the documentation guidelines specific to Fortinet. The addresses used are from the private IP address ranges defined in RFC 1918: Address Allocation for Private Internets, available at http://ietf.org/rfc/rfc1918.txt?number-1918. Most of the examples in this document use the following IP addressing: IP addresses are made up of A.B.C.D A - can be one of 192, 172, or 10 - the non-public addresses covered in RFC 1918. B - 168, or the branch / device / virtual device number. Branch number can be 0xx, 1xx, 2xx - 0 is Head office, 1 is remote, 2 is other. Device or virtual device - allows multiple FortiGate units in this address space (VDOMs). Devices can be from x01 to x99. C - interface - FortiGate units can have up to 40 interfaces, potentially more than one on the same subnet 001 - 099- physical address ports, and non -virtual interfaces 100-255 - VLANs, tunnels, aggregate links, redundant links, vdom-links, etc.
189
Document conventions
Appendix
D - usage based addresses, this part is determined by what device is doing The following gives 16 reserved, 140 users, and 100 servers in the subnet. 001 - 009 - reserved for networking hardware, like routers, gateways, etc. 010 - 099 - DHCP range - users 100 - 109 - FortiGate devices - typically only use 100 110 - 199 - servers in general (see later for details) 200 - 249 - static range - users 250 - 255 - reserved (255 is broadcast, 000 not used) The D segment servers can be farther broken down into: 110 - 119 - Email servers 120 - 129 - Web servers 130 - 139 - Syslog servers 140 - 149 - Authentication (RADIUS, LDAP, TACACS+, FSAE, etc) 150 - 159 - VoIP / SIP servers / managers 160 - 169 - FortiAnalyzers 170 - 179 - FortiManagers 180 - 189 - Other Fortinet products (FortiScan, FortiDB, etc.) 190 - 199 - Other non-Fortinet servers (NAS, SQL, DNS, DDNS, etc.) Fortinet products, non-FortiGate, are found from 160 - 189. The following table shows some examples of how to choose an IP number for a device based on the information given. For internal and dmz, it is assumed in this case there is only one interface being used. Table 8: Examples of the IP numbering Location and device Head Office, one FortiGate Head Office, second FortiGate Branch Office, one FortiGate Office 7, one FortiGate with 9 VDOMs Internal 10.011.101.100 10.012.101.100 10.021.101.100 10.079.101.100 Dmz 10.011.201.100 10.012.201.100 10.021.201.100 10.079.101.100 10.031.201.110 n/a External 172.20.120.191 172.20.120.192 172.20.120.193 172.20.120.194 n/a n/a
Office 3, one FortiGate, web n/a server Bob in accounting on the corporate user network (dhcp) at Head Office, one FortiGate Router outside the FortiGate 10.0.11.101.200
n/a
n/a
172.20.120.195
190
Appendix
Document conventions
Linux PC 10.11.101.20
IN 10 T .11 .10
1.1
01
10
.11
FortiAnalyzer-100B
Switch
10
.11
.14
Po an rt 2 d3 Po rt 1
Switch
f rt 8 r o Po mirro (
rt po
s2
an
d3
H ea d of fic e
01
Linux PC 10.21.101.10
17
2.2
0.1
B ra nc h of fic e B ra nc h
of fic e
1. rt 1 10 Po 0.21. 1
16
Windows PC 10.31.101.10
FortiManager-3000B
10
.2
rt 4 Po .100 01 1 . 2
Cluster
Port 1: 10.21.101.102
FortiGate-5005FA2
Port 1: 10.21.101.102
FortiGate-5005FA2
Port 1: 10.21.101.103
FortiSwitch-5003A
Port 1: 10.21.101.161
FortiGate-5050-SM
Port 1: 10.21.101.104
191
Document conventions
Appendix
Information highlights
A Must Read item details things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled 'Important' will not cause data loss but may cause irritation and frustration.
A Troubleshooting tip provides information to help you track down why your configuration is not working.
A Tip provides shortcuts or alternative approaches to the task at hand. Ignoring a tip should have no negative consequences, but you might miss out on a trick that makes your life easier.
Typographical conventions
Table 9: Typographical conventions in Fortinet technical documentation Convention Button, menu, text box, field, or check box label CLI input Example From Minimum log level, select Notification.
config system dns set primary <address_ipv4> end FGT-602803030703 # get system settings comments : (null) opmode : nat HTTP connections are not secure and can be intercepted by a third party. <HTML><HEAD><TITLE>Firewall Authentication</TITLE></HEAD> <BODY><H4>You must authenticate to use this service.</H4> Visit the Fortinet Technical Support web site, https://support.fortinet.com. Type a name for the remote VPN peer or client, such as Central_Office_1. Go to VPN > IPSEC > Auto Key (IKE). For details, see the FortiOS Handbook.
File content
192
Appendix
193
Appendix
Most web-based manager numeric value fields make it easy to add the acceptable number of digits within the allowed range. CLI help includes information about allowed numeric value ranges. Both the web-based manager and the CLI prevent you from entering invalid numbers.
Training
Fortinet Training Services offers courses that orient you quickly to your new equipment, and certifications to verify your knowledge level. Fortinet provides a variety of training programs to serve the needs of our customers and partners world-wide. Visit the Fortinet Training Services web site at http://campus.training.fortinet.com, or email training@fortinet.com.
Technical Documentation
See the Fortinet Technical Documentation web site, http://docs.fortinet.com, for the most up-to-date technical documentation. The Fortinet Knowledge Base provides troubleshooting, how-to articles, examples, FAQs, technical notes, and more. Visit the Fortinet Knowledge Base at http://kb.fortinet.com.
194
Appendix
195
Appendix
196
FortiOS Handbook
Index
A
accelerated interfaces, 96 Administrative Status, 84 AMC (Advanced Mezzanine Card), 177 anti-spoofing, 94 ARP cache, 57 duplicate packets, 93 resolution, 95 asymmetric routing, 94 av-failopen, 97
F
FIN packet, 158 firewall session setup rate, 46 flow inspection, 15, 16 FortiASIC, 96 FortiGuard Antivirus, 194 services, 194 FortiGuard Distribution System (FDS), 60 Antispam, 11 Antivirus, 11 servers, 62 Fortinet Technical Documentation, conventions, 189 Technical Support, 195 Technical Support, registering with, 194 Technical Support, web site, 194 Training Services, 194 Fortinet customer service, 195 Fortinet documentation, 194 forward domain, 93 FSSO ports, 186
B
Berkeley Packet Filtering (BPF), 52 Bootup issues, 183 brctl,netlink, 93 bridge, Trasnparent mode, 93
C
certification, 194 CLOSE_WAIT, 158 collision domain, 93 connectionless, 14 conserve mode, 97 no SYN-ACK, 41 conventions, 189 CPU usage, 86 Cross-Site Scripting protection from, 193 customer service, 195
G
get system performance status, 86 top, 86 get test app, 81 global, 47
H
HTTP proxy, 156
D
date, 34, 58 dead gateway detection, 114 dead peer detection (dpd), 114 debug flow, 96 default password, 11 Denial of Service (DoS), 18 diag netlink brctl, 81 diagnose commands, 102, 117 diag debug, 96 diag netlink, 93 documentation conventions, 189 Fortinet, 194 domain name server (DNS), 87 Duplicate ARP packet, 93
I
ICAP, 16 identify-based policies, 20 initiator, 114 inspection flow, 15, 16 proxy, 16 security layers, 17 stateful, 13 interface accelerated NP2, 96 link status, 84 pairs, 96 Internet Control Message Protocol (ICMP), 88 Internet Traffic Management Practices (ITMP), 103
197
Index
introduction Fortinet documentation, 194 IP address private network, 189 IP stack validation, 18
powering on, 183 problem scope, 28 product registration, 194 proxy inspection, 16
L
layer 4, 18 Layer-2, 93 layer-2 switch, 81 Layer-3, 88 layer-3 router, 81 LDAP, 59 life of a packet, 13 UDP, 13 link status, 84 Linux, 89, 91 logging alert email did not send, 116 cannot log to log device, 116 FortiGate stopped recording logs, 116
R
RADIUS, 59 registering with Fortinet Technical Support, 194 Return Material Authorization (RMA), 73 Reverse Path Forwarding (RPF), 94 RFC 1918, 189 Round Trip Time (RTT), 62 routing bridge, 93 routing table, 19
S
security association (SA), 114 security layers, 17 serial port parameters, 184 session ACCEPT queue full, 159 ephemeral, 157 half-closed, 158 half-open, 158 SYN queue full, 159 TCP state, 158 timewait, 158 Session creation, 18 session helper, 20 session pick-up, 149 session tables, 20 signature-based IPS, 19 Single instruction, multiple data (SIMD), 128 sniffer, verbosity level, 95 ssl.root, 20 stateful inspection, 13, 94 stateless, 13 SYN_SENT, 158 SYSLOG, 59 system resources, 86
M
MAC table, 93 memory usage, 86 middle-man, 16 MS Windows, 90
N
netlink, 93 network interface card (NIC), 129 Network Time Protocol (NTP), 34, 59 no SYN-ACK, 41 non-conserve mode, 41 NP2 interface, 96
O
OSI Layer-2, 93 Layer-3, 88
P
packet flow, 17, 96 life of, 13 sniffer, 94 Packet verification, 18 password administrator, 11 ping, 88 ports 8000 and 8002, 186 port 1024, 63 port 1025, 63 port 443, 95 port 53, 63 port 8888, 63 UDP ports 33434-33534, 90
T
TCP header flags, 13 TCP SYN packets, 19 TCP/IP stack, 20 technical documentation conventions, 189 support, 195 technical support, 195 Technology Assistance Center (TAC), 30 time, 34, 58 time to live (TTL), 89, 163 TIME_WAIT, 158 tracert (traceroute), 89, 90 traffic shaping, 103
198
Index
Training Services, 194 troubleshooting, 77 alert email did not send, 116 cannot log to log device, 116 debug packet flow, 96 diagnose commands, 102, 117 firewall session list, 101 FortiGate stopped logging, 116 packet sniffing, 94 ping, 87 routing table, 92 traceroute, 87 traffic shaping, 103
Verifications of IP options, 18 vpn error no SA proposal, 114 initiator, 114 P1 proposal, 114 R U THERE, 114 vulnerability Cross-Site Scripting, 193 XSS, 193
W
Wireshark, 82
U
UDP, 13
X
XSS vulnerability protection from, 193
V
VDOM, 31, 46, 47, 100, 105, 119
199