What this Book ​is all about

This book is a follow up to “Fortigate Admin Pocket Guide“ and

“Fortigate Security Pocket Guide” :
Following The basic administration and
applying security profiles, we will now start to
diagnose our FortiGate for troubleshooting
purposes and analyzing what is really
happening on your network

“Fortigate Diagnostics pocket guide”​, will

walk you through the most important
diagnostics commands in different use cases

Every chapter includes hands-on practices

It is written for​ beginners and intermediate users

The following book will help you to diagnose and analyze
sessions, policies, interface issues, network congestions,
and more
Table Of​ ​Contents

What this Book is all about 1

Table Of Contents 3
General Connectivity issues 5
Ping and Traceroute 6
Packet Capture 14
TCP Dump Way 20
Using Filters 25
Examine TCP with the SYN flag on 31
Interfaces, Sessions, Services 35

DNS Settings 41
DHCP Settings 45
Diagnose Your ARP table 52
Diagnosing the Routes 55
Performance Issues 58
System and user Processes 63
Terminating Processes 65
CPU High Usage Stitch 71
Conserve Mode 77
Diagnose Sessions 80
Session List 84
Reduce session time 89
Debug Packet Flow 91
Final Words 98
Knowing how to diagnose your FortiGate is probably one of the most important
tools that you can acquire as a FortiGate professional.
It will make you aware of what is happening on your network, on your FortiGate
kernel, services, and much more. this skill set is unique and the mindset that you
will acquire will serve you not only on your firewall
We will start with a low-level view of our FortiGate traffic, moving on to
General network issues, system performance, and from there to sessions
and packet flow view
General Connectivity ​issues
Ping and ​Traceroute

Before we enter packet capturing and debugging the packet flow, one of the first
steps, that you will want to do in case of connectivity issues, the following 2
commands

“exe ping<destination>
“exe traceroute <destination>

The two will verify connectivity between 2 points, but will also help you to verify
that the correct route is used. so let’s look at them
On your CLI, type: ​“execute ping google.com”
Whenever a Ping is being from your Fortigate, The default settings are:
● One second interval between Pings
● A two-second timeout
● 5 Pings are sent
● The size of the is 56 bytes, not including 8-byte headers, which makes it 64
bytes

Our output shows us that the packets arrive at their destination if there is a
packet loss, how many packets were sent, the size of the ping packet, we can
also, see that the latency ( which can become an issue, if it is too high ) is O.K

But let’s assume, that we aren’t getting an answer, from our Ping request, the
first thing, that should come to your mind is “ IS there a DNS resolve issue”
To check that, we will use a known IP address of Google's DNS server
If the packets do arrive at their destination, then yes, you have a DNS issue, if
not, the problem is at another place

You can with different Ping options using the command

“execute ping-options” ​followed by a question mark
You can play around with the different options, you can view the current settings
using the view-settings

Let's look at the most important ones:

Interval​ - Once an ICMP response, a new ICMP packet is sent. Currently, the
interval between the two is 1 second
Adaptive​ - on the contrary to the interval, once you choose ​adaptive​, then your
ICMP packets will be sent immediately as the last returns. so if we use adaptive,
you will see, a major change in the pace of pings sent
Now let’s change our interface and send All ICMP packets, from the LAN
interface towards our gateway

For that, we will use the source option and IP address of our interface . ​that can
help to check connectivity issues in different interfaces on your FortiGate
Using​ ​Traceroute

To track real-time paths ( routes ) from source to destination, we use traceroute,

a known network diagnostic tool, that is available on your FortiGate CLI

Traceroute uses ICMP packets to make sure that the traffic is following the right
path and that the correct route is being used. it will report the different IP
addresses of the different routers it pinged, and will also report, the time it took
for each hop

So let's open our CLI and start by typing ​“execute traceroute-options”

Let's look at the current settings

As you can see, there aren't many settings as in the ping command, but you can
modify the Source IP or interface for a traceroute

So let's set the source to 10.0.5.7 and look again at the settings

So now our source IP for traceroute is my Linux device on my LAN

Traceroute will help you to discover any DNS resolve issues. If you enter a
domain name, it will try to resolve, if it won’t succeed, then you will know that you
have a DNS issue

So let’s traceroute our ​Fortigate​ to cnn.com

If traceroute is having issues with packets that do not return within the expected
timeout window, then an ​asterisk (*) character ​ will show up, which needs to be
checked
Packet ​Capture

One of the coolest features of your FortiGate is its ability to packet capture
any traffic that flows from different sources towards different destinations.
Packet capture will help you to analyze your network and troubleshoot issues as :

● Routing issues
● Is traffic arriving at your FortiGate on the expected port?
● Missing packets
● Broadcast storm
● Failure in ARP resolution - look for hosts that don’t respond to your
FortiGate arp requests

I have choose Packet Capture as the first technique to troubleshoot and

diagnose your network, as it is will become one of the most valuable tools,
that you will use often

You can capture packets using two ways. The first one is using the Graphical
User Interface.

Navigate to network---packet capture

Create New
And here you can select the interface that you wish to capture the packets that
flow through, you can use different filters, such as :
● Specific host on that interface
● Specific ports ( as ​Port 53​ for DNS Traffic, Port 443 for HTTPS, and so on
)
● VLAN’s on that physical interface
● And the protocol, which you wish to capture ( TCP, UDP…)

Choosing Protocol, is done using the different Protocol numbers, the main ones
are :
● TCP is protocol number 6
● UDP is protocol number 17
● ICMP is protocol number 1
So let’s capture traffic on our WAN interface, and open it using Wireshark
Press OK and you will get back to the packet capture page, where the message
not running appear

Right-click on the “not Running and a pop-up menu will list possible options

Click on ​“Start”
Wait a few seconds, and click again on ​Stop

You will see the message

Right-click again and you will have the option to ​download​ a Pcap ( packet
capture ) file

Double click on the Pcap file and it will open in ​Wireshark

If you want to practice and understand the underneath of TCP, HTTP, and other
protocols and services That are used extensively on your network, then
Wireshark is a must​. www.wireshark.org
TCP​ ​Dump Way
Up until now, we have Captured the traffic, downloaded the Pcap file, and
opened it in Wireshark. we can now look at the output and start to analyze what
is happening.
The second way of doing so is through the CLI.
The syntax is very similar to ​TCP Dump i​ n Linux, I personally find it much
quicker to analyze
.
Let's open up our CLI
The basic syntax for capturing packets is the following

Diagnose sniffer packet <interface><filter><verbosity level>

Following the ​“diagnose sniffer packet “​ which is the standard syntax that you
will use, every time that you wish to capture packets, we can choose a specific
interface ( as Port 2 or any other ) or use the ​“any”​ which means’ look for the
traffic on all interfaces

We can also use a ​filter​ on our command, we will do that very soon

The following number after the​ filter value ​is the​ verbosity ​level from 1-4, each
number represents the amount of data that will be shown
Diagnose sniffer packet <interface><filter><1-4>

● Number 1 ​Inputs the Plain Packets headers

In the screenshot, we can see the source IP​ ( 192.168.1.44 ) ​connection
to the ​192.168.184​ Destination IP address using port 80. The output also
includes the different TCP flags, indicating the connection status. TCP
sequence numbers are also present

Following a few lines, we can see that ​192.168.1.88​ is sending an ARP

request asking everyone in the broadcast domain, ​“who is
192.168.1.198”​, looking for its MAC address

To stop the capture press ​CTRL<C>

● Number 2 ​Inputs header and data from IP packets
Here we can see much more information, including the data from the IP
packets, again represented in hex format
● Number 3 ​Inputs header and data from Ethernet packets ( in hex )

● Number 4 ​Inputs header of packets with interface ( Port ) name

Next to each interface, you will also see, the direction of the traffic, which
tells you if the packet is entering or leaving the interface
Using ​Filters

The filter allows us much more granularity and defines what to look for in the
output as:
● Hosts
● Protocols
● Source
● Destination
● Ports

Let's examine the filter functionality and look for ICMP packets, from a host with
the ​10.0.5.7​ IP address that is connected to our LAN interface
Let’s ping from our ​Host

And now let's use the filter option

We can use different filters, but to capture traffic coming from 10.0.5.7 and ICMP
protocol, the syntax is
Diagnose sniffer packet port2 ‘src 10.0.5.7 and icmp’ 1
Now set the count of packets in the output, let's use 10 packets
<count>​ will appear after the verbosity level

You can play around with the different filters, but you're most
likely to use Source and Destination on your daily FortiGate
routines

You can also combine protocols together, using the ​or​ and the ​and​ statement.
“or”​ will display at least one of our choices
“and”​ will show both
so let's do that
We can see ​TCP and ICMP​ traffic only on our output

Let's continue, with​ DNS traffic​ only on port 53

We can trace ​ 192.168.1.44​ on port 1 connecting Google’s DNS server on port
53 over UDP and the answer coming in from ​8.8.8.8. everything looks good

We can even use​ hexadecimal byte position ​to look for specific data, such as
ARP packets
Or do it the regular way, using the arp keyword in our filter
Examine TCP with the ​SYN flag on

Another cool thing that you can do on your command line, is to view only TCP
Packets with the ​SYN​ flag on

This command is good if you wish to watch packets that are in the 3-way
handshake establishment process ( as ​TCP​ with the ​SYN​ flag ) or that are
terminating the session ( ​TCP​ Packet with the FIN flag )

In our filter, you can see that we are using TCP followed by the number ​13​ in
square parentheses. the number 13 indicates the specific SYN flag position in
the TCP packet ( octet number 13 which is equal to the decimal value 2 )
Lateral ​Movement

Our last packet capture will be done towards a device that is trying to probe our
Gateway using ​Nmap

Packet capture is not usually used to detect such behavior, for that you can use
the IPS signatures in the FortiGate DoS Policy, but it gives us a glimpse, of what
can be analyzed

Lateral movement behavior is when 2 devices on your local area network,

connect often, it is not so common, unless, there is a good purpose to do so.
Another reason could be that a compromised PC, is trying to get credentials from
an administrator PC

So on our ubuntu device, we will scan our gateway using ​Nmap full TCP scan
Which will initiate a full 3-way handshake with the destination target
And when we use the following syntax on our packet capture
We can see that ​10.0.5.7​ is sending syn request towards almost any available
port of our gateway ( on a full TCP scan, it actually sends probes to the most
1000 common ports ) and finding out which one is​ Open or Closed
Interfaces, ​Sessions, Services
If you are having issues with traffic that is not flowing from your FortiGate, then
this could be a hardware issue, routing issue, policy, and so on. Let’s try to
pinpoint the solution, and checkout of everything is O.K with our interfaces

The first thing is to check that our interface is up, we will check our WAN
interface at port 1

config systme interface

Edit port1
Set status up

If you find out that the status is down, that may be an issue of an unplugged
cable, so check your hardware physical connections
Next let’s look at our interface settings, using the “show” command

The​ show​ command will list the default configuration, you can also use the ​get
command to list all the available settings

Show system interface <port>

It seems that our IP address is in place, everything looks O.K

The next Diagnostic command is the get hardware Nic <interface>

diagnose hardware deviceinfo nic <interface>
This command is important, and it will list interface errors (as bad frames,
dropped frames, collisions ... )

Things to look at :

● MTU​ ( the largest packet size in bytes ) size​ -​ if not configured on purpose,
then your ​MTU​ size should be 1500 bytes
● Do you have any dropped packets in your interface?
● Is the status up or down?
● Are you working a full-duplex or maybe half-duplex?
● What is the speed of your interface?
● Look for errors in the ​Rx CRC field​, You might have Checksum value
errors
● If you are witnessing a missed Packet count, then you will see it in the
Rx_Missed_Errors field
DNS ​Settings
Now let’s check our DNS Settings
DNS can be the bottleneck of your network, you might witness latency and
performance issues, or worse than that, traffic might not get out, due to a wrong
configuration

The first place you go is the​ Network---DNS

This is where you will find your ​System DNS​ settings

To the right, you will also find The Latency of each DNS server

If you are using an internal DNS server, check your LOCAL Domain name-
is it correct?

Now, we will check our ​DNS setting​ and performance using the:
“diag test application dnsproxy” ​command
Write it down and press enter
The different options indicate specific things you can check or do

Number 1 . will dump or clear your DNS cache​ - sometimes, you will have
issues connecting to a web site or a web application, that their IP address
changed, yet your DNS cache saved the older IP address
Number 2​ is good to indicate your DNS performance as latency and TTL
Using the ​Number 3​ option, you can check your DNS settings and the ongoing
connections
Looking at our DNS servers configuration, we can also see, their latency ‘ labeled
as rt

Are they using ​TLS​ ( secure connection )

TLS=1 = secure DNS connection

TLS=0- regular, not encrypted

Moving on to our DNS cache settings

Here we can learn 2 important things

1. The number of entries in your ​DNS cache​- currently we are set to 5000
<max-num=5000>

2. The second is the Time To Live setting ​<ttl> ​ for cached entries.
currently, we are set on 1800 seconds ( default setting ). Depending on
your setting, you can enjoy a quicker response, once your DNS entries are
cached for a longer time.
DHCP ​Settings

Your DHCP server is a crucial part of your network, and its settings can indicate
a lot of what is happening on your network, for example, a host that is not
receiving an IP address

We will start by navigating to our ​LAN interface

Edit
DHCP Server
Let's look at our DHCP through the CLI
“config system dhcp server”
“show”
This command will list all available DHCP servers. Our DHCP that serves the
10.0.5.0/24 network is number 2
You can look at the IP ranges, that your DHCP leases, sometimes, it is
configured for a small range of IP address, and new clients will not get an IP
address . ​so make sure, that you have enough IP addresses in your Pool

If you want to disable your DHCP server and then enable it, you could do that
using ( it is not recommended, but it is a safer way to restart your DHCP server
operation than to kill its Process )

If you need to clear all DHCP leases

You can monitor your DHCP using the graphical user interface. I am using
fortiOS 6.4.4. , so this may change in earlier versions
On your Dashboard, click on the Plus sign ​(+) ​to create a new Dashboard
Name your DHCP Dashboard

Click O.K
Here you will be asked to select a widget, we will choose DHCP
The following technique can be used to add up different widgets,
that will give you a detailed overview of different aspects of your
FortiGate, like routing, CPU utilization, and more

Press on the widget, that you have just created

Here you see your Different IP addresses that are connected to the different
DHCP servers. you can also, see the lease time expiration date

Right-click on each IP address, will allow you to take different actions

You can create new IP reservations for specific devices, using their MAC
address
And you can also revoke ( delete ) IP addresses
Diagnose Your ​ARP ​table

Arp s used to determine the MAC address of your hosts, one of the first things
that your hosts do on the local area network, is to send ARP requests asking all
the other hosts on the LAN if they know who has a particular IP address, and if
so, send the MAC address associated with it

If you suspect that there is an IP address conflict, or that an IP

address has been assigned to the wrong interface, then you
should look a the ARP Table

On a packet capture sniffer, as we used earlier in the book, an ARP request

looks like that
Our Gateway (​192.168.1.1​) sends a broadcast request to all the different
interfaces, asking anyone “who has the 192.168.1.126 IP address “
192.168.1.126​ who listens to the broadcast request will reply with its MAC
address and it will be saved in the ARP cache table

To look at your FortiGate ARP table, you can use the following command’s

“Get system arp”

As you can see, the ARP table consists of a :
● IP address
● MAC address
● Interface name

A more detailed view can be seen using the

“Diag IP arp list”
Diagnosing the ​Routes

Whenever you witness, that packets are not arriving at their destination, or that in
general, there is no connectivity one of the first things, that you will want to do, is
to check:

● If there is a route( static, default ) for the destination

● Are there routes with higher distance than others
● Is the correct route being used?

Understanding your routing table will help you to get an overview of the roads
outside and inside your network

To view the routing table, we will start with the following command :
“Get router info routing-table all”
This command will list the active routes only. to list all the routes, we will use
another command

On the left, we can see the type of route, the most used are:
 ● s=static
● c=connected
● o=ospf
● b=bgp

Moving to the right, you will see the destination route, so let’s analyze the static
route

The destination is the default route, which is at the ​0.0.0.0/0​ address, the
following in the square brackets, is the distance, which is ​10​ ( the lower the
better, so if you have several routes, to the same destination, the one that will
take precedence is the lower distance route ), next to it is the priority (the default
is 0, the lower the better )

Following is the next hop to the destination, which is ​192,168.1.1,​ that is our
Gateway ( my ISP modem ), and finally the port number

If you want to list all the routes, including, those that are not active, you will use
the following command
“get router info routing-table database “
Here we can see, that we have another default route, but this time, the next hop
is 10.0.5.4
The asterisk ​(*)​ sign next to our second static route, tells us that although the 2
routes, he is the active one
Performance ​Issues
Performance can be a big issue on your FortiGate
if one or more operations are consuming too much memory, that may lead to
dropped packets and overall bad performance. you can even get into what is
know n as a conserve mode, where your FortiGate will stop performing security
scans or even worse
You can look at system performance using many commands, lets analyze what
is happening with the following

“get system performance status”

This command tells you a lot of your Fortigate performance

At the top output, you will see how many CPUs your FortiGate supports
CPU usage of user and kernel processes

Following that is Memory, your RAM usage.

The first thing that you should look at is the memory usage ​(used)​, if it is over
70%, there could be issues with your memory, either memory leaks, or
overloaded with traffic, processes that are running, it can even be too much
memory in the cache that can be freed

The amount of cache memory that is freeable from the cache is shown next

And then your network sessions as the number of average sessions per minute

In order for you to recognize, if something is wrong, either CPU, Memory Usage,
Amount of sessions, you will need to have a ​baseline​ of your network.
The baseline is nothing more than your normal network behavior in terms of:
● CPU utilization
● Memory Utilization
● Bandwidth
● Traffic Volume

Those metrics can be gathered through your Logs, using the ​Log and Report
menu

or your FortiView, where you can look and analyze, your network performance
over time
The best practice is to use “get system performance status “
several times throughout an hour. this could give you an
understanding of traffic spikes in your network, which can be
seen by the total number of sessions, and CPU utilization
System and user ​Processes
Another way to get a quick and detailed view of Memory and CPU utilization is to
use the Top command on your FortiGate, the same top command that is used on
Linux

To list processes that are running on your FortiGate and find the memory that is
allocated to the process instance, you can use 2 commands
“diag sys top”
“diag sys top summary”
Let’s analyze the output
On the right of the table, you can see the process name. if you are seeing a
duplicate of the process name, then it is probably due to the fact that its instance
is running several times

Following the process name is its​ ID​, it is good, in case that you wish to Kill the
process

The third column is the state of the process :

● Running
● Sleep
● Zombie
● Disk sleep

Moving to the last 2 column’s, we can see that the “​snmpd”​ process, which is
responsible for SNMP monitoring is consuming 0.5% of CPU capacity
To the right, our last column shows the amount of memory that the process is
running. memory usage can range from 0.0 to 5.5. And even higher
As you can see, my processes are in a sleep state, but on a daily basis, you will
see different processes, that consumes your FortiGate CPU and Memory
resources and that should be taken into account

Terminating ​Processes

So you're using high-level encryption in your VPN or using IPS to

scan different patterns and anomalies, all of that consumes lots
and lots of CPU resources and memory.

Let's look at the diag system top command again, add a few more parameters,
and finally terminate a process

We can play with our output refresh interval and the number of processes, that it
will show
So we will display 10 processes in an interval of 20 seconds

“Diag sys top <interval><count>”

Now we can also list the most demanding processes using 2 characters
M- to list the most demanding in terms of memory
P - to list the most demanding in terms of CPU

The most memory-demanding process in our output is the “​cmdbsvr” ( the

command database server application ) ​Usually, you will find other processes
as the IPS and “​scanunitd”​ ( antivirus ) that will consume resources,

You can also find the most CPU demanding processes, using the following
command

“Get system performance top”

Note - Killing Processes is not recommended, since it can
interrupt your network operation. The following should be done,
only as of the last option, knowing exactly what you are doing.

To Kill A processes we will use the ​“diag sys kill”​ command

Following that, you will enter what is called a signal that is a term that comes
from Linux and Unix, which is actually a polite way to ask your system to stop the
process.
We can use different signal numbers, we will use 15, which is an aggressive way
to tell your system to kill that process.
“Diag sys kill 15”
We will also enter the process ​ID​ in our case it is ​9665

We have just killed that process. And here we can see that the DNS proxy
process is actually being terminated.
CPU High Usage ​Stitch
To be notified, when your FortiGate CPU utilization is getting high and exceeds a
specific threshold, you can create a script, known also as an automation stitch
that will trigger an email towards you

Open your CLI and enter:

“config system global”
“set cpu-use-thershold <percentage>”

Here we will enter the percentage of total CPU utilization. Let’s choose 85%
Next, we will enter our Automation script actions, which are mostly debug CLI
commands:
Don’t forget to start and close your script with double quotes
The next step is to set your Automation stitch trigger

I have named it “CPU_HIGH_AUT” but you can name it anything

The event type should be set to ​high-cpu

Next, we will create the stitch itself
As you can see, we are using the trigger that we have just configured

Following that, let’s move to our admin FortiGate page

security-fabric-automation
Here you will find a new stitch named ​“high CPU”

Double click on that stitch

Select email next to our CLI script action and set your email for notification
Conserve ​Mode

One symptom of Memory issues is when your FortiGate gets into a

“conserve mode“

Getting into a conserve mode are determined by thresholds that are

pre-configured, but you can change them

● Red - The first threshold​ is when your FortiGate enters a conserve mode.
The threshold is configured using the percentage of memory that goes
behind the total RAM
● Green - The second Threshold​ is when Your Fortigate exits from
conserve mode
● Extreme - The third Threshold​ is when the sessions are being dropped, -
you're entering an extreme threshold.

When your Fortigate enters Conserve Mode, it will stop accepting new
configuration up to becoming unstable and drop new sessions. so be
aware of Memory and CPU utilization on your machine

To get into the conserve mode setting, we will use global settings
“config system global”
“set memory-use-threshold-green
Using the TAB key, you can scroll between the 3 thresholds percentage

So let's set the red mode to ​85%

If you want to check if you're in the conserve mode, or not, you can just use the
“diag hardware sysinfo conserve”

You can see that you're currently not in a conserve mode. you can also see your
total RAM and the memory that is being used, how much is freeable, and the
threshold for the Green, red and extreme states
Diagnose ​Sessions
Our Network settings seem quite good, so let’s move and look at the
session itself. We can trace the session route, and look at the multiple
sessions that are happening on our network

A higher number of sessions lead to higher memory usage

Let’s start by looking at the number of system sessions

“Get system session status”

We currently have a small number of sessions

We can also look at our sessions and their state using the:

Diag sys session full-stat

Here we can see again the number of sessions, but also:
● Session table size
● State of our sessions - 8 TCP sessions in the established state ( already
running), 1 in the SYN state, which means, that it is starting the 3-way
handshake
● Clashes

To view a list of all the running sessions on your FortiGate, you can use the​ “get
sys session list “
The output shows:
● sessions from source to destination
● Protocols used
● Time of Expiration
● Source and destination NAT

“Get system session list “​ is a great way to list active sessions, and to have a
high-level view of the running sessions
Session​ ​List

Probably the best command to list sessions on your FortiGate is the ​“diag sys
session list”

The output shows all the sessions running on your FortiGate session table, and it
can be overwhelming, so we can use filters to define exactly what we want to see

So let’s filter out our host which is ​10.0.5.7​ and analyze the output, we will use
the src filter, but you can play around with the different filter options using the
TAB key
And here is our output

We will get different sessions that are related to our host. On the last output, you
will also see the total number of sessions which is ​4
Now let’s analyze the first session

proto​= the protocol being used. protocols are mapped using numbers, number ​6
is TCP
State​ = a TCP session has different states:
1= established
2- SYN sent
 3=SYN and SYN-ACK
4=FIN_Wait
5=Time_Wait
6=close
7=Close_wait
8=Last ACK
9=Listen

Each state has its own expiration time, but if we look at the output, we can see
that our session state is 1, which means, that it is already established

Are we using traffic shapers on that session?

We can also see the source NAT and the destination

Is it moving through an acceleration chip or the regular path?

In the session state, we see that it is may dirty

When it is ​dirty​, the session needs to be validated again.
When your FortiGate receives the first packet, it evaluates the packet according
to the policy.
the evaluation is usually done on the first packet unless the firewall policy
changes, routing changes, or network configuration has been done

If the packet follows the policies, then it is labeled ​“may dirty”

When a policy changes or any other condition, then the state for the following
packets changes to “dirty “
Reduce ​session time
One method to overcome performance issues ​( CPU )​, is to reduce session
timers. Using a smaller Timer will free your FortiGate resources faster, as it will
handle fewer sessions

TCP sessions are actually comprised of different steps, each with its own timer,
but you can reduce the timer, so let’s move to our CLI

“Config system global”

“Set tcp-halfclose-timer <value>”

The​ half-close timer​ is in when one side finishes sending the data, throughout a
session. It sends a TCP packet with a FIN flag, but it still wants, to get the data
that is sent from the other side, so it has a timer, which by default is 120
seconds. Example applications that need half-close timers are Databases
You can also change the half-open timer, that is the case where one side of the
TCP connection actually crashes and doesn’t send data anymore. The other will
keep the session for a specif time that you can configure
Debug ​Packet Flow
So you browse the internet and everything seems just okay. Let's just head over
to cnn.com. See what's new. it uploads, Now, let's go to bbc.com. And yeah,
everything seems okay.

But now you decide that you want to​ Ping ( send ICMP Echo request packet )
somewhere just to check that there are no connectivity issues. ​And it seems
that you do not have a DNS resolve.
So let's stop that. And let's try to ping directly to Google's DNS server.

And yet again, you have an issue with ICMP. So how do you get around with
that? And how do you diagnose the issue behind it?

So let's get back to our FortiGate. Let's log in. And one of the first things that I
would do is to use ​“diagnose sniffer packet”​ with my host as the filter

let's just write down the IP address of our host. And see what happens.
It seems that I have an echo request. But I'm not getting any
echo-response​ back.

So that's not enough. packet capture of my traffic shows me the ​symptom​ but
doesn't give me ​medicine​.
So the next thing to do is to work at the CPU level and see how your Fortigate
handles the packets step by step
For that, we will use the ​“diag debug flow”

let's just get back to our CLI. and write the following:

“diag debug enable”

“diag debug flow filter”-​. Let's filter our source, which is the 10.0.5.7
So that will be​ “diag debug flow filter saddr 10.0.5.7”
and let's write
“diag debug flow trace start”

And press enter

I'm actually debugging the source address 10.0.5.7, when your press the TAB
key following the Filter word, you will see that you can also filter :
● Destination address
● VDOM
● Port
● Source port
● Destination port
● Protocol
Following the trace, you can decide how many packets are to be traced

So let’s see if we have any clues, that will help us to resolve our Ping issue

And yes, there it is​ “denied by forward policy check (policy number two)”

There are several reasons for the “denied by forward policy check” :
● There is no firewall policy that matches the traffic
● The traffic matches Firewall Deny Policy
● A firewall policy is in place, but the user hasn’t accepted the disclaimer
page

So we actually have a policy that denies ICMP.

Let's check our FortiGate, and yes, there it is

We could have looked at our policy page, but in production, where your FortiGate
handles hundreds of different policies that we do not have time to check, using
the diag debug flow, can actually make troubleshooting faster

Other messages that you might get is the ​ "reverse path check fail, drop"
That is when your FortiGate drops packets based on its source IP address

Your Fortigate uses an RPF ( reverse path forward ) mechanism, that helps him
to handle spoofing attacks. the Packet source IP is checked against the routing
table for a route back to the source IP. if it doesn’t have one, the packet can be
dropped. RPF has several configurations options, you can set to accept the
packets or to drop them
Final ​Words
You have just Finished “Fortigate Diagnostics Pocket Guide “ Part 3
I hope that you enjoyed the journey. My aim was to give you a head start on
Troubleshooting and diagnostics, on one of the best next-generation firewalls in
the market.

Sincerely yours
Ofer Shmueli