FortiGate III Student Guide Online PDF

DO NOT REPRINT
© FORTINET
FortiGate III
Student Guide
for FortiGate 5.2.1
DO NOT REPRINT
© FORTINET
FortiGate III Student Guide
for FortiGate 5.2.1
Last Updated: 20 July 2015
We would like to acknowledge the following major contributors: Francois Ropert, David Chan, Adrian
Buckley, Ondrej Holecek, Stephane Hamelin, and Mike Lobban
® ® ®
Fortinet , FortiGate , and FortiGuard are registered trademarks of Fortinet, Inc. in the U.S. and other
jurisdictions, and other Fortinet names herein may also be trademarks, registered or otherwise, of
Fortinet. All other product or company names may be trademarks of their respective owners. Copyright
© 2002 - 2015 Fortinet, Inc. All rights reserved. Contents and terms are subject to change by Fortinet
without prior notice. No part of this publication may be reproduced in any form or by any means or
used to make any derivative such as translation, transformation, or adaptation without permission from
Fortinet, Inc., as stipulated by the United States Copyright Act of 1976.
DO NOT REPRINT
© FORTINET
Table of Contents
VIRTUAL LAB BASICS ...................................................................................7
Topology..................................................................................................................................8
Logging In ...............................................................................................................................8
Disconnections/Timeouts .............................................................................................................................13
Transferring Files to the VM....................................................................................................13
Using HTML5 Instead of Java ................................................................................................13
Screen Resolution ...................................................................................................................14
International Keyboards ..........................................................................................................14
Troubleshooting Tips ..............................................................................................................15
SYSTEM RESOURCES ...................................................................................17
Objectives ...............................................................................................................................17
Time to Complete ....................................................................................................................17
System, Processes and Crashlog...........................................................................................18
NETWORK ....................................................................................................21
Objectives ...............................................................................................................................21
Time to Complete ....................................................................................................................21
Exploring the Session Table ...................................................................................................22
Traffic sniffer ...........................................................................................................................25
Break and Fix: Connectivity Issues .........................................................................................28

Tips for Troubleshooting ...............................................................................................................................28
DO NOT REPRINT
© FORTINET
FIREWALL POLICIES .....................................................................................30
Objectives ...............................................................................................................................30
Time to Complete ....................................................................................................................30
Traffic Shaping ........................................................................................................................31
Break and Fix: FTP Traffic ......................................................................................................32

FIREWALL AUTHENTICATION .........................................................................34
Objectives ...............................................................................................................................34
Time to Complete ....................................................................................................................34
Break and Fix: LDAP Authentication ......................................................................................35

FSSO ..........................................................................................................37
Objectives ...............................................................................................................................37
Time to Complete ....................................................................................................................37
Installing FSSO .......................................................................................................................38
Break and Fix: FSSO ..............................................................................................................42

IPSEC ..........................................................................................................44
Objectives ...............................................................................................................................44
Time to Complete ....................................................................................................................44
Break and Fix: IPsec VPN ......................................................................................................45

SECURITY PROFILES ....................................................................................47
Objectives ...............................................................................................................................47
Time to Complete ....................................................................................................................47

DO NOT REPRINT
© FORTINET
Break and Fix: Protection Profiles Part 1 ................................................................................48
Break and Fix: Protection Profiles Part 2 ................................................................................49

EXPLICIT WEB PROXY ..................................................................................51
Objectives ...............................................................................................................................51
Time to Complete ....................................................................................................................51
Break and Fix: Web Proxy ......................................................................................................52

OPERATION MODES......................................................................................55
Objectives ...............................................................................................................................55
Time to Complete ....................................................................................................................55
Transparent Mode ...................................................................................................................56
NAT/Route Mode ....................................................................................................................60
Break and Fix: NAT/Route Mode ............................................................................................62

EXTERNAL BGP ...........................................................................................63
Objectives ...............................................................................................................................63
Time to Complete ....................................................................................................................63
Break and Fix: BGP ................................................................................................................64

OSPF ..........................................................................................................66
Objectives ...............................................................................................................................66
Time to Complete ....................................................................................................................66
Break and Fix: OSPF ..............................................................................................................67

DO NOT REPRINT
© FORTINET
HIGH AVAILABILITY ......................................................................................69
Objectives ............................................................................................................................ 69
Time to Complete ................................................................................................................. 69
Break and Fix: High Availability ............................................................................................ 70

APPENDIX A: ADDITIONAL RESOURCES........................................................72
APPENDIX B: PRESENTATION SLIDES ...........................................................73
Module 1: Troubleshooting Concepts...................................................................................74
Module 2: System Resources...............................................................................................97
Module 3: Network..............................................................................................................147
Module 4: Firewall Policies.................................................................................................174
Module 5: Firewall Authentication.......................................................................................211
Module 6: FSSO.................................................................................................................241
Module 7: IPsec..................................................................................................................275
Module 8: Security Profiles.................................................................................................312
Module 9: Explicit Web Proxy.............................................................................................368
Module 10: Operation Modes.............................................................................................390
Module 11: External BGP...................................................................................................424
Module 12: OSPF...............................................................................................................456
Module 13: High Availability...............................................................................................496

DO NOT REPRINT  Virtual Lab Basics
© FORTINET
Virtual Lab Basics
In this class, you will use a virtual lab for hands-on exercises. This section explains how to connect to
the lab and its virtual machines. It also shows the topology of the virtual machines in the lab.
Note: If your trainer asks you to use a different lab, such as devices physically located in
your classroom, please ignore this section. This applies only to the virtual lab accessed
through the Internet. If you do not know which lab to use, please ask your trainer.
FortiGate III Student Guide 7

© FORTINET
Topology
FORTIMANAGER
port2
10.200.1.241
port1
10.0.1.241 10.200.1.1/24 LINUX
10.200.1.254 10.200.3.254 10.200.3.1/24
port1 eth3 port4 port6
eth1
STUDENT REMOTE 10.0.2.254/24
FortiGate FortiGate
port3 eth2 eth4
10.0.1.254/24 10.200.2.254 10.200.4.254 port5
port2
10.200.2.1/24 10.200.4.1/24
Internet
WIN-REMOTE
WIN-STUDENT 10.0.2.10
10.0.1.10
Logging In
1. Run the System Checker. This will fully verify both:
 compatibility with the virtual lab environment's software, and
 that your computer can connect
It can also diagnose problems with your Java Virtual Machine, firewall, or web proxy.
Use the URL for your location.
North America/South America:
https://remotelabs.training.fortinet.com/training/syscheck/?location=NAM-West
Europe/Middle East/Africa:
https://remotelabs.training.fortinet.com/training/syscheck/?location=Europe
Asia/Pacific:
https://remotelabs.training.fortinet.com/training/syscheck/?location=APAC
If a security confirmation dialog appears, click Run.

© FORTINET
If your computer successfully connects to the virtual lab, the result messages for the browser
and network checks will each display a check mark icon. Continue to the next step.
If a browser test fails, this will affect your ability to access the virtual lab environment. If a
network test fails, this will affect the usability of the virtual lab environment. For solutions,
either click the Support Knowledge Base link or ask your trainer.
2. With the user name and password from your trainer, log into the URL for the virtual lab. Either:
https://remotelabs.training.fortinet.com/

© FORTINET
https://virtual.mclabs.com/
3. If prompted, select the time zone for your location, and then click Update.
This ensures that your class schedule is accurate.
4. Click Enter Lab.
A list of virtual machines that exist in your virtual lab should appear.
From this page, you can access the console of any of your virtual devices by either:
 clicking on the device’s square, or
 selecting System > Open.

© FORTINET

© FORTINET
5. Click Win-Student to open a connection to that server.
A new window should open within a few seconds. (Depending on your account’s preferences,
the window may be a Java applet. If this fails, you may need change browser settings to allow
Java to run on this web site. You also may need to review and accept an SSL certificate.)
Depending on the virtual machine, the applet provides access to either the GUI or a text-
based CLI. Connections to Windows machines will use a Remote Desktop-like GUI. The
applet should automatically log in, then display the Windows desktop. For most lab
exercises, you will connect to this VM.

© FORTINET
Disconnections/Timeouts
If your computer’s connection with the virtual machine times out or if you are accidentally
disconnected, to regain access, return to the initial window/tab that contains your session’s list of VMs
and open the VM again.
If your session frequently times out or does not connect, ask your instructor.
Transferring Files to the VM

When using the Java applet to connect to a VM, you can drag-and-drop files from your computer to
the VM. For example, if you have a FortiGate configuration file that you want to upload to your lab VM,
you could create it on your computer, and then drag it into the Java application window that is
connected to the Windows VM. Usually the destination folder is C:\Uploads.
Alternatively, if you store files in a cloud service such as Dropbox or SugarSync, you can use the web
browser to download them to your VM instead.
Using HTML5 Instead of Java

When you open a VM, your browser may download and use a Java application to connect to the
virtual lab’s VM. This means that Java must be installed, updated, and enabled in your browser.
Alternatively, you can use HTML5 instead. Click the Settings button, and then select Use Java Client.
Click Save & Disconnect, then log in again. (To use this preference, your browser must allow
cookies.)

© FORTINET
When connecting to a VM, your browser should then open a display in a new window or tab.
Screen Resolution
Some Fortinet devices' user interfaces require a minimum screen size.
In the Java client, to configure the screen resolution, click the arrow at the top of the window.
In the HTML 5 client, to configure screen resolution, open the System menu.
International Keyboards
If characters in your language don’t display correctly, keyboard mappings may not be correct.

© FORTINET
To solve this in the HTML 5 client, open the Keyboard menu at the top of the window. Choose to either
display an on-screen keyboard, or send text from your computer to the VM's clipboard.
To solve this in the Java client, copy and paste between your computer and the Java applet. This
sends special characters or combinations using the keyboard icon at the top of the applet window.
Troubleshooting Tips
 If the HTML 5 client does not work, try the Java client instead. Remembering this preference
requires that your browser allow cookies.
 Do not connect to the virtual lab environment through a low-bandwidth or high-latency connection,
including VPN tunnels or wireless such as 3G or Wi-Fi. For best performance, use a stable
broadband connection such as a LAN.
 Do not disable or block Java applets. On Mac OS X since early 2014, to improve security, Java
has been disabled by default. In your browser, you must allow Java for this web site. On
Windows, if the Java applet is allowed and successfully downloads, but does not appear to
launch, you can open the Java console while troubleshooting. To do this, open the Control
Panel, click Java, and change the Java console setting to be Show console.
Network firewalls can also block Java executables.
Note: JavaScript is not the same as Java.

© FORTINET
 Prepare your computer's settings:

o Disable screen savers
o Change the power saving scheme so that your computer is always on, and does not go to
sleep or hibernate
 If disconnected unexpectedly from any of the virtual machines (or from the virtual lab portal),
please attempt to reconnect. If unable to reconnect, please notify the instructor.
 If during the labs, particularly when reloading configuration files, you see a message similar to the
one shown below, the VM is waiting for a response to the authentication server.
 To retry immediately, go to the console and enter the CLI command:
exec update-now

DO NOT REPRINT  System Resources
© FORTINET
System Resources
During this lab, you will learn to use some system and memory debug commands to describe the
status of the unit. Additional, you will generate and analyze a crashlog entry after intentionally killing
one of the FortiGate processes.
Objectives
 Use debug commands to diagnose system problems
 Use the crashlog for diagnostics
Time to Complete
Estimated: 15 minutes

© FORTINET
System, Processes and Crashlog
1. From the Win-Student server, log on to the Student FortiGate’s GUI using the account admin
with no password:
http://10.0.1.254
2. From System -> Dashboard -> Status click the Restore link inside the System Information
widget:
3. Click Browse and select the configuration file for this lab:
Resources\FortiGate III\System\Student\student-system.conf
Click Restore. The Student FortiGate will reboot. Wait a few minutes until it is back up.
4. Using PuTTY, connect SSH to the Student FortiGate CLI (use the account admin with no
password) and execute these commands to check the memory usage:
# get system status
# get system performance status
# diagnose hardware sysinfo memory
# diagnose hardware sysinfo shm

Analyze the outputs from the above commands and answer these questions:
 Is this unit running a 32-bit or 64-bit FortiOS?
 Does it have a hard disk for logging?
 How much memory is available?
 Is the unit in conserve mode?
 Why are the total high memory (HighTotal) and available high memory (HighFree) 0
MB?

© FORTINET
5. Execute now the following command to display the top 50 processes:
# diagnose sys top

6. Try to find one of these three processes: reportd, miglogd, or ipshelper. Write down its
process ID (the first number from left to right):
7. Use the following command to "kill" the chosen process:
# diagnose sys kill 11 <process_id>

11 is the kill signal. In this case the FortiGate kills the process by sending a segmentation fault
(number 11) signal.
Caution: We use the kill command in this exercise to reproduce a process failure. Be
careful although when doing it in a FortiGate that is in production. Improperly killing a
process might make a FortiGate system unstable.
8. Execute the following command one more time:
# diagnose sys top

Observe that the killed process is running again, but this time it is using a higher ID number.
Each time a process starts, it uses the next available process ID number.
9. Now, check the crashlog:
# diagnose debug crashlog read

The output should contain some entries similar to these ones:
93: 2015-03-04 07:47:34 Signal <11> was sent to process <00065> by

user <admin>
94: 2015-03-04 07:47:34 <00065> firmware FortiGate-VM64

v5.2.1,build0618b618,140915 (GA) (Release)
95: 2015-03-04 07:47:34 <00065> application reportd

© FORTINET
96: 2015-03-04 07:47:34 <00065> *** signal 11 (Segmentation fault)
received ***
97: 2015-03-04 07:47:34 <00065> Register dump:
98: 2015-03-04 07:47:34 <00065> RAX: fffffffffffffffc RBX:

0000000000000000
99: 2015-03-04 07:47:34 <00065> RCX: ffffffffffffffff RDX:

0000000000000400
100: 2015-03-04 07:47:34 <00065> R8: 0000000000000000 R9:

0000002a95d49de0
...
120: 2015-03-04 07:47:35 <00065> [0x0043d14f] => /bin/reportd
121: 2015-03-04 07:47:35 <00065> [0x0043abfa] => /bin/reportd
122: 2015-03-04 07:47:35 <00065> [0x2a95c40475] => ../lib/libc.so.6

(__libc_start_main+0x000000f5)
123: 2015-03-04 07:47:35 liboffset 00021475
124: 2015-03-04 07:47:35 <00065> [0x0043aca1] => /bin/reportd
125: 2015-03-04 07:47:35 reportd received a signal - 11
126: 2015-03-04 07:47:36 the killed daemon is /bin/reportd:

status=0x0
Check the first three lines. They contain the FortiOS build number, the name of the process
that failed (or was killed) and the kill signal number.

DO NOT REPRINT  Network
© FORTINET
Network
The following lab exercises show how to use some debug commands to troubleshoot connectivity
problems. You will analyze the information in the FortiGate session table, run the built-in sniffer and
use the debug flow to understand how the FortiGate is processing each IP packet.
Objectives
 Analyze the information in the session table
 Capture traffic using the built-in sniffer tool
 Use some CLI troubleshooting utilities and tools
Time to Complete

© FORTINET
Exploring the Session Table
During this exercise you will analyze the information displayed in the FortiGate session table.
1. From the Win-Student server, log on to the Remote FortiGate’s GUI first using the account
admin with no password:
http://10.200.3.1
2. Find the Resource folder on the desktop and upload the Remote configuration file for this lab:
Resources\FortiGate III\General\Remote\remote-general.conf
The Remote FortiGate will reboot. Wait a few minutes until it is back up.
3. Log on to the Student FortiGate’s GUI using the account admin with no password:
http://10.0.1.254
4. Upload the Student configuration file for this lab:
Resources\FortiGate III\General\Student\student-general.conf
The Student FortiGate will reboot. Wait a few minutes until it is back up.
5. Open a command prompt window in the Win-Student server and execute a ping to the Student
FortiGate's default gateway:
> ping 10.200.1.254

6. Using PuTTY, connect SSH to the Student FortiGate CLI and execute these commands:
# diagnose sys session filter clear
# diagnose sys session filter proto 1
# diagnose sys session filter dst 10.200.1.254
# diagnose sys session list

Analyze the information related with the ICMP session created for the test traffic:
session info: proto=1 proto_state=00 duration=726 expire=63276

timeout=64000 flags=00000000 sockflag=00000000 sockport=0 av_idx=0
use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
ha_id=0 policy_dir=0 tunnel=/
state= may_dirty none app_ntf

© FORTINET
statistic(bytes/packets/allow_err): org=240/4/1 reply=240/4/1
tuples=2
orgin->sink: org pre->post, reply pre->post dev=4->2/2->4

gwy=10.200.1.254/10.0.1.10
hook=post dir=org act=snat 10.0.1.10:1-

>10.200.1.254:8(10.200.1.1:62464)
hook=pre dir=reply act=dnat 10.200.1.254:62464-

>10.200.1.1:0(10.0.1.10:1)
misc=0 policy_id=1 auth_info=0 chk_client_info=0 vd=0
serial=00000243 tos=ff/ff ips_view=0 app_list=0 app=0
dd_type=0 dd_mode=0
Observe the following from the session:
 The may_dirty flag
 The line containing statistics, which should correctly display the amount of ICMP packets
sent and received
 The source NAT information
 The ID of the policy matching the traffic
 The protocol state, whose value is always 00 for the case of ICMP traffic
You will also notice that the expiration timer (expire) and timeout are unusually high. The
default timeout for ICMP sessions is 60 seconds. For the purpose of giving more time to
analyze the session information, the ICMP session timeout was increased to 64.000 seconds.
You can see this configuration by typing the CLI command:
# show system session-ttl

7. Stop the ping (if it is still running) and access the Student FortiGate GUI. Go to Policy &
Objects > Policy > IPv4 and edit the firewall policy with the sequence number 1 (the first one
from top to bottom). Click Enable this policy and then OK.
After a firewall policy configuration change, the FortiGate adds the dirty flag to all the session
with the may_dirty flag. Next time there is traffic matching any of those sessions, the FortiGate
will re-evaluate the action to take.
8. Execute this command in the Student FortiGate CLI and observe that the session has the dirty
flag now:

9. Run the ping one more time from the Win-Student server to 10.200.1.254:
> ping 10.200.1.254

It should fail as the firewall policy enabled earlier is blocking ICMP traffic.
Check quickly the session information one more time:

If you do it fast enough, you will notice that the session is still there but the block flag was
added. All traffic matching a session with that flag is denied. Also, the session expiration

© FORTINET
time is much smaller now. The session will remain in the FortiGate memory until this timer
expires (30 seconds).
10. Before proceeding to the next lab exercise, go to Policy & Objects > Policy > IPv4 and disable
the firewall policy with the sequence number 1 (the one blocking the ICMP traffic.)

© FORTINET
Traffic sniffer
During this exercise, you will use the FortiGate built-in sniffer to capture traffic. After that, you will use
a Perl script to convert the capture to a PCAP file that can be analyzed by a packet analyzer, such as
Wireshark.
1. Open a SSH connection to the Student FortiGate using PuTTY.
2. Click on the upper left icon and select Change Settings:
Go to Session -> Logging and select All session output. Then click Browse and select the
folder c:\Perl64\bin. Click Save, and then Apply. With this change, PuTTY will save all the
sniffer output into a text file name putty.log:

© FORTINET
3. Type the following command in the Student FortiGate CLI to start the sniffer:
# diagnose sniffer packet port1 "host 10.200.1.254 and port 80" 3

4. Open a browser and access this URL:
http://10.200.1.254
You should observe the packets captured in the PuTTY window.
5. Close PuTTY and open a command prompt window. Execute these commands:
> cd \Perl64\bin
> perl fgt2eth.pl –in putty.log

The Perl script fgt2eth.pl converts the output captured to a PCAP file with the name
putty.log.pcap.
6. Use Windows File Explorer and double click the created file:
c:\Perl64\bin\putty.log.pcap
This starts Wireshark and opens the file for analysis.
7. Observe the information in the packets captured. Right click any packet and select Follow
TCP Stream:

© FORTINET
Observe the new Window that pops up. It shows the application-layer data between the client
and the server for that specific TCP session:
Note: Follow TCP Stream is a useful tool to troubleshoot problems at the application
layer.

© FORTINET
Break and Fix: Connectivity Issues
In this exercise, your environment is simulating the following customer network:
10.200.1.254/24
STUDENT
FortiGate Web server
10.200.1.1/24 10.200.3.254
port1
port3
10.0.1.254/24
port2
10.200.2.1/24
10.200.2.10/24
WIN-STUDENT REMOTE HOST
10.0.1.10 10.200.4.1/24
There are however four problems:

1. Although the Telnet protocol is enabled for administrative access in the Student FortiGate's
port3 (10.0.1.254), you cannot access the unit's CLI using telnet.
2. You cannot access the web server (http://10.200.3.254) from Win-Student.
3. You cannot ping the remote host (10.200.4.1) from Win-Student.
4. You cannot access the GUI of the router 10.200.1.254 from Win-Student. The router GUI must
be accessible by using the URL: http://10.200.1.254:88
Find the causes of these problems by using first debug commands, before looking for
configuration mistakes.
In which of the four problems the FortiGate is doing something wrong and in which ones it is
not?
Tips for Troubleshooting

 Can you ping the destination IP address from the Win-Student server?
 Use the sniffer tool to verify that the traffic is actually arriving to the FortiGate's port3. Use
verbosity 4 or 6 and a filter that can capture the traffic both ways
 If the traffic is not intended to terminate in the FortiGate, use the sniffer to check that it is
actually been forwarded to the next hop IP address (use the network diagram provided.) Again,
use a filter in the sniffer that can capture the traffic both ways

© FORTINET
 Check the session table. Is the FortiGate creating the session? Check the session protocol state.
Do you see anything wrong there?
 Clear the related session (if any) from the session table, enable the debug flow and generate more
test traffic. Do you see any debug flow error?
 Try to ping the next hop IP address from the Student FortiGate. Sniffer the traffic to the next hop
IP address while doing it. You can have two simultaneous SSH connections, one running the
sniffer and another one running the ping

DO NOT REPRINT  Firewall Policies
© FORTINET
Firewall Policies
During these lab exercises you will configure and monitor traffic shaping. Additionally, you will
troubleshoot and fix a connection problem to a FTP server.
Objectives
 Monitor statistics related with traffic shaping
 Troubleshoot a FTP connection issue
Time to Complete

© FORTINET
Traffic Shaping
http://10.0.1.254
Resources\FortiGate III\Firewall-Policies\Student\student-policy.conf
3. Go to Policy & Objects > Objects > Traffic Shapers and click Create New. Configure the
following settings:
Type Shared
Name SharedPolicy
Apply shaper All policies using this shaper
Traffic Priority High
Max Bandwidth 10 Kb/s
Click OK.
4. Go to Policy & Objects > Policy > IPv4 and edit the first policy on the top. Enable Shared
Shaper and select SharedPolicy. Click OK.
5. From a browser in Win-Student go to http://www.youtube.com and play some videos.
6. While plying the videos, execute these CLI commands:
# diagnose firewall shaper traffic-shaper stats
# diagnose firewall shaper traffic-shaper list

Locate the counters for packet drops. Execute the above commands a few times more and
notice how those counters increase with the traffic.
7. Before proceeding to the next lab exercise, go to Policy & Objects > Policy > IPv4 and edit the
first policy on the top one more time. Disable Shared Shaper and click OK.

© FORTINET
Break and Fix: FTP Traffic
A FTP server is running in the Linux server 10.200.3.254:222.

The network administrator has installed FileZilla in all the workstations. The administrator has also
added a pre-configured Site profile to FileZilla called FTPsite.
To connect to the server from any workstation, users open FileZilla, click Site Manager, select the site
FTPsite and click Connect:
However, you cannot connect to the FTP server from Win-Student. FileZilla shows this error after each
connection attempt:
The problem only happens with the workstations connected behind the Student FortiGate.
Workstations in other subnets can connect successfully.

© FORTINET
Can you find out what the FortiGate is doing wrong?
What has to be done to fix the problem?

 Understand first which TCP ports are used for this connection. The control channel is using port
TCP 222. The data channel is using the standard port TCP 20
 Understand also how the traffic flows. Is this a FTP server working in active or passive mode? In
active mode the data channel is initiated by the server. In passive mode the data channel is
initiated by the client. Sniffer the traffic in the FortiGate and Linux server to determine who is
initiating the data channel. To run the sniffer in the Linux server follow these steps:
1. Connect SSH to the Linux server (10.200.1.254). Use the username root with the password
password
2. Execute the following command to sniffer the data channel:
# tcpdump -v -i any -nn port 20

You can also sniffer the control channel traffic with this other command:
# tcpdump -v -i any -nn port 222

 Use the FortiGate's built-in sniffer to capture the control channel traffic (port 222) before and after
the FortiGate. Use a verbosity level of either 3 or 6 and save the output to a file. After that, use the
Perl script to convert it to Wireshark (as explained in an earlier lab exercise) and analyze it
 Run the debug flow over the FTP control channel and analyze the output. Is there anything
missing there?

DO NOT REPRINT  Firewall Authentication
© FORTINET
Firewall Authentication
During this lab you will learn to use the authentication and LDAP debug commands to troubleshoot an
authentication issue.
Objectives
 Monitor the status of authenticated users
 Troubleshoot problems related with LDAP authentication
Time to Complete

© FORTINET
Break and Fix: LDAP Authentication
http://10.0.1.254
Resources\FortiGate III\Firewall-Authentication\Student\student-
authentication.conf
An administrator has configured the Student FortiGate to do LDAP authentication against the Windows
AD server located at 10.0.1.10 (Win-Student). However, authentication is failing.
Two LDAP users have been created in the Windows AD Server:
 Username: student, password: Fort1net
Must not have access to information technology sites, such as www.fortinet.com
Belongs to the Windows AD group:
CN=Domain Users,CN=Users,DC=trainingAD,DC=training,DC=lab
Traffic from this user must match the firewall policy crated for the user group LDAPUsers,
which contains the web filter profile NoITSites. Do not change this web filtering configuration.
 Username: administrator, password: password
Must have unrestricted access to the Internet
Belongs to the Windows AD group:
CN=Enterprise Admins,CN=Users,DC=trainingAD,DC=training,DC=lab
Traffic from this user must match the firewall policy created for the user group Enterprise
Admins, which does not have any web filter profile. Do not create any web filter profile for this
policy. Leave it without any.
Use the authentication and LDAP debug commands learned to isolate and fix the problem.
Can you explain why the FortiGate is not challenging users to authenticate?
Can you change the FortiGate configuration to fix the problem?
Can you change the FortiGate configuration to properly restrict the Internet access to the user
student, while leaving unrestricted access to the user administrator?

 First, test the LDAP authentication from the CLI after enabling the real time debug command:
diagnose debug application fnbamd -1
diagnose debug enable

© FORTINET
diagnose test authserver ldap WindowsLDAP administrator password
diagnose test authserver ldap WindowsLDAP student Fort1net

 Check the Distinguished Name (DN) for student and administrator, by running these commands in
Win-Student:
dsquery user -name student
dsquery user -name administrator

 Once the LDAP CLI test works, check the firewall authentication by browsing the Internet from
Win-Student. Look at the session table or run the debug flow to know which firewall policy is
matching the traffic
 The output of the LDAP test command shows the user groups for each user. Compare them with
the groups configured in each firewall policy
 After any configuration change, de-authenticate the users from the FortiGate and clear the
browser cache (or refresh the page with the F5 key). It is also recommended to clear the related
entries in the session table:
# diagnose sys session filter dport 80
# diagnose sys session clear

To de-authenticate a user, go to User & Device -> Monitor -> Firewall, select the user and click on
De-authenticate

DO NOT REPRINT  FSSO
© FORTINET
FSSO
During this lab you will install the FSSO collector agent and troubleshoot a FSSO problem.
Objectives
 Check the connectivity between the FortiGate and the CA
 Track user logon events in the DC, CA and FortiGate
 List the active FSSO users
 Troubleshoot a FSSO problem
Time to Complete

© FORTINET
Installing FSSO
http://10.0.1.254
Resources\FortiGate III\FSSO\Student\student-FSSO.conf
3. On Win-Student, right-click the Fortinet Single Sign On (FSSO) installation file located in
Resources\FSSO, then select Run as administrator.
This should launch the Fortinet Single Sign On Agent Installation Wizard. Follow the wizard to
install the agent on Win-Student.
4. When prompted for the Windows server administrator password, enter "password":
Click Next.
5. In the Install Options window, accept the default settings:
Click Next.
6. Click Install to complete the installation.

© FORTINET
7. At the end of the Single Sign On Agent installation, the Launch DC Agent Install Wizard option
will be selected.
Click Finish to complete the collector agent Installation. This launches the Domain Controller
Agent Installation Wizard.
8. In the Install DC Agent Wizard, accept the Collector Agent IP Address of 10.0.1.10 and the
Collector Agent Listening Port of 8002.
Click Next.
9. Select the TRAININGAD:trainingAD.training.lab domain to monitor.
Click Next.
10. Only the student account needs to be monitored in this exercise. Expand the TRAININGAD
domain and disable all the users in the TRAININGAD domain EXCEPT for student:
Click Next.
11. Set the Working Mode to Polling Mode and select Check Windows Security Event Logs.

© FORTINET
Click Next.
12. After the installation, open the Windows start screen and run the application Configure Fortinet
Single Sign-on.
Perform the following tasks in the Fortinet single sign-on agent configuration window:
 Change the Password to Fortinet.
 Change the Workstation verify interval to 0
 Change the Log level to Information
 Enable Log logon events in separate logs
Click Apply.
13. Click Show Monitored DCs to verify the communication between the collector agent and
the domain controller agent. The IP address of 10.0.1.10 should show as being logged in.
Click Close.
14. Click Select Domains to Monitor and verify that the TRAININGAD:trainingAD.training.lab
domain is selected. Click OK.

© FORTINET
15. Click Set Group Filters. Click Add and enable the Default filter. Click Advanced and expand
the domain name of TRAININGAD. From the expanded list select Users and Domain Admins.
Click Add, then OK.
Click OK.
Click Save & Close to close the Fortinet single sign-on agent configuration window.

© FORTINET
Break and Fix: FSSO
In this network the collector agent has been installed in Win-Student. An administrator has configured
the Student FortiGate to allow Internet access only to active FSSO users. However, it is not working
as desired. Active FSSO users do not have Internet access.
Use the authentication and FSSO debug commands learned to isolate and fix the problem.
To test the FSSO authentication, generate first a login event following these steps:
1. On Win-Student, run the Windows Remote Desktop Connections application.
2. Enter the computer IP address 10.0.1.10:
Log in with these credentials:
Username: Student
Password: Fort1net
Ignore the error message indicating that the user is not authorized for remote login. The
objective of these steps is just to generate a logon event without rebooting the server.
3. After that, test the Internet access from a browser.
Can you explain why the FortiGate is blocking the traffic?
Can you change the FortiGate or/and collector agent configurations to fix the problem?

 Check the active FSSO users in the collector agent by clicking Show logon users
 Use the following command to check the active FSSO users in the FortiGate:
# diagnose debug authd fsso list

 Use the FortiGate real time debug command for FSSO:
# diagnose debug application authd 8256
# diagnose debug enabled

 Check the collector agent logs

© FORTINET
 Use the Windows Remote Desktop Connections application after each configuration change to
generate new login events

DO NOT REPRINT  IPsec
© FORTINET
IPsec
During this lab you will troubleshoot an IPsec VPN problem.
Objectives
 Use the IKE real time debug to isolate problems during the phase 1 and phase 2 negotiations
 Use the debug flow tool to isolate IPsec traffic flow issues
 Monitor the status of an IPsec VPN
Time to Complete

© FORTINET
Break and Fix: IPsec VPN
1. From the Win-Student server, log on to the Student FortiGate’s GUI using the account admin
with no password:
http://10.0.1.254
Resources\FortiGate III\VPN\Student\student-vpn.conf
3. Log on to the Remote FortiGate’s GUI first using the account admin with no password:
http://10.200.3.1
Upload the Remote configuration file for this lab:
Resources\FortiGate III\VPN\Remote\remote-vpn.conf
The Remote FortiGate will reboot and load the new configuration.
An administrator has created an IPsec VPN between the Student FortiGate and the Remote FortiGate.
The objective is to encrypt the traffic both ways between the subnets 10.0.1.0/24 and 10.0.2.0/24.
For the purpose of this lab, assume that the IP address of the Remote FortiGate will be changing
frequently, so the administrator has configured the VPN in the Student FortiGate side as Dialup User.
The name of the VPN in the Student side is RemoteSite. The name of the VPN in the Remote side is
ToHub. There is another VPN created (for a different purpose) in the Student FortiGate with the name
DialUpUsers.
The VPN IPsec between Student and Remote is down. Your objective is to fix the problem, so that the
tunnel comes up and you can ping from Win-Student to Win-Remote.
Use the IPsec debug commands learned in this lesson to isolate and fix the problem. The solution
requires:
 Keeping the Remote Gateway type in the VPN RemoteSite as Dialup User (for the reason
explained before)
 No configuration changes in the DialUPUsers VPN in the Student FortiGate, as this VPN is
already operative and working as expected (you do not need to test this VPN, assume that it is
working)

 Check first why the tunnel is not coming up, use the IKE real time debug in both sides to
troubleshoot the problem:
# diagnose debug application ike -1
# diagnose debug enable

 After the tunnel is established, check that you can ping from Win-Student to Win-Remote. If

© FORTINET
there is a problem, sniffer the traffic and use the debug flow

DO NOT REPRINT  Security Profiles
© FORTINET
Security Profiles
During the following exercises you will use debug commands to fix FortiGuard and web filtering issues.
Objectives
 Troubleshoot FortiGuard problems
 Troubleshoot web filtering problems
 Fix certificate warnings during full SSL inspection
 Investigate virus infections
Time to Complete

© FORTINET
Break and Fix: Protection Profiles Part 1
http://10.0.1.254
Resources\FortiGate III\UTM\Student\student-UTM-1.conf
3. The configuration contains two VDOMs. Go to Virtual Domains -> root -> Policy & Objects ->
Policy -> IPv4.
Check the firewall policy from port3 to port1. It has antivirus and web filtering enabled.
4. Then go to Virtual Domains -> root -> Policy & Objects -> Security Profiles -> Web Filter and
review the profile WebFilterUsers. Some categories, such as malicious websites, streaming
media, hacking and proxy avoidance are being blocked.
Open a browser in Win-Student and go to these restricted web sites:
http://www.youtube.com
http://www.proxyavoidance.com
The FortiGate is not blocking the access to those sites. Indeed, web filter does not seem to be working
at all.
Why isn't the FortiGate blocking the access to any restricted web site?
Note: Your lab environment uses a FortiManager as a local FDS server. It contains a
local copy of the FDS web rating database. The FortiGate devices validate their VM
licenses against the FortiManager. They also send the rating requests to the
FortiManager IP address (10.0.1.241) instead of the public FDS servers. Do not
change this configuration, as it will affect the FortiGate license status.

 Use the web filtering real time debug:
# diagnose debug application urlfilter -1

 Use the FortiGuard real time debug:
# diagnose debug application update -1

© FORTINET
Break and Fix: Protection Profiles Part 2
http://10.0.1.254
Resources\FortiGate III\UTM\Student\student-UTM-2.conf
This configuration is similar to the previous one but it contains the fix to the problem that was
troubleshot during the first exercise of this lab.
The configuration also includes a web filter profile to block, among others, the following FortiGuard
categories:
 Proxy Avoidance
 Streaming Media and Download
 Hacking
All the restricted sites seem to be properly blocked now, such as:
http://www.youtube.com (Streaming Media and Download)
http://www.elite-hackers.com (Hacking)
http://www.proxyavoidance.net (Proxy Avoidance)

However, the administrator complains that the following two sites should be blocked, and they are not.
According to him, they belong to blocked categories:
http://www.metacafe.com
http://www.eicar.org
Additionally, customers are reporting two more problems:
 They receive certificate warnings each time they connect to an HTTPS site
 Even though antivirus is enabled, they can still download the virus sample eicar.com
located at the ftp server 10.200.3.254:222. To test it, open FileZilla and connect to the
preconfigured site FTPSite. Select the Desktop as the local site folder and pub as the
remote site folder. Right click the eicar.com file and select Download:

© FORTINET
Why are those two sites reported by the administrator not being blocked? How can you change the
FortiGate configuration to fix it?
Why are users getting SSL certificate warnings? How can you resolve it?
Why isn't FortiGate detecting the EICAR virus?

For the web filtering problem:
 Enable the following real time debug and attempt to browse the two websites not being blocked:
# diagnose debug application urlfilter -1

The output can be verbose, so save it from PuTTY to a local file.
 Remember to clear the browse cache and FortiGate session after doing any configuration change
For the antivirus problem:

 Sniffer the FTP traffic and analyze the output of the debug flow
 Check the entry in the FortiGate session table for the FTP session

DO NOT REPRINT  Explicit Web Proxy
© FORTINET
Explicit Web Proxy
During this lab you will troubleshoot some explicit web proxy problems.
Objectives
 Monitor web proxy traffic and sessions
 Monitor web proxy DNS traffic
 Use the web proxy real time debug
Time to Complete

© FORTINET
Break and Fix: Web Proxy
http://10.0.1.254
Resources\FortiGate III\Web-Proxy\Student\student-web-proxy.conf
3. Open Firefox and click Open menu. Then click Options:
4. Go to Advanced -> Network and click Settings:

© FORTINET
5. Select Automatic proxy configuration URL and type the following URL:
http://10.0.1.254:8080/proxy.pac
6. Restart the browser.
Test the proxy by accessing any web site. Additionally, access to the Fortinet web site is essential for
users. So, test it using the following URL:
http://www.fortinet.com
Why isn't the web proxy working at all?
After fixing the web proxy, test the access to the Fortinet web site. Why isn't working yet? Can you
also fix it?

 Sniffer the traffic in port 8080 (web proxy traffic)
 Sniffer the traffic coming from the browser:
# diagnose sniffer packet any 'host 10.0.1.254 and not port 22 and
not port 443' 4
 Sniffer the traffic going to the web proxy IP address:
# diagnose sniffer packet any 'host 10.0.1.10 and not port 22 and not
port 443' 4
 Use the following debug commands to check the status of the web proxy connections:
# diagnose wad session list
# diagnose test application wad 2200

 Run the web proxy real time debug using the filter below:
# config web-proxy debug-url
edit fortinet
set url-pattern www.fortinet.com
set status enable
set exact enable
next
edit fortiguard

© FORTINET
set url-pattern www.fortiguard.com
set status enable
set exact enable
next
end
# diagnose wad debug-url enable
# diagnose wad console-log enable

After that, try to browse these two web sites and compare the results:
http://www.fortinet.com
http://www.fortiguard.com
 Remember to restart the browser after any change to the PAC file

DO NOT REPRINT  Operation Modes
© FORTINET
Operation Modes
This lab has 3 exercises. The first exercise includes a FortiGate in transparent mode. During exercises
2 and 3 you will troubleshoot routing problems with two FortiGate devices in NAT/route mode.
Objectives
 Describe how FortiGate routes traffic
 Diagnose routing problems due to reverse path forwarding check
 Identify the existing sessions that will be routed through a different path after a change in the
routing table
 Use debug commands to troubleshoot routing problems
 Segment a layer-2 network into different broadcast domains using a FortiGate in transparent
mode
Time to Complete

© FORTINET
Transparent Mode
Port1 and port3 of a FortiGate in transparent mode are connected to a network. An administrator
wants to create 4 broadcast domains. For that purpose, the administrator segmented the network into
4 VLANs:
VLAN Name VLAN tag FortiGate interfaces
Native VLAN No tag port1

port3
VLAN 20 20 port1-VLAN20
port3-VLAN20
port3-VLAN30
port3-VLAN40
The Win-Student server is connected to the native VLAN in port 3.

The following diagram summarizes this network topology:
1. First, check that Firefox is not configured to use an explicit web proxy.
2. Click Open menu. Then click Options:

© FORTINET
Go to Advanced -> Network and click Settings:
Check that No proxy is selected and click OK.

http://10.0.1.254
Resources\FortiGate III\Operation-Modes\Student\student-operation-
modes-transparent.conf
This changes the FortiGate to transparent mode and adds all the VLAN sub-interfaces.
5. Connect to the Student FortiGate using SSH and start this sniffer:
diagnose sniffer packet any "arp and host 10.0.1.15" 4

© FORTINET
6. From Win-Student command prompt, do a ping to 10.0.1.15:
> ping 10.0.1.15

This IP address is not active, so you will not receive any echo reply. However, the ping
triggers ARP traffic that can be captured by the previous sniffer.
7. The output of the sniffer will be similar to this:
So, broadcast traffic is being forwarded to all the VLAN sub-interfaces. Each VLAN is not a
different broadcast domain, as the administrator wants.
Why is this happening?
What configuration change must be done in the FortiGate to actually make each VLAN a
different broadcast domain?
8. From the FortiGate CLI, execute these configuration changes:
# config system interface
edit port1-VLAN20
set forward-domain 20
next
edit port1-VLAN30
next
edit port1-VLAN40

© FORTINET
next
edit port3-VLAN20
next
edit port3-VLAN30
next
edit port3-VLAN40
next
end
9. Execute the sniffer and ping one more time. Now you will see that the ARP packets are
confined only to the native VLAN.

© FORTINET
NAT/Route Mode
http://10.0.1.254
modes-NAT.conf
3. Log on to the Remote FortiGate’s GUI using the account admin with no password:
http://10.200.3.1
4. Upload the Remote configuration file for this lab:
Resources\FortiGate III\Operation-Modes\Remote\remote-operation-
modes-NAT.conf
5. Check the IPsec VPN configuration in both FortiGate units. Go to VPN -> IPsec -> Tunnels.
Check also the firewall policy and the routing table in both devices. Go to Policy & Objects ->
Policy -> IPv4, then check Router -> Monitor -> Routing Monitor.
You will notice that there is an IPsec VPN created between both units to encrypt the traffic
between the subnets 10.0.1.0/24 and 10.0.2.0/24. You will also see a route in the Student
FortiGate to the subnet 10.0.2.0/24 using the IPsec tunnel.
6. Execute a continuous ping from the Win-Student command prompt to Win-Remote:
> ping -t 10.0.2.10

You will receive the echo reply from Win-Remote as an indication that the tunnel is operating
normally.
7. Without stopping the ping, access the Remote FortiGate and go to System -> Network ->
Interfaces. Click the plus icon besides port4 to expand it, and edit the interface ToStudent:

© FORTINET
Change the Administrative Status of this interface to Down. Click OK.
8. Wait a few seconds and then check the status of the VPN in the Student FortiGate. Go to VPN
-> Monitor -> IPsec Monitor. As the remote virtual IPsec interface is administratively down, the
VPN is down.
Check now the routing table. As the VPN is down, the route to 10.0.2.0/24 was removed.
Check also the ping running in Win-Student. It is failing.
9. Proceed to bring back up the remote IPsec interface. Access the Remote FortiGate, go to
System -> Network -> Interface and edit the ToStudent interface. Change the Administrative
Status to Up and click OK.
10. Go back to the Student FortiGate and check the status of the VPN. Go to VPN -> Monitor ->
IPsec Monitor. If the VPN is still down, right click it and select Bring Up. The tunnel will come
up.
11. Check the routing table. Go to Router -> Monitor -> Routing Monitor. You will notice that the
route to the subnet 10.0.2.0/24 is back to the routing table.
12. Check one more time the ping running in Win-Student. It is not working yet.
13. Sniffer this traffic. Connect to the Student FortiGate's CLI and execute this command:
# diagnose sniffer packet any "icmp and host 10.0.2.10" 4
Why is the ping not working if the VPN is up and the route is back?
Why is the FortiGate still routing the ping traffic through out port1 (and not through
ToRemote)?
What can be done to prevent this problem?

© FORTINET
Break and Fix: NAT/Route Mode
http://10.200.3.1
Resources\FortiGate III\Operation-Modes\Remote\remote-operation-
modes-NAT.conf
http://10.0.1.254
modes-NAT.conf
.
The administrator is reporting two problems:
1. You cannot ping 10.200.4.1 from Win-Student
2. The Student FortiGate configuration includes two default routes, one using port1, and the
other one using port2. However, only one of them is active in the routing table
Can you fix these two problems?

 Sniffer the ping to 10.200.4.1
 Use the debug flow while running the ping to 10.200.4.1
 Use these commands to check the routing table:
# get router info routing-table database
# get router info routing-table all

 Check the status of the link health monitors (if any) under System -> Monitor -> Link Monitor

DO NOT REPRINT  External BGP
© FORTINET
External BGP
During this lab you will troubleshoot some BGP issues between two FortiGate devices.
Objectives
 Monitor and check the status of a BGP communication
 Troubleshoot some common external BGP issues
Time to Complete

© FORTINET
Break and Fix: BGP
http://10.200.3.1
Resources\FortiGate III\BGP\Remote\remote-BGP.conf
http://10.0.1.254
Resources\FortiGate III\BGP\Student\student-BGP.conf
An administrator has configured BGP between Student and Remote. The Student FortiGate belongs to
the autonomous system 65500 and the Remote FortiGate belongs to the autonomous system 65001.
However, the BGP peering is currently down. The objective is to bring up the BGP connection
between both units. Also, each FortiGate must advertise all its locally connected subnets.
Try not to compare both BGP configurations to find mismatches. You should troubleshoot the problem
using the BGP debug commands learned during this lesson.
Explain each problem supporting your arguments with the output of sniffers and BGP debug
commands.

 Use these BGP debug commands and sniffer:
# get router info bgp summary
# get router info bgp network
# get router info bgp neighbors
# get router info bgp neighbors <peer-IP> advertise
# diagnose sniffer packet any “port 179” 4

 Use the BGP real time debug:

© FORTINET
# diagnose ip router bgp all enable
# diagnose ip router bgp level info

 Use this command to restart the BGP connection any time:
# execute router clear bgp all
Stop and Think

After fixing the BGP connectivity, you might notice that you cannot reach Win-Remote from
Win-Student yet, even when both FortiGate routing tables are ok. You do not need to fix
this problem during this lab, but can you find out what is causing this issue?

DO NOT REPRINT  OSPF
© FORTINET
OSPF
During this lab you will troubleshoot some OSPF over IPsec issues between two FortiGate devices.
Objectives
 Establish OSPF adjacency between FortiGate devices
 Use debug commands to troubleshoot some OSPF problems
 Monitor the status of a OSPF network
Time to Complete

© FORTINET
Break and Fix: OSPF
http://10.200.3.1
Resources\FortiGate III\OSPF\Remote\remote-OSPF.conf
http://10.0.1.254
Resources\FortiGate III\OSPF\Student\student-OSPF.conf
An administrator has configured an IPsec tunnel between the Student FortiGate and the Remote
FortiGate. OSPF has been configured to run over the tunnel, so that each FortiGate can advertise its
networks to its remote peer. The tunnel is currently up, however the OSPF adjacency is down.
The objective is to have the OSPF routes correctly learned by both FortiGate units. Also, the IPsec
VPN must remain stable.
Try not to compare both OSPF configurations to find mismatches. You should troubleshoot the
problem using the OSPF debug commands learned in this lesson. Explain each problem supporting
your arguments with the output of the debug commands.

 Check the routing table and OSPF neighbor status:
# get router info ospf status
# get router info ospf neighbor

Is the neighbor adjacency established?
Are OSPF routes present?
 Run the real time debug:
# diagnose ip router ospf all enable
# diagnose ip router ospf level info

© FORTINET
 Once the OSPF issues are resolved, go to the VPN event logs. Is the IPsec VPN stable? Watch
the log messages for a few minutes
Compare the Student routing table when the tunnel is down with the table when it is up.
What is causing the tunnel to bounce?

DO NOT REPRINT  High Availability
© FORTINET
High Availability
During this lab you will troubleshoot some high availability problems between two FortiGate devices.
Objectives
 Monitor a HA cluster
 Check the status of the HA configuration and session synchronization
 Troubleshoot some common HA problems
Time to Complete

© FORTINET
Break and Fix: High Availability
FortiGate
REMOTE
port3 port1
LINUX
port2 10.200.1.254
eth1
port2
port3 port1
10.0.1.254/24 10.200.1.1/24
WIN-STUDENT STUDENT
10.0.1.10 FortiGate
eth0
LAN3 LAN0
0.0.0.0 0.0.0.0
http://10.200.3.1
Resources\FortiGate III\High-Availability\Remote\remote-ha.conf
http://10.0.1.254
Resources\FortiGate III\High-Availability\Student\student-ha.conf
After loading both configurations, the cluster is not forming. The Remote unit cannot join the HA
cluster.
Use the debug commands learned in this lesson to troubleshoot the problem.

© FORTINET
 Run the HA real time debug in the CLI of both units:
# diagnose debug application hatalk 255
# diagnose debug application hasync 255

 Use these additional HA debug commands:
# diagnose sys ha status
# diagnose sys ha showcsum

 For easy access to each unit while the cluster is down, each FortiGate starts with different IP
addresses in their port3:
Student: 10.0.1.254
Remote: 10.0.1.253
So, while Remote cannot join the cluster, you can connect to its port3 IP address via SSH and run
the debug commands
Stop and Think

After the Remote FortiGate joins the cluster, you will notice that you cannot access the
Remote FortiGate using the IP address 10.0.1.253 anymore. Can you explain why?

DO NOT REPRINT  Appendix A: Additional Resources
© FORTINET
Appendix A: Additional Resources
Training Services http://training.fortinet.com
Technical Documentation http://help.fortinet.com
Knowledge Base http://kb.fortinet.com
Forums https://forum.fortinet.com/
Customer Service & Support https://support.fortinet.com
FortiGuard Threat Research & Response http://www.fortiguard.com

DO NOT REPRINT  Appendix B: Presentation Slides
© FORTINET
Appendix B: Presentation Slides

DO NOT REPRINT  Troubleshooting Concepts
© FORTINET
This lesson is about troubleshooting concepts.

© FORTINET
In this lesson, we will review troubleshooting strategies. We will also introduce some of the
troubleshooting tools available in the FortiGate GUI and CLI.

© FORTINET
Let’s being by reviewing some troubleshooting concepts and strategies.

© FORTINET
Good administrators know their network well before any problem happens. That includes an
understanding of the normal behavior related with traffic volume, network applications, traffic flows
and devices' CPU and memory utilization. So, when a problem happens, good administrators identify
quickly what is behaving abnormally. This information speeds up the troubleshooting process and
helps to isolate the cause of the problem.
Many tools can be used to gather statistics and information while the network is operating normally:
SNMP, logging, sFlow, and the monitors located in the FortiGate GUI.

© FORTINET
It is also important to keep the network documentation up-to-date. Network diagrams should include
physical connections, interface names and subnets. Good network documentation also includes
change control records to track any change in the network: Who did the change? When was done?
What was changed?

© FORTINET
If a problem happens, the first step is to define it well. For example, if the problem definition is “web
filtering is not working”, the scope of the problem is too imprecise. Too many things could cause this.
This makes troubleshooting slow. So, we must ask questions to understand the details: Is the
problem happening with one web site? Is it happening with all users? Is it happening randomly? How
can you reproduce the problem?
After answering the right questions, we can define the problem with details. For example: “the web
filtering is not blocking the web site X for the user Y”. This provides a better place to start the
troubleshooting.

© FORTINET
A general approach for troubleshooting network issues is to follow the TCP/IP model and work the
problem either from the highest layer to the bottom or from the lowest layer to the top.
In the first method you check the physical layer first. If a layer operation is ok, you move to the upper
layer, until you find the layer where the problem is happening.
In the second method you check the application layer first, if a layer is not working properly you move
to the layer below to rule out issues in the lower layers.

© FORTINET
During the second part of this lesson, we will review some of the troubleshooting tools available in
the FortiGate GUI.

© FORTINET
The dashboard is the FortiGate GUI welcome screen. Some of its widgets contain information useful
for troubleshooting, such as the system resources and the alert message console widgets.

© FORTINET
Remember that the dashboard is customizable. Widgets can be added, removed and customized.

© FORTINET
In addition to the dashboard’s widgets, the GUI includes some monitor screens for specific features.
For example, the IPsec monitor displays the status of each IPsec tunnel. The firewall monitor shows
the list of authenticated users. Another example is the routing monitor, which lists the routes that
have been loaded (active) the routing table.

© FORTINET
Another important tool for troubleshooting is the FortiGate logs. The log viewer includes a filter
setting that is used to display only the logs entries related with one specific user name, IP address,
URL or event type.

© FORTINET
This table illustrates the expected log behavior depending on the different logging settings.
The first column shows the possible values for the log setting in the firewall policies: no log, log
security events, or log all sessions.
The second column indicates if the AV, web filtering or antispam profile log setting is enabled or
disabled. Remember, DLP and IPS profiles always generate logs in the security log section.
The last column shows the behavior. If you enable any profiles on your policy and logging is not
enabled, you will not get logs of any kind—even when profile is blocking traffic. So if you apply a
security profile, it’s important to consider the logging settings.

© FORTINET
From the information stored in the logs, a FortiGate device with hard disk can generate reports, either
manually or on an automatic schedule.

© FORTINET
Reports in FortiGate devices are fully customizable.

© FORTINET
Let's now introduce some CLI troubleshooting tools.

© FORTINET
The real time debug commands generate information in real time about what a specific FortiGate
process or feature is doing.
The debug level is a bitmask value that specifies which types of messages are displayed. This
depends on each process. For all the cases, although, a debug level of ‘0’ means no output
(disabled), and a debug level of '-1' means enabling all possible message types.

© FORTINET
For example, this slide shows the two commands for enabling all the IPsec real time debug output.
You can also enable the option to prepend the system time to each debug line.
It is important to disable any real time debug after using it. They consume FortiGate resources and
some could be CPU intensive.

© FORTINET
On the other hand, the application layer test commands do not display information in real time, but
statistics and configuration information about a feature or process. Some of these commands can
also be used to restart a process or execute a change in its operation.

© FORTINET
Some CLI debug commands generate a lot of output. If we know that the required information is
contained in one specific line of the output, and if we know a keyword to find that line, we can use the
GREP utility. This utility displays only the lines from the output that match a text string. For example,
in this slide we are using the GREP utility to find the IP address 10.0.0.7 in the ARP table. We use
the debug command ‘diagnose ip arp list’ to get the ARP table, and then we append the GREP utility
to display only the information for one IP address. In this way, the output is only one line (with exactly
the information we need), instead of a long list of entries.

© FORTINET
When displaying the FortiGate configuration via CLI, you can use the GREP utility with the option –f.
It will display only the configuration sections (or tables) where the text string matches at least one
value. This is useful, for example, to find all the references in the configuration to a specific object. In
this slide, we are using the –f option to find all the references to the wan1 interface. The output shows
only the two tables where wan1 is referenced: the definition of the interface itself, and a firewall policy
where wan1 is the destination interface.

© FORTINET
If you need to expand your FortiGate skills, or if you need more information about troubleshooting a
FortiGate issue, these are some of the resources to get more information and help.

© FORTINET
To review, these are the topics that we just talked about.

© FORTINET
In this lesson, we will show you how to troubleshoot system resources on FortiGate.

© FORTINET
You will learn some debug commands for troubleshooting FortiGate system problems, such as high CPU
or memory usage. The lesson also covers new firmware installations from the BIOS, memory optimization
and crashlog diagnostics.

© FORTINET
Let’s see first some general system troubleshooting commands.

© FORTINET
This is usually one of the first debug commands when troubleshooting. The output shows the firmware
version, FortiGuard database versions, license status, operation mode, number of VDOMs and system
time.

© FORTINET
The command 'get system performance status' shows the overall memory and CPU utilization. It also
shows session creation rate, number of viruses caught and number of attacks blocked by the IPS. The last
line displays the system uptime. This output gives a quick overlook at how much traffic the unit is handling.

© FORTINET
During this part of the lesson you will learn how to check the use of he FortiGate memory and CPU.

© FORTINET
To understand how a FortiGate uses its memory, we need to understand the architecture of FortiOS.
The hearth of FortiOS is its kernel. Here, the unit takes some of the most important and basic decisions,
such as how to route a packet, or when to offload a session to a NPU processor.
FortiOS runs over hardware. The device drivers bridge the kernel with the hardware.
Above the kernel, there is the user space. Several application processes or daemons run at this level.
Above all that, there is the configuration layer, which is basically composed of two modules: the command
line interface and the graphical user interface.

© FORTINET
How the memory is segmented depends on the FortiGate model. There are two cases: first, models
running 32-bit FortiOS with more than 1 GB of memory; second, models running 64-bit FortiOS and
models running 32-bit FortiOS with less than 1 GB of memory.
Let’s see the first case.
When the FortiOS is 32 bits and the system memory is more than 1 GB, the kernel cannot access directly
the whole memory space. So, memory paging is used to reach the portion of the memory that cannot be
accessed directly. The part of the memory that the kernel can access directly is called low memory. The
part of the memory that is accessed using memory paging is called high memory. The command
'diagnose hardware sysinfo memory' displays:
- The total amount of low memory (LowTotal)
- The amount of free low memory (LowFree)
- The total amount of high memory (HighTotal)
- The amount of free high memory (HighFree)
- The total amount of system memory (MemTotal = LowTotal + HighTotal)
- The total amount of free memory (MemFree = LowFree + HighFree)

© FORTINET
For models running 32-bit FortiOS with less than 1 GB, and for models running 64-bit FortiOS, the kernel
doesn't need to use memory paging to access the whole memory space. In those cases the command
'diagnose hardware sysinfo memory' shows a size of 0 kB for the total high memory and free high
memory. All the memory space is considered low memory.

© FORTINET
FortiGate allocates memory for five main purposes:

- Kernel memory slabs
- System I/O cache
- Buffers
- Shared memory
- Process memory
We will see each of these in the next slides.

© FORTINET
The kernel memory slabs are collections of objects with a common purpose. They are used by the kernel
to store information in the low memory.
This slide shows example of some slabs. There are slabs for storing information about the TCP sessions.
The entries in the route cache are also stored in low-memory slabs.

© FORTINET
(slide contains animation)

To check how much memory is being allocated to kernel slabs, we use the command 'diagnose hardware
sysinfo slab'.
The first column shows the slab name. The second column the total amount of active objects,
(click)
then the amount of available objects,
(click)
and the size of each object.
The total amount of memory allocated to each slab type can be calculated by multiplying the number of
available objects by their size.

© FORTINET
The system I/O cache is used to speed up the access to information stored in the hard and flash disk
memories. Some processes, such as logging, WAN optimization, and explicit proxy, store information in
the hard disk, so they get the performance boost provided by this memory allocation.
An I/O cache page is labeled as active when it has been recently accessed. It goes to inactive state after
not been used for some time. An inactive page can be reclaimed by the kernel if needed.

© FORTINET

The command 'diagnose hardware sysinfo memory' displays the total amount of memory allocated for
I/O cache.
(click)
It also lists the amount of memory allocated for buffers.

© FORTINET
As we explained, above the kernel layer there are multiple application processes or daemons running. The
operating system allocates separated blocks of memory for each process. One process can access the
memory that was allocated to it, but it cannot access any memory that was allocated to a different process.
So, a process cannot share information with another process by reading or writing data into the memory
allocated to that other process. For that purpose, the operating system dynamically allocates shared
memory (SHM). The SHM can be accessed by multiple processes, allowing the sharing the information
among them.

© FORTINET
We can check how much memory space is being used by each process. The command 'diagnose sys
top' shows that information in the last column. The command also displays, for each process: its ID
number, its state, and the amount of CPU using. You can specify the refresh frequency and the number of
lines to display.
While the command is running, you can press <shift-P> to sort the processes by CPU usage, of <shift-M>
to sort them by memory usage. To stop the command, use <ctrl-C>.

© FORTINET
Another useful command for displaying information about process memory usage is 'diagnose sys top-
summary'. The –h option displays the different options available for this command.

© FORTINET

For example we can use the option –s mem to sort the processes by memory usage; the option –i 60 to
refresh it every 60 seconds; and the option –n 10 to display the top 10 processes.
One advantage of this command over the previous one ('diagnose sys top') is that it shows the total
amount of memory allocated by all forked or child processes, including shared memory. In the case of
'diagnose sys top', each child process is displayed separated. Here, we have an aggregated view of all of
them.
(click)
The output also shows the amount of file descriptors allocated. If the amount of any of the descriptors
keeps constantly increasing, it might indicate that there is a memory leak problem.
(click)
If a process has forked, the number of child processes is displayed after its name.

© FORTINET
This table shows some of the most common processes.

© FORTINET
This other table shows more about the most common processes.

© FORTINET
The command 'diagnose sys top' shows the state of each process. A process can be in one of four
states: Sleeping (S), running (R), do not disturb (D) or zombie (Z).
The S and R states are normal. It is also normal if a process goes briefly to D state.
The Z state is not normal. Also, it is not normal if a process stays in D state for a long time. These usually
indicate that the process is not working properly.

© FORTINET
During this part of the lesson you will learn about conserve mode.

© FORTINET
There are 2 different types of conserve modes:

- Kernel conserve model is triggered when the amount of free low memory is running low
- Proxy conserve mode (also called system conserve mode) is triggered when the amount of free overall
memory is running low.
The command 'diagnose hardware sysinfo shm' can be used to determine the conserve mode status. If
the field conservemode is 1, the unit is in proxy conserve mode. If it is 2, the unit is in kernel conserve
mode.

© FORTINET
Two margins or thresholds determine when the FortiGate enters and exists the kernel conserve mode. The
margins depend on the total amount of low memory.
When a FortiGate is in kernel conserve mode, any proxy inspection is bypassed and administrators cannot
change the unit configuration.

© FORTINET
These are the entries generated in the crashlog, event logs and alert message console when a FortiGate
enters to and leaves the kernel conserve mode.

© FORTINET
Similarly to kernel conserve mode, two margins or thresholds determine when the FortiGate enters and
exists the proxy conserve mode. The margins depend on the total amount of overall memory.

© FORTINET
These are the entries generated in the crashlog and event logs when a FortiGate enters to and leaves the
proxy conserve mode.

© FORTINET
av-fail-open is the CLI setting that controls FortiGate’s behavior while it is in proxy conserve mode.

© FORTINET
An option related to fail-open, is av-failopen-session. This is a setting that kicks in not during a high
memory situation, but when a proxy on the FortiGate runs out of available sessions to process the traffic.
If av-failopen-session is enabled, then FortiGate will act according to the av-failopen setting. Otherwise,
by default, it will block new sessions until proxy connections become available.

© FORTINET
Additionally, FortiGate has one more mechanism to free memory when there is not much available. If the
kernel cannot allocate more memory pages, it deletes the oldest sessions. The command diagnose sys
session stat shows the numbers of sessions that has been deleted by the kernel due to this mechanism.

© FORTINET
During this part of the lesson you will learn how to optimize the memory usage.

© FORTINET
Many FortiGate processes, such as DLP or AV scanning, are memory intensive. So, memory optimization
is important, especially in small units, to guarantee that they will stay away from conserve mode. This slide
shows some recommendations for optimizing the memory usage. These tips might increase significantly
the available memory in a device that is frequently going to conserve mode.
The first and most logical step is to disable the features that are not required. For example, if the network
already has a FortiMail doing antispam, an administrator does not need to do antispam in the FortiGate.
Also, usually not all the IPS signatures are required.
Another important recommendation is to reduce the maximum file size to inspect, which is set to 10MB by
default. This value can be reduced to 2 or 3 MB without reducing significantly the virus caching rate.

© FORTINET
Additionally, you can reduce the amount of allocated memory to some caches, such as the ones for
FortiGuard and DNS.

© FORTINET
The FortiGate session table, especially in networks with high traffic, can consume an important portion of
the memory. By default, a session without traffic remains in the table for up to 1 hour. Although this high
time to live (TTL) might be required by a few applications, in most of the networks, it can be substantially
reduced to, for example, 5 minutes. So, FortiGate will age out idle sessions much quicker, increasing the
amount of available memory. There are four places in the FortiGate configuration where the session TTL
can be changed. Two of them are:
- Globally for all the traffic
- Per IP protocol and port number

© FORTINET
The other two places where the session TTL can be changed are:
- Per firewall policy
- Per application control
If there is one specific application that requires a high session TTL, you can, for example, reduce the TTL
globally to 5 minutes, and set it to 1 hour only for the specific application port number, or firewall policy.

© FORTINET
In the case of TCP, there are other session timers. In most of the cases, they can also be reduced from
their default values without causing problems with the applications. The slide shows some recommended
values (equal to or below the default ones) to optimize the memory usage.
We will see what each of those timers are in the next slide.

© FORTINET
The tcp-halfopen-timer controls for how long, after a SYN packet, a session without SYN/ACK remains in
the table.
The tcp-halfclose-timer controls for how long, after a FIN packet, a session without FIN/ACK remains in
the table.
The tcp-timewait-timer controls for how long, after a FIN/ACK packet, a session remains in the table. A
closed session remains in the session table for a few seconds more to allow any out-of-sequence packet.

© FORTINET
To finish this lesson, we will talk about hardware and firmware troubleshooting.

© FORTINET
Let’s talk about the crashlog. Each time an application crashes, or closes, an entry is generated in the
crashlog. For the case of crashes, the entry contains information about the name of the application, the
time it crashed and the termination signal.
This slide shows a sample of a crash in the crashlog. In this case the application that failed was the
sslvpnd process, which manages SSL VPN connections. The termination signal is 11, which is
segmentation fault.

© FORTINET
This table contains the most common termination signal numbers. Any administrator can manually kill a
process by using the command 'diagnose sys kill', followed by the process ID and the termination signal
number. The command 'diagnose sys top' lists the process ID numbers. Manually killing a process is not
usually required under normal circumstances. Be careful although if, for some reason, you have to do it.
Improperly killing a process might make a FortiGate system unstable.

© FORTINET
So, how de we know if the crashlog is normal or not?.

In most of the cases, entries in the crashlog are normal. A crashlog entry can be considered suspicious if it
happens at the same time than a FortiGate feature that failed or an abnormal FortiGate behaviour.
For example, a crashlog entry generated when the unit unexpectedly reboots might provide information
about the cause. A crash in the sslvpnd process when all SSL VPN users got disconnected is also
relevant. The crashlog includes information that can be used by Fortinet development to determine which
code triggered the problem and the details about the crash.

© FORTINET
If a FortiGate that is rebooting unexpectedly, this is what an administrator can do:

First, check the logs and the crashlog.
Second, a crashdump message is usually generated through the console port when the unit crashes. This
crashdump messages can provide useful information to the Fortinet developers. To capture any
crashdump, keep a laptop connected to the console port, run the commands in this slide, and wait until
another crash happens.

© FORTINET
We say that a FortiGate freezes when stops handling traffic and there is no way to connect to it (there is
even not access to the unit's console port). Only power cycling fixes the problem. For those cases, you
could similarly capture any crashdump in the console port. Additionally, and in the case of models with
more than one CPU, you can enable the nmi-watchdog feature, which crashes the system (and forces
the crashdump) when no more new daemons have been scheduled in the last 10 minutes - This is an
indication that the unit is not operating normally and might have frozen.
Some FortiGate models also have an external NMI button. If the unit is frozen and no crashdump has been
generated, you can press it to force a crash and get a crashdump.

© FORTINET
The FortiGate BIOS menu allows administrators to do some operations over the flash memory and the
firmware images. To access the BIOS menu you must reboot the unit while connected to the console port.
The console port will show the booting process and, at one point, it will show the message:
“Press any key to display configuration menu”
If you press any key while this prompt is displayed, the booting process is interrupted and the BIOS menu
is displayed.

© FORTINET
The options available in the BIOS menu vary depending of the BIOS version and hardware model.
However, in most of the cases you will see an output similar to the one in this slide.

© FORTINET
By pressing F we can format the flash memory.

This might be required if the firmware got corrupted or if the administrator wants to do a clean installation
of a new firmware. Keep in mind, although, that formatting the flash deletes any information stored, such
as firmware images, configuration and digital certificates.

© FORTINET
Fallow these steps to install a firmware image from the BIOS.

© FORTINET
After that, go to the BIOS menu and select the option G.

The BIOS will ask for:
- The IP address of the TFTP server (configure the TFTP server with this IP address)
- The FortiGate IP address (it must be in the same class-C subnet than the TFTP server)
- The name of the firmware image file stored in the TFTP server
If everything is ok, you should see a series of pound signs, indicating that the unit is downloading the
image. The BIOS will then verify the integrity of the file and, in some models, will give you these three
options:
- Save it as the default firmware
- Save it as the backup firmware
- Run the image without saving it

© FORTINET
There is a hardware test images (HQIP) for each model. You can install it by using the BIOS menu and a
TFTP server. You can run it without saving it to the flash. The HQIP runs some hardware diagnostics to
test the NIC interfaces, hard disk, flash disk and other hardware components.

© FORTINET
These are the topics covered in this lesson.

© FORTINET
This lesson is about network troubleshooting.

© FORTINET
You will learn to use some general network troubleshooting tools, the built-in sniffer and the debug flow.
The lesson also shows how to display and analyze the information in the FortiGate session table.

© FORTINET
The first part of this lesson introduces some general network troubleshooting tools.

© FORTINET
This is one of the most useful commands for troubleshooting layer-1 problems. It displays the link status,
negotiated speed and negotiated duplex mode. It also shows counters related with bytes, packets and
errors received and transmitted. The output of this command varies depending on the FortiGate model and
NIC driver version.

© FORTINET
This table contains some of the fields in the output of the 'get hardware nic' command. Not all these are
displayed for the same NIC. As we explained, the output depends on the model and NIC driver version.

© FORTINET
This other table contains more fields that might be present in the output of the 'get hardware nic'
command.

© FORTINET
To check the ARP table, use the command 'get system arp'. It displays the IP addresses, MAC addresses
and interfaces where the devices are connected. It also shows the age, which is for how long there has not
been traffic. After a predefined time, an entry in the ARP table without traffic is aged out.

© FORTINET
Two of the most common troubleshooting tools are the 'ping' and the 'traceroute'. Remember from our
NSE4 training that the options for the ping can be changed with the command 'execute ping-options'. They
include the packet size, amount, timeout and the source IP address for the ping.

© FORTINET
The CLI also includes a telnet client, which cannot only be used to connect to a remote telnet device, but
could also be used to test the connectivity to an application server. For example, in this slide we are testing
the connectivity from the FortiGate to a SMTP server by doing telnet to the server's port 25.

© FORTINET
The command 'get system performance firewall statistics' displays aggregated statistics per network
application.

© FORTINET
The command 'get system performance firewall packet-distribution' displays number of bytes and
packets per packet size range.

© FORTINET
In the last part of this lesson, we will talk about the FortiGate session table, the built-in sniffer and the
debug flow.

© FORTINET
The FortiGate session table, as explained in the NSE4 training, contains detailed information about every
IP connection crossing or terminating at the FortiGate. The command 'get system session status' displays
the total number of sessions in the active VDOM. The command 'get system session list' provides a brief
summary of each session, one session per line, with the most relevant information, such as protocol, and
source and destination IP address and port. We can use the GREP utility with this command, for example,
to list only the sessions for a specific IP address.

© FORTINET
To display detailed information about the sessions, we use the command 'diagnose sys session list'. It is
recommended to set the session filter first, as an unfiltered output displays all the details about all the
existing sessions (which could be in order of thousands, or even millions, for high-end devices). You can
filter the output, for example, by policy ID, or by source or destination IP address and port.

© FORTINET
Some configuration changes, such as security profile or session helper changes, apply only to new
sessions. For those cases, existing sessions keep using the old configuration until they expire or are
terminated. This is important when troubleshooting problems. For example, after a change in a security
profile, you should clear any related existing session and generate new ones. Use the command 'diagnose
sys session clear' to remove all sessions that match the session filter. You must although be careful with
this command as it could potentially clear all the existing sessions if no filter has been set. Double check
first the session filter with the command 'diagnose sys session filter'.

© FORTINET

The slide shows a sample of the information contained in the FortiGate session table. It includes the IP
protocol number,
(click)
the protocol state (we will see what this value is in the next slides),
(click)
how long for the session to expire (if there is not more traffic),
(click)
traffic shaping counters,
(click)
session flags,
(click)
received and transmitted packet and byte counters,
(click)
if the unit is doing NAT - this portion shows the type of NAT (source or destination) for each traffic
direction, and the NAT IP address
(click)
the ID of the matching policy,
(click)
and counters for hardware acceleration.

© FORTINET
The protocol state in the session table is a two-digit number. In the case of TCP, the first number is related
with the client-side state, and is different than 0 when the session is being handled by a FortiGate proxy
(for example, when the FortiGate is doing proxy-based inspection). The second digit is the server-side
state and is used to know the status of the TCP session. This table and the flow graph correlate the
second digit value with the different TCP session states. For example when the FortiGate receives the
SYN packet, the second digit is 2. It goes to 3 once the SYN/ACK is received. After the three-way
handshake, the state value changes to 1.
When a session is closed by both sides, the FortiGate keeps it in the session table for a few seconds
more, to allow any out-of-order packet that could arrive after the FIN/ACK packet. This is the state value 5.

© FORTINET
For case of UDP, the session state can only have two values: 00 when traffic has been only one way, and
01 once there is traffic two ways. For the case of ICMP the protocol state is always 00.

© FORTINET
This table contains the meaning of the most important session flags. For example the log flag indicates
that the session is being logged. The local flag indicates that the session either was originated from the
FortiGate or is terminating in the FortiGate.

© FORTINET
Let’s give more details about the dirty and may dirty flags. When the FortiGate receives the first packet for
a new session (for example, a SYN packet), the unit evaluates if the traffic should or should not be allowed
based on the firewall policies. As long as there are not changes in the firewall policies, this evaluation is
done only on the first session packet. If the traffic is allowed by a firewall policy, the unit creates a session
and flags it as may_dirty.
After that, if there is a change in the firewall policy configuration, all the existing sessions with the
may_dirty flag are also flagged as dirty. This indicates to the FortiGate that it needs to reevaluate the next
session packet to know if the session must now be blocked. If the session is still allowed, the dirty flag is
removed (the may_dirty flag is kept). If the session must be blocked, it is flagged as block and remains in
the session table until it expires. Any packet matching a session with the block flag is dropped.

© FORTINET
We talk about the built-in sniffer in NSE4. This slide reviews this tool. There are 6 verbosity levels. The
table shows what information is displayed in each level. We usually use level 4 to check how the traffic is
flowing and if the FortiGate is not dropping packets. We also use level 3 or 6 to convert the output to PCAP
format, which can be later analyzed with a tool such as WireShark.

© FORTINET

To sniffer the traffic in all the interfaces, use the any keyword as the interface name.
(click)
If you do not specify any option for the timestamp, the debug shows the time, in seconds, since it is
running. As explained in the previous slide, you can instead prepend the local system time to easily
correlate a packet with another recorded event.
(click)
After stopping the sniffer by pressing Ctrl-C, it is important to check the amount of dropped packets. If
there were dropped packets during the sniffer, it means that not all the traffic matching the sniffer filter
could be captured. So, you might need to capture the traffic again, next time using a stricter filter, or
reducing the traffic to capture.

© FORTINET
Another useful FortiGate troubleshoot tool is the debug flow. We also talk about these commands in NSE4,
so we will quickly review them. The debug flow is also called ‘internal sniffer’ as it works similarly to the
built-in sniffer. But, in this case, the output shows step-by-step, and with details, what the kernel is doing
with each packet.

© FORTINET

This is a sample of a debug flow output. Here we have captured the 3 packets of a TCP 3-way handshake.
The output for the SYN packet shows when the kernel creates a new session (with its session ID), finds
the route to the destination and applies NAT. It also shows the ID of the policy that matches this traffic.
(click)
The output of the SYN/ACK and ACK packets also shows the session ID and NAT information.
This tool is useful for many troubleshooting cases, for example when we need to understand why a packet
is taking a specific route, or why a specific NAT IP address is being applied.

© FORTINET
The debug flow is also useful when the FortiGate, for some reason, is dropping packets. In those cases it
usually shows an error message explaining why a packet was dropped.
These are three common messages that we might see in a debug flow output.
The first one means that either there is not firewall policy allowing the traffic or that a disclaimer has not
been accepted yet.
The second one indicates that the IP address has been quarantined by the DLP inspection.
The third one means that the packet was dropped due to a traffic shaper that has exceeded one of its
thresholds.

© FORTINET
These are two more common debug flow error messages. The first error indicates that the packet failed
the reverse path forwarding check.
The second error message is usually because one of these reasons:
if the packet is destined to a FortiGate IP address (for example, management traffic), it might be that the
service is not enabled, is using a different port, or the source IP address if not included in the trusted list.
On the other hand, if the packet is not destined to a FortiGate IP address, but to a device on the other side
of the unit, there might be a virtual IP or IP pool that is wrongly using that IP address. So, check your
virtual IP or IP pool configuration.

© FORTINET

© FORTINET
This lesson is about firewall policy troubleshooting.

© FORTINET
During this lesson, you will learn about how the FortiGate translates the source port when doing
source NAT. The lesson also covers NAT exhaustion, SIP and session helper debug commands.
During the last part of the lesson, you will learn some debug commands for troubleshooting traffic
shaping.

© FORTINET
This part of this lesson is about NAT and NAT exhaustion problems.

© FORTINET
This slide shows the different FortiGate NAT methods. The differences between each of them are
explained in NSE4.
FortiGate can translate the source IP address of an outgoing connection (source NAT or SNAT).
Three different features can be used for that purpose:
- Policy NAT: Overload NAT of many to one
- IP pool: Four types of IP pools – Overload, one-to-one, fixed port range and port block allocation
- Central NAT table
FortiGate can also translate the destination IP address of an incoming connection (destination NAT or
DNAT). Two different features can be used for that purpose:
- Virtual IP (VIP)
- Load Balancer

© FORTINET
For policy NAT and overload IP pool, the number of available NAT connections depends on four
variables:
- IP protocol
- NAT IP address
- Destination IP address
- Destination port
For each NAT IP address, there are up to 60,416 TCP and 60,416 NAT UDP ports available.
For each TCP or UDP connection, two indexes are created to match the traffic and determine the NAT
IP address and port. We will see examples of these indexes in the next slides.

© FORTINET

The slide shows two clients and two servers. There is a FortiGate in the middle doing NAT of the
subnet 172.16.1.0/24 to the IP address 10.10.1.254.
So, if there is a connection, for example, from 172.16.1.1 to the HTTP server 10.10.1.1, the index that
identifies the outgoing traffic includes this information:
Source IP: 172.16.1.1 (client IP). Destination IP: 10.10.1.1 (server IP). Protocol: TCP. Source port:
10000 (random source port used by the client). Destination port: 80 (HTTP port number)
The second index identifies the incoming traffic:

Source IP: 10.10.1.1 (Server IP). Destination IP: 10.10.1.254 (NAT IP). Protocol: TCP. Source port: 80
(HTTP port number). Destination port: 46372 (NAT port selected by the FortiGate)
(click)
If there is now a connection from 172.16.1.2 to the HTTPS server 10.10.1.2, the first index is:
Source IP: 172.16.1.2 (client IP). Destination IP: 10.10.1.2 (server IP). Protocol: TCP. Source port:
25000 (random source port used by the client). Destination port: 443 (HTTPS port number)
And the second index is:

Source IP: 10.10.1.2 (Server IP). Destination IP: 10.10.1.254 (NAT IP). Protocol: TCP. Source port:
443 (HTTPS port number). Destination port: 46372 (NAT port selected by the FortiGate)
In this case is that, as the server IP addresses and ports are different, the sessions can share the
same NAT port. So in both cases the source IP and source port are translated to the same NAT IP
and NAT port.

© FORTINET
So, although both sessions are using the same NAT IP and NAT port, the FortiOS can still use the
indexes to uniquely identify each session.
FortiOS only has to ensure that the chosen NAT port combined with the other four variables are
unique to identify the session.

© FORTINET

So, let see now the case of two users (172.16.1.1 and 172.16.1.2) connecting to the same HTTP
server (10.10.1.1). The FortiGate is again translating 172.16.1.0/24 to 10.10.1.254.
The second index (incoming traffic) for the first client contains this information:
Source IP: 10.10.1.1 (Server IP)
Destination IP: 10.10.1.254 (NAT IP)
Protocol: TCP
Source port: 80 (HTTP port number)
Destination port: 46372 (NAT port selected by the FortiGate)
(click)
As the second client is using the same NAT IP and going to the same destination IP and TCP port
number, the FortiGate must use a different NAT port. The same NAT port cannot be shared by both
sessions. So, the second index for the second client includes:
Source IP: 10.10.1.1 (Server IP)
Destination IP: 10.10.1.254 (NAT IP)
Protocol: TCP
Source port: 80 (HTTP port number)
Destination port: 34020 (different NAT port selected by the FortiGate)
In other words, the source IP, destination IP, protocol and destination port are the same. So, FortiGate
must use different NAT ports to differentiate each session.

© FORTINET
We can calculate the theoretical maximum number (without exhausting the number of NAT ports
available) of simultaneous connections for one NAT IP and one destination IP.
If we multiply the number of NAT IP addresses (1 in our case), by the number of NAT ports per NAT
IP (60,416), by the number of protocols (2, TCP and UDP), by the number of destination IP addresses
(1 in our case) and by the maximum number of destination ports (65,535), we get almost 8 billion
possible connections.

© FORTINET
Although that high number is theoretically possible, in practice, real numbers are much smaller. One
reason is that servers never use all 65,535 possible ports. Indeed, most of the connections to a server
go to the same port number and IP protocol number. For example, all the connections to a HTTP
server are TCP and usually go to port 80 only.

© FORTINET
So, if we calculate again the maximum number of simultaneous connections, this time considering one
IP protocol number, and one destination port number, we get 60,416. This is a more realistic number.

© FORTINET
Some network applications do not support source port address translation. For those cases, enable
Fixed Port so the FortiGate translates only the source IP address and not the source port.

© FORTINET
If at one point the FortiGate does not have any available NAT port for a new connection, the traffic is
rejected and a log is generated.

© FORTINET
This part of this lesson is about session helpers and application layer gateways.

© FORTINET
To understand what a session helper does, let’s see an example of a network protocol that might have
problems when there is a network device doing NAT. The example that we will see is the FTP
protocol, specifically, when working in active mode.
Any FTP file transfer is composed of two TCP sessions: one for the control channel and one for data
transfer. The control channel is always initiated from the client to the server and is used to send the
different FTP commands. The FTP commands allow the client to move through the different server
folder, specify the type of file transfer and initiate the data channel for uploading or downloading a file.
FTP has two modes: active and passive. The mode determines who initiate the data channel. In the
case of passive mode, the data channel is initiated by the client.
In the case of active mode, the client sends the port command through the control channel. The
command includes the client IP address and the TCP port for the incoming data channel. Then, it is
the server who initiates the TCP session to the IP address and port number specified by the port
command.

© FORTINET

So, active FTP will not work if the control channel crosses a network device doing NAT that does not
have a session helper. In this example a FTP client is connecting to an active-mode FTP server.
There is a router in the middle doing NAT of the client IP address 10.0.1.10 to the NAT IP address
10.200.1.1.
(click)
After the control channel is up, the client sends the port command with its IP address 10.0.1.10 as the
destination for the data channel.
When that FTP packet crosses the router, the source IP address in the IP header is changed from
10.0.1.10 to 10.200.1.1. However the IP address in the FTP port command is not translated to
10.200.1.1.
(click)
Once the server receives that FTP command, it tries to bring up the TCP session for the data channel.
It sends the SYN packet to the IP address 10.0.1.10, which is most probably not routable as it is a
private IP behind a device doing NAT.
The file transfer fails.

© FORTINET

The FTP session helper fixes this problem. Let’s replace the router by a FortiGate. This is what the
FortiGate session helper does.
(click)
The packet with the FTP port command arrives to the FortiGate.
(click)
The FortiGate not only translates the source IP address in the IP header, but the session helper also
translates the IP address inside the FTP port command. If the source port is also translated in the
TCP header, the session helper also does the same in the port command.
(click)
Another important function of the session helper is to temporally create an expected session (or pin
hole) for the data channel connection that will come from the server. So, that means that the
administrator does not need to manually create firewall policies to allow these incoming TCP sessions
(which use random TCP ports numbers). The session helper automatically creates the session and
opens the door for the incoming connection on the fly.
(click)
After that, the server connects the data channel to the right IP address 10.200.1.1.
(click)
That incoming TCP connection is allowed by the expected session previously created by the session
helper, even when there is not firewall policy allowing it.

© FORTINET
This slide shows a packet capture of the previous FTP traffic before the port command reaches the
FortiGate. We see the original client IP address 10.0.1.10.

© FORTINET
This other slide shows another packet capture, this time after the port command crosses the
FortiGate. The session helper has translated the IP address inside the port command to 10.200.1.1.

© FORTINET

SIP is another example of a protocol that requires a session helper in a NAT environment. Similarly to
FTP, SIP uses control channels and data channels. In the case of SIP, 4 data channels, 2 per each
traffic direction, are required for each call. In this slide, there are two SIP phones with the IP
addresses 10.0.1.10 and 172.168.100.205. Additionally, a FortiGate is doing NAT of 10.0.1.10 to
66.171.121.44.
(click)
Once the control channel is up, a SIP phone sends an invite packet with its IP address and port
numbers for two of the four data channels.
(click)
The FortiGate session helper creates two expected sessions,
(click)
and translates the IP address inside the invite packet to 66.171.121.44.
(click)
The remote phone sends now an OK packet to the right destination IP address (66.171.121.44). The
packets includes the IP address and ports for the other two data channels.
(click)
The session helper creates two more expected sessions, this time using the information coming in the
OK packet.
(click)
After that, the 4 data channels can be connected through the 4 expected sessions. Again, firewall
policies are not needed to allow this traffic.

© FORTINET
There is actually a way to list the expected sessions created by the session helpers. In this example,
the command lists an expected session to allow traffic from 10.171.121.38 to 10.0.1.10, port TCP
50365.

© FORTINET
The debug flow shows the name of the session helper (if any) that is inspecting the traffic. In this case,
it is the FTP session helper.
Also, for traffic that matches an expected session previously created by a session helper, the debug
flow shows the message: 'find an EXP session'.

© FORTINET
There are other protocols that, in some circumstances, also require a session helper, for example
PPTP, H323 and RSH. We can list the active session helpers by typing the command 'show' after
'config system session-helper'. The output lists the TCP or UDP port numbers that each session
helper is listening to. If one of those protocols is using a different port number, you need to change the
FortiGate configuration to match it.

© FORTINET
For SIP traffic inspection, the FortiGate also includes a feature smarter and more versatile than the
SIP session helper. It is the SIP application layer gateway (SIP ALG).
The SIP ALG has all the same functions than the SIP helper, but provides more features.
Another difference is that the all session helpers run in the kernel. SIP ALG, on the other hands, runs
as a user space process.

© FORTINET
A FortiGate uses either the SIP helper or the SIP ALG depending on the configuration. The system
setting 'default-voip-alg-mode' specifies which one is used when no VoIP profile is applied. If it is set
to 'proxy-based' (default), SIP ALG is used. If it is set to 'kernel-helper-based', SIP helper is used.
If the SIP traffic matches a firewall policy with a VoIP profile, SIP ALG is always used regardless of the
'default-voip-alg-mode' setting.
Fortinet recommends the use the SIP ALG. The SIP helper should be used only as an alternative
when, for some reason, the SIP ALG is not working as expected.

© FORTINET
These are the commands to change the ports for the SIP ALG. The SIP ALG supports SIP over UDP,
SIP over TCP and encrypted (SSL) SIP.

© FORTINET
The command 'diagnose sys sip-proxy calls list' displays all the active SIP calls.
The command 'diagnose sys sip-proxy calls clear' disconnects all the active SIP calls.

© FORTINET
There are two real time debugs that display information about SIP traffic:
diagnose debug application im 31
and
diagnose debug application sip <debug_level>

© FORTINET
The last part of this lesson is about traffic shaping.

© FORTINET
Three QoS techniques are available in FortiGate:

- Traffic policing
- Traffic queuing
- Traffic shaping

© FORTINET
Traffic policing limits the amount of traffic ingressing or egressing an interface. A packet is allowed
only if the traffic rate is below the threshold for the respective traffic direction.

© FORTINET
Traffic queueing assigns packets to one of different queues, depending on the ToS field of the IP
header. Each queue has different priorities: high, medium and low.

© FORTINET
Traffic shaping can:

- Guarantee bandwidth
- Limit bandwidth
- Increase or decrease traffic priority

© FORTINET
Two types of traffic shapers: Shared and Per-IP.

A shared shaper applies aggregated bandwidth limits to all traffic using that shaper. The scope can
be per-policy or for all policies referencing that shaper.
A per-ip shaper applies the bandwidth limits to each source IP address.

© FORTINET
There are two CLI commands that display the amount of packets dropped by shared traffic shapers:
diagnose firewall shaper traffic-shaper list
diagnose firewall shaper traffic-shaper stats

© FORTINET
For the case of per-ip shapers, the equivalent commands are:

diagnose firewall shaper per-ip-shaper list
diagnose firewall shaper per-ip-shaper stats

© FORTINET

© FORTINET
This lesson is about firewall authentication.

© FORTINET
During this lesson you will learn to monitor the authentication status of the network users. You will also
learn to troubleshoot the most common problems related with local, LDAP and RADIUS
authentication.

© FORTINET
Let’s start with some general authentication troubleshooting commands, tools and tips.

© FORTINET
As in the case of any other type of issue, the first step in troubleshooting authentication problems is to
get the specifics: Is the problem happening with more than one user? Is the problem really related with
user authentication and not with user permissions?
The first place to check is usually the logs. What do they show? Is the username correct? Is the traffic
being blocked after the authentication? What is the user profiled assigned to the user?
In the case of remote server authentication (such as RADIUS or LDAP), check also the logs in the
server side. In those cases the result of the FortiGate authentication depends on the server response.
If the RADIUS credentials are invalid, what do the RADIUS server’s logs show? Is the user account
still active in the RADIUS server?

© FORTINET
This is sample of the user authentication logs. A log can be generated each time a user authentication
action is successful or fails.

© FORTINET
You can check the list of active users from either the user monitor in the GUI or from the CLI. The CLI
command for that purpose is 'diagnose firewall auth list'. You can previously filter the output with the
command 'diagnose firewall auth filter'.

© FORTINET
Any session for traffic coming from an authenticated user contains the authed flag. Additionally, the
username and user group is added to the session information.

© FORTINET
There is a real time debug for troubleshooting problems related with any user authentication method:
local, remote or even FSSO. It is 'diagnose debug application authd'.

© FORTINET
The second part of this lesson is about debug commands and tools for troubleshooting LDAP
authentication problems.

© FORTINET
Let’s start with a quick review of what we learned about the LDAP protocol in NSE4.
The hierarchy of an LDAP schema is not required to hold any resemblance to the organization.
However, generally the name conventions used and the group structure match with the name of the
company and corporate hierarchy very closely.
On the top, we have the root or DC. This is where an LDAP tree always starts, with any schema.
After that, the groups (or branches) are defined using C, OU, and/or O. The exact behavior and
options used depend on the schema and what exactly is being defined. Branches may contain objects
and each object contains attributes. Objects are uniquely identified by their distinguish names (DNs).
The full DN specifies where the object is and the name and value of an attribute that can be used to
find it.

© FORTINET

There are 3 different methods, or bind types, for a FortiGate to access a LDAP server: simple,
anonymous and regular. During this lesson we will talk only about regular bind, which is the most
complex, versatile and commonly used method.
LDAP authentication using regular bind does four steps:
First, the FortiGate logs into (bind) the LDAP server using a LDAP administrator account.
(click)
At this point the FortiGate knows only the user name, but it doesn't know the branch where the user is
located. So during the second step, the FortiGate does a search query in the LDAP database to find
where the user is located. In other words, to find the user’s DN.
If the user is found, the server replies with the user’s DN.
After that the FortiGate logs out (unbind) from the LDAP server.
(click)
The third step is another bind, but this time using the user credentials. The FortiGate sends the DN,
learned in the step before, together with the password.
(click)
The last step is to get the user group information. How this work depends on the type of LDAP server,
but it is generally a LDAP query.
When isolating a LDAP problem, we must find first which of these four steps is failing.

© FORTINET
Most of the LDAP authentication problems are caused by misconfigurations. So, let’s see how to
quickly check if our regular-bind LDAP configuration is ok. We will analyze the case of an LDAP
server based on Windows AD, which is the most common case.
Misconfigurations usually happen in one of these LDAP settings:
- Common name identifier
- Distinguished name
- User DN
- Password

© FORTINET
How do you check if the distinguished name is ok? You can run either of these two commands in the
Windows AD server’s command prompt:
dsquery user –name <full_user_name>
dsquery user –samid <login_username>
The output displays the user DN. The distinguished name setting specifies a parent branch under
which all users are located. The FortiGate searches users in any sub-branch below this parent branch.
For the case in this slide, we can set the distinguished name setting, for example, to:
dc=tac,dc=ottawa,dc=fortinet,dc=com

© FORTINET
The user DN (or bind DN) setting is the full DN of the LDAP admin account. We can use the same
Windows LDAP server command (dsquery) to find that information.

© FORTINET
You can simply copy and paste the full DN output from the server command prompt to the FortiGate
configuration.

© FORTINET

This is a summary of how to properly configure regular bind for Windows AD. A different type of LDAP
server might require a different approach.
First the common name identifier is usually either cn or sAMAccountName. If you set it to CN, users
must authenticate using their full names (ex: John Smith). If you set it to sAMAccountName, users
must authenticate using their login names (ex: jsmith).
(click)
You can get the distinguished name by querying the users DNs with the Windows AD command
dsquery.
(click)
You can get the user DN by querying the admin DN with the same Windows AD command dsquery.
(click)
Finally, the password setting is the LDAP administrator’s password.

© FORTINET
The CLI includes a LDAP authentication test command. It is 'diagnose test authserver ldap'. If the
credentials are ok, and if the LDAP configuration is ok, you get not only an authentication confirmation,
but also a list of the user groups for that user.
You can run this test command as soon as you have completed you LDAP server configuration, even
before any user group, or authentication firewall policy have been added to the FortiGate. It tests only
the LDAP server configuration and the LDAP communication between the FortiGate and the server.

© FORTINET

There is a real time debug for LDAP and RADIUS. This is a sample of the output if everything is ok.
A start_search_dn message indicates that the FortiGate is doing the second step mentioned earlier:
searching for the user in the LDAP tree. The message includes the “base” branch (distinguished name
setting) and the name of the attribute used to locate the user (common name identifier setting).
(click)
If the LDAP server finds the user, the output shows Found DN, follow by the user full DN.

© FORTINET

After that, the FortiGate does the third step: biding using the user credentials.
(click)
Finally, the output shows the step four: getting the list of user groups.

© FORTINET
If there is any problem with either step 1 (admin bind) or step 3 (user bind) you can sniffer the traffic
between the FortiGate and the LDAP server to get the error code. They are very explicit about the
reason why the biding is failing.

© FORTINET
Let's see the most common problems and how to identify them by the output of the real time debug.
If you see the error invalid credentials right after an authentication attempt, this means a problem in
step 1. There is probably an issue with the admin account credentials.

© FORTINET
The message No more DN left indicates a problem in step 2. The LDAP server couldn't find the user
in the LDAP tree.

© FORTINET
The message auth denied after finding the user indicates a problem in step 3. Either the user
credentials are wrong, or the user account is not active.

© FORTINET
Finally, the error:

get_member_of_groups-attr=<attribute_name> found 0 values
indicates a problem in step 4. The user credentials are ok, but no user group information was found.
In some LDAP implementations, the user group information is not used. All LDAP users have the
same privileges regardless of their AD group memberships. In those implementations, this error can
be ignored.

© FORTINET
So, what to do if the problem is in step 1 (admin bind not working)?

- Use the dsquery query to check the admin DN
- Check the admin password
- Sniffer the error code coming from the server
What to do if the problem is in step 2 (LDAP server could not find the user):
- If the common name identifier is set to sAMAccountName, the user must use the login name. If it is
set to cn instead, the user must use the full name.
- Check the distinguished name setting with the dsquery command.

© FORTINET
As FortiGate RADIUS configuration is simpler than LDAP configuration, the troubleshooting is also
usually simpler. Let’s see quickly how to troubleshoot RADIUS problems.

© FORTINET
Normal authentication queries with the RADIUS protocol begin with an "Access-Request" being sent
from the FortiGate to the RADIUS server. Valid responses to this are "Access-Accept" and "Access-
Reject" (yes and no effectively).
If two-factor authentication is enabled on the server, it will come back with an "Access-Challenge"
message, where it is essentially looking for more information.

© FORTINET
Similarly to LDAP, there is a CLI test command for RADIUS.

In this case, you must provide not only the credentials for a test user, but also the authentication
scheme. FortiGate supports the following RADIUS schemes: chap, pop, mschap and mschap2.
Also, as in the case of LDAP, this command tests only the FortiGate RADIUS server configuration. It
does not require any user group or firewall policy in the FortiGate configuration.

© FORTINET
RADIUS is either a one-step or two-step process (depending on the use of two-factor authentication).
It is not as long as the 4-step process that happens with LDAP regular bind. So, the output of the real
time debug is usually shorter.

© FORTINET

© FORTINET
This lesson reviews FSSO operation and provides tools and tips for troubleshooting FSSO problems.

© FORTINET
You will learn to check the connectivity between a FortiGate and a collector agent (CA), and between
a CA and the domain controllers (DCs).
You will also learn to track user logon events from the DC, to the CA and to the FortiGate.
After that, the lesson explains how to list the active FSSO users both in the CA and the FortiGate.
Additionally, the lesson covers FSSO troubleshooting.

© FORTINET
FSSO operation and configuration is covered in NSE4. So, let's do a review of how this authentication
method works.

© FORTINET
FSSO enables FortiGate to leverage your network’s existing authentication system for firewall
authentication. Once a user logs in, he or she can access other network resources without having to
authenticate again.
These are the two most common FSSO methods: DC agent and polling.
DC agent requires one collector agent. It also requires DC agents installed in all Windows domain
controllers. The DC agents detect the login events and "push" this information to the CA. The CA
forwards the login events to the FortiGate, together with the workstation IP address and user's group
information.
The polling mode does not require any DC agent installed. In this mode, the CA frequently polls each
DC to get the latest login events. There are three types of polling modes:
NetAPI: Polls NetSessionEnum API every 9 seconds.
WinSecLog: Polls all security event logs every 3 seconds.
WMI: Polls specific security event logs every 3 seconds.
These poll intervals are estimated times that depend on the number of servers and network latency.
If a standalone CA is used, the 3 types of polling methods are supported. Alternatively, the polling can
be done directly from the FortiGate (agentless polling mode), so a standalone CA is not required.
However, the FortiGate acting as a CA only supports the WinSecLog polling type.
NSE4 covers the advantages and disadvantages of each FSSO method.

© FORTINET
This slide shows how the FSSO traffic flows and which ports are used.
For the case of polling mode, the polling is done from either the FortiGate or the CA, and in both cases
port TCP 445 is used.
The communication between the CA and the FortiGate, by default, uses port TCP 8000. The
communication between the DCs and the CA uses, also by default, port UDP 8002.
FSSO commonly works by identifying each user based on the source IP address of the traffic. Each
active FSSO user is associated with one or more IP addresses and one IP address is associated only
with one user. This creates conflicts in network using terminal or Citrix servers, where more than one
user are sharing the same IP address. For those cases, you can install a terminal server (TS) agent.
The TS agent provides the CA and FortiGate with not only the source IP address of each user, but
also with the source TCP/UDP ports they are using. So, multiple users sharing the same IP address
can be identified by combining the source IP address with the source port. The communication
between the TS agent and the CA also uses port UDP 8002.

© FORTINET

Each time the CA receives a logon event, it checks first if the user is in the ignore list.
(click)
If it is, the logon event is discarded. After that, the CA checks if the user's group information is still in
the cache.
(click)
If it is not, the CA does either a LDAP or an API query to get the group information from Windows AD.
Once the CA knows the user groups, it checks if any of them is included in the list of monitored
groups.
(click)
If none of the groups are included, the logon event is not forwarded to the FortiGate. If at least one
group is included, the logon event is forwarded to the FortiGate.

© FORTINET

An external CA periodically checks if each active FSSO user is still logged on. It does this by polling
all known active workstations based on a configurable interval.
(click)
The way of how this checking is done depends on the FSSO mode. For WMI polling mode, the CA
checks the WMI service. For all the other modes, the CA checks the HKEY_USERS hive via remote
registry services.
(click)
If a workstation does not respond to these checks, the CA starts a timer (dead entry timeout interval).
After the expiration of that timer, the workstation goes to not verified status and the CA assumes that
the user has logged out.

© FORTINET

We can also configure the external CA to periodically check if the IP address of an active FSSO user
has changed. This might happen for example, when a user switches from a wired to a wireless
network. When this CA feature is enabled, the CA periodically uses the DNS to resolve the
workstation name.
(click)
If the DNS server replies with a new IP address, the CA sends first a logoff event to the FortiGate,
followed by a logon event with the new IP address.
So it is important for the DNS server to be immediately and automatically updated each time a user
changes the IP address.

© FORTINET
There are three important requirements for any FSSO network, and most of the FSSO problems
happen when any of these requirements are not fulfilled.
As explained, the CA frequently polls each active workstation. For this polling to work properly, TCP
ports 139 and 445 must be open between the CA and the workstations. Firewalls must allow this
traffic. Additionally, the remote registry service must be up and running on the workstations.
As explained in NSE4, DNS is used to get the user IP address. So, administrators must ensure that
each workstation name is registered in the DNS server with the right and up-to-date IP address.

© FORTINET
We will start now with FSSO troubleshooting.

© FORTINET
What steps should an administrator do when troubleshooting a FSSO problem? This slide offers a
general checklist.
In the next slides, you will learn how to check each of those steps.

© FORTINET
If you can reproduce the FSSO problem, there are different tools to track the logon event as it travels
from the DC, to the CA and to the FortiGate. Follow these steps:
- Perform the login.
- Check which DC received the login using the Windows command: echo %logonserver%
- Go to that DC and use the Windows event viewer to find the generated logon event.
- Check the CA logs and the list of active FSSO users.
- Check that at least one user's group is listed as monitored in the group filter.
- Go to the FortiGate and check that the logon event was received. List the active FSSO users.
Generate traffic from the user workstation and verify that it is added to the FortiGate user monitor.

© FORTINET

Before starting the troubleshooting, it is recommended to do some changes to the CA logging settings
to ensure that all the required debug information will be captured. First, change the log level to
Information.
(click)
Second, increase the log file size limit. If the log file limit is too low, especially in big FSSO networks,
you might not have time to see the relevant debug information, as it will be overridden quickly by new
log events.
(click)
Third, you can optionally record the logon events on a different file.

© FORTINET
By clicking on show service status in the CA, you can check the connectivity between the FortiGate
and the CA. The window displays the serial numbers of all the active FortiGate units.

© FORTINET
When a FortiGate is successfully connecting to a CA, the CA logs show these messages. Additionally
you can confirm a successful connection while running the FSSO real time debug in the FortiGate.

© FORTINET
While the TCP session between the FortiGate and the CA is up, the CA sends heartbeat messages to
the FortiGate unit. Both the CA logs and the FortiGate real time debug show this heartbeat
interchange.

© FORTINET

The error "server authentication failed, aborting" in the FortiGate real time debug might indicate a
mismatch in the password shared between the CA and the FortiGate.
(click)
The error "connection refused" might indicate that the TCP communication between the FortiGate
and the CA is blocked by a firewall or another device.
(click)
The error "no route to host" might indicate that the CA IP address is not routable from the FortiGate.

© FORTINET
By clicking on the show monitored DCs button on the CA, you can check the communication between
the CA and each DC. The table includes the time when the last logon event was received from each
DC.

© FORTINET
The Event Viewer is a Windows tool that displays all the server logs. You can use it to search for
logon event logs.

© FORTINET
The FortiGate FSSO real time debug generates messages each time a logon event arrives. They
include the name and IP address of the user, together with the workstation name and user's group
information.

© FORTINET
You can check the list of logon users by selecting show login users. The status OK indicates that the
user is active. A user goes to not verified status when either she or he has logged out, or when there
is a problem in the polling done by the CA to the workstation.

© FORTINET
To get the list of active users from the FortiGate, use the command 'diagnose debug authd fsso
list'. You can set up a filter first with the command 'diagnose debug authd fsso filter'.

© FORTINET
The CLI command 'diagnose debug authd fsso refresh-logons' refreshes the active FSSO user list
in the FortiGate by getting this information again from the CA.
The CLI command 'diagnose debug authd fsso clear-logons' flushes the list of active FSSO users
in the FortiGate.
The CLI command 'diagnose debug authd fsso refresh-groups' refreshes the user group
information in the FortiGate by getting this information again from the CA.
To list the monitored user groups, use the command 'get user adgrp'.

© FORTINET
Agentless polling mode allows the FortiGate to directly poll each DC. The FortiGate acts as a CA, so
no standalone CA is required. There are some specific FSSO commands for troubleshooting this
mode.

© FORTINET
The command 'diagnose debug fsso-polling detail' displays the status and some statistics related
with the polls done by the FortiGate to each DC.
The command 'diagnose debug fsso-polling refresh-user' flushes the information about all active
FSSO users.
In agentless polling mode, the FortiGate frequently polls all workstations (as a standalone CA does) to
check which users are still logged on. You can sniffer this traffic in port 445.

© FORTINET
There is a specific FortiGate daemon that handles the polling mode. It is the fssod daemon. To enable
agentless polling mode real time debug use:
diagnose debug application fssod -1
The slide shows the most common real time debug errors and the possible causes.

© FORTINET
To finish this lesson, we will discuss the most common FSSO problems.

© FORTINET
What should you do if the CA does not have logon information?

- Verify that the CA is monitoring all DCs
- Check that the CA is receiving logon events from the DCs
- Test the user account and check the CA logs
What should you do if the CA does have the logon information, but the FortiGate does not?
- Check that the FortiGate is connected to the CA
- Run the real time debugs while testing the user account

© FORTINET
If a FSSO user is listed as active in the FortiGate but it cannot browse the Internet, you should first
confirm the IP address listed in the FortiGate. Also check the user's group information, firewall policies
and CA logs.
If the FortiGate is randomly blocking some users after some time, you should check that the CA
service, or any FortiGate process, is not crashing. Confirm that the connection between the FortiGate
and the CA is up and stable. Try to reproduce the problem and check the CA logs after that.

© FORTINET
DNS resolution is essential for FSSO operation. If a CA cannot resolve a workstation name, the user
will not be listed as active and the CA logs show the errors in this slide.

© FORTINET
Another common problem with FSSO authentication happens when there are applications generating
logon events with an AD account different than the users'. In those cases the logon event overrides
the information about the user for a workstation IP address (a different user is listed as active for the
IP address). The CA is coded to ignore any logon event for anonymous accounts, and account whose
name starts with '$' (which are system accounts). If any application is using an account that is
overriding the user information, the solution is to add that account to the ignore user list.

© FORTINET
Active users with the status not verified are also a common problem. That status is normal after a user
has logged out. However, it is not normal for a user that is still logged on, meaning that the polling
from the CA to the workstation is failing. In those cases, check the CA logs. They provide more
information about the cause of the problem.

© FORTINET
What should you do if a user lost internet access after the IP address changed?
This type of problem might happen when the user moves back and for from a wired to a wireless
network. It might also happens when a workstation gets back from hibernate mode (a gets a new IP
address from the DHCP server).
The first step is to check the DNS resolution. As explained in a previous slide, when a user changes
the IP address, the DNS server must be automatically updated with the new IP information. If that is
not possible, one workaround to this problem is to configure FSSO guest access. So, users whose IP
addresses have changed can still have some (probably limited) access to some network resources.
For the cases where a workstation was assigned more than one IP address (for example one address
for the wired network and another one for the wireless), the DNS server should be able to return all
those IP addresses when resolving the workstation name. The user is then listed multiple times in the
FSSO active user list, one time for each IP address.

© FORTINET

© FORTINET
This lesson is about IPsec troubleshooting. It is one of the most important lessons, as a significant
percentage of the support tickets received by Fortinet TAC are related with IPsec.

© FORTINET
At the end of this lesson you will know how to bring down, bring up and restart an IPsec VPN. The
lesson also teaches how to check the IPsec offloading and use the real time debug to troubleshoot
problems during the negotiations of phase 1 and 2. Additionally, you will learn about IPsec traffic
capture using the sniffer and debug flow.

© FORTINET

When isolating IPsec problems, it is useful to understand that an IPsec connection can be described
as a 4-step process:
(click)
1- "Interesting" traffic triggers the VPN negotiation. Traffic is called "interesting" when it must travel
through an IPsec tunnel (encrypted and encapsulated) to reach a remote network.
(click)
2- Phase 1 goes up through the negotiation of an IKE security association (SA)
(click)
3- One or more phases 2 go up. Two IPsec security association are negotiated per each phase 2
(click)
4- Traffic crosses the tunnel
So, if you have an IPsec issue, you should determine in which of these four steps the problem is.

© FORTINET

We explained the differences between aggressive mode and main mode in NSE4. Let's review those concepts.
This shows main mode. 6 packets are exchanged.
(click)
First, the client initiates by proposing that the tunnel will use one or more security policies.
(click)
The responder selects which security policy it will agree to use, and replies.
(click)
Then, the initiator sends its Diffie Hellman public value.
(click)
The responder replies with its own Diffie Hellman public value.
(click)
Finally, the initiator sends its peer ID and hash payload,
(click)
and the responder replies in the same way.
(click)
As the peer ID is not included in the first packet, it cannot be used by the FortiGate as part of the criteria for
selecting the VPN configuration to use. This is not a problem when using point-to-point VPNs as the FortiGate
uses the source IP to identify the VPN. It is not a problem either when using one dial-up VPN. But, as we will see
later, it can be a problem when using multiple dial-up VPNs.

© FORTINET

In comparison, let’s show aggressive mode negotiation. Only 3 packets are exchanged:
(click)
First, the client initiates by suggesting a security policy, and providing its Diffie Hellman public value
and peer ID.
(click)
The responder replies with the same information, plus a hash.
(click)
Finally, the initiator sends its hash payload.
(click)
Unlike main mode, the first packet contains the initiator’s peer ID. Therefore, the responder can use
this ID as part of the criteria for identifying the VPN configuration to use.

© FORTINET
This slide details the selection criteria for dial-up. A FortiGate uses the first VPN configuration (in
alphabetical order) that matches:
- Local gateway IP
- Mode (aggressive or main)
- Peer ID, if aggressive mode is used. As explained, only aggressive mode include the Peer ID in the
first packet.
- Authentication method (pre-shared or PKI)
- Digital certificate information, if PKI is used as the authentication method
So, if a FortiGate has multiple dial-up VPNs with pre-shared key sharing the same local gateway, you
must use aggressive mode and different peer IDs. In this way, the FortiGate identifies the right VPN
configuration for each incoming IPsec proposal.

© FORTINET
If you need to capture the IPsec traffic, keep in mind that the IP protocol and UDP port numbers
depend on NAT-T and the use of NAT.
If there is no device in the middle doing NAT, IKE traffic uses UDP port 500 and ESP traffic uses IP
protocol 50. The slide shows the two sniffer filters to capture each of those traffic protocols.
If, on the other hand, NAT-T is enabled and there is a device in the middle doing NAT, the sniffer
command must use a different filter. In this cases IKE traffic starts using port UDP 500, but it switches
to UDP port 4500 during the tunnel negotiation. Additionally, ESP traffic is encapsulated inside the
UDP-4500 channel.

© FORTINET

To check the status of the IKE SA, use the command 'diagnose vpn ike gateway list'. It shows, amount
other information, when the SA was created,
(click)
who initiated the VPN,
(click)
and the time of the last DPD messages sent and received.
To manage the IKE SA, use the commands:
diagnose vpn ike gateway flush name
diagnose vpn ike restart

© FORTINET

For information about the IPsec SAs (phases 2), use the command 'diagnose vpn tunnel list'.
For each phase 2, the output shows:
- Gateways IP addresses
(click)
- DPD counters
(click)
- NAT-T mode: There are three possible values. none means that there is no device doing NAT
between both peers. silent indicates that the remote peer is NATed. keepalive indicates that the local
peer is NATed. In both silent and keepalive modes, ESP traffic is encapsulated into UDP packets.
(click)
- SPIs, negotiated encryption, authentication and keys for each traffic direction
(click)
- Traffic counters

© FORTINET
The same command 'diagnose vpn tunnel' offers options for shutting down, activating or flushing
any phase 2.

© FORTINET
'get vpn ipsec stats tunnel' shows the number of IPsec VPNs in the configuration.
Both 'get vpn ipsec tunnel summary' and 'get ipsec tunnel list' can be used to list summarized
information about each IPsec VPN.

© FORTINET

For detailed information about each tunnel, use the command 'diagnose vpn ipsec tunnel detail'.
The output shows traffic counters,
(click)
negotiated quick mode selectors,
(click)
and negotiated encryption, authentication and keys.

© FORTINET
In some FortiGate models, the encryption and decryption of IPsec traffic can be offloaded to hardware.
The supported algorithms depend on the model and type of processor unit doing the offloading.
For the cases of NP2, and NP4 with FortiOS 5.2.2 or lower, there is an additional requirement. If IPsec
anti-replay is enabled, you must check that IPsec offloading is enabled under 'config system npu'.
If the hardware is NP6, NP4lite, or NP4 with FortiOS 5.2.3 or newer, enabling those settings is not
required for IPsec offloading.
NP2 has an important limitation regarding anti-replay. We will talk about it later.

© FORTINET
For the case of IPsec traffic, the FortiGate session table includes a field that describes when the
encryption and decryption is being offloaded.
First, when the phase 2 goes up, the IPsec SAs are created and loaded to the kernel. As long as there
is not traffic crossing the tunnel, the SAs are not copied to the NPU and the npu_flag shows 00. The
value of that field also remains in 00 when IPsec offloading is disabled or not supported.
Second, when the first IPsec packet arrives, if it is an outbound packet that can be offloaded, the
outbound SA is copied to the NPU and the npu_flag changes to 01. If, on the other hands, the first
IPsec packet is inbound and can be offloaded, the inbound SA is copied to the NPU and the npu_flag
changes to 02.
Finally, once both SAs are copied to the NPU, the npu_flag changes to 03.

© FORTINET
This command shows number of packets encrypted and decrypted by software (CPU), and by each
type of processor unit in the FortiGate.

© FORTINET
The IPsec or IKE real time debug is the most useful tool for troubleshooting problems during the
tunnel negotiation. Before enabling the debug, you should first set the filter. The recommended filter
option is dst-addr4 (or dst-addr6 for IPv6). With this filter, the debug shows only and all the
information about the IPsec tunnel whose remote IP address matches the filter.

© FORTINET
After setting the filter, enable the IKE real time debug. The table shows what type of output is enabled
by each bit in the bit-mask. The most common values for the bit-mask are -1 (all outputs enabled) and
63 (shorter output). They both show the DPD packets and all the information required for
troubleshooting IPsec negotiation problems.

© FORTINET

Let’s see what the real time debug shows under normal circumstances. We analyze the case of main
mode first.
As explained earlier, main mode requires the interchange of 6 packets, 3 for each traffic direction. The
real time debug shows when the first packet (first main mode message) arrives.
(click)
Then the debug shows the negotiated settings for the phase 1.
(click)
A message is generated once the FortiGate identifies the VPN configuration to use (with the name of
the VPN).

© FORTINET

The second main mode message arrives.
(click)
The third main mode message arrives.
(click)
Pre-shared keys match.
(click)
And a message is generated to indicate that the phase 1 is up.

© FORTINET

Let’s see aggressive mode. In this case the negotiation is faster as only three packets are
interchanged.
After the first aggressive mode message arrives,
(click)
the debug shows the result of the phase 1 negotiation.
(click)
Then the VPN is identified.
(click)
The second aggressive mode message arrives.
(click)
A message is generated to confirm that the pre-shared keys match.
(click)
Another message is generated to indicate that the phase 1 is up.

© FORTINET

Now, let’s see the output for a phase 2 negotiation.
The debug shows the phase 2 proposal from the local gateway,
(click)
and the phase 2 proposal coming for the remote gateway.

© FORTINET

Then, the output shows the negotiated phase 2 settings.
(click)
The last messages confirm that the phase 2 is up.

© FORTINET
We explained extended authentication (XAuth) in NSE4. Here, you will see what packets are
interchanged during extended authentication.
XAuth happens after the phase 1 is up and before any phase 2 negotiation. That is why XAuth is also
called phase 1.5.
In any XAuth communication there is always one client and one server. The server sends a
CFG_REQUEST packet, which must be replied by the client with a CFG_REPLY packet. The
CFG_REPLY packet includes the user credentials. If the authentication is ok, the server sends
CFG_SET and the client replies with CFG-ACK.

© FORTINET
The IKE real time debug shows this interchange.

© FORTINET
The messages in the real time debug, related with the CFG_REPLY, show the XAuth user and group
names.

© FORTINET
If a VPN is using XAuth, the output of the command 'diagnose vpn ike gateway list' shows the
username. The same information is displayed in the IPsec monitor in the FortiGate GUI.

© FORTINET
A FortiGate has two different methods for automatically configuring the IP settings of an IPsec client:
DHCP over IPsec and IKE mode configuration.
We analyze the DHCP over IPsec method first.
After the phase 1 is up, and the extended authentication is completed, the client negotiates a
provisional phase 2 of short duration only for the DHCP traffic. Once that provisional phase 2 is up,
standard DHCP traffic is interchanged to set the client IP settings: DHCP_DISCOVER,
DCHP_OFFER, DHCP_REQUEST and DHCP_ACK.
After the client gets the IP settings (IP address, default gateway, DNS, among others) the phase 2
created for DHCP is turned down. After that, a new phase 2 is negotiated for user traffic.

© FORTINET

The IKE real time debug shows the negotiation of the phase 2 for DHCP. In this first phase 2, the
client IP address is used in the quick mode selectors.
(click)
Then the DHCP phase 2 is deleted,
(click)
and a new phase 2 is created for use traffic. In this second phase 2, the IP address assigned by the
DHCP server is used in the quick mode selectors.

© FORTINET
When troubleshooting DHCP over IPsec problem, you can also use the DHCP real time debug. It
shows information about the DHCP traffic, including all the IP settings assigned to the IPsec client.

© FORTINET
The other method for assigning IP settings to an IPsec client is IKE mode configuration. Similarly to
DHCP over IPsec, it happens after the phase 1 and extended authentication and before any phase 2
negotiation. The client sends a CFG_REQUEST message listing the required IP settings (or
attributes). The server replies with a CFG_REPLY which contains the assigned values for each of the
attributes requested.

© FORTINET
The IKE real time debug shows when the CFG_REQUEST and CFG_REPLY packets are sent.

© FORTINET
Let's say now that the phase 1 goes up, the phase 2 also goes up, but, for some reason, the traffic is
not crossing the tunnel.
For those cases, the best troubleshooting tool is the debug flow. If possible, run it in both gateways. It
will let you know if the local gateway is dropping the packets, if the local gateway is not routing the
packets through the tunnel, or if the remote gateway is the one dropping the packets.
This slide shows the normal output. You should see a message indicating that the packet goes to the
tunnel (with the tunnel name), and another one explaining that the packet is being encrypted.

© FORTINET
We will do now a summary of the most common problems and solutions.

If the tunnel is not coming up, use the IKE real time debug. In those cases, you usually get one of this
error messages.
When the tunnel is bouncing, you usually see DPD packets lost, as an indication that the problem
might be on the ISP side.

© FORTINET
If the tunnel is up, but traffic does not go through, use the debug flow. It could be that packets are
being dropped at either the local or remote gateway. It could be that packets are not being routed
properly. Or it could be that packets do not match the quick mode selectors, so they are dropped.

© FORTINET
Most of the IPsec problems happen when creating tunnels between FortiGate and a third-party device.
Differently than FortiGate, some vendors do not support quick mode selectors that are either 0.0.0.0/0,
or use subnets with different sizes. So, they require a different SA (and so a different phase 2) for
each pair of local and remote protected subnets. On those cases, the FortiGate configuration might
get complicated and long as administrators need to create all the different phases 2.
One alternative is to use the 'mesh-selector-type' setting in the phase 1. When it is set to 'subnet'
the FortiGate automatically creates the phases 2 on the flight and per demand. This might simplify the
FortiGate configuration considerably.

© FORTINET
Anti-replay is an IPsec mechanism that adds a sequence number to the ESP headers. The number is
incremented each time a packet is sent. It protects the tunnel against replay attacks.
We mentioned earlier that there is a restriction related with NP2 and IPsec anti-replay. NP2 cannot
properly generate this sequence number. So, if encryption offloading is enabled in a NP2 platform, you
might get warning and packets dropped if the remote site has anti-reply enabled. Therefore, you would
need to disable either anti-replay or encryption offloading. This limitation does not apply to NP4 and
NP6 platforms.

© FORTINET

© FORTINET
During this lesson, you will learn to troubleshoot some of the security profiles (or UTM inspection) features.

© FORTINET
You will learn to troubleshoot FortiGuard, web filtering and antivirus issues. You will also learn to monitor
the IPS and fix certificate warning problems during full SSL inspection.
Additionally, the lesson compares proxy-based and flow-based inspection modes and describes the
FortiGate packet inspection steps.

© FORTINET
The first part is about FortiGuard.

© FORTINET
The FortiGate GUI can be used to quickly check the status of the communication between FortiGate and
FortiGuard. A green checkmark beside each FortiGuard service indicates that FortiGate can reach
FortiGuard and that the license for that service is valid.

© FORTINET

To learn how to troubleshoot FortiGuard problems, we need to understand first how FortiGuard
communication works. Also, the communication between FortiGate and FortiGuard for web filtering and
antispam is different than in the case of antivirus and IPS. Let's see how FortiGuard web filtering and
antispam works first:
Initially, FortiGate contacts the DNS server to resolve the name service.fortiguard.net. FortiGate should get
a list of IP addresses for servers (usually two or three) that can be contacted to validate the FortiGuard
license.
(click)
FortiGate contacts one of those servers to check the license and get a list of servers that can be used to
submit web filtering and antispam rating queries.
(click)
FortiGate starts sending rating queries to one of the servers in the list. We will explain soon how FortiGate
chooses the server.
(click)
If the chosen server does not reply, FortiGate uses the next server in the list.

© FORTINET
For web filtering and antispam, the communication between FortiGate and FortiGuard uses either port 53
or 8888. The GUI also offers a test button to check this connectivity.

© FORTINET
The command 'diagnose debug rating' shows the list of servers for web filtering and antispam queries.
For each IP address, the table shows:
- The round trip delay
- The server's time zone
- The amount of recent and consecutives queries without reply
- The historical total amount of queries without reply

© FORTINET
This is how FortiGate selects the server from the list to send the rating requests.
FortiGate initially uses the delta between the server's time zone and the FortiGate's system time zone
multiplied by 10. This is the server initial weight.
The weight goes up with each packet lost.
The weight goes down over time if there are no packets lost. But, it never goes below the initial value.
FortiGate uses the server with the lowest weight as the one for the rating queries.

© FORTINET
The output of the command 'diagnose debug rating' shows some flags beside some of the servers:
I: This was the server initially contacted to validate the license and get the server list. Usually, there is only
one server with this flag
D: FortiGate got this IP address when resolving the name service.fortiguard.net. If the administrator has
not overwritten the FortiGuard FQDN or IP address in the FortiGate configuration, there are usually two or
three servers with this flag
S: FortiGate got this IP address from a FortiManager
T: Server is not replying to FortiGate queries
F: Server is down

© FORTINET
In many cases, problems related with FortiGuard are caused by ISPs.

Some ISPs block traffic in port 53 that is not DNS, or that contains big-size packets (as FortiGuard does).
In those cases the solution is to switch FortiGuard traffic from port 53 to port 8888.
Some other ISPs (or upstream firewalls) block traffic to port 8888. In those cases the solution is to use port
53.
There are also a few cases where ISPs block traffic based on source ports. Changing the source port
range for FortiGuard to the one in this slide usually fixes the issue.

© FORTINET
Let's move to FortiGuard antivirus and IPS. For these two services the communication between FortiGate
and FortiGuard happens much less frequently. In the case of web filtering and antispam, FortiGate goes to
FortiGuard each time it needs to rate a web site or email (if the information is not in the FortiGate cache).
In the case of antivirus and IPS, FortiGate goes to FortiGuard usually once a day to check and download
any new version of the AV or IPS databases and engines. This is done using port TCP 443. By default,
FortiGate connects to update.fortiguard.net.
If FortiGate must connect through a web proxy, these are the CLI commands to use. Usually, clients
connecting to a web proxy do not contact the DNS server to resolve names, as it is the web proxy who
does it. But, in the case of FortiGuard, FortiGate always requires DNS access, even when connecting
through a web proxy.

© FORTINET
Both the AV and IPS databases can be updated either automatically or manually. Automatic updates
download the portions of the databases that have changed since the last update.
Manual updates download the whole database if there is new version available.
In a few cases, FortiGuard problems are fixed by doing a manual update. This forces FortiGate to
download and install the whole database. For example, if the AV database is corrupted, a manual update
might fix the problem.

© FORTINET
The command 'diagnose test application dnsproxy 7' displays the FQDN and IP addresses of the
FortiGuard servers available for AV and IPS updates.
The command 'diagnose autoupdate status' provides a summary of the FortiGuard configuration in the
FortiGate.

© FORTINET
'diagnose autoupdate versions' lists all the FortiGuard databases and engines installed. The information
includes the version, contract expiration date, time it was updated and what happened during last update.
The output displays information about the AV engine, AV database and IPS database.

© FORTINET
It also shows the IPS engine, device identification database and IP geography database.

© FORTINET
If there are problems updating the AV or the IPS, or if there are problems validating the license, you can
use the FortiGuard real time debug:
diagnose debug application update -1
diagnose debug enable
After enabling the debug, you can force a manual update from the CLI with the command:
execute update-now
For FortiGuard web filtering and antispam, the commands for enabling the real time debugs are different.
We will see the web filter real time debug later.

© FORTINET
Remember that FortiGuard traffic originates always from the management VDOM. So, the management
VDOM (which is root by default) must have Internet access.
Proper DNS access from the management VDOM is also important. The FortiGate must be able to resolve
the names:
update.fortiguard.net
service.fortiguard.net
Also, keep in mind that updates to FortiGuard contracts are NOT instantaneous. It usually takes around 1
or 2 hours to update a contract in all the servers. However, in some cases, it could take up to 24 hours. So,
if you have just changed or renewed your FortiGuard contract and you do not see the change in the
FortiGate, most probably you just need to wait a bit more, to give FortiGuard time to synchronize the
information in all the servers.

© FORTINET
Before talking about how to troubleshooting the most common UTM features, let's talk about inspection
modes and life of a packet.

© FORTINET
Proxy and flow-based inspection modes are covered in NSE4. Let's do a quick review.
Most of the UTM features (AV, web filtering) can work in either proxy or flow-based mode.
In proxy-based mode, the FortiGate transparently proxies any TCP session between a client and a server.
So, actually two TCP sessions are created: one between the client and the FortiGate, and another one
between the FortiGate and the server. This mode is usually slower than flow-based, but it offers all the
functionality.

© FORTINET
Flow-based mode, on the other hand, does not proxy TCP sessions. The traffic is inspected as it flows
through the unit as a single TCP session. This mode is usually faster that proxy-based, but offers less
functionality.

© FORTINET
If traffic matches a firewall policy that combines proxy-based with flow-based inspection profiles, FortiGate
does proxy-based inspection for all the profiles that support both modes.
This does not apply to IPS or application control profiles, which are always and only flow based.

© FORTINET
Application proxies are used during proxy-based inspection. They split the TCP session into two and
decide what actions to take. They also buffer the traffic and files, generates logs and archives information.

© FORTINET
Two session flags indicate if the traffic is being inspected in either proxy or flow-based mode. The flag redir
means proxy-based mode. The flag ndr means flow-based mode. Also, and for the case of proxy-based
inspection, the debug flow shows the message "send to application layer".

© FORTINET
During the following slides we are going to analyze how a packet is processed by FortiGate since arrives
to the unit until egresses the unit. We will see what actions the FortiGate takes and the order of those
actions. We have split the whole process into four stages. The first stage happens when the packet is
ingressing:
First, you can set a limit in the incoming bandwidth ('inbandwidth' parameter). If the traffic has exceeded
that limit, the packet is drop.
After that, FortiGate checks the thresholds in the DoS policies.
The next steps are the reverse path forwarding and IP header integrity checks.
If the traffic terminates at the FortiGate (for example management traffic, web proxy, DNS) the packet is
processed by the specific daemon that handles the requested feature. If the traffic does not terminate at
the FortiGate, it goes to the second stage: routing and firewall policy.

© FORTINET
Before doing the routing lookup, the FortiGate (if required) applies the destination NAT. If there is no route
to the destination, the packet is dropped. In a similar way, if the packet is denied by a firewall policy, it will
not be allowed. If the packet is allowed, FortiGate checks if the policy requires authentication. After those
steps, FortiGate proceeds to identify the traffic, which is then inspected by any session helper that might
be required.

© FORTINET
The third stage is UTM inspection. If the traffic is encrypted and full SSL inspection is used, the unit
proceeds to decrypt the traffic. After that, the inspection profiles are applied in this order: IPS, application
control, VoIP, DLP, antispam, web filtering and antivirus.

© FORTINET
After inspection, FortiGate applies traffic shaping and source NAT. Finally, if the packet must be routed
through an IPsec VPN, it is encrypted before egressing the unit.

© FORTINET
We will talk now about the most common UTM problems. We will start with antivirus.

© FORTINET
This is the order of how antivirus inspection is executed. First, the unit does the virus scan, then the
grayware scan, and finally the heuristic scan.

© FORTINET
One of the best tools for troubleshooting antivirus issues is the FortiGate logs. This is a sample of a log
generated when the FortiGate detects a virus. You can see the name of the file, time and name of the
virus.

© FORTINET
What should an administrator do if a virus is detected in a workstation behind a FortiGate doing antivirus.
Again, the first step is to get specific information: What AV software did detect the virus? What is the name
of the virus? How did it get inside the network? It might have got inside, for example, through a CD or USB
stick, so the infected file never went through the FortiGate.

© FORTINET
Test if the FortiGate antivirus is properly inspecting the user traffic. Go to eicar.org from a user workstation
and try to download the EICAR sample file.
If you have a sample of the virus, submit it to FortiGuard and check if it is present in any of the FortiGate
virus databases. If it is, check the name of the AV database where it is present. Does your FortiGate have
that database installed?

© FORTINET
There are three FortiGate AV databases: normal, extended and extreme. Not all the models support the 3
databases, and not all the databases might be installed in your FortiGate device.

© FORTINET
Also remember the restrictions for antivirus inspection:

- SSL traffic requires SSL deep inspection
- Archives files, such as ZIP files, are examined to certain limits, such as the maximum number of
subdirectories and nested archives
- Password protected archives cannot be scanned

© FORTINET
The next part of this lesson is about web filtering.

© FORTINET
During web filtering inspection, FortiGate first checks the static URL filter list, then the FortiGuard
categories and content filtering. Finally, the web filtering can execute some advanced options.

© FORTINET
Similarly to other UTM features, one of the best troubleshooting tools for web filtering is the FortiGate logs.
The unit can generate a log each time a web site is blocked. The log lists the URL, the category, action
taken and quota info.

© FORTINET
Error counters related with web filtering can be listed with the command 'diagnose webfilter fortiguard
statistics list'.

© FORTINET
The output also shows counters for the number of sites allowed and blocked, and statistics about the
cache (including hit and miss rates).

© FORTINET
Another tool for web filtering troubleshooting is the web filter real time debug. You can set a filter first by
user (source) IP address with the command:
diagnose debug urlfilter src-addr
After that, you can enable the real time debug:
diagnose debug application urlfilter -1
This slide shows a sample output of the real time debug when the URL to categorize is not in the
FortiGuard cache. The two messages display the URL, category, source, destination IP addresses and
service.

© FORTINET
This is another sample of the real time debug. But in this case, the URL category was found in the
FortiGuard cache.

© FORTINET

Even after filtering the real time debug by source IP address, the output might still be too much to easily
find the URL that you are troubleshooting. In those cases, it might be easier just to search the category
information in the FortiGuard cache.
To list the content of the FortiGuard web filtering cache, use the command 'diagnose test application
urlfilter 3'.
For each URL, the output lists its rating by domain name and its rating by IP address.
The rating by domain name is the first two digits of the first number from left to right. It is the category ID
represented in hexadecimal.
(click)
The rating by IP address is the first two digits of the second number. It is also the category ID represented
in hexadecimal.
(click)
The command 'get webfilter category' lists all the categories with their respective ID numbers. In this list,
the IDs are represented in decimal.
(click)
So, if you want to find the category name for a URL in the cache, use the first command to list the cache,
and convert the ID number from hexadecimal to decimal. Then, use the second command to find the
category name for that ID number.

© FORTINET
The command 'diagnose test application urlfilter 2' clears the web filter
cache.
The command 'diagnose test application urlfilter 99' restarts the web filtering
daemon.

© FORTINET
Tips for troubleshooting web filtering:

- Get the specifics first: What are the URLs that are having the problem? Is it random? Does it happen with
all the users?
- Check the logs
- Isn't the problem caused by a wrong user group configuration? Are the user access privileges right?
- Run the real time debug while reproducing the issue

© FORTINET
We are going to briefly talk about monitoring IPS and DoS operation.

© FORTINET
IPS logs show information about attacks detected by the IPS. We can use them to see the source and
destination IP addresses, and time and name of the attack.

© FORTINET
'diagnose ips packet status' shows general IPS statistics, including amount of packet inspected (by IP
protocol) and counters for each action taken.

© FORTINET
The IPS test application command can be used to restart the IPS process. If, for any reason, you need to
restart the IPS daemon, this is the right way to do it (do not use the command 'diagnose sys kill' to restart
it).
There is also an option for bypassing the IPS. To "toggle" the bypass status (switch it back and for from
enabled to disabled) use the option 5. When IPS bypass is enabled, the FortiGate does not do any kind of
IPS inspection over the traffic. All IPS inspection is actually bypassed. This is useful when troubleshooting
high CPU problems caused by the IPS engine. In those cases, if after enabling IPS bypassing, the CPU
usage is still high, it might indicate a problem in the IPS engine that should be reported to Fortinet TAC.

© FORTINET
The anomalies (or DoS) logs are aggregated. When several incidents occur together, this reduces the
number of log messages.
In large attacks, the number of incidents can easily reach 100,000 in a few seconds. Generating a log
entry for every packet that matches would completely utilize the CPU. So instead, FortiGate collapses
incidents by periodically recording only one message for all of them, and noting the number of incidents.
Here, the detection threshold was 50, and the total count is 75. So, FortiGate doesn’t make 24 separate
log entries (1 for each incident above 50). It’s just one log message.

© FORTINET
The command 'diagnose ips anomaly list' lists the statistics for traffic matching any DoS policy. For each
IP address and DoS policy, the output displays the expiration time (when the entry will be removed from
the DoS table) and the packets per seconds.

© FORTINET
NSE4 introduces the different SSL inspection modes. Here, we are going to review and expand those
concepts.

© FORTINET
There are two SSL inspection modes: SSL certificate inspection and full SSL inspection. When using SSL
certificate inspection, the FortiGate is not decrypting the traffic. It is only inspecting the server digital
certificates and the name indication (SNI) field, which are interchanged before the encryption.
The FortiGate tries first to get the URL from the SNI field. The SNI field is a TLS extension that contains
the complete URL that the user is connecting to. It is supported by most of the modern web browsers.
If the SNI field is not present (because the web client may not support it), the FortiGate proceeds to inspect
the server digital certificate. It specifically uses the FQDN in the common name (CN) field as the URL to
categorize. Usually, but not always, this FQDN matches the site the user is connecting to. But, it is not
always accurate. That is why it is only use as an alternative when the first method (SNI) fails.
If you see that the FortiGate is wrongly identifying a HTTPS URL, check the CN field in the site's digital
certificate. It might be that the client does not support SNI and the FortiGate is getting the wrong URL from
the certificate. In those cases the solution would be to use either full SSL inspection, or a client that
supports SNI.

© FORTINET
Full SSL inspection is available only in some FortiGate models. In this case, the device does indeed
decrypt (and re-encrypt) the SSL traffic.
Under normal circumstances (without full SSL Inspection), encrypted traffic cannot be inspected, as the
firewall does not have the key required to decrypt the data.
In order to work, the FortiGate must be located in the middle of the communication between the user’s
browser and the web site. When the browser connects to the site, the web server sends its certificate,
which contains its public key.
The FortiGate intercepts the web server certificate and generates a new one on the fly. The new certificate
is issued by the CA installed on the FortiGate, which is not and well-known public CA. The FortiGate also
generates on the fly a new pair of public and private keys.

© FORTINET
When enabling SSL Inspection, browsers start displaying a certificate warning each time a user is
connecting to a HTTPS site. The reason is that the certificates received by the browsers are now being
signed by the FortiGate, which is a CA that the browsers do not trust. There are two ways to fix this
warning:
The first option is to download the default FortiGate certificate for SSL proxy inspection and install it in all
the workstations.
The second option is to generate a new SSL proxy certificate from a private CA. In this case, the private
CA certificate must still be installed in all the workstations.
This is not a limitation in FortiGate, but a consequence of how digital certificates were designed to work.

© FORTINET
Replacing the certificate on the traffic can cause problems. Some software and servers have specific
limitations on the certificates that are allowed to be used.
HSTS and PKP are security features designed to detect man-in-the-middle SSL attacks by making sure
that any certificate presented when accessing a server resource is signed by a specific CA. If it detects
any other CA it will simply refuse to continue the SSL handshake and prevent access to the website.
The options available for these are limited. One option is to bypass SSL inspection of that traffic. Another
option is to use SSL certificate inspection instead.

© FORTINET

© FORTINET
NSE4 covers explicit web proxy configuration. This lesson is about explicit web proxy troubleshooting.

© FORTINET
At the end of this lesson you will be able to troubleshoot explicit web proxy issues by checking the
status of the proxy users, user traffic, sessions and DNS traffic. You will also learn to use the proxy
real time debug.

© FORTINET
With explicit web proxy, clients do not send their requests directly to the servers, but to the proxy. The
proxy processes the requests and forwards them to the servers. Clients must be explicitly configured
with the proxy IP address (or FQDN) and the port that the proxy is listening to. A web proxy can
optionally cache the web traffic.

© FORTINET
There are 3 methods for configuring the explicit web proxy settings in a browser: Manual, using a PAC
file, and using the web proxy autodiscovery protocol (WPAD). WPAD is explained in NSE4.

© FORTINET
A PAC file defines when the browser must use a proxy and how to choose it (when more than one
proxy is available). If you are not using WPAD, and want to use a PAC file, you must configure the
browser with the URL of the HTTP server where the file can be downloaded. Usually, the PAC file is
stored in the proxy itself. In the case of FortiGate, by default, the PAC file can be downloaded from the
URL:
http://<FortiGate_IP>:<Port>/proxy.pac
Both the port and the PAC file name are configurable.

© FORTINET

What does a PAC file contain?
A PAC file is a JavaScript function. When browsers run it, it determines whether the request will be
proxied, and what proxy to use.
In this example:
The PAC file allows any connection to example.com to bypass the proxy
(click)
Connections to servers in the 10.0.0.0/24 subnet use the proxy named fastproxy.example.com
(click)
All other requests are made through proxy.example.com

© FORTINET
Given that cache consumes system resources, do you want all users to be able to use the cache?
You can configure the proxy to allow access only to authenticated users that belong to specific user
groups. Authentication can be either based on either source IP address or HTTP session cookies.
How should you decide which to use?
IP-based authentication requires less memory. However, it should only be used when each user has a
different IP address. If your users are sharing the same IP address, use HTTP session-based
authentication instead. In this mode, each browser inserts an HTTP cookie in its requests. The
cookies identify each user. This method requires slightly more memory because FortiGate must
remember all session cookies.

© FORTINET
The GUI user monitor shows the active explicit web proxy users. To get that list from the CLI, use the
command:
diagnose wad user list
To clear (de-authenticate) all the explicit web proxy user, use the command:
diagnose wad user clear

© FORTINET

For any explicit web proxy connection (if the content is not in the cache) there are two TCP sessions
listed separately in the FortiGate session table: one from the client to the proxy, and another one from
the proxy to the server. This is a sample of the session entry created for the first TCP session.
It has the local flag as it terminates at the FortiGate.
(click)
The client IP address is the source,
(click)
and the proxy is the destination.

© FORTINET

This is a sample of the second TCP session.
The local flag is here too, and this session also terminates at the FortiGate.
(click)
However, in this case the source IP address is the proxy,
(click)
and the destination is the server.

© FORTINET
The command 'diagnose wad session list' shows a summary of the all the sessions related with web
proxy. In this case, each pair of TCP sessions (client to proxy and proxy to server) is displayed as one
entry.

© FORTINET
FortiOS limits the number of simultaneous explicit web proxy users. This limit is hard coded, applies
globally (to all VDOMs) and depends on the model. However, there are no limits in the number of
simultaneous sessions per user.
If user authentication is not used, each source IP address is treated as a different user.

© FORTINET
Before running any explicit proxy debug command, you must enable them with the command:
diagnose test application wad 2200
After that, use 'diagnose test application wad 112' to get the maximum number of simultaneous
proxy users.

© FORTINET
The command 'diagnose test application wad 110' shows the amount of concurrent proxy users.
The output also shows the list of users, their IP addresses and the authentication method they are
using (either IP-based or session-based).
As explained in the previous slide, type first the command 'diagnose test application wad 2200' if
you have not done it yet.

© FORTINET

Keep in mind than in the case of explicit web proxy, the proxy itself is responsible for resolving the
web site FQDNs. So, the proxy requires a reliable DNS connection.
This commands displays web proxy DNS traffic statistics. It shows the number of DNS requests sent,
(click)
the number of times that the DNS could not resolve a name,
(click)
and the number of times that the DNS server did not reply.

© FORTINET

The way of how you enable the real time debug for web proxy is different than the usual way.
First, you can optionally create a filter by adding it to the FortiGate configuration. Multiple URLs can be
added to the filter.
(click)
After that, you must enable the use of the filter previously created with the command 'diagnose wad
debug-url enable'.
(click)
Finally, you enable the real time debug with the commands 'diagnose wad console-log enable' and
'diagnose debug enable'.

© FORTINET
In this example we are enabling the real time debug to display information about connections to the
URL www.fortinet.com.
The debug shows good details about what is happening step-by-step with the proxied traffic.
First, you should see the HTTP GET request coming from the client. The output shows part of the
HTTP header.

© FORTINET

After that, the debug shows messages related with the DNS resolution of the FQDN in the URL.
(slide)
The HTTP GET request is forwarded to the cache first, and then to the server if the content is not in
the cache.

© FORTINET
If it is not in the cache and the server replies, the debug shows an output similar to this one.

© FORTINET
Finally, the reply from the server is forwarded to the client.

© FORTINET
If, on the contrary, the content was found in the cache, the output is a bit different.

© FORTINET

© FORTINET
During this lesson, we will talk about the two FortiGate operation modes: NAT/route and transparent.

© FORTINET
At the end of this lesson you will know how to identify the route path for any traffic session, diagnose and
troubleshoot routing problems and segment a layer-2 network into multiple broadcast domains.

© FORTINET
Traditional firewalls and NAT mode FortiGate units are routers. So, each interface has to be in different
subnets and each forms different broadcast domains. The FortiGate routes IP packets based on the IP
header information, overriding the source MAC address. So, if a client sends a packet to a server
connected to a different FortiGate interface, the packet arrives to the server with a FortiGate’s MAC
address, instead of the client’s.
In the case of transparent mode, FortiGate forwards frames without changing the MAC addresses. When
the client receives a packet from a server connected to a different FortiGate interface, the frame contains
the server’s real MAC address. A transparent mode FortiGate is a Layer-2 bridge or switch. So, the
interfaces do not have IP addresses and all belong (by default) to the same broadcast domain.
This means that a transparent mode FortiGate can be installed in a customer network without changing the
customer’s IP address plan.
A FortiGate can combine VDOMs in NAT mode with VDOMs in transparent mode.

© FORTINET
The first part of this lesson is about NAT or route mode.

© FORTINET
A FortiGate is a stateful device, so a lot of information is decided right at the beginning of a session, on the
first packets. For any traffic session, usually only two routing lookups are done: one on the first packet sent
by the originator, and another one on the first reply packet coming from the responder. After that, all the
routing information is written in the FortiGate session table. However, after any change to the routing table,
the route information is flushed from the affected entries in the session table. So, additionally routing table
lookups are done after to repopulate the session table with the new routing information.

© FORTINET
How does FortiGate decide routes? FortiGate has multiple routing modules. This diagram shows their
logic.
First FortiGate searches its policy routes. You can view them with the command 'diagnose firewall
proute list'. If there is a match in a policy route and the action is Forward Traffic, FortiGate routes the
packet accordingly. If the action is Stop Policy Routing, FortiGate goes to the next table, which is the route
cache. You can view that content with the CLI command 'diagnose ip rtcache list'.
Finally, FortiGate searches the forwarding information base (FIB). The FIB is generated by the routing
process, and is the table used for packet forwarding. Think of the routing table’s purpose as for
management, while the FIB is for forwarding. This separation becomes clearer in FortiGate active-active
HA. In an HA cluster, both route management and forwarding tables exist on the master FortiGate. But on
the slave FortiGate, only the forwarding table (FIB) exists.
If there’s no match in any of those tables, FortiGate drops the packet because it is unroutable.

© FORTINET
When there is more than one route to a destination, this is process for selecting which route to use.
First, FortiGate uses the most specific route, which is the one with the longest net mask (smallest subnet).
If there are two or more routes with the same longest net mask, the unit selects the ones with the lowest
distance.
After that, the lowest metric is used as tie breaker for dynamic routes. In the case of static routes, the
priority is used instead.
If there multiples routes with the same net mask, distance, metric, and priority, FortiGate shares the traffic
among all of them. This is what is called equal cost multipath (ECMP), which is supported for static, BGP
and OSPF routes.

© FORTINET
The FortiGate adds a static route to the routing table only if all these requirements are met:
1- The outgoing interface is up
2- There is no other route with the same net mask and lower distance
3- The link health monitor (if configured) is up
4- The next-hop IP address is in the same subnet as one of the IP addresses assigned to the outgoing
interface

© FORTINET
Let's quickly review an important routing concept that was introduced in NSE4: reverse path forwarding
check.
It protects against IP spoofing attacks and routing loops by checking the route to the source IP address.
This check is done only on packets on the original direction (not on reply packets) and only when the
session was not created yet, or when the routing table has changed.
If the check fails, the packet is dropped and the debug flow shows the error:
reverse path check fail, drop

© FORTINET

There are two RPF check modes: Loose (which is the default mode) and strict.
In the loose mode, the packet is accepted as long as there is one valid route to the source IP through out
the incoming interface. It does not have to be the best route, just a valid one. In this example, the packet
from 10.4.0.1 to 10.1.0.1 is accepted because the FortiGate has a valid route (the default route) to 10.4.0.1
through out port2.
(click)
However, the packet from 172.16.1.1 to 10.1.0.1 is NOT accepted, as there is not any valid route to the IP
address 172.16.1.1 through out port3.

© FORTINET

In the case of strict mode, the FortiGate checks that the best route to the source IP address is through the
incoming interface. The route not only has to be valid (as in the case of loose mode), but it also has to be
the best one.
If in the same example, we change the FortiGate configuration from loose mode to strict mode. This is the
result:
1- The packet from 172.16.1.1 to 10.1.0.1 is still blocked as there is not route at all to that source IP
address through out port3
(click)
2- Now the packet from 10.4.0.1 to 10.1.0.1 is also blocked. There is a valid route to 10.4.0.1 through out
port2, but it is NOT the best route to that source IP address. The best route to 10.4.0.1 is through out
port3. So, in this case, strict mode accepts traffic from the subnet 10.4.0.0/24 only when port3 is the
incoming interface.

© FORTINET
UTM traffic inspection requires symmetric routing. Traffic both ways must follow the same path. So,
FortiGate routes traffic symmetrically. This means that, under some network topologies, FortiGate might
not route the return traffic through the best path, but through the path that the originating traffic is using.
For that purpose the unit “remembers” the “interface to source” and uses that interface to route the return
packets even when a better route using a different interface exists. Let’s see an example in the next slides.

© FORTINET

Let’s analyze this network topology. The “local” network 10.1.0.0/24 has three network devices: a local
workstation, a local router and a FortiGate (port1). Also the FortiGate port2 is directly connected to the
local router (using the subnet 10.2.0.0/24). There is a remote router connected to FortiGate port3 and,
behind that, a remote server (10.4.0.1). So, any traffic destined to the remote server must be routed
through the FortiGate. One important detail in this network is that the local workstation default gateway is
10.1.0.254.
(click)
This means that if you send a ICMP echo request from the local workstation to the remote server, the
packet goes to the local router first; then to the FortiGate; then to the remote router; and finally to the
destination. When the ICMP packet arrives to the FortiGate, an entry for the originating traffic is created in
the unit route cache. This entry contains the interface to source, or the incoming interface where the
packet arrived, which in this case is port2.

© FORTINET

Additionally, FortiGate creates an entry in the session table. This entry also contains information about the
interface to source.
(click)
As explained earlier, the FortiGate does a first routing lookup to find the next-hop to destination. That IP
address is also stored in the session information.
(click)
As there is not ICMP echo reply yet, you will notice that the next-hop to source is still undetermined (it is
0.0.0.0). It will be determined with the second routing lookup that happens with the first reply packet.

© FORTINET
Now, let’s see how the FortiGate routes the return packet.
When the FortiGate receives the ICMP echo reply, as there is already a session and a route cache
created, it uses the interface to source. So, in this case the unit routes the packet through out port2
towards the local router, even when there is a better route to the destination 10.1.0.1. The FortiGate
routing table shows port1 as the best route to 10.1.0.1 (locally connected), but it still uses port2. The
objective, as explained, is to keep the traffic flows symmetric.
With the first ICMP echo reply, a second entry is added to the route cache, this time for the return traffic.

© FORTINET
Additionally, and as explained earlier, the unit does a second routing lookup, this time to find the next-hop
(or gateway) to source. That IP address is added to the session (where 0.0.0.0 was before).

© FORTINET
Now, this raises one question: What happens is the traffic is originated from the server side instead? Let’s
say that the ping is sent from the remote server to the local workstation.
In this case, when the ICMP echo request arrives to the FortiGate, there is not session yet. So, the unit
uses the best route to 10.1.0.1, which is through port1.
These examples show how a FortiGate might, on some network topologies, route packets to the same
destination differently depending on who initiated the session.

© FORTINET
Something interesting in this last example is the reply traffic. As the local workstation default gateway is
10.1.0.254, the ICMP echo reply goes to the local router first. Then the packet arrives to the FortiGate
port2. The result is asymmetric routing as the return traffic is following a different path than the originating
traffic. The return packet is arriving to port2 instead of port1 (where the originating traffic was sent). In
these cases, the FortiGate accepts this asymmetry, no packets are dropped and UTM inspection is not
affected.

© FORTINET

After a change in the routing table, the routing information is removed from the sessions affected by the
change. Additionally, related route cache entries are deleted.
So, two more routing lookups are done again for the next packets to learn the new routing information and
store it in the routing table. Also, RFP check is done again for the first packet in the original direction.
This is a sample of a session just after a routing change. The gateways in both directions change to
0.0.0.0/0 and the interfaces to 0, indicating that this information must be learned again.
(click)
Additionally, the dirty flag is added.

© FORTINET
However, there is an important exception to this rule. After a routing change, sessions with source NAT
keep using the same outbound interface, as long as the old route is still valid. Let's see an example. In this
case, the FortiGate is connected to two different ISPs. There is client with the IP address 10.1.0.1/24
connected behind the FortiGate. The FortiGate is doing source NAT of the client traffic to a public IP
address that depend on which ISP is using. The unit routing table contains two default routes with the
same distance, but different priorities, one default route for each ISP. The route with the lowest priority
(port1) is the primary one. When both ISP connections are up, this is route selected by the FortiGate for
Internet traffic. So, all sessions to the Internet are created using port1 as the outbound interface.

© FORTINET
If the administrator increases the priority for port1 to a value higher than the priority in port2, this is what
happens:
1- All new sessions start using port2 as it is now the one with the lowest priority
2- However, all the existing sessions keep using port1. The default route through port1, although it is not
the best one now, it is still valid and the unit is doing source NAT. Those sessions will keep the old route
until they expire
If the unit were not doing source NAT, all the existing sessions would switch to port2 after the change.

© FORTINET
Most of the debug commands for routing are under the 'get router info' tree. Let's see some of them in
the next slides.

© FORTINET

The command 'get router info routing-table all' displays all the active routes in the routing table. The left
column indicates the source for the route.
(click)
The first number inside the brackets is the distance,
(click)
and the second one is the metric.
This command doesn't show inactive routes. For example, when two static routes to the same destination
subnet have different distances, the one with the lowest distance is active. The one with the highest
distance is inactive. So, this command displays only one of them (the active one).

© FORTINET
If you want to display both the active and the inactive routes, use instead the command:
get router info routing-table database
In this example the command shows one inactive route at the end. It is inactive because it has a higher
distance than the one above.

© FORTINET
This low-level command shows the actual forward information database (FIB), which is the routing
information that the kernel uses to route traffic. All the active routes in the routing table must be also
present in the FIB. Additionally, the FIB may contain routes that are not in the routing table and were
automatically added by the FortiGate.

© FORTINET
The route cache contains recently used routing entries in a fast-to-search table. It is consulted before the
routing table to speeds up the routing lookup process.

© FORTINET
This table contains all the IP addresses used by the FortiGate and assigned to FortiGate interfaces.

© FORTINET
The command 'get router info protocols' gives a quick look at the configuration of all the dynamic routing
protocols running in the unit.

© FORTINET
Now, we will move to transparent mode.

© FORTINET
In transparent mode, each VDOM forms a separate broadcast domain. Interfaces, by default, don’t. Until
you change the initial VDOM configuration, all interfaces, regardless of their VLAN ID, are part of the same
broadcast domain.
The interface command 'forward-domain' is used to segment the VDOM into multiple broadcast domains.
Interfaces with different forward domain IDs belong to different broadcast domains. Interfaces with the
same ID belong to the same broadcast domain.

© FORTINET

Here’s an illustration that explains the concept. Let's see first a broadcast with all the interfaces on the
same forwarding domain. An ARP request is sent by a single device. It reaches FortiGate through one of
the interfaces in the VDOM.
(click)
Because they all belong to the same forwarding domain, FortiGate then re-broadcasts the packet to all the
interfaces, even to interfaces that belong to a different VLAN.

© FORTINET

As we explained, forwarding domains are like broadcast domains.
Here’s we have the same network, but this time, VLAN 101 is only on 2 interfaces. Placing those
interfaces in a separate forward domain ID (101) segregates them.
(click)
Traffic arriving to one interface is only broadcasted to interfaces that are part of the same forwarding
domain.

© FORTINET
This debug command lists the MAC address forwarding table in a VDOM operating in transparent mode.

© FORTINET

© FORTINET
This lesson is about troubleshooting external BGP issues.

© FORTINET
The lesson has three main objectives. First, we will do a quick review of BGP concepts and configuration
on FortiGate devices. Second, you will learn to monitor and check the status of a BGP communication.
Finally, the lesson covers BGP troubleshooting.
BGP is a deep topic more extensive than what is covered here. This lesson shows only the basics about
the protocol and the FortiGate to troubleshoot of the most common external BGP issues.

© FORTINET
Let's start with a review of basic BGP concepts.

© FORTINET
An autonomous system (AS) is a set of routers and networks under the same administration. Each AS is
identified by a unique number and usually runs an interior gateway protocol, such as OSPF or RIP.

© FORTINET
An external routing protocol (EGP) exchanges routing information between autonomous systems. BGP4,
which runs in the Internet, is the dominant EGP protocol today. BGP is typically used when strict control is
required over a large number of routes.
Two BGP routers exchange AS path information for destination prefixes or subnets. When two routers
start a BGP communication, the whole BGP routing table is interchanged. After that, only network updates
are sent.

© FORTINET
A BGP speaker or peer is a router that sends and receives BGP routing information. The connection
between two BGP peers is called a BGP session.

© FORTINET
There are three types of autonomous systems:

-A Stub AS handles and routes only local traffic and it has only one connection to another AS. One
example is a company running BGP, with its own AS number, and one ISP connection
-Multi-homed AS also handles and routes only local traffic, but it has multiple connections to different
autonomous systems. One example is a company running BGP, with its own AS number, and multiple ISP
connections
-Transit AS handles and routes not only local traffic, but traffic that is originated and terminates in different
autonomous systems (transit traffic). An ISP is an example of a transit AS

© FORTINET
A BGP router stores the routing information into three logical tables. The RIB-In table contains all the
routing information received from other BGP routers before any filtering. The local RIB table contains that
same information after the filtering. The RIB-Out table contains the BGP routing information selected to
advertise to other BGP routers.

© FORTINET
This flow graph summarizes the BGP process. BGP routes received from other routers are stored in the
RIB-In table. A filter is applied and the resulting routes are stored in the local RIB table. Then routes
redistributed from the routing table are added, and another filter (outbound) is applied. The resulting routes
are the ones being advertised by the BGP router.

© FORTINET
BGP routes traffic based on AS paths. Each AS patch includes attributes, which some are used to select
the "best" route to each destination. One of the attributes is the AS list, which contains the autonomous
systems through which the traffic must pass to reach the destination.

© FORTINET
There are four types of BGP attributes:

- Well-known mandatory
- Well-known discretionary
- Optional transitive, which can be passed from one AS to another
- Optional non-transitive, which cannot be passed from one AS to another

© FORTINET
This is a list of the supported attributes with their types.

© FORTINET
Some of the attributes are used during the routing selection process. If all those attributes for multiple
routes to the same destination match, and if ECMP is enabled, FortiGate shares the traffic among up to 10
BGP routes. If ECMP is not enabled, the FortiGate uses the route to the router with the lowest BGP router
ID.

© FORTINET
There are three important considerations related with the way of how BGP is implemented in FortiGate
devices.
First, there are not hard-code limits. The limitations regarding the number of neighbors, routes and policies
depend exclusively on the system memory available.
Second, FortiGate by default does not announce any prefix. An administrator must enable redistribution or
manually indicate the prefixes to advertise.
Third, all prefixes received, by default, are accepted. Administrators can optionally filter out or modify some
prefixes.

© FORTINET
So, FortiGate BGP, by default, does not advertise any prefix. One way to configure FortiGate to do it is by
using the 'redistribution' commands. Connected, static and routes learned from other routing protocols
can be redistributed into BGP. You can optionally add route maps to filter the prefixes or modify some of
their BGP attributes.

© FORTINET
The other way to indicate the prefixes to advertise is by using the 'network' command. It is important to
know that, by default, an exact match of the prefix in the 'network' command must be active in the routing
table. If there isn't any active route whose destination subnet matches the prefix, FortiGate does not
advertise it. This default behavior can be changed by disabling the setting 'network-import-check.' Once
disable it, all prefixes in the BGP network table are always advertised, regardless of the actives routes
present in the routing table.

© FORTINET
This is a sample of a basic FortiGate BGP configuration. In this case the administrator has configured the
local AS (65075) and one neighbor with the IP address 10.127.0.117 and AS 65117. The BGP ID of the
local router is 0.0.0.75, and the redistribution of static routes into BGP is enabled.

© FORTINET
Also by default, FortiGate accepts BGP sessions only from BGP neighbours that are "locally" connected.
In other words, neighbours that are in the same subnet (and broadcast domain) than the local FortiGate.
There are at least two cases where this requirement creates problems:
First, when there are routers between the two BGP peers, in other words, when both peers are not in the
same subnet.
Second, when a loopback interface is used as the common BGP neighbour IP address for all the BGP
peers. For example, a FortiGate running BGP with two or more ISPs, and all the ISPs using the FortiGate
loopback IP address as their remote neighbor.
In those cases, administrators must enable 'ebgp-enforce-multihop'.

© FORTINET
We will move to BGP troubleshooting.

© FORTINET
This graph shows the BGP neighbor states and how they change:
Idle: Initial state.
Connect: Waiting for a successful 3-way TCP connection.
Active: Unable to establish the TCP session.
OpenSent: Waiting for an OPEN message from the peer.
OpenConfirm: Waiting from the keepalive messages from the peer.
Established: Peers have successfully interchange OPEN and keepalive messages.

© FORTINET
Most of the BGP diagnostic commands are under the 'get router info bgp' tree.

© FORTINET

This is usually the first debug command we use to get a quick overview of how BGP is working and the
status of all neighbors. It shows the local router ID and AS. For each neighbor, the output also displays:
Their AS,
(click)
packet counters,
(click)
and for how long the neighbor has been up.
(click)
The last column is the neighbor state and number of prefixes. If the state is not established, this column
displays the BGP state. If it is established, this column displays the number of prefixes received by the
local FortiGate from that neighbor.

© FORTINET
The command 'get router info bgp neighbors' provides detailed information about each BGP neighbor. It
includes packet counters, and number of prefixes announced and accepted.

© FORTINET
It also shows the number of times that the session has dropped and last time it was reset.

© FORTINET
This command provides details about the prefixes being advertised by the local router.

© FORTINET
This is the command to display the routes advertised by a neighbor.

© FORTINET
These are the commands for enabling and disabling the BGP real time debug.

© FORTINET
The next two slides show a sample of the real time debug output during the successful establishment of a
BGP session. It shows when the session goes to OpenSent state.

© FORTINET

When the session goes to OpenConfirm,
(click)
and then to Established.
(click)
It also lists the prefixes received once the BGP session is established.

© FORTINET
The command 'execute router clear bgp' is used to restart a BGP session between two peers. It is also
used to execute a BGP soft reset, which forces both peers to interchange their complete BGP routing
tables.

© FORTINET
Follow these steps if you are troubleshooting a BGP problem between two peers:
- Check that the local router can reach the remote peer
- If the remote peer is not on a connected network, enable EBGP multihop
- Check the TCP session
- Check the BGP session
- If the BGP session is established, check the prefixes received and advertised by each peer.

© FORTINET

© FORTINET
This lesson is about OSPF configuration and troubleshooting.

© FORTINET
This lesson reviews some OSPF concepts, such as link state advertisements, cost, adjacencies, and
areas. The lesson also covers OSPF troubleshooting and basic OSPF configuration.
Similarly to BGP, OSPF is a deep topic more extensive than what is covered in this lesson. This lesson
shows only the basics about the protocol and the FortiGate to troubleshoot of the most common OSPF
issues.

© FORTINET
Let's start with a review of basic OSPF concepts.

© FORTINET
In a link state protocol, such as OSPF, every router has a complete view of the network topology. Some of
the advantages of OSPF are scalability and fast convergence. Every 30 minutes routers re-advertise their
OSPF information. Between those 30-minute intervals, only updates are sent if a topology change is
detected. So, it is a relatively quiet protocol as long as the network topology is stable.
OSPF in large networks, although, may require good planning and be difficult to troubleshoot.

© FORTINET

Each router (in the same area) has identical and synchronized databases. We will talk about OSPF areas
soon.
(click)
An OSPF router uses the information in the LSDB and the Dijkstra's algorithm to generate an OSPF tree,
which contains the shortest path from the local router to each other router and network.
(click)
The tree gives the best route to each destination,
(click)
which is the information that OSPF can inject into the device's routing table.

© FORTINET
The topology information interchanged by OSPF peers is contained in link state advertisements (LSAs). A
router's LSBD is populated with the information from the local LSAs and all the LSAs received from other
routers.
A router sends link state update packets, which contain one or more LSAs.

© FORTINET
If there are multiple OSPF routes to the same destination subnet, OSPF selects the route with the lowest
cost. Each router interface is associated with an interface cost, which is usually related with how fast or
preferable that interface is. An OSPF route cost is the sum of all interfaces' costs to the final destination.

© FORTINET
The following two slides explain briefly how an OSPF router builds its OSPF tree. The initial information for
each router is the locally connected networks, together with the OSPF cost for each interface. For
example, the router R2 has three locally connected subnets: subnet Net 1 with a cost of 2, subnet Net 2
with a cost of 3, and subnet Net 3 with a cost of 1. Router R1 has only one subnet connected: Net 1 with a
cost of 2, and so on.
Each router starts advertising its locally connected subnets by sending LSAs.

© FORTINET
Then, OSPF routers use the Dijkstra’s algorithm to determine the best route to each destination. The best
routes can be represented as a tree with the local router at the root. The Dijkstra’s algorithm is a recursive
process that is repeated multiple times until the best routes are found. For example, this slide shows the
OSPF tree for router R2. It indicates that the best route to Net 5 and Net 4 is through R3; and that Net 1,
Net 2 and Net 3 are locally connected.

© FORTINET
An OSPF network can be segmented into areas. Each area is identified by a unique number, which can be
represented either in decimal or IP address format.

© FORTINET
Each area has its own separated LSDB. All routers in the same area maintain an identical copy of the
area's LSDB. As we will see later, a router can belong to more than one area. In those cases, the router
maintains multiple LSDBs, one LSDB for each area connected to it.
Segmenting big OSPF networks into areas reduces the sizes of the LSDB tables. Additionally, a topologies
change does not impact the whole network, but only the area where the change happens.
Using OSPF areas, although, requires good planning and may complicate the troubleshooting process.

© FORTINET
Any OSPF network must have at least one area, the backbone area. The backbone is the core of the
network with all the other areas connected to it in a hub-and-spoke topology.

© FORTINET
An internal OSPF router has all its interfaces connected to the same area. So, it maintains only one LSDB.
On the other hand, an area border router is connected to multiple areas, so it keeps multiple LSDBs.
A backbone router has at least one interface connected to the backbone area.
An autonomous system boundary router (ASBR) redistributes non-OSPF routes into the OSPF network.

© FORTINET
This is example that shows each router type.

© FORTINET
An OSPF session between two OSPF peers is called an adjacency. This slide shows the initial
interchange between two peers that are forming an adjacency. Any new adjacency goes through different
states: Init, 2-way, ExStart, Exchange, Loading and Full. The Full state indicates that the adjacency has
successfully formed, and both routers have identical copies of the LSDB.

© FORTINET
There are the requirements for two peers to form an OSPF adjacency. If any of the requirements is not
met, the adjacency fails and will not reach the Full state.

© FORTINET
There are two types of OSPF networks. Point-to-point networks contain only two peers, one at each end of
a point-to-point link. Multi-access networks support two or more peers.
There are two sub-types of multi-access networks. Broadcast networks support multicasting. An example
of a broadcast multi-access network is an Ethernet network. Non-broadcast multi-access networks
(NBMA) do not support multicasting. An example of a NBMA network is Frame Relay.

© FORTINET
In any multi-access network there are one designated router (DR) and one backup designated router
(BDR). The router with the highest priority is elected as the DR. If two or more routers are tied with the
highest priority, the router with the highest OSPF ID is elected.
The BDR monitors the DR status. If the DR fails, the BDR takes the DR role.
Other routers form adjacencies only with the DR and the BDR. The DR forwards the link state information
from one router to another. This simplifies the amount of adjacencies required in multi-access networks.

© FORTINET
These are the multicast addresses used by OSPF in broadcast multi-access, and point-to-point networks.
Keep in mind that OSPF also uses unicast addresses, for LSA retransmissions and database description
packets.

© FORTINET
There are 11 LSA types. This lesson covers the five more commonly used:
-Type 1 describes all the links connected to a router
-Type 2 describes all the routers (if more than one) in a multi-access network
-Type 3 describes the networks within an area (only generated by a ABR)
-Type 4 describes the path to reach an ASBR
-Type 5 describes the external destinations originated by an ASBR
We will see examples of each of these five types in the next slides.

© FORTINET
Type 1 describes the networks connected to a router. They are advertised by all the routers in an area.
Type-1 LSAs are not advertised outside the area where they originate.

© FORTINET
Type-2 LSAs are advertised only by DRs. In this example, the area has two multi-access networks, each
of them with one DR. The two DRs advertise type-2 LSAs, which contain information about the other
routers connected to their multi-access networks.

© FORTINET
Type-3 LSAs contain summarized link state information. They are advertised only by area border routers
(ABRs). In this example, the ABR in the left sends type-3 LSAs to the area 1. They contain link state
information for the summarized subnets in areas 0 and 2. This same ABR also sends type-3 LSAs to the
backbone area with a summary of the subnets in area 1.
Something similar happens with the ABR in the right. It sends type-3 LSAs to the area 2. They contain
links state information for the summarized subnets in areas 0 and 1. This same ABR also sends type-3
LSAs to the backbone area with a summary of the subnets in area 2.

© FORTINET
An ASBR advertises itself by sending type-1 LSAs. These LSAs have the E-bit on in the OSPF header. As
any other type 1, the LSAs with the E-bit are confined to the area where they originate. However, ABRs in
the same area send a type-4 LSA to the other areas with information about how to reach the ASBR. In this
example, an ASBR that is redistributing RIP routes into OSPF announces itself by sending type-1 LSAs to
the backbone area. The ABR receives that LSA and sends a type-4 LSA to the area 1.

© FORTINET
The last type of LSA covered in this lesson is type 5. They are sent only by the ASBRs and are not
confined to one area. Indeed, they reach all the standard areas. They contain link state information for
routes redistributed to OSPF (also called external routes).
Note: All the area examples in this lesson are standard areas. There are also stub and NSSA areas, which
are not covered in this lesson. Type-5 LSAs are not advertised to stub or NSAA areas.

© FORTINET
Each external route is assigned a metric. There are two types of external-route metrics. A type-1 metric is
the sum of the external cost plus the internal cost to reach the ASBR. A type-2 metric is only the external
cost (the internal cost is not considered). If there are two external routes to the same destination, one type
1 and one type 2, an OSPF router selects the type 1 over the type 2.

© FORTINET
This is a basic FortiGate OSPF configuration. It has the list of areas, the list of OSPF networks, and the
OSPF router ID.

© FORTINET
Any FortiGate that is redistributing non-OSPF routes into OSPF is an ASBR.

© FORTINET
The next part of this lesson is about OSPF troubleshooting.

© FORTINET
'get router info ospf status' shows general OSPF information and some statistics about OSPF.

© FORTINET
The output of 'get router info ospf status' also shows information about each area the router belongs to.

© FORTINET

For OSPF information about each interface, use the command 'get router info ospf interface'. It shows:
-Network type, in this case broadcast multi-access
(click)
- If it is a DR, or a BDR
(click)
- DR and BDR IP addresses
(click)
- Number of adjacencies and traffic statistics

© FORTINET
This command shows a summary of the status of all the OSPF neighbors. For each neighbor, it displays
the adjacency state; and if it is a DR, a BDR, or none of them (DROther). A dash is displayed after the
state if the neighbor is in a point-to-point network.

© FORTINET
The command 'get router info ospf database brief' provides a summary of all the LSDB entries in the
FortiGate, order by LSA types. It shows first the LSAs type 1 (router link states), then type 2 (net link
states).

© FORTINET
After that, the command shows the LSAs type 3 (summary link state), type 4 (ASBR-summary link states)
and type 5 (AS external link states).

© FORTINET
The command 'get router info ospf database self-originate ' lists the LSAs originated in the local
FortiGate.

© FORTINET
To get the details about each LSA, use the command 'get router info ospf database router lsa'.

© FORTINET
This is a sample of more output from the command 'get router info ospf database router lsa'.

© FORTINET
The OSPF real time debug displays information about adjacency establishments and OSPF errors. It also
shows information about network topology changes.

© FORTINET

© FORTINET
This lesson is about HA troubleshooting.

© FORTINET
The lesson reviews how virtual MAC addresses are assigned in a HA cluster. It also covers HA monitoring
and troubleshooting.

© FORTINET
We will start with a quick review of the HA operation material covered in NSE4. There are two HA modes:
active-passive and active-active.
Let’s examine first the active-passive mode. In any of the two HA operation modes, the configurations of
the secondary FortiGates are synchronized with the configuration in the primary device.
In the case of the active-passive mode, the primary FortiGate is the only device that actively processes
traffic. The secondary FortiGates remain in passive mode monitoring the status of the primary unit.
If a problem is detected in the primary FortiGate, one of the secondary devices will take over the primary
role. This event is what we call HA failover.
Session synchronization is an optional setting (which is disabled by default) that enables seamless failover
for some traffic. The information of some sessions is synchronized, so when the primary fails, the new
primary can take over those sessions where they were left and keep them open. Traffic might be
interrupted for a few seconds, but the network applications will not need to reconnect the sessions again.

© FORTINET
Like with active-passive HA, in active-active, all FortiGates’ configurations are synchronized. Also, if a
problem is detected with the primary device, one of the secondary units will take over the primary role.
However, one of the main differences with active-passive is that in active-active all of the FortiGates are
processing traffic. Indeed, one of the tasks of a primary FortiGate in active-active is to balance some of the
sessions among all the secondary devices.

© FORTINET
There are three reasons why a failover can happen:

1- When the primary stops replying to the heartbeats
2- When the link status of a monitored interface goes down - you can configure a HA cluster to monitor the
link status of one or more interfaces
3- When a server (IP address) stops replying to the ping sent by the primary - you can configure a HA
cluster to periodically send a ping to one or more servers to test the connectivity between the primary unit
and the network services

© FORTINET
To forward traffic correctly, a FortiGate HA solution uses virtual MAC addresses.

When a primary joins an HA cluster, each interface is given a virtual MAC. The primary informs all
secondary units about the assigned virtual MACs. Upon failover, a secondary adopts the same virtual
MAC addresses for equivalent interfaces.

© FORTINET
After fail over, the new primary also broadcasts gratuitous ARP packets, notifying the network that each
virtual MAC address is now reachable through a different switch port.
In most of the networks that is enough for the switches to update their MAC forwarding tables with the new
information. However, some high-end switches might not clear properly their MAC tables after a fail-over.
So, they keep sending packets to the former primary even after receiving the gratuitous ARPs. In those
cases, enable the command 'link-failed-signal' to force the former primary to shut down all its non-
heartbeat interfaces for 1 second when the failover happens. This simulates a link failure that clears the
related entries from the switches' MAC tables.

© FORTINET
The HA virtual MAC addresses assigned to each interface are determined by the HA group ID, the virtual
cluster ID and the interface index. So, if you have two or more HA clusters in the same broadcast domain,
and using the same HA group ID, you might get MAC address conflicts. For those cases, it is strongly
recommended to assign different HA group IDs to each cluster.

© FORTINET
The same command we used to get interface traffic statistics can be used to display the HA virtual MAC
address assigned to an interface.

© FORTINET

Let’s show how a HA cluster in active-active mode distributes traffic.
(click)
First, the client side sends a SYN packet. It’s forwarded always to the primary FortiGate using the internal
interface’s virtual MAC address as the destination.
(click)
If the primary decides that the session is going to be inspected by a secondary, the primary forwards the
SYN packet to the respective secondary. In this case, the destination MAC address is the physical MAC
address of the secondary FortiGate.
(click)
The secondary responds with SYN/ACK to the client and starts the connection with the server by directly
sending a SYN packet.

© FORTINET

Next the client acknowledges the SYN/ACK. It’s forwarded again to the primary using the virtual MAC
address as the destination.
(click)
The primary device forwards the packet to the secondary inspecting that session, using the secondary’s
physical MAC address.

© FORTINET

When the server responds to the TCP SYN, again, the packet is sent to the primary using the external
interface’s virtual MAC.
(click)
So the primary signals the secondary.
(click)
And it is the secondary the one who replies to the server.
As you can see, the idea of active-active mode is not to load balance bandwidth. The traffic is always sent
first to the primary. The main objective is to share CPU and memory among multiple FortiGates for traffic
inspection.

© FORTINET
If you connect to a secondary's console port while it is joining a HA cluster, these are the messages you
should see. The secondary tries to synchronize first what are called the "external files". They include the
FortiGuard databases and digital certificates. After that, the secondary synchronizes the configuration. The
last message indicates that the secondary has successfully joined the cluster.

© FORTINET
If the HA cluster has formed successfully, the GUI displays all the FortiGates, together with their
hostnames and serial numbers.

© FORTINET
When troubleshooting any problem in a HA cluster, it is useful to know that you can connect to the CLI of
any secondary from the primary CLI. You have to use the command 'execute ha manage' with the
secondary HA index for that purpose. To get the list of secondary FortiGates with their HA indexes, use the
question mark at the end of that same command.

© FORTINET
(slide contains animations)

From the CLI, although, you can get more information about the status of the HA. For example, the
command 'diagnose sys ha status' displays heartbeat traffic statistics,
(click)
as well as the serial number and HA priority of each FortiGate. The command also shows the heartbeat
interface IP address automatically assigned to the primary FortiGate.

© FORTINET
As explained in NSE4, the HA uptime is one of variables used to elect the primary unit. Depending on
other variables and configuration, at one point the units might compare their system uptimes to elect the
primary. If that happens and if there is one unit whose system uptime is 5 minutes more than the system
uptimes of all the other units, it is elected primary.
This command can be used to compare the system uptimes of all the units in a cluster.

© FORTINET
A good indication of the health of a HA cluster is the status of the configuration synchronization. To check
that all the secondary configurations are synchronized with the primary configuration, you can execute the
command 'diagnose sys ha showcsum' in all the HA units. If a secondary FortiGate displays exactly the
same sequence of numbers than the primary, its configuration it's well synchronized. Also, in each of the
devices, the debugzone and the checksum zone must display the same sequence of numbers. We will
suggest later some troubleshooting tips if that is not case.

© FORTINET
Instead of running the previous command in each of the cluster units, you can alternatively run this other
command only on the primary. It shows the checksum for all the cluster members. This command is easier
to use. However, if there are communication problems between one of the secondary units and the
primary, you might need to use the previous command instead.

© FORTINET
FortiGate HA uses FGCP, the FortiGate clustering protocol, for HA-related communications. FGCP travels
among the clustered FortiGate units over the links that you have designated as the heartbeats.
The FGCP traffic uses a different Ethernet type than the IP protocol. It actually uses three different
Ethernet types depending on the operation mode (transparent or NAT/route).

© FORTINET
HA session synchronization, as explained, is disabled by default. If it is enabled, you can check in the
primary's session table which sessions have been synchronized to the secondary units. They are the ones
with the synced flag. Additionally and for the case of all sessions, the ha_id field shows the HA member ID
of the unit that is processing the traffic.

© FORTINET
If a failover happens, the best tool to get information about it is the FortiGate logs. In case the failover
happened because the primary unit failed, the secondary unit's logs should show these log entries.

© FORTINET
If a new primary was elected because one or more monitored interfaces failed, the former primary displays
logs similar to these ones. In this case the primary unit is reporting a problem with the monitored interface
port1.

© FORTINET
If a unit cannot join a cluster follow these steps:

- Verify the HA settings
- Verify the firmware versions and hardware models
- Verify the physical layer connections
- Use the HA real time debug while the unit tries to join the cluster. Run the debug both in the primary and
the unit with the problem
If the checksums between the debugzone and checksum zones do not match, you can force the
recalculation.

© FORTINET
Traffic from session synchronization is bandwidth intensive. If the session creation rate is high, session
synchronization traffic might interfere with the heartbeat traffic, creating delays in the heartbeat replies.
There two configuration changes that might help in those cases:
1- Use an interface different than the heartbeat interface for session synchronization
2- Delay the synchronization of new sessions by 30 seconds, so short-lived sessions are not synchronized
High CPU issues could also create HA heartbeat problems. In those cases, troubleshoot and fix the high
CPU problem first before checking the HA status.

© FORTINET

FortiGate III Student Guide Online PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FortiGate III Student Guide Online PDF

Uploaded by

Copyright:

Available Formats

DO NOT REPRINT

VIRTUAL LAB BASICS ...................................................................................7

Transferring Files to the VM....................................................................................................13

Using HTML5 Instead of Java ................................................................................................13

Screen Resolution ...................................................................................................................14

International Keyboards ..........................................................................................................14

Troubleshooting Tips ..............................................................................................................15

SYSTEM RESOURCES ...................................................................................17

Time to Complete ....................................................................................................................17

System, Processes and Crashlog...........................................................................................18

Time to Complete ....................................................................................................................21

Exploring the Session Table ...................................................................................................22

Traffic sniffer ...........................................................................................................................25

Break and Fix: Connectivity Issues .........................................................................................28

Time to Complete ....................................................................................................................30

Traffic Shaping ........................................................................................................................31

Break and Fix: FTP Traffic ......................................................................................................32

FIREWALL AUTHENTICATION .........................................................................34

Time to Complete ....................................................................................................................34

Break and Fix: LDAP Authentication ......................................................................................35

Time to Complete ....................................................................................................................37

Installing FSSO .......................................................................................................................38

Break and Fix: FSSO ..............................................................................................................42

Time to Complete ....................................................................................................................44

Break and Fix: IPsec VPN ......................................................................................................45

SECURITY PROFILES ....................................................................................47

Time to Complete ....................................................................................................................47

Break and Fix: Protection Profiles Part 2 ................................................................................49

EXPLICIT WEB PROXY ..................................................................................51

Time to Complete ....................................................................................................................51

Break and Fix: Web Proxy ......................................................................................................52

Time to Complete ....................................................................................................................55

Transparent Mode ...................................................................................................................56

NAT/Route Mode ....................................................................................................................60

Break and Fix: NAT/Route Mode ............................................................................................62

EXTERNAL BGP ...........................................................................................63

Time to Complete ....................................................................................................................63

Break and Fix: BGP ................................................................................................................64

Time to Complete ....................................................................................................................66

Break and Fix: OSPF ..............................................................................................................67

Time to Complete ................................................................................................................. 69

Break and Fix: High Availability ............................................................................................ 70

APPENDIX A: ADDITIONAL RESOURCES........................................................72

APPENDIX B: PRESENTATION SLIDES ...........................................................73

Module 1: Troubleshooting Concepts...................................................................................74

Module 2: System Resources...............................................................................................97

Module 4: Firewall Policies.................................................................................................174

Module 5: Firewall Authentication.......................................................................................211

Module 8: Security Profiles.................................................................................................312

Module 9: Explicit Web Proxy.............................................................................................368

Module 10: Operation Modes.............................................................................................390

Module 11: External BGP...................................................................................................424

Module 12: OSPF...............................................................................................................456

Module 13: High Availability...............................................................................................496

FortiGate III Student Guide 7

FortiGate III Student Guide 8

FortiGate III Student Guide 9

FortiGate III Student Guide 10

FortiGate III Student Guide 11

FortiGate III Student Guide 12

Transferring Files to the VM

Using HTML5 Instead of Java

FortiGate III Student Guide 13