JUN

29

CLI Commands for Troubleshooting Palo Alto Firewalls

CLI Commands for

Troubleshooting Palo Alto
Firewalls
https://blog.webernetz.net/cli-commands-for-troubleshooting-palo-alto-firewalls/

When troubleshooting network and security issues on many different devices/platforms I

am always missing some command options to do exactly what I want to do on the device
I am currently working with. Therefore I list a few commands for the Palo Alto
Networks firewalls to have a short reference / cheat sheet for myself. Maybe some
other network professionals will find it useful.
However, since I am almost always using the GUI this quick reference only lists
commands that are useful for the console while not present in the GUI.
This blog post will be a living document. Whenever I use some “new” commands for
troubleshooting issues, I will update it. If there are any useful commands missing,
please send me a comment!
For a complete list of all CLI commands, use the CLI Reference Guides from PAN. Or
use the official Quick Reference Guide: Helpful Commands PDF.

Standard Show & Restart Commands

The following commands are really the basics and need no further description. I list them
just as a reference:
 show system info //shows the uptime of the device

show system environmentals //e.g. power supply failures

3 show ntp

6 show session info //packet rate, number of sessions, fastpath active, etc.

9 show session id <id>

10

11

12 show interface { all | <interface-name> }

13

14

15 show routing route //routing table (all routes)

16

17

18 show routing fib //forwarding table (only used routes)

19

show routing protocol <protocol> ...

show arp { all | <interface-name> }

 show neighbor interface { all | <interface-name> } //IPv6 neighbor cache

show mac all //only with layer 2 interfaces

show jobs all

show jobs id <id>

show running resource-monitor //resource statistics

show system resource follow //="top", CPU usage and processes

show system disk-space //="df -h"

debug software restart <service> //Restart a certain process

request restart system //Reboot the whole device

Problems with SFPs

To troubleshoot SFP problems use the following command such as shown here:, where
XXX is the slot and YYY is the port:

1
show system state filter-pretty sys.sXXX.pYYY.phy

Sample output with one non functional and one functional SFP in port ethernet1/19:
Click To Expand Code

Find
Since PAN-OS 6.0, the “find” command helps searching for the needed command in case
you do not fully know the whole set of commands. With “find command”, all possible
commands are displayed. With “find command keyword xyz”, all commands containing
“xyz” are shown.

find command
1

find command keyword <word-to-search-for>

Ping, Traceroute, and DNS

A standard ping command looks like that:

1
ping host 8.8.8.8
Note that this ping request is issued from the management interface! To use a data
interface as the source, the option source can be used. To use IPv6, the option is inet6
yes. For example:

1
ping inet6 yes source 2003:51:6012:120::1 host 2a00:1450:4008:800::1017

A traceroute command looks like that:

1
traceroute host 8.8.8.8

The source can be used to specify the outgoing interface. However, for IPv6, the option
is dissimilar to the ping command: ipv6 yes.
To resolve DNS names, e.g., to test the DNS server that is configured on the
management interface, simply ping a name:

1
ping host ip.webernetz.net

Routing
(For a “show” of the routing table refer to the “Standard Show Commands” above.)
Debugging dynamic routing protocols functions like this:

2
debug routing pcap <routing-protocol> on
3

5
debug routing pcap show
 debug routing pcap <routing-protocol> view

debug routing pcap <routing-protocol> off

debug routing pcap <routing-protocol> delete

Or follow the routed.log:

1
tail follow yes mp-log routed.log

Test
The Palo offers some great test commands, e.g., for testing a route-lookup, a VPN
connection, or a security policy match. Use the question mark to find out more about
the test commands. Here are some useful examples:

test routing fib-lookup virtual-router default ip <ip>

1

3
test vpn ipsec-sa tunnel <value>
4

test security-policy-match ?
 test security-policy-match from trans-internet to pa-trust-server source 192.168.86.5 destination 192.168.120.2 protocol

6 application ssl destination-port 443

Viewing Management-Plane Logs

In order to view the debug log files, “less” or “tail” can be used. The keyword “mp-log”
links to the management-plane logs (similar to “dp-log” for the dataplane-logs). The tail
command can be used with “follow yes” to have a live view of all logged messages. And
as always: Use the question mark in order to display all possibilities.
Examples:

less mp-log ?

1
less mp-log dnsproxyd.log
2

4
tail follow yes mp-log dhcpd.log

tail follow yes mp-log routed.log

Capturing Management Packets

To view the traffic from the management port at least two console connections are
needed. The first one executes the tcpdump command (with “snaplen 0” for capturing
the whole packet, and a filter, if desired),

1
tcpdump snaplen 0 filter "port 53"

while the second console follows the live capture:

1
view-pcap follow yes mgmt-pcap mgmt.pcap

Test traffic can be generated with a third console session, e.g.:

1
ping host webernetz.net

Later on, the pcap file can be moved to another computer with the following command:

1
scp export mgmt-pcap from mgmt.pcap to <username@host:path>

Alternatively, tftp can be used:

1
tftp export mgmt-pcap from mgmt.pcap to <host>

Live Viewing of Packet Captures

When using the Packet Capture feature on the Palo Alto, the filter settings can easily be
made from the GUI (Monitor -> Packet Capture). These settings as well as the current
size of the running packet capture files can be examined with:

1
debug dataplane packet-diag show setting

Now, the current capturing in follow mode can be viewed with:

1
view-pcap follow yes filter-pcap

And for a really detailed analysis, the counters for these filtered packets can be viewed.
This exactly reveals how many packets traversed which way, and so on. With the “delta
yes” option, only the counter values since the last execution of this command are
shown. The “packet-filter yes” option uses the packet filter from the GUI (Monitor ->
Packet Capture) to filter the counters:

1
show counter global filter packet-filter yes delta yes

For example, here are the delta counters after a few DNS lookups:
Click To Expand Code
Or, even more interesting, filtered on “drop” severity. (Note the reasons on the right-
hand side):
Click To Expand Code

Zone Protection Logging

Beginning with PAN-OS 8.1.2 you can enable an option to generate a threat log entry for
dropped packets due to zone protection profiles. More information here. You must enable
this feature through the CLI. (Hopefully it will be default at a later date.)

1
set system setting additional-threat-log on

Examining the Session Table

If a network connection failure is not found in the traffic log, the session table can be
asked for sessions in DISCARD state, filtered based on its source, or whatever. This is
useful at the console because the session browser in the GUI does not store the
filter options and is therefore a bit unhandy. All commands start with “show session
all filter …”, e.g.:

show session all filter state discard

2
show session all filter application dns destination 8.8.8.8
3

show session all filter from trust to untrust application ssl state active

To have an overview of the number of sessions, configured timeouts, etc.:

1
show session info

For investigating a single session in more detail, use:

1
show session id <id>

Watch out for the: “Hardware session offloading” line. If it is “true” you might want to
disable the fastpath during troubleshooting (inside the config mode):

set session offload no

1

set deviceconfig setting session offload no //= persistent, even after reboot. CAUTION!

To see whether there are some “predict” sessions in which the Palo Alto uses a ALG
(appliation layer gateway) to predict dynamic ports (e.g., SIP, active FTP), use this
command:

1
show session all filter type predict

A specific session can then be cleared with:

1
clear session id <value>

Reason for Session Close

[UPDATE] Since PAN-OS 6.1 the session end reason is a column within the GUI at
Monitor -> Logs -> Traffic. Hence this is not needed anymore.[/UPDATE]
You cannot see the reason for a closed session in the traffic log in the GUI. For this
purpose, find out the session id in the traffic log and type in the following command in the
CLI (Named the “Session Tracker“). Note the last line in the output, e.g. “tracker stage
firewall : Aged out” or “tracker stage firewall : TCP FIN”. This shows what reason the
firewall sees when it ends a session:

1
show session id <id>

Alternatively, the traffic log on the CLI can display the session tracker when used with
the option “show-tracker equal yes” such as:

show log traffic show-tracker equal yes

2
show log traffic show-tracker equal yes direction equal backward
3

show log traffic show-tracker equal yes direction equal backward app equal ipv6-icmp from equal pa-ripe-atlas

VPN Issues
The general show commands for VPN sessions are:

1 show vpn gateway

show vpn ike-sa

(Palo Alto: How to Troubleshoot VPN Connectivity Issues). Though you can find many
reasons for not working site-to-site VPNs in the system log in the GUI, some more CLI
commands might be useful. To reveal whether packets traverse through a VPN
connection, use this: (it shows the number of encap/decap packets and bytes, i.e., the
actual traffic flow)

1
show vpn flow name <value>

Or use the counter values for ipsec issues:

1
show counter global filter delta yes | match ipsec

And for a detailled debugging of IKE, enable the debug (without any more options)

1
debug ike pcap on

then follow the pcap with

1
view-pcap follow yes debug-pcap ikemgr.pcap

and do NOT forget to set the debugging off!

1
debug ike pcap off
The complete ikemgr.pcap can be downloaded from the Palo with scp or tftp, e.g.:

1
scp export debug-pcap from ikemgr.pcap to <username@host:path>

To clear or to initiate an IPsec connection use the following commands for either
phase 1 (IKE) or phase 2 (IPsec):

clear vpn ike-sa gateway <value>

1
clear vpn ipsec-sa tunnel <value>
2

4
test vpn ike-sa gateway <value>

test vpn ipsec-sa tunnel <value>

GlobalProtect
Current users and flow:

1
show global-protect-gateway current-user
2
 show global-protect-gateway flow

Displaying the Config in Set Mode

The XML output of the “show config running” command might be unpractical when
troubleshooting at the console. That’s why the output format can be set to “set”
mode:

1
set cli config-output-format set

Now, enter the configure mode and type show. This reveals the complete configuration
with “set …” commands. (Click here for more information.) Here is a sample output of a
particular show command:

weberjoh@fd-wv-fw02# show network interface ethernet ethernet1/1

1
set network interface ethernet ethernet1/1 layer3 ip 172.16.1.2/24
2

4
set network interface ethernet ethernet1/1 layer3 untagged-sub-interface no
5

7
set network interface ethernet ethernet1/1 layer3 interface-management-profile ping

set network interface ethernet ethernet1/1 link-speed auto

 set network interface ethernet ethernet1/1 link-duplex auto

set network interface ethernet ethernet1/1 link-state auto

The pipe (|) can be used to grep certain values with the “match” keyword, such as:

weberjoh@fd-wv-fw02# show | match 192.168.120.2

2
set deviceconfig system ip-address 192.168.120.2
3

set address h_fd-wv-fw02_mgmt ip-netmask 192.168.120.2

To show the complete config without breaks (which is “terminal length 0” on Cisco
devices), the following command can be used (BEFORE the configure mode is entered):

1
set cli pager off

To omit line breaks (carriage returns), use this one:

1
set cli terminal width 500
High Availability
Some show commands for the HA:

show high-availability ?

show high-availability all

1
show high-availability state
2

4
show high-availability link-monitoring
5

7
show high-availability path-monitoring

show high-availability control-link statistics

show high-availability state-synchronization

The following request can be used to trigger an HA failover, either for the local device or
the “peer” device:
1

3 request high-availability state suspend

4

request high-availability state functional

request high-availability state peer suspend

request high-availability state peer functional

To verify the session synchronization (HA2), you can either use the show high-
availability state-synchronization as shown above on both devices (to verify that “sent” is
increasing on the active unit while “received” is increasing on the passive unit) or you can
look at the session browser on the passive device whether there are the same count of
sessions as on the active device.
Following is a demo output of the “state-synchronization” from both devices in a cluster:
Click To Expand Code

Export/Import Files
To copy files from or to the Palo Alto firewall, scp or tftp can be used. The commands
have both the same structure with “export … to” or “import … from”, e.g.:

scp export log system to <username@host:path_to_destination_filename>

1

3
scp import software from <username@host:path>
4

tftp export configuration from running-config.xml to <tftp-host>

 tftp import url-block-page from <tftp-host>

User-IDs and Groups

State of the LDAP server connections incl. the listing of all groups:

1
show user group-mapping state all

Group mapping and user-id agent refresh (=update) and reset (=delete and reload):

debug user-id refresh group-mapping all

1 debug user-id refresh user-id agent all

debug user-id reset group-mapping all

debug user-id reset user-id-agent all

Show the group memberships for a particular user:

1
show user user-IDs match-user <value>

Show the members of a particular group:

1
show user group name "AD\name-of-the-group"

IP to User mapping for all users or for a particular user. (The match value does not work
with a backslash, so the username must be specified without the domain):

show user ip-user-mapping all

1

show user ip-user-mapping all | match <username>

User-ID cache clearance. Note that you must clear both, the dataplane AND the
management plane (…-mp), to really delete an IP mapping. Since the MP pushes the
mapping to the DP you should clear the MP first. More info here.

clear user-cache-mp all

1

3
clear user-cache-mp ip <ip>
4

5
 clear user-cache all

clear user-cache ip <ip>

IP Addresses of FQDN Objects

When using objects with FQDNs, the current IP addresses are not shown in the GUI. The
following command displays respectively refreshes them:

1
request system fqdn { show | refresh }

To set the refresh timer to another value, use the following commands:

configure

2
set deviceconfig system fqdn-refresh-time <600-14399>
3

commit

To verify this setting you can “show” the configuration with pipe and match. If you are in
the default cli config-output-format it looks like this:
1

3 weberjoh@pa# show | match fqdn-ref

 fqdn-refresh-time 600;

[edit]

When you are in the “cli” config-output-format it looks like that:

weberjoh@pa# show | match fqdn-ref

2
set deviceconfig system fqdn-refresh-time 600
3

[edit]

Now, as in my case I am updating the FQDNs every 600 s = 10 m, I can see the
appropriate job every 10 minutes:

2
weberjoh@pa> show jobs all
3

7
Enqueued Dequeued ID PositionInQ Type Status Result Completed
8

9
 ------------------------------------------------------------------------------------------------------------------------------------------

2017/02/22 09:55:35 09:55:35 185 FqdnRefresh FIN OK 09:55:37

2017/02/22 09:45:31 09:45:31 184 FqdnRefresh FIN OK 09:45:32

2017/02/22 09:35:28 09:35:28 183 FqdnRefresh FIN OK 09:35:31

2017/02/22 09:25:24 09:25:24 182 FqdnRefresh FIN OK 09:25:25

2017/02/22 09:15:21 09:15:21 181 FqdnRefresh FIN OK 09:15:21

IP Addresses of External Dynamic Lists

Similar, the entries in a external dynamic (block) list can be viewed or refreshed with:

request system external-list show type {ip|name|url} name <name-of-the-list>

1

request system external-list refresh type {ip|name|url} name <name-of-the-list>

DNS Proxy
To verify the functionality of DNS proxy objects, at least two commands are useful. Both
outputs should speak for themselves:

show dns-proxy statistics all

1

show dns-proxy cache all

Active URL Vendor/Database

I had some issues with the two different URL databases “brightcloud” and “PAN-DB”. This
is the command to show unambiguously which vendor is active on the PA (independent
of the licenses):

1
show system setting url-database

The output is either brightcloud or paloaltonetworks. The standard URL DB up to PAN-

OS 5.0 is brightcloud. Beginning with PAN-OS 6.0, the default is PAN-DB (refer to the
release notes, section “Changes to Default Behavior”). To change the vendor (of course
only if it is licensed), click the “Activate” link under licenses in the GUI.

PAN-DB URL Test & Cache

To show the category of a specific URL, use one of the following commands:
1

2
3 test url <fqdn>

test url-info-cloud <fqdn>

test url-info-host <fqdn>

To display the current URL cache from the PAN-DB, two steps are required. The first one
is the creation of a logfile which contains all entries and the second one is to display this
logfile:

show system setting url-cache all

1

less dp-log dp_url_DB.log

Fan Speed
Ok, this is not a troubleshooting command, but nevertheless very useful. It sets the fan
speed to “auto” which immediately drops the noise of the fan, e.g. on a PA-200:

1
set system setting fan-mode auto

Defaults
 Just for reference:
 Default Management Interface IP: 192.168.1.1
 Login: admin
 Password: admin
To change the static IP settings of the management inferface via the console:

configure

2 set deviceconfig system ip-address 192.168.1.5 netmask 255.255.255.0 default-gateway 192.168.1.1 dns-setting servers

3 primary 8.8.8.8

commit

Or to change it to a DHCP client (of the management interface), use this:

configure

2 set deviceconfig system type dhcp-client send-hostname yes send-client-id no accept-dhcp-domain no accept-dhcp-host

3 name no

commit

And wait for a console message such as DHCP: new ip 10.100.20.175 : mask255.255.255.128 .
Otherwise you can show the management IP address via showinterface management . If you
later on want to change back to static IP addresses you must not only use the set
command above (for the mere IP address) but also change the type back to
static: set deviceconfig system type static.
To perform a factory reset without direct access to the firewall via a console cable,
you can use this procedure.

COMMAND DESCRIPTION
General System Health
Shows the system’s management IP, serial #, and
show system info
code version
Shows when commits, downloads, upgrades are
show jobs processed
completed
show system disk-space Shows percent usage of disk partitions
show system logdb-quota Shows the maximum log file sizes
show system software status Shows running processes
Monitor CPUs
Shows processes running in the Management
show system resources
Plane
show running resource-monitor Shows the resource utilization in the Dataplane
Dropped Packet Troubleshooting
Ping from a specified device source interface to
ping source host
destination IP
ping host Ping from the management interface
Shows specific sessions in the sessions table for
show session all filter source destination
source and destination IPs.
show session info Shows usage, pps rates, etc
Shows session details by entering the session ID
show session id
number.
Packet Filters and Capture - WARNING: Running debug commands on a production device may cause
instability or other undesirable results!
debug dataplane packet-diag clear all
Clear/delete settings and files previously created.
debug dataplane packet-diag clear log log
delete debug-filter file * Removes all packet capture files
debug dataplane packet-diag set filter match source
x.x.x.x destination y.y.y.y destination-port
Sets filter with the source IP, destination IP and
debug dataplane packet-diag set filter match source
port to capture from/to packets.
y.y.y.y destination x.x.x.x destination-port debug
dataplane packet-diag set filter on
debug dataplane packet-diag set capture stage receive
file pantacrx.pcap
debug dataplane packet-diag set capture stage
transmit file pantactx.pcap
Configures the different stage of capture types to
debug dataplane packet-diag set capture stage drop
be executed.
file pantacdrop.pcap
debug dataplane packet-diag set capture stage firewall
file pantacfw.pcap
debug dataplane packet-diag set capture on
debug dataplane pack-diag show setting Verifies packet filters are setup correctly.
While test is running, run the command 2-3 times
show counter global filter delta yes packet-filter yes
to verify filtered traffic is being captured.
debug dataplane packet-diag set capture off Turns off packet capture and filter
tcpdump filter “src net ”
tcpdump snaplen 1500 filter “src net ” Captures PCAP on management interface.
view-pcap mgmt-pcap mgmt.pcap
Packet Flow Logs - WARNING: Always set specific packet filters to minimize CPU usage. See above Packet
Filters and Capture commands.
debug dataplane packet-diag set log feature flow basic Set packet-diag log to capture flow basic
debug dataplane packet-diag set log on Turns on packet-diag log.
Capture traffic then immediately disable packet-
debug dataplane packet-diag set log off
diag log.
Aggregates pack-diag logs to a single file. After
debug dataplane packet-diag aggregate-logs disabling packet-diag log, wait 1-2 minutes before
running this command.
View packet-diag log output. Note: PA-5000 series
less dp-log pan_packet_diag.log
writes to individual dp0-log, dp1-log or dp2-log
Log/Forward Device Issues
Shows the log statistics, like logging incoming rate,
debug log-receiver statistics log written rate, corrupted packets and logs
discarded due to a full queue.
less mp-log logrcvr.log Shows debug logging issues on the device.
debug software restart log-receiver Restarts log-receiver process.
Log Viewing/Deleting
show log [system | traffic | threat] direction equal Goes to the beginning/end of a log. Note:
[forward | backward] Arguments shown with square bracket [] and pipe
 | symbols mean choose one of the arguments
listed.
Monitor Management or Device Server
Shows management server messages for commit
show system resources follow
failures, updates, licenses, link status, policy
tail follow yes mp-log ms.log
details, etc.
Shows device server message for commit failures,
tail follow yes mp-log devsrv.log
updates, licenses, link status, policy details, etc.
Authentication Logs
Shows the detail authentication logs on the
less mp-log authd.log
device.
NAT
show running nat-policy Shows current NAT policy table.
show running ippool
Shows NAT pool utilization.
show running global-ippool
Routing
show routing route Shows routing table.
Policies
show running security-policy Shows current policy set.
User-ID Agent
Shows agent’s status. Status should be connected
show user user-id-agent state all
OK and there should be numbers shown under
show user user-id-agent statistics
users, groups, and IPS.
show user user-ids show user user-IDs
show user group-mapping state all
show user group-mapping statistics Shows the groups pulled from User-ID Agent.
show user group list
show user group name
show user ip-user-mapping all Shows IP to username mappings.
clear user-cache all
Clears user-ID cache.
clear user-cache ip
BrightCloud URL Filtering
test url Tests categorization of a URL on the device.
tail follow yes mp-log pan_bc_download.log Shows the BrightCloud database update logs.
debug dataplane show url-cache statistics Shows statistics on the URL cache
clear url-cache url Clears URL cache for a site.
Shows the URL log, most recent entries first.
show log url direction equal backward Note: Cache contains 100k of the most popular
URLs on the network.
ping host service.brightcloud.com Tests connectivity to the BrightCloud servers.
PAN-DB URL Filtering
show url-cloud status Check URL cloud status.
Tests categorization of a URL on Dataplane cache.
debug dataplane test url-resolve-path
Tests categorization of a URL on Management
test url-info-host
Plane cache.
test url-info-cloud
Tests categorization of a URL on Cloud.
clear url-cache url Clears URLs from the Dataplane cache.
delete url-database url Clears URLs from the Management Plane cache
show running url-cache statistics Show statistics on URL Dataplane cache.
debug device-server pan-url-db show-stats Show statistics on URL Management Plane cache
IPSEC
show vpn flow Shows encap/decap counters
show vpn gateway Shows list of IKE gateway configurations.
show vpn ike-sa Shows IKE Phase 1 SA
show vpn ipsec-sa Shows IPSEC Phase 2 SA.
show vpn tunnel Shows list of auto-key IPSec tunnel configurations.
show log system subtype equal vpn direction equal
backward Shows detail debug information for IPSec
debug ike global on debug tunneling.
less mp-log ikemgr.log
High Availability
show high-availability state Shows the HA state of the device.
Shows the HQ settings configured on the device
show high-availability all
and peer.
show high-availability state-synchronization Shows if the devices are synchronized
Suspends active device and makes passive device
request high-availability state suspend
active
request high-availability state functional Changes the state from suspend to passive.
Software, Content and Licenses
request restart system Reboots the system.
request content upgrade Upgrades content.
> check Gets info from Palo Alto Networks server.
> download Downloads content packages.
> info Displays available content packages info.
> install Installs content packages.
request content downgrade install previous Downgrades to previous content version
request license info Shows the license installed on the device.
delete license key Deletes a license file.
Miscellaneous
configure
set deviceconfig setting session tcp-reject-non-syn no Ignore SYN when creating sessions.
commit
show session info Confirms command took effect
configure Make all packets go through CPU, otherwise all
set deviceconfig setting session offload no fastpath packets go through the chip. Turns
commit session offload to fastpath.
show session info Confirms command took effect.
Shows the different dataplane buffers and
debug dataplane pool statistics
capacity
Posted 29th June 2018 by http://networksecurityknowledge2016.blogspot.com