Professional Documents
Culture Documents
Troubleshooting ASA
Firewalls
Kevin Klous, CCIE #43604
Technical Leader, Services
BRKSEC-3020
#CLUS
Why do we need to troubleshoot an ASA?
• The Cisco ASA is a widely deployed, feature-rich, enterprise-class firewall that is
usually a critical component of the network environment
• Any misconfiguration or unexpected behavior of the device can quickly lead to network
outages
• Without a solid understanding of the ASA architecture, traffic flow, and troubleshooting
tools available, problems may be difficult to resolve and often have significant business
impact
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Presentation Objectives and Outcomes
• To combat this, today we're going to arm you with knowledge, skills, and tools to more
effectively troubleshoot and resolve incidents on the Cisco ASA platform
• Think about past or potential future experiences where you can apply these skills
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Introduction
Your Presenter
Kevin Klous
• Network and security enthusiast in Cisco Security TAC since October 2012
• Cisco Certified Internetwork Expert (Security – CCIE #43604)
• TAC Security Podcast host & panelist
• Pursuing M.S. in I.S. Engineering – Cybersecurity at JHU
• Serves as a Spanish translator for Guatemala missions
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Presentation Focus Areas
• Focused on architecture of the ASA firewall and datapath
troubleshooting
• Since ASA features have been integrated into the Firepower Threat
Defense (FTD) product, FTD may be mentioned throughout the
presentation
• The latest ASA hardware platforms are based on FXOS which is
outside the scope of this presentation (Firepower 2100, 4100, and
9300 series)
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Agenda
• Architecture Overview
• Path of the Packet
• Troubleshooting Tools
• Case Studies
• Best Practices
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Cisco Firepower Sessions: Building Blocks
Monday Tuesday Wednesday Thursday
BRKSEC-2031 BRKSEC-2064
08:00
16:00 13:30 10:30 08:00
13:30 08:00
BRKSEC-2050 BRKSEC-3455
BRKSEC-3035 We are here!
Firepower NGFW Dissecting
Internet Edge Firepower Firepower Platform
Deployment Installation & Deep Dive
Scenarios Troubleshooting
BRKSEC-2050 BRKSEC-2066
Firepower NGFW Optimizing Your
Internet Edge Firepower/FTD
Deployment Deployment
BRKSEC-2020
Firepower
BRKSEC-2058
Deployment Data Deep Dive into
Center & Enterprise Firepower Manager
Network Edge #CLUS © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Architecture Overview
ASA Memory
• Free memory may not recover immediately after conn spike due to caching
• Connections, Xlates, and ACL configuration are top users of shared memory
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
ASA Memory Blocks (Direct Memory Access)
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
ACE Explosion with Object Groups
• All configured ACLs are expanded before programming
10 source IP 21 destination IP
addresses addresses
33 TCP ports 6930 rules
• Nested Object Groups magnify the impact
• Add a new source Object Group with 25 additional objects
• Result: (10+25) x 21 x 33 = 24,255 rules (ACEs)
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Failover/HA Basics
Internet
• Active/Standby vs. Primary/Secondary
Corp
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Verifying Failover Operation
asa# show failover
Failover On
Failover unit Primary
Failover LAN Interface: failover Redundant5 (up)
Unit Poll frequency 200 milliseconds, holdtime 1 seconds
Interface Poll frequency 500 milliseconds, holdtime 5 seconds
Interface Policy 1
Monitored Interfaces 2 of 250 maximum
Version: Ours 9.7(1), Mate 9.7(1)
Last Failover at: 10:37:11 UTC May 14 2017
This host: Primary - Active
Active time: 1366024 (sec)
slot 0: ASA5585 hw/sw rev (1.0/8.1(2)) status (Up Sys)
Interface outside (10.8.20.241): Normal
Interface inside (10.89.8.29): Normal
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5585 hw/sw rev (1.0/8.1(2)24) status (Up Sys)
Interface outside (10.8.20.242): Normal
Interface inside (10.89.8.30): Normal
Stateful Failover Logical Update Statistics
Link : stateful Redundant6 (up)
Stateful Obj xmit xerr rcv rerr
General 424525 0 424688 0
sys cmd 423182 0 423182 0
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
What to Do After a Failover Event
• Always check the syslogs to determine root cause
• Example: switch port failed on inside interface of active firewall
• Check show failover history to see the state transition times and reasons
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
TAC Tips and Tricks - Failover TAC Tip
• Manually configure MAC addresses on interfaces
• Execute commands on the mate’s CLI with failover exec mate <command>
ASA-SM# failover exec mate show memory
Used memory: 31432840 bytes ( 0%)
------------- ----------------
Total memory: 25769803776 bytes (100%)
ASA-SM#
ASA-SM# config t
ASA-SM(config)# prompt hostname state priority
ASA-SM/act/pri(config)# exit
ASA-SM/act/pri# Active vs. Standby
Primary vs. Secondary
• Prevent config changes made directly to the standby (9.3(2) and later)
ASA-SM(config)# failover standby config-lock
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Clustering Basics
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
TAC Tips and Tricks - Clustering
• Use the ‘show cluster info’ command to view basic member information
• Use the ‘cluster exec’ command to execute any exec level command on all units
• Especially important for packet captures!
asa# cluster exec cap capin interface inside match ip any any
unit-2-1(LOCAL):******************************************************
unit-1-1:*************************************************************
asa# cluster exec show cap
unit-2-1(LOCAL):******************************************************
capture capin type raw-data trace interface inside [Capturing - 83 bytes]
match ip any any
unit-1-1:*************************************************************
capture capin type raw-data interface inside [Capturing - 46627 bytes]
match ip any any
asa#
• Use ‘cluster exec show cluster history’ command to view the cluster state
transitions of all units in the cluster.
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Best Practices - Clustering
• Set the cluster control link (CCL) MTU to 100 bytes larger than largest data
interface to avoid CCL fragmentation
• Many common problem domains for ASA clustering involve NAT
• Only use NAT in conjunction with clustering if necessary
• Do NOT use a single interface PAT in a cluster
• Results in ALL traffic matching the rule to be sent to the master unit
• Use a PAT pool with at least as many IPs as there are units in the cluster
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
The Path of the Packet
Understanding Packet Flow
Effective troubleshooting requires an understanding of the packet path in network
1. Attempt to isolate the problem down to a single device
2. Perform a systematic walk of the packet through device to identify problem
TCP outside 172.16.164.216:5620 inside 192.168.1.150:50141, idle 0:00:00, bytes 0, flags saA
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Example Flow
• TCP Flow
• Source IP : 10.1.1.9 Source Port : 11030
• Destination IP : 198.133.219.25 Destination Port : 80
• Interfaces
• Source: Inside Destination: Outside
10.1.1.9
Servers
Packet Flow With the Flow defined,
Eng Accounting examination of configuration
issues boils down to just the two
Interfaces: Inside and Outside
Outside
198.133.219.25
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Packet Processing: Ingress Interface
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
• If no existing connection
• TCP SYN or UDP packet, pass to ACL and other policy checks in Session Manager
• TCP non-SYN packet, drop and log
ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on
interface inside
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Packet Processing: NAT Un-Translate
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
• NAT rules that translate the destination of the packet can override the routing table to
determine egress interface (NAT divert)
• Could also override policy-based routing (PBR)
• In ASA 8.4(2) and later versions you can use the ‘show nat divert-table command’ to view
compiled NAT divert structure
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Packet Processing: ACL Check
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Packet Processing: Stateful Inspection
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Packet Processing: FirePOWER Redirection
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
• Traffic that matches a redirect policy to an SFR module will be sent to the module for additional
inspection. Configuration example:
policy-map global_policy
class sfr_class
sfr fail-open <- enables redirection
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Packet Processing: NAT IP Header
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Object-NAT (Auto-NAT)
• Object NAT is the simplest form of NAT, and is defined within an object
Host NAT
object network obj-WebServer
host 10.3.19.50
nat (inside,outside) static 198.51.100.50
Network NAT
object network Servers
subnet 10.0.54.0 255.255.255.0
nat (inside,outside) static 203.0.113.0
• Twice NAT can specify the source and the destination translation
Network Objects
object network 10.10.10.0-net
subnet 10.10.10.0 255.255.255.0
!
object network 192.168.1.0-net
subnet 192.168.1.0 255.255.255.0
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
NAT Order of Operation version For your
reference
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Packet Processing: Egress Interface
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
172.16.0.0/16
nat (inside,outside) source static 172.16.0.0-net 192.168.0.0-net
nat (dmz,outside) source static 172.16.12.0-net 192.168.12.0-net
172.16.12.4
172.16.12.0/24
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Packet Processing: L3 Route Lookup
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
%ASA-6-110003: Routing failed to locate next hop for TCP from inside:192.168.103.220/59138
to dmz:172.15.124.76/23
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Packet Processing: L2 Address Lookup
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
• Once a Layer 3 route has been found and a next hop IP address identified, Layer 2
resolution is performed
• Layer 2 rewrite of MAC header
• If Layer 2 resolution fails — no syslog
• show arp will not display an entry for the L3 next hop
• debug arp will indicate if we are not receiving an ARP reply
arp-req: generating request for 10.1.2.33 at interface outside
arp-req: request for 10.1.2.33 still pending
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Packet Processing: Transmit Packet
FirePOWER
Module
Yes
No Yes Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Troubleshooting Tools
Syslogs and their Usage
• Primary mechanism for recording connections to and through the firewall
Console
Syslog/FTP Server
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Defining Custom Syslog Levels
• Assign any syslog message to any available level
Levels
• Problem:
0—Emergency
You want to record what exec commands are
being executed on the firewall; syslog ID 111009 1—Alert
records this information, but by default it is at 2—Critical
level 7 (debug)
3—Errors
ASA-7-111009: User ‘johndoe’ executed cmd: show run
4—Warnings
The problem is we don’t want to log all 1775 5—Notifications
other syslogs that are generated at debug level
6—Informational
asa(config)# logging message 111009 level 3
7—Debugging
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Logging – Common Issues
• logging flash-bufferwrap should only be used when logging to buffer at Level 1
• logging history should only be used when you really have an SNMP server that you want to
receive all syslogs
• logging console should only be enabled while actively troubleshooting on the console
• logging standby should only be used if you want to receive double the syslogs
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Debug Commands
• Debugs should not be the first choice to troubleshoot a problem
• Debugs can negatively impact the CPU complex and affect performance
• Know how much traffic of the matching type is passing through the firewall before enabling
the respective debug
• For example, when troubleshooting a SIP issue, be sure to check the number of SIP connections
(‘show sip’) before enabling SIP debugging to estimate the volume of debug output
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
See
Show Output Filters Appendix
• Filters limit the output of show commands to only what you want to see
• Use the pipe character “|” at the end of show <command> followed by
• begin Start displaying the output beginning at the first match of the RegEx, and
continue to display the remaining output
• include Display any line that matches the RegEx
• exclude Display any line that does not match the RegEx
• grep Same as include
• grep –v Same as exclude
• redirect Send output to a file (flash, tftp, ftp…)
• append Append output to an existing file (flash, tftp, ftp…)
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
CPU Utilization by Processes
• show processes cpu-usage command displays the amount of CPU used on a per-process
basis for the last 5 sec, 1 min, and 5 min
Heavy CPU load from
SNMP traps.
asa# show process cpu-usage sorted non-zero
PC Thread 5Sec 1Min 5Min Process
0x08dc4f6c 0xc81abd38 14.4% 8.2% 8.0% SNMP Notify Thread
0x081daca1 0xc81bcf70 1.3% 1.1% 1.0% Dispatch Unit
0x08e7b225 0xc81a28f0 1.2% 0.1% 0.0% ssh
0x08ebd76c 0xc81b5db0 0.6% 0.3% 0.3% Logger
0x087b4c65 0xc81aaaf0 0.1% 0.1% 0.1% MFIB
0x086a677e 0xc81ab928 0.1% 0.1% 0.1% ARP Thread
• If you have high CPU utilization for a generic process such as DATAPATH, contact the
TAC as there are more granular CPU profiling tools available to investigate deeper
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Traffic Rates
Uptime statistics is useful to determine historical average
packet size and rates:
asa# show traffic
52128831 B/sec / 39580 pkts/sec = ~1317 B/packet
[…]
TenGigabitEthernet5/1:
received (in 2502.440 secs):
99047659 packets 130449274327 bytes
39580 pkts/sec 52128831 bytes/sec
transmitted (in 2502.440 secs):
51704620 packets 3581723093 bytes
20661 pkts/sec 1431292 bytes/sec
1 minute input rate 144028 pkts/sec, 25190735 bytes/sec
1 minute output rate 74753 pkts/sec, 5145896 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 131339 pkts/sec, 115953675 bytes/sec
5 minute output rate 68276 pkts/sec, 4748861 bytes/sec
5 minute drop rate, 0 pkts/sec
One-minute average is useful to detect bursts and small packets:
25190735 B/sec / 144028 pkts/sec = ~174 B/packet
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Xlate Table (NAT / PAT translations)
• show xlate displays information about NAT translations through the ASA
• Second biggest memory consumer after conn table, no hardcoded size limit
• You can limit the output to just the local or global IP
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Detailed NAT Information TAC Tip
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Example: TCP Connection Establishment
1. Client sends TCP SYN to 2. Permit flow, create half-opened
10.1.1.1/80 through FTD stateful conn with flags saA
(awaiting SYN ACK, ACK)
5. Complete 3-way
6. Create full conn, update flags to
handshake with
U (Up)
TCP ACK
inside outside
192.168.1.101 10.1.1.1
7. Apply stateful checks, update
6. Send first data packet flags to UI (inside data seen)
TCP outside 10.1.1.1:80 inside 192.168.1.101:50141, idle 0:00:00, bytes 153, flags UIO
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Example: TCP Connection Termination
TCP outside 10.1.1.1:80 inside 192.168.1.101:50141, idle 0:00:00, bytes 153, flags UIO
inside outside
192.168.1.101 10.1.1.1
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
For your
TCP Connection Flags reference
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
TCP Connection Termination Reasons
• If a TCP flow was built through the ASA, it will always log a teardown reason
• For problems with abnormal connection termination, temporarily increase logging level and
check the teardown reason
What do these termination reasons mean in the Teardown TCP connection syslog?
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
For your
Reason Description
Conn-Timeout Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout
Idle Timeout Connection Timed Out Because It Was Idle Longer than the Timeout Value
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
For your
Reason Description
Force Termination After Twenty Seconds Awaiting
SYN Timeout
Three-Way Handshake Completion
TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission
TCP Fins Normal Close Down Sequence
TCP Invalid SYN Invalid TCP SYN Packet
TCP Reset-I TCP Reset Was Sent From the Inside Host
TCP Reset-O TCP Reset Was Sent From the Outside Host
TCP Segment Partial Overlap Detected a Partially Overlapping Segment
TCP Unexpected Window Size Connection Terminated Due to a Variation in the
Variation TCP Window Size
Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down
Unauth Deny Connection Denied by URL Filtering Server
Unknown Catch-All Error
Xlate Clear User Executed the ‘Clear Xlate’ Command
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Local Host Table
• A local-host entry is created for every IP tracked by the ASA
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Accelerated Security Path (ASP)
• Packets and flows dropped in the ASP will increment a counter
• Frame drop counters are per packet
• Flow drops are per flow
• See command reference under show asp drop for full list of counters
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Packet Capture Inside Capture Outside Capture
Inside Outside
Capture IN Capture OUT
• Inline capability to record packets passing through the ASA
• Apply capture under unique name to ingress and egress interfaces
• Define the traffic that you want to capture, use pre-NAT “on the wire” information
• Tcpdump-like format for displaying captured packets on the box
Match covers both directions
asa# capture OUT interface outside match ip any host 172.18.124.1
asa# capture IN interface inside match ip any host 172.18.124.1 of the flow
asa# show capture IN
4 packets captured
Yes
No Yes Yes Yes
RX Ingress Existing NAT ACL Stateful NAT IP Egress L3 L2 TX
Pkt Interface Conn Untranslate Permit Inspection Header Interface Route Addr Pkt
No No No No No
DROP
• Packets are captured at the first and last points they can be in the
flow
• Ingress packets are captured before most packet processing (Frags)
• Egress packets are captured after all processing
• Transit packets show the destination MAC address rewritten
• Self-sourced packets may show an empty MAC address (0000.0000.0000)
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Packet Capture
• Capture buffer maintained in RAM (512KB by default, 33 MB max) Much larger
• Stops capturing when full by default, circular option available capture sizes
coming soon!
• Default recorded packet length is 1518 bytes
• May elevate CPU utilization on multiple-core ASAs when applied under high packet rates
• Copy captures off via TFTP or retrieve through HTTPS with your web browser
• Do this before removing the capture with no capture
https://x.x.x.x/admin/capture/OUT/pcap/outsidecapture.pcap
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Capturing ASP Drops
• Capture all frames dropped in the ASP
‘match’ command for
asa# capture drops type asp-drop all filtering ASP drops was
added in ASA 9.6(2)
• Capture all frames with a specific drop reason
asa# capture drop type asp-drop ?
acl-drop Flow is denied by configured
rule
all All packet drop reasons
bad-crypto Bad crypto return in packet
bad-ipsec-natt Bad IPSEC NATT packet
bad-ipsec-prot IPSEC not AH or ESP
bad-ipsec-udp Bad IPSEC UDP packet
bad-tcp-cksum Bad TCP checksum
bad-tcp-flags Bad TCP flags
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Capturing IPv6 Traffic
• Packet capture and other troubleshooting tools support IPv6
• Only major caveat for capture is that we must match via ACL not inline ‘match’
asa# show run access-list IPV6
access-list IPV6 extended permit ip any6 any6
asa# show cap capin
1: 11:21:14.228641 2002:db8:1:1::20 > ff02::1:ff00:254: icmp6: neighbor sol: who has 2002:db8:1:1::254
2: 11:21:15.230411 2002:db8:1:1::20 > ff02::1:ff00:254: icmp6: neighbor sol: who has 2002:db8:1:1::254
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Packet Tracer
• Unique capability to record the path of a specially tagged packet through ASA
• Best way to understand the packet path in the specific software version
• Inject a simulated packet to analyse the behaviour and validate configuration
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Sample Packet Tracer Output
asa# packet-tracer input outside tcp 172.18.124.66 1234 172.18.254.139 3389
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list
Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside
Additional Information:
NAT divert to egress interface dmz
Untranslate 172.18.254.139/3389 to 192.168.103.221/3389
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Sample Packet Tracer Output (Cont’d)
Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_in in interface outside
access-list outside_in extended permit tcp any any eq 3389
Additional Information:
……
Phase: 8
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside
Additional Information:
Dynamic translate 172.18.124.66/1234 to 192.168.103.221/1234
……
Phase: 12
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 16538274, packet dispatched to next module
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Packet Tracer in ASDM
Launch from Tools >
Packet Tracer
Associated
configuration
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Packet capture with trace
• Enable packet tracer within an internal packet capture
asa# capture IN interface inside trace trace-count 200 match tcp any any
Trace inbound
Traced packet count per
packets only
capture (1-1000, 50 by default)
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Packet capture with trace (continued)
• Likely the most used datapath troubleshooting tool in the TAC
• You can now capture traffic post-decryption across a VPN tunnel w/ ASA as VPN endpoint:
asa# capture OUT interface outside trace include-decrypted match tcp any any
asa# packet-tracer input inside tcp 10.1.1.20 10000 10.1.2.100 80 transmit detailed
asa# sh cap capout
1 packet captured
1: 12:08:30.837709 10.1.1.20.10000 > 10.1.2.100.80: S 1119191062:1119191062(0) win
Without this option, the packet is never transmitted onto the wire. This can be useful for troubleshooting.
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Packet Capture with Trace Demo
Lab Topology for Demo
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
TCP Ping
• Powerful troubleshooting tool added in ASA 8.4(1)+
• Verify bi-directional TCP connectivity from the ASA or host to a remote server using
injected packet
• Provides ASA policy and upstream path verification without client host access
• TCP RST and ICMP error responses are intercepted and displayed as well
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Embedded Event Manager (EEM)
• Troubleshooting tool added in 9.2(1), similar to IOS EEM
• Powerful way to run CLI commands based on ASA events (syslogs) and
save the output
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
EEM – Triggers and Uses
• Time-based events
– Every midnight back up the ASA configuration to your TFTP server
– Every 3 hours gather the output of ‘show memory detail’ and save it to the flash
• Syslog based events
– If the available 1550 byte blocks become depleted, gather ‘show blocks pool 1550
dump’ and save to the disk
– If the AAA server is marked down: ‘ping tcp’ to the server on port 49, ‘show aaa-
server’ to gather statistics, save to a file on disk, use SCH to email the file contents
• Manual events
– Gather the output of 10 different commands and save to a file
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
EEM Example – Configuration backup
• Goal: Backup the config when a user logs in and when they log off an SSH session
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
EEM – WannaCry detection example
• Goals:
1. Identify any DNS lookups to the WannaCry kill switch domain
2. Drop the traffic and store relevant syslogs to flash for investigation
– Add regex to default DNS inspection policy-map:
regex wannacry "www(.)iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea(.)com”
policy-map type inspect dns preset_dns_map
match domain-name regex wannacry
drop log
– Configure EEM
event manager applet WANNACRY event syslog id 410003
action 0 cli command "show logging”
output file new
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Case Study 1:
UDP connections fail
after reload
Problem Summary
• After reloading the ASA, wireless mobility traffic (UDP and IP Protocol 93) from inside WLC
to DMZ WLC fails
• Other traffic (TCP) recovers successfully
1. Standalone
ASA is reloaded
10.10.1.2 DMZ
2. UDP/16666 and
10.10.9.0/28
IP/93 connections fail
10.10.9.3
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Checking Connection Table and Drops
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Checking Packet Tracer
• Packet Tracer shows that a new UDP flow will be correctly passed to DMZ
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Reviewing Packet Captures
asa# capture IN interface inside match udp host 10.10.1.2 host 10.10.9.3
asa# capture OUT interface dmz match udp host 10.10.1.2 host 10.10.9.3
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
The Problem: We have a U-Turn Connection
• Traffic is looping back out the inside interface back towards the sender
10.10.1.2 DMZ
10.10.9.0/28
10.10.9.3
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Root Cause
• When conn entry was first created, route lookup for 10.10.9.3
resolved to inside interface
• Since DMZ interface was not “up/up”, the route to 10.10.9.0/28 was
not present 1. Standalone ASA
is reloaded
10.10.9.3
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Floating Connection Timeout
• The “bad” connection never times out since the UDP traffic is constantly flowing
• TCP is stateful, so the connection would terminate and re-establish on its own
• ASA needs to tear the original connection down when the corresponding route changes
• ASA 8.4(2)+ introduced timeout floating-conn to accomplish this goal
asa# show run timeout
timeout xlate 9:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 9:00:00 absolute uauth 0:01:00 inactivity
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00 Schedule the conn entry for termination
asa#
in 1 minute if a matching packet yields a
asa# configure terminal
different egress interface on route
asa(config)# timeout floating-conn 0:01:00
lookup
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Case Study 2:
Clients cannot reach
server
Case Study 2: Clients cannot reach server
The customer states that clients traversing the ASA are not able to access an internal web server.
However, other clients on the server subnet (10.1.2.0/24) are able to access the server.
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Using our ASA troubleshooting tools
%ASA-6-302013: Built inbound TCP connection 46927 for inside:10.1.1.20/2286 (10.1.1.20/2286) to outside:10.1.2.100/80 (10.1.2.10 0/80)
%ASA-6-302014: Teardown TCP connection 46927 for inside:10.1.1.20/2286 to outside:10.1.2.100/80 duration 0:00:30 bytes 0 SYN Tim eout
So, now we know that we are receiving the packet but either the server is not responding or the ASA
is not forwarding it. Let’s dig deeper.
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Packet Captures – The single source of truth
What do we know at this point?
The ASA is receiving the packet. The ASA is building the TCP connection for the flow.
The next step here is to determine if the ASA is actually forwarding the packet.
Let’s use our awesome packet capture tools for this.
Houston…we have a
capture capout type raw-data interface outside problem.
[Capturing - 0 bytes]
match tcp any host 10.1.2.100 eq www No packets going to the
destination server?
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Visual troubleshooting recap
10.1.1.20 10.1.2.100
TCP/80 (HTTP)
• Packet is received
• ASA is building connection
• However, ASA is not forwarding
Let’s review! What are possible reasons that the ASA may
drop traffic without a syslog?
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Checking ASA inspection and L2 adjacency
Remember, we can use packet capture with the ‘trace’ command to see policy deci
asa# show cap capin trace packet-number 1
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
Next-hop ARP resolution?
Root cause:
asa# show route A static, more specific /32 route to the
… server via 10.1.2.50 is configured and
S 10.1.2.100 255.255.255.255 [1/0] via 10.1.2.50, outside
that host is not responding to ARP.
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
ASA CLI Analyzer
cway.cisco.com/go/sa
Best Practices
ASA Best Practices
• Avoid interface oversubscription: maximize packet size and minimize rate
• Baseline CPU load, connection and xlate counts, and per-interface traffic rates
• Upgrade major feature trains only for new features or when they mature
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
ASA Best Practices
• Remove ACL entries that remain at 0 hit counts over time
• Log to at least one syslog server, do not configure more than 3
• Move syslog messages you want to see to lower levels or create logging lists instead of
raising logging levels and capturing messages you don’t want to see
• Use NSEL for recording connection information and disable redundant syslogs
• Troubleshoot with a variety of tools, including syslogs, show commands, Packet Tracer,
packet captures
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
TAC Security Podcast
Knowledge from TAC… On the go!
• Periodic podcast episodes with
troubleshooting tips from TAC
• Focus on new Cisco Security technology
trends like Firepower Threat Defense, ISE
2.1, Anyconnect, voice security, etc…and
CCIE study tips!
@CiscoTACPodcast
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Podcast Episodes
Ep. # Topic Ep. # Topic
Finding Your Firepower - A discussion on Firepower
50 36 Network Management at Cisco Live! 2013
Technologies
Cryptic Thoughts - A discussion on changing crypto
49 35 Identity Services Engine v1.2
standards
48 Cisco Live! 2015 - San Diego 34 Cisco Live! 2013 Orland, FL
Virtual Security: The ASA 1000v and Virtual Security Gateway
47 The Cisco Security Ninja Program 33
(VSG)
46 New features in ASA version 9.3(2) 32 Investigating Syslogs: Tips and Tricks
45 Introduction to Cisco Wireless Security 31 A look into ASA Quality with the Quality Assurance Team
44 Cisco Live! 2014 in San Francisco 30 Introducing FlexVPN
43 ASA Version 9.2 and Interesting TAC Cases 29 Cisco Live! 2012 San Diego
42 The Cisco Secure Development Lifecycle 28 The History of the PIX
41 Troubleshooting ASA Clustering 27 IOS Embedded Event Manager (EEM)
40 Introduction to ASA Clustering 26 Troubleshooting IPSec VPNs
39 Voice Security Concepts and Best Practices 25 Understanding DMVPN and GETVPN
38 Introduction to OnePK 24 The Cisco Identity Services Engine (ISE)
37 ASA Network Address Translation (NAT) 23 The Cisco ASA Services Module
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Security Beta Programs
Beta Software Product Access to Test Hardware Bugs Fixed for Influence
Access Training Dev Teams and Licenses Release Product Roadmap
#CLUS © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Wrapping it up
• Take new knowledge, skills, and tools and apply them to your daily ASA troubleshooting.
• Check out additional resources as well as slides in the appendix for future reference
purposes.
• You should now have a better understanding of ASA architecture, traffic flow, and
troubleshooting tools to help you more quickly resolve issues.
• If you leverage those newfound skills and resources, before you know it you’ll be
troubleshooting ASA like a TAC engineer!
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
Complete your online session evaluation
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
Continue
your Demos in
the Cisco
Walk-in
self-paced
Meet the
engineer
Related
sessions
education campus labs 1:1
meetings
#CLUS BRKSEC-3020 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
Thank you
#CLUS
#CLUS