Troubleshooting Traffic Flow Issues Through A VPN Tunnel On The ASA

3/23/2020 Troubleshooting traffic flow issues through a VPN tunnel on the ASA - cisco
akashtha My Settings Messages (1) Help Sign Out

ASA Virtual Private Network (VPN)
Tech Zone Discussion Board Tribal Knowledge Base Content Request Queue Topic (TZ Only) Search
Tech Zone Tech Zone Knowledge Base Security Knowledge Base Virtual Private Networks (VPN) Knowledge Base ASA VPN
Troubleshooting traffic flow issues through a VPN tunnel on the ASA

40
Troubleshooting traffic flow issues through a VPN tunnel on the ASA Kudos
Started 10-27-2012 by Modified 04-29-2016 by Edit Article Options

atbasu atbasu
Troubleshooting traffic flow issues through a VPN tunnel on the ASA Reminder: Link Your Cases
(21,480 Views)
Please remember to link your support
cases to Tech Zone articles or
by atbasu 10-27-2012 09:21 PM - edited 04-29-2016 12:56 PM discussions that help you solve them.
This helps everyone understand which
Activity: Debug, Troubleshooting content is most useful, gives credit to the
Content: Troubleshooting Guide contributors, and could impact what
appears in Topic Search results and which
Feature: IKEv2, VPN
articles get published externally for our
Product (Cisco): ASA customers to read.
Protocol, Standards & Languages: IPsec
Instructions for linking cases are here, and
Table of Contents additional information is here.
Introduction
Prerequisites
Requirements
Components Used Publishing Life Cycle
Background Information
Basic things to know about the ASP table entries Step 1: Internal
SA information
ASP Drop Data Step 2: External Preview
Packet Processing
Troubleshooting Methodology Step 3: External
Step 1 - confirm tunnel is up
Step 2 - identify which rule is being used for the faulty traffic
Step 3 - identify why it's not hitting the right rule
Outputs to collect Link Your Case
Known Problems
Problem 1 Traffic is blackholed and the syslog "Duplicate entry already in Tunnel Manager" is seen
Enter case #
What does this syslog mean?
Why do we see this syslog? Same Problem Link
When is it normal to see this syslog?
Solution
Related Information
Link An Automation Task
Enter BDB App Name Link

Introduction
This document describes how to troubleshoot one/more of the following symptoms:

1. In a site-to-site tunnel one of the peers is unable to bring up a tunnel, but the minute traffic is initaited from the other end everything Actions
works fine
2. Remote access clients or even a site-to-site peer are able to bring up the tunnel just fine, but traffic is only unidirectional, i.e. traffic
Edit Article
in the opposite direction reaches the ASA but doesn't get encrypted.
3. Traffic suddenly stops flowing in either/or both directions for a working SA. Flag for Improvement
Nominate for External Publication
In any of these scenarios the ipsec debugs by themselves are usually useless for troubleshooting the issue primarily because the Article Options
problem is either that:
1. the ASA isn't even negotiating the SA, in which case there won't be any debugs, or
2. the tunnel is already up, and the debugs, again, have little to tell. Labels
This document describes what tools to use for troubleshooting such issues and how to do it. It also explains why the
Activity:
syslog/debug "Duplicate entry already in Tunnel Manager" is seen on the ASA.
Debug, Troubleshooting
Content:
Contributed by Cisco Engineers. Troubleshooting Guide
Feature:
Prerequisites IKEv2, VPN
Product (Cisco):
Requirements
ASA
There are no specific requirements for this document. Protocol, Standards & Languages:
IPsec
Components Used
This document is not restricted to specific software and hardware versions.

Contributors
https://techzone.cisco.com/t5/ASA-Virtual-Private-Network-VPN/Troubleshooting-traffic-flow-issues-through-a-VPN-tunnel-on-the/ta-p/118722 1/24
atbasu
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document
started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any
command. anchacko
malhyari
Background Information
Basic things to know about the ASP table entries
dlia
Once a packet comes to the ASA, the connection table is looked up. If it isn't there, then its punted to the ASP, where its decided if the
packet has to be NATed, encrypted,etc. Before ike is enabled on an interface the ASP table is empty:
ciscoasa(config)# sh asp tabl clas cry brquinn
Input Table
Output Table: josemed
L2 - Output Table:
L2 - Input Table: ballowe
Last clearing of hits counters: Never
dasilver
As soon ike is enabled on the outside interface, the ASP table is populated with two entries that indicate that should the ASA recieve
any encrypted ipv4 or ipv6 packets that don't match any other rule in the table they should be dropped and a third entry for packets
destined to the outside interace's ip address that will allow the packets to be processed: dwhitejr
ciscoasa(config)# cry isa en outside

ciscoasa(config)# sh asp tabl clas cry
tac-ws
Input Table
in id=0x7fffdbe01bc0, priority=13, domain=ipsec-natt, deny=false
hits=0, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=17
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=1.1.1.1, mask=255.255.255.255, port=4500, dscp=0x0
input_ifc=outside, output_ifc=any Case Links
in id=0x7fffda80ac60, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=0, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
623187221
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0 1110287160 Same Problem
input_ifc=outside, output_ifc=any
in id=0x7fffdbcf1d20, priority=13, domain=ipsec-tunnel-flow, deny=true 1110494501 Same Problem
src ip/id=::/0, port=0 1110581338 Same Problem
dst ip/id=::/0, port=0
input_ifc=outside, output_ifc=any 622585211
Output Table: 623855883 Same Problem
L2 - Output Table: 624019633
L2 - Input Table: 624067375
Last clearing of hits counters: Never 624321433
At this time the domain classifier table and the vpn-context tables are all empty as there's no configuration: BDB Tasks
ciscoasa(config)# sh asp table vpn-context
ciscoasa(config)# sh asp table class domain encrypt There were no tasks found in Big Data Broker
that match the labels associated with this
Input Table board.
You should write one!
Output Table: Get started here: BDB Training Material
Feedback/Info
L2 - Output Table:
L2 - Input Table: Top Tags
Last clearing of hits counters: Never

ciscoasa(config)# sh asp table class domain decrypt Add Tag...
View All
Input Table
Output Table:
Related Links
L2 - Output Table:
621042699: ASA IPsec doesn't
L2 - Input Table:
start Quick Mode for some SA with
Last clearing of hits counters: Never IKEv1 SA up
ciscoasa(config)# Troubleshooting: ASA-7-752008:
Duplicate entry already in Tunnel
Now we add the following configuration to the ASA: Manager
BGL-O-15-ASA5500-1(config)# sh run cry

crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec ikev2 ipsec-proposal AES256
protocol esp encryption aes-256
protocol esp integrity sha-1 md5
crypto ipsec security-association pmtu-aging infinite
crypto map outside_map 1 match address l2l_list
crypto map outside_map 1 set peer 1.1.1.1
crypto map outside_map 1 set ikev2 ipsec-proposal AES256
crypto map outside_map 478 match address outside_0478
crypto map outside_map 478 set pfs
crypto map outside_map 478 set ikev1 transform-set ESP-3DES-MD5
crypto map outside_map 478 set security-association lifetime seconds 28800
crypto map outside_map 478 set security-association lifetime kilobytes 4608000
crypto map outside_map interface out
crypto ca trustpool policy
crypto ikev2 policy 1
encryption aes-256
integrity sha
group 2
prf sha
lifetime seconds 86400
crypto ikev2 enable out
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400
BGL-O-15-ASA5500-1(config)# sh run access-l

access-list outside_0478 extended permit ip host 192.168.1.1 host 192.168.2.1
access-list l2l_list extended permit ip host 1.1.1.2 host 1.1.1.1
Once you configure "match address" under the crypto map,the ASA install the encrypt entries into the ASP domain classifier table with
a user_data value of 0x0, which indicates to the ASA that it needs to bring up a tunnel. Once the tunnel is established, the user_data
value will be a non-zero value. So, in the ASA we see the following ASP table entry before the packet is encrypted( i am sending traffic
that matches sequence number 1):
BGL-O-15-ASA5500-1(config)# sh asp table classify domain encrypt
Input Table
Output Table:
out id=0x7ffd902111d0, priority=70, domain=encrypt, deny=false
hits=0, user_data=0x0, cs_id=0x7ffd9020d410, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=255.255.255.255, port=0, tag=0
dst ip/id=0.0.0.0, mask=255.255.255.255, port=0, tag=0, dscp=0x0
input_ifc=any, output_ifc=out
out id=0x7ffd90205ef0, priority=70, domain=encrypt, deny=false
out id=0x7ffd901803b0, priority=70, domain=encrypt, deny=false
hits=0, user_data=0x0, cs_id=0x7ffd7d5b0ee0, reverse, flags=0x0, protocol=0
out id=0x7ffd9097a290, priority=70, domain=encrypt, deny=false
src ip/id=::/128, port=0, tag=0
dst ip/id=::/128, port=0, tag=0
out id=0x7ffd8e5e8d60, priority=70, domain=encrypt, deny=false
The cs_id is the cascade delimiter and it contains a crypto map entry identifier. The very first entry with src and dst IP addresses as
0.0.0.0 is the beginning of a particular crypto map. The cs_id value will be identical for all crypto ACL entries of a particular crypto map.
The following document though outdated, explains the ASP table concepts well:
Crypto tunnel and the ASP Table
Once the entry is in the ASP table, the NP checks the domain and in this case it says "encrypt", so this packet is supposed to be
encrypted and sent over the tunnel. But, the user data is 0x0, which means that the tunnel is not up. This is where this packet is sent to
the tunnel manager. The tunnel manager decides whether this should be an IKEv1 or an IKev2 tunnel. This is how it makes this
decision:
The tunnel manager looks at the configuration and goes through the crypto map configured for that ASP table rule. It records the proxy
IDs(1.1.1.1 and 1.1.1.2), the name of the crypto map(outside_map) and the sequence number of the crypto map(10). Now, if both ikev1
and ikev2 are enabled for this crypto map, the tunnel manager dispatches this request to IKEv2 first.
Oct 06 08:00:53 [IKE COMMON DEBUG]Tunnel Manager dispatching a KEY_ACQUIRE message to IKEv2. Map Tag = o
Once the tunnel manager has already dispatched this packet to IKEv2, if it gets another packet that has the same proxy IDs, it will print
this syslog, since it has already dispatched a packet to IKEv2. The tunnel manager then discards this packet.
SA information
1. show crypto ipsec sa [entry | identity | map map-name | peer peer-addr] [detail
2. show asp table vpn-context detail
3. show asp table classify crypto
To understand the correlation between these show commands refer to the diagram below:
So in order for a packet to be properly encrypted each of these outputs must allign correctly. This will be further clarified in the next
section where the data is analysed.
Tip: The ASP crypto classifier table will always have two entries for every outbound SPI once the tunnel comes up:
out id=0x7fffa3856080, priority=70, domain=encrypt, deny=false

hits=0, user_data=0x5c7824, cs_id=0x7fff9f663350, reverse, flags=0x0, protocol=0
dst ip/id=192.168.0.0, mask=255.255.0.0, port=0, tag=0 dscp=0x0
input_ifc=any, output_ifc=outside
out id=0x7fffa3817160, priority=70, domain=encrypt, deny=false
hits=54010, user_data=0x0, cs_id=0x7fff9f663350, reverse, flags=0x0, protocol=0
dst ip/id=192.168.0.0, mask=255.255.0.0, port=0, tag=0 dscp=0x0
input_ifc=any, output_ifc=outside
One of the the entries will have an appropriate user_data value, while the second entry will have the user_data set as 0x0. This second
entry is a catchement rule that is generated as soon as the crypto map is created and before the VPN tunnel is initiated. So when
interesting traffic reaches the ASA for the first time, it hits this rule and forces the ASA to trigger the creation of an SA for that traffic.
For more information on what the rules in the "show asp table classify crypto" command mean please refer to Understanding the rules
in the "show asp table classify crypto" output
ASP Drop Data
1. Capture type asp drop - The command used to configure this capture is:
rtpvpnoutbound6# capture <capname> type asp-drop all match ip host x.x.x.x host y.y.y.y
This command will capture the packet matching the rule specified if the the packet is dropped by the accelerated security path and
it will list the drop-type
2. show asp drop frame - This output lists all the packets dropped by the accelerated security path along with the drop-type so you can
match the data from the asp-drop capture
There are quite a few types of drops so I've tried to narrow down the usual suspects when it comes to troubleshooting any VPN issue:
no-route
acl-drop
np-sp-invalid-spi
natt-keepalive
ipsecudp-keepalive
bad-crypto
bad-ipsec-prot
ipsec-ipv6
bad-ipsec-natt
bad-ipsec-udp
ipsec-need-sa
ctm-error
ipsec-spoof
ipsec-clearpkt-notun
ipsec-tun-down
Packet Processing
1. Capture with trace - This command combines two of the most useful tools used in inspecting the flow of a packet through the ASA:
- the packet capture tool
- the packet tracer tool
The command used to configure this capture is:
rtpvpnoutbound6# capture <capname> trace detail interface <interface_name> match ip host x.x.x.x host
2. ISAKMP Captures - Collected on the interface on which the crypto map is applied. The ISAKMP subsystem does not have access
to the upper layer protocols. The capture is a pseudo capture, with the Physical, IP, and UDP layers combined together to satisfy a
PCAP parser. The peer addresses are obtained from the SA exchange and are stored in the IP layer.
rtpvpnoutbound6# capture <capname> type isakmp interface <interface_name>
To extract these captures use the command:

copy /pcap capture:my_cap tftp:///my_cap.pcap
To view the output of this command run:

rtpvpnoutbound6# sh cap capi trace
The output of this capture is particularly useful in cases were the VPN tunnel is up but the traffic isn't getting encrypted.
Troubleshooting Methodology
For the purposes of this document I will be using an example from SR 622830245 and bug CSCub83428 to better explain the process.
Step 1 - confirm tunnel is up
Irrespective of what the complaint is the first step is always to confirm that phase 1 and phase 2 SAs between the two affected peers is
up.
Step 2 - identify which rule is being used for the faulty traffic
Ask the user to run a continuous ping between two hosts that are behind the two peers and capture the incoming clear text packets on
the affected ASA with the trace option enabled as described above. If the packet is matching a configured VPN rule the output of the
capture will provide the user_data value and then you can use this value to track down the crypto map being matched.
Here's what happened in SR 622830245 :
Problem Description: Everytime the customer enables the default dynamic crypto map for ikev2, everything w
The Vpn client was using the following subnet as the client pool:
ip local pool VPN_IP_Pool 192.168.254.5-192.168.254.254 mask 255.255.255.0
I've cleaned my troubleshooting logs and presented it here with annotations to explain the steps described here as well as how I used
the various show commands to get to the result:
the dynamic crypto map and build the tunnel thus blackholing all traffic to the 192.0.0.0/8 subnet which i
Step 3 - identify why it's not hitting the right rule

If the traffic isn't matching any VPN rule or matching the correct VPN rule but still not getting encrypted then it's time to check the ASP
drops to identify why the packet is getting dropped. To do this you can follow either of the two processes:
1. clear the asp drop output by running the command:
rtpvpnoutbound6# clear asp drop
Run 1000 pings between the affected hosts with timeout set to 0 and then check the "show asp drop frame" output to see if any of
the counters went up by 1000. This process may not work all the time especially if there's a lot of traffic going through the ASA.
2. Run and collect the asp-drop captures as defined above. If any packets are captured then compare the drop type with the output
from "show asp drop frame" to identify why the packet is getting dropped. If there are no packets captured then ASP isn't dropping
this packet.
One important thing to keep in mind here is, just because you see the need-ike counter go up doesn't indicate an error. The need-ike
counter only indicates that the packet was dropped because it requires encryption but has no established IPSec security association.
This is generally a normal condition for LAN-to-LAN IPSec configurations. This indication will cause the appliance to begin ISAKMP
negotiations with the destination peer. It only becomes abnormal when the counter goes up but ikev1 or ikev2 debugs don't indicate any
ongoing negoatiations for the crypto domain to which that packet belongs. Bugs
CSCtl86372 and more recently CSCub83428 have been filed for this issue.
Tip: Keep in mind you could also be running into bug CSCtq92619 which is why it's important to check if both phase 1 and
phase 2 are up in step 1.
Outputs to collect
If you are runinng into this issue then please collect the following data and get in touch with Cisco TAC:
Common Data&colon;
-------------
1. show crypto isakmp sa detail
2. show crypto ipsec sa detail
3. show crypto ipsec sa inactive
4. show crypto protocol statistics all
5. show asp table classify crypto
6. show asp table vpn-context detail
7. show asp drop
8. debug crypto ike-common 5
9. debug menu ike-common 1
12. isakmp capture as described above 12. capture with trace detail option on inside for interesting traf
13. Syslogs(and preferable, debugs as well) from the ASA taken at (present time on the ASA seen in "show
For IKEv1:
----------
1. debug crypto ikev1 5
For IKEv2:
----------
1. Turn on exit path debugging using "debug menu ikev2 2 1"
2. Collect the exit path debugs using "debug menu ikev2 13 0"
3. debug menu ikev2 13 0
4. debug menu ikev2 9
Known Problems
Problem 1 Traffic is blackholed and the syslog "Duplicate entry already in Tunnel Manager" is seen
This message means that there already is a IKE negotiation in progress and that a request to build a crypto tunnel has already been
dispatched to IKEv1/IKEv2 crypto engine.
Why do we see this syslog?
This is seen if the tunnel is taking longer than usual to come up. So, if the inside host is sending packets continously to the ASA, the NP
will send one packet to the tunnel manager every 2 seconds(the NP rate limits this traffic so that the tunnel manager does not get
flooded with packets). Thus, the syslog is printed once every 2 seconds and the tunnel manager drops this packet.
Assume a situation where the the response to the packets that this ASA is sending out is not being received. It could be that port 500 is
blocked or that the remote side is completely down or the reply packet is not reaching the local side. After125 seconds, IKEv2 gives
up(32 seconds for IKEv1). In such cases, it is perfectly normal to see this syslog.
BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

Number of DSH in-use: 1
NODE:
VCID: single_vf
Entry Age: 60 secs
Protocol: ip
IP Source: 1.1.1.2
IP Destination: 1.1.1.1
Source Net: 1.1.1.2 - 1.1.1.2, port range: 0 - 0
Destination Net: 1.1.1.1 - 1.1.1.1, port range: 0 - 0
Map Tag: outside_map
Map Sequence Number: 1
IKEv1 Supported: NO
IKEv2 Supported: YES
NODE:
VCID: single_vf
Entry Age: 63 secs
Protocol: ip
IP Source: 1.1.1.2
IKEv1 Supported: NO

NODE:
VCID: single_vf
Entry Age: 67 secs
Protocol: ip
IP Source: 1.1.1.2
IKEv1 Supported: NO
<snip>

NODE:
VCID: single_vf
Entry Age: 122 secs
Protocol: ip
IP Source: 1.1.1.2
IKEv1 Supported: NO

Number of DSH in-use: 0 <<at this point, the tunnel manager deletes the entry since IKEv2 informed tha
BGL-O-15-ASA5500-1(config)# debug menu ike-common 3

Tunnel Manager Counters Summary:
--------------------------------
Dispatched to IKEv2: 3
Failed callbacks from IKEv2: 3
Success callbacks from IKEv2: 0
Outstanding callbacks from IKEv2: 0
Failed callback processing: 0
If IKEv2 successfully establishes a tunnel, it immediately informs the tunnel manager about this and teh tunnel manager deletes the
entry for these proxy IDs.
Solution
In some cases, the crypto tunnel suddenly fails to come up without any changes made to the setup. In such cases, look for a stuck
tunnel manger entry, where the communication between IKEv2 and the tunnel manager may fail. So, even after the tunnel is up, IKEv2
will not infomr the tunnel manager that the tunnel is up and it thus needs to delete the entry. Hence, the entry stays in the tunnel
manager and it refuses to process new requests coming in from the NP for those particular proxy IDs.
Check the output of "debug menu ike-common 1" and you will see a stuck entry in the tunnel manager with a very high "Entry age"
value. For example, the following was seen in a customer setup:
NODE:
VCID: single_vf
Entry Age: 38507 secs
Protocol: ip
IP Source: 10.91.43.115
Map Tag: outside_map0
IKEv1 Supported: NO
This isue is seen due to bug CSCug49382. This bug has been resolved.
As a workaround, either reload the ASA or re-configure the crypto map with a different sequence number. You can also choose to
delete the tunnel manager entry using "deb menu ike-common 10", but please be aware that this will delete all the active tunnel
manager entries as well.
Related Information
Known Caveats
1. CSCud41507 Traffic destined for L2L tunnels can prevent valid L2L from establishing
"This is the newest bug and currently it is being worked with the DEs. Basically what is happening here is that we have an
inside rate limiter and if we have traffic for a bad peer ( a bursty traffic ) then for some SAs we will not even try to negotiate IKE"
. From the repro attached to the bug :
"If you send continuous traffic to the remote side of the unreachable peer, you will not be able to
2. CSCub83428 IPsec L2L tunnel fails to establish - need-ike drops but no IKE exchange
3. CSCtq92619 ASA IPsec doesn't start Quick Mode for some SA with IKEv1 SA up
4. CSCud28106 IKEv2: ASA does not clear entry from asp table classify crypto
5. CSCty91877 Duplicate VPN-context/ASP classifier for crypto
6. CSCuo58411 ASA IKEv2 "Duplicate entry in tunnel manager" (post 9.1.5)
7. CSCuv35483 ASA IKEv2 "Duplicate entry in tunnel manager" (post 9.1.6.6)
8. CSCug49382- L2L tunnel fails with error "Duplicate entry in Tunnel Manager"
9. CSCuj46707- Stale entry in vpn-context doesn't allow SA to be initiated
10. CSCuj29123- ENH: "Duplicate entry in Tunnel Manager" error lacks clarity
11. CSCup37416 Stale VPN Context entries cause ASA to stop encrypting traffic
These symptoms can also be observed with stuck tunnel manager entries. This was originally exposed with:
1. CSCty16864 ASA doesn't start quick mode negotiation - stuck tunnel manager entries
Technical Support & Documentation - Cisco Systems
Everyone's Tags: tz:km:1110287160 tz:km:1110494501 tz:km:1110581338 tz:km:622585211 tz:km:623187221 View All (141)
Add Tag...
40 Kudos
Cisco Internal Information (access controlled / confidential)
Hide Comments
Comments
Options
by anchacko
10-06-2013 02:56 PM - edited 03-27-2014 03:16 AM
Table of Contents
Introduction
Traffic is blackholed and this syslog is seen
Outputs to collect
Caveats
Introduction
This article explains why the syslog/debug "Duplicate entry already in Tunnel Manager" is seen on the ASA.
This message means that there already is a IKE negotiation in progress and that a request to build a crypto tunnel has already been
dispatched to IKEv1/IKEv2 crypto engine.

Once a packet comes to the ASA, the connection table is looked up. If it isn't there, then its punted to the ASP, where its decided if the
packet has to be NATed, encrypted,etc. This is how my crypto maps and crypto ACLs look:
BGL-O-15-ASA5500-1(config)# sh run cry

crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec ikev2 ipsec-proposal AES256
protocol esp encryption aes-256
protocol esp integrity sha-1 md5
crypto ipsec security-association pmtu-aging infinite
crypto map outside_map 1 match address l2l_list
crypto map outside_map 1 set ikev2 ipsec-proposal AES256
crypto map outside_map 478 match address outside_0478
crypto map outside_map 478 set pfs
crypto map outside_map 478 set ikev1 transform-set ESP-3DES-MD5
crypto map outside_map 478 set security-association lifetime seconds 28800
crypto map outside_map 478 set security-association lifetime kilobytes 4608000
crypto map outside_map interface out
crypto ca trustpool policy
encryption aes-256
integrity sha
group 2
prf sha
lifetime seconds 86400
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400
BGL-O-15-ASA5500-1(config)# sh run access-l

access-list outside_0478 extended permit ip host 192.168.1.1 host 192.168.2.1
access-list l2l_list extended permit ip host 1.1.1.2 host 1.1.1.1
once you configure "match address" under the crypto map,the ASA install the encrypt entries into the ASP table with a user_data value
of 0x0, which indicates to the ASA that it needs to bring up a tunnel. Once the tunnel is established, the user_data value will be a non-
zero value. So, in the ASA we see the following ASP table entry before the packet is encrypted( i am sending traffic that matches
sequence number 1):
BGL-O-15-ASA5500-1(config)# sh asp table classify domain encrypt
Input Table
Output Table:
out id=0x7ffd902111d0, priority=70, domain=encrypt, deny=false
out id=0x7ffd901803b0, priority=70, domain=encrypt, deny=false
out id=0x7ffd9097a290, priority=70, domain=encrypt, deny=false
out id=0x7ffd8e5e8d60, priority=70, domain=encrypt, deny=false
The cs_id is the cascade delimiter and it contains a crypto map entry identifier. The very first entry with src and dst IP addresses as
0.0.0.0 is the beginning of a particular crypto map. The cs_id value will be identical for all crypto ACL entries of a particular crypto map.
The following document though outdated, explains the ASP table concepts well:
Crypto tunnel and the ASP Table
Once the entry is in the ASP table, the NP checks the domain and in this case it says "encrypt", so this packet is supposed to be
encrypted and sent over the tunnel. But, the user data is 0x0, which means that the tunnel is not up. This is where this packet is sent to
the tunnel manager. The tunnel manager decides whether this should be an IKEv1 or an IKev2 tunnel. This is how it makes this
decision:
The tunnel manager looks at the configuration and goes through the crypto map configured for that ASP table rule. It records the proxy
IDs(1.1.1.1 and 1.1.1.2), the name of the crypto map(outside_map) and the sequence number of the crypto map(10). Now, if both
ikev1 and ikev2 are enabled for this crypto map, the tunnel manager dispatches this request to IKEv2 first.
Oct 06 08:00:53 [IKE COMMON DEBUG]Tunnel Manager dispatching a KEY_ACQUIRE message to IKEv2. Map Tag = o
Once the tunnel manager has already dispatched this packet to IKEv2, if it gets another packet that has the same proxy IDs, it will print
this syslog, since it has already dispatched a packet to IKEv2. The tunnel manager then discards this packet.

This is seen if the tunnel is taking longer than usual to come up. So, if the inside host is sending packets continously to the ASA, the
NP will send one packet to the tunnel manager every 2 seconds(the NP rate limits this traffic so that the tunnel manager does not get
flooded with packets). Thus, the syslog is printed once every 2 seconds and the tunnel manager drops this packet.

Assume a situation where the the response to the packets that this ASA is sending out is not being received. It could be that port 500
is blocked or that the remote side is completely down or the reply packet is not reaching the local side. After125 seconds, IKEv2 gives
up(32 seconds for IKEv1). In such cases, it is perfectly normal to see this syslog.

NODE:
VCID: single_vf
Entry Age: 60 secs
Protocol: ip
IP Source: 1.1.1.2
IKEv1 Supported: NO

NODE:
VCID: single_vf
Entry Age: 63 secs
Protocol: ip
IP Source: 1.1.1.2
IKEv1 Supported: NO

NODE:
VCID: single_vf
Entry Age: 67 secs
Protocol: ip
IP Source: 1.1.1.2
IKEv1 Supported: NO
<snip>

NODE:
VCID: single_vf
Entry Age: 122 secs
Protocol: ip
IP Source: 1.1.1.2
IKEv1 Supported: NO

Number of DSH in-use: 0 <<at this point, the tunnel manager deletes the entry since IKEv2 informed tha
BGL-O-15-ASA5500-1(config)# debug menu ike-common 3

Tunnel Manager Counters Summary:
--------------------------------
Failed callback processing: 0
If IKEv2 successfully establishes a tunnel, it immediately informs the tunnel manager about this and teh tunnel manager deletes the
entry for these proxy IDs.
Traffic is blackholed and this syslog is seen

In some cases, the crypto tunnel suddenly fails to come up without any changes made to the setup. In such cases, look for a stuck
tunnel manger entry, where the communication between IKEv2 and the tunnel manager may fail. So, even after the tunnel is up, IKEv2
will not infomr the tunnel manager that the tunnel is up and it thus needs to delete the entry. Hence, the entry stays in the tunnel
manager and it refuses to process new requests coming in from the NP for those particular proxy IDs.
Check the output of "debug menu ike-common 1" and you will see a stuck entry in the tunnel manager with a very high "Entry age"
value. For example, the following was seen in a customer setup:
NODE:
VCID: single_vf
Entry Age: 38507 secs
Protocol: ip
IP Source: 10.91.43.115
Map Tag: outside_map0
IKEv1 Supported: NO
This isue is seen due to bug CSCug49382. This bug has been resolved.
Outputs to collect

2. Syslogs(and preferable, debugs as well) from the ASA taken at (present time on the ASA seen in "show c
3. Turn on exit path debugging using "debug menu ikev2 2 1"
4. Collect the exit path debugs using "debug menu ikev2 13 0"
5. debug menu ikev2 13 0
As a workaround, either reload the ASA or re-configure the crypto map with a different sequence number. You can also choose to
delete the tunnel manager entry using "deb menu ike-common 10", but please be aware that this will delete all the active tunnel
manager entries as well.
Caveats
CSCug49382- L2L tunnel fails with error "Duplicate entry in Tunnel Manager"
CSCuj46707- Stale entry in vpn-context doesn't allow SA to be initiated
CSCuj29123- ENH: "Duplicate entry in Tunnel Manager" error lacks clarity
Permalink 10 Kudos
Options
by otipisov
on 12-26-2016 01:30 PM
@atbasu I don't know how to edit this article without breaking formatting and pictures. It would be good to replace last section with this:
Bugs
Older bugs:
CSCsh48962 Duplicate ASP table entry causes FW to encrypt traffic with invalid SPI
CSCsd48512 Duplicate ASP crypto table entry causes firewall to not encrypt traffic
CSCtb53186 Duplicate ASP crypto table entry causes firewall to not encrypt traffic
CSCtb62022 VPN-Context does not replicate via statefull failover sometimes. ASA PIX
CSCtb65794 Identical encryption rules in the table with diffent encryption values
CSCtd36473 IPsec: Outbound context may be deleted prematurely
CSCty16864 ASA doesn't start quick mode negotiation - stuck tunnel manager entries
CSCud41507 Traffic destined for L2L tunnels can prevent valid L2L from establishing
If you send continuous traffic to the remote side of the unreachable peer, you will not be able to bring the tunnel up with traffic destined
for the good peer.
CSCub83428 IPsec L2L tunnel fails to establish - need-ike drops but no IKE exchange
CSCtq92619 ASA IPsec doesn't start Quick Mode for some SA with IKEv1 SA up
CSCud28106 IKEv2: ASA does not clear entry from asp table classify crypto
CSCty91877 Duplicate VPN-context/ASP classifier for crypto
CSCuo58411 ASA IKEv2 "Duplicate entry in tunnel manager" (post 9.1.5)
CSCuv35483 ASA IKEv2 "Duplicate entry in tunnel manager" (post 9.1.6.6)
CSCug49382 L2L tunnel fails with error "Duplicate entry in Tunnel Manager"
CSCuj46707 Stale entry in vpn-context doesn't allow SA to be initiated
Most popular ones:
CSCup37416 Stale VPN Context entries cause ASA to stop encrypting traffic
CSCuz94862 IKEv2: Data rekey collisions can cause inactive IPsec SAs to get stuck
CSCvb29688 Stale VPN Context entries cause ASA to stop encrypting traffic despite fix for CSCup37416
Enhancement Requests
CSCuj29123 ENH: "Duplicate entry in Tunnel Manager" error lacks clarity

CSCuj84243 ENH: Add syslog for drops caused by "vpn-overlap-conflict"
CSCup37434 ENH: Packet tracer should detect stale ASP entry
CSCuu58775 ENH: ASA IPSEC should provide more granular counters
CSCuu58812 ENH: ASA should allow debugs for IPSEC datapath triggered issues
CSCuw10852 ENH: ASA VPN ASP table Hardening
CSCuy80545 ENH: Additional counters for serviceability of ASP table with IKE
CSCuy80557 ENH: "service internal" command to remove hanged ASP Table entries
Don't forget to link your cases to them!
Permalink
Options
by malhyari
on 12-27-2016 02:22 PM
hehehe ,
That is a a good punch of defects.
Moh,
Permalink
Options
by otipisov
on 12-28-2016 03:29 AM
Yeah,
@malhyari
And another good news ;) is that stale ASP rules can still be created even in 9.5.x software. This can happen during ASA
reconfiguration, maybe when static crypto maps are edited or a new static crypto map block is added with overlapping ACL and a
lower sequence number. Exact steps are not known and current customer configuration doesn't have overlapping ACLs (SR
681518524 ). The stale ASP encryption rules below definitely relate to a static crypto map which doesn't exist in the configuration,
because cs_id is non-zero (cs_id=0x2aaad69b8400). So far as I know, dynamic ASP rules have zero cs_id. Clearing all SAs doesn't
help, as well as "clear crypto ipsec sa inactive". They remain in the ASP table. So, ASA reboot is the only option.
access-list rt_gpb extended permit ip host 172.20.17.125 host 10.200.171.151
crypto dynamic-map dyno 500 set ikev1 transform-set my-transform-set-ikev1

crypto map IPSEC 50 match address rt_gpb <-- problematic crypto map block
crypto map IPSEC 50 set pfs group5
crypto map IPSEC 50 set peer 195.225.38.72
crypto map IPSEC 50 set ikev1 transform-set 3des-SHA
crypto map IPSEC 60 match address uni-rt
crypto map IPSEC 60 set pfs
crypto map IPSEC 60 set ikev1 transform-set AES-256-SHA
crypto map IPSEC 60 set security-association lifetime seconds 86400
crypto map IPSEC 60 set reverse-route
crypto map IPSEC 70 match address rt_uni
crypto map IPSEC 70 set ikev1 transform-set 3des-SHA
crypto map IPSEC 70 set security-association lifetime seconds 3600
crypto map IPSEC 70 set reverse-route
crypto map IPSEC 500 ipsec-isakmp dynamic dyno
crypto map IPSEC interface uplink_1
---------
vpn# sh ver
Cisco Adaptive Security Appliance Software Version 9.5(2)
Compiled on Sat 28-Nov-15 00:16 PST by builders

System image file is "disk0:/asa952-lfbff-k8.SPA"
Config file at boot was "startup-config"
vpn up 27 days 13 hours

failover cluster up 35 days 5 hours
...
//
// No SAs initially
//
vpn# sh cry isa sa
There are no IKEv1 SAs
vpn# sh cry ipsec sa
There are no ipsec sas
//
// Send traffic to bring L2L tunnel up
//
vpn# sh cry isa sa
IKEv1 SAs:
Active SA: 2
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 2
1 IKE Peer: 46.188.36.26

Type : user Role : responder
Rekey : no State : MM_ACTIVE
2 IKE Peer: 195.225.38.72
Type : L2L Role : initiator
Rekey : no State : MM_ACTIVE
vpn# sh capture
capture cap1 type raw-data trace detail interface prod2 [Capturing - 590 bytes]
match icmp host 172.19.21.83 host 192.168.15.123
vpn# sh capture cap1
8 packets captured
1: 17:21:04.852449 802.1Q vlan#449 P0 172.19.21.83 > 192.168.15.123: icmp: echo request

8 packets shown
vpn# sh capture cap1 packet-number 1 trace detail
27 packets captured
1: 17:21:04.852449 5254.009e.1028 00c1.6485.2dd5 0x8100 Length: 102
802.1Q vlan#449 P0 172.19.21.83 > 192.168.15.123: icmp: echo request (DF) (ttl 64, id 0)
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaad95fd7c0, priority=13, domain=capture, deny=false
hits=5, user_data=0x2aaad4fb6670, cs_id=0x0, l3_type=0x0
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0000.0000.0000
input_ifc=prod2, output_ifc=any
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
in id=0x2aaad61a81e0, priority=1, domain=permit, deny=false
hits=941564, user_data=0x0, cs_id=0x0, l3_type=0x8
src mac=0000.0000.0000, mask=0000.0000.0000
dst mac=0000.0000.0000, mask=0100.0000.0000
Phase: 3
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
found next-hop 217.107.111.169 using egress ifc uplink_1
Phase: 4
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,any) source static TUCS TUCS destination static test test
NAT divert to egress interface uplink_1
Untranslate 192.168.15.123/0 to 192.168.15.123/0
Phase: 5
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group GLOBAL_ACCESS global
access-list GLOBAL_ACCESS extended permit ip host 172.19.21.83 host 192.168.15.123
in id=0x2aaad4fb85a0, priority=12, domain=permit, deny=false
hits=4315, user_data=0x2aaacd974040, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=172.19.21.83, mask=255.255.255.255, port=0, tag=any
dst ip/id=192.168.15.123, mask=255.255.255.255, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any
Phase: 6
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (any,any) source static TUCS TUCS destination static test test
Static translate 172.19.21.83/23135 to 172.19.21.83/23135
in id=0x2aaad53b5050, priority=6, domain=nat, deny=false
hits=2576, user_data=0x2aaad4fbbe30, cs_id=0x0, flags=0x0, protocol=0
Phase: 7
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
in id=0x2aaad483fc00, priority=0, domain=nat-per-session, deny=true
hits=5633351, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
Phase: 8
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
in id=0x2aaad61af0b0, priority=0, domain=inspect-ip-options, deny=true
Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
in id=0x2aaad615f720, priority=70, domain=inspect-icmp, deny=false
hits=5743, user_data=0x2aaad6fe26a0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
Phase: 10
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
in id=0x2aaad61ae8c0, priority=66, domain=inspect-icmp-error, deny=false
hits=5743, user_data=0x2aaad55b5170, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=any, dscp=0x0
Phase: 11
Type: VPN
Subtype: encrypt
Result: DROP <-- DROP
Config:
out id=0x2aaad4a430d0, priority=70, domain=encrypt, deny=false
hits=2577, user_data=0x0, cs_id=0x2aaad69b8400, reverse, flags=0x0, protocol=0 <-- no VPN context
input_ifc=any, output_ifc=uplink_1
Result:
input-interface: prod2
input-status: up
input-line-status: up
output-interface: uplink_1
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule
1 packet shown
vpn# sh asp table class crypto
Input Table
in id=0x2aaad58a6460, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=0, user_data=0x0, cs_id=0x2aaad69b8400, reverse, flags=0x0, protocol=0
input_ifc=uplink_1, output_ifc=any
in id=0x2aaad55bf280, priority=70, domain=ipsec-tunnel-flow, deny=false
in id=0x2aaad7c0aeb0, priority=70, domain=ipsec-tunnel-flow, deny=false
in id=0x2aaad6e767f0, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=0, user_data=0x0, cs_id=0x2aaad7c0a4a0, reverse, flags=0x0, protocol=0
in id=0x2aaad4a8afe0, priority=70, domain=ipsec-tunnel-flow, deny=false
in id=0x2aaad5443930, priority=70, domain=ipsec-tunnel-flow, deny=false
in id=0x2aaad69c4ee0, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=0, user_data=0x89fffc, cs_id=0x2aaad7c0a4a0, reverse, flags=0x0, protocol=0
in id=0x2aaad58add30, priority=70, domain=ipsec-tunnel-flow, deny=false
in id=0x2aaad4feb240, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=0, user_data=0x0, cs_id=0x2aaad69b59b0, reverse, flags=0x0, protocol=0
in id=0x2aaad6f61560, priority=70, domain=ipsec-tunnel-flow, deny=false
in id=0x2aaad69a9400, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=0, user_data=0x0, cs_id=0x2aaad4fc31e0, reverse, flags=0x0, protocol=0
in id=0x2aaad491ea80, priority=70, domain=ipsec-tunnel-flow, deny=false
in id=0x2aaad66a35f0, priority=70, domain=decrypt, deny=false
hits=1, user_data=0x910d4, cs_id=0x0, reverse, flags=0x0, protocol=17
in id=0x2aaad6f4c120, priority=70, domain=decrypt, deny=false
hits=1, user_data=0x89fffc, cs_id=0x0, reverse, flags=0x0, protocol=17
in id=0x2aaad5f4a210, priority=69, domain=ipsec-tunnel-flow, deny=false
hits=108, user_data=0x910d4, cs_id=0x0, reverse, flags=0x0, protocol=17
in id=0x2aaad6f47cb0, priority=13, domain=decrypt, deny=true
in id=0x2aaad69ca0f0, priority=13, domain=ipsec-tunnel-flow, deny=true
in id=0x2aaad69d7c70, priority=13, domain=ipsec-natt, deny=false
in id=0x2aaad6f67ba0, priority=13, domain=decrypt, deny=true
in id=0x2aaad69b5250, priority=70, domain=ipsec-tunnel-flow, deny=false
src ip/id=::/128, port=0, tag=any
dst ip/id=::/128, port=0, tag=any
in id=0x2aaad69bcf00, priority=70, domain=ipsec-tunnel-flow, deny=false
in id=0x2aaad6f66890, priority=70, domain=ipsec-tunnel-flow, deny=false
Output Table:
out id=0x2aaad6e77260, priority=70, domain=encrypt, deny=false <-- stale entry
out id=0x2aaad55bf390, priority=70, domain=encrypt, deny=false <-- stale entry
out id=0x2aaad4a430d0, priority=70, domain=encrypt, deny=false <-- we hit this stale entry
hits=2835, user_data=0x0, cs_id=0x2aaad69b8400, reverse, flags=0x0, protocol=0 <-- without VP
out id=0x2aaad69c02a0, priority=70, domain=encrypt, deny=false <-- start of 'crypto map IPSEC 50'
out id=0x2aaad4a4b8b0, priority=70, domain=encrypt, deny=false
out id=0x2aaad44c6ba0, priority=70, domain=encrypt, deny=false
out id=0x2aaad55bdf70, priority=70, domain=encrypt, deny=false <-- we should have hit this one
hits=0, user_data=0x92974, cs_id=0x2aaad7c0a4a0, reverse, flags=0x0, protocol=0 <-- with a VPN c
out id=0x2aaad69bcbf0, priority=70, domain=encrypt, deny=false
out id=0x2aaad693dc30, priority=70, domain=encrypt, deny=false <-- start of 'crypto map IPSEC 60
out id=0x2aaad505a7f0, priority=70, domain=encrypt, deny=false
out id=0x2aaad4fc6540, priority=70, domain=encrypt, deny=false
out id=0x2aaad4fc2b30, priority=70, domain=encrypt, deny=false
out id=0x2aaad6a90c10, priority=70, domain=encrypt, deny=false
hits=108, user_data=0x8ec1c, cs_id=0x0, reverse, flags=0x0, protocol=17
out id=0x2aaad69dc870, priority=70, domain=encrypt, deny=false
out id=0x2aaad4c73360, priority=70, domain=encrypt, deny=false
out id=0x2aaad69bec30, priority=70, domain=encrypt, deny=false
L2 - Output Table:
L2 - Input Table:
vpn# sh asp table vpn-context det
VPN CTX = 0x0089FFFC
Peer IP = 192.168.15.123
Pointer = 0xD586E630
State = UP
Flags = DECR+ESP+NATT
SA = 0x022685CF
SPI = 0xFDC876CE
Group = 0
Pkts = 0
Bad Pkts = 0
Bad SPI = 0
Spoof = 0
Bad Crypto = 0
Rekey Pkt = 0
Rekey Call = 0
VPN Filter = <none>
VPN CTX = 0x00092974 <-- correct VPN context
Peer IP = 192.168.15.123
Pointer = 0xD5F50360
State = UP
Flags = ENCR+ESP+NATT
SA = 0x00286E85
SPI = 0x2C7F7F7B <-- correct SPI
Group = 0
Pkts = 0
Bad Pkts = 0
Bad SPI = 0
Spoof = 0
Bad Crypto = 0
Rekey Pkt = 0
Rekey Call = 0
VPN Filter = <none>
VPN CTX = 0x000910D4
Peer IP = 46.188.36.26
Pointer = 0xD6894E40
State = UP
Flags = DECR+ESP+NATT
SA = 0x002730EF
SPI = 0x6C8D0BEA
Group = 1
Pkts = 30957
Bad Pkts = 0
Bad SPI = 0
Spoof = 0
Bad Crypto = 0
Rekey Pkt = 0
Rekey Call = 0
VPN Filter = <none>
VPN CTX = 0x0008EC1C
Peer IP = 46.188.36.26
Pointer = 0xD69AB8D0
State = UP
Flags = ENCR+ESP+NATT
SA = 0x0027D7FD
SPI = 0x07F33818
Group = 1
Pkts = 21955
Bad Pkts = 0
Bad SPI = 0
Spoof = 0
Bad Crypto = 0
Rekey Pkt = 0
Rekey Call = 0
VPN Filter = <none>
vpn# sh cry ipsec sa

interface: uplink_1
Crypto map tag: dyno, seq num: 500, local addr: 217.107.111.170
local ident (addr/mask/prot/port): (217.107.111.170/255.255.255.255/17/1701)

remote ident (addr/mask/prot/port): (46.188.36.26/255.255.255.255/17/0)
current_peer: 46.188.36.26, username: stanislaff
dynamic allocated peer ip: 10.253.253.10
dynamic allocated peer ip(ipv6): 0.0.0.0
#pkts encaps: 30056, #pkts encrypt: 30056, #pkts digest: 30056

#pkts decaps: 43299, #pkts decrypt: 43299, #pkts verify: 43299
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 30056, #pkts comp failed: 0, #pkts decomp failed: 0
#post-frag successes: 0, #post-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0
local crypto endpt.: 217.107.111.170/4500, remote crypto endpt.: 46.188.36.26/4500

path mtu 1500, ipsec overhead 46(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: 07F33818
current inbound spi : 6C8D0BEA
inbound esp sas:

spi: 0x6C8D0BEA (1821182954)
transform: esp-3des esp-sha-hmac no compression
in use settings ={RA, Transport, NAT-T-Encaps, IKEv1, }
slot: 0, conn_id: 139264, crypto-map: dyno
sa timing: remaining key lifetime (kB/sec): (4348692/2624)
IV size: 8 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0x07F33818 (133380120)
in use settings ={RA, Transport, NAT-T-Encaps, IKEv1, }
slot: 0, conn_id: 139264, crypto-map: dyno
IV size: 8 bytes
Anti replay bitmap:
0x00000000 0x00000001
Crypto map tag: IPSEC, seq num: 50, local addr: 217.107.111.170

local ident (addr/mask/prot/port): (172.19.21.83/255.255.255.255/0/0)
remote ident (addr/mask/prot/port): (192.168.15.123/255.255.255.255/0/0)
current_peer: 195.225.38.72
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0
local crypto endpt.: 217.107.111.170/4500, remote crypto endpt.: 195.225.38.72/4500

path mtu 1500, ipsec overhead 66(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: 2C7F7F7B
current inbound spi : FDC876CE
inbound esp sas:

spi: 0xFDC876CE (4257773262)
in use settings ={L2L, Tunnel, NAT-T-Encaps, PFS Group 5, IKEv1, }
slot: 0, conn_id: 143360, crypto-map: IPSEC
IV size: 8 bytes
Anti replay bitmap:
0x00000000 0x00000001
outbound esp sas:
spi: 0x2C7F7F7B (746553211)
in use settings ={L2L, Tunnel, NAT-T-Encaps, PFS Group 5, IKEv1, }
slot: 0, conn_id: 143360, crypto-map: IPSEC
IV size: 8 bytes
Anti replay bitmap:
0x00000000 0x00000001
vpn# debug crypto ike-comm 3

vpn# Dec 26 17:34:34 [IKE COMMON DEBUG]Tunnel Manager dispatching a KEY_ACQUIRE message to IKEv1. Map Ta
Dec 26 17:34:34 [IKE COMMON DEBUG]IKEv1 was unsuccessful at setting up a tunnel. Map Tag = IPSEC. Map S
Dec 26 17:34:34 [IKE COMMON DEBUG]Tunnel Manager has failed to establish an L2L SA. All configured IKE v
Dec 26 17:34:34 [IKE COMMON DEBUG]Tunnel Manager Removed entry. Map Tag = IPSEC. Map Sequence Number =
Dec 26 17:34:36 [IKE COMMON DEBUG]Tunnel Manager dispatching a KEY_ACQUIRE message to IKEv1. Map Tag = I
vpn# no deb all
vpn# debug crypto ipsec 3

vpn# IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=1, saddr=172.19.21.83, spor
IPSEC(crypto_map_check)-3: Checking crypto map IPSEC 50: matched.
IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=1, saddr=172.19.21.83, sport=252
no deb all
vpn# clear asp drop
vpn# sh asp drop
Frame drop:
Reverse-path verify failed (rpf-violated) 4
Flow is denied by configured rule (acl-drop) 7
IKE new SA limit exceeded (ike-sa-rate-limit) 2
Last clearing: 17:17:09 MSK/MSD Dec 26 2016 by stanislaff
Flow drop:
Need to start IKE negotiation (need-ike) 6

vpn# sh asp drop
Frame drop:
Reverse-path verify failed (rpf-violated) 9
Flow is denied by configured rule (acl-drop) 17
IKE new SA limit exceeded (ike-sa-rate-limit) 4
Flow drop:
Need to start IKE negotiation (need-ike) 16
vpn# show crypto ikev1 stats
Global IKEv1 Statistics

Active Tunnels: 0
Previous Tunnels: 32
In Octets: 201328
In Packets: 1854
In Drop Packets: 49
In Notifys: 1661
In P2 Exchanges: 25
In P2 Exchange Invalids: 0
In P2 Exchange Rejects: 11
In P2 Sa Delete Requests: 2
Out Octets: 186976
Out Packets: 1911
Out Drop Packets: 0
Out Notifys: 3315
Out P2 Exchanges: 10
Out P2 Exchange Invalids: 0
Out P2 Exchange Rejects: 0
Out P2 Sa Delete Requests: 22
Initiator Tunnels: 19
Initiator Fails: 11
Responder Fails: 8
System Capacity Fails: 0
Auth Fails: 0
Decrypt Fails: 0
Hash Valid Fails: 0
No Sa Fails: 0
IKEV1 Call Admission Statistics

Max In-Negotiation SAs: 50
In-Negotiation SAs: 0
In-Negotiation SAs Highwater: 2
In-Negotiation SAs Rejected: 0
vpn# show crypto ipsec stats
IPsec Global Statistics

-----------------------
Active tunnels: 0
Previous tunnels: 310
Inbound
Bytes: 744712936
Decompressed bytes: 744712936
Packets: 3733234
Dropped packets: 9
Replay failures: 1
Authentications: 3733234
Authentication failures: 0
Decryptions: 3733234
Decryption failures: 0
TFC Packets: 0
Decapsulated fragments needing reassembly: 0
Valid ICMP Errors rcvd: 0
Invalid ICMP Errors rcvd: 0
Outbound
Bytes: 1828620558
Uncompressed bytes: 1828620558
Packets: 3453553
Dropped packets: 2
Authentications: 3453553
Authentication failures: 0
Encryptions: 3453553
Encryption failures: 0
TFC Packets: 0
Fragmentation successes: 0
Pre-fragmentation successses: 0
Post-fragmentation successes: 0
Fragmentation failures: 0
Pre-fragmentation failures: 0
Post-fragmentation failures: 0
Fragments created: 0
PMTUs sent: 0
PMTUs rcvd: 8
Protocol failures: 0
Missing SA failures: 11
System capacity failures: 0
vpn# show crypto protocol statistics ikev1

[IKEv1 statistics]
Encrypt packet requests: 2003
Encapsulate packet requests: 2003
Decrypt packet requests: 1945
Decapsulate packet requests: 1945
HMAC calculation requests: 3023
SA creation requests: 36
SA rekey requests: 3
SA deletion requests: 529
Next phase key allocation requests: 162
Random number generation requests: 0
Failed requests: 0
vpn# show crypto protocol statistics ipsec

[IPsec statistics]
Encrypt packet requests: 580748
Encapsulate packet requests: 580748
Decrypt packet requests: 582113
Decapsulate packet requests: 582113
HMAC calculation requests: 1162861
SA creation requests: 156
SA rekey requests: 6
SA deletion requests: 165
Next phase key allocation requests: 0
Random number generation requests: 0
Failed requests: 0
vpn# debug menu ike-common 4

Tunnel Manager Counters:
------------------------
Severe TM Errors
----------------
Failed to removed record: 0
Invalid record error: 0
Keyacquire callback errors: 0
Internal Severe Errors

----------------------
Alloc_failure: 0
DSH alloc failure: 0
IPC chunk alloc failure: 0
IPC enqueue failure: 0
Unknown timer pops: 0
Block ptr invalid errors: 0
IKEv2 dispatch failed: 0
IKEv1 dispatch failed: 0
Unknown Msg Received: 0
Unknown Event Received: 0
BP in record invalid error: 0
Config errors
----------------------
Map check failed: 0
No protocol configured: 0
Multi-peer not supported errors: 0
Backup L2L not supported errors: 0
No proposal configured: 0
No transform configured: 0
Warnings
----------------------
Duplicate initiate attempts: 172
Initiate via IKEv2 failed: 0
Initiate via IKEv1 failed: 3426
Tunnel initiate failed: 3426
Invalid record during callback(dup cb): 0
Status
----------------------
Initiates dispatched to IKEv2: 0
Initiates dispatched to IKEv1: 3435
Initiate fallback dispatched to IKEv2: 0
Initiate fallback dispatched to IKEv1: 0
Initiate via IKEv2 succeeded: 0
Initiate via IKEv1 succeeded: 9
NP indications enqueued: 3607
Permalink
Post a Comment
Rich Text HTML Preview Quote
        OVP    Photos  Video    Paragraph Arial Size
                    

Email me when someone replies
Cancel Post Your Comment

Troubleshooting Traffic Flow Issues Through A VPN Tunnel On The ASA - Cisco

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Troubleshooting Traffic Flow Issues Through A VPN Tunnel On The ASA - Cisco

Uploaded by

Copyright:

Available Formats

3/23/2020 Troubleshooting traffic flow issues through a VPN tunnel on the ASA - cisco

akashtha My Settings Messages (1) Help Sign Out

Started 10-27-2012 by Modified 04-29-2016 by Edit Article Options

Enter BDB App Name Link

This document describes how to troubleshoot one/more of the following symptoms:

This document is not restricted to specific software and hardware versions.

Output Table: josemed

L2 - Input Table: ballowe

Last clearing of hits counters: Never

ciscoasa(config)# cry isa en outside

Output Table: 623855883 Same Problem

L2 - Output Table: 624019633

L2 - Input Table: 624067375

Last clearing of hits counters: Never 624321433

L2 - Input Table: Top Tags

Last clearing of hits counters: Never

BGL-O-15-ASA5500-1(config)# sh run cry

BGL-O-15-ASA5500-1(config)# sh run access-l

BGL-O-15-ASA5500-1(config)# sh asp table classify domain encrypt

out id=0x7fffa3856080, priority=70, domain=encrypt, deny=false

ASP Drop Data

To extract these captures use the command:

To view the output of this command run:

Step 1 - confirm tunnel is up

Here's what happened in SR 622830245 :

Step 3 - identify why it's not hitting the right rule

What does this syslog mean?

Why do we see this syslog?

When is it normal to see this syslog?

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# debug menu ike-common 3

Failed callback processing: 0

Number of DSH in-use: 1

Technical Support & Documentation - Cisco Systems

Cisco Internal Information (access controlled / confidential)

What does this syslog mean?

Basic things to know about the ASP table entries

BGL-O-15-ASA5500-1(config)# sh run cry

BGL-O-15-ASA5500-1(config)# sh run access-l

BGL-O-15-ASA5500-1(config)# sh asp table classify domain encrypt

Why do we see this syslog?

When is it normal to see this syslog?

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# deb menu ike-common 1

BGL-O-15-ASA5500-1(config)# debug menu ike-common 3

Failed callback processing: 0

Traffic is blackholed and this syslog is seen

Number of DSH in-use: 1

1. debug menu ike-common 1

Most popular ones:

CSCuj29123 ENH: "Duplicate entry in Tunnel Manager" error lacks clarity

Don't forget to link your cases to them!

That is a a good punch of defects.

crypto dynamic-map dyno 500 set ikev1 transform-set my-transform-set-ikev1

Cisco Adaptive Security Appliance Software Version 9.5(2)

Compiled on Sat 28-Nov-15 00:16 PST by builders