You are on page 1of 15

JNCIE-SEC Traceoptions & IPSEC

troubleshooting
In IPSEC topic, I am continuing with traceoptions and troubleshooting section. In this post, I will
try to explain how I troubleshoot IPSEC VPNs mostly initial setup.
IPsec VPNs

Implementation of IPsec VPNs

Multipoint tunnels

Policy and route-based VPNs

Traceoptions

Dual and backup tunnels

On-demand tunnels

DRP over a tunnel

Dynamic VPNs

Certificate-based VPNs

PKI

Interoperability with 3rd party devices

NAT

Implementation of NAT

Source NAT

Destination NAT

I enabled IKE traceoptions and simulated several type of possible problems and observed the error logs.2 and 212.2 IKE & IPSEC SUCCESSFUL LOG Phase 1 [Aug 22 20:40:14]ike_calc_mac: Star [Aug 22 20:40:14]ike_st_i_cert: [Aug 22 20:40:14]ike_st_i_priva 1 2 3 4 [Aug 22 20:40:14]ike_calc_mac: Start. initiator = true. PS: All errors below are between ike peers 192. local = false [Aug 22 20:40:14]ike_st_i_cert: Start [Aug 22 20:40:14]ike_st_i_private: Start [Aug 22 20:40:14]ike_st_o_wait_done: Marking for waiting for done . Because of this.179. Static NAT  Implementation of NAT with IPSec  Overlapping IPs between sites One of the challenging parts of JNCIE-SEC must be the troubleshooting part for which I need to understand under what sort of problems what type of error logs are generated. But first let’s see how a successful IKE Phase 1 and IKE Phase 2 log looks like.168.64.45.

nego = 0 22 [Aug 22 20:40:14]ike_init_qm_negotiation: Start.498aaa01 19 01d0dd21} 20 [Aug 22 20:40:14]ssh_ike_connect_ipsec: SA = { e4d65d2e a7bf1c17 . Initiator.2 IKEv1 12 with status: Error ok 13 [Aug 22 20:40:14]Added (spi=0xaebf2827.168.64. Found SA = { e4d65d2e a7bf1c17 17 .2.0. prf = hmac-s ha1 [Aug 22 20:40:14]192. remote:212.64. auth_method = Pre shared keys. cipher = 3des-cbc. nego = -1 10 [Aug 22 20:40:14]iked_pm_ike_sa_done: local:192.168.2:500 { e4d65d2e a7bf1c17 5 498aaa01 01d0dd21 [-1] / 0x00000000 } IP.498aaa01 01d0dd21} 18 [Aug 22 20:40:14]ike_alloc_negotiation: Start.45.168.168. protocol=0) entry to the spi table 14 [Aug 22 20:40:14]Added (spi=0x3037b766. remote_name = :500.498aaa01 21 01d0dd21}. hash = sha1.64. flags = 00000000 16 [Aug 22 20:40:14]ike_sa_find_ip_port: Remote = all:500. protocol=0) entry to the spi table 15 [Aug 22 20:40:14]ssh_ike_connect_ipsec: Start. remote:212.179.179.179.2. message_id = 5aa9f0f2 23 [Aug 22 20:40:14]ike_st_o_qm_hash_1: Start 24 [Aug 22 20:40:14]ike_st_o_qm_sa_proposals: Start 25 [Aug 22 20:40:14]ike_st_o_qm_nonce: Start 26 [Aug 22 20:40:14]ike_policy_reply_qm_nonce_data_len: Start 27 [Aug 22 20:40:14]ike_st_o_qm_optional_ke: Start 28 [Aug 22 20:40:14]ike_st_o_qm_optional_ids: Start [Aug 22 20:40:14]ike_st_qm_optional_id: Start [Aug 22 20:40:14]ike_st_qm_optional_id: Start [Aug 22 20:40:14]ike_st_o_private: Start [Aug 22 20:40:14]Construction NHTB payload for local:192. remote:212. prf = hmac7 sha 8 [Aug 22 20:40:14]ike_send_notify: Connected. Phase 2 [Aug 22 20:40:14]ike_policy_reply_p [Aug 22 20:40:14]ike_st_o_encr [Aug 22 20:40:14]<none>:500 (I 1 [Aug 22 20:40:14]ike_policy_reply_private_payload_out: Start .64. SA = { e4d65d2e a7bf1c17 .2 IKEv1 11 [Aug 22 20:40:14]IKE negotiation done for local:192.45. initiator = 1.45. version = 1.[Aug 22 20:40:14]ike_st_o_all_done: MESSAGE: Phase 1 { 0xe4d65d2e a7bf1c17 0x498aaa01 01d0dd21 } / 00000000.2. SA = { e4d65d2e a7bf1c17 . hash = sha1.0. xchg = Identity protect.179. MESSAGE: Phase 1 version = 1.498aaa01 9 01d0dd21}.2:500 (Initiator) <-> 212.2 IKEv1 P1 SA index 8160872 sa-cfg vpn-hub You can see the “IKE negotiation done” log in here. auth_method 6 = Pre shared keys. cipher = 3des-cbc.45.

2) entry to the 11 peer hash table 12 [Aug 22 20:40:14]Added (spi=0xdfa0760c. No PFS. key rounds = 0 8 [Aug 22 20:40:14]iked_pm_ipsec_sa_install: local:192.179.45.2:500 { e4d65d2e a7bf1c17 3 498aaa01 01d0dd21 [0] / 0x5aa9f0f2 } QM.45.2) entry to the peer 13 hash table 14 [Aug 22 20:40:14]Hardlife timer started for inbound vpn-hub with 3600 seconds/0 kilobytes 15 [Aug 22 20:40:14]Softlife timer started for inbound vpn-hub with 2978 seconds/0 kilobytes 16 [Aug 22 20:40:14]In iked_ipsec_sa_pair_add Adding GENCFG msg with key. tunnel.64.[Aug 22 20:40:14]ike_st_o_encrypt: Marking encryption for packet [Aug 22 20:40:14]<none>:500 (Initiator) <-> 212. life = 0 4 kB/3600 sec.168.168. No PFS. tunnel. nego = 0 You can see “QM.45.45. hmac-sha1-96. Extended seq not used.45.2 . key len = 0.64. SA = { 0xe4d65d2e a7bf1c17 .498aaa01 01d0dd21 } / 5aa9f0f2. MESSAGE: Phase 2 connection succeeded” message. MESSAGE: SA[0][0] = ESP 3des. protocol=ESP dst=212.SPI-In = 0xaebf2827 [Aug 22 20:40:14]Added dependency on SA config blob with tunnelid = 131073 [Aug 22 20:40:14]Successfully added ipsec SA PAIR [Aug 22 20:40:14]ike_st_o_qm_wait_done: Marking for waiting for done [Aug 22 20:40:14]ike_encode_packet: Start.45. group = 0 [Aug 22 20:40:14]ike_qm_call_callback: MESSAGE: Phase 2 connection succeeded.64. root@J23-London> show secu Index State Initiator cookie Re 8160872 UP e4d65d2ea7bf1c 1 2 3 4 5 6 7 8 root@J23-London> show security ike sa Index State Initiator cookie Responder cookie Mode Remote Address 8160872 UP e4d65d2ea7bf1c17 498aaa0101d0dd21 Main 212. key le 5 n= 6 [Aug 22 20:40:14]ike_qm_call_callback: MESSAGE: SA[0][0] = ESP 3des. hmac-sha1-96. MESSAGE: Phase 2 connection succeeded. life = 0 kB/3600 7 sec.2:500 { e4d65d2e a7bf1c17 498aaa01 01d0dd21 [0] / 0x5aa9f0f2 } QM.64. group = 0 2 [Aug 22 20:40:14]<none>:500 (Initiator) <-> 212. group = 0.179. For this specific connection here is the CLI outputs.64.root 500 212.2 root@J23-London> show security ipsec sa Total active tunnels: 1 ID Algorithm SPI Life:sec/kb Mon lsys Port Gateway <131073 ESP:3des/sha1 aebf2827 3469/ unlim . Extended seq not used.2. 9 remote:212.2 IKEv1 for SA-CFG vpn-hub 10 [Aug 22 20:40:14]Added (spi=0xaebf2827. Tunnel = 17 131073. group = 0. protocol=ESP dst=192.64.

2 ERROR 1: “IKEv1 Error : Invalid payload type” If your pre-shared keys aren’t matching you will get a similar error log like below.64.45.64. Offending message id = 0x00000000 14 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212. Offending payload type = 145 7 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.64.1dd10036 25 f9452830}.45.64.45. nego = 0 23 [Aug 22 20:49:08]ike_free_negotiation: Start. SA = { 287c0904 ed0108cf .2:500 { 287c0904 ed0108cf 10 1dd10036 f9452830 [0] / 0x28e022de } Info. nego = 0 22 [Aug 22 20:49:08]ike_free_negotiation_info: Start.root 500 212.45.45.2:500 { 287c0904 ed0108cf 4 1dd10036 f9452830 [0] / 0x28e022de } Info. nego = -1 26 [Aug 22 20:49:08]192.2:500 (Initiator) <-> 212. SA = { 287c0904 ed0108cf . delete it 17 [Aug 22 20:49:08]ike_st_i_private: Start 18 [Aug 22 20:49:08]ike_send_notify: Connected.2:500 { 287c0904 ed0108cf 1dd10036 f9452830 [-1] / 0x00000000 } IP. nego = -1 [Aug 22 20:49:08]ssh_ike_tunnel_table_entry_delete: Deleting tunnel_id: 0 from IKE tunnel table [Aug 22 20:49:08]ssh_ike_tunnel_table_entry_delete: The tunnel id: 0 doesn't exist in IKE tunnel table [Aug 22 20:49:08]ike_sa_delete: Start.1dd10036 f9452830 } [Aug 22 20:49:08]ike_free_negotiation_isakmp: Start.2:500 { 287c0904 ed0108cf 8 1dd10036 f9452830 [0] / 0x28e022de } Info.64.1dd10036 Notification data has attribute lis 1 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.1dd10036 21 f9452830}. Notification data has attribute list 3 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212. nego = -1 .64.2:500 { 287c0904 ed0108cf 15 1dd10036 f9452830 [0] / 0x28e022de } Info. Error text = Incorrect pre-shared key (Invalid 11 next payload value) 12 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.1dd10036 f9452830}. [Aug 22 20:49:08]<none>:500 (Resp 287c0904 ed0108cf .168. nego = 0 20 [Aug 22 20:49:08]ike_delete_negotiation: Start.179.45. Notify message version = 1 5 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212. nego = 0 24 [Aug 22 20:49:08]ike_remove_callback: Start. SA = { 287c0904 ed0108cf .45.1dd10036 19 f9452830}. Connection got error = 1.64.45.2:500 { 287c0904 ed0108cf 6 1dd10036 f9452830 [0] / 0x28e022de } Info. Offending payload data offset = 0 9 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212. SA = { 287c0904 ed0108cf .45.2:500 { 287c0904 ed0108cf 2 1dd10036 f9452830 [0] / 0x28e022de } Info.64.64.9 >131073 ESP:3des/sha1 dfa0760c 3469/ unlim .2:500 { 287c0904 ed0108cf 13 1dd10036 f9452830 [0] / 0x28e022de } Info. calling callback [Aug 22 20:49:08]ike_delete_negotiation: Start. Received notify err = Invalid payload type (1) 16 to isakmp sa. delete SA = { 287c0904 ed0108cf .

nego = -1 [Aug 22 20:49:08]ike_free_id_payload: Start. remote:212.  if st0..45. d proposal chosen (14).16 800c0001 00060022 . id type = 1 [Aug 22 20:49:08]ike_free_sa: Start [Aug 22 20:49:08]IKE negotiation fail for local:192..64.0 interface isn’t created with family inet and/or assigned to a security zone [Aug 22 20:53:10]ike_st_i_n: Start..45..46] = 800c0001 00060022 .64.16] = f39ace76 bde7864a .2. you will get the same error under the following conditions  if you forget to set “bind-interface st0.168.0” under your vpn configuration. doi = 1..168. 2 spi[0. code = No proposal chosen (14). Notify message version = 1 7 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212. remote:212..  dh-group  authentication algorithm  encryption algorithm WARNING!!!: In addition to these mismatches.179.64. data[0..2:500 { f39ace76 bde7864a 10 0d31547b 7e819258 [0] / 0xf638af05 } Info.2..64.2:500 { f39ace76 bde7864a 8 0d31547b 7e819258 [0] / 0xf638af05 } Info. Error text = Could not find acceptable proposal 9 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212.2:500 { f39ace76 bde7864a 6 0d31547b 7e819258 [0] / 0xf638af05 } Info..2:500 { f39ace76 bde7864a - .2:500 { f39ace76 bde7864a 4 0d31547b 7e819258 [0] / 0xf638af05 } Info. You can also see “Error text = Incorrect pe-shared-key” Error 2: “IKEv1 Error : No proposal chosen” You will get the following error if one of the followings mismatches in your IKE config.179.64.45.64. protocol = 1.. spi[0.45.45.2 IKEv1 with status Invalid syntax Error message “IKEv1 Error: Invalid payload type” is a likely indication of a pre-shared key mismatch. Notification data has attribute list 5 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212.64.[Aug 22 20:49:08]ike_free_negotiation: Start.45.2 IKEv1 with status: Invalid syntax [Aug 22 20:49:08] IKEv1 Error : Invalid payload type [Aug 22 20:49:08]IPSec Rekey for SPI 0x0 failed [Aug 22 20:49:08]IPSec SA done callback called for sa-cfg vpn-hub local:192.45. 1 [Aug 22 20:53:10]ike_st_i_n: Start. Offending message id = 0x00000000 11 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212. 3 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212.

remote = 212.2:500.2.2 IKEv1 with status: No proposal chosen [Aug 22 20:53:10] IKEv1 Error : No proposal chosen [Aug 22 20:53:10]IPSec Rekey for SPI 0x0 failed [Aug 22 20:53:10]IPSec SA done callback called for sa-cfg vpn-hub local:192. ISSU 4 pending=no 5 [Aug 22 20:59:24]iked_pm_ike_spd_notify_request: Sending Initial contact 6 [Aug 22 20:59:24]ssh_ike_connect: Start.00000000 00000000 } 9 [Aug 22 20:59:24]ike_init_isakmp_sa: Start.0d31547b 15 7e819258}.0d31547b 7e819258}. Connection got error = 14.64. nego = 0 [Aug 22 20:53:10]ike_delete_negotiation: Start. remote_name = 212. xchg = 2. [Aug 22 20:59:08]KMD_INTERNAL_E received [Aug 22 20:59:13]iked_spu_ha_ 1 [Aug 22 20:59:08]KMD_INTERNAL_ERROR: iked_ifstate_eoc_handler: EOC msg 2 received 3 [Aug 22 20:59:13]iked_spu_ha_ipc_get_server_addr.45.168.2 IKEv1 with status No proposal chosen ERROR 3: “IKEv1 Error : Timeout” If IKE port 500 isn’t reachable at the peer. remote:212.179.45. delete SA = { f39ace76 bde7864a . server tnp addr (standalone): 0x1.45. nego 11 = -1 . This means peer A can’t be the initiator but only responder. Another problem you might encounter is that for example. you forget to enable IKE service in a zone only in one peer (e. Received notify err = No proposal chosen (14) to isakmp sa.168.0d31547b 7e819258}.0d31547b 7e819258 [0] / 0xf638af05 } Info.00000000 00000000}. flags = 7 00090000 8 [Aug 22 20:59:24]ike_sa_allocate: Start.2. nego = 0 14 [Aug 22 20:53:10]ike_remove_callback: Start.2:500.64.179.64.168.45.45. calling callback 18 [Aug 22 20:53:10]ikev2_fb_v1_encr_id_to_v2_id: Unknown IKE encryption identifier -1 19 [Aug 22 20:53:10]ikev2_fb_v1_hash_id_to_v2_prf_id: Unknown IKE hash alg identifier -1 20 [Aug 22 20:53:10]ikev2_fb_v1_hash_id_to_v2_integ_id: Unknown IKE hash alg identifier -1 [Aug 22 20:53:10]IKE negotiation fail for local:192. SA = { f39ace76 bde7864a .64. remote:212.g Peer B) but Peer A is still allowing IKE. nego = -1 16 [Aug 22 20:53:10]192. Because A can’t connect to IKE port but B can. initiator = 1 10 [Aug 22 20:59:24]ssh_ike_connect: SA = { e6ed730d 487d645f . delete it [Aug 22 20:53:10]ike_st_i_private: Start [Aug 22 20:53:10]ike_send_notify: Connected. SA = { f39ace76 bde7864a .2:500 { f39ace76 bde7864a 17 0d31547b 7e819258 [-1] / 0x00000000 } IP. nego = 0 13 [Aug 22 20:53:10]ike_free_negotiation: Start.64.2:500 (Initiator) <-> 212. SA = { e6ed730d 487d645f .179. you will get an error like this. nego = 0 12 [Aug 22 20:53:10]ike_free_negotiation_info: Start.

22 [Aug 22 20:59:54]iked_pm_ike_sa_delete_done_cb: For p1 sa index 2299946. nego = -1 18 [Aug 22 20:59:44]ike_send_packet: Start. retransmit SA = { e6ed730d 487d645f 13 00000000 00000000}.[Aug 22 20:59:24]ike_st_o_sa_proposal: Start [Aug 22 20:59:24]ike_policy_reply_isakmp_vendor_ids: Start [Aug 22 20:59:24]ike_st_o_private: Start [Aug 22 20:59:24]ike_policy_reply_private_payload_out: Start [Aug 22 20:59:24]ike_encode_packet: Start. nego = -1.45.00000000 00000000 } / 00000000.45.00000000 00000000}. Connection timed out or error.64. .2:500 routing table id = 0 20 [Aug 22 20:59:54]P1 SA 2299946 timer expiry. flags 0x0. dst = 212. delete SA = { e6ed730d 487d645f . ref cnt 2.64. nego = -1. Thus if one of the following two mismatches. send SA = { e6ed730d 487d645f .e if peer A has 3600secs and peer B has 7200secs.179.2:500 routing table id = 0 16 [Aug 22 20:59:44]ike_retransmit_callback: Start. 23 status: Error ok 24 [Aug 22 20:59:54]ike_remove_callback: Start. remote:212. you will get this error. retransmit previous packet SA = { e6ed730d 19 487d645f .00000000 00000000}.45. they agree on 3600secs. retransmit SA = { e6ed730d 487d645f 17 00000000 00000000}. nego = -1 26 [Aug 22 20:59:54]192.179. dst = 212.168.00000000 25 00000000}. nego = -1.2. routing table id = 0 12 [Aug 22 20:59:34]ike_retransmit_callback: Start.64.2:500 { e6ed730d 487d645f 27 00000000 00000000 [-1] / 0x00000000 } IP. calling callback 28 [Aug 22 20:59:54]ikev2_fb_v1_encr_id_to_v2_id: Unknown IKE encryption identifier -1 [Aug 22 20:59:54]ikev2_fb_v1_hash_id_to_v2_prf_id: Unknown IKE hash alg identifier -1 [Aug 22 20:59:54]ikev2_fb_v1_hash_id_to_v2_integ_id: Unknown IKE hash alg identifier -1 [Aug 22 20:59:54]iked_pm_ike_sa_done: UNUSABLE p1_sa 2299946 [Aug 22 20:59:54] IKEv1 Error : Timeout [Aug 22 20:59:54]IPSec Rekey for SPI 0x0 failed [Aug 22 20:59:54]IPSec SA done callback called for sa-cfg vpn-hub local:192.00000000 00000000}.45. ref cnt 2. nego = -1 14 [Aug 22 20:59:34]ike_send_packet: Start.2:500.2 IKEv1 with status Timed out ERROR 4: “IKEv1 Error : No proposal chosen” This is the same error like ERROR 2 but it is actually caused by IPSEC proposals not IKE.168. SA = { 0xe6ed730d 487d645f .45.64.  Authentication algorithm  Encryption algorithm Note: I had thought that ipsec lifetime is also something that has to match but my tests showed a different result.2:500 (Initiator) <-> 212. nego = -1 [Aug 22 20:59:24]ike_send_packet: Start. timer reason Force delete timer 21 expired (1).64. As far as I can see peers agree on the lowest lifetime configured i. retransmit previous packet SA = { e6ed730d 15 487d645f . dst = 212.

45. flag basic-datapath.64.63. source-prefix 212.63.2/32.log size 5m.63.1 count 1 size 1000 2 PING 212.45.1): 1000 data bytes 3 1008 bytes from 212.45 PING 212.63.45.1: icmp_seq=0 ttl=64 time=46.55.55.2/32.63.63. } packet-filter incoming-filter { protocol icmp. source-prefix 192.1/32. source-prefix 55.45. destination-prefix 212.63.55.63. } packet-filter outgoing-esp { protocol esp.1 (212. I enable traceoptions for the traffic that I am going to generate.1/32.168.1 (212.63. destination-prefix 55.45.204 ms . destination-prefix 212.log size 5m.55.45.179.45. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 [edit] root@J23-London# show security flow traceoptions { file ipsec-traf.1: ic 1 root@J23-London> ping 212. Now I will do some flow troubleshooting.55.55.45. packet-filter outgoing-filter { protocol icmp.1/32.1) 1008 bytes from 212. [edit] root@J23-London# show secu traceoptions { file ipsec-traf.45.1/32.FLOW Troubleshooting So far I have done IKE troubleshooting.45.1 source 55. } } First two packet filters will show us clear text packets but outgoing-esp is for the encrypted packets Let’s send 1 ICMP packet with 1000 bytes (1008 bytes with ICMP header) root@J23-London> ping 212.

in ifp .574883:CID-0:RT:<55. sp 0.45..0.45.55.63.1.63.45.. da 212. dp 2 29 3813.55.45. out ifp N/A sp 28 0.55.63.204/0. call 27 flow_route_lookup(): src_ip 55.55. rtbl_idx = 0 6 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Over-riding lpak->vsys with 0 13 Aug 22 20:01:06 20:01:06.local.0 12 Aug 22 20:01:06 20:01:06.1 nsp2 0. 1 packets received.125 >212. dp 23813 23 Aug 22 20:01:06 20:01:06.55.574883:CID-0:RT:(flow_first_create_session) usp_tagged set 20 session as mng session 21 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: in_ifp <junos-host:.0->212.local.0 as incoming nat if.1. sa 55.204/46. start first path.4 5 --.574883:CID-0:RT: routed (x_dst_ip 212.55.63.flow_process_pkt: (thd 1): flow_ctxt type 5 0. 24 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_rule_dst_xlate: packet 55.55..45.574883:CID-0:RT:flow_first_routing: vr_id 0.0..574883:CID-0:RT:<55..1->212.. mbuf 0x4d10d480. ip_proto 1.0 -> 11 to:.1. tos 0 30 Aug 22 20:01:06 20:01:06. dp 23813.45.1.0:55.1.5 matched filter outgoing-filter: 1 Aug 22 20:01:06 20:01:06. @0x4d10d6c1 4 Aug 22 20:01:06 20:01:06. from_cp_flag .574883:CID-0:RT: find flow: table 0x552e9c90.574883:CID-0:RT:Using vr id from pfe_tag with value= 0 10 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: no session found.574883:CID-0:RT: flow_first_in_dst_nat: in <.1.574883:CID-0:RT: flow_first_create_session 19 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_process_pkt_exception: setting rtt in lpak 8 to 0x5d22ca70 9 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:---.574883:CID-0:RT: chose interface . sp 0.574883:CID-0:RT:Doing DESTINATION addr route-lookup 31 Aug 22 20:01:06 20:01:06. 0% packet loss 7 round-trip min/avg/max/stddev = 46.0>. (8/0) 14 Aug 22 20:01:06 20:01:06. tok 2 16 Aug 22 20:01:06 20:01:06.63.local. hash 15 28352(0xffff).45.63.55.63.local.local.63.45. icmp.55.574883:CID-0:RT:Changing lpak->in_ifp from:. proto 1.log outgoing-filter match Aug 22 20:01:06 20:01:06.1.63.45.local.55.574883:CID-0:RT:packet [1028] ipid = 3812.. common flag 0x0.1> matched 2 filter outgoing-filter: 3 Aug 22 20:01:06 20:01:06.204/46.1/23813.1/0->212. in_tunnel 17 0x0.local.1) from junos-host .0> 7 Aug 22 20:01:06 20:01:06.000 ms Now examine the file ipsec-traf.1 ping statistics --6 1 packets transmitted. 26 Aug 22 20:01:06 20:01:06. out 22 <N/A> dst_adr 212.574883:CID-0:RT: .212.0.0 18 Aug 22 20:01:06 20:01:06. x_dst_ip 212.

574883:CID-0:RT:-jsf int check: plugin id 22. svc_req 0x0.1 33 Aug 22 20:01:06 20:01:06.55. impl mask 0x0.0x5d05. svc_req 0x0. rc 4 66 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: dip id = 0/0.574883:CID-0:RT:is_loop_pak: No loop: on ifp: st0.574883:CID-0:RT:flow_first_get_out_ifp: tunnel out 0x577cf0ec.63.0.1/0->55. curr ageout 60s 40 Aug 22 20:01:06 20:01:06.. impl 65 mask 0x0.574883:CID-0:RT:-jsf : Alloc sess plugin info for session 8 58 Aug 22 20:01:06 20:01:06. impl 63 mask 0x0.55.574883:CID-0:RT: permitted by policy self-traffic-policy(1) 41 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:[JSF]Normal interest check. 59 enabled impl mask 0x0 60 Aug 22 20:01:06 20:01:06.1/38837 38 proto 1 39 Aug 22 20:01:06 20:01:06. impl 72 mask 0x0.574883:CID-0:RT:-jsf int check: plugin id 15. impl 69 mask 0x0.45.574883:CID-0:RT:-jsf int check: plugin id 2. 46 Aug 22 20:01:06 20:01:06. count 0) enabled for session .0 50 Aug 22 20:01:06 20:01:06.45.574883:CID-0:RT:flow_first_policy_search: policy search from 34 zone junos-host-> zone vpn (0x0. svc_req 0x0. rc 2 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf int check: plugin id 16.0x5d05) 35 Aug 22 20:01:06 20:01:06. tun id 131073 52 Aug 22 20:01:06 20:01:06. svc_req 0x0.local. svc_req 0x0.55.1. rtt_idx:0 57 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Policy lkup: vsys 0 zone(2:junos-host) -> 36 zone(7:vpn) scope:0 37 Aug 22 20:01:06 20:01:06.0 in 0) to st0. timeout 60s. rc 4 73 Aug 22 20:01:06 20:01:06. svc_req 0x0.574883:CID-0:RT:flow_first_src_xlate: nat_src_xlated: False.63.63.574883:CID-0:RT: Found tunnel for if (non-vpn or vpn without 49 nhtb) st0. 43 nat_src_xlate_failed: False 44 Aug 22 20:01:06 20:01:06. impl mask 0x0.574883:CID-0:RT:[JSF]Plugins(0x0. 53 tun id 131073 54 Aug 22 20:01:06 20:01:06.55. rc 4 70 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_get_tun_info: tunnel out 51 0x577cf0ec. Next-hop: 212. 67 plugin_id 0 68 Aug 22 20:01:06 20:01:06.0 as outgoing phy if 55 Aug 22 20:01:06 20:01:06. Permitted by policy. regd plugins 12. pst_nat: False.574883:CID-0:RT:flow_first_src_xlate: src nat returns status: 0.574883:CID-0:RT:-jsf int check: plugin id 3. svc_req 0x0. 42 Aug 22 20:01:06 20:01:06.0.1/2048 -> 212. addr: 56 212.574883:CID-0:RT: choose interface ge-0/0/0.1/0 47 protocol 0 48 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: 55. 55.45.574883:CID-0:RT:-jsf int check: plugin id 12. rc 4 64 Aug 22 20:01:06 20:01:06. 45 rule/pool id: 0/0.55. rc 4 62 Aug 22 20:01:06 20:01:06. impl 61 mask 0x0.574883:CID-0:RT: packet passed.574883:CID-0:RT:-jsf int check: plugin id 27.574883:CID-0:RT: Error : parameter wrong natp 0x577cfcc8.32 (. rc 4 Aug 22 20:01:06 20:01:06.55.574883:CID-0:RT: app 0.574883:CID-0:RT:+++++++++++jsf_test_plugin_data_evh: 3 71 Aug 22 20:01:06 20:01:06.

574883:CID-0:RT: make_nsp_ready_no_resolve() Aug 22 20:01:06 20:01:06. nsp2 0x577cfd48 Aug 22 20:01:06 20:01:06.55.574883:CID-0:RT: flow_encrypt: tun 0x577cf0ec.local. pak_ptr: 0xbf97d578. post_nat cnt 0 svc req(0x0) Aug 22 20:01:06 20:01:06.574883:CID-0:RT:ttl vector.574883:CID-0:RT: going into tunnel 131073 (nsp_tunnel=0x577cf0ec).flow_process_pkt rc 0x0 (fp rc 0) What happens here is ..574883:CID-0:RT: flow session id 8 Aug 22 20:01:06 20:01:06. type 1 Aug 22 20:01:06 20:01:06.0 orig-zone 2 out-zone 2 vsd 0 Aug 22 20:01:06 20:01:06.local.0> Aug 22 20:01:06 20:01:06.local. Aug 22 20:01:06 20:01:06.574883:CID-0:RT:pre-frag not needed: ipsize: 1028.574883:CID-0:RT: route lookup: dest-ip 55. Aug 22 20:01:06 20:01:06. in_tunnel: 0x0 Aug 22 20:01:06 20:01:06. Aug 22 20:01:06 20:01:06.574883:CID-0:RT: service lookup identified service 0. out_tunnel = 0x577cf0ec Aug 22 20:01:06 20:01:06.574883:CID-0:RT: existing vector list 0x204-0x50a92108.1 orig ifp .574883:CID-0:RT:avt_get_config_by_lsys_id: Not supported on low memory platforms. 0(0). Aug 22 20:01:06 20:01:06.574883:CID-0:RT: encap vector Aug 22 20:01:06 20:01:06.574883:CID-0:RT: vector bits 0x204 vector 0x50a92108 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: Session (id:8) created for first pak 204 Aug 22 20:01:06 20:01:06.0>.574883:CID-0:RT:flow_process_pkt_exception: Freeing lpak 0xbf97d578 associated with mbuf 0x4d10d480 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:no need update ha Aug 22 20:01:06 20:01:06.574883:CID-0:RT: nsp 0x577cfcc8. exit nh 0x30010 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:mbuf 0x4d10d480. free sess plugin info Aug 22 20:01:06 20:01:06.574883:CID-0:RT: route to 55.574883:CID-0:RT: flow_first_install_session======> 0x577cfcc8 Aug 22 20:01:06 20:01:06. out <ge-0/0/0.55..1 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_complete_session.574883:CID-0:RT:construct v4 vector for nsp2 Aug 22 20:01:06 20:01:06.55.55.574883:CID-0:RT: flow got session.574883:CID-0:RT: ----.0 output_ifp . Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Installing c2s NP session wing Aug 22 20:01:06 20:01:06. nsp: 0x577cfcc8. Aug 22 20:01:06 20:01:06.= 34359738368. impli mask(0x0). nsp2->pmtu: 1438 Aug 22 20:01:06 20:01:06.  packet with size 1028 (extra 20 byte from IP header) with identification number 3812 matches: packet [1028] ipid = 3812  A new session is created “flow_first_create_session” . mtu: 1438.574883:CID-0:RT: flow_first_final_check: in <.574883:CID-0:RT:flow_first_service_lookup(): natp(0x577cfcc8): app_id..574883:CID-0:RT:-jsf : no plugin interested for session 8.

574883:CID-0:RT: post_encap: nsp_tunnel 0x577cf0ec. mbuf 0x4d10d480.574883:CID-0:RT:<192 matched filter outgoing-esp: Aug 22 20:01:06 20:01:06.168.579966:CID-0:RT:<212. @0x4d10d69d 3 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:mbuf 0x4d10d480.45.574883:CID-0:RT:<192. rtbl_idx = 0 6 Aug 22 21:02:44 21:02:44. outgoing-esp filter caught this traffic.579966:CID-0:RT:---.50> matched 1 filter outgoing-esp: 2 Aug 22 20:01:06 20:01:06. exit nh 0xa0010 Aug 22 20:01:06 20:01:06. is_valid 1 6 Aug 22 20:01:06 20:01:06. tun id 131073″  Physical outgoing interface is chosen: “choose interface ge-0/0/0.e total length of the new IP packet with ESP header and encryption) and with outside ip id 405 incoming encrypted traffic I have seen that though I haven’t configured returned esp traffic filter. common flag 0x0.0 as outgoing phy if”  and finally packet is encrypted “flow_encrypt: tun 0x577cf0ec. self-traffic-policy allows the traffic as it is locally generated  Tunnel id is identified for the traffic “flow_first_get_tun_info: tunnel out 0x577cf0ec.579966:CID-0:RT: flow process pak fast ifl 70 in_ifp ge-0/0/0.flow_process_pkt rc 0x0 (fp rc 0) In this filter we can see that: Packet is in the tunnel and grew in size to 1080 bytes (i.574883:CID-0:RT: ----. type 1″ outgoing-esp filter match Aug 22 20:01:06 20:01:06.179. mbuf 0x49fa8180.50> matched 2 filter outgoing-esp: 3 Aug 22 21:02:44 21:02:44.579966:CID-0:RT:<212 matched filter outgoing-esp: 1 Aug 22 21:02:44 21:02:44.2/0->192.flow_process_pkt: (thd 1): flow_ctxt type 2.179.168.2/0.579966:CID-0:RT:packet [1080] ipid = 4.0 .45. 4 common flag 0x0.574883:CID-0:RT:packet [1080] ipid = 405. @0x49fa83ce 4 Aug 22 21:02:44 21:02:44. Aug 22 21:02:44 21:02:44.2/0->212. rtbl_idx = 0 5 Aug 22 20:01:06 20:01:06.64.flow_process_pkt: (thd 1): flow_ctxt type 5 15.64.574883:CID-0:RT:---.2/0.

0 interface : ge-0/0/0.168.2->192. Policy name: N/A.179.64.179. Status: Normal Flag: 0x10000 Policy name: N/A Source NAT pool: Null Maximum timeout: N/A. If: ge-0/0/0.45.179.2.esp. Timeout: N/A.64. FIN state: 0. 50  So called source port: 58759 and destination port: 3018  It hits the flow session with id 1 which is a one direction ESP session root@J23-London> show secu Session ID: 1.0. Current timeout: N/A Session State: Valid Start time: 14. tok 8 8 Aug 22 21:02:44 21:02:44.179. Bytes: 0 .45.64.168. Flag: 0x100621 Route: 0xa0010.64.2/3018.45. Pkts: 0.2/3018.64. Interface: ge-0/0/0. Bytes: 0 Total sessions: 2 root@J23-London> show security flow session session-identifier 1 Session ID: 1.2/58759 --> 192. 50 Aug 22 21:02:44 21:02:44.0.168.Aug 22 21:02:44 21:02:44.45.2/58759 --> 192.64.168.2->192. proto 50.45.0:212. da 192. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 root@J23-London> show security flow session protocol esp Session ID: 1. Pkts: 0.2.579966:CID-0:RT: flow_decrypt: tun 0x577cf0ec(flag 0x82).2/58759 --> 192. iif 70  Return traffic enters from ge-0/0/0.1. Tunnel: 0 Port sequence: 0.64. Timeout: N/A.45.579966:CID-0:RT: ge-0/0/0.179. dp 3018.esp.45. In: 212. Pkts: 0.579966:CID-0:RT: flow session id 1 Aug 22 21:02:44 21:02:44.2. Session token: 0x8.0. Valid In: 212.168.0:212.579966:CID-0:RT: flow got session 9 Aug 22 21:02:44 21:02:44. sa 212.579966:CID-0:RT: find flow: table 0x552e9c90.168. Policy name: N/A. Gateway: 192. Duration: 7792 In: 212. Bytes: 0 Session ID: 2. sp 58759. hash 7 44389(0xffff).esp. Policy name: N/A. Valid In: 212.2/0. FIN sequence: 0.179.2.179. If: ge-0/0/0.2/0 --> 192.168.

I will do some troubleshooting if required. If you have any other error you have received which isn’t covered here. DISCLAIMER: Views expressed in this blog are my own and do not necessarily reflect those of Juniper Networks .I hope to have covered various scenarios in this post related to traceoptions and troubleshooting of IPSEC VPN sessions. please do share. In the future posts.