Contents :

Cisco Router Troubleshooting

1. SOP To Find Interface Status..........................................................................................................2

2. SOP To Troubleshoot Packet Drops & High Latency Problems.......................................................3
3. SOP To Find out BGP neighborship & state....................................................................................4
4. SOP To Troubleshoot Finacle Application Not Working From Branch Side....................................7
5. SOP To Switch over WAN link from BSNL MPLS (Primary Link) to Tata MPLS (Secondary link) in
case of Reliability issue / Packet Drops Issue or Flapping Issue in BSNL Link................................8
6. SOP To Switch over WAN link from BSNL MPLS (Primary Link) to BSNL VSAT (Secondary Link) in
case of Primary Link Failure...........................................................................................................9
7. SOP To Switch over WAN link from BSNL VSAT (Active Link) to BSNL (Primary Link) when BSNL
MPLS Link UP................................................................................................................................12
Cisco Router Troubleshooting
SOP To Find Interface Status

Router#sh ip int brief

Interface IP-Address OK? Method Status Protocol

FastEthernet0/0 172.18.36.1 YES NVRAM up up
FastEthernet0/1 10.72.27.154 YES NVRAM up up
Serial0/0 192.16.1.10 YES NVRAM administratively down down
Serial0/1 unassigned YES NVRAM administratively down down
BRI0/1/0 unassigned YES NVRAM administratively down down
Loopback0 172.24.255.36 YES NVRAM up up
Tunnel1 172.24.241.7 YES NVRAM up up

If any IP Assign used interface administratively down , then make it up with `no shut’ command

Router#conf t
Router(config)#
Router(config)#int s0/0
Router(config-if)#no shut
Router(config-if)#Ctrl Z

Router#sh ip int brief

Interface IP-Address OK? Method Status Protocol

FastEthernet0/0 172.18.36.1 YES NVRAM up up
FastEthernet0/1 10.72.27.154 YES NVRAM up up
Serial0/0 192.16.1.10 YES NVRAM up up
Serial0/1 unassigned YES NVRAM administratively down down
BRI0/1/0 unassigned YES NVRAM administratively down down
Loopback0 172.24.255.36 YES NVRAM up up
Tunnel1 172.24.241.7 YES NVRAM up up

Now Serial interface status showing UP if link is properly terminated & working ok.

If still serial interface status shows down then there will be link or modem issue.
SOP To Troubleshoot Packet Drops & High Latency Problems
Step -1 :Check for interface reliability.Ideally reliability should be 255/255 means link is ok.

Router#sh int fa0/1

FastEthernet0/1 is up, line protocol is up

Hardware is Gt96k FE, address is b8be.bf7c.4e19 (bia b8be.bf7c.4e19)
Description: *** WAN ***
Internet address is 10.72.27.154/30
MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload /255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Mb/s, 100BaseTX/FX

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 output buffer failures, 0 output buffers swapped outTo copy the current system image file from
the router to the TFTP network server, use below command in privileged EXEC mode.

In case of link issue it may shows 225/255 or 200/255 etc.

Step -2 : If link reliability is ok , still packet drops occurs then check for CRC error.

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

There should be 0 count for CRC errors.

If any CRC errors showing , then clear it & recheck again by below command.

Router#clear counters

Router#sh int fa0/1

Step -3 : If CRC error not showing , still packet drops occurs then check for Tx/Rx load.

It may depends upon data flow between link. Ideally it should not be overloaded eg. txload250/255,
rxload235/255. Means bandwidth of link is more utilized.

Step -4 :If it is overloaded , then check for which system utilized more bandwidth by sh ip accounting.

Router#sh ip accounting

Source Destination Packets Bytes

172.18.36.11 172.18.45.66 19348 11001742
172.18.36.72 172.18.45.64 160509 120302912
172.18.36.108172.18.45.54 365 91784

As per above o/p system having IP add. 172.18.36.72 utilized more bandwidth.
Router#clear ip accounting (To clear ip accounting)
SOP To Find out BGP neighborship & state.
Router#show ip bgp

BGP table version is 4, local router ID is 172.17.1.99

Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal, r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 10.1.1.0/24 192.168.1.2 0 0 65536 i
*> 10.2.2.0/24 192.168.3.2 0 0 65550 i
*> 172.17.1.0/24 0.0.0.0 0 0 32768 i

In show ipbgp output, Networks are listed in numerical order, lower to higher. An (*) asterisk indicates it
is valid route with proper next-hop. You can see many options in front of route, Below are the meaning
of options:

 “s” for Suppressed : BGP knows about this network but this network is not advertised. Since it is
part of summarized route.
 “d” for dampened: BGP can stop advertising a network that flaps too often until it is stable for a
period of time.
 “h” for history: BGP knows about this network but this network but does not currently have a
valid route.
 "r” for rib failure: The route was advertised to BGP but it was not installed in IP routing table,
Since it is learned by some other routing protocol with better administrative distance.
 “S” for Stale: Used with nonstop forwarding to indicate that the route is stale and needs to be
refreshed when the peer is re-established.

This symbol *> indicates that this route is valid with proper next hop and same time it is accessible.
Next-hop indicates which router is advertising this route and to reach this network you have to reach
this next hop router. 0.0.0.0 Indicates this route is originated locally.
Metric indicates MED attribute, if MED value was received with the route, you can see the med in metric
column.
Local Pref indicates Local preference which is associated with routes, It is locally significance.
Weight, It’s a BGP attribute.
Path: It will show you AS path associated with route. Reading this path left to right.
Origin of route – i – means ibgp and locally originated , e- external bgp and externally received, ?- means
incomplete origin that means the routes are originated by redistribution.

Router#show ip bgp summary

This command allows us to see if the router has been able to set up the Ibgp and Ebgp session with the
other peers.

a) The “BGP table version” which increases and tracks the changes of the router which has been
advertised tothe peers.This table should always increment and never be static. It is important to
compare the table version of the router with the table version of each neighbor.As you can deduce, if
the table version of the neighbor is lower than the main table version, it means that the neighbor is not
yet fully updated. The BGP timer advertisement value is by default 30 seconds.

b) The up/down time session will indicate if you are a victim of flap issues (transmission line or IP) with
yourneighbor.Generally, by ruling an ISP, the session should not flap, and the up/down should not be
reset often. If the up/down is resetting every day or week, it can deteriorate your quality of service given
to your customers. Some upstream/transit providers also process to a damping case if flaps occurred too
many times from a customer.

c) The number of prefixes that you receive from your neighbor.

d) How long the neighbor has been in the current state and the name of the current state.

e) The “InQ” shows how many messages have been received but not yet processed. A high InQ number
indicates an insufficiency of CPU resources to process the input.“OutQ” shows how many outgoing
messages are queued. A high OutQ number indicates a lack of bandwidth to transmit on the outgoing
messages or CPU overload from the other router.

f) The amount of memory that is being used for the BGP data structures. This information is very
important to check and compare with the “show memory command”, in order to see if the router has
enough memory or if the memory is not fragmented.

Router#show ip bgp summary

BGP router identifier 131.108.255.13, local AS number 1
BGP table version is 12, main routing table version 11
6 network entries and 10 paths using 854 bytes of memory
3 BGP path attribute entries using 280 bytes of memory
BGP activity 50/44 prefixes, 73/63 paths
Neighbor V AS MsgRcvdMsgSentTblVerInQOutQ Up/Down State/PfxRcd
131.108.1.2 4 1 194 195 11 0 0 00:03:22 2
131.108.255.6 4 1 84 83 11 0 0 00:03:23 3
131.108.255.14 4 1 152 152 11 0 0 00:03:23 3
141.199.1.1 4 1001 0 0 0 0 0 never Idle

It shows the local router identifier 131.108.255.13, the local AS of 1, and the BGP table version
of 11. (An increasing version number indicates a network change is occurring; if no changes
occur, this number remains the same.) It also shows six network paths on R1, using 854 bytes of
memory.

Memory is important in BGP because in a large network, such as the Internet, memory can be a
limiting factor. As more BGP entries populate the IP routing table, more memory is required.
Example displays four configured remote peers: the first three are IBGP (because the AS is 1 and
the same as the local AS) and one remote peer that has never been active. (The output indicates
an idle session, and the up/down time displays this connection was never established.)

The BGP table displays information, such as remote and local network entries, BGP attributes,
and selected paths. Entries are then inserted into the IP routing table.
The BGP table version increases by 1 for every new BGP best path, not for every received BGP
update/advertisement.  The BGP table version ONLY increments when the is a change to a BGP best
path.

We could see that the table version of the router is slightly different to the one of the BGP neighbors.
This is a correct result; BGP neighbor updates are correctly incremented to the router (BGP and main
routing table).

The InQ and OutQ columns are empty, and additionally, all memory outputs, show that the router is
perfectly capable of handling the multiple BGP sessions.

*** To see what routes you're sending to a neighbor, use the command

Router# show ip bgp neighbors <ip-address of neighbor> advertised-routes

BGP table version is 27, local router ID is 172.16.232.181

Status codes: s suppressed, d damped, h history, * valid, > best, i–
internalOrigin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i10.0.0.0 172.16.232.179 0 100 0 ?
*> 10.20.2.0 10.0.0.0 0 32768i

*** To see what routes you're getting from a neighbor, use the command

Router#show ip bgp neighbors <ip-address of neighbor> routes

*** To clear a BGP session:

Router#clear ip bgp <ip-address of neighbor>

SOP To Troubleshoot Finacle Application Not Working From Branch
Side.
Step1 : Ask for branch code to branch person. According to branch code findout LAN IP address of
that branch & Telnet it.

Eg. Branch Code  7320 , LAN IP Address of that Branch  172.17.10.1

User Name:cmsfmnw
Password:
AHD-White-HouseJunagadh-7320>en
Password:
AHD-White-HouseJunagadh-7320#sh ip int brie
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 172.17.10.1 YES NVRAM up up
Serial0/0 unassigned YES NVRAM down down
Serial0/1 10.201.7.25 YES NVRAM up up
BRI1/0 unassigned YES NVRAM up up
Dialer100 172.17.122.30 YES NVRAM up up
Dialer200 172.16.132.26 YES NVRAM up up

Step2 : Ping Finacle IP as per RO using source IP as a LAN IP as per below

AHD-White-HouseJunagadh-7320# ping 172.16.3.37 so 172.17.10.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.3.37, timeout is 2 seconds:
UUUUU
Success rate is 0 percent (0/5)

Step3 : If it is pinging OK , means Fincle application working fine.

If it is not pinging then do as per below.

AHD-White-HouseJunagadh-7320#clear crypto isakmp

AHD-White-HouseJunagadh-7320#clear crypto sa

Step4 : Again Ping Finacle IP using source IP as a LAN IP as per below

AHD-White-HouseJunagadh-7320#ping 172.16.3.37 so 172.17.10.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.3.37, timeout is 2 seconds:
Packet sent with a source address of 172.17.10.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/70/77 ms

Means Fincle application working fine.

SOP To Switch over WAN link from BSNL MPLS (Primary Link) to
Tata MPLS (Secondary link) in case of Reliability issue / Packet Drops
Issue or Flapping Issue in BSNL Link.
router#sh ip bgp sum
BGP router identifier 172.30.17.113, local AS number 65003
BGP table version is 4229525, main routing table version 4229525

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.201.22.46 4 9829 9754 385 4229525 0 0 05:45:26 5200 Established
10.210.141.113 4 4755 12841 6769 4229525 0 0 2d03h 3765 Active

router#conf t

router(config)#router bgp 65003

router (config-router)#neighbor 10.201.22.46 shutdown

(To ignore neighbor ship with BSNL peer 10.201.22.46)

router#sh ip bgp sum

BGP router identifier 172.30.17.113, local AS number 65003
BGP table version is 4229525, main routing table version 4229525

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.201.22.46 4 9829 9754 385 4229525 0 0 05:45:26 Active
10.210.141.113 4 4755 12841 6769 4229525 0 0 2d03h 3765

Now packet will pass through Tata MPLS Link (via 10.210.141.113)

Remember again when BSNL Link will stable , we have to switch over from Tata to BSNL link with below
commands

router#conf t
router(config)#router bgp 65003
router (config-router)#no neighbor 10.201.22.46 shutdown
(To remove shutdown neighbor ship with BSNL peer 10.201.22.46)

router#sh ip bgp sum

BGP router identifier 172.30.17.113, local AS number 65003
BGP table version is 4229525, main routing table version 4229525

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.201.22.46 4 9829 9754 385 4229525 0 0 00:5:26 5200
10.210.141.113 4 4755 12841 6769 4229525 0 0 2d03h 3765

Now packet will pass through BSNL link (via 10.201.35.18)

SOP To Switch over WAN link from BSNL MPLS (Primary Link) to
BSNL VSAT (Secondary Link) in case of Primary Link Failure.
Step1 : Ask for branch code to branch person. According to branch code findout VSAT IP address of
that branch. Login to DC Secondary core router IP add. 172.16.5.83 via ssh (eg. Branch Code : 1280)

login as: vijuser

vijuser@172.16.5.83's password: password
<DC-Secondary>

Step2 : To check BSNL VSAT Link working or not by first pinging IDU IP (-1 from Router IP) & then
Router IP. If router IP is 10.103.184.42 , then IDU IP will be 10.103.184.41

<DC-Secondary>ping 10.103.184.41
PING 10.103.184.41: 56 data bytes, press CTRL_C to break
Reply from 10.103.184.41: bytes=56 Sequence=0 ttl=61 time=802 ms
Reply from 10.103.184.41: bytes=56 Sequence=1 ttl=61 time=727 ms
--- 10.103.184.41 ping statistics ---
2 packet(s) transmitted
2 packet(s) received
0.00% packet loss
round-trip min/avg/max = 727/744/802 ms (Means IDU is reachable)

If IDU is not pinging means not reachable, then there is problem in VSAT link.
As a first level troubleshooting , you can reboot the IDU & check again.

<DC-Secondary>ping 10.103.184.42
PING 10.103.184.42: 56 data bytes, press CTRL_C to break
Reply from 10.103.184.42: bytes=56 Sequence=0 ttl=251 time=659 ms
Reply from 10.103.184.42: bytes=56 Sequence=1 ttl=251 time=734 ms
--- 10.103.184.42 ping statistics ---
2 packet(s) transmitted
2 packet(s) received
0.00% packet loss
round-trip min/avg/max = 659/717/736 ms (Means Router is reachable)

If IDU is pinging but router IP is not pinging , then there is connectivity problem between router & IDU.

Step3 : Then telnet to router with VSAT IP of that branch.

<DC-Secondary>telnet 10.103.184.42
User Name:cmsfmnw
Password:
KAL-Balichakkur-1280>en
Password:
KAL-Balichakkur-1280#
KAL-Balichakkur-1280#sh ip int brie
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 172.19.172.1 YES NVRAM up up
FastEthernet0/1 10.103.184.42 YES NVRAM up up
Serial0/1/0 10.201.1.69 YES NVRAM up up
Dialer100 unassigned YES NVRAM up up
Dialer200 unassigned YES NVRAM up up
Loopback100 172.30.19.181 YES NVRAM up up
Tunnel100 172.16.120.22 YES NVRAM standby mode down
Tunnel200 172.16.154.22 YES NVRAM up up

KAL-Balichakkur-1280#sh backup
Primary Interface Secondary Interface Status
----------------- ------------------- ------
Serial0/1/0 Tunnel100 backup mode

KAL-Balichakkur-1280#sh run int s0/1/0

Building configuration...
Current configuration : 392 bytes
interface Serial0/1/0
description *** Connected to BSNL MPLS ***
backup delay 5 60
backup interface Tunnel100
ip address 10.201.1.69 255.255.255.252
crypto map vpn
service-policy output QOSout
end

KAL-Balichakkur-1280#sh run int tun100

Building configuration...

Current configuration : 192 bytes

!
interface Tunnel100
description *** Connected to VSAT DC ***
ip address 172.16.120.22 255.255.255.252
tunnel source 10.103.184.42 (VSAT IP address as a Tunnel Source)
tunnel destination 192.168.193.189 (DC side BSNL VSAT IP add. as a Tunnel Destination)
crypto map vpn
end

Step4 : Remove ‘no backup interface’ command from Serial Interface.

KAL-Balichakkur-1280#config t
KAL-Balichakkur-1280 (config)#interface Serial0/1/0
KAL-Balichakkur-1280 (config-if)#no backup interface
KAL-Balichakkur-1280#sh ip int brie
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 172.19.172.1 YES NVRAM up up
FastEthernet0/1 10.103.184.42 YES NVRAM up up
Serial0/1/0 10.201.1.69 YES NVRAM up up
Tunnel100 172.16.120.22 YES NVRAM up up

Now Tunnel100 status & protocol will shows up.

KAL-Balichakkur-1280#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
172.16.5.34 0 EXCHANGE/ - 00:00:39 172.16.119.209 Tunnel100

KAL-Balichakkur-1280#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
172.16.5.34 0 FULL/ - 00:00:40 172.16.119.209 Tunnel100

Step5 : Check Tunnel Interface status on DC Secondary router

<DC-Secondary>dis int tun 162 (Tunnel no. of that branch code)

Tunnel162 current state: UP
Line protocol current state: UP
Description: *** MUM HUB * HBL-Balichakkur-1280 ***
Internet Address is 172.16.120.21/30 Primary
Tunnel source 192.168.193.189, destination 10.103.184.42

<DC-Secondary>dis ospf peer Tunnel 162 (Tunnel no. of that branch code)
OSPF Process 1 with Router ID 172.16.5.34
Neighbor Brief Information

If OSPF neighborship not form then , tell HP Team to shut & unshut tunnel 162

<DC-Secondary>dis ospf peer Tunnel 162

OSPF Process 1 with Router ID 172.16.5.34
Neighbor Brief Information
Area: 0.0.0.0
Router ID Address Pri Dead-Time Interface State
172.30.22.49 172.16.120.22 1 40 Tun162 Exchange/ -

<DC-Secondary>dis ospf peer Tunnel 162

OSPF Process 1 with Router ID 172.16.5.34
Neighbor Brief Information
Area: 0.0.0.0
Router ID Address Pri Dead-Time Interface State
172.30.22.49 172.16.120.22 1 35 Tun162 Full/ -

Now that branch traffic will pass through BSNL VSAT Link.
SOP To Switch over WAN link from BSNL VSAT (Active Link) to
BSNL (Primary Link) when BSNL MPLS Link UP
Step1 :Login to DC Secondary core router IP add. 172.16.5.83 via ssh (eg. Branch Code : 1280)

login as: vijuser

vijuser@172.16.5.83's password: password
<DC-Secondary>

Step2 : Then telnet to router with VSAT IP of that branch.

<DC-Secondary>telnet 10.103.184.42
User Name:cmsfmnw
Password:
KAL-Balichakkur-1280>en
Password:
KAL-Balichakkur-1280#

KAL-Balichakkur-1280#sh ip int brie

Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 172.19.172.1 YES NVRAM up up
FastEthernet0/1 10.103.184.42 YES NVRAM up up
Serial0/1/0 10.201.1.69 YES NVRAM up up
Tunnel100 172.16.120.22 YES NVRAM up up

KAL-Balichakkur-1280#config t
KAL-Balichakkur-1280 (config)#interface Serial0/1/0
KAL-Balichakkur-1280 (config-if)# backup interface Tunnel100

KAL-Balichakkur-1280#sh ip int brie

Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 172.19.172.1 YES NVRAM up up
FastEthernet0/1 10.103.184.42 YES NVRAM up up
Serial0/1/0 10.201.1.69 YES NVRAM up up
Tunnel100 172.16.120.22 YES NVRAM standby mode down
Tunnel200 172.16.154.22 YES NVRAM up up

KAL-Balichakkur-1280#sh ip bgp sum

BGP router identifier 172.30.19.181, local AS number 65003
BGP table version is 827931, main routing table version 827931

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.201.1.70 4 9829 331655 12505 827908 0 0 00:03:09 345

Now that branch traffic will pass through BSNL MPLS Link.