You are on page 1of 41

Troubleshooting Cisco

Catalyst 2960, 3560,
3560e,3750 and 3750e,
Series Switches

BRKRST-3141

BRKRST-3141
14493_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr

Agenda

ƒ Product Overview
ƒ General Switch Health
ƒ Local Link Issues
ƒ Layer 2 Forwarding
ƒ Layer 3 IP Unicast
ƒ Quality of Service
ƒ Access Control Lists
ƒ IP Multicasting

BRKRST-3141
14493_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

Catalyst Switching Portfolio
Features, Scalability, Longevity

tion/Core
Distribu Catalyst 6500

Catalyst 4500/E

ss
ter Acce
Datacen
Catalyst 6500

Catalyst 4900

Blade
Switches

loset
Wiring C Catalyst 6500
Catalyst 4500/E

Catalyst 3750
Catalyst 3560
Catalyst 2900
Catalyst Express 500

Small Medium-sized Large

BRKRST-3141
14493_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr

Catalyst Fixed Switching
Catalyst 3750-E

Catalyst 3750

Scalability
ƒ 24/48 GE w/ 2x10 Gig ports
ƒ PoE—Up to 15.4W on
High Availability 48 ports
ƒ Modular power ƒ Cisco StackWise™ Plus
High Availability
supply and fan for enhanced scalability (3750-E)
ƒ Layer 3 routed access
ƒ Enhanced ƒ TwinGig for 10 second
Advanced and IPv6
availability 10 Gig upgrade
QoS and Multicast ƒ Virtualization support
with RPS 2300 ƒ Enhanced PoE for 802.11n device
ƒ PIM and Source Specific w/ VRF
support (20W)
Multicast
Scalability
ƒ 8Kbps and per VLAN
ƒ FE and GE Layer 2 Policing, Q-in-Q
switching Catalyst 3560-E
ƒ 8/24/48-ports w/ dual-
purpose Gig uplinks Catalyst 3560

ƒ PoE configurations
ƒ RPS 2300 support Catalyst 2960
w/ LAN Base

Catalyst 2960
w/ LAN Lite
Scalability
Advanced Security ƒ 8/24/48 FE and GE w/ up to 4 GE
Enhanced Layer 2+ uplink ports
ƒ Expanded and dynamic
ƒ Availability ACLs, DARP Inspection, ƒ PoE—370W total for up to 48 ports
ƒ Enhanced security IP Source Guard, Private VLAN
ƒ Advanced QoS

BRKRST-3141
14493_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

Before We Start
ƒ Most outputs taken in this presentation are taken from a
Catalyst 3750
ƒ Troubleshooting the 2960/2970/3560/3560e and 3750e
is done similar
ƒ For some commands the port-asic and the port on that asic
is needed
Show platform pm if-number shows this mapping
Switch#sh platform pm if-numbers

interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Gi3/0/1 109 109 1 1/1 3 1 1 local Yes Yes
Gi3/0/2 110 110 2 1/0 3 2 2 local Yes Yes
Gi3/0/3 111 111 3 1/3 3 3 3 local Yes Yes
Gi3/0/4 112 112 4 1/2 3 4 4 local Yes Yes
Gi3/0/5 113 113 5 1/5 3 5 5 local Yes Yes
Gi3/0/6 114 114 6 1/4 3 6 6 local Yes Yes
Gi3/0/7 115 115 7 1/7 3 7 7 local Yes Yes

BRKRST-3141
14493_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr

Agenda

ƒ Product Overview
ƒ General Switch Health
ƒ Local Link Issues
ƒ Layer 2 Forwarding
ƒ Layer 3 IP Unicast
ƒ Quality of Service
ƒ Access Control Lists
ƒ IP Multicasting

BRKRST-3141
14493_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

Memory Utilization

ƒ Processor memory is the memory used by IOS
ƒ I/O memory is used for packet buffers for traffic send to
the CPU (this is not used for normal packet switching)

Switch#sh memory statistics
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 2641D6C 81519252 31192204 50327048 49241540 48621848
I/O 7400000 12574720 8532852 4041868 3821068 4039616

ƒ Free(b) shows how much memory is available now
ƒ Lowest(b) shows what was the lowest free since boot
ƒ Largest(b) shows what the largest block of memory the
switch could allocate if it should be needed
BRKRST-3141
14493_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved.
Presentation_ID.scr

00% 0. All rights reserved. Inc.00% 0. one minute: 6%. All rights reserved. Cisco Public 10 © 2006.00% 0. Inc.00% 0 Load Meter 3 0 1 0 0.00% 0. Cisco Public 9 High CPU Due to Network Traffic ƒ The switches have 16 different CPU queues for different types traffic ƒ Each queue can only send a certain amount of traffic to the CPU. All rights reserved. Cisco Systems.00% 0. five minutes: 5% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 0 9 0 0.00% 0. Presentation_ID.00% 0 Chunk Manager 2 0 3779 0 0. Processor Utilization ƒ Processor is not involved in the normal switching of traffic ƒ CPU can become high due to Traffic send to CPU for processing Processes running on the CPU taking up resources *Note: Running show tech causes the virtual exec process to use some resources ƒ Using CPU cycles is not a problem Switch#sh processes cpu CPU utilization for five seconds: 4%/0%.scr . The port asic will drop the rest ƒ An overload on one queue should not cause problems for the other queues ƒ As long as you know why there are a lot of packets hit a specific queue there is no reason to panic BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.00% 0 DiagCard4/-1 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc.

1. The 16 Different Queues 0:rpc 1:stp 2:ipc 3:routing protocol 4:L2 protocol 5:remote console 6:sw forwarding 7:host 8:broadcast 9:cbt-to-spt 10:igmp snooping 11:icmp 12:logging 13:rpf-fail 14:dstats 15:cpu heartbeat ƒ To see what packets are located in the buffers use the show buffer command ƒ CPU buffer pools are named RxQ0 to RxQ15 ƒ The port asic can drop packets before reaching the CPU queue.. Inc.e042. SrcGID:2.00bb.1. Vlan:101 SrcGPN:2. Inc. switch 1 CPU queues: 6 14. Cisco Systems.. All rights reserved.100 IP_DA:10.. New destIndex is 0x02C7. ========================================== Egress: Asic 0. Cisco Public 11 The Software Forwarding Queue ƒ Traffic that needs to be forwarded by the switch that the hardware cannot handle will be sent to software forwarding queue ƒ Performance of software forwarding is much lower then when routing is done by the ASIC’s SW-FWD-Q:Consumed by SW-Bridging: Remote Port Blocked L3If:Vlan101 L2If:GigabitEthernet1/0/2 DI:0x2FD. ACLLogIdx:0x0. LT:7. DestIndex: 02FD.101.100 255 Station Descriptor: 02F30000.100 10.100 IP_Proto:255 TPFFD:D0000002_80048065_004D0040-000002FD_374CF00A_00000008 Switch#sh plat for gi 1/0/2 00.1.f7e8. Inc.. MacDA:000f.e042 ip 10. Presentation_ID. Cisco Public 12 © 2006. MacSA: 0000.scr .99. Queue 7: 10000 Å--.10000 packets dropped before reaching the CPU Queue BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved.101.87df 000f.f7e8..00bb. both locations should be checked Switch#sh platform port-asic stats drop Supervisor TxQueue Drop Statistics Queue 0: 0 . RewriteIndex: F00A Redirected by Input ACL. All rights reserved. BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.1.99.87df IP_SA:10.

260: HSRP: Vl100 Grp 0 Hello in 10.1. EIGRP. Inc.0002.55 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. LT:7.0.5e00. ƒ Drops on these queues can cause instability on the network BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. SrcGID:1. ACLLogIdx:0x0. OSPF. All rights reserved.scr . HSRP.1. Cisco Public 13 Routing Protocol Queue ƒ Receives all traffic for the routing protocols.1fc1 IP_SA:10.1.1. dropped BPDU’s are a danger in the network ƒ Layer 2 protocols queue for the rest CDP PAGP DTP LLDP UDLD Etc.ba88.2 IP_Proto:17 TPFFD:CC41C001_00640064_00A60042-000012FC_CDF80000_00000000 *Mar 6 00:47:39. MacSA: 0018. Switch#debug platform cpu-queues routing-protocol-q debug platform cpu-queue routing-protocol-q debugging is on Switch#debug standby HSRP debugging is on *Mar 6 00:47:39.0. etc. MacDA:0100. Presentation_ID.2 IP_DA:224. Inc. Cisco Systems. All rights reserved.2 Standby pri 100 vIP 10. like BGP. Layer 2 Control Protocol Queues ƒ Spanning tree has its own queue. All rights reserved.260: RT-Q:Queued: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0x12FC. Vlan:100 SrcGPN:1.1. Cisco Public 14 © 2006. Inc.1.

All rights reserved.1.2 IP_DA:10.1.ba88. SrcGID:488. ACLLogIdx:0x0.1. Host CPU Queue ƒ The host queue is used for all unicast traffic sent to the switch.e041. SrcGID:488.1. MacSA: 0018.1. MacSA: 0018. All rights reserved. LT:7. LT:7. eg. telnet.1. icmp (ping).1/32 receive for Vlan100 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.scr .1.1 IP_Proto:1 TPFFD:DC0001E9_00000064_00B00076-000000B0_A68A0000_00000000 Switch#sh ip cef 10. Cisco Public 15 Host CPU Queue—Drops ƒ Show buffer shows current buffer usage (RxQ7) ƒ When free buffers reaches below watermark(32). ACLLogIdx:0x0.1. permanent 192): 64 in free list (0 min. Vlan:100 SrcGPN:489. MacDA:000f.648: Host-Q:Queued L3If: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0xB0. 192 max allowed) 294 hits.1.648: Host-Q:Queued L3If: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0xB0. tacacs. MacDA:000f.1fc1 IP_SA:10.1. Inc. 0 misses Í=============== Misses equals drops BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.f7e8.1fc1 IP_SA:10. Presentation_ID. Inc.2 IP_DA:10.1 IP_Proto:1 TPFFD:DC0001E9_00000064_00B00076-000000B0_A68A0000_00000000 Switch#show buffer | begin RxQ7 RxQ7 buffers. Inc.ba88. Cisco Public 16 © 2006. All rights reserved. throttling might occur resulting in packet drop Switch#debug platform cpu-queues host-q debug platform cpu-queue host-q debugging is on Switch# *Mar 6 00:01:46.f7e8.1.1. Vlan:100 SrcGPN:489. Cisco Systems. Switch#debug platform cpu-queues host-q debug platform cpu-queue host-q debugging is on Switch# *Mar 6 00:01:46. etc.1 10. ssh. 2040 bytes (total 192.e041.

Hardware forwarding of the packet still occurs Switch#debug ip icmp ICMP packet debugging is on Switch#debug platform cpu-queues software-fwd-q debug platform cpu-queue sw-fwd-q debugging is on *Mar 9 21:34:30.1.1. MacSA: 000f.1. ICMP CPU Queue ƒ Receives all traffic for which an ICMP message should be generated (excluding PING) ƒ Receives a copy of the traffic for which an ICMP packet needs to be generated.1.1.695: ICMP-Q:Queued to Process.1fc1.1. Presentation_ID. Vlan:100 SrcGPN:163.3 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc.scr .e041 IP_SA:10.f7e8. Switch#show diagnostic post Makes sure faulty hardware is taken Switch#show diagnostic post out of service (POST = Power On Self Test) Runtime diagnostics Health-Monitoring Switch(config)#[no] diagnostic monitor interval { switch <1-9> } test { test-id | test. MacDA:0018. All rights reserved.1.1. for troubleshooting purposes. Inc.ba88.3: Remote Port Blocked L3If:Vlan100 L2If:GigabitEthernet4/0/1 DI:0xB4. All rights reserved. LT:7. use GW:10.1 for dest 77.1.1. Cisco Public 18 © 2006. It can also be used as a pre-deployment tool.695: ICMP: redirect sent to 10.1. use gw 10. Cisco Public 17 GOLD (Generic Online Diagnostics) 3750E/3750 and 3560E/3560 Boot-Up diagnostics Run During System Bootup.1.1 IP_Proto:1 TPFFD:EFC100A3_00640064_00B00076-000000B4_00A30000_00010000 *Mar 9 21:34:30.1 IP_DA:77. Inc. All rights reserved. Scheduled Switch(config)#[no] diagnostic schedule { switch <1-9> } test { Switch(config)#[no] diagnostic schedule { switch <1-9> } test { All diagnostic tests can be test-id | test-id-range | all } daily {hh:mm} test-id | test-id-range | all } daily {hh:mm} Scheduled. To run Non-disruptive Switch(config)#[no] diagnostic monitor interval { switch <1-9> } test { test-id | test- id-range | all } hh:mm:ss { ms <0-999> } { days <0-20> } id-range | all } hh:mm:ss { ms <0-999> } { days <0-20> } tests in the background Serves as HA trigger On-Demand Switch#diagnostic start {switch <1:9>} test {test-num | test range | Switch#diagnostic start {switch <1:9>} test {test-num | test range | All diagnostics tests can be run all | basic | non-disruptive } all | basic | non-disruptive } on demand. SrcGID:163. Cisco Systems.1. ACLLogIdx:0x0. for verification and troubleshooting purposes BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.

Cisco Public 20 © 2006.Per port test / Per device test / NA D/N/* .Always enabled monitoring test / NA A/I .Disruptive test / Non-disruptive test / NA S/* .Switch will reload after test list completion / NA P/* . All rights reserved.Only applicable to standby unit / NA X/* .Not a health monitoring test / NA F/* . Cisco Systems. Inc. Inc.05 n/a 2) TestPortAsicLoopback ----------------> B*D*X**IR* not configured n/a 3) TestPortAsicCam -----------------------> B*D*X**IR* not configured n/a 4) TestPortAsicRingLoopback ----------> B*D*X**IR* not configured n/a 5) TestMicRingLoopback ----------------> B*D*X**IR* not configured n/a 6) TestPortAsicMem ----------------------> B*D*X**IR* not configured n/a 7) TestInlinePwrCtlr -----------------------> B*D*X**IR* not configured n/a Diagnostics test suite attributes: B/* . All rights reserved. Switch 6: Running test(s) 2 will partition stack Switch 6: Running test(s) 2 may disrupt normal system operation Do you want to continue? [no]: Note: Tests Run to Completion (No Stop Command) BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.Monitoring is active / Monitoring is inactive R/* . Cisco Public 19 GOLD: OnDemand diagnostic start {switch <1:9>} test {test-num | test range | all | basic | non-disruptive } 3750E# diagnostic start switch 1 test 1 00:24:33: %DIAG-6-TEST_RUNNING: Switch 1: Running TestPortAsicStackPortLoopback{ID=1} 00:24:34: %DIAG-6-TEST_OK: Switch 1: TestPortAsicStackPortLoopback{ID=1} has completed successfully Disruptive Test: Users will be prompted if the test causes a lose of stack connectivity: Switch 3: Running test(s) 2 will cause the switch under test to reload after completion of the test list. Presentation_ID.scr . Inc.ms Threshold ==== ====================== ============ ========== ==== ======== 1) TestPortAsicStackPortLoopback ---> B*N****I** 005 01:10:25.will partition stack / NA BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. GOLD: OnDemand What Tests Can I Run? 3750E# show diagnostic content switch 1 Test Interval ID Test Name Attributes day hh:mm:ss.Basic ondemand test / NA P/V/* . All rights reserved.Fixed monitoring interval test / NA E/* . Switch 3: Running test(s) 2 may disrupt normal system operation Do you want to continue? [no]: Disruptive Test: Users will be prompted if the test causes stack partitioning: Switch 6: Running test(s) 2 will cause the switch under test to reload after completion of the test list.

Error code ----------------------> 0 (DIAG_SUCCESS) Total run count ----------------> 21 Last test execution time ----> Mar 13 1993 11:35:00 First test failure time ---------> n/a Last test failure time ---------> n/a Last test pass time -----------> Mar 13 1993 11:35:00 Total failure count -------------> 0 Consecutive failure count ---> 0 BRKRST-3141 __________________________________________________________________ 14493_04_2008_c2© 2008 Cisco Systems.scr . Inc. Inc. Cisco Public 21 On-Board Failure Logging (OBFL) 3750E/3560E ƒ Provides “flight recorder” capability ƒ It is enabled by default ƒ Collects operational data about the switch and the field-replaceable unit (FRU) including power supplies. All rights reserved. Presentation_ID. = Pass. F = Fail. All rights reserved. Cisco Public 22 © 2006. U = Untested) _________________________________________________________________ 1) TestPortAsicStackPortLoopback ---> . Cisco Systems. redundant power systems and small form-factor pluggable (SFP) modules ƒ Stores the data as a circular buffer on the flash (2Mbytes). All rights reserved. GOLD: OnDemand 3750E# show diagnostic status shows what diagnostics are currently running 3750E# show diagnostic result switch 1 detail Switch 1: SerialNo : CAT1033R1FS Overall diagnostic result: PASS Test results: (. Inc. Older data is compressed with less detail ƒ Each switch on the stack records its own OBFL data ƒ Information can be seen with Show logging onboard BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.

Inc. port 4’s led will light up) BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Public 23 Catalyst 3750E/3750 Stack Commands 3750# show switch detail Current Switch# Role Mac Address Priority State ------------------------------------------------------ 1 Slave 000c. (for example.4f00 9 Ready *2 Master 000d. Stack Ring Protocol : StackWisePlus 1 5781 2 4928 Total frames sent to stack ring : 10709 Note: these counts do not include frames sent to the ring by certain output features such as output SPAN and output ACLs. Presentation_ID. Stack Partitioning (Catalyst 3750E/3750) How Not to Remove Switches from a Stack! M #1 M #1 M #1 S #3 S #3 S #3 S #2 S #2 S #2 S #4 M #4 M #4 S #5 S #5 S #5 S #6 S #6 S #6 ƒ After a stack has been split.scr . All rights reserved. ƒ You can also use the mode button on the front of the switch to determine its stack switch number. Cisco Systems. if the switch is switch# 4 in the stack. Cisco Public 24 © 2006. All rights reserved.1680 15 Ready Stack Port Status Neighbors Switch# Port 1 Port 2 Port 1 Port 2 ------------------------------------------------------ 1 Ok Ok 2 2 2 Ok Ok 1 1 3750E# show switch stack-ring speed 3750# show switch stack-ring activity Stack Ring Speed : 32G Switch Frames sent to stack ring (approximate) Stack Ring Configuration: Full -----------------------------------------------. All rights reserved.bd5c.30ae. both stacks have the M = Master same config S = Slave ƒ For Layer 2 switching this is not an issue ƒ Layer 3 will become broken if SVI’s and loopbacks are used as both stacks will use these BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc. Inc. the LED on the port with the corresponding switch number will illuminate.

Agenda ƒ Product Overview ƒ General Switch Health ƒ Local Link Issues ƒ Layer 2 Forwarding ƒ Layer 3 IP Unicast ƒ Quality of Service ƒ Access Control Lists ƒ IP Multicasting BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved. All rights reserved. All rights reserved. Presentation_ID. Cisco Systems. Inc. Inc.scr . Cisco Public 26 © 2006. Cisco Public 25 Troubleshooting Link Issues ƒ Is the link coming up as expected ƒ Are packets being sent and received on the port? ƒ Are there errors on the port ƒ Are there drops on the port BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc.

Cisco Public 27 Checking Physical Cabling Issues ƒ Use the TDR feature on the port to determine possible cabling issues as miswiring or cable breaks ƒ Interfaces will be brought down and up when run on active ports Switch#test cable-diagnostics tdr interface GigabitEthernet4/0/1 TDR test started on interface Gi4/0/1 A TDR test can take a few seconds to run on an interface Use 'show cable-diagnostics tdr' to read the TDR results. changed state to up %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/1. All rights reserved. -------------------- Gi4/0/1 1000M Pair A 3 +/. Inc.scr . All rights reserved. Inc.1 meters Pair A Normal Pair B 2 +/. Link Not Coming Up ƒ Verify the configured duplex and speed on both switch and attached host.1 meters Pair B Normal Pair C 3 +/.-----------------. changed state to down *%LINK-3-UPDOWN: Interface GigabitEthernet4/0/1. Switch# %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/1.----.1 meters Pair C Normal Pair D 3 +/. Inc.1 meters Pair D Normal BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Systems. All rights reserved. ----------. Cisco Public 28 © 2006. changed state to down %LINK-3-UPDOWN: Interface GigabitEthernet4/0/1.---------. changed state to upw Switch#show cable-diagnostics tdr interface GigabitEthernet4/0/1 TDR test last run on: March 01 03:11:11 Interface Speed Local pair Pair length Remote pair Pair status --------. Presentation_ID. fixing speed and duplex should be done on both sides ƒ Upgrade the NIC drivers on the host to the latest version available from the vendor ƒ Try a different cable/NIC and switchport to exclude faulty hardware Switch#show interfaces status | inc connected Gi1/0/1 connected trunk a-full 10 10/100/1000BaseTX Gi1/0/2 connected 101 a-full a-100 10/100/1000BaseTX Gi1/0/24 connected 1 a-full a-1000 10/100/1000BaseTX BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.

Presentation_ID. rxload 1/255 Encapsulation ARPA. All rights reserved. Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec. 40593 multicast. 0 pause input 0 input packets with dribble condition detected 350898 packets output. 0 throttles 0 input errors. output 00:00:12. 0 no buffer Received 40607 broadcasts (40593 multicasts) 0 runts. 0 frame. Cisco Public 29 Showing What Kind of Errors There Are Switch#show interfaces GigabitEthernet 1/0/1 counters errors Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards Gi1/0/1 0 0 0 0 0 0 Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Gi1/0/1 0 0 0 0 0 0 0 Switch#sh interfaces counters errors Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards Gi1/0/1 0 0 0 0 0 0 Gi1/0/2 0 0 0 0 0 0 <snip> Gi2/0/12 0 0 0 0 0 0 Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Gi1/0/1 0 0 0 0 0 0 0 Gi1/0/2 0 0 0 0 0 0 0 See Appendix A for Error Explanation BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes).scr .e001) MTU 1500 bytes. 0 packets/sec 5 minute output rate 1000 bits/sec. reliability 255/255. All rights reserved. ARP Timeout 04:00:00 Last input 00:00:15. 0 overrun. 0 output buffers swapped out BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. 0 deferred 0 lost carrier. Port Status and Counters Overview Switch#show interface GigabitEthernet 1/0/1 GigabitEthernet1/0/1 is up. Inc. 0 CRC. 35603065 bytes. All rights reserved. Inc. 0 PAUSE output 0 output buffer failures. 1000Mb/s. address is 000f. 4 interface resets 0 babbles. Inc. 0 ignored 0 watchdog.f7e8. Cisco Public 30 © 2006. output flow-control is unsupported ARP type: ARPA. 1 packets/sec 75390 packets input. media type is 10/100/1000BaseTX Media-type configured as connector input flow-control is off.f7e8. 0 underruns 0 output errors. DLY 10 usec. 0 no carrier. txload 1/255. 9856388 bytes. 0 giants. Cisco Systems.e001 (bia 000f. line protocol is up (connected) Hardware is Gigabit Ethernet. 0 collisions. loopback not set Keepalive set (10 sec) Full-duplex. BW 1000000 Kbit. 0 late collision.

too large 0 Excess defer frames 0 Invalid frames. All rights reserved. Cisco Systems. All rights reserved. Inc. All rights reserved. Presentation_ID. Inc. Cisco Public 31 Statistics per Port-Asic ƒ Shows the statistics per port-asic ƒ Remote command <switch> should be used to show these statistics for member switches ƒ Looking at the statistics per port-asic will give a quick overview of possible drops/issues on the switch BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.scr . too large 0 VLAN discard frames 0 Valid frames. too small 1061 64 byte frames 0 Valid frames. too small 263 127 byte frames 0 255 byte frames 0 Too old frames 136 511 byte frames 0 Valid oversize frames 18 1023 byte frames 0 System FCS error frames 0 1518 byte frames 0 RxPortFifoFull drop frame BRKRST-3141 14493_04_2008_c2 0 Cisco © 2008 Too large Systems. Additional Statisics from the Port-Asic Switch#show controller ethernet-controller GigaBitEthernet 1/0/1 Transmit GigabitEthernet4/0/1 Receive 159038 Bytes 803050 Bytes 791 Unicast frames 862 Unicast frames 669 Multicast frames 5402 Multicast frames 18 Broadcast frames 1602 Broadcast frames 0 Too old frames 60417 Unicast bytes 0 Deferred frames 419857 Multicast bytes 0 MTU exceeded frames 314886 Broadcast bytes 0 1 collision frames 0 Alignment errors 0 2 collision frames 0 FCS errors 0 3 collision frames 0 Oversize frames 0 4 collision frames 0 Undersize frames 0 5 collision frames 0 Collision fragments 0 6 collision frames 0 7 collision frames 6093 Minimum size frames 0 8 collision frames 1158 65 to 127 byte frames 0 9 collision frames 26 128 to 255 byte frames 0 10 collision frames 294 256 to 511 byte frames 0 11 collision frames 295 512 to 1023 byte frames 0 12 collision frames 0 1024 to 1518 byte frames 0 13 collision frames 0 Overrun frames 0 14 collision frames 0 Pause frames 0 15 collision frames 0 Excessive collisions 0 Symbol error frames 0 Late collisions 0 Invalid frames. Cisco Public 32 © 2006. frames Inc.

wt-1 drop frames 0 RxQ-0. wt-0 enqueue frames 0 RxQ-0. Inc. All rights reserved. All rights reserved. All rights reserved. wt-2 drop frames 0 RxQ-1. wt-1 enqueue frames 0 RxQ-0. Inc. wt-0 drop frames <snip> 100 TxBufferFull Drop Count 0 Rx Fcs Error Frames 0 TxBufferFrameDesc BadCrc16 0 Rx Invalid Oversize Frames 0 TxBuffer Bandwidth Drop Cou 0 Rx Invalid Too Large Frames 0 TxQueue Bandwidth Drop Coun 0 Rx Invalid Too Large Frames 0 TxQueue Missed Drop Statist 0 Rx Invalid Too Small Frames 0 RxBuffer Drop DestIndex Cou 0 Rx Too Old Frames 0 SneakQueue Drop Count 0 Tx Too Old Frames 0 Learning Queue Overflow Fra 0 System Fcs Error Frames 0 Learning Cam Skip Count 0 Sup Queue 0 Drop Frames 0 Sup Queue 8 Drop Frames <snip> 0 Sup Queue 7 Drop Frames 0 Sup Queue 15 Drop Frames BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. PortASIC 0 Statistics --------------------------------------------------------------------------- 0 RxQ-0. Presentation_ID. Inc. wt-2 enqueue frames 0 RxQ-0. PortASIC 0 Statistics --------------------------------------------------------------------------- 100 TxBufferFull Drop Count 0 Rx Fcs Error Frames 0 TxBufferFrameDesc BadCrc16 0 Rx Invalid Oversize Frames 0 TxBuffer Bandwidth Drop Cou 0 Rx Invalid Too Large Frames 0 TxQueue Bandwidth Drop Coun 0 Rx Invalid Too Large Frames 0 TxQueue Missed Drop Statist 0 Rx Invalid Too Small Frames 0 RxBuffer Drop DestIndex Cou 0 Rx Too Old Frames 0 SneakQueue Drop Count 0 Tx Too Old Frames 0 Learning Queue Overflow Fra 0 System Fcs Error Frames 0 TxBufferFrameDesc BadCrc16 0 Rx NP Packet Count0 0 RxBuffer Drop DestIndex Cou 0 Rx NP Packet Count1 0 Sup Queue 0 Drop Frames 0 Sup Queue 8 Drop Frames 0 Sup Queue 1 Drop Frames 0 Sup Queue 9 Drop Frames 0 Sup Queue 2 Drop Frames 0 Sup Queue 10 Drop Frames 0 Sup Queue 3 Drop Frames 0 Sup Queue 11 Drop Frames 0 Sup Queue 4 Drop Frames 0 Sup Queue 12 Drop Frames 0 Sup Queue 5 Drop Frames 0 Sup Queue 13 Drop Frames 0 Sup Queue 6 Drop Frames 0 Sup Queue 14 Drop Frames 0 Sup Queue 7 Drop Frames 0 Sup Queue 15 Drop Frames BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. wt-0 drop frames 8811506 RxQ-0. Cisco Public 34 © 2006. wt-0 enqueue frames 0 RxQ-1. Cisco Public 33 Port-Asic Statistics 3560E and 3750E Switch#remote command 4 show controller ethernet-controller port-asic statistics Switch : 4 : ------------ =========================================================================== Switch 4. Port-Asic Statistics 2960/3560 and 3750 Switch#sh controllers ethernet-controller port-asic statistics =========================================================================== Switch 2. Cisco Systems.scr .

All rights reserved. Presentation_ID. All rights reserved. Cisco Systems. Inc. Cisco Public 35 Agenda ƒ Product Overview ƒ General Switch Health ƒ Local Link Issues ƒ Layer 2 Forwarding ƒ Layer 3 IP Unicast ƒ Quality of Service ƒ Access Control Lists ƒ IP Multicasting BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved. Looking for Egress Queue Drops ƒ Queue and weight are 0-based in these ƒ Tuning of buffers is only possible when QoS is enabled ƒ Drops on egress indicate oversubscription Switch#show platform port-asic stats drop gigabitEthernet 2/0/1 Interface Gi2/0/1 TxQueue Drop Statistics Queue 0 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 1 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 2 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 3 Weight 0 Frames 100000 Weight 1 Frames 0 Weight 2 Frames 0 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Public 36 © 2006.scr . Inc. Inc.

Inc. All rights reserved. ----------.0000.2 P2p ƒ Step 3: Check if the packets are being received/send on the port Switch#show interfaces gigabitEthernet 1/0/2 counters Port InOctets InUcastPkts InMcastPkts InBcastPkts Gi1/0/2 2108289 48 0 6813 Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts Gi1/0/2 36817803 48229 252940 72564 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc.Nbr Type ------------------.-------------------------------- VLAN0100 Desg FWD 19 128. All rights reserved. Presentation_ID. -------. ----- 100 0000. Cisco Public 37 Layer 2 Forwarding Troubleshooting ƒ Step 4: Verify if the Mac-address is correcly learned on the port Switch#sh mac address-table interface gigabitEthernet 1/0/2 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---.--. ----- 100 0000. Layer 2 Forwarding Troubleshooting ƒ Step 1: Verify if the link is up Switch#show interface gi 1/0/2 status Port Name Status Vlan Duplex Speed Type Gi1/0/2 connected 100 a-full a-100 10/100/1000BaseTX ƒ Step 2: Verify if the port is in the right vlan and is forwarding Switch#show spanning-tree interface Gi1/0/2 Vlan Role Sts Cost Prio. All rights reserved.--------.1234 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---.0000. Inc. Cisco Systems. Cisco Public 38 © 2006.0000.---.-------. ----------. -------.1234 DYNAMIC Gi1/0/1 Total Mac Addresses for this criterion: 1 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.4321 DYNAMIC Gi1/0/2 Total Mac Addresses for this criterion: 1 ƒ Step 5: Verify if the destination Mac-address is learned on the switch on the expected port Switch#sh mac address-table dynamic address 0000.scr .

L2EncapType 0. is the port authorized? ƒ Does port security allow more Mac-addresses? ƒ Is the port in spanning tree forwarding? BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. RewriteIndex: F009 ========================================== Egress: Asic 1. Presentation_ID. Cisco Public 39 No Mac-Address Learned on Port ƒ Are packets being received? ƒ Is the expected Mac-address learned on another port? ƒ Check if dot1x is in use.4321 0000. All rights reserved. All rights reserved.0000.1234 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Public 40 © 2006.0000.4321 0000. if so. Inc. L2EncapType 0. Cisco Systems. non-SPAN portMap 0x8 Output Packets: ------------------------------------------ GigabitEthernet1/0/1 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 30_00B00000_00001234-00_00000000_00004321 01FFC 01000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/1 0100 0000. Mapped 9. Inc.1234 Ingress: Global Port Number: 2. All rights reserved.scr . L3EncapType 3 Hashes: L2Src 0x00 L2Dst 0x0B L3Src 0x00 L3Dst 0x0B Lookup Key-Used Index-Hit A-Data Classify 68_00B00000_00001234-00_00000000_00004321 017FC 00000000 InputACL 20_00B00000_00001234-00_00000000_00004321 01FF8 01000000 L2LrnMsk FF_03FFFFFF_FFFFFFFF-00_000003FF_00000000 L2Learn 83_00090000_00004321-C3_00002402_00000000 00E5C 0000005D L2FwdMsk FF_03FFFFFF_FFFFFFFF L2Fwd 83_00090000_00001234 00E52 000000B8 Station Descriptor: F001F009. Mapped 9. L3EncapType 3 portMap 0x8. lpn: 2 Asic Number: 1 Source Vlan Id: Real 100. Inc. switch 1 Source Vlan Id: Real 100.0000. Layer 2 Forwarding Troubleshooting ƒ Step 6 : Use show platform forward to verify the hardware forwarding Switch#show platform forward gigabitEthernet 1/0/2 0000. DestIndex: F001.0000.

Inc. Cisco Public 41 Agenda ƒ Product Overview ƒ General Switch Health ƒ Local Link Issues ƒ Layer 2 Forwarding ƒ Layer 3 IP Unicast ƒ Quality of Service ƒ Access Control Lists ƒ IP Multicasting BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. notification 0. received 2 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved. All rights reserved. detected flag not set Number of topology changes 100 last change occurred 3d01h ago from GigaBitEthernet 3/0/1 Times: hold 1. topology change 35.scr . aging 300 Port 1 (GigabitEthernet1/0/1) of VLAN0100 is designated forwarding <output removed> Number of transitions to forwarding state: 1 Link type is point-to-point by default BPDU: sent 131991. topology change 0. Mac-Address Disappears from a Port ƒ Check for spanning tree topology changes ƒ Does the link remain up ƒ Is it learned on another port Switch#sh spanning-tree vlan 100 detail VLAN0100 is executing the ieee compatible Spanning Tree protocol Topology change flag not set. Cisco Public 42 © 2006. Inc. Cisco Systems. notification 2 hello 2. Presentation_ID. Inc. max age 20. forward delay 15 Timers: hello 0. All rights reserved.

round-trip min/avg/maz = 1/4/9 ms 3750#sh ip arp vlan 100 Protocol Address Age (min) Hardware Addr Type Interface Internet 100.1.2 Type escape sequence to abort.1. Inc.2 23 0018.2.2 source lo0 Type escape sequence to abort.1. Cisco Systems.1. Layer 3 IP Unicast Routing ƒ Verify source reachability from the switch ƒ Verify destination reachability from the switch ƒ Verify hardware forwarding from source to destination (and back) 3750 3750 3750 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Public 44 © 2006. round-trip min/avg/max = 1/4/9 ms 3750#ping 100. timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5). Sending 5.1.ba88.1. Sending 5. Inc. All rights reserved.5c5c.1.ba88.1.1fc1 ARPA Vlan100 3750#sh mac address-table address 0018. timeout is 2 seconds: Packet sent with a source address of 99. Cisco Public 43 Verify Source Reachability 3750#ping 100.1.1 .01c1 ARPA Vlan100 Internet 100.2. All rights reserved.1.1 !!!!! Success rate is 100 percent (5/5). Inc. ----- 100 0018. ----------. Presentation_ID. 100-byte ICMP Echos to 100. 0011. All rights reserved. -------.1.scr .1.1.1fc1 DYNAMIC Gi1/0/1 Total Mac Addresses for this criterion: 1 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.1fc1 Mac Address Table ------------------------------------------- Vlan Mac Address Type Ports ---.ba88.1. 100-byte ICMP Echos to 100.

the vlan of the source BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc. L3EncapType 0 portMap 0x0. DestIndex: 00B0.1. All rights reserved.scr . lpn: 3 Asic Number: 1 Source Vlan Id: Real 100. Cisco Public 46 © 2006. RewriteIndex: 0000 ========================================== <output removed> Output Packets: ========================================== Egress: Asic 0.1. Cisco Systems. Cisco Public 45 Verify Destination Reachability ƒ Verify there is a route to reach the destination ƒ Verify there is a valid ARP for the next hop ƒ PING the destination ƒ PING the destination as source. Inc. Inc.ba88. non-SPAN portMap 0x0 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.1 icmp 0 0 Ingress: Global Port Number: 1.1.01c1 ip 100.2 100. Verify Source Reachability(2) ƒ Verify packets from the source are getting to the CPU 3750#show platform forward 0018.1fc1 0011. Mapped 9. Presentation_ID.5c5c. L2EncapType 0. L3EncapType 0 Hashes: L2Src 0x03 L2Dst 0x05 L3Src 0x09 L3Dst 0x03 Lookup Key-Used Index-Hit A-Data Classify 78_64010101_64010102-00_01000000_00000100 017FE 00000000 InputACL 40_64010101_64010102-00_01000000_00000100 01FFA 03000000 L2LrnMsk FF_03FFFFFF_FFFFFFFF-00_000003FF_00000000 L2Learn 80_00090018_BA881FC1-C0_00002401_00000000 00E54 00000040 L3LclMsk FF_FF8FFC00_FFFFFFFF L3Local C0_00302401_64010101 01CF0 00000000 L3Scndr 10_64010101_64010102-00_00000000_00000100 008AA 000A0008_00000000 Lookup Used: Secondary Station Descriptor: 00B00000. Source Vlan Id: Real 100. switch 2 CPU queues: 7 14.1. L2EncapType 0. All rights reserved. Mapped 9. All rights reserved.

Cisco Public 47 Verify Hardware Forwarding ƒ Show platform forward verifies how the HW is setup 3750#sh plat for Gi1/0/2 0.168. Presentation_ID.100 source vlan 101 Type escape sequence to abort. 100-byte ICMP Echos to 172. RewriteIndex: F001 <snip> Output Packets: ------------------------------------------ GigabitEthernet1/0/24 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 50_AC106464_C0A86464-00_00000000_00000100 01FFE 03000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/1 0100 0000.1. DestIndex: 02F4.16.1. Inc. type extern 2. timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5).100. via Vlan100 Route metric is 20.1.ba88.100. L3EncapType 0 Hashes: L2Src 0x0B L2Dst 0x0F L3Src 0x0C L3Dst 0x0D Lookup Key-Used Index-Hit A-Data Classify 78_64010102_C0A86464-00_00000000_00000100 017FE 00000000 InputACL 40_64010102_C0A86464-00_00000000_00000100 01FFA 03000000 L2LrnMsk FF_03FFFFFF_FFFFFFFF-00_000003FF_00000000 L2Learn 80_00010000_00001234-C0_00000401_00000000 01820 00000000 L2FwdMsk FF_03FFFFFF_FFFFFFFF L2Fwd 80_00010011_5C5C01C2 01820 00000040 Station Descriptor: 02F30000.1.1.100 Type escape sequence to abort.2 172.100. 00:08:54 ago.2 9 0018.2 Protocol Address Age (min) Hardware Addr Type Interface Internet 10.100. All rights reserved. Mapped 1.100.1. Verify Destination Reachability 3750#sh ip route 172. Inc.16.1fc1 ========================================== BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. traffic share count is 1 3750#sh ip arp 10. from 100. distance 110. round-trip min/avg/max = 1/2/8 ms BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. 100-byte ICMP Echos to 172. round-trip min/avg/max = 1/1/1 ms 3750#ping 172. All rights reserved.100. 00:08:54 ago Routing Descriptor Blocks: * 10.1234 11.100.0/24 Known via "ospf 1". Sending 5.1.1.1. Cisco Public 48 © 2006.16. Sending 5.5c5c.01c2 ip 100. L2EncapType 0.ba88.1 !!!!! Success rate is 100 percent (5/5).16. Inc.100.1fc1 ARPA Vlan100 Switch#ping 172.2 on Vlan100.1234 0018. Cisco Systems.100.16. All rights reserved.16. metric 20. timeout is 2 seconds: Packet sent with a source address of 192.1.scr . forward metric 1 Last update from 10.0000.100. lpn: 3 Asic Number: 1 Source Vlan Id: Real 1.100 icmp 0 0 Ingress: Global Port Number: 1.1.100 Routing entry for 172.1.0.2.16.2.

scr . All rights reserved. All rights reserved. Inc. All rights reserved.34 : 0 0 0 0 10000 <output remove> Policer: Inprofile: 1467 OutofProfile: 8533 ƒ 10000 packets were received. Cisco Systems. Inc. Cisco Public 49 QoS Troubleshooting (Ingress) access dot1q 3750 10000 IP packets with DSCP 34 Ingress policer with trust DSCP Switch#sh mls qos interface gi 1/0/2 statistics GigabitEthernet1/0/2 (All statistics are in packets) dscp: incoming ------------------------------- 0 . ƒ 1467 packets were in profile ƒ 8533 were dropped due to exceeding the policer BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Agenda ƒ Product Overview ƒ General Switch Health ƒ Local Link Issues ƒ Layer 2 Forwarding ƒ Layer 3 IP Unicast ƒ Quality of Service ƒ Access Control Lists ƒ IP Multicasting BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Presentation_ID. Inc. Cisco Public 50 © 2006.4 : 0 0 0 0 0 30 .

Possible reasons: Attached service policy does not mark or trust dscp value Traffic is being routed via the CPU BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Systems.29 : 0 0 0 0 0 30 . Inc. Inc.34 : 0 0 0 0 1467 ƒ 10000 packets were received. Cisco Public 51 QoS Troubleshooting (Egress) access dot1q 3750 10000 IP packets with DSCP 34 Ingress policer with trust DSCP Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics GigabitEthernet1/0/1 (All statistics are in packets) <output removed> 0 – 4 : 1467 0 0 0 0 30 . All rights reserved. All rights reserved. Inc.34 : 0 0 0 0 0 ƒ 1467 packets were in profile and made it to the egress port but with DSCP 0 in stead of 34. Presentation_ID. All rights reserved.scr . ƒ 1467 packets were in profile and made it to the egress port BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Public 52 © 2006. QoS Troubleshooting (Egress) access dot1q 3750 10000 IP packets with DSCP 34 Ingress policer with trust DSCP Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics GigabitEthernet1/0/1 (All statistics are in packets) <output removed> 25 .

Cisco Systems. All rights reserved. Inc. 8920 were dropped BRKRST-3141 on egress 14493_04_2008_c2 © 2008 Cisco Systems. threshold 1 14493_04_2008_c2 © 2008 Cisco Systems.34 : 0 0 0 0 10000 Switch#sh mls qos maps dscp-output-q Dscp-outputq-threshold map: d1 :d2 0 1 2 3 4 5 6 7 8 9 ------------------------------------------------------------ 0 : 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 1 : 02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01 2 : 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 3 : 03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 4 : 01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01 5 : 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 6 : 04-01 04-01 04-01 04-01 ƒ 10000 packets were received and will go on egress to BRKRST-3141 Q4. Inc. Cisco Public 54 © 2006.scr . QoS Troubleshooting (WTD Drops) 100Mb/s 10Mb/s 3750 10000 IP packets with DSCP 34 Switch#sh mls qos interface gi 1/0/2 statistics GigabitEthernet1/0/2 (All statistics are in packets) dscp: incoming ------------------------------- 0 . Cisco Public 53 QoS Troubleshooting (WTD Drops) 100Mb/s 10Mb/s 3750 10000 IP packets with DSCP 34 Switch#sh platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 0 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 1 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 2 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 3 Weight 0 Frames 8920 Weight 1 Frames 0 Weight 2 Frames 0 ƒ 10000 packets were received. All rights reserved. All rights reserved. Presentation_ID.4 : 0 0 0 0 0 30 . Inc.

at what level fill are packets going to be dropped ƒ Reserved. Inc. Cisco Public 56 © 2006. Cisco Systems. Presentation_ID. upper limit of what the port can use Switch#showSwitch#sh mls qos queue-set Queueset: 1 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 200 200 100 100 threshold2: 200 200 100 100 reserved : 50 50 50 50 maximum : 400 400 400 400 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved. Inc. All rights reserved.scr . Cisco Public 55 QoS Troubleshooting (Buffer Tuning) 100Mb/s 10Mb/s 3750 400 IP packets with DSCP 34 Switch#show platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 3 Weight 0 Frames 37 Å---. Inc. Buffer Tuning ƒ 2 queue-sets are available ƒ Threshold1 and Treshold2 . ƒ Maximum.Packet drops occured Switch#sh mls qos queue-set 1 Queueset: 1 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 100 200 100 100 threshold2: 100 200 100 100 reserved : 50 50 50 50 maximum : 400 400 400 400 Switch(config)#mls qos queue-set output 1 threshold 1 200 200 50 400 Switch#show platform port-asic stats drop gigabitEthernet 1/0/1 Interface Gi1/0/1 TxQueue Drop Statistics Queue 3 Weight 0 Frames BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved. how much will be reserved for this port.

Egress Packet Drops ƒ Packet drops don’t always indicate a problem Before tuning the buffers make sure there is a real performance problem. All rights reserved. Cisco Public 57 Agenda ƒ Product Overview ƒ General Switch Health ƒ Local Link Issues ƒ Layer 2 Forwarding ƒ Layer 3 IP Unicast ƒ Quality of Service ƒ Access Control Lists ƒ IP Multicasting BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Gigabit attached servers can easily oversubscribe 100Mb/s attached clients Most protocols react well to drop and will slow down so maximum performance can be achieved BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Systems. Cisco Public 58 © 2006. All rights reserved. Inc. All rights reserved. Inc. Presentation_ID.scr . Inc.

etc do not take up TCAM space BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.scr . The above information is meant to provide an abstract view of the current TCAM utilization BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Systems. All rights reserved. Cisco Public 59 Access-List Space in TCAM ƒ TCAM space is limited Switch#sh platform tcam utilization CAM Utilization for ASIC# 0 Max Used Masks/Values Masks/values Unicast mac addresses: 784/6272 14/40 IPv4 IGMP groups + multicast routes: 144/1152 7/27 IPv4 unicast directly-connected routes: 784/6272 14/40 IPv4 unicast indirectly-connected routes: 272/2176 11/55 IPv4 policy based routing aces: 0/0 0/0 IPv4 qos aces: 768/768 260/260 IPv4 security aces: 1024/1024 723/723 Note: Allocation of TCAM entries per feature uses a complex algorithm. Software forwarding is slower. Inc. Cisco Public 60 © 2006. All rights reserved. All rights reserved. VACL and PACL are all compressed and loaded into the TCAM ƒ The number of available entries depends on the switch model and the choosen SDM template ƒ When an ACL does not fit in hardware it will be processed in software. more latency with lower capacity ƒ Access-list used for software features like BGP. Inc. Inc. Hardware Support ƒ RACL. SNMP. Presentation_ID.

Inc.1.scr . Cisco Systems.. Current configuration : 134 bytes ! interface GigabitEthernet1/0/2 switchport access vlan 101 ip access-group 123 in mls qos trust dscp spanning-tree portfast end BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved. Inc. TCAM Overload ƒ An error message will get generated ƒ Traffic forwarding will be done (partly) in Software Syslog: %ACLMGR-4-UNLOADING: Unloading ACL input label 1 VLAN interfaces 101 IPv4/Mac feature %ACLMGR-4-ACLTCAMFULL: ACL TCAM Full. Inc. All rights reserved.2 any Switch#sh run int gi 1/0/2 Building configuration. Presentation_ID. Cisco Public 61 Port Access Lists Switch#sh run | inc access-list access-list 123 permit ip host 10.100. Cisco Public 62 © 2006.. Software Forwarding packets on Input label 1 on L3 L2 Switch#sh platform acl oacltcamfull Vlan oacl_tcam_full_bitmap notify_apps 101 0x 0 NOT-FULL Vlan ipv6_oacl_tcam_full_bitmap notify_apps Switch#sh platform acl label 1 detail IPv4/MAC ACL label ------------------ Unloaded due to lack of space: BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved.

1.255. Presentation_ID.255. Inc.2 any Switch#sh platform acl interface vlan 101 Input Label: 1 Output Label: 0 (default) Input IPv6 Label: 1 Output IPv6 Label: 0 (default) BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.1 255. All rights reserved. Cisco Public 63 Router Access-List Configuration : ! interface Vlan101 ip address 10.scr . 0 VMRs BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.1.Permit IP Source address Mask: 00000000 00000000 00000000 00000000 00000000 Value: 00000000 00000000 00000000 00000000 00000000 Result: 0x00 Å--.101. All rights reserved. All rights reserved.0 ip access-group 123 in ! access-list 123 permit ip host 10. Inc. Cisco Public 64 © 2006.Deny Mask & Value all 0 = any any Mask: 00000000 00000000 00000000 00000000 00000000 Value: 00000000 00000000 00000000 00000000 00000000 Result: 0x09 IP Source Guard: 0 VMRs LPIP: 0 VMRs MAC Access Group: (none). Inc. Cisco Systems. 3 VMRs Mask: 00000000 FFFFFFFF 00000000 00000000 00000000 Value: 00000000 0A640102 00000000 00000000 00000000 Result: 0x09 Å--.100. Port Access Lists Switch#sh platform acl interface gigabitEthernet 1/0/2 portlabels detail Port based ACL: (asic 1) ---------------------------- Input Label: 4 Op Select Index: 255 Interface(s): Gi1/0/2 Access Group: 123.

Presentation_ID.1. Inc. All rights reserved. Cisco Systems. 5 VMRs.scr .2 any Switch#sh vlan filter VLAN Map FilterMap is filtering VLANs: 101 Switch#show platform acl vlan 101 Input Label: 1 Output Label: 1 Input IPv6 Label: 1 Output IPv6 Label: 1 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc.100. Inc. All rights reserved. Cisco Public 66 © 2006. Mask: 00000000 FFFFFFFF 00000000 00000000 00000000 Value: 00000000 0A640102 00000000 00000000 00000000 Result: 0x09 Mask: 00000000 00000000 05000000 00000000 00000000 <output removed> BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Router Access-List Switch#sh platform acl label 1 detail IPv4/MAC ACL label ------------------ Input Op Select Index 255: Output Op Select Index 255: Input Features: Interfaces or VLANs: Vl101 Vlan Map: (none) Access Group: 123. Cisco Public 65 Vlan Access-List Configuration: vlan access-map FilterMap 10 action drop match ip address 123 ! vlan filter FilterMap vlan-list 101 ! access-list 123 permit ip host 10. All rights reserved.

Cisco Systems. Inc. Inc. Cisco Public 67 Agenda ƒ Product Overview ƒ General Switch Health ƒ Local Link Issues ƒ Layer 2 Forwarding ƒ Layer 3 IP Unicast ƒ Quality of Service ƒ Access Control Lists ƒ IP Multicasting BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Action 0x00. All rights reserved. All rights reserved. Cisco Public 68 © 2006. Inc. Seq 10. Vlan Access-List Switch#sh platform acl label 1 detail IPv4/MAC ACL label ------------------ Input Op Select Index 255: Output Op Select Index 255: Input Features: Interfaces or VLANs: Vl101 Vlan Map: FilterMap IP Access-lists: 123. 2 VMRs. All rights reserved.scr . Mask: 00000000 FFFFFFFF 00000000 00000000 00000000 Value: 00000000 0A640102 00000000 00000000 00000000 Result: 0x09 Mask: 00000000 00000000 00000000 00000000 00000000 Value: 00000000 00000000 00000000 00000000 00000000 Result: 0x00 <output removed> BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Presentation_ID.

scr . Cisco Public 70 © 2006. packet ingressed on this port ------------------------------------------ GigabitEthernet1/0/1 Packet 2 Lookup Key-Used Index-Hit A-Data OutptACL 50_EF646464_0A010101-00_40000000_0000A87E 01FFE 03000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/1 0100 0001.5e64.1.5e64. Layer 2 Mcast—IGMP Disabled ƒ Traffic is flooded inside the vlan to all forwarding ports 3750#sh plat for Gi1/0/2 1. Inc.1. All rights reserved.6464 ========================================== BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Presentation_ID. Inc.0001 0100.100.0001.1 239. Å deja vu check. Inc.100.1 0100.1. Cisco Public 69 IGMP Snooping Troubleshooting ƒ Verify the multicast router port is learned ƒ Verify that the join from the clients are received by the switch ƒ Verify that multicast traffic get’s forwarded as per the IGMP table BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.6464 ip 10.100 udp 0 0 Output Packets: ------------------------------------------ GigabitEthernet1/0/2 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 50_EF646464_0A010101-00_40000000_0000A87E 01FFE 03000000 Dropped due to failed deja vu check. All rights reserved. Cisco Systems. All rights reserved.

100.100.100.175: IGMPSN: router: Is not a router port on Vlan 101.100 on vlan 101.175: l2mcm_group_create: timer stop: vlan 101.101. on Vlan 101 *Mar 6 04:19:39. All rights reserved.075: IGMPSN: router: PIMV2 Hello packet received in 100 *Mar 1 03:33:44. Inc. on Vlan 101 *Mar 6 04:19:39.5e64. IGMP Multicast Router Port ƒ Gets learned dynamically by listening either to PIM/DVMRP or to CGMP packets ƒ Mrouter port should be learned dynamically 3750#sh ip igmp snooping mrouter vlan 100 Vlan ports ---.100.6464. All rights reserved.175: IGMPSN: Received IGMPv2 Report for group 239.100.175: IGMPSN: mgt: created gce 0100.1 IP_DA:224. Presentation_ID.469: Pak recvd on IGMP-SNOOP-Q: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1 DI:0x12FC.100. Cisco Public 71 IGMP Client Join IGMP Joins Received Are Sent to the CPU to Be Processed Switch#Debug ip igmp snooping group 239.6464 *Mar 6 04:19:39.5e64. MacSA: 0011. on Vlan 101 *Mar 6 04:19:39.175: IGMPSN: Can not Locate gce 0100. on Vlan 101 *Mar 6 04:19:39.175: IGMPSN: group: Skip client info adding .5e64.100. Inc.5e00. MAC address 0100.175: l2mcm_group_create: creating a group 239. LT:7.100.5e64.5a40 IP_SA:10. SrcGID:24.21e6.175: IGMPSN: MCAST IP address 239.100.100.175: IGMPSN: Can not Locate gce 0100.scr .0005. Cisco Systems. MacDA:0100.100.0.6464.075: IGMPSN: router: Learning port: Gi1/0/1 as rport on Vlan 100 debug platform cpu-queue igmp-snooping *Mar 1 03:39:09.100. Cisco Public 72 © 2006.100. ACLLogIdx:0x0.175: L2MM: setting Gi1/0/2 in gce->mbr_blist BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.ip 10.175: IGMPSN: MCAST IP address 239. port Gi1/0/2 *Mar 6 04:19:39.100. port Gi1/0/1 *Mar 1 03:33:44.075: IGMPSN: router: Received non igmp pak on Vlan 100. Vlan:100 SrcGPN:24.6464 add port Gi1/0/2 *Mar 6 04:19:39. on vlan 101 *Mar 6 04:19:39. port Gi1/0/1 *Mar 1 03:33:44.5e64.100.100.100.075: IGMPSN: router: Is a router port on Vlan 100.100 received on Vlan 101. Inc.6464. port_id Gi1/0/2.175: IGMPSN: mgt: created group 239. ----- 100 Gi1/0/1(dynamic) debug ip igmp snooping mrouter *Mar 1 03:33:44.0.5 IP_Proto:89 TPFFD:E841C018_00640064_00A0005E-000012FC_43330000_00000000 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. MAC address 0100.16.175: IGMPSN: mgt: Vlan 101 gce 0100. port Gi1/0/2 *Mar 6 04:19:39.100.100 *Mar 6 04:19:39.100 *Mar 6 04:19:39. All rights reserved.5e64.6464 *Mar 6 04:19:39.160.1. group 239. dummy NO *Mar 6 04:19:39.

L2EncapType 0. Inc. IP Multicast Routing ƒ Verify PIM is working fine (not covered in this session) ƒ Verify client is correctly joined via IGMP ƒ Verify the switch is routing the flow correctly Switch#sh ip mroute 239. Presentation_ID.6464 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.1fc2 0100. L3EncapType 0 <output removed> Output Packets: ------------------------------------------ GigabitEthernet1/0/2 Packet 1 Lookup Key-Used Index-Hit A-Data OutptACL 50_EF646464_0A630164-00_41000000_0000A87E 01FFE 03000000 Port Vlan SrcMac DstMac Cos Dscpv Gi1/0/2 0101 000f. flags: JT Incoming interface: Vlan100.99.100.100.1. Forward/Sparse-Dense.5e64.6464 ip 10.99. Inc.99. RPF nbr 10.scr . Cisco Public 74 © 2006. All rights reserved.100.100.100).100 239.100 udp 0 0 Ingress: Global Port Number: 1.f7e8.e042 0100. 239.100 Vlan Group Type Version Port List ----------------------------------------------------------------------- 101 239.1.100 igmp v2 Gi1/0/2 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.100. Inc.100.100.100.100 10.100. Mapped 9. lpn: 3 Asic Number: 1 Source Vlan Id: Real 100.1.5e64. 11:32:59/00:02:56. All rights reserved. Cisco Systems. All rights reserved.100.100.1 Outgoing interface list: Vlan101.1. Cisco Public 73 IP Multicast Routing ƒ The show forward command can be used to verify if the ASIC’s are setup correctly to route the multicast flow Switch#show platform forward Gig 1/0/1 vlan 100 18.100.100 IP Multicast Routing Table <output removed> (10. 11:32:59/00:02:22 Switch#sh ip igmp snooping groups vlan 101 239.ba88.

Cisco Public 75 Recommended Sessions: ƒ BRKRST-3142: Troubleshooting Catalyst 4500 Switches ƒ BRKRST-3143 Troubleshooting Catalyst 6500 Switches ƒ BRKRST-3131 Troubleshooting LAN Protocols ƒ BRKRST-3437 Catalyst 3750 Switch Architecture BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc. Starting from general switch health to advance features like QoS and access-list ƒ The show platform forward command has been used in many situations and is a powerful troubleshooting tool to diagnose issues quickly and effectively BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. All rights reserved.scr . Cisco Public 76 © 2006. Summary ƒ During this session we’ve shown various commands that should assist in diagnosing issues on these switches. Inc. All rights reserved. All rights reserved. Cisco Systems. Inc. Presentation_ID.

scr . Inc. All rights reserved. Inc. All rights reserved. All rights reserved. Q and A BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Cisco Systems. Cisco Public 77 Recommended Reading ƒ Continue your Cisco Live learning experience with further reading from Cisco Press ƒ Check the Recommended Reading flyer for suggested books Available Onsite at the Cisco Company Store BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Presentation_ID. Inc. Cisco Public 78 © 2006.

All rights reserved. Inc.scr .com. Solutions or visit www. All rights reserved. Inc. BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. on-demand and return for our live virtual event ƒ Complete your session evaluation online now in October 2008.cisco-live. (open a browser through our wireless network Go to the Collaboration to access our portal) or visit one of the Internet Zone in World of stations throughout the Convention Center. your Cisco Live virtual account for access to ƒ Receive 20 Passport points for each session all session material evaluation you complete. Complete Your Online Session Evaluation ƒ Give us your feedback and you could win Don’t forget to activate fabulous prizes. Inc. Winners announced daily. Cisco Systems. All rights reserved. Cisco Public 80 © 2006. Presentation_ID. Cisco Public 79 BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems.

try to find the offending device and remove it from the network ƒ http://www. excessive collisions is typically an indication that the load on the segment needs to be split across multiple segments but can also point to a duplex mismatch with the attached device.scr . some of these errors may occur. this can be caused by duplex mismatch and physical problems like a bad cable. NIC card. NIC card.…) but can also indicate a duplex mismatch ƒ Align-Err is the number of frames with alignment errors (frames that do not end with an even number of octets and have a bad CRC) received on the port. collisions should not be seen on ports configured as full duplex ƒ Carri-Sen (Carrier Sense) occurs every time an Ethernet controller wants to send data on a half duplex connection. when the cable is first connected to the port. Presentation_ID. bad port. for the duplex mismatch scenario the late collision would be seen on the half duplex side. the controller senses the wire and check if it is not busy before transmitting. if collisions are increasing dramatically this points to a highly utilized link or possibly a duplex mismatch with the attached device ƒ Multi-Coll (Multiple Collision) is the number of times multiple collisions occurred before the port transmitted a frame to the media successfully. Inc. collisions are normal for port configured as half duplex but should not be seen on full duplex ports.html#show_interface BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. Inc. the full duplex side does not wait its turn and transmits simultaneously causing a late collision. five hundred and twelve bit-times corresponds to 51. All rights reserved.2 microseconds on a 10 Mbit/s system. port. bad port. but including FCS octets) that were otherwise well formed. check the device sending out these frames ƒ Runts are frames received that are smaller than the minimum IEEE 802. an excessive collision happens when a packet has a collision 16 times in a row. Cisco Systems. if there is a hub connected to the port then collisions between other devices on the hub may cause these errors ƒ Late-Coll (Late Collisions) is the number of times that a collision is detected on a particular port late in the transmission process.3 frame size of 64bytes long (excluding framing bits.3 frame size (1518 bytes for non-jumbo Ethernet). late collisions can also indicate an Ethernet cable/segment that is too long. as the half duplex side is transmitting. Appendix A: Error Counters ƒ FCS-Err is the number of valid size frames with FCS (Frame Check Sequence) errors but no framing errors: this is typically a physical issue (cabling.com/warp/public/473/164.cisco. All rights reserved. Cisco Public 82 © 2006. Inc. these usually indicate a physical problem (cabling. All rights reserved. for a 10mbit/s port this is later than 512 bit-times into the transmission of a packet. if collisions are increasing dramatically this points to a highly utilized link or possibly a duplex mismatch with the attached device BRKRST-3141 14493_04_2008_c2 © 2008 Cisco Systems. also. Cisco Public 81 Appendix A: Error Counters(2) ƒ Excess-Coll (Excessive Collisions) is a count of frames for which transmission on a particular port fails due to excessive collisions.…) but can also indicate a duplex mismatch.3 frame size (64 bytes for Ethernet) and with a bad CRC. collisions are normal for port configured as half duplex but should not be seen on full duplex ports. or NIC card on the attached device ƒ Giants exceed the maximum IEEE 802. this error can indicate a duplex mismatch among other things. this is normal on an half-duplex Ethernet segment ƒ Undersize are frames received that are smaller than the minimum IEEE 802. collisions should not be seen on ports configured as full duplex ƒ Single-Coll (Single Collision) is the number of times one collision occurred before the port transmitted a frame to the media successfully. the packet is then dropped.