You are on page 1of 8

[Brocade]SAN Troubleshooting(11) - Common SAN Issue Part5 - Mar...

http://windowspeople.com/brocade/brocade-san-troubleshooting-11-...

[Brocade]SAN Troubleshooting(11) - Common
SAN Issue Part5 - Marginal Links
Marginal Links caused by physical connectivity includes physically hardware issue, transmissión of light/signal,
negotiation of speed and synchronization of Transmission characters and words.

Physical Components
Marginal Links involves the connection between the switch and the edge device. Isolating the exact cause of a marginal
link involves analyzing and testing many of the components that make up the link.
1. SwitchShow

Use switch to confirm switch is online, SFP installed.
2. portErrShow
check portErrShow to see if there is high number of errors on CRC or ENC_OUT, You should always reset(clear) port
counters to assist the problem determination:
Clear port counters using portStatsClear. you can now clear the stats by slot, if Error counters increasing in a short period
of time, could indicate issue.
To clear hardware statistics for all ports on slots 3-5:
switch:admin> portstatsclear -s 3-5
To clear hardware statistics for all ports on slots 3-5 and 7-10:
switch:admin> portstatsclear -s 3-5 7-10
here is sample portErrShow output,
DCX01:admin> porterrshow
frames
tx

rx

enc
in

crc
err

crc

too

too

g_eof

shrt

long

bad
eof

enc

disc

out

c3

link
fail

loss

loss

sync

sig

frjt

fbsy

=========================================================================================================
0:

336.0m

1.8g

0

0

0

0

0

0

0

0

0

0

0

0

0

1:

898.0m 116.9m

0

0

0

0

0

0

0

0

0

2

2

0

0

0

0

0

0

0

0

0

0

0

2

2

0

0

2:

38.6m

60.2m

3:

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

4:

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

5:

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

6:

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

(output truncated)

porterrshow can be divided in 3 sections

Section1 - Encoding Statistics
enc_in - Encoding errors inside frames
1 of 8

02-01-2017 14:02

In isolation. or a bad or damaged device in the path. it means that when this counter gets incremented. b). look at the crc_err and crc_g_eof counters: a). However. . indicate electrical signal is lost on a link. patch panel. it points to the traffic coming from the remote end .Mar. further investigate with fabriclog -s output Note:One of the common cause is related to is ISL over-subscription..The crc_g_eof counter is used to locate the source of CRC problems. below situation could cause disc_c3 error. since the first port marked the frame with a bad EOF. If you notice the crc_err counter go up.it is not routable to a destination address b).com/brocade/brocade-san-troubleshooting-11-.The enc_out are errors that occur outside the frame and usually indicating a bad primitive. Due to this. and other devices. . To find the source of CRC errors. indicate a bit/word synchronization on a link is lost. Because switch is cut-through. which means that the switch is holding onto the frame longer than the hold time allows. Locking a port to the speed supported by the end device can be used to isolate enc_out issues.. The crc_err counter are frames with CRC errors. so the crc_err counter will increment on these ports. indicate that the link to that port was the source of the problem.The disc_c3 errors are discarded class 3 errors. The link associated with the port with an increasing crc_g_eof counter is the marginal link and the source of the errors.Detected on the ingress port. CRC errors from a single device maybe seen on multiple ports in a fabric. Check the SFP by swapping it with a known good working SFP. If these are all verified to be good. Incremented during speed negociation and link transitions prior to login. Look for marginal SFPs on each side of the link and problems in the optical pathway between the SFPs.Also could caused by FC node or Switch port status change(online/offline)continuously. you have a bad or damaged cable. it means that a systems is trying to send frames through a channel with errors.. the problem could be a failing switch port. a).[Brocade]SAN Troubleshooting(11) . It represents frames with CRC errors and a good EOF marker. It does not. Subsequent ports in the path will also detect the CRC error.crc_err means frames with CRC errors which Indicates corruption of byte(s) within the data frame. frjt: Frames rejected with F_RJT fbsy: Frames rejected with F_BSY link_fail: Link failures (LF1 or LF2 states) 2 of 8 02-01-2017 14:02 .Incremented when 8b/10b encoding errors are detected within a frame (2112 transmission characters) .Traffic is only impacted when enc_in gets incremented. then the physical path should be inspected.Frame Statistics . Check the cables to and from the switch. by itself. crc_g_eof on the subsequent ports will not increment. This means that it was the first port detecting the CRC error.Timeout exceeded in the ASIC (500 ms) d). http://windowspeople. so that the destination can proactively request retransmission.Signal/Sync/Link Statistics loss_sig: Loss of signal. If this counter goes up. The crc_err counter represents frames with CRC errors. Section3 . this usually indicates that a problem exists upstream of that link. Section2 .Destination ID (DID) is not online c). This simply means that a port received a frame with a bad CRC.. CRC is checked and validated as frames leaves the switch.Common SAN Issue Part5 . loss_sync: Loss of synchronization. It will mark the frame with a “bad” EOF and pass the frame on to its destination. .

portdisable/portenable).. Link Failure state consists of: LF1 => No operational Receive LF2 => No operational Transmit To conclude. path panel).[Brocade]SAN Troubleshooting(11) . As for loss_sync. The link failure also indicates that loss of signal or loss of sync lasting longer than the R_T_TOV value was detected while not in the Offline state. power cycles. disconnections. a Link Reset Protocol Timeout will be detected that results in a Link Failure condition and the port enters the NOS (Not Operational State) Transmit State. 100ms per default)..also verify the SFP is not faulty .com/brocade/brocade-san-troubleshooting-11-. Note that If the switch running on FOS7. http://windowspeople. verify that the link was not brought down manually after setting the baseline. it means a possible bad media (SFP. cable. Loss of Signal and Enc_out are errors expected every time a user brings a port up or down (reboots.and Tx/Rx Power is inside recommended values according to SFP datasheets.x..Common SAN Issue Part5 . sfpshow Use sfpshow to check .. sfpshow –all sfpshow <port> 3 of 8 02-01-2017 14:02 .Mar. if found enc_in and/or crc_err. 3. .wavelength and speed. Loss of Sync. you can also use sfpshow -health. make sure Optical Modules (GBIC/SFP /SFP+/XFP) must match cables. portflagsshow Use portflagsshow to check if LED is seen SAN01:FID128:admin> portflagsshow Slot Port SNMP Physical Flags -----------------------------------1 0 Online In_Sync PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI 1 1 Online In_Sync PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI 1 2 Online In_Sync PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT 1 3 Online In_Sync PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT 1 4 Online In_Sync PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI 1 5 Online In_Sync PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI 4. If a Port remains in the LR Receive State for a period of time greater than R_T_TOV (Receiver-Transmitter Timeout Value.

0 uW uW SAN01:FID128:admin> 4 of 8 02-01-2017 14:02 .Tx_Disable BR Max: 0 BR Min: 0 Serial No: UYA109491157932 Date Code: 091203 DD Type: 0x68 (units nm) Enh Options: 0xf0 Status/Ctrl: 0x90 Alarm flags[0.0 uW TX Power: -2.400.400. SAN01:FID128:admin> sfpshow Slot 1/Port 0: id (sw) Vendor: BROCADE Serial No: UYA109491147982 Speed: 200.400.500 2.800_MB/s Slot 1/Port 1: id (sw) Vendor: BROCADE Serial No: UYA109491157932 Speed: 200.400.800_MB/s M5.0 100. 0x0 Alarm low Warn high low high Temperature: 35 Centigrade -15 90 -10 85 Current: 6.400.800_MB/s Slot 1/Port 3: id (sw) Vendor: BROCADE Serial No: UYA109531208472 Speed: 200.0 dBm (506.[Brocade]SAN Troubleshooting(11) .M6 sw Inter_dist Encoding: 1 8B10B Baud Rate: 85 (units 100 megabaud) Length 9u: 0 (units km) Length 9u: 0 (units 100 meters) Length 50u: 5 (units 10 meters) Length 62.400.0 uW 1100.800_MB/s SAN01:FID128:admin> sfpshow 1/1 Identifier: 3 SFP Connector: 7 LC Transceiver: 540c402000000000 200.1] = 0x0. 0x0 Warn Flags[0.7 uW) 50.0 uW 800.0 3800.0 3630.Tx_Fault.0 RX Power: -3.800_MB/s Slot 1/Port 2: id (sw) Vendor: BROCADE Serial No: UYA109501179322 Speed: 200.0 uW 49.com/brocade/brocade-san-troubleshooting-11-.Common SAN Issue Part5 ..1] = 0x0.0 uW 6550.000 8.800_MB/s Slot 1/Port 5: id (sw) Vendor: BROCADE Serial No: UYA109521204452 Speed: 200.832 mAmps 2. http://windowspeople.5u:3 (units 10 meters) Length Cu: (units 1 meter) 0 Vendor Name: BROCADE Vendor OUI: 00:05:33 Vendor PN: 57-1000046-01 Vendor Rev: A Wavelength: 850 Options: 001a Loss_of_Sig.0 uW) 0..Mar.000 8.0 uW 700..800_MB/s Slot 1/Port 4: id (sw) Vendor: BROCADE Serial No: UYA109531213772 Speed: 200..6 dBm (546.400.500 Voltage: 3294.2 mVolts 2800.0 2970.

See below sample of output 5 of 8 02-01-2017 14:02 .. Targets and local ISLs .. It should be enforced at the switch port.Mar.Common SAN Issue Part5 ..Targets port speeds must be manually set to their own speed from the device point of view.Auto Negotiate for Initiators. http://windowspeople.[Brocade]SAN Troubleshooting(11) .com/brocade/brocade-san-troubleshooting-11-. . Port Parameters use portcfgshow to verify all port parameters .If Targets port speed couldn't set to manual..Manually set port speeds for long distance ISLs . Port Speed & Synchronization Brocade best practice for the port speed configuration: .

. .. . . Locked G_Port ... .... . . . .. ...... .. . .. . . ..... ...... . ... . .... Locked L_Port .. . . .. .... . ... Fault Delay 0 0 0 Ports of Slot 4 16 0 0 17 0 18 0 19 0 0 20 0 21 0 22 0 23 0 0 24 0 25 26 27 0 28 29 30 31 -----------------+---+---+---+---+-----+---+---+---+-----+---+---+---+-----+---+---+--Fill Word 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 AL_PA Offset 13 ... .. .. NPIV capability ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON NPIV PP Limit 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 QOS E_Port AE AE AE AE AE AE AE AE AE AE AE AE ... . . ... .. . ... output truncated. .... .. . . . .. . . .. . ... . .. ... . ..... .. .. .... . .. . . .. . CSCTL mode .. . . .. . .. ...Common SAN Issue Part5 . .. . .... . ..... .. ... . . . .. .... .. ... VC Link Init .... . Credit Recovery ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON Fport Buffers .. ... .. . . . Trunk Port ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON Long Distance .. . . . . .... .. . Disabled E_Port ..... ISL R_RDY Mode ... . . . . . .. ... ....... .... Trunk Port ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON Long Distance .... . .... . . ... .. . .. . ... . . . . . . .... . . .. . ... .... . .. . . . . ...... . .… Ports of Slot 4 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -----------------+---+---+---+---+-----+---+---+---+-----+---+---+---+-----+---+---+--Speed AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN Fill Word 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 AL_PA Offset 13 . .. . ...... . . . ...... . . .... . .. . .. . ..... . . . . .. ... .. .. .. . . . . .. Persistent Disable ... .. RSCN Suppressed . .. . ..... . .. .. .. . . . ..Mar. ... . . ... . .. . .... LOS TOV enable . ..... . .. . . VC Link Init . .... .... .. . .. .. . ..... ... . . .. . ... . .... . .. .... ... .. . . . . .. . . .. ... . . .... . . . . Rate Limit .... .com/brocade/brocade-san-troubleshooting-11-.... . EX Port . .. . ... 6 of 8 02-01-2017 14:02 ..... . . . .. . .... ...... .... .... ..[Brocade]SAN Troubleshooting(11) . .. .. . ... . . . . . ISL R_RDY Mode . RSCN Suppressed . . .... ... . .. . .. .. . . ... . . .. . . . .. . ..... .... .. . . .... ..... . . .... Locked G_Port .. . .. Locked E_Port .... ... . . . . . . Mirror Port ..... . . . . .. . . .... Disabled E_Port .. . ... ... .. SAN01:FID128:admin> portcfgshow . . .... . .. ... Locked E_Port .. . . . . .... . .... ... .... .. .... . . . .. ........ . . .. .. . http://windowspeople.. ... ..... .. .. . . .. . . .. . ...... . ... .. . .. . .. . .. .... .. .. . .. .. ... .... .. . ... . . . ... .. ... .. .. . . .. . . .... .. . . ...... .. . .. . .. . ..... .. ... .. . . Locked L_Port .. Port Auto Disable ... . .. . . ... ....

... . . .. NA:NotApplicable.. Port Auto Disable . ... ??:INVALID.... Displays (. LOS TOV enable .. . .. . . .5 . . otherwise. . .... . . . .... . Displays (. This value is set by the portcfglongdistance command.... Port Type (E_Port portcfgeport disabled) Disabled E_Port – Displays ON when the port is not allowed to be an E_Port.....) or OFF when L_Port lock mode is disabled and the port behaves as a U_Port). . http://windowspeople. .. . (.. ....com/brocade/brocade-san-troubleshooting-11-. ... Fport Buffers .. . output truncated. .. displays long distance levels as shown below. .) or OFF when the port is allowed to func on as an E_Port.. .. LM:L0..) or OFF when G_Port lock mode is disabled and the port behaves as a U_Port....… See below spreadsheet on the command to use to make changes to the port parameters: Command Comments portcfgspeed Speed – Displays AN for auto speed nego a on mode... .. . .. . .[Brocade]SAN Troubleshooting(11) .. . ... • LE – The link is up to 10 km • LD – The distance is determined dynamically • LS – The distance is determined sta cally by user input 02-01-2017 14:02 ..:OFF.. . .. Persistent Disable .... ... ...... . Reset to Defaults portcfgdefault Port Type (L_Port) portcfglport Locked L_Port – Displays ON when the port is locked to L_Port only....Common SAN Issue Part5 . .... or 8 Parameter Port Speed Gbits/sec. This value is set by the portcfggport command. .. .... . .. . . .. Long Distance and VC Link Init 7 of 8 portcfglongdistance Long Distance – Displays (.. .. . .. . This value is set by the portcfgspeed command. . . EX Port . . .. . ... . . Port Type (E or portcfggport F_Port only) Locked G Port Displays ON when the port is locked to G Port only Displays ( ) TS300 End Device Connec vity G_– G_only. .. . . ... .. . . .. 4.. . .. . 2. or a specific speed of 1... NPIV capability ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON NPIV PP Limit 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 QOS E_Port . This value is set by the portcfgeport command..... AN:AutoNegotiate. . . . .. . Fault Delay 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 where AE:QoSAutoEnable.. . . .. . . .) or OFF when long distance mode is off.. .... .. .... . ... .Mar. . . This value is set by the portcfglport command.. .. ...

Displays (.) or OFF when ISL R_RDY mode is disabled..Common SAN Issue Part5 . This value is set by the portcfgislmode command.Mar.. By default... ISL R_RDY Mode portcfgislmode http://windowspeople..[Brocade]SAN Troubleshooting(11) .com/brocade/brocade-san-troubleshooting-11-. ISL R_RDY Mode – Displays ON when ISL R_RDY mode is enabled on the port.) or OFF when disabled.. Displays (. This value is set by the portcfgcreditrecovery command. The credit recovery feature is enabled by default.) or OFF when QoS is disabled. but only ports configured as long distance ports can u lize this feature.. Credit Recovery 8 of 8 portcfgcreditrecovery Credit Recovery – Displays ON when Credit Recovery is enabled on the port or (. This value is set by the portcfgqos command. 02-01-2017 14:02 . QoS is enabled by best effort based on availability of buffers. QoS portcfgqos QOS E_Port – Displays ON when Quality of Service (QoS) is enabled on the port.