-y9r2020 ‘port-:0-0: blocked FC remote port time out saving binding - Red Hat Customer Postal 2 cusTomen httpsi//access.redhat.com/) 2 PORTAL rport-1:0-O: blocked FC remote port time out: saving binding @ SOLUTION VERIFIED - Updated January 19 2016 at 8:07 PM - English > () Environment * Red Hat Enterprise Linux(RHEL) 7 » Red Hat Enterprise Linux(RHEL) 6 * Red Hat Enterprise Linux(RHEL) 5 Issue + System /var/log/messages , /var/log/dmesg , or vcore contain the following logged events indicating FC path lost between host’s HBA and storage: {RHEL6,7} Jun 5 23:46:53 HOST kernel: rport-1:0-0: blocked FC remote port time out: removing target and saving binding Jun 5 23:46:53 HOST kernel: rport-1:0-1: target and saving binding blocked FC renote port time out: removing {RHELS) Jun § 23:46:53 HOST kernel: rport-1:0-@: blocked FC remote port time out: saving binding Jun 5 23:46:53 HOST kernel: rport-1:@-1: blocked FC remote port time out: saving binding sor if option fc_remove_on_dev_loss set- Jun § 23:46:53 HOST kernel: rport-1:0-0: target and saving binding Jun § 23:46:53 HOST kernel: rport-1:0-1: target and saving binding blocked FC renote port time out: removing blocked FC renote port time out: renoving * Device mapper multipath paths going down and multipath -11 shows the path down hitpsiaccess.tedhat.convsohutonsi58169 18-y9r2020 ‘port-:0-0: locked FC remote port time out saving binding - Red Hat Customer Portal ie rene ezocenecmoasendotins tice apsonaot com/) 2 i2e=306] [FeatR@RTAdueve_if_no_path]|hwhandler=@] [rw] round-robin @ [prio=2][active] @:1 sdb 8:16 [active] [ready] 30:1 sdd 8:48 [Failed] [ready] v vu 3: + Found following messages logged in /var/1og/messages , what could be possible cause and solution of this issue: Jan 20 15:54:28 HOST kernel: qla2xxx @200:0f:02.1: LOOP DOWN detected (2 3 0). Jan 2@ 15:54:36 HOST kerne: binding Jan 28 15:54:36 HOST kernel: sd 3:0:0:1: SCSI error: return code = @x00010000 Jan 20 15:54:36 HOST kernel: end_request: 1/0 error, dev sdd, sector 593496 Jan 20 15:54:36 HOST kernel: device-mapper: multipath: Failing path 8:48. pport-3:0-@: blocked FC remote port time out: saving + Extd filesystem mount goes read-only and cause system hang, observed blocked FC remote port time out: saving binding, in messages. * Server kernel panic and vcore contain the blocked FC remote port error message, what does cause this issue? Resolution + "Always’: When these messages are occurring persistently against specific ports, check zoning with storage administrator. + “Sporadic”: When observed to be occurring intermittently, check with storage administrator with regards to congestion or other health issues with the storage configuration coinciding with the times of the logged events. + "Started suddenly’: If this started suddenly on a previously working configuration and it's persistent, check SAN/storage side hardware for errors. Engage SAN vendor support to diagnose the issue further. Root Cause These messages mean that the remote Fibre Channel storage port could not be contacted so the server could not access the devices behind these ports. As such, there are underlying hardware issues as root cause. The messages are usually accompanied by other io errors and recovery actions, for example: hitpsiaccess.tedhat.convsohutonsi58169 28‘97020 ‘port-:0-0: blocked FC remote port time out saving binding - Red Hat Customer Postal Jan quae? YosT GAME ME RGOEES// acon badhaGAD/ waited 30s Jan 18 17:00:32 HOST kerneM® sd 5:0:0:0: tRMLRFALE command, waited 305 Jan 18 17:¢1:32 HOST kernel: qla2%«x 000: 1a:00.0: scsi( qla2wex 0000:12:00. kernel: qla2xxx 2000: 17:01:33 HOST kere :@): Abort command issued -- 1 56438 2002. Jan 18 17:01:32 HOST kernel: scsi(5:5:8): Abort command issued -- 1 564389 202. Jan 18 17:01:33 HOST 108.0: scsi(5:5:@): Abort conmand issued -- 1 56438b 2002. Jan 18 sd 5:0:5:0: timing out conmand, waited 38 Jan 18 17:01:33 HOST kernel: sd 5:0:5:0: timing out command, waited 3s Jan 18 17:03:03 HOST kernel: qla2xxx 0000:1a:00.2: scsi(5:5:0): Abort command issued -- 1 5643c@ 2002. Jan 18 17:03:43 HOST kernel: rport-5:0-5: blocked FC renote port time out: saving binding Jan 18 17:03:43 HOST kernel: qla2xxx 2000:10:00.0: scsi(5:5:0): DEVICE RESET ISSUED. Jan 18 17:03:54 HOST kernel: qla2xxx 000:10:00.0: qla2xxx_eh_device_reset: device reset failed Jan 18 17:03:54 HOST kernel qla2xxx @000:12:00.0: scsi(5:5:): LOOP RESET ISSUED. Jan 18 17:@5:17 HOST kernel: qla2xxx 2000:10:00.0: qla2xxx_eh_bus_reset: reset succeeded Jan 18 17:06:52 HOST kernel: sd 5:0:5:0: SCSI error: return code = x0001290 Jan 18 17:26:50 HOST kernel: end_request: 1/0 error, dev sdd, sector 260496 Jan 18 17:06:50 HOST kernel: sd SCSI error: return code = @x99010000 Jan 18 17:06:50 HOST kernel: end_request: I/ error, dev sdd, sector 1463842996 NOTE!: When a port leaves the current running SAN configuration, there are several different types of messages that can be logged depending on kernel configuration as well as the state of the target port at the time it was no longer accessible: + blocked FC remote port time out: removing target and saving binding (https://access.redhat.com/site/solutions/58169) + blocked FC remote port time out: saving binding (https://access.redhat.com/site/solutions/58169) RHELS only (1] * blocked FC remote port time out: removing rport, (https://access.redhat.com/site/solutions/65596) RHEL6,7 only [2] + blocked FC remote port time out: removing target (https://access.redhat.com/site/solutions/65596) RHELS only [2] + blocked FC remote port time out: no longer a FCP target, removing starget + blocked FC remote port time out: leaving rport alone RHEL6,7 ONLY [2] * blocked FC remote port time out: leaving target alone RHELS ONLY [2] While "removing rport’, for example, is done when the SAN port initiated contact to the host and then logged out (as some SAN appliances do) then the target port was never associated with storage LUNs. In this case the kernel is removing the target and reclaiming all of the local rport resources it allocated for that port. All traces of connectivity to that port will cease to exist within the kernel However, in the more common case where the host logs into a remote SAN port, finds it to be an FCP-SCSI storage port, and discovers LUNs behind that port, then a local target number, as part of the typical SCSI address host: bus: target: Lun, is assigned. When the associated remote port leaves the SAN after getting one of these binding target addresses, the kernel will save the target hitpsiaccess.tedhat.convsohutonsi58169 a8-y9r2020 ‘port-:0-0: blocked FC remote port time out saving binding - Red Hat Customer Portal binding information for if/when the remote port returns tp the SAN configuration. This wl allow CUSTOMER (hltpsy/Accessredhat.com, returrfibcevices to comigwente have the Same host :bus: target :1un SCSI address. In this cas the "removing target and saving binding" message is output. This doesn’t change the cause of why the remote port is not longer accessible on the SAN, just that if or when it returns devices will continue to have the same SCSI addresses and sdN names Footnotes: [1] this is default message on RHELS, only if the option fc_remove_on_dev_loss is set will the alternative message be ‘removing target and saving bindin. found in /sys/module/sesi_transport_fe/parameters/remove_on_dev_loss. That option is not available within RHELS. The current set value can be [2] message has “rport” in RHEL6-7, and “target” in RHELS. Terminology changed to more accurately reflect remote port classification. Product(s) Red Hat Enterprise Linux (/taxonomy/products/red-hat-enterprise-linux) Component device-mapper (/components/device-mapper) kernel (/components/kernel) Category Troubleshoot (/category/troubleshoot) Tags device mapper (/tags/device mapper) eme (/tags/eme) fibre. channel (/tags/ibrechannel) file_systems (/tags/file_systems) hang (/tags/hang) ba (/tags/hba) _io_error(/tags/ioerror) multipath (/tags/muttipath) panic /tags/panie) logic (/tags/alogic) read-only (/tags/read-only) rhel.S (/tags/rhel.S) rhel_6 (/tags/rhel_6) san (/tags/san) scsi /tags/scsi) storage (/tags/storage) This solution is part of Red Hat's fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form People who viewed this solution also viewed /Nat/log/messages reports "blocked FC remote port time out: removing rport” or hitpsiaccess.tedhat.convsohutonsi58169 48-y9r2020 ‘port-:0-0: blocked FC remote port time out saving binding - Red Hat Customer Postal 4 terest" CUSTOMER (https://access.redhat.com/) ei stot ota 2 Solution - Mar 8, 2019 Dual port FC card is seen in RHELZ, but driver is not bringing the FC ports UP Solution - Sep 11, 2018 RHEL: kernel crashes when disabled FC port is reenabled with Emulex Ipfc driver Solution - Mar 1, 2016 1 Comments Tr " 2 28 October 2013 7:08 AM (https://access.redhat.com/solutions/58169#comment-564423) “TT I (/user/3123033m also getting same error MEMBER "Oct 20 04:03:03 qalermrac02 syslogd 1.4.1 restart. 42Points Oct 22 00:01:20 galermracO2 kernel: st!: Failed to read 524288 byte block with 32768 byte transfer. Oct 23 00:08:36 qalermracO2 kernel: st3: Failed to read 524288 byte block with 32768 byte transfer. Oct 23 14:35:04 qalermrac02 auditd[16662]: Audit daemon rotating log files Oct 26 09:16:40 galermrac02 kernel: qla2xxx 0000:06:00.0: scsi(0:5:18): Abort command issued -- 1 56e6e218e 2002. Oct 26 09:16:40 galcrmracO2 kernel: qla2xxx 0000:06:00.0: scsi(0:4:20): Abort command issued -- 1 56e6e218d 2002. Oct 26 17:31:40 qalcrmracO2 kernel: qla2xxx 0000:06:00.0: scsi(0:2:0): Abort command issued -- 1570d6eda7 2002. Oct 26 17:31:50 qalermrac02 kernel: qla2xxx 0000:06:00.0: sesi(0:2:0): Abort hitpsiaccess.tedhat.convsohutonsi58169 98 Thillai Thayanidhi (/user/312303)-y9r2020 a Ipore.0-0 blocked FC remote pot be out saving binding - Red Hat Customer Poca command soso (SS IBM ARC ednatcomy) Ser 25 7PSPygatiermracd2 Kernel qladxxx 6006.06:00.0: scsi(0:3:17): Abort command issued -- 1570d6a36e 2002. Oct 26 17:32:00 qalermrac02 kernel: qla2xxx 0000:06:00.0: sesi(0:3:17): Abort command issued -- 1570d6a36e 2002. Oct 26 17:32:00 qalermracO2 kernel: qla2xxx 0000:06:00.0: sesi(0:2:17): Abort command issued -- 1570d6eda3 2002 Oct 26 17:32:10 qalermrac02 kernel: qla2xxx 0000:06:00.0: scsi(0:2:17): Abort command issued -- 1570d6eda3 2002. Oct 26 17:32:10 qalermrac02 kernel: qla2xxx 0000:06:00.0: scsi(0:2:16): Abort command issued -- 1570d6eda4 2002 Oct 26 17:32:20 gatermrac02 kernel: gla2xxx 0000:06:00.0: sesi(0:2:16): Abort command issued -- 1570d6eda4 2002 Oct 26 17:32:20 qalermracO2 kernel: gla2xxx 0000:06:00.0: sesi(0:4:10): Abort command issued -- 1570d6eda5 2002. Oct 26 17:32:37 qalermracO2 kernel: rport-0:0-2: blocked FC remote port time out: saving binding Oct 26 17:32:41 galermracO2 kernel: qla2xxx 0000:06:00.0: scsi(0:4:10): Abort command issued -- 1570d6eda5 2002. Oct 26 17:32:41 galermracO2 kernel: qla2xxx 0000:06:00.0: scsi(0:5:19): Abort command issued -- 1570d6eda6 2002 Oct 26 17:33:21 qalcrmracO2 kernel: qla2xxx 0000:06:00.0: scsi(0:5:19): Abort command issued -- 0 570d6eda6 2003. Oct 26 17:33:21 qalcrmracO2 kernel: qla2xxx 0000:06:00.0: scsi(0:5:19): DEVICE RESET ISSUED. Oct 26 17:33:22 galcrmracO2 kernel: qla2xxx 0000:06:00.0: qla2xxx_eh_device_reset: device reset failed Oct 26 17:33:22 galcrmracO2 kernel: qla2xxx 0000:06:00.0: scsi(0:5:19): LOOP RESET ISSUED. Oct 26 17:34:02 galcrmracO2 kernel: INFO: task MpxPeriodicCall:13954 blocked for more than 120 seconds. Oct 26 17:34:02 qalermrac02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 26 17:34:02 galcrmracO2 kernel: MpxPeriodicCa D ffffffff801546d1 0 139541 13955 13953 (L-TLB) Oct 26 17:34:02 qalermrac0? kernel: ff##810c039af9d0 0000000000000046, FfFf810c09048418 ffff81074ed57e40 Oct 26 17:34:02 qalermrac02 kernel: fff#810c090484f8 000000000000000a FfFFBIObFfd6f040 #ffF81057803b820 Oct 26 17:34:02 qalermrac02 kernel: 002b5att39665d9a 0000000000011597 hitpsiaccess.tedhat.convsohutonsi58169 8-y9r2020 a (Dor 10-0; Blocked FC remote por tne ou saving bing Red Hol Customer Pol TsO GOR RGR OOP ES BEE com) Set 25 17:44 G4 galermracd? kernel: Call Trace Q Oct 26 17:34:02 qalermrac02 kernel: [] elv_next_request+Ox168/Ox178 Oct 26 17:34:02 qalermrac02 kernel: [] wait_for_completion+0x79/Oxa2 Oct 26 17:34:02 galermrac0? kernel: [] default_wake_function+Ox0/Oxe Oct 26 17:34:02 qatermrac02 kernel: [] blk_execute_rq_nowait*Ox7e/Ox92 Oct 26 17:34:02 qalermrac02 kernel: [] blk_execute_rq*0x98/OxcO Oct 26 17:34:02 qalermrac02 kernel: [] blk_recount_segments+Ox17/0x28 Oct 26 17:34:02 qatermrac02 kernel: [] emep:emep_sesi_emd_joctl*Ox2ad/0x400 Oct 26 17:34:02 qalermrac02 kernel: [] emep:PowerPlatformBottomDispatch+Ox461/0x550 Oct 26 17:34:02 qalermrac02 kernel: [] ‘emep:PowerSyncloBottomDispatch+0x7a/Oxd0 Oct 26 17:34:02 qalermrac02 kernel: [] emcp:PowerDispatchX+0x353/0x410 Oct 26 17:34:02 qatermrac02 kernel: [] emep:Emsinquiry+0x9d/Oxla0 Oct 26 17:34:02 qalermrac02 kernel: [] ‘emepmpx-ClariionKLam_getPathLunStatus+0x8a/Oxi10 Oct 26 17:34:02 qalermrac02 kernel: [] emcpmpx:MpxDefaultTestPath+Ox21/0x80 Oct 26 17:34:02 qalermrac02 kernel: [] emcpmpx:MpxLnxTestPath+Ox31/0x270 Oct 26 17:34:02 qalermrac02 kernel: [] emcpmpx:MpxTestPath+0x244/OxIfSO Oct 26 17:34:02 qalermrac02 kernel: [] thread_retun+0x62/Oxfe Oct 26 17:34:02 qalermrac0? kernel: [] emepmpx:MpxPeriodicTestPath+Ox9b/0x230 Oct 26 17:34:02 qalermrac02 kernel: [] emep:PewerTimeout+Oxla4/OxleO Oct 26 17:34:02 galermrac0? kernel: [] do_gettimeofday+0x40/0x90 Oct 26 17:34:02 qatermrac02 kernel: [] emcpmpx:MpxPeriodicCallout+0x2f2/Ox410 Oct 26 17:34:02 qalermrac02 kernel: [] emcpmpx:MpxPeriodicCallout+Ox0/0x410 Oct 26 17:34:02 qalermrac02 kernel: [] emcp:PowerServiceDaemon@+Oxad/Oxd0 Oct 26 17:34:02 galermrac02 kernel: [] child_rip+Oxa/Oxtl Oct 26 17:34:02 qatermrac02 kernel: [] emep:PowerDaemonStart+0x0/0x20 Oct 26 17:34:02 qalermrac02 kernel: [] child_rip+Ox0/OxIl Oct 26 17:34:02 qatermrac02 kernel Oct 26 17:34:07 qatermrac02 kernel: qla2xxx 0000:06:00.0: LOOP DOWN detected (4 000). Oct 26 17:34:07 qalermrac02 kernel: qla2xxx 0000:06:00.0: LOOP UP detected (8 Gbps). Oct 26 17:34:09 qalermrac0? kernel: gla2xxx 0000:06:00.0: qla2x0x_eh_bus_reset: reset succeeded ‘% Reply (/Ajax_comments/Reply/58169/564423) hitpsiaccess.tedhat.convsohutonsi58169 78-y9r2020 ‘port-:0-0: blocked FC remote port time out saving binding - Red Hat Customer Postal 2 cusTomen httpsi//access.redhat.com/) PORTAL All systems operational _(https://status.redhat.com) Privacy Statement (httpi//mww.rechat.com/en/about/privacy- policy) Customer Portal Terms of Use (https//access.rechat.com/help/terms/) All Policies and Guidelines (httpy/www.redhat.com/en/about/all-policies-guidelines) Copyright © 2020 Red Hat, Inc. hitpsiaccess.tedhat.convsohutonsi58169 88