Professional Documents
Culture Documents
Today I am going to explain different issues faced during IOS upgradation & workaround to resolve
the issue.
General Issue:
If post upgradation or for any other reason, any RSP goes to unknown state, then first of all ask
FE to take a clear snap shot of LED status
JOJI the RSP & observe if it is coming online
Ask FE to connect the console cable at problematic RSP & check whether it is stuck to Rommon>
mode
If it is stuck in rommon mode then try to boot the RSP from bootflash or usb drive after copying
the file from the active RSP(in case of USB)
If standby RSP is not coming online or reloading continuously even after multiple RSP insertion &
stuck in standby cold state followed by continuous booting then follow below steps:
sh ip int brief
sh environment
sh platform
sh isis neighbors
sh int description
sh int description | i up
sh int description | i down
sh hw-module all fpd
sh ver
sh mpls l2transport vc
sh ip bgp all summary
sh redundancy
show ip ospf neighbor
show rep topology detail
show standby brief
show vrrp brief
show ip arp
sh running-config
sh tech-support
Bulk-sync issue:
If from console/terminal log, it is observed that standby RSP is reloading due to bulk-sync error
logs then do the following step
Please run below command to check the sync mismatch in active and standby RPs.
#show redundancy config-sync failures bem
#show redundancy config-sync failures mcl
#show redundancy config-sync failures prc
If you find mismatching lines There are two ways to avoid this, as below:
1- Remove problematic config lines from configuration and re-add them after a successful bootup
of standby RSP
2- no policy config-sync lbl prc “ and “ no policy config-sync bulk
3- Run command “#redundancy config-sync ignore mismatched-commands” and reload the RSP
CHCHEVIRCAG02CA903#format bootflash:
Format operation may take a while. Continue? [confirm]
Format operation will destroy all data in "bootflash:". Continue? [confirm]
The system was booted from bootflash: and is running from media.
In order to reformat bootflash: you must boot from consolidated package
or boot from a different media device.
mount: /dev/bootflash1 already mounted or /bootflash busy
mount: according to mtab, /dev/bootflash1 is already mounted on /bootflash
%Error formatting bootflash: (I/O error)
CHCHEVIRCAG02CA903#mkdir bootflash:XE313ES
Create directory filename [XE313ES]?
%Error Creating dir bootflash:/XE313ES (File exists)
For an example:
CHCHEVIRCAG02CA903#sh redundancy
------------------------------
Standby failures = 0
Communications = Up
In case redundancy mode is sso then change it to rpr in order to make the state standby-hot. During
the change standby RSP will get rebooted.
CHCHEVIRCAG02CA903(config)#redundancy
CHCHEVIRCAG02CA903(config-red)#mode ?
CHCHEVIRCAG02CA903(config)#exitCHCHEVIRCAG02CA903(config-red)#mode sso
CHCHEVIRCAG02CA903(config-red)#exit
Case study:
Observation:-
Router#sh platform
Chassis type: ASR-903
++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Router#sh platform
Chassis type: ASR-903
Router#sh platform
Chassis type: ASR-903
*Dec 7 08:23:04.641: %IOSXE-5-PLATFORM: F0: kernel: scsi 0:0:0:0: Direct-Access SanDisk Cruzer
Blade 1.00 PQ: 0 ANSI: 6
*Dec 7 08:23:04.641: %IOSXE-5-PLATFORM: F0: kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
*Dec 7 08:23:04.645: %IOSXE-5-PLATFORM: F0: kernel: sd 0:0:0:0: [sda] 30464000 512-byte logical
blocks: (15.5 GB/14.5 GiB)
*Dec 7 08:23:04.660: %IOSXE-5-PLATFORM: F0: kernel: sd 0:0:0:0: [sda] Attached SCSI removable
disk
*Dec 7 08:23:26.316: %BTRACE_ROTATE-3-ARCHIVE_FAIL: SIP0: btrace_rotate.sh: Error archiving
trace file - cman_fp_F0-0.log.13499.20171207081124.gz(error:'cp: cannot create regular file
`/harddisk/tracelogs/cman_fp_F0-0.log.13499.20171207081124.gz': No space left on device' for file
created:2017-12-07:08:23:26, closed:2017-12-07:08:11:24)
*Dec 7 08:23:36.762: %BTRACE_ROTATE-3-ARCHIVE_FAIL: SIP0: btrace_rotate.sh: Error archiving
trace file - cmcc_0-0.log.13363.20171207081133.gz(error:'cp: cannot create regular file
`/harddisk/tracelogs/cmcc_0-0.log.13363.20171207081133.gz': No space left on device' for file
created:2017-12-07:08:23:36, closed:2017-12-07:08:11:33)
*Dec 7 08:28:35.904: %IOSXE_OIR-6-OFFLINECARD: Card (rp) offline in slot R0
*Dec 7 08:28:35.921: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault
(PEER_NOT_PRESENT)
*Dec 7 08:28:35.921: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_DOWN)
*Dec 7 08:28:35.921: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault
(PEER_REDUNDANCY_STATE_CHANGE)
*Dec 7 08:28:36.015: %IOSXE_OIR-6-OFFLINECARD: Card (fp) offline in slot F0
*Dec 7 08:28:35.945: %CMRP-6-FP_HA_STATUS: R1/0: cmand: F1 redundancy state is Active with
no Standby
*Dec 7 08:28:35.962: %CMRP-6-RP_SB_RELOAD_REQ: R1/0: cmand: Reloading Standby RP: initiated
by RF reload message
*Dec 7 08:28:36.968: %IOSXE_OIR-6-REMCARD: Card (rp) removed from slot R0
*Dec 7 08:28:37.079: %IOSXE_OIR-6-REMCARD: Card (fp) removed from slot F0
*Dec 7 08:28:37.185: %IOSXE_OIR-6-REMCARD: Card (cc) removed from slot 0
*Dec 7 08:28:40.268: %IOSXE_OIR-6-INSCARD: Card (rp) inserted in slot R0
*Dec 7 08:28:40.270: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0
*Dec 7 08:28:40.281: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 0
*Dec 7 08:28:46.553: %RF-5-RF_RELOAD: Peer reload. Reason: EHSA standby down
We replaced the flash card & inserted again the faulty card & it came up after normal booting
but again it went down
Router#sh platform
Chassis type: ASR-903
Router#sh platform
Chassis type: ASR-903
We got the RSP rebooted from usb with image copied from active RSP. This time it was not
coming up but observed multibit error on ECC memory.
#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#
.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.
#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#
.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.
#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#.#
.#.#.#.#.#.#.#.#.#.#.#.#.#.#.
Image loaded
Image validated
Kernel panic - not syncing: Multi Bit error on ECC memory: address: 0x95C096D0, proc: chasync.sh
ÿ
Router#
Router#sh plat
Router#sh platform
Chassis type: ASR-903