You are on page 1of 15

N Narayan Rao

Customer Service Engineer Platform & Storage


Enterprise Services
Narayan.rao2@wipro.com
Subject - How to Replaced a failed disk in VXVM
Date 06/12/2011

Enterprise Services ,Wipro Infotech .

Page 1

How to Replaced a failed disk in VXVM


Please check below mentioned command before the activity
1) # vxdisk list

/* it will show you the failed disk in a long listing disk details ,
find the log below */

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto: none - - online invalid
c1t1d0s2 auto:none - - online invalid
c2t8d0s2 auto:sliced rootdiskc2t8 rootdg online
c2t9d0s2 auto - - error
c2t10d0s2 auto:sliced c2t10d0 scdg online
c2t11d0s2 auto:sliced c2t11d0 scdg online
c2t12d0s2 auto:sliced c2t12d0 scdg online
c2t13d0s2 auto:sliced - - online
c3t8d0s2 auto:sliced rootmirrorc3t8 rootdg online
c3t9d0s2 auto:sliced c3t9d0 scdg online
c3t10d0s2 auto:sliced c3t10d0 scdg online
c3t11d0s2 auto:sliced c3t11d0 scdg online
c3t12d0s2 auto:sliced c3t12d0 scdg online
c3t13d0s2 auto:sliced - - online
- - c2t9d0 scdg failed was:c2t9d0s2
2) #echo | format

/* check & verify which disk not showing in Solaris, in format


command it will show drive not available , find the log below */

#echo | format
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000000871d69d0,0
1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000000871d1f69,0
2. c2t8d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/pci@1/scsi@4/sd@8,0
3. c2t9d0 <drive not available>
4. c2t10d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/pci@1/scsi@4/sd@a,0 5. c2t11d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/pci@1/scsi@4/sd@b,0

Enterprise Services ,Wipro Infotech .

Page 2

3) # vxdiskadm

/* run this command to start the disk replacement, this command


show you lot of options, for removing the failed disk select 4th
option, find the log below */

# vxdiskadm
Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
22 Change/Display the default disk layouts
23 Mark a disk as allocator-reserved for a disk group
24 Turn off the allocator-reserved flag on a disk
list List disk information
? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: 4
Enterprise Services ,Wipro Infotech .

/*we should enter 4, for continue the disk


replacement activity */
Page 3

Remove a disk for replacement


Menu: VolumeManager/Disk/RemoveForReplace
Use this menu operation to remove a physical disk from a disk
group, while retaining the disk name. This changes the state
for the disk name to a "removed" disk. If there are any
initialized disks that are not part of a disk group, you will be
given the option of using one of these disks as a replacement.
Enter disk name [<disk>,list,q,?] list
/* enter list to see available dg details where the
harddisk need to be replaced */
Disk group: rootdg
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
dm rootdiskc2t8 c2t8d0s2 auto 20351 143328960 dm rootmirrorc3t8 c3t8d0s2 auto 20351 143328960
Disk group: scdg
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
dm c2t9d0 - - - - NODEVICE
dm c2t10d0 c2t10d0s2 auto 10175 143339136 dm c2t11d0 c2t11d0s2 auto 10175 143339136 dm c2t12d0 c2t12d0s2 auto 10175 143339136 dm c3t9d0 c3t9d0s2 auto 10175 143339136 dm c3t10d0 c3t10d0s2 auto 10175 143339136 dm c3t11d0 c3t11d0s2 auto 10175 143339136 dm c3t12d0 c3t12d0s2 auto 10175 143339136
Enter disk name [<disk>,list,q,?] c2t9d0 /*please enter the failed disk path here to deatached
the old disk from VXVM */
VxVM NOTICE V-5-2-371
The following volumes will lose mirrors as a result of this
operation:
cdl-L01 db09aux-L01 db09mib-L01 db09root-L01 home-L03 sc-L01 screl-L05
No data on these volumes will be lost.
VxVM NOTICE V-5-2-381
The requested operation is to remove disk c2t9d0 from disk group
scdg. The disk name will be kept, along with any volumes using
Enterprise Services ,Wipro Infotech .

Page 4

the disk, allowing replacement of the disk.


Select "Replace a failed or removed disk" from the main menu
when you wish to replace the disk.
Continue with operation? [y,n,q,?] (default: y) y

/*please enter Y for yes option to continue


the activity for removing
the old/faulty disk completely */

VxVM INFO V-5-2-265 Removal of disk c2t9d0 completed successfully.


Remove another disk? [y,n,q,?] (default: n) n
/*please enter n for selecting NO option to stop
the activity for Physically removing the removed
hard disk from the server/storage */
Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
22 Change/Display the default disk layouts
23 Mark a disk as allocator-reserved for a disk group
24 Turn off the allocator-reserved flag on a disk
Enterprise Services ,Wipro Infotech .

Page 5

list List disk information


? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: q
Goodbye.
#
/* now server completely stop the VXVM controller on the failed disk ,
now we need to replaced the same failed form solaris OS using cfgadm coammnd , please find
the log below */
4) # cfgadm al
5) # cfgadm -c unconfigure system:disk slot

/* showing the failed disk path and status*/


/* unconfigure the disk from system */

#cfgadm -c unconfigure c2::dsk/c2t9d0


*/ Note:- this command will not work if the same device used by VXVM , it will give error like
May 4 01:28:48 bhvomc rcm_daemon[8326]: rcm script es_rcm.pl: VxVM vxdmpadm ERROR V-51-5123 Attempt to disable path failed. No active path is associated to the node. Use the -f option if
you still want to disable this node.
cfgadm: Component system is busy, try again: failed to offline:
Resource
Information
------------------ ------------------------/dev/dsk/c2t9d0s2 Device being used by VxVM

/* For that solution we need the disable the same failed device from VXVM services.
Please find below mentioned steps for disable the failed device from VXVM */

6) vxdiskadm

/* it will showing so many options , chose 11th option in that list for
disable the selected disk */

Volume Manager Support Operations


Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
Enterprise Services ,Wipro Infotech .

Page 6

4 Remove a disk for replacement


5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
22 Change/Display the default disk layouts
23 Mark a disk as allocator-reserved for a disk group
24 Turn off the allocator-reserved flag on a disk
list List disk information
? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: 11
Disable (offline) a disk device
Menu: VolumeManager/Disk/OfflineDisk
Use this menu operation to disable all access to a disk device
by the Volume Manager. This operation can be applied only to
disks that are not currently in a disk group. Use this operation
if you intend to remove a disk from a system without rebooting.
NOTE: Many systems do not support disks that can be removed from a system during normal
operation. On such systems, the offline operation is seldom useful
Select a disk device to disable [<disk>,list,q,?] list
Enterprise Services ,Wipro Infotech .

Page 7

DEVICE DISK GROUP STATUS


c1t0d0 - - online invalid
c1t1d0 - - online invalid
c2t8d0 rootdiskc2t8 rootdg online
c2t9d0 - - error
c2t10d0 c2t10d0 scdg online
c2t11d0 c2t11d0 scdg online
c2t12d0 c2t12d0 scdg online
c2t13d0 - - online
c3t8d0 rootmirrorc3t8 rootdg online
c3t9d0 c3t9d0 scdg online
c3t10d0 c3t10d0 scdg online
c3t11d0 c3t11d0 scdg online
c3t12d0 c3t12d0 scdg online
c3t13d0 - - online
Select a disk device to disable [<disk>,list,q,?] c2t9d0 /*need to enter disk path ,for offline */
May 4 01:33:21 bhvomc scsi: WARNING: /pci@8,600000/pci@1/scsi@4/sd@9,0 (sd38):
May 4 01:33:21 bhvomc offline
May 4 01:33:24 bhvomc scsi: WARNING: /pci@8,600000/pci@1/scsi@4/sd@9,0 (sd38):
May 4 01:33:24 bhvomc offline
Disable another device? [y,n,q,?] (default: n) n
/*enter n for NO,to exit from this command */
/*Note after disabling the disk , go to the Solaris part and run cfgadm command for
unconfigured the devices ,find the log below*/
# cfgadm -f -c unconfigure c2::dsk/c2t9d0
7 ) /* Now you can go for the physical replacement of the disk .
/*Note check the amber led disk for identifying the failed disk .
If the amber led not glowing, then you have to check the previous and post disk slot of the failed
disk. For the checking yor can follow this below mentioned steps */
->format -> select disk2 ->analyze -> read

Enterprise Services ,Wipro Infotech .

Page 8

/* it will blink the particular disk , same you can check the 4th and 2nd disk of the storage and it
will helps you to identify the failed disk 3 easily which is in between 2 and 4 */
After replacing the disk----

8) #cfgadm -c configure c2:: dsk/c2t9d0


9) # echo | format

/* it will configured the new disk in solaris */


/* please check the 3rd number disk failed disk
showing the disk size ,sec,cy,find the log below */

AVAILABLE DISK SELECTIONS:


0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000000871d69d0,0
1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000000871d1f69,0
2. c2t8d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/pci@1/scsi@4/sd@8,0
3. c2t9d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/pci@1/scsi@4/sd@9,0
4. c2t10d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@8,600000/pci@1/scsi@4/sd@a,0
5. c2t11d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/* If the disk showing in normal mode, then we need to
enable the disk in VXVM access */
10 ) vxdiskamd

/* now the time to enable the disk volume in vxvm , select


the option 10 for enable the disk path in vxvm,find the log below */

# vxdiskadm
Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
Enterprise Services ,Wipro Infotech .

Page 9

11 Disable (offline) a disk device


12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
22 Change/Display the default disk layouts
23 Mark a disk as allocator-reserved for a disk group
24 Turn off the allocator-reserved flag on a disk
list List disk information
? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: 10
Enable (online) a disk device
Menu: VolumeManager/Disk/OnlineDisk
Use this operation to enable access to a disk that was disabled
with the "Disable (offline) a disk device" operation.
You can also use this operation to re-scan a disk that may have
been changed outside of the Volume Manager. For example, If a
disk is shared between two systems, the Volume Manager running on
the other system may have changed the disk. If so, you can use
this operation to re-scan the disk.
NOTE: Many vxdiskadm operations re-scan disks without user
intervention. This will eliminate most needs to online a
disk directly, except when the disk is directly offlined.
Select a disk device to enable [<disk>,list,q,?] list /*enter list to view available offline disk*/
DEVICE DISK GROUP STATUS
c1t0d0 - - online invalid
c1t1d0 - - online invalid
c2t8d0 rootdiskc2t8 rootdg online
c2t9d0 - - offline
c2t10d0 c2t10d0 scdg online
c2t11d0 c2t11d0 scdg online
Enterprise Services ,Wipro Infotech .

Page 10

c2t12d0 c2t12d0 scdg online


c2t13d0 - - online
c3t8d0 rootmirrorc3t8 rootdg online
c3t9d0 c3t9d0 scdg online
c3t10d0 c3t10d0 scdg online
c3t11d0 c3t11d0 scdg online
c3t12d0 c3t12d0 scdg online
c3t13d0 - - online
Select a disk device to enable [<disk>,list,q,?] c2t9d0
Enable another device? [y,n,q,?] (default: n) n
/*enter n for No to exit from vxvm command*/
Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
22 Change/Display the default disk layouts
23 Mark a disk as allocator-reserved for a disk group
24 Turn off the allocator-reserved flag on a disk
list List disk information
? Display help about menu
Enterprise Services ,Wipro Infotech .

Page 11

?? Display help about the menuing system


q Exit from menus
Select an operation to perform: q
Goodbay
/*Then check the disk list for vxvm enable the disk or not */
11) vxdisk list
/* it will check all disk latest status and shows the latest enable command */
Example-# vxdisk list
NAME STATE ID
rootdg enabled 1170010227.18.bhvomc
scdg enabled 1097437955.1151.bbsromcr-1c2t8

12) # vxdisk list


/*check the disk status in vxvm */
DEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:none - - online invalid
c1t1d0s2 auto:none - - online invalid
c2t8d0s2 auto:sliced rootdiskc2t8 rootdg online
c2t9d0s2 auto:none - - online invalid
c2t10d0s2 auto:sliced c2t10d0 scdg online
c2t11d0s2 auto:sliced c2t11d0 scdg online
c2t12d0s2 auto:sliced c2t12d0 scdg online
c2t13d0s2 auto:sliced - - online
c3t8d0s2 auto:sliced rootmirrorc3t8 rootdg online
c3t9d0s2 auto:sliced c3t9d0 scdg online
c3t10d0s2 auto:sliced c3t10d0 scdg online
c3t11d0s2 auto:sliced c3t11d0 scdg online
c3t12d0s2 auto:sliced c3t12d0 scdg online
c3t13d0s2 auto:sliced - - online
- - c2t9d0 scdg removed was:c2t9d0s2
Now time to take the final step
13) #vxdiskadm

/* run the vxdiskamd command to use the 5th option for replaced
the hardisk option to start the syn with its sub-disk/mirrored part */

bash-2.05# vxdiskadm
Voleme Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
Enterprise Services ,Wipro Infotech .

Page 12

3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
22 Change/Display the default disk layouts
23 Mark a disk as allocator-reserved for a disk group
24 Turn off the allocator-reserved flag on a disk
list List disk information
? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: 5
Replace a failed or removed disk
Menu: VolumeManager/Disk/ReplaceDisk
Use this menu operation to specify a replacement disk for a disk
that you removed with the "Remove a disk for replacement" menu
operation, or that failed during use. You will be prompted for
a disk name to replace and a disk device to use as a replacement.
You can choose an uninitialized disk, in which case the disk will
be initialized, or you can choose a disk that you have already
initialized using the Add or initialize a disk menu operation.
Enterprise Services ,Wipro Infotech .

Page 13

Select a removed or failed disk [<disk>,list,q,?] list


Disk group: rootdg
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
Disk group: scdg
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
dm c2t9d0 - - - - REMOVED
Select a removed or failed disk [<disk>,list,q,?] c2t9d0
/*enter replaced disk path to add disk in VXVM */
The following devices are available as possible replacements after being
initialized (or reinitiliazed):
c1t0d0 c1t1d0 c2t9d0
You can choose one of these devices to replace c2t9d0.
Choose "none" to abort the replacement of c2t9d0.
Choose a device, or select "none"
[<device>,none,q,?] (default: c1t0d0) c2t9d0
The following disk device has a valid VTOC, but does not appear to have
been initialized for the Volume Manager. If there is data on the disk
that should NOT be destroyed you should encapsulate the existing disk
partitions as volumes instead of adding the disk as a new disk.
Output format: [Device_Name]
c2t9d0
Encapsulate this device? [y,n,q,?] (default: y) n
c2t9d0
Instead of encapsulating, initialize? [y,n,q,?] (default: n) y
VxVM INFO V-5-2-378
The requested operation is to initialize disk device c2t9d0 and
to then use that device to replace the removed or failed disk
c2t9d0 in disk group scdg.
Continue with operation? [y,n,q,?] (default: y) y
Use FMR for plex resync? [y,n,q,?] (default: n) y
VxVM INFO V-5-2-282
Replacement of disk c2t9d0 in group scdg with disk device
c2t9d0 completed successfully.
Replace another disk? [y,n,q,?] (default: n) n /*enter n for exit form vxvm control completely */
_
Enterprise Services ,Wipro Infotech .

Page 14

Volume Manager Support Operations


Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
22 Change/Display the default disk layouts
23 Mark a disk as allocator-reserved for a disk group
24 Turn off the allocator-reserved flag on a disk
list List disk information
? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: q
/* Now the sync is started .. you can check using the vxtask list command*/
14) #vxtask list
bash-2.05# vxtask list
ExampleTASKID PTID TYPE/STATE PCT PROGRESS
171 PARENT/R 0.00% 7/0(1) VXRECOVER c2t9d0 scdg
172 172 ATCOPY/R 05.74% 0/114973184/6596608 PLXATT cdl-L01 cdl-P01 scdg
*it will take time of sync, after that the vxtask list command output not
showing any process id and percents , it means Activity done successfully.
Enterprise Services ,Wipro Infotech .

Page 15

You might also like