Professional Documents
Culture Documents
sanspire.com/emc-notes-for-day-to-day-operation/
By sanaswati
DISCLAIMER:Information mentioned below as been compiled from various sources such as first hand experience by myself and
my colleagues, from webpages searched via google etc. I do not take credit for the tips mentioned below.
If a linux host using QLogic card is seeing 18 paths per disk instead of 2, chances are you might need to Enable
SPC2 version for that host’s wwn on a particular dir/port
# symmask -sid 1234 -wwn 1000000000000000 set hba_flags on SPC2 -enable -dir 08C -p 1
# symmaskdb -sid 1234 list database -v -dir 08C -p 1
Originator Port wwn : 1000000000000000
Type : Fibre
User-generated Name : hostA/qla1
Visibility : No
FCID Lockdown : No
Lun Offset : No
Heterogeneous Host : No
Port Flag Overrides : Yes
Enabled : SPC2_Protocol_Version(SPC2)
Disabled : N/A
Dynamic Addressing : No
Authentication State : N/AG
Devices : 1003 1007 100B
3) After we split the BCV device DEV001 from testdg, we need to issue “vxdctl enable” or “vxdisk scandisks” to update the configuration in
order to seee clone disk is in “online udid_mismatch” state issuing “vxdctl enable” ( or “vxdisk scandisks”).
DEVICE TYPE DISK GROUP STATUS
EMC0_1 auto:cdsdisk EMC0_1 testdg online
EMC0_2 auto:cdsdisk - - online udid_mismatch
Using the first and last serial numbers as examples, the serial number is broken out as follows:
1/9
So, the first example, device 009 is mapped to director 15, processor A, port 0 while the second example has device 01A mapped to
director 12, processor B, port 0.
How are hypers laid out across underlying hard drives> (Courtsey – James Spurin)
The recommendation internally with the V-Max is for 8 hyper splits across the disk, in testing this configuration has been proven to
provide the best pormance as each lun has dedicated access to one of the disks internal queues, in essence.
Disk1 would service part of 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008
Disk2 would service part of 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008
Disk3 would service part of 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008
Disk4 would service part of 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008
Disk5 would service part of 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008
Disk6 would service part of 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008
Disk7 would service part of 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008
Disk8 would service part of 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008...
You can work out the size necessary for this by dividing the size of the disk by 8, i.e. a 450GB disk would be 56.25GB … in our case the
magic number 53872 cylinders for accuracy.
Another good point to know is that also confirmed is that there is no degradation in performance when you configure these through
symconfigure instead symwin as per a bin file configuration … it.s just more difficult to get your initial figures correct with symconfigure
and you don.t get the pretty phical view and the number optimization.
With regards to the ACLX setting on the directors … by default the ACLX lun is presented to all directors and subsequently any host will
always see t lun which can be annoying and cause confusion … you can resolve this by presenting gatekeepers to the management host and
then unpresenting the ACLXn from the directors.
How to use a single Unix server (client) to run commands on various remote management servers (symapi
servers) managing different arrays?
It involves 2 steps:
1. On client, edit /var/symapi/config/netcnfg on local Unix server (client) to define remote management servers.
DC1_SERVER - TCPIP hostA 192.168.0.1 2707 ANY #
DC2_SERVER - TCPIP hostB 192.168.1.1 2707 ANY #
2. On client, connect to DC1_SERVER or DC2_SERVER by exporting following variables (can be done either manually or via profile files)
# export SYMCLI_CONNECT_TYPE=REMOTE
# export SYMCLI_CONNECT=DC1_SERVER
ERROR
CLIENT# symcfg list
The remote connection is refused. The symapi server is probably not running
SOLUTION
ERROR
2/9
CLIENT# symcfg list
The remote client/server handshake failed. Please consult symapi and storsrvd log files
SOLUTION
This is because remote server is expecting either a secure/nonsecure protocol for connection. Change the parameter
storsrvd:security_level in /var/symapi/config/daemon_options on server.
Before change:
# storsrvd:security_level = SECURE
Shutdown and start storsrvd using stordaemon command and try again.
ERROR
SOLUTION
Its for an obviously stated reason – Solution Enabler on client is of a higher version than that on server. Check the version by running
command symcli
CLIENT# symcli
Symmetrix Command Line Interface (SYMCLI) Version V7.6.1.0 (Edit Level: 1755) built with SYMAPI Version V7.6.1.0
(Edit Level: 1755)
DC1_SERVER# symcli
Symmetrix Command Line Interface (SYMCLI) Version V7.1.2.0 (Edit Level: 1059) built with SYMAPI Version V7.1.2.0
(Edit Level: 1059)
ERROR
SOLUTION
On remote server that is running storsrvd, check /var/symapi/config/nethost file and add an entry in the format “FQDN_of_client
userid_allowed_from_client” (example: hostnameA username). Restart storsrvd daemon by running
stordaemon shutdown storsrvd and stordaemon start storsrvd .
ERROR
CLIENT# symcfg list
The gatekeeper device (while using the Base Daemon) has an error (Please see the Log file)
CLIENT# symcfg discover
This operation may take up to a few minutes. Please be patient...
Unable to obtain unique ID for host
CLIENT#
SOLUTION
On remote server that is running storsrvd, change SYMAPI_USE_ACCESS_ID = LOCAL to SYMAPI_USE_ACCESS_ID = SERVER in
/var/symapi/config/options file and restart the daemons by running
stordaemon shutdown storsrvd and stordaemon start storsrvd .
3/9
/usr/emc/ECC/data/swdata/symmWin.passwd – ECC Passwords are stored in this file.
/var/symapi/db/symapi_db.bin – default symapi database file (used to sync the configuration from the array to the management host
and is creatupdated when you do a symcfg discover. No problem with renaming it and then re-doing a symcfg discover to recreate it)
/var/symapi/config/netcnfg – lists the network services available from that host
/var/symapi/config/options – the options file contains behavior parameters that can be set to critically change default behavior of
SYMCLI oations, SYMAPI calls and their control actions
/var/symapi/config/symavoid – identifies devices to skip over when looking for devices
/var/symapi/config/gkavoid – identifies devices not chosen as gatekeepers
/var/symapi/config/inqfile – lists devices to be added to the symapi database
/var/symapi/log/symapi-.log – logfile for symcli.symapi functions, calls, activities
Resolution: The setting asb:iopf stands for “Auto standby intermittent I/O failure” algorithm.
This feature, the intermittent I/O failure algorithm, means the path will not be used unless all other active paths are dead.
asb path mode settings do not persist across reboots.
This can be turned on using the following options:
Turn on: powermt set autostanby=on trigger=iopf
Turn off: powermt set autostanby=off trigger=iopf
Usages: powermt set autostandby={on|off|reinitialize} [trigger={prox|iopf}]
This determines whether or not a path that has intermittent I/O failures should be placed into autostandby mode (asb:iopf). You can
override autostandby for a path by running powermt set mode={standby|active} dev= force .
If the path is already in the requested mode, then the command is ignored.
dev=||all sets the mode for the specified path, or all paths to the specified device (all specifies all paths to all devices.) The default is
all.
Example for device path:
# powermt set mode=active dev=/pci@8,700000/QLGC,qla@3/sd@
Example for all devices:
# powermt set mode=active dev=all devices
# powermt set mode=active dev=all devices
To verify if the asb:iopf is disabled or enabled check powermt display options or powermt
display dev=all.
4/9
A note on FAN in and FAN out rations:
Fan in Raito : The Fan ratio defines the number of storage connections to a particular server.
For example Server A can access storage from Storage A, B, C, and D So the fan in ratio is 1:4
Fan out ratio: The fan out ratio is the number of initiators per storage port. So you could have Server A, B, C and D accessing Storage port-
A. The fant ratio would be 3:1
Also remember that each storage port has a definite queue length so connecting more hosts means sharing of that queue depth and thus
can lead to perforce bottlenecks.
If device comes up as “Reserved” on server after masking on the array (the dev was in use before somewhere)
Find out disks which have SCSI reservation
# symdev -sid 1234 -resv list | grep 0CD8
/dev/rdsk/c2t5006048C52A90B46d172s2 0CD8 RAID-5 08 03B:1 Group
/dev/rdsk/c2t5006048C52A90B46d172s2 0CD8 RAID-5 08 03B:1 Group
/dev/rdsk/c2t5006048C52A90B46d172s2 0CD8 RAID-5 08 10B:1 Group
/dev/rdsk/c2t5006048C52A90B46d172s2 0CD8 RAID-5 00 07C:0 Group
/dev/rdsk/c2t5006048C52A90B46d172s2 0CD8 RAID-5 00 14C:0 Group
/dev/rdsk/c2t5006048C52A90B46d172s2 0CD8 RAID-5 00 14C:0 Group
Create device group, add the devices in it, break the device away from device group, delete the device group
# symdg -type regular create testdg (chose -type rdf1 for R1s and rdf2 for R2s)
# symld -sid 1234 -g testdg addall -range 0CD8:0CE8
# symld -g testdg break DEV0001 (Do this for all the dev)
# symdg -force delete testdg
http://knowledgebase.emc.com/emcice/documentDisplay.do?
docType=1006&clusterName=DefaultCluster&resultType=5002&groupId=1&page=&docProp=$solution_id&docpValue=emc81931&passedT
S Y M M E T R I X L O C K S
The specified Symmetrix External Lock cannot be released except if the force flag is used
5/9
Interrogating Configuration Server for config change session information
Lock 15 was obtained to protect a configuration change session.
Abort the configuration change session for Symmetrix unit 000290101234 (y/[n]) ? y
A few devices on server can’t pick up the pseudo name (last column). What is the solution?
# syminq |egrep -i "hdisk5|hdisk25"
/dev/rhdisk5 EMC SYMMETRIX 5772 8300562000 N/A
/dev/rhdisk25 EMC SYMMETRIX 5772 8300562000 N/A
/dev/rhdisk50 EMC SYMMETRIX 5772 8300610000 23809920
Manufacturer................EMC
Machine Type and Model......SYMMETRIX
ROS Level and ID............5772
Serial Number...............83562000
Part Number.................000000000000510026000290
EC Level....................102783
LIC Node VPD................0562
Device Specific.(Z0)........00
Device Specific.(Z1)........51
Device Specific.(Z2)........577201040000000000093010
Device Specific.(Z3)........12000000
Device Specific.(Z4)........54110008
Device Specific.(Z5)........FF80
Device Specific.(Z6)........4D
SOLUTION:
Go into /etc and look for any files with a .FCS extension. If there are, remove them.
Then move the /kernel/drv/emcp.conf file and any .saved versions to a different location and rename them.
Once this is done, do a touch /kernel/drv/emcp.conf to recreate the file.
Run powercf -q and a powermt config.
Check with the powermt_display_dev=all command to see if all the devices now have Pseudo Names.
When server logs the errors about paths going dead and coming back, chances are they running short of
gatekeepers, and are using data devices as gatekeepers instead.
May 18 20:18:38 hostA kern:err|error unix: emcp: Error: Killing bus 0 to Symmetrix 000294901111 port 7gA.
May 18 20:18:38 hostA kern:err|error unix: emcp: Error: Path hdisk390 to 000294901111 is dead.
May 18 20:18:38 hostA kern:err|error unix: emcp: Error: Path hdisk123 to 000294901111 is dead.
May 18 20:18:38 hostA kern:err|error unix: emcp: Error: Path hdisk395 to 000294901111 is dead.
...
SOLUTION:
Add extra gatekeepers, and also add following parameter to file /var/symapi/config/daemon_options as described in emc217815,
emc231545.
storapid:parallel_inquiry_size = 0
While running symmigrate for VLUN migration, the client host receives an error when establishing clone session
# symclone -sid 1234 -f clone_1234 -nop establish
'Clone Incremental Establish' operation execution is in progress for the device list in device file
'clone_1234'. Please wait...
6/9
Thin device is not bound to any pool
Establish failed
This is because if incorrect version of Solution Enabled being used. It should be V7.2.1.0 or above.
PowerPath Pseudo names for newly presented devices are missing upon reboot. Rnning “powercf -q; powermt
config” does not help OR powermt.custom is missing the entries for all the newly presented devices OR
emcp.conf has the entries for the missing pseudo names
Environment:
OS: Solaris 10 SPARC
EMC SW: PowerPath
EMC SW: PowerPath 5.3 P02
Cause:
The boot archive had not been updated after the configuration changes. The update normally occurs during the shutdown process but a
hard reset can interrupt this scheme.
Resolution/Workaround:
For each missing pseudo name, run a powermt unmanage/powermt manage command using the native name such as:
# powermt unmanage dev=c3t50000974C0149158d88
# powermt manage dev= c3t50000974C0149158d88
This will fix PowerPath configuration. Once this is done for all the devices, run a “powermt save” to update powermt.custom. Then run a
“bootadm update-archive” command to avoid the issue at the next reboot.
Permanent fix:
After each PowerPath configuration change, run a “bootadm update-archive” command to be sure that the changes are persistent upon
reboot even in case of “not so clean” reboot.
Server having issues in picking up newly allocated devices, configuring them or powerpath. Useful commands in
troubleshooting the issue
fcinfo hba-port
cfgadm -lao show_FCP_dev
powermt display
powermt display dev=all
powermt display paths
sympd list
syminq hba
syminq -wwn
syminq -pdevfile
symcfg list
7/9
Powerpath not creating emcpowerpath devices for newly assigned devices
For this, unmanage and then remanage the device
# /var/tmp/uplift# syminq |grep 264C
/dev/rdsk/c1t500009740842B514d10s2 R1 EMC SYMMETRIX 5875 690264C000 23809920
/dev/rdsk/c1t500009740842B550d10s2 R1 EMC SYMMETRIX 5875 690264C000 23809920
/dev/rdsk/c3t500009740842B56Cd10s2 R1 EMC SYMMETRIX 5875 690264C000 23809920
/dev/rdsk/c3t500009740842B528d10s2 R1 EMC SYMMETRIX 5875 690264C000 23809920
Solution
#powermt config
powermt unmanage dev=c1t500009740842B514d10s0
powermt manage dev=c1t500009740842B514d10s0
# syminq |grep 264C
/dev/rdsk/c1t500009740842B514d10s2 R1 EMC SYMMETRIX 5875 690264C000 23809920
/dev/rdsk/c1t500009740842B550d10s2 R1 EMC SYMMETRIX 5875 690264C000 23809920
/dev/rdsk/c3t500009740842B56Cd10s2 R1 EMC SYMMETRIX 5875 690264C000 23809920
/dev/rdsk/c3t500009740842B528d10s2 R1 EMC SYMMETRIX 5875 690264C000 23809920
/dev/rdsk/emcpower0c R1 EMC SYMMETRIX 5875 690264C000 23809920
In most of the cases, this occurs because of mismatch between the users/groups defined in symauth database on the array, and credentials
of user running command from host. Find out the list of users/groups defined in symauth database using the command:
# symauth list -user -sid 1234
Ensure the users are defined as user and groups are defined as group. symauth will gleefully accept group name as user and vice versa,
and you will end up scratching your hairs why it isn’t working. To delete an entry from and to add a new entry into symauth database:
# cat delusergrp
delete user xyz; group=grpname;
# symauth -sid 1234 -f delusergrp comm -nop
# cat addusergrp
assign abc to role storageadmin; newgroup admin;
# symauth -sid 1234 -f addusergrp comm -nop
8/9
powerpath fails to load with powershift script message
Error messages in /var/adm/messages:
Cancelling powershift script: PowerPath driver version () does not match expected version (53000)
The empty parenthesis in the error reported in /var/adm/messages (PowerPath driver version () does not..) is showing that the powershift
script can not find the emcp release with the “modinfo” command because it is not loaded.
Solution is to uninstall PowerPath and reinstall it.
Error while creating devices on local and remote array using symconfigure
# symconfigure -sid 1234 -cmd "create dev count=4, size=50 GB, emulation=FBA, dynamic_capability=dyn_rdf,
config=RDF1+TDEV, remote_config=RDF2+TDEV, ra_group=21, binding to pool=localpool_1,
remote_pool=remotepool_1,sg=localhostA_sga,remote_sg=remotehostB_sga; " prep -nop
A Configuration Change operation is in progress. Please wait...
Establishing a configuration change session...............Failed.
Error occurred while Starting
Remote configuration change sessions cannot define or abort RDF changes.
The configuration change session has failed.
Solution
Ensure the user you are running the command as has appropriate group asigned in symauth/symacl across both the arrays.
You’ve created new SRDF paired devices on local and remote array using symconfigure. However, when you try
to add the remote devices into storage groups, storage management server gives error “Devices not configured”
Solution
This happens because the either storage management servers (local and remote) have not refreshed their configurations with devices
remote to them. To refresh the configuration, run the following command:
symcfg -sid 1234 list -tdev -dev xxx:xxx
9/9