Professional Documents
Culture Documents
Mpio SDD PCM SDD
Mpio SDD PCM SDD
Agenda
Multi-path basics Multi Path I/O (MPIO)
Useful Path
MPIO Commands
priorities
Failed
path management SDD and SDDPCM Multi-path code choices for DS4000, DS5000 and DS3950 SAN Boot
MPIO
Server
FC Switch
Storage
Disk configuration
The disk vendor dictates what multi-path code can be used, supplies the filesets for it and supports it A fileset is loaded to update the ODM to support the storage AIX then recognizes and appropriately configures the disk Without this, disks are configured using a generic ODM definition Performance and error handling may suffer as a result # lsdev Pc disk displays supported storage The multi-path code will be a different fileset Unless using the MPIO thats included with AIX
Disks always use MPIO and all IO for a LUN normally goes to one VIOS VIOS uses the multi-path code specified for the disk subsystem
VIO Server
Multi-path code
VIO Server
Multi-path code
Set path priorities for hdisks so half the hdisks use one VIOS and half use the other
Disk Subsystem
Potentially
one vFC adapter for every real FC adapter in each VIOC of 64 vFC adapters per real FC adapter recommended
Maximum
VIO Server
VIO Server
VIOC uses multi-path code that the disk subsystem supports IOs for a LUN can go thru both VIOSs
Disk Subsystem
What is MPIO?
MPIO is an architecture designed by AIX development MPIO is also a commonly used acronym for Multi-Path IO
In this presentation MPIO refers explicitly to the architecture, not the acronym
With the advent of SANs, each disk subsystem vendor wrote their own multi-path code These multi-path code sets were usually incompatible Mixing disk subsystems was usually not supported on the same system, and if they were, they usually required their own FC adapters Integration with AIX IO error handling and recovery Several levels of IO timeouts: basic IO timeout, FC path timeout, etc
Compliant code requires a Path Control Module (PCM) for each disk subsystem Default PCMs for SCSI and FC exist on AIX and often used by the vendors Capabilities exist for different path selection algorithms Disk vendors have been moving towards MPIO compliant code
for 32 K paths
more than 16 is necessary PCMs exist for FC, SCSI may write optional PCMs
Allows various algorithms to balance use of paths Full support for multiple paths to rootvg Hdisks can be Available, Defined or non-existent Paths can also be Available, Defined, Missing or non-existent Path status can be enabled, disabled or failed if the path is Available One must get the device layer correct, before working with the path status layer
MPIO support
Storage subsystem family IBM ESS, DS6000, DS8000, DS4000, DS5000, SVC XIV NSeries EMC Symmetrix HDS HP (not EVA) some members of XP family SCSI VIO VSCSI MPIO code SDDPCM Default FC PCM Default FC PCM Default FC PCM HDLM Default FC PCM Default SCSI PCM Default SCSI PCM Multi-path algorithm fail_over, round_robin, load balance, load balance port fail_over, round_robin fail_over, round_robin fail_over, round_robin fail_over, round robin, extended round robin fail_over, round_robin fail_over, round_robin fail_over
10
Storage subsystem family IBM DS4000 and DS5000 EMC HP HDS Vertias supported storage
Multi-path code RDAC Power Path AutoPath HDLM (older versions) DMP
11
disk subsystem vendor supports their storage, the server vendor generally doesnt
You can mix multi-path code compliant with MPIO and even share adapters
Exception
Generally one non-MPIO compliant code set can exist with other MPIO compliant code sets
Except The
you cant used SDDPCM for one DS8000 and SDD for another DS8000 on the same AIX instance
12
13
MPIO Commands
lspath list paths, path status and path attributes for a disk chpath change path status or path attributes
Enable
or disable paths
a path into the defined mode means it wont be used (from available to defined)
chdev change a devices attributes (not specific to MPIO) cfgmgr add new paths to an hdisk or make defined paths available (not specific to MPIO)
14
scsi_id
0x30400
SCSI ID
False
Priority
15
True
Path priorities
A Priority Attribute for paths can be used to specify a preference for path IOs. How it works depends whether the hdisks algorithm attribute is set to fail_over or round_robin. algorithm=fail_over
the the Set
path with the higher priority value handles all the IOs unless there's a path failure. other path(s) will only be used when there is a path failure. the primary path to be used by setting it's priority value to 1, and the next path's priority (in case of path failure) to 2, and so on.
algorithm=round_robin
If
the priority attributes are the same, then IOs go down each path equally.
In
the case of two paths, if you set path As priority to 1 and path Bs to 255, then for every IO going down path A, there will be 255 IOs sent down path B.
chpath -l hdisk0
-p vscsi1 -a priority=25
16
Path priorities
# lsattr -El hdisk20 PCM PCM/friend/otherapdisk algorithm fail_over hcheck_interval 60 hcheck_mode nonactive lun_id 0x5000000000000 node_name 0x20060080e517b6ba queue_depth 10 reserve_policy single_path ww_name 0x20160080e517b6ba Path Control Module Algorithm Health Check Interval Health Check Mode Logical Unit Number ID FC Node Name Queue DEPTH Reserve Policy FC World Wide Name False True True True False False True True False
# lspath -l hdisk9 -F"parent connection status path_status" fscsi1 20160080e517b6ba,5000000000000 Enabled Available fscsi1 20170080e517b6ba,5000000000000 Enabled Available # lspath -AEl hdisk9 -p fscsi1 -w"20160080e517b6ba,5000000000000" scsi_id 0x10a00 SCSI ID False node_name 0x20060080e517b6ba FC Node Name False priority 1 Priority True
17
priorities for half the LUNs to use VIOSa and half to use VIOSb both VIOSs CPU and adapters is the only option at the VIOC for VSCSI disks
Uses
algorithm=fail_over
With NSeries have the IOs go the primary controller for the LUN
Set
18
True
hcheck_interval Defines how often the health check is performed on the paths for a device. The attribute supports a range from 0 to 3600 seconds. When a value of 0 is selected (the default), health checking is disabled hcheck_mode Determines which paths should be checked when the health check capability is used: enabled: Sends the healthcheck command down paths with a state of enabled failed: Sends the healthcheck command down paths with a state of failed nonactive: (Default) Sends the healthcheck command down paths that have no active I/O, including paths with a state of failed. If the algorithm selected is failover, then the healthcheck command is also sent on each of the paths that have a state of enabled but have no active IO. If the algorithm selected is round_robin, then the healthcheck command is only sent on paths with a state of failed, because the round_robin algorithm keeps all enabled paths active with IO. Enable path health checking if paths arent normally used, and only for enough LUNs to ensure physical paths are checked Often turning on health checking for a single LUN is sufficient to monitor all physical paths status An error shows up in the error log if a path fails Consider setting up error notification Failed Path Recovery To enable a failed path: # chpath -l hdisk1 -p <parent> -s enable
19
Adapter failure and replacement VIOS upgrades (VIOS or multi-path code) Cable failure and replacement Storage controller/port failure and repair Paths will not be in use if the adapter has failed, paths will be in the failed state Remove paths with # rmpath l <hdisk> -p <parent> -w <connection> [-d] -d will remove the path, without it the path will changed to Defined Remove the adapter with # rmdev Rdl <fcs#> Replace the adapter cfgmgr Check the paths with lspath Avoid timeouts, application delays or performance impacts and potential error recovery bugs
Adapter replacement
Its better to stop using a path before you know the path will disappear
20
a controller is active for some LUNs, but passive for the others
IOs for a LUN are only be sent to the Active controllers port for disk subsystems with Active/Passive controllers
ESS,
DS6000, and DS8000 have active/active controllers DS4000, DS5000, DS3950, NSeries have active/passive controllers The NSeries passive controller can accept IOs but IO latency is affected The passive controller takes over in the event the active controller or all paths to it fail MPIO recognizes Active/Passive disk subsystems and sends IOs only to the primary controller
Except
under failure conditions, then the active/passive role switches for the affected
LUNs
21
SDD: An Overview
SDD = Subsystem Device Driver Used with IBM ESS, DS6000, DS8000 and the SAN Volume Controller, but is not MPIO compliant.
A
host attachment fileset (populates the ODM) and SDD fileset are both installed attachment: ibm2105.rte devices.sdd.<sdd_version>.rte
Host
SDD:
paths maximum per LUN, but less are recommended with more than 600 LUNs
One installs SDDPCM or SDD, not both. No support for rootvg, dump or paging devices
One can exclude disks from SDD control using the excludesddcfg command Mirror rootvg across two separate LUNs on different adapters for availability
22
SDD
Load balancing algorithms
fo: rr: lb:
failover round robin load balancing (aka. df or the default) and chooses adapter with fewest in-flight IOs load balancing sequential optimized for sequential IO round robin sequential optimized for sequential IO
lbs: rrs:
The datapath command is used to examine vpaths, adapters, paths, vpath statistics, path statistics, adapter statistics, dynamically change the load balancing algorithm, and other administrative tasks such as adapter replacement, disabling paths, etc. mkvg4vp is used instead of mkvg, and extendvg4vp is used instead of extendvg SDD automatically recovers failed paths that have been repaired via the sddsrv daemon
23
SDDPCM: An Overview
SDDPCM = Subsystem Device Driver Path Control Module SDDPCM is MPIO compliant and can be used with IBM ESS, DS6000, DS8000, DS4000 (most models), DS5000, DS3950 and the SAN Volume Controller
A
host attachment fileset (populates the ODM) and SDDPCM fileset are both installed attachment: devices.fcp.disk.ibm.mpio.rte devices.sddpcm.<version>.rte
Host
SDDPCM:
Provides a PCM per the MPIO architecture One installs SDDPCM or SDD, not both. SDDPCM is recommended and strategic
24
SDDPCM
Load balancing algorithms
rr lb fo
- round robin - load balancing based on in-flight IOs per adapter - failover policy
lbp
- load balancing port (for ESS, DS6000, DS8000 and SVC only) based on in-flight IOs per adapter and per storage port
The pcmpath command is used to examine hdisks, adapters, paths, hdisk statistics, path statistics, adapter statistics, dynamically change the load balancing algorithm, and other administrative tasks such as adapter replacement, disabling paths SDDPCM automatically recovers failed paths that have been repaired via the pcmserv daemon
MPIO
health checking can also be used, and can be dynamically set via the pcmpath command. This is recommended. Set the hc_interval to a non-zero value for an appropriate number of LUNs to check the physical paths.
25
26
* Indicates path to passive controller 2145 is a SVC which has active/passive nodes for a LUN DS4000, DS5000 and DS3950 also have active/passive controllers IOs will be balanced across paths to the active controller
27
28
29
30
depend on model and AIX level uses MPIO and is recommended not supported on VIOS yet for these disk subsystems so use MPIO
MPIO is strategic
SDDPCM SDDPCM
requires fcsA be connected to controllerA and fcsB connected to controllerB with no cross connections IO for a LUN goes to its primary controller
Unless the paths to it fail, or the controller fails, then the other controller takes over the LUN
Storage
administrator assigns half the LUNs to each controller vary among models and AIX levels
31
AIX_AAPCM - MPIO with active/active controllers AIX_APPCM - MPIO with active/passive controllers AIX_SDDAPPCM - SDDPCM AIX_fcparray - RDAC
2010 IBM Corporation
32
Frame id 0: Storage Subsystem worldwide name: 608e50017be8800004bbc4c7e Controller count: 2 Partition count: 1 Partition 0: Storage Subsystem Name = 'DS-5020' hdisk LUN # Ownership hdisk4 0 A (preferred) hdisk5 1 B (preferred) hdisk6 2 A (preferred) hdisk7 3 B (preferred) hdisk8 4 A (preferred) hdisk9 5 B (preferred)
# sddpcm_get_config Av
33
Some older systems dont support FC boot, if in doubt, check the sales manual SAN disk configuration Create the SAN LUNs and assign them to the system's FC adapters WWPNs prior to installing the system
For
AIX installation Boot from installation CD or NIM, this runs the install program
When
you do the installation you'll get a list of disks that will be on the SAN for the system the disks for installing rootvg You may want to know the SAN disk volume IDs if you want a specific LUN for rootvg. You should also see the size of the disk, so that's a good clue about which one(s) to use for rootvg.
Choose
34
problems can cause loss of access to rootvg ~ not an issue as app data is on SAN anyway loss of system dump and diagnosis if loss of access to SAN is caused by a kernel bug to change multi-path IO code Not an issue with dual VIOS
Potential Difficult
35
rootvg to internal SAS disks, e.g., using extendvg, migratepv, reducevg, bosboot and bootlist, or use alt_disk_install the multi-path code rootvg back to SAN versions of AIX require a newer version of SDD or SDDPCM
Change Move
Newer
Follow procedures in the SDD and SDDPCM manual for upgrades of AIX and/or the multi-path code Not an issue when using VIO with dual VIOSs
36
How this adapter is CONNECTED False Dynamic Tracking of FC Devices True FC Fabric Event Error RECOVERY Policy True Adapter SCSI ID False FC Class for Fabric True
37
Documentation
Infocenter Multiple Path IO http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/ baseadmndita/dm_mpio.htm SDD and SDDPCM Support matrix: www.ibm.com/support/docview.wss?rs=540&uid=ssg1S7001350 Downloads and documentation for SDD www.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D430&uid=ssg1S4000065 &loc=en_US&cs=utf-8&lang=en Downloads and documentation for SDDPCM: www.ibm.com/support/docview.wss?rs=540&context=ST52G7&dc=D430&uid=ssg1S4000201 &loc=en_US&cs=utf-8&lang=en
38