PowerPath Configuration and Administration - SRG

Welcome to PowerPath Configuration and Administration.
Copyright ©2015 EMC Corporation. All Rights Reserved. Published in the USA. EMC believes the information in this publication is
accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES
OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. The trademarks,
logos, and service marks (collectively "Trademarks") appearing in this publication are the property of EMC Corporation and other parties.
Nothing contained in this publication should be construed as granting any license or right to use any Trademark without the prior written
permission of the party that owns the Trademark.
EMC, EMC² AccessAnywhere Access Logix, AdvantEdge, AlphaStor, AppSync ApplicationXtender, ArchiveXtender, Atmos, Authentica,
Authentic Problems, Automated Resource Manager, AutoStart, AutoSwap, AVALONidm, Avamar, Bus-Tech, Captiva, Catalog Solution, C-
Clip, Celerra, Celerra Replicator, Centera, CenterStage, CentraStar, EMC CertTracker. CIO Connect, ClaimPack, ClaimsEditor, Claralert
,CLARiiON, ClientPak, CloudArray, Codebook Correlation Technology, Common Information Model, Compuset, Compute Anywhere,
Configuration Intelligence, Configuresoft, Connectrix, Constellation Computing, EMC ControlCenter, CopyCross, CopyPoint, CX,
DataBridge , Data Protection Suite. Data Protection Advisor, DBClassify, DD Boost, Dantz, DatabaseXtender, Data Domain, Direct Matrix
Architecture, DiskXtender, DiskXtender 2000, DLS ECO, Document Sciences, Documentum, DR Anywhere, ECS, elnput, E-Lab, Elastic
Cloud Storage, EmailXaminer, EmailXtender , EMC Centera, EMC ControlCenter, EMC LifeLine, EMCTV, Enginuity, EPFM. eRoom, Event
Explorer, FAST, FarPoint, FirstPass, FLARE, FormWare, Geosynchrony, Global File Virtualization, Graphic Visualization, Greenplum,
HighRoad, HomeBase, Illuminator , InfoArchive, InfoMover, Infoscape, Infra, InputAccel, InputAccel Express, Invista, Ionix,
ISIS,Kazeon, EMC LifeLine, Mainframe Appliance for Storage, Mainframe Data Library, Max Retriever, MCx, MediaStor , Metro,
MetroPoint, MirrorView, Multi-Band Deduplication,Navisphere, Netstorage, NetWorker, nLayers, EMC OnCourse, OnAlert, OpenScale,
Petrocloud, PixTools, Powerlink, PowerPath, PowerSnap, ProSphere, ProtectEverywhere, ProtectPoint, EMC Proven, EMC Proven
Professional, QuickScan, RAPIDPath, EMC RecoverPoint, Rainfinity, RepliCare, RepliStor, ResourcePak, Retrospect, RSA, the RSA logo,
SafeLine, SAN Advisor, SAN Copy, SAN Manager, ScaleIO Smarts, EMC Snap, SnapImage, SnapSure, SnapView, SourceOne, SRDF, EMC
Storage Administrator, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix, Symmetrix DMX, Symmetrix VMAX,
TimeFinder, TwinStrata, UltraFlex, UltraPoint, UltraScale, Unisphere, Universal Data Consistency, Vblock, Velocity, Viewlets, ViPR, Virtual
Matrix, Virtual Matrix Architecture, Virtual Provisioning, Virtualize Everything, Compromise Nothing, Virtuent, VMAX, VMAXe, VNX, VNXe,
Voyence, VPLEX, VSAM-Assist, VSAM I/O PLUS, VSET, VSPEX, Watch4net, WebXtender, xPression, xPresso, Xtrem, XtremCache,
XtremSF, XtremSW, XtremIO, YottaYotta, Zero-Friction Enterprise Storage.
Revision Date: 02/2015

Revision Number: MR-1WP-PPADMN
Copyright 2015 EMC Corporation. All rights reserved. PowerPath Configuration and Administration 1
This course focuses on the details for implementing and managing a SAN environment using
PowerPath. It reviews the commands used to display and manipulate the PowerPath configuration
as well as troubleshooting of problems within a SAN.
Upon completion of this module, you should be able to explain path failover operations under
PowerPath management and define path and device attributes.
The lesson explains how PowerPath arranges paths and handles I/O failures and how PowerPath
works with different array configurations.
PowerPath has traditionally been viewed only as path management software supporting load
balancing and failover. With the addition of migration and encryption functionality, PowerPath has
expanded its scope of functionality as well as its supported platforms. Centralized licensing
platforms as well as standardized interfaces allow for the bundling of PowerPath Products
together.
• PowerPath Multipathing, sometimes referred to simply as PowerPath, is the same trusted
product it has always been. The product automates data path management and failover and
recovery, as well as optimizes load balancing to ensure application availability and
performance. PowerPath Multipathing runs on Windows, Solaris, HP-UX, and Linux.
• PowerPath/VE automates and optimizes data path pools in virtual environments such as
VMware to ensure business process availability and performance and the ability to scale-out
mission-critical applications.
• PowerPath Viewer is a centralized monitoring utility that provides more visibility into
PowerPath-supported storage environments. It consolidates events and allows customers to
view and monitor PowerPath hosts through a single graphical user interface (GUI).
• PowerPath Configuration Checker (PPCC) is a software program that verifies that a host is
configured to the hardware and software required for PowerPath multipathing features (failover
and load-balancing functions, licensing, and policies) as specified in the EMC Support Matrix
and in the EMC Simple Support Matrix.
• PowerPath Migration Enabler leverages the same underlying technology as PowerPath
Multipathing to enable data mobility technologies to perform data migrations to and from any
network or supported array type. This includes EMC and qualified non-EMC storage. The
product allows completion of these tasks without application downtime.
• PowerPath Encryption uses software agents on hosts and centralized RSA® Data Protection
Manager appliances for key management to secure data from the host to disk on storage. This
product will reach end of service life (EOSL) January 31, 2016 and will not be supported after
that date.
PowerPath is host-based software that provides storage I/O path management. It operates with
various operating systems and storage arrays to intelligently manage the I/O paths that connect
them.
The illustration shows a typical PowerPath configuration where there are multiple paths between
the host and storage devices in the array. Each host path has its dedicated host bus adapter
(HBA) port, cable and Fibre Channel or iSCSI switch port. Through a switched SAN, it is possible
for any host HBA port to establish a connection to any storage device across a wide selection of
path components between the SAN switch and storage system interface port.
This is typically done with Fibre Channel Zoning or iSCSI Discovery Domains, which limit the full
number of available pathways which PowerPath can take advantage of through the SAN fabric.
Due to operating system limitations, without path management software installed there can be
only one active path to each storage device.
When I/O path components are placed under management of path failover and load balancing
software such as PowerPath, the ability to access devices through multiple paths increases both
application availability and I/O efficiency. To best take advantage of this benefit, the devices
should be assigned equally across all available storage array interface cards. This ensures that the
full capacity of all array interfaces are being used.
The PowerPath driver groups all equivalent paths into a volume path set, creates a volume path
set for each multi-ported LUN in the array, and then populates the path set with all the available
paths to the device.
Once PowerPath creates a volume path set, it can use any other path in the set to service an I/O
request. If a path fails, PowerPath can redirect an I/O request from that path to any other viable
path in the set. This redirection is transparent to the application. Occasionally, a short delay
occurs due to the low level disk driver trying to recover from an error or waiting for a request to
time out.
To distinguish between paths to the same volume and paths to a different volume, PowerPath
uses a unique volume - or LUN - serial number.
PowerPath performs load-balancing on a path set with more than one active path, based on the
path set’s load-balancing policy. By balancing the I/O load, PowerPath takes full advantage of all
active paths to the LUN. Applications run faster when the I/O is not constrained to a single path,
as is the case without PowerPath installed on the host.
PowerPath cannot redirect I/O to an alternate path until the HBA driver recognizes the path failure
and returns a failure message to the PowerPath driver. Therefore, it is necessary to set the
timeout value in the HBA driver and HBA according to the EMC Support Matrix and PowerPath
documentation. Setting these values incorrectly can increase the length of time PowerPath
requires before failing over an I/O request.
Load-balancing occurs only in one direction; from the host to the storage system. There is no
load-balancing from the storage system to the host.
PowerPath maintains a table of available paths in its host memory space and refers to this table
when determining which path to use.
This illustration shows an example of multi-pathing on a VMAX. In an active/active storage array,
if multiple interfaces exist to a LUN, they all provide equal access to the logical device. Active-
active means all interfaces to a device are active simultaneously.
In a configuration that includes an active-active array, PowerPath can spread the work load across
all paths. In addition, if a path fails, PowerPath will simply rebalance the I/O across the remaining
available paths.
PowerPath works differently with arrays that are active/passive. In this example, a LUN is
assigned to port 0 and port 1 on storage processor, or SPA. In this active/passive system, SPA is
designated as the primary or active route to the device and all I/O is directed down the paths
through SPA to the device. PowerPath load balances I/O across these active paths as shown by
the green arrows.
The LUN can also be accessed through SPB but only after it has been reassigned to Storage
Processor B, or SPB. These paths are referred to as a passive paths. PowerPath does not send I/O
down the passive paths. These paths are highlighted with red rectangles.
When all active paths to a LUN become unavailable, the LUN must be logically moved to the other
storage processor. This reassignment may be initiated by the functioning SP, but if PowerPath is
installed on the host, the reassignment is initiated by PowerPath, which instructs the storage
system to make the reassignment.
On a VNX array, these reassignments are known as trespassing. Trespassing can take several
seconds to complete; however, I/Os do not fail during this process. After devices are trespassed,
PowerPath detects the changes and seamlessly routes data via the new route, through the new
SP. After a trespass, logical devices can be trespassed back to their original paths.
In order to take advantage of all array interfaces, the devices should be assigned equally among
interface cards. This ensures that the full capacity of the array interfaces is being used.
In this illustration, the port on the storage array marked with a red X has been disabled using
powermt set port disable=true dev=A.
When you disable a storage array port, all logical paths to all devices configured through that port
are marked dead, and I/O is no longer sent to the port. To disable a port, enter the name of a
native-named device configured through the port. To determine which devices are configured
through a particular port, use the powermt display and powermt display dev command
output.
In this example, the storage port was disabled and device B became unavailable because it was
only mapped to that single array port. The logical path through that FA to device A was also
marked as dead. However PowerPath manages the logical paths through all HBA and storage
ports so that device A can still be accessed through the other array ports to which it is mapped.
The PowerPath command powermt display port_mode reports the mode of all ports on
connected arrays.
Note: A disabled port state persists across host reboots; however, the complete suppression of
I/O from the host to the disabled array port is not guaranteed during early boot (of the host)
before the PowerPath driver starts.
This lesson explains how PowerPath monitors the paths from the HBA to a LUN and the state and
mode in which these paths can be placed.
PowerPath keeps track of each HBA and the states of all paths from that HBA to each LUN. In this
illustration there are four logical paths to both LUNs A and B. The paths from one HBA to the
LUNs are highlighted in red. These paths are able to simultaneously access LUN’s A and B because
any HBA port to any storage array port connection can be made through switching in the SAN
network. This being the case, PowerPath monitors the state of the HBA as part of any logical path
through the SAN to the LUNs. From the PowerPath driver perspective, an HBA path to a LUN can
be in one of the following states: optimal, degraded, or failed.
The PowerPath command powermt display reports the state of all logical paths that originate
from an HBA.
The HBA marked with a red X has been disabled using the powermt disable hba command. The
command disables all paths configured through the specified HBA. These logical paths are shown
in red. When an HBA is disabled, all I/O on the HBA port is stopped, and all devices configured
through this port are closed. These devices are reported as dead in the powermt display dev
command output.
In this example, all logical paths through the HBA to devices A and B are closed (dead) and are no
longer accessible through the disabled HBA. There remains another viable path set through the
other HBA to the LUNs (shown in green). PowerPath can send I/O to the LUNs through this other
HBA port.
The PowerPath command, powermt display hba_mode, reports the mode for a path that
originates from an HBA.
PowerPath manages the state of each path to each LUN independently. From the PowerPath
driver's perspective, a path is alive or dead.
In the example, one logical path to the LUNs has a red X. It includes the HBA, cable to the switch,
the switch, the cable from the switch to the array, and the LUN.
The PowerPath powermt display dev command reports the state of the logical paths to a LUN.
When PowerPath connects to Active/Passive storage like an EMC VNX array, paths connected to
the SP that owns a LUN operate in Active mode, and paths to the other SP are put in Standby
mode.
When a path is active, load-balancing is performed for a device with more than one active path.
Load-balancing and failover are based on the policy set for the device. As shown in the
illustration, for active/passive and ALUA-compliant storage systems, I/O is load-balanced among
the ports of the storage processor which owns the LUN. In this example, active paths go through
SPA, which owns LUN A.
When a path is in standby mode, the path is held in reserve. Standby paths will only be used if
all active paths have failed, and LUNs are trespassed from one SP to the other SP. PowerPath will
initiate the trespass operation when it senses that all active paths are down.
A path that is reported as unlic indicates that unlicensed PowerPath is running for a Symmetrix
VMAX or CLARiiON/VNX storage system. In this scenario, all paths are marked unlicensed except
one path to each SP. Unlicensed paths cannot become candidates for path failover. Policies are
discussed later in this module.
Use the PowerPath powermt set mode command to modify the mode of the path to the LUN.
Host Registration for EMC arrays is a process that sends special I/O commands down each path to
an EMC array to register the initiators. The information sent in the registration includes the
initiator type, hostname, and IP address. This information is used by the array to determine the
correct SCSI behavior to present to the host and in the EMC array to display hosts for connecting
to storage. Host Registration is carried out automatically and simplifies the configuration process.
However, at present PowerPath Auto Host Registration cannot register a new host on an array as
it needs valid LUNs to send the SCSI commands, and hence the initial registration has to be done
manually or with the help of array agents. The PowerPath code is designed to support VNX, VMAX,
VPLEX and XtremIO.
This module covered PowerPath automatic path failover and attributes.
Upon completion of this module, you will be able to navigate PowerPath Management Interfaces,
view your Environment using PowerPath Tools, explain PowerPath Path Testing and Restore,
configure PowerPath Dynamic load-balancing, enable and disable Array and HBA ports, and
reconfigure PowerPath.
This lesson covers the PowerPath user interfaces, the operating systems the PowerPath CLI and
Administrative Console support, and the required environment to use PowerPath CLI and
Administrator GUI.
PowerPath runs on many Operating System platforms, and there is a command line interface for
each one. However, all CLI commands are not available on all platforms. This table shows which
commands are available for each platform. Many of these commands have additional arguments
to complete a specific task. Not all arguments of a command are available for all platforms. Use
the help pages to get a listing of commands and arguments supported on a given platform.
In this course, we will cover the most common commands:

• powermt - This is the main command used to administer PowerPath. It is used to monitor
and manage PowerPath devices and paths.
• emcpreg - This command is used to install and manage Licensing on the PowerPath host.
• emcpadm - This command is used to manage Pseudo Devices.
• emcpcfg - This command is used to save and restore PowerPath configurations.
• emcphostid - This command is used to set a host ID.
The powermig command is covered extensively in the EMC training course: PowerPath Migration
Enabler Implementation.
The rpowermt command is covered in the EMC training course: PowerPath/VE Implementation
and Configuration.
Here is a partial list of the powermt commands and their functions. Note that not all of these
commands are supported on all operating system platforms.
This list of supported commands can be displayed by issuing the powermt help command at the
OS command line.
The EMC PowerPath 6.x Family CLI and System Messages Reference provides complete
information for all commands, including command line syntax and arguments within each
supported operating system.
Here is an example of the structure of a typical PowerPath command and argument fields when
using the CLI interface. Powermt is the most commonly used PowerPath CLI command.
This slide lists the access level that is required to run the PowerPath management utilities.
The PowerPath Administrator GUI for Windows consists of two applications: the PowerPath
Administrator Console and PowerPath Monitor. They are automatically installed and started as
part of the PowerPath installation.
There are two methods of starting the PowerPath Administrator Console in Windows 2012.
In the first method, you start at the main screen and click on the tile for PowerPath Administrator.
The second method is used if you are already on the Windows desktop. Simply right-click on the
PowerPath icon found in the Windows system tray, and then click PowerPath Administrator.
The PowerPath Administrator GUI runs as a snap-in program within Microsoft Management
Console. It is used to configure, monitor, and manage PowerPath devices.
The PowerPath Monitor continuously monitors the status of the PowerPath configuration and can
be set to alert with visual and audible cues when the status of the PowerPath configuration
changes. Consult the PowerPath Installation and Administration Guide for Windows for details on
how the PowerPath Monitor icon reports configuration status.
This graphic shows a sample of the PowerPath Administrator Console. The PowerPath
Administrator has two panes. The scope pane displays PowerPath objects in a hierarchical list that
can be expanded or collapsed. The results pane provides a view of configuration statistics for
PowerPath objects selected in the scope pane. PowerPath Monitor (available on the taskbar of the
Windows desktop) continuously monitors the status of the PowerPath configuration. PowerPath
Monitor can be set to alert when there are operational status changes.
PowerPath Administrator icons represent the PowerPath objects in the configuration. The type of
icon reflects the state or status of the particular object selected.
The table lists the PowerPath Administrator icons that are documented in the EMC PowerPath for
Windows Installation and Administration Guide.
This lesson covers how to use the PowerPath display command.
Use the powermt display command to display information about HBAs and/or devices configured
for and managed by PowerPath.
Under the Host Bus Adapters, ### is the number PowerPath assigned to the HBA. The value is
preserved across boots but is not preserved after configuration changes.
The Host Bus Adapters Hardware Path is the host specific hardware address where the HBA is
installed. The values in the HW Path column are platform specific.
This example shows the results of executing the command on a Windows host. There are two
HBAs in the host (HBA 5 and HBA 6).
This example shows the same command output as shown on the previous slide.
The first lines of output list the storage system types and the logical device count for each of
those systems. This is the total number of unique logical devices sorted by array type that are
configured by PowerPath and accessible by the host. In this example, the host is connected to a
VNX with five LUNs.
The I/O paths Summary is the status of the paths originating from this HBA. An optimal status
means all paths are alive (usable). A status of degraded means one or more, but not all, paths
from this HBA are dead (not usable). A status of failed means all paths are dead and no I/O is
passing through this HBA.
The I/O Paths Total is the total number of paths that originate from this HBA. The maximum
number of logical devices supported by one HBA is platform specific. In this example, as can be
seen from the illustration, there are five LUNs that can be connected on two paths per HBA. So
the total is ten I/O paths per HBA.
This slide continues the explanations we left off with on the previous slide.
I/O Paths Dead displays the total number of paths originating from this HBA that are dead.
These paths are dead (meaning not usable) because of a path failure or a change to the
configuration. This value is modified when a path fails or is restored.
Stats IO/sec displays the average number of I/Os sent across each HBA each second. This field
is blank for powermt display, unless it is used with the every parameter. Subsequent powermt
display iterations show values in the I/O sec column. This parameter is discussed later in this
course.
Stats Q-IOs is the total number of outstanding I/Os on the HBA queue at the time the command
is issued.
Stats Errors is the total number of times any logical I/O paths on this bus transitioned from alive
to dead. This is always equal to or less than the total number of HBA I/O path errors. It is cleared
at system boot time or when powermt restore executes.
The class=<Array Type> parameter can be added at the end of any powermt display
command to limit the results to only show devices/paths from the array you specify. This is handy
id you have two or more array types attached to your host, and you want to limit the command
output to only one array type. If the value is not specified, the class variable defaults to all. This
default can be modified by altering the PP_DEFAULT_STORAGE_SYS environment variable within
the OS.
This table lists each array class that may be specified.
Shown is the powermt display dev=1 output for a LUN that resides in a VNX array. “1” is a
shortened version of the full PowerPath pseudo name harddisk1 on a Windows system. The
information that is shown is similar to that for a Symmetrix VMAX array, but is VNX specific.
The output shows that there are two paths through each HBA to LUN 0 (also known as Dev 1, or
harddisk1).
The load-balancing policy is CLAROpt (CLARiiON Optimized) which is the default policy for VNX
arrays. Load balancing policies are discussed later in this course. It can be seen in this display
that the LUN is currently owned by SP A.
The next step is to use the output of the powermt display paths command to determine the
number of paths from the HBA to the array. The example shows the output for the command on a
Windows host that matches the illustration.
The command output points out that there are two paths from each HBA to the VNX. In this
example, each HBA has five logical devices mapped to a SP port in the VNX.
Note that the SP ports, such as SP A1, are listed more than once. This means that each HBA port
is zoned to two SP ports, for example SP A1 and SP B1.
The next step is to look at the output of powermt display ports. The output shows the same
output of the powermt display paths except that it is a sum of the number of devices that have
a path through the HBA to the array port. Otherwise, the port number and storage system ID are
already known.
EMC PowerPath Viewer is a centralized monitoring utility that provides more visibility into
PowerPath-supported storage environments. It consolidates events and allows customers to view
and monitor up to 500 PowerPath hosts through a single graphical user interface (GUI).
With EMC PowerPath Viewer, PowerPath customers can view hosts, host groups, LUNs, individual
paths to each LUN, and buses. PowerPath Viewer also provides email alerts regarding changes in
the status of PowerPath devices.
PowerPath Viewer includes two components:

• PowerPath Viewer Console: This interface displays information and alerts about PowerPath-
supported devices in windows and panes that can be tailored by customers to meet their
specific data center needs.
• PowerPath Management Component: This host-based software monitors host, LUN, data
path, and bus events and sends information and alerts to the PowerPath Viewer Console over
IP. The PowerPath Management Component supports Windows (including Hyper-V), Linux, and
VMware vSphere operating systems and appropriate PowerPath versions.
This lesson describes the characteristics and benefits of proactive path testing and autorestore.
Autoprobe identifies failed paths by periodically probing the paths PowerPath believes to be
alive; especially paths that have not been recently used. Testing helps prevent application I/O
from being sent on dead paths that PowerPath otherwise would not detect as dead. As a result,
I/O timeout and retry delays are reduced. Periodic testing of live paths is a low priority task. It
does not test all paths within a specific time, but rather tests all paths within a reasonable time.
This testing scheme minimizes the impact that the testing has on application I/O. PowerPath does
path testing on the paths that have not been tested for at least one hour and on idle paths that
have not been used for I/O within the last minute.
If one probe fails down a path, it kicks off error checking routines.
Typically, all live idle paths are tested at least hourly, although this is not guaranteed. On an
active system, failed paths are recognized immediately when an application I/O fails. In this case,
path testing is triggered promptly because of the I/O failure.
PowerPath autorestore periodically tests dead paths and automatically restores them to service.
Like periodic testing of live paths, periodic autorestore is a low priority task. It is not designed to
restore a path immediately after it is repaired, but rather to restore the path within a reasonable
time after it is repaired. Dead paths are tested when the path test process is run, provided the
paths have not been tested for at least one hour. This frequency limits the number of I/Os that
fail, so the impact on normal operations is negligible.
Using a combination of autoprobe and autorestore features means that if a path fails, it is marked
failed but the application continues to run. Autorestore then tests the path’s viability and
automatically restores the connection.
The time it takes for all paths to be restored varies greatly. In lightly loaded or small
configurations, paths typically are restored within an hour after they are repaired (on average,
much sooner). In heavily loaded or large configurations, it may take several hours for all paths to
be restored after they are repaired because periodic autorestore is preempted by higher priority
tasks.
The fastest way to restore paths is to restore paths manually using the PowerPath CLI command,
powermt restore.
Use powermt set periodic_autorestore=on/off to enable or disable PowerPath autorestore.

By default, Autorestore is enabled when PowerPath is installed.
PowerPath periodically runs the path test process. This process sequentially visits each path and
tests it if it meets the following requirements:
• Live paths are tested periodically if they have not been tested for at least one hour and are
idle.
• Dead paths are tested periodically if they were marked for testing at least one hour ago and
are idle.
• Any paths are marked for testing as a result of the conditions listed here.
Tests are spaced out, so at least one path on every HBA and every port is tested at least hourly.
The more paths that need testing, the longer it takes to complete the path test process. As a
result, it is hard to predict exactly when a path will be tested.
When a path state change is detected, it is propagated quickly to all related paths. After all paths
are visited and those marked for testing have completed their tests, the process sleeps for 10
seconds, and then restarts. This 10 second period is a compromise between using non-application
system resources (CPU and I/O cycles) and keeping the state of paths current so that the
maximum number of paths are always available for use.
This lesson covers how to identify the appropriate load-balancing policy and change the load-
balancing policy of devices within your PowerPath configuration.
PowerPath has an automated load-balancing policy configuration mechanism. When the operating
system boots and PowerPath is initialized, device-specific information on the managed LUNs is
read by PowerPath. The optimized load-balancing policy is automatically set on a per-LUN basis. If
multiple arrays are accessible on the SAN and zoned to the PowerPath host, each LUN that is
masked to the sever will be assigned the policy that provides the best load-balancing algorithm.
There are multiple load-balancing policies intended to support various application requirements.
Optimized policies are the default policies for EMC Symmetrix VMAX, and VNX devices. Use of
other policies for EMC arrays should be used only under the direction of EMC Customer Support.
EMC has developed multiple load-balancing policies intended to support the most common
application requirements in production and test environments. Here is a list of the default load-
balancing policies available when the PowerPath installation is licensed:
CLARiiON Optimized - EMC proprietary algorithm that is designed for VNX and CLARiiON
models. This policy is recommended for all VNX and CLARiiON implementations. Asynchronous
Logical Unit Access (ALUA) and non-ALUA modes are also supported. When used with ALUA-
compliant arrays, this policy recognizes the difference between optimized and non-optimized
paths as specified in the ALUA design. The Symmetrix Optimized and CLARiiON Optimized policies
are the default policies for Symmetrix VMAX and CLARiiON/VNX arrays.
Symmetrix Optimized - EMC proprietary algorithm that is designed for past and present
Symmetrix models. This policy is recommended for all Symmetrix implementations including
VMAX.
Adaptive - A load-balancing and failover policy for PowerPath devices in which I/O requests are
assigned to paths based on an algorithm that takes into account path load and logical device
priority.
Note that Symmetrix Optimized and CLARiiON Optimized policies should be used by most EMC
customers for Symmetrix, VNX, and CLARiiON arrays, and Adaptive used with VPLEX and XtremIO
storage systems. Use of other policies with EMC arrays should be considered only under the
direction of EMC Customer Support. For more information on other PowerPath policies, consult the
EMC PowerPath Product Guide and Release Notes for the PowerPath version installed.
Here is listed the load-balancing policies that can be assigned for devices belonging to all
supported arrays when PowerPath is licensed.
Least Blocks and Least I/O policy are based on observed I/O load to the device. These policies
are commonly used in large database environments. This policy looks at the number of blocks
going through the path or the size of the data overall going through, regardless of the number of
I/O’s.
Round Robin is useful when all active paths to a LUN will be used at the same time. I/O requests
are assigned to each available path in rotation.
Stream IO uses the same path to the volume that was most recently used for I/O. If the next
I/O exceeds the volume’s threshold value, a new path is selected based on the adaptive policy
algorithm. This policy may be applied to all arrays that PowerPath supports.
All of these polices include path failover to any logical path that is part of a volume path set to the
device.
For more information on each policy, consult the PowerPath Product Guide and Release Notes for
the version of PowerPath installed.
Here are special load-balancing and failover policies that can be used on licensed and unlicensed
PowerPath installations. All of these policies are not recommended; they are only used in special
circumstances because they disable load balancing and/or failover capability.
With the No Redirect policy in effect, PowerPath does not load balance or support path failover.
This policy is not available on platforms attached to a VNX or CLARiiON array. It is the default
policy for Symmetrix systems on platforms with a valid PowerPath Base license. It is also the
default policy for all third-party storage systems on platforms without a valid PowerPath license.
No Redirect policy is not recommended for use in production environments. It can be used as a
diagnostic tool.
When the Request policy is used, PowerPath does not load balance but does support path
failover to an alternate path. Request is the default policy for VNX and CLARiiON storage systems
on platforms with a PowerPath Base license. It is also available on platforms with a valid
PowerPath license that are attached to Symmetrix, VNX, CLARiiON, Invista, or supported 3rd
party arrays.
The Basic Failover policy protects against VNX and CLARiiON SP failures, Symmetrix FA port
failures, and backend failures. It allows non-disruptive upgrades to work when running PowerPath
without a license key. It does not protect against HBA failures when load-balancing is not in
effect. I/O routing on failure is limited to one HBA and one port on each storage system interface.
For more information on each policy, consult the PowerPath Product Guide and Release Notes for
the version of PowerPath installed.
Use the powermt display dev CLI command, or the disk view in the PowerPath Administrator
GUI, to determine the current load-balancing policy for a device. To change the load-balancing
policy, use the powermt set policy CLI command.
In the example, the powermt set policy CLI command is used to change the load-balancing policy
for device dev=2, and from CLARiiON optimize to round robin. A powermt display dev=2 is run to
verify that the load-balancing policy has changed.
Load-balancing policies can be set on a device-by-device basis or for all devices. One device or a
subset of devices can be using one load-balancing policy and another subset can be using
another. The powermt set policy command also accepts the class parameter to modify only the
devices belonging to an array class.
This example illustrates two methods to change load-balancing policies using the PowerPath
Administrator GUI and Windows.
You can right-click a disk in the Disks tree and choose Change Policy to in the drop-down menu.
A drop-down menu displays the policies that can be set on the disk. Choose a policy from the list
to change the current policy.
You may also change the load-balancing policy by choosing the Policy tab in the Disk Properties
window. The Disk Properties window is accessed by right-clicking on the disk and choosing
Properties from the drop-down menu. The window has a box that lists the available policies.
Choose a policy and click Apply to change the policy.
This lesson covers using the write_throttle setting to tune the PowerPath environment,
introduces the stream I/O threshold and explains the performance implications of setting a
path in standby verses active mode.
Write throttling is enabled to limit the number of queued writes to the common I/O queue in the
HBA driver; instead, the writes are queued in PowerPath. As a result, read requests do not get
delayed behind a large number of write requests. Write throttling is disabled by default.
powermt set write_throttle_queue sets the write throttling queue depths for a storage system
port connected to a specified device. The queue depth setting limits the number of writes to all
devices enabled for write throttling that can be outstanding (from PowerPath's perspective) on the
storage system port. The queues are allocated within PowerPath, one per storage system port.
Use the powermt set streamio_threshold command only if “si” (streamio) policy is in effect.
This threshold determines if PowerPath will continue to stream I/O down a given path, or switch
to a different path.
Also shown here are typical event messages to indicate if a bus has died or come back to life.
PowerPath allows you to form a channel group of dedicated paths to a logical device to increase
application performance. (Note however, that reserving paths for one application makes those
paths unavailable for other applications, potentially decreasing their performance.)
Channel groups keep a second set of paths in reserve in case the first set fails. As a form of
manual load balancing, channel groups reserve bandwidth more precisely than automatic means.
Channel groups require at least two paths; they work best in environments with more than two
paths and at least two separately managed applications on the same host that use different
logical devices. You create a channel group by using the powermt set mode command to label a
group of paths to a logical device as active or standby. An application accessing one or more
logical devices designates one group of paths as active and another group as standby. A second
application accessing different logical devices designates the first group of paths as standby and
the second group as active. Each application has its own dedicated group of active paths, while
the overall configuration provides channel failover protection.
If a path in an application’s active group fails, the application’s I/O is redirected automatically to
another active path in the group. If all paths in the active group fail, the application’s I/O is
redirected automatically to the standby paths.
Note that the display at the top of this screenshot shows powermt display dev=2 output with
all paths active. We can set paths to standby mode at the device level.
This example shows how to set adapter 5 into standby mode.
Use the powermt set mode=standby hba=xx dev=yy CLI command to set all paths from a
specific HBA to a specific LUN into standby mode.
To verify, we execute the powermt display dev=2 CLI command again (as shown at the
bottom of the screenshot) and notice that the two paths from HBA 5 to LUN 2 are now in standby
mode.
By setting an adapter into standby mode, the adapter is not used to satisfy I/O’s unless all active
paths through the other adapters are unavailable. PowerPath automatically uses the standby path
when it is needed to satisfy I/O requests, but only when all other resources are unavailable.
Use powermt set mode=active hba=xx dev=xx to set the mode back to active.
This example illustrates two methods to set the path mode to standby using the PowerPath
Administrator GUI in Windows.
You may right-click on a disk in the scope pane and then choose Change Mode to and Standby
from the drop-down menu.
You may also set the path to standby mode using the Disk Properties window by choosing the
Mode tab and selecting the radial button next to Standby. In either case, the disk properties
window shows the current mode setting on the Details tab.
This lesson covers the process to disable and enable HBA and array ports using PowerPath and
identify HBA and array ports that have been disabled.
PowerPath may be used to disable host HBAs (on certain hosts) and array ports.
Disabling a HBA Port stops I/O to a port on a HBA and closes all devices configured through the
port. Disable a HBA when you want to replace a HBA. A disabled HBA is not persistent across host
reboots.
Disable array Port enables or disables a storage system port, and closes all devices configured
through the port. When you disable a port, all paths configured through the port are marked
dead, and I/O is not sent to the port. It is useful when I/O needs to be isolated away from the
port in order to perform maintenance on the port hardware. For example, disabling the
appropriate array ports to isolate paths to LUNs accessed through a disk array controller which is
to be replaced.
The disabled port state persists across host reboots; however, the complete suppression of I/O
from the host to the disabled array port is not guaranteed during early boot of the host. When
you disable a port with powermt set port disable in an R1/R2 boot failover environment, the
disabled port does not persist through a failover or failback. The command may be used with AIX
and Solaris platforms that are connected to storage by Fibre Channel, HP-UX platforms connected
by Fibre Channel and iSCSI, and Linux platforms connected by Fibre Channel and iSCSI, but only
where iSCSI is implemented at the host using iSCSI HBAs.
This example shows the steps to follow when disabling a HBA. First powermt display hba_mode
is executed to determine the current status of the HBAs and the devices accessible from the
HBAs. Next a powermt disable hba=1280 is executed. Finally, a powermt display
hba_mode is re-run to view the results of disabling the HBA.
This example shows the steps to follow when enabling a HBA. First, a powermt display
command is executed and we see the HBA port summary shows failed, with a count of 30 Dead
paths. Notice here that the error count is 0. This is because the dead paths are due to a user
configuration change and are not unexpected. This is a clue when diagnosing dead paths, to
determine whether someone has inadvertently disabled the HBA or if it is truly a dead path.
To enable the HBA in this example, we use the powermt enable hba=1280 and see the results
here. The HBA port has gone from failed to optimal.
This example from a Solaris host shows the steps to follow when disabling an array port. First, the
powermt display command is executed to determine the current status of the paths from the
HBAs to the array port and the devices accessible from the port. Next, disable array port using:
powermt set port_disable=true dev=c2t28d5so. Note that the device name of one of the
devices accessed through the port is specified. This device name is only used to identify the array
port that it uses. The command will disable the port for all devices, not just this one.
We then see the result of a powermt display port_mode, which shows us all ports accessible
by this host and their modes. In this example, we can see that port FA5bA has been set to
Disabled.
Another thing to keep in mind with this command is that it only disables the array port for all
devices on this host. Any other server accessing this same array port is not affected.
This example shows the steps to follow when enabling an array port. First, the user must do a
powermt display ports to show that all 3 paths through FA 5bA are dead. Notice here that there
are no errors counted. This is because the paths are dead due to user changes, not unexpected
changes. Use the powermt set port_disable=false dev=c2t28d5s0 to enable the port. You
can use this device or any other device that uses this array port. Use the powermt display
port_mode to verify the array port is enabled.
This lesson explains how to use the PowerPath CLI and Administrator GUI to save and load the
PowerPath configuration, add and remove devices and paths within the PowerPath configuration,
and reconfigure the PowerPath configuration.
This example shows how to execute a powermt save and powermt load using both the CLI and
the Administrator GUI.
Use powermt save (and its GUI equivalent) to save the PowerPath settings and configuration
data in an ASCII formatted file. The command forces PowerPath to save all logical device serial
numbers, policy, priority, write throttle setting, write throttle queue depth, and pseudo device
names. Powermt save also records the mode for each configured path, and the periodic
autorestore setting for each storage system class. For performance tuning purposes, you can use
powermt save to save different configurations under different filenames.
By default, PowerPath saves the configuration file to the powermt.custom file. (On Unix
systems, the file is saved in /etc.).
To save the configuration to a user-specific configuration file, use powermt save

file=<filename>. If a filename is not specified, the information is saved in the default file
powermt.custom.
Use powermt load to load a saved configuration file. Executing powermt load without specifying
a file name loads the default powermt.custom file from the default location.
The saved configuration is also loaded when the system is booting. The exception is AIX, which
always loads default settings at boot time. If the default filename is not used, the configuration
does not load automatically at boot time. In this case, you must load the configuration manually
with powermt load.
Use powermt config to configure all detected logical devices as PowerPath devices and add
these devices to the PowerPath configuration. The command also creates pseudo devices on
operating system platforms that support pseudo devices such as Solaris and AIX. The powermt
config command also configures all detected paths to PowerPath devices and adds these paths to
the PowerPath runtime configuration. Powermt config also configures all detected third-party
storage system logical devices as PowerPath devices if their storage system classes are set to
‘managed’.
The command adds paths to logical devices based on the storage system frame serial number and
the logical device serial number. Together, these values uniquely identify a logical device.
By default, powermt config adds devices under PowerPath’s control with the following settings:
• Symmetrix optimization (For Symmetrix-based arrays including VMAX), CLARiiON optimization
(for CLARiiON-based arrays including VNX), or Adaptive load-balancing and failover policy (for
other arrays);
• Priority of 0;
• Write throttling set to off;
• Write throttle queue depth of 256.
Powermt config does not remove previously configured paths when they become dead paths. Use
powermt remove to remove these paths.
The powermt config command is not available with PowerPath for Windows platforms, as Windows
does this automatically.
This slide lists the steps needed to add a new device to the PowerPath configuration. The
procedure applies to managed storage system classes only. It does not apply to any storage
system class that has been excluded from PowerPath control using the powermt unmanage
command. Also note that the procedure is an overview and therefore does not include the fine
points. Consult the PowerPath Product Guide for detailed information on each step. Also, the
procedure to use a new logical device is platform-specific and may require service interruption.
First, use the PowerPath CLI or Administrator GUI to view the existing environment. Verify that all
devices and paths are correctly configured to the host. Next, make the devices visible to the host.
This step could entail a number of steps as shown on the slide. Verify the devices are correctly
seen by the host before continuing with the next step.
Then use powermt config to reconfigure the PowerPath environment and add the new devices.
When the reconfiguration is complete, use the PowerPath CLI or GUI to verify the devices have
been added correctly. The next step is to adjust device-specific values such as load-balancing
policy and device priority.
Always save the PowerPath configuration using the powermt save command. This ensures that
PowerPath always loads the correct configuration.
Shown here is an example of using powermt config to add a device to the PowerPath
configuration. The powermt display output provides a view of the configuration prior to adding
the new device. Next, a new device is added using an array management tool (not shown). The
next step is to execute a powermt config to add the device to the PowerPath configuration. The
subsequent powermt display output now includes the new device(s).
Before using powermt config to add a device, it must be visible to the operating system.
The example screenshots were shot on a Solaris host. When making devices available to Solaris,
verify that the sd.conf file includes an entry for the target and LUN. Next, run the operating
system commands to configure the device. In the example, devconfig and disks commands
were used (not shown).
In this example, the before and after view of adding a device using the PowerPath Administrator
GUI is shown. On Windows, there is no need to use powermt config because PowerPath
automatically configures new devices. Prior to addition to the PowerPath configuration, the device
must be seen in Windows device manager (and you may have to use Device Manager to rescan
for new devices).
These are the general steps that should be followed when removing a path from the PowerPath
configuration.
Use the PowerPath CLI or Administrator GUI to view the existing environment. When using
PowerPath CLI, use powermt display commands to identify the paths that you wish to remove.
In other words, determine the HBA, zone, etc, that you want to remove. The output of this step
is the pathname for the path that is to be removed.
Use the powermt set mode command and specify the pathname found in the previous step to
set the path mode to standby. This is optional, but a good best practice as it ensures that the
paths are not in use when you unplug cables or HBAs. The next step is to remove the components
of the path whether it is a zone, or cable, etc. Next, run the powermt restore command to verify
that the path you removed in the previous step is dead.
The next step is to run the powermt check command to remove the dead paths from the
configuration file. When devices and adapters are added or removed, the configuration file may
not reflect these recent changes. Running the powermt check command synchronizes the
configuration file with the running configuration. If a path is marked dead or the serial numbers
encoded in the path configuration information do not match the serial numbers on the logical
device, powermt check prompts the user to remove the path.
Next, use the operating system-specific commands to remove the paths and all devices down the
path. This step includes removing the operating system device file and the PowerPath device that
corresponds to the device(s) being removed. Always remove the PowerPath device file first. Save
the configuration changes and then reboot the system for these to take effect. When the reboot is
finished, check the new PowerPath configuration.
Shown here is an example of the powermt check command discussed in the previous slide.
Notice that there are 27 total and one dead device through HBA 2304 on the first powermt
display. Then we see only 26 devices after running the check command.
This module covered PowerPath management interfaces, viewing your environment using
PowerPath CLI, path testing and restore, dynamic load balancing, enabling and disabling HBA and
array ports, and reconfiguring PowerPath.
Upon completion of this module, you will be able to monitor I/O throughput, monitor and
configure Path Latency, analyze error logs, troubleshoot your PowerPath Environment, use
PowerPath Configuration Checker to verify host configuration, and describe how PowerPath
Management Pack works with MOM and SCOM to monitor your environment.
This lesson explains how to use the PowerPath CLI and GUI to investigate and improve I/O
performance at the path and HBA level, other methods of improving I/O performance, and
considerations for modifying load-balancing and device priority settings to improve performance.
This slide lists the steps a PowerPath Administrator can follow to view current path I/O
performance and use the information to tune the PowerPath environment.
Start by viewing performance at the HBA level by using the powermt display every=xx
command. The value after the every parameter specifies how many seconds elapse between each
time the command is run. Any integer between 1 and 86400 is accepted as an argument for the
every parameter.
Use the information in the report to determine if the I/O is unbalanced across HBAs.
When unbalanced I/O is found, the administrator should use the powermt command with the dev
option to drill down and investigate the devices that are part of the path from the HBA. First, look
for dead path(s). If dead paths are present, the alive paths are handling more of a share of the
I/O. Check the load-balancing policy and the device priority. Try adjusting the load-balancing and
device priority values and check the results using powermt display. Load-balancing and device
priority settings can usually be changed during operation without interrupting the application.
The every=n parameter can be added at the end of any powermt display command to reprint the
results of the display command every n seconds. When using this feature, keep in mind that
interface delay time must be taken into account. For example, if you are connected through a
hyper terminal and the screen is printing slowly, you are better off setting n to a higher value to
avoid output that is constantly scrolling.
Here we see output from the command after five seconds.
This example shows repeated screen output on 5 second intervals after issuing the command
powermt display every=5.
Note that the first time the report runs the I/O per sec field is blank.
In the example, the number of I/O’s per second and the QIO’s column is shown in the red box.
The IO/Sec column displays the average number of I/Os sent across this bus each second.
The Stats QIOs column reports the total number of outstanding I/Os on this HBA now.
In the example, there are no dead paths to the devices. So, you can discount a performance issue
because of dead paths to devices.
There are a small number of QIO’s (two and four on each HBA path respectively), listed in the
report. These outstanding I/O’s may point to a problem where the paths are at maximum
utilization and tuning may be needed. In order to tune the paths, the powermt display dev
command is used to view load-balancing and device priority settings for the disks attached to the
system.
This screen capture shows the I/Os per second HBA performance view using the Administrator
GUI.
The IOs/sec and QIOs columns are boxed in red. In this snapshot example we have queued 10
and 14 I/Os on our two HBAs. To determine how much impact this has on performance, we need
to find out how much time it takes an I/O request to complete. We will see how to do that when
we look at monitoring path latency.
In this example, a portion of the powermt display dev=all every=5 output is shown. There is a
sample output for a Symmetrix model and VNX array.
When you specify the device option of the powermt display command, the IO/Sec column is
removed. However, the QIOs field is available to use as a gauge on how the paths to the device
are utilized.
In the Symmetrix portion of the output, there is a queued I/O on one path to the device
suggesting that there is enough traffic being sent down that path to make it busy enough to
queue up I/Os. This command produces output every 5 seconds, so observe the display for a
couple of minutes and check for queued I/Os. If I/Os are consistently queued up on one path,
then this device may be a candidate for tuning by adjusting the load-balancing policy for the
device. You may also see what effect of changing the device priority setting has on device
performance.
The VNX device has zero QIOs on all paths. There is probably no need to tune these devices.
This command can be used on a single device as well to limit the amount of data shown on each
repeat of the every parameter. Simply replace dev=all with dev=n where n is the pseudo device
you want to look at.
When an underperforming device is found, PowerPath can be used to adjust load-balancing and
device priorities to improve performance. Information on how to modify load-balancing and device
priority settings using the PowerPath CLI and Administrator GUI is covered later in this course.
It is important to understand the application I/O profile before changing load-balancing or device
priority values. For example, some applications perform sequential I/O while others perform
random I/O.
When adjusting device priorities, the key is to know in which devices the application data resides.
It is equally important to determine which devices contain the data to which the application needs
efficient access. An example is a database log device. The idea is not to set a higher device
priority on any of the other devices that share the path to the database log device. Doing so
improves performance to the wrong devices at the expense of the database log device. A better
solution is to raise the priority of the database log device.
Also listed are other methods of improving I/O performance to a disk. The list is not complete but
provides a good starting point.
Shown here are a few guidelines to keep in mind when modifying load-balancing policies.
In most cases, the default load-balancing policies for the array class delivers the best
performance. They are optimized to use with devices for the array types they support. As
mentioned, these policies are Symmetrix optimize (so), CLARiiON optimize (co), and adaptive
(ad). Others, such as round robin, are great for balancing I/O across all paths. Least blocks and
least I/O policies are best for applications that perform sequential I/O or block I/O’s.
The basic failover and request polices should not be used on a fully licensed PowerPath
environment. The reason is that these policies only enable failover functionality and do not
provide load balancing.
This lesson explains how to use the PowerPath CLI to configure and monitor path latency.
PowerPath 6.0 includes an I/O latency monitoring capability on UNIX, Linux, and Windows
platforms. Path latency enables an administrator to set a time threshold in seconds for how long it
should take a host I/O to complete. To use the path monitoring facility, it must be turned on
using a PowerPath CLI command. Administrators can use another PowerPath CLI command to
monitor the actual, current, and maximum latency and any incident where I/O latency exceeds
the threshold logged to the system log.
Path latency monitoring is a good tool to use when observing I/O performance on SAN’s. Some
SAN configurations that may benefit from I/O latency scrutiny are listed on this slide.
Starting with version 5.0, PowerPath monitors the time it takes I/O requests to complete on all
paths in your environment. When you enable or disable path latency monitoring, this setting
applies globally to all paths.
As shown here, the powermt display options command reports whether path latency
monitoring is enabled, and the path latency threshold, if one is set.
You may monitor the latency on all paths to your environment by first enabling the path latency
monitor. The next step is to set a path latency threshold to set a time interval in seconds within
which I/Os should complete. Each customer environment is different and therefore different
thresholds may be used.
Use powermt display latency to display the latency information including the last and longest
time required to deliver an I/O. The diagnostic information reported by path monitoring allows
you to determine normal I/O times for your environment, which can help to determine an
appropriate path latency threshold.
Use the powermt display options command to check whether path latency monitoring is turned
on or off. This example illustrates the command usage and representative output of a system that
has been turned off.
In order to turn on or enable path monitoring, use the command powermt set
path_latency_monitor=on. The next step is to set the threshold using powermt set
path_latency_threshold=n. In this example, the latency threshold is set to three seconds.
Then use the powermt display options output to view the path latency state and latency threshold
to ensure they were entered correctly.
These commands are often scripted to temporarily change the thresholds to higher amounts while
a batch job is running, or for a number of other reasons.
In this example, the administrator is monitoring the threshold using powermt display latency.
The results show the current and maximum latency for all devices.
To monitor latency over a period of time you can use the ‘every’ option of the display command.
For example, powermt display latency every=5.
The powermt display latency output includes the following informational columns:
• The Current column reports the time in microseconds it took the last I/O on the path to
complete.
• The Max column reports the time in microseconds it took the longest I/O on the path to
complete.
You can use the Current and Max values to determine the appropriate threshold value for your
environment.
In this example, the Max column for the VNX devices are well above the current values,
suggesting that the current latency is low and the maximum latency is roughly 0.54 seconds. In
the previous slide, we set the threshold to three seconds (3000 milliseconds or 3,000,000
microseconds), so we are well within limits and don’t need to increase our threshold.
For more detailed information, you can specify a particular device or all devices when monitoring
Path Latency. In this example, the output of powermt display latency dev=2 is shown. We can
see here that because this is an active/passive VNX array there is latency on the active paths
through SP-A, and the standby SP-B paths show zero latency. The current latency is well below
the maximum, and the maximum is 0.54 seconds, which is well below the 3 second threshold.
This lesson explains how to find and deconstruct PowerPath Error Log Entries.
PowerPath logs error messages to the Windows event log or the Unix / Linux system log.
Each error message includes a platform-specific prefix referencing PowerPath. Note in this
Example from a Windows host, that PowerPath messages show EMCpMpx as the source of the
event messages from PowerPath.
The error message also includes the message type and a character string indicating the type of
array to which the message applies. Message types are Panic, Warning, Error, Info, or Debug.
Please refer to the PowerPath CLI and System Messages Reference manual for information about
the error messages for Each PowerPath family product.
This example is the output of a powermt display command on a Solaris machine showing that
all paths to devices from HBA 2304 have failed. Notice that all 42 devices are dead and there are
42 errors. The path to the devices from the other HBA is still available as shown by the optimal
status.
On the bottom of the slide is a portion of the Solaris messages file (/var/adm/messages) showing
the messages that PowerPath wrote to the file when the path to the devices was lost. We can see
from this output that the problem occurred at 11:05:37 on Jan 7th. We can derive that all paths
through HBA 2304 have died to the Symmetrix with the serial number ending in 308. This error
log will detail all of the devices affected.
We can get the same information from a powermt display dev=all, but the error log is helpful
for Administrators. In most cases, an Administrator would have a tool that monitors the error log
and sends a notification when a path is dropped. This way the administrator can take action to
ensure that no data is lost and no applications go offline due to path failures.
This slide is an example of a log entry showing that the latency threshold has been exceeded. This
is good as an early warning system for your environment. If thresholds are being routinely
exceeded, then an administrator can be notified, troubleshoot the paths, and take steps to avoid
a future failure.
This lesson explains how to turn autorestore on and off, how to identify dead paths, what actions
to take when dead paths are found, and how to manually restore paths using the PowerPath
command line and Administrator GUI.
Use powermt set periodic_autorestore=on to enable PowerPath autorestore functionality if it
has been turned off. When autorestore is enabled, PowerPath periodically tests dead paths. If the
paths pass the test, PowerPath restores them to service. PowerPath then can use the restored
path for load-balancing and failover. Even when periodic autorestore is off, path testing continues
to be done on all paths to the devices, not just the paths that are dead.
When autorestore is off, PowerPath does not restore the path automatically.
Autorestore is enabled by default when PowerPath is installed.
The process of finding and resolving a path to a disk failure is shown on this slide. Use the
powermt display CLI command to determine if there are any dead paths to devices.
The output for the powermt display command used in step 1 shows that our host HW Path 6 is
degraded. That means we still have at least one path, but at least one path is dead on that HBA.
We can also see that one path is dead to each of our five devices.
In step 2, the powermt display dev=2 command is run to determine which path is having a
problem. In the example, it is VNX SP-B2 as shown. This is because we shutdown the port on the
SAN switch that connects to this storage array port.
In Step 3, the user determines the cause of the failure.
The administrator must determine the reason and follow the appropriate procedure to correct the
problem. If the problem is due to a configuration change, then reconfigure PowerPath to make the
change permanent. If the problem is due to a hardware problem, then fix the problem and restore
manually or wait for the autorestore process to restore the path.
Step 4 can be done automatically by the autorestore feature, or manually if autorestore is

disabled. To manually restore the path, run the powermt restore command. This command will
scan through all devices and retest any dead paths to see if they have been repaired. If they have
been repaired, then PowerPath restores them to an alive state.
If there are many devices visible to your host, you have the option of running powermt restore
hba=6 dev=2. This only touches the device specified and does not scan for dead devices. This
example shows all the paths to dev 2 though both HBAs are up and operational after we re-
enabled the SAN switch port that we had previously shutdown.
Shown here is a screenshot of the PowerPath Administrator GUI showing the same dead path to
device 002 that we saw using the CLI commands. Device 002 is owned by SP-A and we can see
IO on both of those paths. However we shutdown a port on the switch connected to SP-B2 (as
shown in the illustration), so that path is shown as Failed and Dead in the PowerPath Console.
Notice in the scope pane that a red slash (/) is shown for all the disk, the storage array, and one
Adapter. The slash symbol states that one or more, but not all, paths to the storage array and
adapters have failed or are in a degraded state. The particular path to the device that has failed
shows a red x, indicating that it is dead.
Note that the EMC PowerPath Administrator icon in the taskbar will also have a red slash
indicating that one or more paths are in a degraded state, even if the PowerPath Console window
is not currently open.
This example shows how to restore a path using the PowerPath Administrator GUI. To restore a
path, right-click the EMC PowerPathAdmin node in the scope panel. Next, choose All Tasks
from the drop-down window. Choose Restore All Devices to restore the path.
In addition to logs that can be used for troubleshooting, there is a VBScript available that can get
all pertinent information about the host and put it into an XML file. The name of this script is
PP_HealthCheck_Tool.vbs. This tool can be used to detect install/upgrade failures and capture the
state of the host at that instance.
This script is not packaged with PowerPath, but will be available through EMC Customer Support.
The VBScript generates an easily readable XML file named PowerPath.xml found under C:\. This
report can be attached along with other EMC grab reports to get to the root cause of an issue
faster.
This lesson covers PPCC function, installation, use procedures, and examination of PPCC output to
determine how to proceed with installing, upgrading or troubleshooting PowerPath.
PowerPath Configuration Checker (PPCC) is a software program that verifies that a host is
configured to the hardware and software required for PowerPath multipathing features (failover
and load-balancing functions, licensing, and policies) as specified in the EMC Support Matrix and
in the EMC Simple Support Matrix.
PPCC can assist the PowerPath user in the successful deployment of the PowerPath solution prior
to and after a PowerPath installation. It also provides efficiency in that it allows the host
administrators to independently effect their own planning and installation on hosts where
PowerPath is not yet installed. It also allows administrators to more easily prepare for upgrading
an existing installation and engage in troubleshooting. For example, PPCC supports
administrators after configuration changes are made on a host that includes PowerPath, such as
hardware changes or replacements or the installation or de-installation of software.
PPCC is intended to support the planning and upgrading of a PowerPath host environment. It can
also serve as a diagnostic tool when host system configuration changes affect PowerPath
operation. Using PPCC is an easy and efficient method to identify what changes should be made
to the host’s PowerPath configuration to ensure continued support for failover and load balancing.
PPCC can assist the administrator in three major areas that can cause a change in the host-to-
PowerPath relationship:
Planning - This task applies to a host on which PowerPath has never been installed or is not
currently installed. PPCC can identify the software that needs to be installed to support a specific
version of PowerPath.
Upgrade - This task applies to a host on which some version of PowerPath is installed. An
upgrade (or downgrade) to a different version is required. PPCC can identify components of a
configuration that need to change when a different version of PowerPath is to be installed.
Diagnostic - This task applies to a host on which some version of PowerPath is installed or on
which configuration changes have been made to PowerPath, to the host OS, and/or to other
software on the host. Also, if PowerPath does not appear to be operating correctly, running EMC
Reports along with using PPCC can assist with configuration problem analysis. Please see the
PowerPath configuration Checker User Guide for more information.
PPCC can advise you of changes you should make on a PowerPath host before you install or
configure a new version of PowerPath.
The process for this is to first run the EMC Grab utility on a ESXi or UNIX host (Sun Solaris, HP-
UX, IBM AIX, Linux) or run EMC Reports on a Windows host. Note that EMC Grab or EMC Reports
should be run on hosts that will be connecting to EMC arrays.
Then run PPCC on a Windows computer. PPCC reads the Grab and EMC Reports files and then
writes a report for the host.
Check EMC Support for the most up-to-date PPCC support information.
PPCC can be downloaded from the EMC Online Support website (support.emc.com). It is packaged
as a ZIP file containing the complete PPCC product. You can uncompress and run the installer
from any folder of a system running Windows 2012, Windows 7, Windows 2003, Windows 2008,
or Windows XP. PPCC writes its output report and log files to the installation folder by default.
PPCC works by analyzing data collected by EMC Reports for Windows or the latest EMC Grab
version for VMware vSphere and UNIX systems. EMC Reports and EMC Grab are widely used
diagnostic utilities that collect host configuration data, system logs, and other host information
used for EMC software problem analysis and resolution. PPCC verifies data in the EMC Reports or
Grab output file against an internal database of valid PowerPath configurations. It is because PPCC
analyzes the input file using its internal configuration database that the user must be sure to use
the latest version of PPCC available.
In this example, the EMC Reports utility was run on a Windows host that already has PowerPath
installed. PPCC was run against the EMC Reports file.
Note: When PowerPath is not currently installed on the host, PPCC must be informed through a
command line argument of the PowerPath version anticipated to be installed.
After analysis, PPCC generates an HTML results file and automatically opens a web browser to
display it. The information is presented in the form of a configuration report that validates the
host environment and warns about potential configuration problems. With this information, the
administrator can then install or update any missing or outdated software as required. EMC
Reports or Grab should be run and analyzed by PPCC again after each remedial action has been
taken.
Here is a partial list of the conditions that PPCC checks.
This lesson describes SCOM, and MOM and how they work with the PowerPath management pack.
The PowerPath Management Pack (MP) is a package designed and developed to help system
administrators monitor PowerPath installations. The management pack is separate and is not
included with PowerPath. It is installed on a management server running either Microsoft
Operation Manager (MOM) or System Center Operations Manager (SCOM). An associated MOM or
SCOM agent is installed on each PowerPath host and communicates with the MOM/SCOM server.
PowerPath Management Pack is available on support.emc.com for download. It includes

PowerPath specific event monitoring features and rules. The PowerPath Management Pack
provides the ability for MOM and SCOM to effectively monitor and manage triggers, alerts, and
SNMP traps associated with PowerPath in the data center environment.
The PowerPath management pack monitors PowerPath paths and volumes, and triggers alerts and
SNMP traps based on pre-defined criterion. You can modify this criterion.
The PowerPath management pack addresses the following PowerPath states:

• PowerPath path is dead — If the path does not come back alive within one minute (default
interval), then an alert and an SNMP trap is generated with the path information.
• PowerPath volume is dead — If the paths to the volume remain dead for more than five
minutes (default interval), then an alert and an SNMP trap is generated with the volume
information.
• PowerPath path latency threshold is exceeded - When path latency monitoring is enabled,
PowerPath measures the amount of time each I/O request takes to complete on the selected
path. If the latency threshold is exceeded, then an alert and an SNMP trap is generated with
the path, volume, and latency information.
The default time intervals can be altered by editing a config file named pp_delay.config.
This table provides a snapshot of some of the events monitored by the PowerPath management
pack and the actions taken on these events.
SCOM and MOM server software communicate with the associated agent software that is installed
on the PowerPath hosts. Agents on the hosts watch the Windows System Event log for PowerPath
events related to dead paths and volumes. Based on a predefined criterion, the agent generates
alerts and SNMP traps with the Path/Volume information. The alerts are then sent to the SCOM or
MOM server. These alerts can be viewed through the PowerPath Alerts Window.
The PowerPath Management Pack must be imported into the MOM or SCOM server after the
operations manager is up and running. Once imported, the server pushes the PowerPath
intelligence into the managed hosts where MOM or SCOM agents are running. Operations
manager is then able to monitor PowerPath devices that reside on the managed hosts. The MOM
agent monitors the PowerPath dead paths/volumes and generates MOM alerts and SNMP traps
when the paths/volumes do not come up within a configurable time period.
MOM and SCOM can be configured to send out emails to the system administrator when events
occur.
Shown here are supported operating systems for PowerPath management pack. Notice there are
three different versions, one for MOM 2005, one for SCOM 2007, and one for SCOM 2012.
See the EMC PowerPath Management Pack for Operations Manager User Guide for more details.
This module covered monitoring I/O throughput, path latency, error log reporting,
troubleshooting, PowerPath Configuration Checker, and PowerPath Management Pack.
Upon completion of this module, you will be able to explain how PowerPath is used in clustered
hosts environments and use PowerPath consistency protection.
This example shows an Open Systems HA cluster with PowerPath installed. Both nodes of the
cluster have shared access to the same devices in the storage array. To prevent data corruption,
device access is managed by the cluster software. Open Systems clustering technology manages
application availability by detecting failures. It restarts high availability applications on a surviving
cluster node.
The example shows a typical PowerPath/cluster implementation in which each node of the cluster
has multiple paths to the devices that are shared by the cluster.
The deployment of PowerPath in the cluster eliminates the application downtime due to a failure
of one path to the devices. PowerPath detects the path failure and uses alternate paths to deliver
the I/O rather than letting the application failover to the standby node. Path failover occurs within
PowerPath, and the cluster software is not aware of the failure.
Without PowerPath installed, the cluster technology would detect the path failure and restart the
application in the other node of the cluster. Cluster failover is not desired because it takes time to
perform the failover and restart the application. The result is an application outage until the
failover and restart operation is complete.
With PowerPath installed, cluster failover is limited to situations where the primary node goes
down or all paths to the devices are lost.
Another advantage of using PowerPath in a clustered environment is that its load-balancing ability
can help the server administrators maximize performance and get the greatest value from their
cluster investment. PowerPath balances the I/O across all paths to the devices from the primary
node of the cluster.
Dependent write operations preserve the integrity of data that is being written to disk, ensuring
that dependent writes are applied in the intended sequence of the application. The typical
sequence of write operations for a database update transaction proceeds in this way: A first write
operation is sent to the database transaction log indicating that a database data update is about
to take place. A second write operation follows which updates the database itself. A third write
operation then updates the database log indicating that the database update has completed
successfully. With this technique, the database ensures correct ordering of these writes by
waiting for each step to complete before starting the next.
With EMC TimeFinder, the Enginuity Consistency Assist (ECA) feature can be used to perform
consistent splits between source and target device pairs. Consistent split helps to avoid
inconsistencies and restart problems, which can occur if database-related devices are split without
first quiescing the database. When using consistent split on a group of devices, the database
writes are held at the storage level momentarily while the foreground split occurs, maintaining
dependent-write order consistency on the target devices comprising the group.
SRDF/Consistency Group (SRDF/CG) is an SRDF feature designed to ensure the dependent-write

consistency of the data distributed across multiple devices. A consistency group is a set of SRDF
devices that may reside on multiple Symmetrix and VMAX systems and are enabled for database
consistency. SRDF devices that belong to the same consistency group act in unison to preserve
dependent-write consistency of a database distributed across multiple devices within the
consistency group.
When databases become very large (larger than one terabyte), the time and resources required
for host-based software to protect or run decision support queries on these databases becomes a
critical factor. The time required to quiesce or shutdown the database for offline operations is no
longer acceptable for most database applications. When you need to perform such database
operations, the use of TimeFinder or SRDF consistent split allows you to split off a consistent,
DBMS-restartable copy of the database in seconds with no interruption to online service.
This is done by Either Enginuity Consistency Assist (ECA), or by PowerPath knowing when a
split will happen and holding I/O from the host until the split is completed. Doing this ensures that
the copy of the data being split off is sequential and restartable, and thereby a “gold copy.”
PowerPath suspends I/O to the devices at the host level. Split is then executed, and PowerPath
resumes I/O to the original devices.
ECA will hold I/O at the Symmetrix level within cache, which eliminates the need for PowerPath to
suspend I/O. In this way consistency is maintained across multiple servers.
EMC Solutions Enabler is host-based storage management software that provides an API-shared
library and a special command set that interfaces with the SYMCLI. Using command-line entries
as well as scripts, it interfaces with devices and data objects in the Symmetrix or VMAX array
complex to perform control operations on them. For Solutions Enabler versions prior to version
6.1, consistency protection for SRDF devices required PowerPath support. Creating an RDF type
composite group and specifying it for PowerPath places the group in the host’s PowerPath
database, and gives the group the capability of being enabled for consistency protection.
PowerPath-enabled RDF consistency group operations do not currently support concurrent RDF
devices.
The Microsoft Cluster Server (MSCS) provides a clustering technology that keeps server-based
applications highly available, regardless of individual component failures. When migrating devices
in an MSCS environment with PowerPath Migration Enabler, failover groups need not be disabled.
You do not have to shutdown any nodes in the cluster to perform a cluster migration. PowerPath
Migration Enabler supports multiple cluster migrations from multiple cluster nodes. You can
perform non-cluster migrations between non-cluster disks in any cluster node.
This module covered PowerPath and clustering, consistency, PowerPath consistent splits and
PowerPath Consistency Protection Considerations.
This course covered PowerPath theory of operations, PowerPath operation and administration,
using PowerPath for performance monitoring and troubleshooting, and PowerPath interoperability
concepts.
This concludes the training. Thank you.

PowerPath Configuration and Administration - SRG

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PowerPath Configuration and Administration - SRG

Uploaded by

Copyright:

Available Formats

Welcome to PowerPath Configuration and Administration.

Revision Date: 02/2015

In this course, we will cover the most common commands:

This table lists each array class that may be specified.

PowerPath Viewer includes two components:

Use powermt set periodic_autorestore=on/off to enable or disable PowerPath autorestore.

This example shows how to set adapter 5 into standby mode.

To save the configuration to a user-specific configuration file, use powermt save

Here we see output from the command after five seconds.

Autorestore is enabled by default when PowerPath is installed.

Step 4 can be done automatically by the autorestore feature, or manually if autorestore is

PowerPath Management Pack is available on support.emc.com for download. It includes

The PowerPath management pack addresses the following PowerPath states:

SRDF/Consistency Group (SRDF/CG) is an SRDF feature designed to ensure the dependent-write

This concludes the training. Thank you.

You might also like