Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013

Auckland New Zealand | August 14 - 17 2013

AHY24

#include <std_disclaimer.h> •These notes have been prepared by an Australian, so beware of unusual spelling and pronunciation.

PowerHA SystemMirror for AIX: New Features and Best Practice Antony “Red” Steel - ATS
Advanced Technical Skills

© 2013 IBM Corporation

Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013

Contents

Introduction to PowerHA – Standard and Enterprise PowerHA maintenance and features PowerHA directions

PowerHA SystemMirror

© 2013 IBM Corporation

Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013 Standard Edition
Centralised Management C-SPOC Cluster resource management Shared Storage management Cluster verification framework Integrated disk heartbeat SMIT management interfaces AIX event/error management Integrated heartbeat PowerHA DLPAR HA management Smart Assists Multi Site HA Management PowerHA GLVM async mode IBM Metro Mirror support IBM Global Mirror support DS8700 EMC SRDF sync/async Hitachi Truecopy Stretched or linked clusters

Agenda

Enterprise Edition               pending  
© 2013 IBM Corporation

PowerHA Standard and Enterprise Editions Cluster Aware AIX General changes What's new in PowerHA 7.1.1 and 7.1.2 Walk through PowerHA configuration and demo of application

         

PowerHA SystemMirror

DS8000 Hyper Swap

0.1 AIX 7.1 PowerHA SystemMirror 6.1 TL2 with RSCT 2.1 AIX 7.1 TL7 SP2 PowerHA SystemMirror 7.1.12.5.0.1 TL8 SP1 PowerHA SystemMirror 7.1 with RSCT 3.1 TL6 with RSCT 3.1.3 TL9 with RSCT 2.0 AIX 5.1.1.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.0 AIX 6.0.5 went EOS 30/4/2012 GeoRM went EOS 30/9/2009 PowerHA SystemMirror © 2013 IBM Corporation .4.1 with RSCT 3.4.1 TL1 SP2 AIX 6.2 AIX 7.1 AIX 7.0 Standard Edition 5765 H39 Standard Edition 5765 H23 Standard Edition 5765 H23 Standard Edition 5765 H23 Enterprise Edition 5765 H40 Enterprise Edition N/A Enterprise Edition N/A Enterprise Edition 5765 H24 Fixpack 1 GA EOS Fixpack 1 GA EOS Fixpack 4 GA EOS Fixpack 6 GA EOS Feb 2013 Nov 2012 N/A Feb 2012 Dec 2011 N/A Sept 2011 Sept 2010 Sept 2014 Aug 2011 Oct 2009 Sept 2014 ● ● ● HACMP 5. 2013 Introduction to PowerHA SystemMirror Standard ● Introduction to PowerHA What is high availability Planning – designing high availability Features of PowerHA to keep your applications available PowerHA SystemMirror 7.1 TL2 SP1 AIX 6.1 AIX 6.1.

2013 Introduction to High Availability ● PowerHA SystemMirror for AIX Standard Edition Cluster management for the data centre – Monitors. detects and reacts to events – Establishes a heartbeat between the systems – Enables automatic switch-over Causes of downtime Application errors Operating system errors Hardware failure Operator error ● ● IBM shared storage clustering – Can enable near-continuous application service – Minimize impact of planned & unplanned outages Standish Group Research 2008-2010 – Ease of use for HA operations Smart Assists – application agents – Out of the box deployment for SAP and other popular applications Mature Product – 22 Major releases (averaging one a year) – Over 12.Advanced Technical Skills ● IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.000 customers worldwide PowerHA SystemMirror for AIX Enterprise Edition – Cluster management for the Enterprise – Multi-site cluster management – Includes the Standard Edition function ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Introduction to High Availability ● High availability is: – The reduction to close to zero for downtime (not fault tolerance) – Solution may address planned or unplanned down time – Solution need not be fault tolerant but should be fault resistant – Solution should eliminate single points of failure (SPOF) PowerHA is not the answer if – Cannot afford any downtime – life critical systems .Need a fault tolerant solution – Environment is not secure ● Many users with root access – Then environment is not stable ● Change management is not respected ● You do not have trained administrators ● Procedures are not well documented ● Environment is prone to user fiddle factor – Applications cannot be controlled ● Scripts cannot be used to start/stop and recover applications ● PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. configuring application monitor – VIO server Implementing dual VIO servers Site – ● ● ● ● ● ● Adding an additional site PowerHA SystemMirror © 2013 IBM Corporation . 2013 Eliminate single points of failure by: ● Node – Using multiple nodes Power source – Using multiple circuits or un-interruptible power supplies Network adapter – Using redundant network adapters and bonding (etherchannel etc) – Network Using multiple networks to connect nodes / clients TCP/IP subsystem – Using non-IP networks to connect nodes Disk adapter – Using redundant disk adapter or multipath hardware – Disk Using multiple disks with mirroring or raid Application – Adding node for takeover.

9% (3-nines) 99.999% (5-nines) 99.9999% (6-nines) Downtime 36. 2013 Setting realistic expectations  What is considered an outage in your environment?  Unexpected downtime  Maintenance Tasks Availability 90% (1-nine) 99% (2-nines) 99.76 hours/year 52 minutes/year 5 minutes/year 31 seconds/year  What are the desired: – RTO – Recovery Time Objective – RPO – Recovery Point Objective PowerHA SystemMirror © 2013 IBM Corporation .65 days/year 8.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.5 days/year 3.99% (4-nines) 99.

Workload management. and make sure all configuration redundancy requirements have been met – Use PowerVM Suspend / Resume to reduce CPU and active memory PowerHA SystemMirror © 2013 IBM Corporation . Energy management.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. IBM may recommend quiescing critical applications on running partitions – Have current backups before beginning. SCSI Backplane. 2013 Building for availability  Infrastructure planning – Power Redundancy. Virtualized or Dedicated Deployments. Firmware updates without stopping / restarting the application  Charm – Available on high end models (>= 770) – Perform CHARM during low-use periods – LPM critical partitions to other servers if possible – Depending on the repair. suspend low priority workloads. Hardware management  Partition Suspend/Resume – Resume where stopped. SAN HBAs / Multipathing. Application Fallover Protection  LPM – Live move of OS/Application between frames. Backup Strategies. I/O Drawers.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Introduction to High Availability ● Planned – Maintenance – Upgrades – Testing – Development ● Unplanned – User Error – Application Failure – Component Failure – Operating System Failure – Environmental Disasters Becoming a more important area PowerHA as an administration tool LPM is an alternative for But not for (or software upgrades etc) PowerHA will help to mask or eliminate PowerHA SystemMirror © 2013 IBM Corporation .

.. – Replacement hardware may be at unrecognisable firmware levels.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. – Application may need to be upgraded. – Hardware may need to be upgraded (6 monthly f/w update – 1/year may not be concurrent).. which may require new software levels or fixes – OS and/or application out of support – Business expands – PowerHA designed to manage/support upgrade process ● Rolling upgrades ● Snapshot conversions PowerHA SystemMirror © 2013 IBM Corporation . 2013 You cannot let sleeping clusters lie ● Why touch the system ?? – has been working now for 2 years.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. Storewise. ● PowerHA managing application and storage replication – GLVM – SVC. GlobalMirror – EMC SRDF / Hitachi TrueCopy/HUR ● >> Planning and preparation PowerHA SystemMirror © 2013 IBM Corporation . shared storage ● Site only single point of failure Disaster Recovery – Replication ● GLVM ● Storage / Database – PowerHA SystemMirror Enterprise Ed. 2013 High Availability options ● One site – HA – PowerHA SystemMirror ● Dual servers. MetroMirror.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17... VIOS. PowerHA.) use CAA – CAA is a toolset – doesn’t form a cluster (now concept of quorum or fencing nodes – but provides tools to manage these) All interfaces are monitored – lscluster -i All nodes monitored – lscluster -m Changes from 2010 – No consistent view of devices – SolidDB no longer used – No zones / sub-clusters – Secure communication between nodes – Deadman switch (DMS) – A node is detected if isolated – can generate an AHAFS event or crash the node – clctrl -tune -o deadman_mode (clctrl -tune -L to list) ● ● ● PowerHA SystemMirror © 2013 IBM Corporation . 2013 Cluster Aware AIX ● IBM Cluster products (RSCT.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Cluster topology nA_n1_boot1 nA_n1_boot2 nB_n1_boot1 network1 network2 nA_n2_boot1 nA_n2_boot2 NodeA NodeB NodeC Repository disk hdiskn hdisko hdiskp PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Cluster topology nA_al nA_n1_boot1 rg1_n2_svc1 nA_n1_boot2 rg1_n1_svc1 network2 nA_n2_boot1 nA_n2_boot2 nB_n1_boot1 network1 rg2_n2_svc1 nC_al rg2_n1_svc1 RG1 RG2 NodeB NodeC NodeA app_mon1 NodeA rmt0 app_mon2 Repository disk vg1 hdisko hdiskp vg2 hdiskn PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Cluster topology nA_al nA_n1_boot1 rg1_n2_svc1 nA_n1_boot2 rg1_n1_svc1 network2 nA_n2_boot1 nA_n2_boot2 nB_n1_boot1 nC_al network1 rg2_n2_svc1 rg2_n1_svc1 Policies RG1 Policies NodeA app_mon1 NodeA NodeB RG2 NodeC rmt0 app_mon2 Repository disk vg1 hdisko hdiskp vg2 hdiskn PowerHA SystemMirror © 2013 IBM Corporation .

2013 Cluster behaviour ● Resource Group Policies – Startup ● Online on home node only ● Online on first available ● Online on all available ● Start up distribution – Failover ● Failover to next node in the list ● Failover using Dynamic node priority (CPU. Paging space.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. Adaptive (user defined)) ● Bring offline – Fallback ● Fallback to higher priority node ● Never fallback ● Resource group dependencies IP distribution preferences Inter site management policies – Online on Both Sites – Online on Either Site – Prefer Primary Site – Ignore ● ● PowerHA SystemMirror © 2013 IBM Corporation . Disk IO.

n3. Can have up to 3 levels Online on different node dependency – High. will go offline if the parent goes offline.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. intermediate and low – High will force intermediate and low to move.Child n1 DB n2 App n3 Test n2 App n3 Test n2 App n3 DB PowerHA SystemMirror © 2013 IBM Corporation .n3.n2 High App: n2. App .n1 Intermediate Test n3.n1 Low Parent / Child DB – parent. 2013 Resource Group dependencies ● Online on same node dependency – Resource groups come online on the same node Parent child dependency – Child will come online after the parent is stable. intermediate will force low to move – Same priority cannot come online on same node – Same priority will not cause a movement ● ● On same node dependency DB: n1.n2.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. but if no other is available then they will occupy the same interface. that is. Anti-collocation with Persistent Labels – service labels will almost never be on the same “adapter” as the persistent IP. 2013 IP distribution preferences ● Collocation – All Service labels will be on the same “adapter” Collocation with persistent – all service labels will be on the same “adapter” as the persistent IP. Collocation with Source – all service labels will be on the same “adapter” and the customer can choose the source IP of the outgoing packets Anti-collocation – all resources of this type will be allocated on the first “adapter” which is not already serving (or serving the least number of) addresses Anti-collocation with 1st Source – Same as above with the service IP being the source address of all outgoing packets. Anti-collocation with Persistent Labels and Source – Same as above with all outgoing packets having the service IP as the source address. service will occupy a different interface as long as one is available. ● ● ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .

2013 Are you using PowerHA features ● Are you aware of / using – Fast failure detection – File collections – Application monitoring ● Startup.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. the developers used feedback from the field/PMRs to fix common problems PowerHA SystemMirror © 2013 IBM Corporation . long running or both ● Process or custom – CSPOC – Cluster Test tool missing heartbeat check ● Remember that in the new versions of PowerHA.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Cluster Aware AIX (cont) ● Debugging – snap caa Logging via syslog – lscluster -s for stats lsattr -El cluster0 – Obtains node and repository disk UUID /usr/lib/cluster/clras lsrepos – Lists valid cluster repository disks /usr/lib/cluster/clras sfwinfo -d hdisk2 – Displays storage framework UUID for disks /usr/lib/cluster/clras dumprepos – Displays contents of cluster repository disk ● ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .

– CAA services can assist in the management and monitoring of an arbitrary set of nodes and/or running a third-party cluster. (If 20 nodes of a 21 node cluster are down. It is a tool set. 2013 Cluster Aware AIX ● Kernel based A set of services/tools embedded in AIX to help manage a cluster of AIX nodes and/or help run cluster software on AIX – IBM cluster products (including RSCT. CAA does not form a cluster by itself. CAA still runs on the remaining node). – CAA does not eject nodes from a cluster.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. PowerHA. – There is no notion of quorum. and the VIOS) will use and/or call CAA services/tools. CAA provides tools to fence a node but never fences a node and will continue to run on a fenced node Requires a repository disk (protected at the storage level) By default all interfaces monitored snap caa to collect PD data ● ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .

– Cluster Aware AIX tells you what nodes are in the cluster plus information on those nodes . – A point-of-contact “down” state indicates that the packet flow does not continue between the nodes. A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast.including state. – A point-of-contact “up” state indicates that the packet flow continues between the nodes. Note: The ability to monitor this particular condition is very important. An interface in the “up” state and a point-of-contact in a “down” state can occur because of hardware or other network issues between these particular nodes.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Cluster Aware AIX (cont) ● All nodes are monitored. Gossip packets travel over all interfaces. even though the interface may be in an “up” state. including storage. CAA monitors both communication interface states and points-of-contact between nodes on a node-by-node basis – A point-of-contact indicates that a node has received a packet from the other node over the interface. No traditional heartbeat mechanism is employed. ● ● PowerHA SystemMirror © 2013 IBM Corporation .

Synchronized between sites. Single Repository Disk.Networks. – 3rd party disk support added – Synchronous changes allowed across the cluster – Improved logging and RAS tools In 2012 added: – 2 sites – Linked or stretched clusters ● Stretched Cluster (Single CAA cluster. Cluster communication:. 2013 Cluster Aware AIX (cont) ● Cluster disks.Networks) ● ● ● PowerHA SystemMirror © 2013 IBM Corporation . Cluster communication:. One local repository on each site. Require multicast across 2 sites.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. SAN. (3 rd party disks do not participate in the monitoring). SolidDB and cluster disk naming dropped in 2010 In 2011 added: – Deadman switch for isolated nodes – tuneable and response options. 2 Separate Repository Disks. – CAA has information on all disks in the cluster – including their state. or Disk) ● Linked Cluster (Linked CAA cluster.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.1 RSCT Resource Monitoring and Control Resource Manager Group Services Topology Services AIX PowerHA SystemMirror © 2013 IBM Corporation . 2013 PowerHA 6.

1 RSCT Resource Monitoring and Control Resource Manager Group Services AIX PowerHA SystemMirror CAA © 2013 IBM Corporation . 2013 PowerHA 7.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.

1 Heartbeat Rings: detailed protocol – Leader.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.1 Multicast based protocol – Discover and use as many adapters as possible – Use network and SAN as needed – Adapt to the environment: delay. 2013 Cluster aware AIX – Topology management Host 2 Host 2 Host 1 Host 3 Host 3 Host 1 MULTICAST Host 4 Host 4 ● PowerHA 6. Successor. subnet etc Kernel based cluster message handling ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation . Mayor etc – Difficult to add/delete nodes Requires IP aliases management in the subnet ● PowerHA 7.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Default Multi Channel Health Management  Minimal Setup  Multiple channels of communication – Network – SAN – Central Repository Host 1 Reliable Messaging Host 2 Reliable Messaging Heartbeats Heartbeats First line of Defence Network SAN Heartbeats Second line of Defence Third line of Defence Cluster Repository 3 lines of (redundant) independant communications PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Configure SAN heart beating in virtual environment PowerHA SystemMirror © 2013 IBM Corporation .

no override allowed ● Done by call to cl_makecm out of node_up ● C-SPOC creates all volume groups as ECM ● Either “Fast Disk Takeover” or “Concurrent Access” ● Active/Passive mode used for non-concurrent resource groups – No SCSI-2 disk reserves set or broken ● Most disk differences now irrelevant ● Disk reserve handling code – cl_disk_available – retained for migration ● Fast path through code if ECM and no reserves PowerHA SystemMirror © 2013 IBM Corporation .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 General Changes ● Disk Handling Changes – ECMVG required – Existing volume groups automatically converted ● No user action required.

1.1 (only) – no 7.1.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.cluster.1 TL 1 with bos.cluster.rte 7.1.1 TL 7 with bos.1 EE RSCT and AIX requisites – AIX 6.1.1.ibm.com/software/support/lifecycle/index_h.rte 6.2 (SP2) APAR IV09868 – RSCT 3.1 ● Key dates: – Announce: October 12 – General Availability: December 16 Lifecycle information: – http://www-01.html Offerings: – Standard Edition has “base” function plus Smart Assists – New features added to Enterprise Edition 6.0 – works with either versions of AIX ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .1.7. 2013 PowerHA 7.2 (SP2) APAR IV09929 OR AIX 7.

migration. Role Based Access Control.1 (cont) ● New features – Standard edition – Security features – Encrypted Filesystem.1. 2013 PowerHA 7. LDAP Smart Assists – Expanded middleware support including SAP MaxDB HotStandby and Websphere MQ Series IBM Systems Director plug-in – Extends features available through Director Cluster Aware AIX – New features Miscellaneous updates – CSPOC enhancements.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. synchronous application startup ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .

1) – XIV replication support – Global Mirror support enhancements – Enterprise Edition 6.1 with Service Pack 7 SP7 and new install images available from FixCentral – http://www.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.1. 2013 PowerHA 7.com/support/fixcentral/aix/selectFixes  follow the links to select IV11782 (packaging APAR) – New support included in existing genxd fileset (updates only) ● PowerHA SystemMirror © 2013 IBM Corporation .1 (cont) ● New features – Enterprise edition (6.ibm.1 requires SystemMirror 6.

6 .1 6.1 Smart Assist support SystemMirror 7.5 AIX print server AIX 6.0 EHP1 for SVT .0 EHP1 for FVT SAP SCM 7.0.1 6.5.1.5 9.0 with Netweaver 7.1 AIX DHCP AIX 6.1 HTTP Server 6.1 AIX DNS AIX 6.1 WAS N/D 6.5.1 PowerHA SystemMirror © 2013 IBM Corporation .DB2 9.5.Oracle 10g r2 10g r2 .2 TDS 5.1 DB2 Enterprise Edition 9.3 Filenet 4.1.1 6.7 MQ Series 7.0 SystemMirror 7. 2013 PowerHA 7.1 Lotus Domino Server 8.1 4.1 6.1.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.1.2 6.7 WAS 6.0 with Netweaver 7.MaxDB V7.1 TSM 6.1 Oracle Databse 11g r1 Oracle Application Server 10g r2 SAP SAP ERP netweaver 2004s SAP SCM 7.

2013 PowerHA 7.1.1 Federated Security ● All user. encrypted FS credentials in a central store – Can use existing LDAP or Windows server Role based access (RBAC) – Roles: ha_admin: ha_op: ha_mon: ha_view: ● Administrator Operator Monitor Viewer ● Support for Encrypted filesystems – Shared filesystem or LDAP for keystore PowerHA SystemMirror © 2013 IBM Corporation . RBAC.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.

linked clusters with unicast communications & dual repositories HyperSwap capability is introduced.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.1. cluster wide AIX commands. – The Enterprise Edition provides for Disaster Recovery solutions with both host based mirroring and storage based mirroring IPv6 support is enabled with this version for v7 product Simpler to deploy and easier to manage multi-site configurations with IBM Systems Director. – HyperSwap with DS8800 storage subsystems provides for continuous availability against storage failures. Cluster wide AIX commands. kernel based event management single repository multicast communications – Linked Clustering. Cluster Split/Merge technology for managing split-site policy scenarios ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation . multi-site install wizard – Stretched Cluster. intuitive interfaces. 2013 PowerHA 7. kernel based event management.2 offers both a Standard and an Enterprise Edition.1.2 ● Version 7.

2013 PowerHA 7.1 TL2 SP1 – PowerHA SystemMirror 7.2 APAR IV27586 ● ● PowerHA SystemMirror © 2013 IBM Corporation .2 ● Cross Site Mirroring using LVM mirror pools Enhancements to the Director plugin to facilitate the use of these new features Software Levels Required: – OS – AIX 6.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.2 SP1 – Additonal software requirements for Enterprise Editionand HyperSwap – PowerHA SystemMirror 7.1 TL8 SP1 – OS – AIX 7.1.1.1.

PPRC SVC.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 PowerHA 7.1.2 ● High Availability and disaster recovery across multiple sites – PowerHA SystemMirror for AIX Enterprise Edition – Adds long distance failover for Disaster Recovery – Low cost host based mirroring support – Extensive support for storage array replication – Short distance (Campus to 80-100km) deployment: Synchronous – Long distance ( >100km) deployment: Asynchronous Replication Technology Sync Async New York IBM DS8K Series Storage .Truecopy * HP – Continuous Access * Network Host Mirroring Host Replication Geo LVM London Storage Array Replication Site 1 Site 2 Fiber Storage Mirroring Enterprise Edition © 2013 IBM Corporation PowerHA SystemMirror . Storevize. XIV EMC – SRDF * Hitachi – Universal Replicator.

2 Site 1 Site 2 Multi Sites Inter site communication Repository disk Cluster Communication Stretched Cluster Multicast Shared Networks SAN Disk Linked Cluster Unicast Separate Networks SAN in future Repository Disk Cross site LVM mirroring HyperSwap Multi site Conncurrent RG with HyperSwap Standard Enterprise Fig 1: Multi Sites with Stretched Cluster Site 1 Site 2 Multi Site Definition  Site Service IP  Site Policies Stretched Cluster Links Repository Disk 1 Repository Disk 2 Linked Clusters HADR with Storage Replication Management HyperSwap © 2013 IBM Corporation Fig 2: Multi Sites with Linked Clusters PowerHA SystemMirror .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 PowerHA 7.1.

2013 Tie breaker support ●  PowerHA 7.2 Tie Breaker Support – Separate Site Split and Merge policies – Split/Merge: Tie Breaker policy – FC/iSCSI Tie Breaker – SCSI 3 reservation disk – Losing side is quiesced Site 1 Cluster Site 2 SCSI or iSCSI Shared Disk Tie Breaker More suited for Linked Clusters Policy Setting Tie Breaker Majority Rule Split Merge Comments  Tie break Holder side wins  >N/2 side wins Site 3  In case of N/2. side that includes node with the smallest node id Manual  Manual steps needed for recovery to continue PowerHA SystemMirror © 2013 IBM Corporation .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.1.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.applications keep running – Key value add to HA/DR deployments Customer Benefits – Unplanned HyperSwap: HA/D R Application Cluster Hyperswap • • • Continuous Availability against storage failures Storage Maintenance without downtime Storage migration without downtime Primary DS8K Site 1 Sync Mirror – Planned HyperSwap: Secondary DS8K Site 2 Legend: Active Path Passive Path PowerHA SystemMirror © 2013 IBM Corporation . 2013 HyperSwap Technology  Continuous Availability against Storage failures  Substitutes storage secondary to take the place of failed primary device – Non-disruptive .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 HyperSwap Support by AIX-PowerHA • HyperSwap device configuration transparent to application – Application can continue to use the device as before Application/LVM/Middleware Application/LVM/Middleware /dev/hdiskX HyperSwap Pair /dev/hdiskX /dev/hdiskY Configure HyperSwap /dev/hdiskX /dev/hdiskY SYNC SYNC Primary DS8K Secondary DS8K Primary DS8K Secondary DS8K PowerHA SystemMirror © 2013 IBM Corporation .

2013 HyperSwap Multi Site Deployments: Oracle RAC Example PowerHA Cluster  Compute Node outages: – Active-Active workload provides continuous availability Site 1 Oracle RAC (Active) N1-1 (Active) N1-2 N2-1 Site 2 (Passive) (Passive) N2-2  Storage outages: – HyperSwap provides continuous availability  Active-Passive Sites – Active-Active workload within a site – Active-Passive across sites – Continuous availability for site storage outages S1 SYNC < 100 KM S2 Fig 1: Active-Passive HyperSwap  Active-Active Sites (Future) – Active-Active workload across sites – Continuous availability of site compute infrastructure and storage outages – Oracle RAC long distance deployment N1-1 Site 1 (Active) (Active) N1-2 Oracle RAC (Active) N2-1 Site 2 (Active) N2-2 S1 SYNC < 100 KM S2 Fig 2: Active-Active HyperSwap PowerHA SystemMirror © 2013 IBM Corporation .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.

cluster.0.enh ● ● Configuration Files – /etc/cluster/rhosts on the node where cluster will be created – /etc/hosts the nostname is the first alias for that IP address Topology services daemon is no longer used – CAA uses Scalable Reliable Multicast (SRM) for monitoring all network and storage interfaces using a single cluster-wide multicast IP address Can automatically define Multicast Address for you – Range 224.255 ● ● PowerHA SystemMirror © 2013 IBM Corporation .rte ← CAA Fileset – bos.1 – not pre-reqd but required ← bos.1.solid ← Solid DB (not required in 7.cluster.0.1) – bos.255. 2013 Pre-requisites ● Additional AIX Fileset Requirements: – bos.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.0 –239.clvm.255.ahafs ← Autonomic Health Advisor Filesystem ECM VGs are Required in 7.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Implementation differences ● New LAN Switch Settings – IP Multicasting Enabled ● Address Automatically selected during cluster configuration ● Set on Network Switches – IGMP_snooping Enabled ● Will reduce the amount of Multicast Traffic on LAN switches TME must be enabled on HBAs to leverage SAN heartbeating – List of supported Adapters in the slide notes – Additional steps for virtual HBAs (later slide) Repository Disk requirement – CAA Requirement (documented size has changed) ● This value can now be altered to 512MB or higher (max is 460GB) ● Larger disks will only result in wasted space – VSCSI volumes are supported ● ● PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013

Implementation differences (cont).

All network adapters will be discovered and used – To exclude adaters, use: /etc/cluster/ifrestrict: en4 en5 IPAT via Aliasing Only – No IPAT via Replacement – No Heartbeating over Aliases Network types supported mping to test – Ether broadcast – Infiniband (soon) – Notice that FDDI, TMSSA, TMSCSI and others are gone Removed Serial Network Types – RS232 Serial network – Disk heartbeat networks – No Multi-node disk heartbeat

PowerHA SystemMirror

© 2013 IBM Corporation

Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013

Zoning requirements for HBA heartbeating

W W P N

W W P N

W W P N

W W P N

W W P N

W W P N

optional heartbeat zone

W W P N

W W P N

shared storage zone

Storage subsystem
PowerHA SystemMirror
© 2013 IBM Corporation

Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013

Zoning requirements for HBA heartbeating

W W P N

W W P N

W W P N

W W P N

W W P N

W W P N

optional heartbeat zone

W W P N

W W P N

individual shared storage zone

Storage subsystem
PowerHA SystemMirror
© 2013 IBM Corporation

2013 Tools ● Cluster test tool Application availability analysis tool File collections Automatic cluster verification Automatic Error Notification (can also be customized) Auto-corrective/Self healing clusters Custom Pager notification methods (including SMS) OEM Volume and Filesystem Support (Veritas) and Custom disk methods Non-disruptive startup (create cluster around existing environment) Cluster snapshots to save/restore clusters (XML format allows easy editing) ● ● ● ● ● ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.

not the operation of the cluster manager ● ● PowerHA SystemMirror © 2013 IBM Corporation . Custom test procedure .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. There are some limitations. network and application down. then preform node down with and without takeover on random nodes.user defined plan Designed to test the configuration. 2013 Cluster test tool ● Automated test plan – Important part of install process – Still important as regular procedure once in production – Many cluster administrators believe testing too time consuming and costly – Lack of testing leads to failures – Conducts a series of tests and then analyzes them – Will start all nodes.

2013 Tests ● NODE_UP: start one or more nodes NODE_DOWN_FORCED: stop a node forced NODE_DOWN_GRACEFUL: stop one or more nodes NODE_DOWN_TAKEOVER: stop a node with takeover CLSTRMGR_KILL: catastrophic software failure NETWORK_DOWN_LOCAL: stop a network on a node NETWORK_UP_LOCAL: restart a network on a node SERVER_DOWN: stop an application server WAIT: pause testing RG_ONLINE. FAIL_LABEL: Interface fail and join VG_DOWN: loss of VG ● ● ● ● ● ● ● ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation . RG_OFFLINE. RG_MOVE. RG_MOVE_SITE: Resource Group online.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. move and site move JOIN_LABEL.offline.

Stop cluster services with takeover on a node ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .node1.node1. SITE_MERGE: Site isolation and re-integration Non-IP networks now tested 1/10/2005_07:20:24: ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 11/10/2005_07:20:24: | Validate NODE_UP 11/10/2005_07:20:24: ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 11/10/2005_07:20:24:    Event node: ALL 11/10/2005_07:20:24:    Configured nodes: ha1 ha2 11/10/2005_07:20:24:       Event 2: NODE_DOWN_GRACEFUL:     NODE_DOWN_GRACEFUL. SITE_DOWN_GRACEFUL.    Restart cluster services on the node that was stopped 11/10/2005_07:20:24: ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 11/10/2005_07:20:24: | Validate NODE_UP 11/10/2005_07:20:24: ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 11/10/2005_07:20:24:    Event node: ha1 11/10/2005_07:20:24:    Configured nodes: ha1 ha2 11/10/2005_07:20:24:       Event 4: NODE_DOWN_TAKEOVER:     NODE_DOWN_TAKEOVER.Stop cluster services gracefully on a node 11/10/2005_07:20:24: ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 11/10/2005_07:20:24: | Validate NODE_DOWN_GRACEFUL 11/10/2005_07:20:24: ­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­­ 11/10/2005_07:20:24:    Event node: ha1 11/10/2005_07:20:24:    Configured nodes: ha1 ha2 11/10/2005_07:20:24:       Event 3: NODE_UP: NODE_UP.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. SITE_DOWN_TAKEOVER: site up and down graceful or takeover SITE_ISOLATION. 2013 Tests (cont) ● NETWORK_UP/DOWN_LOCAL: Local network up and down SITE_UP. node2.

 1 minutes. 3 seconds Downtime:       Amount: 0 days. 22 hours. 2013 Application availability analysis tool                  Application Availability Analysis  Type or select values in entry fields. 30­June­2005. 01­Jan­2005. 23 hours. 59 minutes. 23:59 Application analyzed:         test_appl01 Total time:                   180 days. 1 hours. 00:00 Analysis ends:                Thursday. 59 seconds Uptime:       Amount:                180 days.97%       Longest period:        98 days. 58 minutes. 48 minutes.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 16 hours. 29 seconds       Percentage:            99.    [Entry Fields] * Select an Application                            [test_appl01]    + * Begin analysis on YEAR (1970­2038)               [2005]           # * MONTH (01­12)                                    [01]             # * DAY (1­31)                                       [01]             # * Begin analysis at HOUR (00­23)                   [00]             # * MINUTES (00­59)                                  [00]             # * SECONDS (00­59)                                  [00]             # * End analysis on YEAR (1970­2038)                 [2005]           # * MONTH (01­12)                                    [06]             # * DAY (1­31)                                       [30]             # * End analysis at HOUR (00­23)                     [23]             # * MINUTES (00­59)                                  [59]             # * SECONDS (00­59)                                  [59]             # Analysis begins:              Saturday. 30 seconds PowerHA SystemMirror Good log for initial PD © 2013 IBM Corporation .  Press Enter AFTER making all desired changes.

X releases is different than prior releases – Migration is disruptive – Requires the use of clmigcheck utility – Requires some reconfiguration of cluster topology If running older versions of HA you have a decision to make: – Migrate or Start at PowerHA version 7.1 is disruptive – (7.1.1.X – Can use non-disruptive upgrade to load patches: ● ie.4 Migration to 7. Source 7. 2013 Upgrade considerations ● Non-Disruptive Upgrade functionality is NOT available to get to 7.1 – Migrating from 7.0.0 or 7.1.0.1.1.1 to Target 7.1 requires newer AIX levels which provide CAA enhancements) ● ● PowerHA SystemMirror © 2013 IBM Corporation .1.1.0 to 7.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. not always adhered to – Cut to reduce cost – effects of the failure of a single component not always thought through eg single adapter networks. 2013 Designing High Availability ● Designing High Availability – A spare should be available for every single hardware and software component that is required to keep application running ● No ‘Single Point of Failure” – Whilst a generally accepted principle. no serial/failed serial network – Nodes – Power feed – Storage – Networks – Adapters – Administrators (good documentation 'clear' design) – Applications PowerHA SystemMirror © 2013 IBM Corporation .

1 restores ability to make network as private.filesystems 7.1. ● PowerHA SystemMirror © 2013 IBM Corporation .5 Private Networks – Reserve a network for Oracle – Oracle needs a network with no heartbeat etc. 2013 PowerHA usability changes ● Mount Guard – A new JFS2 facility to help prevent accidental double mounts – LVM and CAA can help. – PowerHA < 6. can be changed by chfs and logredo – Available in bos. without an intervening unmount will be rejected. 7.1 or 6.1.1 didn’t – PowerHA 7.1 supported.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.7 – Available in HA 6. but not ensure – A second mount.1 and 5. Mount state is maintained on the disks – Set by chfs option.1.

2013 PowerHA usability changes (cont) ● Application start in debug mode DARE Progress Indicators – What’s going on.1 didn’t support Xsite mirroring.1.1 has concept of sites and uses Mirror Pools to handle cross site mirroring Renaming Physical Volumes optional since 7. but poor scripts can lead to config_too_long Startup in Debug mode – warning exit code currently not checked. User can respond immediately to start failure ● ● ● ● ● ● ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .1 and 5. and when is it done – Terminal is locked – Back ported to HA 6.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. but PowerHA 7.5 Mirror Pools – NB: PowerHA 7.1 Shared physical volumes can be given consistent names across the cluster Cannot be part of a VG when renamed Foreground Application Start Application server can now be started in foreground Simplifies design of scripts.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 PowerHA usability changes (cont) ● Network changes – New ways to specify the source IP address for outgoing network traffic – The following are the new policies for Service IP Distribution Preference: ● Anti-Collocation with 1st Service – Each Service label will be placed on a different adapter and the service address is the source address of all outgoing traffic ● Collocation with 1st Service – All the Service labels are placed on one adapter and the customer can choose an address as a source for all outgoing traffic ● Anti-collocation with Persistent with 1st Service – Each service label will be the source address The swap adapter will use the new “transfer” option of ifconfig – This should help with problems associated with default and user specified routes CLCOMD now uses all unrestricted interfaces ● ● PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.1) New Heartbeat Tuning Parameters – Grace Period: The amount of time (seconds) the node will wait before marking a node as DOWN. Accepted values are between 5 and 30 Seconds. Accepted values are between 1 and 20 seconds – Settings apply to all networks across the cluster. Notes on Migration – Check carefully as not many configurations can be migrated ● ● PowerHA SystemMirror © 2013 IBM Corporation . – Failure Cycle: The frequency of the heartbeat.5 and 6. 2013 PowerHA usability changes (cont) ● Two-Node disk heartbeat – Easy set up. change and test (only 5.

out ● Repeated on config_too_long pattern ● DARE and sync continue to function.1.7 SP4 or AIX 7.1.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. the node shuts down on when the repository disk fails ● Disk failure or lost connection – CAA will provide Repository Resiliency ● Requires AIX 6.0.1.1. but any CAA topology changes are rejected – User must recognise repository failure. and allocate a new disk ● SMIT path under Manage the Cluster -> Select a new Repository Disk PowerHA SystemMirror © 2013 IBM Corporation . 2013 PowerHA usability changes (cont) ● Repository Resiliency – In PowerHA 7. PowerHA 7.1 SP1 ● Node continues running even on repository disk failure. using locally cached information ● Kept in the kernel ● User can provide a new disk on which to rebuild the repository ● No changes allowed while repository is out of service – On repository failure ● Message posted to hacmp.0 SP3.

2013 7.log” Fully globalized. clvt is a binary – the *only* binary in the clmgr code base. – all other code is ksh93 /usr/es/sbin/cluster/utilities/clmgr Added 100% tracing coverage. Difficult to use clmgr is a hard link to clvt. – Simplify management of clusters from Director – Reduce maintenance overhead Replacement to CLVT – Current Smart Assists utilize CLVT Overcomes previous CLVT limitations – Limited trace output and logging.1 . uses the “command. and large amounts of automatic help – Consolidated the set of supported actions and attributes ● ● ● ● ● PowerHA SystemMirror © 2013 IBM Corporation .cat” message catalog.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. with multiple levels – all STDERR output is written to “/var/hacmp/log/clutils.clmgr – cluster command line ● Director plug-in neededs a consistent interface for SystemMirror. Added dozens of consistent error messages.

clmgr – cluster command line (cont) ● Supported actions – add – delete – manage – modify – move – offline – online – query – recover – sync – view ● Supported object classes – cluster – site – node – interface – network – resource_group – service_ip – persistent_ip – application_controller – application_monitor – dependency – – – – – – – – – – file_collection fallback_timer volume_group * logical_volume * file_system * physical_volume * method* report snapshot tape * incomplete coverage of features PowerHA SystemMirror © 2013 IBM Corporation . 2013 7.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.1 .

Exit status = 0 PowerHA SystemMirror © 2013 IBM Corporation . Aug 12 2012 21:04:35 complete.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.. 2013 clmgr examples 7801p20:/usr/local/scripts# clmgr online cluster WHEN=now MANAGE=auto \ BROADCAST=false CLINFO=true FORCE=false FIX=interactively <snip> s7801p22: s7801p22: s7801p22: s7801p22: s7801p22: s7801p22: s7801p22: s7801p22: s7801p22: Aug 12 2012 21:04:35 Checking for srcmstr active.. Aug 12 2012 21:04:35 /usr/es/sbin/cluster/utilities/clstart: called with flags -m -G -i -B -A Aug 12 2012 21:05:10 Completed execution of /usr/es/sbin/cluster/etc/rc.cluster with parameters: -boot -N -A -i interactively -P cl_rc_cluster.

120 Cluster id for node s7801p21 is 2 Primary IP address for node s7801p21 is 10.2. s7801p20:/usr/local/scripts# lscluster -c Cluster query for cluster pleiades returns: Cluster uuid: 527e26c4-99b8-11e1-a0e3-1293071a2808 Number of nodes in cluster = 3 Cluster id for node s7801p20 is 1 Primary IP address for node s7801p20 is 10.55. -n Allows the cluster name to be queried for all interfaces -s Lists the cluster network statistics on the local node.121 Cluster id for node s7801p22 is 3 Primary IP address for node s7801p22 is 10.120 PowerHA SystemMirror © 2013 IBM Corporation .55. -m Lists the cluster node configuration information. -c Lists the cluster configuration.2. -d Lists the cluster storage interfaces.55.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.2. 2013 lscluster command ● lscluster flags -i Lists the cluster configuration interfaces on the local node.55.2.122 Number of disks in cluster = 0 Multicast address for cluster is 228.

2013 clRGinfo command s7801p20:/usr/local/scripts# clRGinfo -p Cluster Name: pleiades Resource Group Name: test_rg Node ---------------------------s7801p20 s7801p21 s7801p22 Group State --------------ONLINE OFFLINE OFFLINE PowerHA SystemMirror © 2013 IBM Corporation .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.

s7801p20 PgSpFree = 126661 PvPctBusy = 0 PctTotalTimeIdle = 98.523127 DNP Values for NodeId .3 NodeName . my_handle 1 ml_idx[1]=0 ml_idx[2]=1 ml_idx[3]=2 There are 0 events on the Ibcast queue There are 0 events on the RM Ibcast queue CLversion: 12 local node vrmf is 7103 cluster fix level is "3" The following timer(s) are currently active: Current DNP values DNP Values for NodeId .1 NodeName .2 NodeName .s7801p21 PgSpFree = 127610 PvPctBusy = 0 PctTotalTimeIdle = 98. 61haes_r710. 2013 lssrc command output changed s7801p20:/usr/local/scripts# lssrc -ls clstrmgrES Current state: ST_STABLE sccsid = "$Header: @(#) 61haes_r710_integration/14 43haes/usr/sbin/cluster/hacmprd/ main.945318 DNP Values for NodeId . 1038A_61haes_r710 2010-08-27T05:11:44-05:00$" i_local_nodeid 0.s7801p22 PgSpFree = 126483 PvPctBusy = 0 PctTotalTimeIdle = 98.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.801866 PowerHA SystemMirror © 2013 IBM Corporation . hacmp. i_local_siteid -1.C.

PowerHA SystemMirror © 2013 IBM Corporation . Cluster services are not active on any nodes. There are a number of possible causes: clinfoES or snmpd subsystems are not active.. snmp is not configured correctly...... Failed retrieving cluster information.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. snmp is unresponsive... Refer to the HACMP Administration Guide for more information..... 2013 cldump command s7801p20:/usr/local/scripts# cldump cldump: Waiting for the Cluster SMUX peer (clstrmgrES) to stabilize.

2.121 NODE s7801p22: Network net_ether_02 srvc1 10.120 There are 3 node(s) and 1 network(s) defined NODE s7801p20: Network net_ether_02 srvc1 10.2.2.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.120 s7801p20b 172.55.55.3.2.50.2. 2013 cltopinfo command s7801p20:/usr/local/scripts# cltopinfo Cluster Name: pleiades Cluster Connection Authentication Mode: Standard Cluster Message Authentication Mode: None Cluster Message Encryption: None Use Persistent Labels for Communication: No Repository Disk: caa_private0 Cluster IP Address: 228.1.122 <snip> Resource Group test_rg Startup Policy Online On Home Node Only Fallover Policy Fallover To Next Priority Node In The List Fallback Policy Never Fallback Participating Nodes s7801p20 s7801p21 s7801p22 Service IP Label srvc1 PowerHA SystemMirror © 2013 IBM Corporation .120 S7801p22 10.50.20 NODE s7801p21: Network net_ether_02 srvc1 10.2.55.120 s7801p21 10.50.

distributes command to all nodes (or a subset of the nodes) in cluster (or clusters) – Similar to dsh clcmd lssrc -g caa ------------------------------NODE s7801p22 ------------------------------Subsystem Group cld caa clcomd caa clconfd caa solidhac caa solid caa ------------------------------NODE s7801p20 ------------------------------Subsystem Group cld caa clcomd caa solid caa solidhac caa clconfd caa ● PID 6750432 7012576 7798794 6815926 8847410 Status active active active active active PID 5832952 6553816 7929910 8454150 8388622 Status active active active active active © 2013 IBM Corporation PowerHA SystemMirror .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Cluster wide execution ● Command is – /usr/sbin/clcmd Provided by CAA.

2013 IBM Systems Director: PowerHA management interface No charge plug-in Masks complexity Central management Real-time status Smart Assist integration Deployment wizards 73 PowerHA SystemMirror © 2013 IBM Corporation .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.

Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013

PowerHA 7.1.2 Director Plugin Enhancements

Wizards

– Cluster Create Wizard • Single Site and Multi Site deployment – Resource Group Creation Wizard • Custom and Smart Assist based RG deployment – SAP liveCache HotStandby solution Wizard – Federated Security Setup Wizard – Volume Group Create Wizard • Support for LVM Mirror Pools – Replication (Mirror) Group Wizard • HyperSwap Setup

Management Enhancements

– Repository Disk/s Management – Resource Groups management • Snapshots, networks, log files etc – Reports Management – Notifications management – Event driven callouts – Capacity upgrade based fallovers – HyperSwap Management – File collections

PowerHA SystemMirror

© 2013 IBM Corporation

Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013

PowerHA 7.1.2 Director Plugin: Multi Site Management

PowerHA SystemMirror

© 2013 IBM Corporation

Advanced Technical Skills

IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17, 2013

System Director Plug-in: Basic Architecture
Three-tier architecture provides scalability: User Interface  Management Server  Director Agent
Director Agent
 Automatically installed on AIX 7.1 & AIX V6.1 TL06
User Interface   Web-based interface Command-line interface

AIX PowerHA Director Agent

P

D

P

D

P

D

Secure communication
P D

Director Server
P D   P D  Central point of control Supported on AIX, Linux, and Windows Agent manager

P

D

Discovery of clusters and resources
© 2013 IBM Corporation

76

PowerHA SystemMirror

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 System Director Plug-in – Getting Started PowerHA SystemMirror © 2013 IBM Corporation .

PowerHA SystemMirror © 2013 IBM Corporation . A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast. 2013 Monitoring Services All communication interfaces are monitored • Cluster Aware AIX tells you what interfaces have been discovered on a node and information on those interfaces including state All cluster disks are monitored • Cluster Aware AIX tells you what disks are in the cluster and information on those disks including state • All monitors implemented at a low-level of the AIX kernel. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces including storage.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. therefore they are largely insensitive to system load All nodes are monitored • Cluster Aware AIX tells you what nodes are in the cluster and information on those nodes including state.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 LVM Split Site (Cross Site) Equivalent ● Assumes SAN connected disks and nodes at two locations ● Define shared volume group with super strict mirror pools – Mirror pool for each location – Disks must be manually assigned to each mirror pool ● Knowing which disks are where is a user responsibility – LVM mirrors logical volume between two locations – Resource group definition should allow forced varyon In the event of node and disk loss at one location – Volume group forced on line at other location by PowerHA ● Mirror pool set up guarantees a local copy of the data – Manual recovery of repository using Repository Resiliency ● PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Problem determination ● # clctrl -tune -L NAME                      DEF    MIN    MAX    UNIT           SCOPE      ENTITY_NAME(UUID)                                                 CUR      pleiades(361d4ace­5eb0­11e2­91f0­1293071a2807)                    240 config_timeout            240    0      2G­1   seconds        c n deadman_mode              a                                   c n hb_src_disk               1      ­1     3                     c hb_src_lan                1      ­1     3                     c hb_src_san                2      ­1     3                     c link_timeout              30000  0      1171K  milliseconds   c n node_down_delay           10000  5000   600000 milliseconds   c n node_timeout              20000  10000  600000 milliseconds   c n packet_ttl                32     1      64                    c n remote_hb_factor          10     1      100                   c repos_mode                e                                   c n site_merge_policy         p                                   c PowerHA SystemMirror © 2013 IBM Corporation .

tar drwxr­xr­x 0   0        0 Jan 30 09:55:30 2013 s7801p20/ ­rw­r­­r­­ 0   0     1123 Jan 30 09:55:31 2013 s7801p20/LOG ­rw­r­­r­­ 0   0     2554 Jan 30 09:55:30 2013 s7801p20/bootstrap_repository ­rw­r­­r­­ 0   0      978 Jan 30 09:55:30 2013 s7801p20/caa_tunables ­rw­r­­r­­ 0   0   194671 Jan 30 09:55:29 2013 s7801p20/clcomd_log. 2013 Problem determination ● # snap caa – Creates /tmp/ibmsupt/caa Contains data from each node in Data/data_time – – – – – – – – – – – – – – – – – tar ­tvf s7801p20.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.Z ­rw­r­­r­­ 0   0  5618196 Jan 30 09:55:30 2013 s7801p20/clcomddiag_log.Z ­rw­r­­r­­ 0   0     1362 Jan 30 09:55:30 2013 s7801p20/detail_repository ­rw­r­­r­­ 0   0      548 Jan 30 09:55:30 2013 s7801p20/lscluster_clusters ­rw­r­­r­­ 0   0     6144 Jan 30 09:55:30 2013 s7801p20/lscluster_network_interfaces ­rw­r­­r­­ 0   0     1968 Jan 30 09:55:30 2013 s7801p20/lscluster_network_statistics ­rw­r­­r­­ 0   0     2484 Jan 30 09:55:30 2013 s7801p20/lscluster_nodes ­rw­r­­r­­ 0   0     1067 Jan 30 09:55:30 2013 s7801p20/lscluster_storage_interfaces ­rw­r­­r­­ 0   0       76 Jan 30 09:55:30 2013 s7801p20/lsrepos_all ­rw­r­­r­­ 0   0      396 Jan 30 09:55:30 2013 s7801p20/swfinfo_uuids ­rw­r­­r­­ 0   0  10017023 Jan 30 09:55:28 2013 s7801p20/syslog_caa ­rw­r­­r­­ 0   0       93 Jan 30 09:55:30 2013 s7801p20/system_proc_version ­rw­r­­r­­ 0   0       30 Jan 30 09:55:30 2013 s7801p20/system_uname PowerHA SystemMirror © 2013 IBM Corporation .

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Moving to Disaster Recovery ● Requirements for HADR Solution – Recovery Time Objective ● Time application is unavailable – Recovery Point Objective ● Last data point at which production is recovered in event of a failure – Planned downtime ● Maintenance / Testing – Geographic dispersion ● To meet compliance regulations – Ease of management ● Degree of skill required compared with practicality of swaps – Ease of deployment ● Desire from customers for a simple solution – Integration and support ● Degree of integration with the OS and application will affect the success of failover PowerHA SystemMirror © 2013 IBM Corporation .

1 Cluster Aware AIX ● – IPv6. Rolling upgrade. 2013 Summary of changes  PowerHA 6.1. ● – Drop topology services for MultiCast protocol Linked and stretched clusters ● Split / merge site options with – Storage Monitoring tie-breaker – HADR Storage Framework ● Hyperswap – Support for DS8k for 2 sites PowerHA SystemMirror © 2013 IBM Corporation .2 – PowerHA 7.1 – DSCLI Metro Mirror VIOS – Packaging & Pricing Changes – p6/p7 CoD DLPAR Support – EMC SRDF Integration – GLVM Config Wizard – Full IPV6 Support   PowerHA 7.1. – DS8700 Global Mirror Integration – Enterprise Edition.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. Linked Cluster Aware AIX clusters – IBM Director Integration – IBM Systems Director plug-in – Hitachi TrueCopy & HUR async Integration ● New wizards. 2 site clusters.1 – CAA Repository Resilience – JFS2 Mount Guard support – SAP Hot Standby Solution – Federated Security – SAP & MQ Smart Assists – XIV Replication Integration – Director Plug-in Updates  PowerHA 7.

2013 PowerHA roadmap ● PowerHA release Life cycle strategy – Current model: Major release every year ● Requires ISV certification for every major release – New model: Implement technology level release strategy ● Major releases as necessary ● Minor release updates (Technology Leve 0 to Major release) ● At lease two technology levels per major release – Proposed ● Additional 2 year service offering for last TL (under review) New command – halevel -s ● PowerHA SystemMirror © 2013 IBM Corporation .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.

Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. 2013 Support planning PowerHA SystemMirror © 2013 IBM Corporation .

1.1+ ● ● Technology level release PowerHA failover reversal 3 or more sites support Operator override support PowerHA Enterprise Edition 6.1 TL01 ● PowerHA SystemMirror 7.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. Peoplesoft PowerHA Enterprise Edition 7.1 director plugin ● X 2013 X Federated security management Replicated storage management Wizards update – SAP liveCache HotSwap – GLVM express wizard – Multi-site cluster wizard PowerHA 7.1. 2013 Roadmap PowerHA SystemMirror 7.1 director plugin ● X 2012 PowerHA 7.1.1 director plugin ● Hyperswap HA/DR support Cluster modeling ● Three site support Cluster modeling Failover reversal LPAR HA management ● ● ● ● LPAR HA management ● ● PowerHA SystemMirror © 2013 IBM Corporation .1+ Hyperswap HA/DR ● ● VM HA management – VM restart – VM DR restart MQSeries smart assist ● PowerHA Enterprise Edition 7. Sybase.1 ● ● HA/DR support for XIV ● 2011 PowerHA 7.3 ● SAP liveCache Hot Standby Solution PowerHA federated security ● Smart Assists – Weblogic.2 ● PowerHA SystemMirror 7.

IBM PowerHA SystemMirror for AIX v7.com/developerworks/community/forums/html/forum?id=1111111 1-0000-0000-0000-000000001611 PowerHA Comments & Questions: – hafeedbk@us.com/systems/power/software/availability/ PowerHA portal – http://www-03.ibm.ibm.redbooks.ibm.com/systems/p/library/hacmp_docs.html PowerHA SystemMirror Marketing Page – http://www-03.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.ibm.ibm.1 – http://www.com/systems/power/software/availability/aix/index.com/abstracts/sg247845.html PowerHA Web site: – www.com http://www-03.html PowerHA technical forum – https://www.com PowerHA SystemMirror © 2013 IBM Corporation ● ● ● ● ● ● ● ● .ibm.html Online Documentation – http://www-03.com/systems/power/software/availability/aix/index. Mike and the US team for notes and detailed information.html PowerHA landing page on IBM. 2013 References ● Thanks to Shawn.com/systems/power/software/availability/aix/index.ibm.ibm.

com/abstracts/redp4669.ibm.html?Open Education: PowerHA for AIX Implementation.com/services/learning. Configuration and Administration AN610 – Go to www. 2013 Other useful info ● IBM Technology Service Offering for PowerHA SystemMirror XD deployment – http://www-935.com/systems/resources/systems_power_software_availability_clmgr_tech_guide. search for AN610 or PowerHA – coming soon GLVM white paper – www.wss/offering/its/a1000032 Redbooks – SG24-7739 : PowerHA for AIX Cookbook – SG24-7841 : Exploiting IBM PowerHA SystemMirror Enterprise Edition – SG24-7845 : IBM PowerHA SystemMirror 7.ibm.redbooks.com/developerworks/wikis/display/WikiPtype/High%20Availability PowerHA SystemMirror © 2013 IBM Corporation .1 for AIX RedGuide – High Availability and Disaster Recovery Planning: Next-Generation Solutions for Multi server IBM Power Systems Environments ● http://www.pdf clmgr white paper – www.ibm.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.com/systems/storage/virtualization Wiki – ● ● ● ● ● ● ● http://www.com/systems/resources/systems_p_os_aix_whitepapers_pdf_aix_glvm.com/services/us/index.ibm.pdf IBM storage virtualization offerings – www.ibm.ibm.ibm.

com/support/techdocs/atsmastr.ibm.ibm.ibm.nsf/searchsite?SearchView&query=powerha  Redbook if using PowerHA Enterprise Edition with Hitachi TrueCopy – http://www-03. 2013 Other useful info  PowerHA SystemMirror for AIX v7.nsf/WebIndex/WP102245 Implementing PowerHA with Storwize V7000  Tips for Configuring PowerHA on Flex System POWER7 Compute Nodes – http://w3-03.com/support/techdocs/atsmastr.com/support/techdocs/atsmastr.ibm.redbooks.ibm.nsf/WebIndex/WP102181 PowerHA SystemMirror © 2013 IBM Corporation 89 .1 Two-Node Quick Configuration Guide – http://www-03.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.com/support/techdocs/atsmastr.1 – – http://www-03.nsf/WebIndex/WP102216  Current Redbook – http:// www.com/redbooks.nsf/WebIndex/PRS5098  Disaster recovery using IBM Storwize family storage with IBM PowerHA SystemMirror Enterprise Edition 7.

ibm.ibm.nsf/WebIndex/TD105440 90 PowerHA SystemMirror © 2013 IBM Corporation .com/support/techdocs/atsmastr.nsf/WebIndex/TD105638  PowerHA for AIX Version Compatibility Matrix – http://w3-03. 2013 Resources – matrices and cross references  PowerHA Hardware Support Matrix – http://www-03.ibm.com/support/techdocs/atsmastr.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.nsf/WebIndex/TD101347  PowerHA Enterprise Edition Support Cross Reference – http://www-03.com/support/techdocs/atsmastr.

com/watch?v=bV9JdzPWTVQ PowerHA Enterprise Edition with XIV replication failover http://www.youtube.2 using IBM Systems Director Demo http://www.2 cluster http://www.youtube.com/watch?v=fZpYiu8zAZo PowerHA cluster test tool demonstration http://www.com/watch?v=RJ5O0030agM 91 PowerHA SystemMirror © 2013 IBM Corporation .youtube.1. 2013 Resources – demo videos           Configuring PowerHA v7.youtube.com/watch?v=zZHhCXhg1L8 Dynamically add a node into an active PowerHA cluster http://www.1.youtube.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.com/watch?v=zxHURigatQc Apply Updates (Service Packs) to an active PowerHA 7.

ibm.1 heartbeat over SAN (Talor Holloway – Advent One) – http://www.html  PowerHA 7.DeveloperWorks  PowerHA cluster migration to POWER7 (Chris Gibson – IBM) – http://www.htm l 92 PowerHA SystemMirror © 2013 IBM Corporation .com/developerworks/aix/library/au-cluster-migration/index.ibm. 2013 Resources .Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17.com/developerworks/aix/library/au-aix-powerha-heartbeat/index.

This ensures a product that continually improves over time into an extremely robust HA clustering solution. and the fixes are baked into the next release of the product.html “…Such deep integration enables innovative features unavailable in other products… In addition.” 93 PowerHA SystemMirror © 2013 IBM Corporation .ibm. because the clustering solution and operating system evolve together. 2013 Edison Group whitepaper – The Value of Deep Integration http://www-03.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. any flaws in the synthesis between the two discovered in the field are addressed.com/systems/power/advantages/whypower/powerha.

2013 Case study . and IT staff ensured data access by reducing failover time from several hours to five minutes. The 610-bed hospital based in New Brunswick. News & World Report. Solution The hospital deployed IBM Power 740 Express servers running IBM AIX.Robert Wood Johnson University Hospital  Download case study Overview Consistently ranked as one of “America’s Best Hospitals” by U. 94 PowerHA SystemMirror © 2013 IBM Corporation . Robert Wood Johnson University Hospital needed to improve IT performance and implement a failover system to ensure reliable data access. IBM PowerHA SystemMirror for AIX. New Jersey functions as one of the nation’s leading academic medical centers and is the only Level 1 Trauma Center for Central New Jersey. Benefits Hospital staff and patients noticed vast performance improvements in accounts and records systems. IBM System Storage DS4300 and NTT DATA Optimum Revenue Cycle Management software. Business need To remain competitive and ensure business continuity.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17. Robert Wood Johnson University Hospital provides state-of-the-art care through a wide range of health care services.S.

com +61 41980 3049 IBMTECHU.COM IBM STG Technical Universities & Symposia web portal download password: nz2013  KEY FEATURES. – Create a personal agenda using the agenda planner – View the agenda and agenda changes – Use the agenda search to find the sessions and/or – Download presentations – Submit Session and Conference Evaluations PowerHA SystemMirror © 2013 IBM Corporation ? ibmtechu..com/nz . 2013 AHY24 PowerHA SystemMirror for AIX: New Features and Best Practice Questions ? Antony (Red) Steel .ibm.Advanced Technical Skills IBM Systems and Technology Group Technical Symposium Auckland New Zealand | August 14 – 17..ATS Senior IT Specialist IBM Aust/NZ red_steel@au.

Sign up to vote on this title
UsefulNot useful