You are on page 1of 26

The￿Linux-HA￿User’s￿Guide

Florian￿Haas￿<florian.haas@linbit.com> The￿Linux-HA￿Project<linux-ha@lists.linux-ha.org>

The￿Linux-HA￿User’s￿Guide
by Florian Haas Copyright © 2009, 2010 LINBIT HA-Solutions GmbH, The Linux-HA Project

License information
The text of and illustrations in this document are licensed under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). • A summary of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. • The full license text is available at http://creativecommons.org/licenses/by-sa/3.0/legalcode. • In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.

.2.. RHEL and CentOS ................. 11 4.............. Building Packages ............................. 12 5................ Restarting Heartbeat services . 13 5.......................................................... 10 4............................ 21 7.........3................................................ The authkeys file ....... Building Packages ...........................1....... 18 6.......... Upgrading from previous Heartbeat versions ..... Communication module .........................8....1..4....... 6 3......1..........................1............... Stopping Heartbeat services ................................................9.................................. Upgrade software ................. Reporting problems ........................... 19 6..2............................ Administrative Tasks ....... 2 2...............................1.............. 14 5.....................2........... 20 7....... Linux-HA-dev ......................... 6 3......... IPC Library ........................Preface ........ Non-blocking logging daemon ...........................2.......1..............................................................................................3.................... 18 6.......... Downloading Cluster Glue sources ............. iv I.......5. Starting Heartbeat services ..........2...................2..........2.... Introduction to Heartbeat .4.. IRC .... 21 7....................1....................... 6 3......4................................... Restoring the CIB ............................................ Bug Tracking System . 15 6..................3......2................................................ Propagating the cluster configuration to cluster nodes ...................................2............... 1 1.................... OpenSUSE and SLES ........2.. Building Heartbeat ............1...........2................ 21 8.............2........................... Enabling the Heartbeat cluster to use Pacemaker .......................................................................... Placing the cluster in unmanaged mode .......2.... 19 6...................2.............................................................. Upgrading from CRM-enabled Heartbeat 2...........1....................................2.........3..................... Restarting Heartbeat .......... 21 7.....6....... Fedora...... 3 2.. 16 6.............................................................1......1 clusters not using the CRM ...................... 3 2........... Wiping files related to the CRM .......4....................................1............................................................................. 4 II............... 8 3...................................................................1.......... Heartbeat build prerequisites ....................................................................... Upgrading software ...........................................3................ Installing pre-built packages ................ Upgrading the CIB schema . Linux-HA .......................... Submitting patches ...... 21 7.................................... The ha............................................ 13 5...................................................... Upgrading from Heartbeat 2.............. Returning the cluster to managed mode ............2....................1........ 5 3..... Getting Help and Helping Out ..........1.......... 19 6......... 11 4............. 13 5............................ Heartbeat as a Cluster Messaging Layer ..................................... Creating an initial Heartbeat configuration ............................. Building and installing Cluster Glue from source ......3....... 16 6.............................1.......2............................................4.... 16 6....... Stopping Heartbeat services .... Mailing Lists .................................. Where to go from here ... 16 6..................... 18 6....................................... Debian and Ubuntu ...................2..................... Building Cluster Glue ......................1.7........... 16 6......3.................... Installing Heartbeat ................5............................................................................................................... 17 6...................................... 3 2.............................. 22 iii ................................................................. Backing up the CIB ............... Building and installing Heartbeat from source ................cf file ..........1.......................... Cluster Consensus Membership ............. 3 2..................1 clusters ................... 21 7..........................5.......................................... 14 5.1. Downloading Heartbeat sources ......2....2............1........ 11 4... 11 III........................................4... 8 3. 9 3......................................2.......................................................2......................... 17 6......1...................................................... 4 2.....1.............2.. 8 3................... Building and installing from source ...............................2...... 7 3.............................................................................................................. 19 IV.......................................................................... Cluster Plumbing Library ..... 8 3....................................... 6 3...................... Components ............... 18 6................................. Cluster Glue build prerequisites ..........3..................................................................................... 17 6......................

We welcome and actively encourage any and all feedback about this document.org [mailto:linux-ha@lists. the Linux-HA cluster resource agents.linux-ha.org] mailing list for comments.linux-ha. corrections. Please use the linux-ha@lists. iv .Preface This guide aims to be the definitive reference guide for users of the Heartbeat cluster messaging layer. and suggestions for improvement. and other clustering building blocks maintained by the Linux-HA project.

 Introduction￿to￿Heartbeat .Part I.

while maintaining full support for Heartbeat cluster messaging. Heartbeat￿as￿a￿Cluster Messaging￿Layer Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. In order to be useful to users.clusterlabs. the Pacemaker project was spun off as a separate project and continues as such. the Heartbeat daemon needs to be combined with a cluster resource manager (CRM) which has the task of starting and stopping the services (IP addresses.Chapter 1. After this release. Note Up until Heartbeat release 2. the Corosync [http://www.org] cluster messaging layer. etc) that cluster will make highly available. alternatively.org]. The canonical cluster resource manager typically associated with Heartbeat is Pacemaker [http://www.corosync.1. Pacemaker was developed jointly with Heartbeat as part of the Linux-HA umbrella project. 2 .3. a highly scalable and feature-rich implementation that supports both Heartbeat and. web servers. This allows clients to know about the presence (or disappearance!) of peer processes on other machines and to easily exchange messages with them.

 Communication￿module The Heartbeat communication module provides strongly authenticated. The heartbeat layer has an API which provides the following classes of services: • intra-cluster communication . serial links should be avoided. Cluster￿Consensus￿Membership CCM provides strongly connected consensus cluster membership services. Warning Please consider some important caveats with regard to serial link connectivity (see the ha.1. locally-ordered multicast messaging over basically any media. • broadcast UDP over IPv4. 2.sending and receiving packets to cluster nodes • configuration queries • connectivity information (who can the current node hear packets from) .both for queries and state change notifications • basic group membership services 2. Heartbeat can detect node failure reliably in less than a half-second.cf man page. IP-based or not. man ha. Cluster￿Plumbing￿Library The Cluster plumbing library is a collection of very useful functions which provide a variety of services used by many of our main components. Components The Heartbeat messaging layer comprises several interlocking but distinct components. • multicast UDP over IPv4. A few of the major objects provided by this library include: • compression API (with underlying compression plugins) • Non-blocking logging API • memory management oriented to continuously running services 3 . It ensures that every node in a computed membership can talk to every other node in this same membership.cf).Chapter 2. CCM Implements both an OCF draft membership API. It will register with the system watchdog timer if configured to do so. Typically it computes membership in sub-second time. • serial link communications.3.2. As a general rule: when in doubt. and the SAF AIS membership API. Heartbeat supports cluster communications over the following network link types: • unicast UDP over IPv4. 2.

promoting capture of core dumps in a uniform way.Components • Hierarchical name-value pair messaging facility promoting portability and version upgrade compatibility (also provides optional message compression facilities) • Signal unification . • 64-bit HZ-granularity time manipulation (longclock_t) • User id management for security purposes. logd prints a lost message count. Once it is capable of outputting messages again.but they work even when the time of day clock jumps) • child process management .setting and unsetting high priorities. Heartbeat provides simple and convenient integration with mainloop. and under all circumstances • timers (like glib mainloop timers . Although use of Glib mainloop with these APIs is not required. signals. This API also includes built-in authentication and authorization of peer processes. 2. 4 . • Mainloop integration for IPC. but the currently available implementations use UNIX (Local) Domain sockets. This means that all these different event sources are managed and dispatched consistently. etc. to files. or both. for processes which need some root privileges. capable of logging to a syslog daemon. and on a per-application basis.4. This IPC API does not require sockets. with configurable death-of-child messages • Triggers (arbitrary events triggered by software) • Realtime management . Non-blocking￿logging￿daemon logd is Heartbeat’s logging daemon. 2. and includes integrated flow control.death of children causes invocation of process object. logd never blocks. plain file descriptors. instead. IPC￿Library All interprocess communication is performed using a very general IPC library which provides nonblocking access to IPC using a flexible queueing strategy. Queue sizes are controllable both overall.allowing signals to appear as mainloop events • Core dump management utilities .5. and is portable to most POSIX-like OSes. it discards messages lagging too far behind. and locked into memory attributes of processes.

 Installing￿Heartbeat .Part II.

1. for example. Since heartbeat has a build dependency on cluster-glue.bz2 # tar -vxjf glue-1.1. Note This list applies to the default software configuration.y.tar. 3. Downloading￿Cluster￿Glue￿sources Several options are available for retrieving the Cluster Glue source code. it is necessary to build and install cluster-glue first.y. other dependencies may apply. the correct sequence of commands would be: # wget http://hg. Release tags follow the format glue-x. Building￿and￿installing￿from source Building and installing the Heartbeat cluster messaging layer from source amounts to building the following packages: • heartbeat itself. where x. for building locally on a target system. • net-snmp development headers. • the flex scanner generator and the bison parser compiler.2. If. not library headers).bz2 6 . • the cluster-glue package containing the Heartbeat local resource manager (LRM) and STONITH plugins.0.1.1 release. to enable IPMI related functionality. Building￿and￿installing￿Cluster￿Glue￿from source 3. • OpenIPMI development headers. one wants to download the 1. Downloading￿a￿release￿tarball Downloading a released version of Heartbeat as a compressed tarball is equivalent to fetching a tagged snapshot from the Mercurial source code repository.0. before one proceeds with building and installing heartbeat. If you configure the source with non-standard options. Cluster￿Glue￿build￿prerequisites Building Cluster Glue requires the presence of the following tools and libraries on the build system: • A C compiler (typically gcc) and associated C development libraries.tar.1. 3.linux-ha.Chapter 3.z.1.org/glue/archive/glue-1.z is the released version of Cluster Glue you wish to download.1. to enable SNMP related functionality.0. • Python (just the language interpreter.

tar. A customized build may thus comprise these steps: $ $ $ $ $ . Checking out the sources amounts to cloning the repository: $ hg clone http://hg./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var make sudo make install 7 .tar./configure --help . and libtool. 0 files removed. --sysconfdir. use this sequence of commands: # wget http://hg.sh . 0 files merged. as shown in this example: $ $ $ $ $ .linux-ha. 0 files unresolved 3./autogen./autogen. autoheader. it usually amounts to just the following sequence of commands: $ $ $ $ .org/glue/archive/tip. To retrieve a list of configuration options./autogen. Building￿Cluster￿Glue Building Cluster Glue is an automated process making extensive use of GNU Autotools./configure make sudo make install Note The autogen. To download a tarball auto-generated from the tip./configure --help . and --localstatedir. autoconf.sh script is a convenience wrapper around automake.linux-ha. When building and installing on the same machine.Building and installing from source Downloading￿the￿latest￿Mercurial￿snapshot The latest development code is always available in the Mercurial repository as the tip revision.1. A number of configuration options are supported.sh .bz2 # tar -vxjf tip.sh .bz2 Checking￿out￿sources￿from￿Mercurial This is the method you would apply if you have the Mercurial utilities locally installed. and you may tweak some of them to optimize Heartbeat for your system.org/glue cluster-glue requesting all changes adding changesets adding manifests adding file changes added 12491 changesets with 34830 changes to 2632 files updating working directory 356 files updated. you may invoke configure with the --help option./configure configuration-options make sudo make install Some typical configuration options that you may wish to set are --prefix.3.

See Section 3.4. rather than cloning or downloading from the upstream Mercurial repository. 3. to enable SNMP related functionality.org/hg/debian-ha/cluster-glue/].y.2. Note This list applies to the default software configuration.debian. use the one hosted on alioth.2. Downloading￿Heartbeat￿sources Downloading￿a￿release￿tarball Downloading a released version of Heartbeat as a compressed tarball is equivalent to fetching a tagged snapshot from the Mercurial source code repository. Thus. not library headers) • the cluster-glue development headers. for example. Red Hat Enterprise Linux.Building and installing from source 3. to enable IPMI related functionality. • the flex scanner generator and the bison parser compiler. “Building and installing Cluster Glue from source” [6] for details on how to build these from source. If you configure the source with non-standard options. the correct sequence of commands would be: 8 .1. other dependencies may apply. where x.2. Debian￿packages Debian packaging for Cluster glue is maintained in a Mercurial repository on alioth. Heartbeat￿build￿prerequisites Building Heartbeat requires the presence of the following tools and libraries on the build system: • A C compiler (typically gcc) and associated C development libraries.2.org [http://hg. 3.y. Building￿Packages RPM￿packages RPM spec files are provided in the Cluster Glue source tree both for SuSE and Red Hat based distributions: • cluster-glue-suse.spec is for Fedora.4 release. one wants to download the 3. and CentOS. • cluster-glue-fedora.0.z.1.1. Release tags follow the format STABLE-x. If.z is the released version of Heartbeat you wish to download.spec should be used for OpenSUSE and SLES installations. Building￿and￿installing￿Heartbeat￿from source 3. simply invoke dpkg-buildpackage from the top of the source tree — like you would with any other Debian package. • OpenIPMI development headers. • Python (just the language interpreter.debian. Once you have checked out or unpacked the source tree from the alioth repository. • net-snmp development headers.

--sysconfdir. A number of configuration options are supported.4.Building and installing from source # wget http://hg. autoheader. To download a tarball auto-generated from the tip. ConfigureMe is a convenience wrapper for the autoconf-generated configure script.0.tar./bootstrap .org/dev heartbeat-dev requesting all changes adding changesets adding manifests adding file changes added 12491 changesets with 34830 changes to 2632 files updating working directory 356 files updated./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var 9 .0./configure make sudo make install Note The bootstrap script is a convenience wrapper around automake.org/dev/archive/tip.linux-ha. A customized build may thus comprise these steps: $ $ $ $ $ . autoconf.tar. Checking out the sources amounts to cloning the repository: $ hg clone http://hg. and you may tweak some of them to optimize Heartbeat for your system. you may invoke configure with the --help option.org/dev/archive/STABLE-3./configure <configuration-options> make sudo make install Some typical configuration options that you may wish to set are --prefix.linux-ha. When building and installing on the same machine./bootstrap $ . 0 files merged.tar. and --localstatedir. as shown in this example: $ .bz2 Downloading￿the￿latest￿Mercurial￿snapshot The latest development code is always available in the Mercurial repository as the tip revision.3.2. To retrieve a list of configuration options. it usually amounts to just the following sequence of commands: $ $ $ $ .linux-ha./configure --help $ .bz2 # tar -vxjf STABLE-3./configure --help . Building￿Heartbeat Building Heartbeat is an automated process making extensive use of GNU Autotools.4. use this sequence of commands: # wget http://hg. 0 files unresolved 3.bz2 Checking￿out￿sources￿from￿Mercurial This is the method you would apply if you have the Mercurial utilities locally installed./bootstrap . and libtool.bz2 # tar -vxjf tip.tar. 0 files removed.

simply invoke dpkg-buildpackage from the top of the source tree — like you would with any other Debian package. Thus. 10 .2.spec should be used for OpenSUSE and SLES installations. Once you have checked out or unpacked the source tree from the alioth repository. Building￿Packages RPM￿packages RPM spec files are provided in the Heartbeat source tree both for SuSE and Red Hat based distributions: heartbeat-suse. use the one hosted on alioth.org/hg/debian-ha/heartbeat/].debian.spec is for Fedora. heartbeatfedora. rather than cloning or downloading from the upstream Mercurial repository. Red Hat Enterprise Linux.Building and installing from source $ make $ sudo make install 3.debian. and CentOS. Debian￿packages Debian packaging for Heartbeat is maintained in a Mercurial repository on alioth.org [http://hg.4.

• Fedora (since release 12). install the two packages with the following commands: zypper install heartbeat cluster-glue Since you most likely will also want to install Pacemaker (beyond the scope of this manual). This section describes the steps necessary to install binary packages on these platforms.3.￿RHEL￿and￿CentOS On Red Hat platforms. Assuming you have the correct package repositories configured for APT. OpenSUSE￿and￿SLES On SUSE platforms. Debian￿and￿Ubuntu Installing the cluster-glue and heartbeat packages on Debian and Ubuntu is a straightforward process.repos.1. • OpenSUSE (since release 11). backports packages are available for lenny). you install the cluster-glue and heartbeat packages with the YUM package manager. 4.linbit. install the two packages with the following commands: aptitude install heartbeat cluster-glue Since you most likely will also want to install Pacemaker (beyond the scope of this manual). do so by issuing the following commands as well: yum install resource-agents pacemaker 4. Fedora. do so by issuing the following commands as well: zypper install resource-agents pacemaker 11 . Assuming you have the correct package repositories configured. including • Debian (fully included in squeeze and up. you install the cluster-glue and heartbeat packages with the Zypper package manager.2.d/. Assuming you have the correct package repositories configured in +/etc/ yum. Installing￿pre-built￿packages Cluster Glue and Heartbeat are available as pre-built binary packages on a number of platforms. • Ubuntu (since lucid). install the two packages with the following commands: yum install heartbeat cluster-glue Since you most likely will also want to install Pacemaker (beyond the scope of this manual).Chapter 4. Commercially supported enterprise packages for Red Hat Enterprise Linux and SUSE Linux Enterprise server are available from LINBIT [http://www.com]. do so by issuing the following commands as well: aptitude install cluster-agents pacemaker 4.

 Administrative￿Tasks .Part III.

The next options configure node failure detection.d/ha. the time after which Heartbeat considers a node confirmed dead (deadtime). The￿authkeys￿file /etc/ha. but pacemaker is the preferred syntax.0.1. whereas on eth2 broadcast is acceptable as eth2 is not a shared network. and ensures that Pacemaker is automatically restarted in case of a failure. pacemaker respawn enables the Pacemaker cluster manager.0. This speeds up cluster start-up in clusters with a fixed small number of nodes.cf￿file The following example is a small and simple ha.d/authkeys — a file containing keys for mutual node authentication. and the maximum time it waits for other nodes to check in at cluster startup (initdead). • /etc/ha. Creating￿an￿initial Heartbeat￿configuration For any Heartbeat cluster. The option values listed here must match the exact host names of cluster nodes as given by uname -n. The￿ha. Thus. keepalive sets the interval at which Heartbeat keep-alive packets are sent.2. 5. This example assumes that bond0 is the cluster’s interface to the shared network. using the node options. bond0 can be used for Multicast heartbeat.0.d/authkeys contains pre-shared secrets used for mutual cluster node authentication. It should only be readable by root and follows this format: auth <num> 13 . 5. the pacemaker keyword was named crm.cf file: autojoin none mcast bond0 239.Chapter 5.4.cf — the global cluster configuration file. the following configuration files must be available: • /etc/ha. Note Prior to Heartbeat release 3. The node option identifies cluster members. They set the time after which Heartbeat issues a warning that a no longer available peer node may be dead (warntime). All these options are given in seconds.43 694 1 0 bcast eth2 warntime 5 deadtime 15 initdead 60 keepalive 2 node alice node bob pacemaker respawn Setting autojoin to none disables cluster node auto-discovery and requires that cluster nodes be listed explicitly. Newer versions still retain the old name as a compatibility alias. and that eth2 is the interface dedicated for DRBD replication between both nodes.

You may use either md5 or sha1. Depending on your system platform. 5. After a few seconds. the use of crc (a simple cyclic redundancy check. which you would invoke using either /usr/lib/heartbeat/ha_propagate or /usr/lib64/heartbeat/ha_propagate This utility will copy the configuration files over to any node listed in /etc/ha. not secure) is not recommended. It will afterwards also connect to the nodes using ssh and issue chkconfig heartbeat on in order to enable Heartbeat services on system startup. Starting￿Heartbeat￿services The Heartbeat services are started just as you would any other system service on your machine. You may create an authkeys file.4. using a generated secret. secret is the actual authentication key.d/authkeys chmod 0600 /etc/ha.cf and authkeys configuration files. Heartbeat will now start the Pacemaker daemon (named crmd for historical reasons) along with the rest of its services. \ dd if=/dev/urandom bs=512 count=1 | openssl md5 ) \ > /etc/ha. starting with 1.d/authkeys 5. you might be using one of the following commands: /etc/init. you should be able to detect Heartbeat processes in your process table: # ps -AHfww | grep heartbeat root 2772 1639 0 14:27 root 4175 1 0 Nov08 root 4224 4175 0 Nov08 root 4227 4175 0 Nov08 root 4228 4175 0 Nov08 root 4229 4175 0 Nov08 root 4230 4175 0 Nov08 102 4233 4175 0 Nov08 102 4234 4175 0 Nov08 pts/0 ? ? ? ? ? ? ? ? 00:00:00 00:37:57 00:01:13 00:01:28 00:01:29 00:01:35 00:01:32 00:03:37 00:15:02 grep heartbeat heartbeat: master control proc heartbeat: FIFO reader heartbeat: write: bcast eth2 heartbeat: read: bcast eth2 heartbeat: write: mcast bond heartbeat: read: mcast bond0 /usr/lib/heartbeat/ccm /usr/lib/heartbeat/cib 14 .d/heartbeat start service heartbeat start rcheartbeat start Through the pacemaker entry in your ha. with the following shell hack: ( echo -ne "auth 1\n1 sha1 ". you will only have one key in your authkeys file.d/ha.cf using scp.3.Creating an initial Heartbeat configuration <num> <algorithm> <secret> num is a simple key index.cf [13]. Usually. you may use the ha_propagate command. Propagating￿the￿cluster￿configuration￿to cluster￿nodes In order to propagate the contents of the ha. algorithm is the signature algorithm being used.

via the Pacemaker crm_mon command: # crm_mon -1 ============ Last updated: Mon Dec 13 14:29:36 2010 Stack: Heartbeat Current DC: alice (083146b9-6e26-4ac8-a705-317095d0ba57) .9-74392a28b7f31d7ddc86689598bd23114f58978b 2 Nodes configured.clusterlabs. 15 .org/doc/en-US/Pacemaker/1.5. The document primarily covers Pacemaker on Corosync.drbd. Where￿to￿go￿from￿here Now that you have a working Heartbeat configuration.1/html/ Clusters_from_Scratch/] is an excellent guide for configuring Pacemaker clusters.linbit. you should also be able to confirm that the cluster is working. unknown expected votes 24 Resources configured. • Several Technical Guides covering Heartbeat/Pacemaker clusters for a variety of applications are available from the LINBIT web site [http://www. you will want to continue with configuring Pacemaker and adding cluster resources. • The DRBD User’s Guide [http://www.0. but starting with its chapter "Using Pacemaker Tools" applies identically to Heartbeat clusters.partition with quorum Version: 1. The following documents are available for your perusal: • Clusters From Scratch [http://www.com/en/education/tech-guides]. ============ Online: [ alice bob ] 5.Creating an initial Heartbeat configuration root root 102 102 102 4235 4236 4237 4238 5724 4175 4175 4175 4175 4238 0 0 0 0 0 Nov08 Nov08 Nov08 Nov08 Nov08 ? ? ? ? ? 00:17:14 00:02:48 00:00:54 00:08:32 00:04:47 /usr/lib/heartbeat/lrmd -r /usr/lib/heartbeat/stonithd /usr/lib/heartbeat/attrd /usr/lib/heartbeat/crmd /usr/lib/heartbeat/pengine Finally.org/users-guide/] has a chapter dedicated to integrating DRBD with Pacemaker clusters.

uninstall the heartbeat package first.1 tree has been split up into modular parts. Enabling￿the￿Heartbeat￿cluster￿to￿use￿Pacemaker The cluster messaging layer must now be instructed to start Pacemaker on cluster start-up. it is important to recall that the monolithic Heartbeat 2.1￿clusters￿not using￿the￿CRM For Heartbeat 2. Stopping￿Heartbeat￿services You should commence the upgrade procedure on your current standby node.d/heartbeat stop 6. 6. add 16 . Upgrading￿from￿previous Heartbeat￿versions 6. and Heartbeat 3 which comprises just the cluster messaging layer. However. this down time amounts to minutes. stop Heartbeat services: # /etc/init. To do so. If your cluster is using an active-active configuration (both nodes running resources).0 involves converting the current configuration to one that is suitable for Pacemaker. Upgrade￿software While upgrading. clusters configured with the haresources file. Note This upgrade procedure does incur application down time. • Upgrading using a package repository: When upgrading using an APT. or Zypper repository. and on that node only. Do not restart Heartbeat services at this point. install Cluster Glue and Heartbeat.2. Thus you will replace Heartbeat with three individual pieces of software: Cluster Glue. and executed. you should just be able to run the install command for heartbeat version 3 and pacemaker.1 clusters not using the CRM (i. Then. Upgrading￿from￿Heartbeat￿2.1. 6.e. • Upgrading using locally built packages: When installing packages manually. that is. YUM.. Pacemaker. the version 3 heartbeat package. select one and issue the following command to transfer all resources to the peer node: # hb_standby Then. run make uninstall.Chapter 6. Then install cluster-glue.1. resource-agents. the cluster node currently not running any resources. and the dependencies will be resolved automatically.3.1 from.1. • Upgrading from source: In the unpacked archive that you installed Heartbeat 2. and pacemaker. when the upgrade is properly planned. an upgrade to 3. tested. possibly even seconds (depending on configuration).1.1.

cf(5) manual page. This will start the local Heartbeat instance and Pacemaker. Run /etc/init. and will not rectify and application or node failures during this time. you should also check your ha.1 cluster with the built-in CRM enabled. Note When properly planned and executed. Restarting￿Heartbeat Your cluster is now ready to be restarted in Pacemaker-enabled mode.1. It is strongly recommended to read and understand the steps outlined in this section before attempting a production cluster uqpgrade.cf modifications are complete.1￿clusters This section outlines the steps necessary to upgrade a Heartbeat 2. will continue to run. and start applications. 6. and remove any deprecated options.cf configuration file.1.4. 6. Run /etc/init.d/heartbeat start on your standby node (the one where you created your CIB). with no application down time. 2.d/heartbeat start on your the other node.2. This will shutdown your cluster resources. 6. This means that the cluster no longer monitors its nodes or resources for the duration of the upgrade. copy the file to the peer node.Upgrading from previous Heartbeat versions crm respawn to your ha. to Heartbeat 3.0 with Pacemaker. individual resources do not set the is_managed attribute individually. complete the procedure on each node before continuing with the next. the upgrade procedure can be completed in a manner of minutes. Instead. If in your specific configuration you do have resources that have this attribute set. 3. To do so: 1. Important At this point. All commands stated in this section must be run as root.d/heartbeat stop on your still-active node. fetch the CIB automatically. This will start the local Heartbeat instance and Pacemaker. and hence the cluster-wide attribute is_managed_default applies to all of them. When your ha. Run /etc/init. Do not perform the individual steps in parallel on all of your cluster nodes. Upgrading￿from￿CRM-enabled￿Heartbeat 2. you should remove it to make sure the default applies: # crm_resource -t primitive -r <resource-id> -p is_managed -D 17 . # crm_attribute -t crm_config -n is_managed_default -v false In most configurations.cf file against the ha. and wait for other cluster nodes to check in. Currently running resources.2. the cluster temporarily relinquishes control of its resources. Placing￿the￿cluster￿in￿unmanaged￿mode With this step. however.

# cp /var/lib/heartbeat/crm/cib. Backing￿up￿the￿CIB At this point. etc. is is important to save a copy of the Cluster Information Base (CIB). and all cluster nodes have had their CIB contents wiped. In this case. # rm /var/lib/heartbeat/crm/* 6.2. commonly /var/lib/heartbeat/crm.). “Wiping files related to the CRM” [18].2.2. remove all files from the directory where the CRM stores CIB information.2. rcheartbeat stop. Restoring￿the￿CIB Note You should only proceed with this step if Heartbeat is still stopped on all cluster nodes. namely the first node on which you are about to upgrade the cluster software. the /var/lib/heartbeat/crm directory must remain empty — Pacemaker distributes the CIB automatically. “Backing up the CIB” [18].3. as outlined in Section 6. The CIB to save is stored in a file named cib.Upgrading from previous Heartbeat versions 6. kill the crmd process manually: • use ps -AHfww to retrieve the process ID of crmd.xml ~ You need to perform this step on only one node currently connected to the cluster. normally located in /var/lib/heartbeat/crm. On all other nodes.xml You must perform this step on one node only.xml # chown hacluster:haclient /var/lib/heartbeat/crm/cib. verify that you have created a backup copy of the CIB on one of your cluster nodes. You should now wipe local CRM-related files from your node.xml /var/lib/heartbeat/crm/cib. it will be restored later.d/heartbeat stop or the preferred command to stop a system service on your distribution (service heartbeat stop. If you still have remaining nodes that have a residual CIB configuration. 6. after you have initiated a graceful service shutdown with the above command.4.2. Wiping￿files￿related￿to￿the￿CRM Warning Before proceeding with this section.2. 6.xml. To do so. Do not delete this file.2.2.2. “Backing up the CIB” [18] to the /var/lib/heartbeat/crm directory.4. 18 . Stopping￿Heartbeat￿services You may now stop Heartbeat with /etc/init.2. # cp ~/cib. • kill crmd with a TERM signal. In case you are running a legacy version of Heartbeat affected by a shutdown bug. proceed as outlined in Section 6.xml # chmod 0600 /var/lib/heartbeat/crm/cib.5. Restoring the CIB means copying the CIB backup described in Section  6. then graceful crmd shutdown will not work properly in unmanaged mode.

or monitor any resources).2. To do so. “Restarting Heartbeat services” [19]. • the cluster redistributes the old CIB among its nodes.1 from. Returning￿the￿cluster￿to￿managed￿mode Once the cluster software has been upgraded. Then. stop. and your package management system did not restart Heartbeat services after the software upgrade.2.7. “Restoring the CIB” [18]) with /etc/init. and • the cluster is still using the pre-upgrade CIB schema. repeat this command on the remaining cluster nodes.2.d/heartbeat start.1 tree has been split up into modular parts.8. “Stopping Heartbeat services” [16] Section 6.9.6.2. you should move to the next node and proceed as outlined in Section  6.Upgrading from previous Heartbeat versions 6. • Upgrading using locally built packages: When installing packages manually.2. uninstall the heartbeat package first. install Cluster Glue and Heartbeat.5. Upgrading￿the￿CIB￿schema Although an upgraded cluster can theoretically operate on the pre-upgrade CIB schema indefinitely. • Upgrading using a package repository: When upgrading using an APT. and the dependencies will be resolved automatically.7. Upgrading￿software While upgrading. run the following command after cluster communications between all nodes have been reestablished: # cibadmin --upgrade --force 19 . it is strongly recommended to upgrade the CIB to the current schema. or Zypper repository. Pacemaker. “Upgrading software” [19]. it is recommended to return the cluster into managed mode: # crm_attribute -t crm_config -n is_managed_default -v true 6. • the cluster is still in unmanaged mode (meaning it does not start. Otherwise. Restarting￿Heartbeat￿services Note This step may be omitted in case your package management system automatically restarts Heartbeat services during post-install. Then. run make uninstall. At this time. 6. If this is the last node to be upgraded in your cluster. First. and Heartbeat 3 which comprises just the cluster messaging layer.2. you should just be able to run the install command for heartbeat version 3 and pacemaker.1. the version 3 heartbeat package. YUM. Thus you will replace Heartbeat with three individual pieces of software: Cluster Glue. restart Heartbeat on the node where you restored the CIB (see Section 6. and pacemaker. you should now proceed to Section  6. resource-agents. Then install cluster-glue. • Upgrading from source: In the unpacked archive that you installed Heartbeat 2. 6.6.2.1. it is important to recall that the monolithic Heartbeat 2.

Part IV. Getting￿Help￿and￿Helping￿Out .

2. 21 . Both are subscriber-only lists. which requires a valid email address. you must create an account. hosted at lists. 7. 7.Chapter 7.linux-ha.linux-ha. in the #linuxha channel. The post address for this list is linux-ha-dev@lists.1.1. The post address for this list is linux-ha@lists. Linux-HA This is the general-purpose community mailing list. Bug￿Tracking￿System The Linux-HA project uses a public Bugzilla installation run by the Linux Foundation at http:// developerbugs. 7.linux-ha.org].org.1. Post here if you have enhancement suggestions or think you have found a bug.org [mailto:linuxha-dev@lists. IRC Linux-HA developers can usually be found on the irc.org.net server. Reporting￿problems This chapter describes the appropriate channels for dealing with bugs and issues releated to the software. 7. Linux-HA-dev This is the developer mailing list.linux-ha.org [mailto:linux-ha@lists. 7.3.freenode.linux-ha.2. To report bugs or comment on existing entries. Mailing￿Lists The Linux-HA project maintains two mailing lists.linuxfoundation.1. Post here if you have a general question about Heartbeat or the resource agents.org].

one of the upstream developers will push it into the repository.linux-ha. To do so. hg qpop -a hg pull --update hg qdelete fix-superfrobnication 22 .Chapter 8.org/dev heartbeat Create a new Mercurial queue. and remove your own patch set. you can update your checkout from upstream. Create a working copy (a Mercurial clone) of the upstream repository with the following command: hg clone http://hg. and you are able to fix it.org fix-superfrobnication Once your patch has been accepted for merging. Now the patch set is good for review on the mailing list: hg email --to=linux-ha-dev@lists. superfrobnication breaks on 63-bit middle-endian Linux platforms when ACPI is disabled. Fencepost align foobar_create_reqqueue() and guard against memory spinlock starvation to fix this. and a new patchset: cd heartbeat hg qinit hg qnew --edit --force fix-superfrobnication Note The --force option rolls your local modications into the patch set you are just creating. the developers would love to see a patch from you. be sure to include a meaningful description. for example: High: IPC: fix superfrobnication on 63-bit platforms In the IPC layer. Submitting￿patches If you have found a problem in Cluster Glue or Heartbeat. In your patch message. please follow the steps outlined in this section.linux-ha. At that point.