Fencing in Red Hat Enterprise Linux:

Methods, Use Cases, and Failover

Author Name: Shane Bradley and Allison Pranger
Technical Reviewer: Shane Bradley
Editor: Chris Negus
09/06/2011

OVERVIEW
The instability or failure of one node in a cluster can severely impact the other interconnected nodes. To
prevent data damage or loss across the cluster, Red Hat requires the use of fencing in all clustered
environments.
Fencing is the disconnection of a node from a cluster's shared storage. This cuts off I/O from shared
storage, which ensures data integrity, and can remove a node from the cluster.
This document provides a brief overview of fencing methods, describes how to manually call fence agents
from the command line, outlines fencing use cases, and lists steps to test failover.

Environment
• Red Hat Enterprise Linux 5 Advanced Platform
• Red Hat Enterprise Linux 6 (High Availability and Resilient Storage Add-Ons)

FENCING METHODS
Red Hat supports a variety of fencing methods, including network-power fencing, network-switch fencing,
virtual-machine fencing, and storage fencing. You can configure a node with one fencing method or multiple
fencing methods. When you configure a node for one fencing method, that is the only fencing method
available for fencing that node. When you configure a node for multiple fencing methods, the fencing
methods are cascaded from one fencing method to another, according to the order of the fencing methods
specified in the cluster configuration file, until a node is successfully fenced.
For a general overview of fencing and supported fencing types, see the official product documentation at
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/High_Availability_Add-
On_Overview/ch-fencing.html.

SCSI Persistent Reservations

If your storage devices support the use of SCSI persistent reservations, Red Hat will support SCSI
persistent reservations as a fencing method through the use of the fence_scsi agent. However, you
should note that SCSI reservations differ significantly from traditional methods such as network-power,
network-switch, and virtual-machine fencing (for more information about SCSI persistent reservations, see
the tech brief at https://access.redhat.com/knowledge/techbriefs/how-control-access-shared-storage-
devices-using-scsi-persistent-reservations-re).

Fencing in Red Hat Enterprise Linux | Shane Bradley & Allison Pranger 1
Manual Fencing
The use of manual fencing is not supported in production environments. To guarantee file-system integrity,
Red Hat recommends using a network power switch or a Fibre Channel switch fencing device for production
clusters.
Red Hat Enterprise Linux 5 includes the fence_manual fence agent, which was meant to act solely as a
secondary fence agent. However, this use of fence_manual was not optimal as it required the primary
agent to fail multiple times before it could be overridden, and fence_manual has often been misused as a
primary fence agent.
To eliminate the issues seen with fence_manual, the fence_ack_manual tool was enhanced in Red Hat
Enterprise Linux 6 so that it works with all standard fence agents and allows any fence agent to respond to
an override. This removes the need to configure fence_manual as a secondary or backup fence device
while still providing administrators the ability to continue cluster operation by manually verifying that a failed
node is truly down.
Outlined below is a scenario that explains how manual fencing might lead to file system corruption:
1. A node stops sending heartbeats long enough to be dropped from the cluster, but it has not
panicked or the hardware has not failed. There are a number of ways this could happen, including
a faulty network switch or a rogue application on the system locking out other applications.
2. Another member of the cluster initiates the fencing of the node. Fence_manual is called and lock-
manager operations are put on hold until the fencing operation is complete (existing locks are still
valid and I/O still continues for those activities not requiring additional lock requests).
3. Prior to checking on the status of the failed node, the system administrator sees the
fence_manual and immediately enters fence_ack_manual to start the cluster running again.
4. Journals for the fenced node are replayed and locks are cleared for those entries so other
operations can continue.
5. The fenced node continues to perform read/write operations based on its last lock requests. The
file system is now corrupt.
Because of these issues, the use of fence_manual is not supported in any production cluster. You may
use fence_manual for development or debugging purposes only.

NOTE: The fence_manual fence agent is not available in Red Hat Enterprise Linux 6.

The use of fence_ack_manual is fully supported in conjunction with any fence agent shipped by Red Hat.
Before issuing a fence_ack_manual override, a system administrator must first verify that the node being
overridden has been cut off from I/O to shared storage or file systems. If the node has not been disabled
from the cluster correctly, the cluster will operate in an undesired state, causing corruption of file systems or
other strange behavior. You can ensure a node is cut off from I/O to shared storage in one of several ways:
• Pull the power on the node
• Pull network and/or Fibre Channel cables as appropriate
• Disable Fibre Channel or Ethernet switch ports
• Verify that the cluster node has become completely disabled and has no chance of returning to
operation

Fencing in Red Hat Enterprise Linux | Shane Bradley & Allison Pranger 2
MANUALLY CALLING FENCE AGENTS FROM THE COMMAND LINE
When called by fenced, fence agents take arguments from standard input. There might be times, however,
when you want to run a fence agent from the command line, such as when testing failover.
There are three ways to execute a fence-agent script from the command line:
• Call the fence agent with command-line arguments for the particular fence device script, which is
located either in /sbin/fence_* (Red Hat Enterprise Linux 5) or /usr/sbin/fence_* (Red Hat
Enterprise Linux 6).
• Call fence_node, which uses /etc/cluster/cluster.conf as arguments for the fence agent.
• Call the fence agent by piping or redirecting standard input into the fence agents. Calling the fence
agent and sending stdin into the agent can be accomplished in one of two ways:
• Redirecting with file contents:
# cat /tmp/fence_ipmilan-args
ipaddr=192.168.1.42
login=<username>
password=<password>
action=status
# /sbin/fence_ipmilan < /tmp/fence_ipmilan-args

• Echoing a string via stdin into the fence agent:

# echo -e "ipaddr=192.168.1.42 \nlogin=<username> \
\npasswd=<password> \naction=status" | fence_ipmilan

USE CASES
This section describes two common fencing use cases: fencing a node with dual power supplies and
delaying fencing in a two-node cluster.

Fencing a Node with Dual Power Supplies

If a node has dual power supplies, then the fencing method for the node must specify at least two fencing
devices: one fencing device for each power supply. You must also configure your fence devices so that both
power supplies can be shut off and the system can be taken completely down. Figure 1 below illustrates
fencing with dual power supplies.

Fencing in Red Hat Enterprise Linux | Shane Bradley & Allison Pranger 3
 Figure 1: Fencing a Node with Dual Power Supplies

For detailed instructions on how to configure fence devices, see the official product documentation at
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Cluster_Administration/s1-config-
fence-devices-conga-CA.html.
Below is an example of the clusternodes and fencedevices sections in a
/etc/cluster/cluster.conf file for a node with dual power supplies connected to two different APC
switches:

<clusternode name="node-01" votes="1">

<fence>
<method name="1">
<device name="pwr01" action="off" switch="1" port="1"/>
<device name="pwr02" action="off" switch="1" port="1"/>
<device name="pwr01" action="on" switch="1" port="1"/>
<device name="pwr02" action="on" switch="1" port="1"/>
</method>
</fence>
</clusternode>
...
<fencedevices>
<fencedevice agent="fence_apc" ipaddr="192.168.0.101"
login="admin" name="pwr01" passwd="XXXXXXXXXXX"/>
<fencedevice agent="fence_apc" ipaddr="192.168.1.101"
login="admin" name="pwr02" passwd="XXXXXXXXXXX"/>
</fencedevices>

NOTE: In order for a fence to complete successfully, the machine must completely reboot. When a machine has
redundant power supplies, then each power supply must be turned off so that the machine can be properly
fenced.

Delaying Fencing in a Two-Node Cluster

Two-node clusters in Red Hat Enterprise Linux operate in a special mode. In most configurations, fencing
requires quorum, which is the minimum set of hosts required to provide service. In Red Hat Enterprise Linux,

Fencing in Red Hat Enterprise Linux | Shane Bradley & Allison Pranger 4
the specific quorum algorithm used is called simple-majority quorum, meaning that a majority of hosts must
be online in order for a quorum to be present. For example, in an eight-node cluster, at least five must be
online in order to provide service; in a five-node cluster, at least three nodes must be online; and so forth.
Generally speaking, quorum is a means to prevent a split-brain scenario, where two subsets of a cluster
operate independently of one another.
In two-node clusters, there is no majority when one node is active. Instead, the CMAN component relies on
a special mode known as two_node. In two_node mode, both hosts always have quorum. Split-brain
scenario, where each node thinks that it is the only surviving member, is common in two-node clusters. This
can lead to a fencing loop, where each node fences the other over and over again.
When a two-node cluster partitions for any reason, both nodes, since they have quorum, enter what is called
a fence race. Both try to cut the other off in order to establish a new leader so recovery can complete,
thereby allowing the cluster to continue providing service to clients. In many cases, you can correct the
issue with a cable pull on a single host or with a single fencing device that serializes access.
In cases where there are multiple fencing devices (especially integrated power management, such as HP
iLO), the outcome can be undesirable. It is possible for both nodes to issue a fencing action at the same
time, causing both nodes to turn off and resulting in a cluster outage: both nodes are powered off and will
not power on because a power-on command was not issued.
As of the noted releases of Red Hat Enterprise Linux (and respective Add-On products), most fencing
agents now support a delay parameter, where preference for which node wins the fence race is
predetermined by an administrator. The delay attribute can be added to the relevant fencedevice or
device tag in cluster.conf:

<cluster alias="mycluster" config_version="1" name="mycluster">

<clusternodes>
<clusternode name="node1.example.com" votes="1" nodeid="1">
<fence>
<method name="1">
<device name="node1-fence"/>
</method>
</fence>
</clusternode>
<clusternode name="node2.example.com" votes="1" nodeid="2">
<fence>
<method name="1">
<device name="node2-fence"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="node1-fence" agent="fence_ilo"
ipaddr="node1-ilo" login="user" passwd="passwd" delay="30" />
<fencedevice name="node2-fence" agent="fence_ilo"
ipaddr="node2-ilo" login="user" passwd="passwd" />
</fencedevices>
....
</cluster>

Fencing in Red Hat Enterprise Linux | Shane Bradley & Allison Pranger 5
The preceding example introduces a 30-second delay when fence_ilo is called by the cluster to fence
node1.example.com. This prevents both hosts from having a simultaneous fence request, resolving the
fence race and preventing both hosts from turning each other off at the same time. The cluster node without
the delay will have a 30-second head start to fence off the cluster node with the delay. The delay attribute
should be tested because each cluster might have a different optimal delay. The delay should be long
enough for the fencing to complete, plus a few seconds in case there is bandwidth congestion at the time of
the fence.

TESTING FAILOVER
Once you have set up your cluster and have assigned a supported fencing device to each participating
cluster node, you might want to test to see if fencing will work as expected in the event that it is needed.

NOTE: Before testing, ensure that the cluster is in a state that will have minimal impact on your data or
services. Fencing a node will cause a machine to power off ungracefully. Testing should ideally occur before a
cluster node is in production. Do not test fencing with data that cannot be recovered.

Set Up a Simple Service

The first step in the test procedure is to set up a virtual IP address to use as a simple service. In this
example, VIP refers to the service that will be created, and the IP address used is 192.168.1.100:
1. Using system-config-cluster or luci, create a single simple service with one private
resource that is a virtual IP with monitoring enabled.
2. Once the service is created, propagate the /etc/cluster/cluster.conf file to all nodes in the
cluster.
3. Verify that rgmanager has started on all nodes. If it has not, then start rgmanager on the nodes:
# service rgmanager status
# service rgmanager start

4. Check to see if the IP address is attached to a network device, then disable the service to verify
that it starts and stops correctly (substitute your service name and IP address if they are different):
# clusvcadm -e VIP
# ip addr | grep "192.168.1.100"
# ping 192.168.1.100
# clusvcadm -d VIP

5. If the service is enabled (and appears to work) and disabled correctly, then start the service again:
# clusvcadm -e VIP

Test Fencing
To see if fencing is working correctly, you can use one of three methods:
• fence_<device name>: This method allows you to manually call the fence agent with command-
line arguments for the particular fence device script. These scripts all begin with fence_ and are
located either in the /sbin/ (Red Hat Enterprise Linux 5) or /usr/sbin/ (Red Hat Enterprise Linux
6) directory.

Fencing in Red Hat Enterprise Linux | Shane Bradley & Allison Pranger 6
 • fence_node: This method uses /etc/cluster/cluster.conf as arguments for the fence agent.
• Fence via a sysrq crash: This method is as close as you can get to a real event that would trigger
fencing.
The fence_<device_name> and fence_node methods can help with troubleshooting. If fencing fails
when fence_node is called, there is most likely an issue with configuration values for the fence device in
the /etc/cluster/cluster.conf file. If fencing fails when manually invoked, then either the options
supplied are incorrect or there are configuration issues with the fence device.
Fencing via a sysrq crash is described in the next section.

Test Failover
This procedure demonstrates how to cause a cluster node to trigger a kernel panic. Once the panic occurs,
the cluster will notice that the cluster node is offline and will fence the cluster node out of the cluster.
1. Enable the clustered service that will be used to test failover of the service in the event that a node
is fenced that is running the service. For this procedure would do the following if the service is not
running:
# clusvcadm -e VIP

2. Once the service is started, use clustat to find out where the service is running:
# clustat

3. Enable sysrq for the cluster node running the clustered service. Note that enabling in proc will
only turn on sysrq for the running kernel and will not persist across reboots:
# echo 1 > /proc/sys/kernel/sysrq

4. Trigger the panic with the following command on the node running the cluster service:
# echo 'c' > /proc/sysrq-trigger

If the fencing worked, the cluster node should reboot and report as offline. It might take a minute or two for
the cluster to get the status and fence the node.
You can monitor the sequence of events in the log messages. You can use any of the following commands
to monitor the process on the surviving nodes (nodes that were not fenced and that are still participating
members of the cluster):

# watch clustat
# tail -f /var/log/messages
# watch cman_tool nodes

If the node was not fenced correctly or the service did not fail over correctly, you must resolve those issues
before proceeding to more complex cluster service configurations. Reviewing /var/log/messages on all
the nodes can give you a good idea of the problem. You should first determine the fence was successful or if
a failed service caused a problem. When a cluster node is fenced successfully, a message should be printed
to the log.

Fencing in Red Hat Enterprise Linux | Shane Bradley & Allison Pranger 7
RELATED DOCUMENTATION
• Official Product Documentation:
• Cluster Suite Overview: Fencing
• Configuring ACPI for Use with Integrated Fence Devices
• Configuring Fence Devices
• Troubleshooting Configuration of Fence Devices
• Tech Brief:
• How to Control Access to Shared Storage Devices Using SCSI Persistent Reservations with Red
Hat Enterprise Linux Clustering and High Availability
• Knowledgebase Articles:
• Fence Device and Agent Information for Red Hat Enterprise Linux
• Fence Support Matrix for Red Hat Enterprise Linux High Availability/Clustering and Red Hat
Enterprise Virtualization
• Red Hat Enterprise Linux High Availability: Manual Fencing (fence_manual vs.
fence_ack_manual)

Fencing in Red Hat Enterprise Linux | Shane Bradley & Allison Pranger 8

Copyright © 2011 Red Hat, Inc. “Red Hat,” Red Hat Linux, the Red Hat “Shadowman” logo, and the products www.redhat.com
listed are trademarks of Red Hat, Inc., registered in the U.S. and other countries. Linux® is the registered
trademark of Linus Torvalds in the U.S. and other countries.