How can I obtain quorum in a Pacemaker cluster with less than

or equal to half the nodes online?

access.redhat.com/solutions/6100541

Environment
Red Hat Enterprise Linux 7 (with the High Availability Add-on)
Red Hat Enterprise Linux 8 (with the High Availability Add-on)

Issue
I have a 3-node cluster. node2 and node3 are shut down, and I need to run resources on
node1 . But node1 can't run resources because it doesn't have quorum. How do I obtain
quorum?
Resources are stopped and the corosync-quorumtool -s command says "Activity
blocked" , but the pcs quorum unblock command doesn't unblock it:

Raw

[root@fastvm-rhel-7-6-21 ~]# corosync-quorumtool -s

Quorum information
------------------
Date: Fri Jun 4 16:10:46 2021
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 1
Ring ID: 1/247225
Quorate: No

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 1
Quorum: 2 Activity blocked
Flags:

# # RHEL 7
[root@fastvm-rhel-7-6-21 ~]# pcs quorum unblock
Error: cluster is not waiting for nodes to establish quorum

# # RHEL 8
[root@fastvm-rhel-8-0-23 ~]# pcs quorum unblock
Error: unable to check quorum status

Resolution

Runtime (temporary) change

1/4
To allow an inquorate cluster partition to obtain quorum at runtime, use the pcs quorum expected-
votes command.

Raw

# pcs quorum expected-votes --help

Usage: pcs quorum <command>

expected-votes <votes>
Set expected votes in the live cluster to specified value. This only
affects the live cluster, not changes any configuration files.

For example, if 2 nodes of a 3-node cluster are down, you can run the following command to allow the
remaining node to become quorate. Warning: You should do this only if you can ensure that the
other two nodes are down. If the other nodes are online, then this could result in a split-brain
situation and possibly data corruption.

Raw

# pcs quorum expected-votes 1

Permanent change

If you want the cluster to remain quorate after losing more than half the nodes in a cascade fashion (i.e.,
when fewer than half the nodes fail at the same time but more than half the nodes fail eventually), then
consider configuring the last_man_standing option.

Further reading

For more information on the pcs quorum unblock command, the pcs quorum expected-votes
command, and the last_man_standing option, refer to the appropriate section of the product
documentation below.

RHEL 7: High Availability Add-On Reference | Chapter 10. Cluster Quorum

RHEL 8: Configuring and managing high availability clusters | Chapter 25. Cluster quorum

Root Cause
By default, corosync requires greater than 50% of the cluster nodes to be online in order for a cluster
partition to be quorate (i.e., to have quorum). Obtaining quorum in a cluster partition with less than or
equal to 50% of nodes requires a runtime override or the configuration of non-default quorum options.

Note that the pcs quorum unblock command is used only for unblocking quorum when waiting for all
nodes to join the cluster. In a cluster with the wait_for_all option enabled, this command can be used
when one or more nodes have joined the cluster and are still waiting for the rest of the nodes to join.

Raw

2/4
# pcs quorum unblock --help

Usage: pcs quorum <command>

unblock [--force]
Cancel waiting for all nodes when establishing quorum. Useful in
situations where you know the cluster is inquorate, but you are
confident that the cluster should proceed with resource management
regardless. This command should ONLY be used when nodes which
the cluster is waiting for have been confirmed to be powered off and
to have no access to shared resources.

WARNING: If the nodes are not actually powered off or they do have
access to shared resources, data corruption/cluster failure can occur.
To prevent accidental running of this command, --force or interactive
user response is required in order to proceed.

The pcs quorum unblock command is not used to obtain quorum when more than half the nodes have
left the cluster. In that situation, the remaining nodes are inquorate, but they are not waiting for the rest
of the nodes to come online. You can use the pcs quorum expected-votes command to obtain
quorum, as described in the Resolution section.

The key point is that waiting only happens when one or more nodes join the cluster and the rest of the
nodes have not yet joined. If a node has joined the cluster and then left the cluster, the remaining nodes
are not waiting on that node.

If the cluster is inquorate but the wait_for_all option is not enabled, RHEL 7 reports Error:
cluster is not waiting for nodes to establish quorum , while RHEL 8 reports Error: unable
to check quorum status . This is because the pcs quorum unblock command runs corosync-
cmapctl -g runtime.votequorum.wait_for_all_status to check whether the cluster is in a waiting
state.

Raw

output, retval = utils.run(

["corosync-cmapctl", "-g", "runtime.votequorum.wait_for_all_status"]
)
if retval != 0:
utils.err("unable to check quorum status")
if output.split("=")[-1].strip() != "1":
utils.err("cluster is not waiting for nodes to establish quorum")

If the wait_for_all option is not enabled, then the runtime.votequorum.wait_for_all_status

key doesn't exist. On RHEL 7, the corosync-cmapctl -g command returns 0 if the key doesn't exist.
On RHEL 8, it returns 1 if the key doesn't exist.

Raw

3/4
[root@fastvm-rhel-7-6-21 ~]# corosync-cmapctl -g runtime.votequorum.wait_for_all_status; echo $?
Can't get key runtime.votequorum.wait_for_all_status. Error CS_ERR_NOT_EXIST
0

[root@fastvm-rhel-8-0-23 ~]# corosync-cmapctl -g runtime.votequorum.wait_for_all_status; echo $?

Can't get key runtime.votequorum.wait_for_all_status. Error CS_ERR_NOT_EXIST
1

That is the reason for the difference in error messages.

Bug 1968088 - Misleading error message from pcs quorum unblock if wait_for_all=0 is open to address
this discrepancy. As of 4 June 2021, this bug is in NEW state.

If you would like to track the progress of this bug, please subscribe to the Bugzilla or open a case with
Red Hat Global Support Services.

Diagnostic Steps
See Issue section.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that
Red Hat engineers have created while supporting our customers. To give you the knowledge you need
the instant it becomes available, these articles may be presented in a raw and unedited form.

4/4