Configure Multiple Fencing Devices Using pcs

Updated 416 Days AgoPublic

Actions

This describes how to Configure Multiple Fencing Devices (using that page's example of IPMI followed by two switched PDUs) using the higher-level pcs tool.

Starting Point

For a frame of reference, the cluster starts with this configuration:

Cluster Name: an-cluster-03
Corosync Nodes:
  pcmk-1 pcmk-2 
Pacemaker Nodes:
  pcmk-1 pcmk-2 

Resources:   

Stonith Devices: 

Fencing Levels: 

Location Constraints:

Ordering Constraints:

Colocation Constraints:

Cluster Properties:
  cluster-infrastructure: corosync
  no-quorum-policy: ignore
  stonith-enabled: false

Assumptions

We will need to make a few assumptions about our example cluster:

• It is a two-node cluster with the node names pcmk-1 and pcmk-2.
• The two PDUs are accessible at the network addresses pdu-1 and pdu-2, and will be accessed using the fence_apc_snmp fence agent.
• The fencing details for pcmk-1 are:
  • IPMI device address is pcmk-1.ipmi, the login name is admin and the password is secret.
  • Its power supplies are connected to port 1 of both pdu-1 and pdu-2.
• The fencing details for pcmk-2 are:
  • IPMI device address is pcmk-2.ipmi, the login name is admin and the password is secret.
  • Its power supplies are connected to port 2 of both pdu-1 and pdu-2.

Adapt the example below to the names, addresses, credentials, and fence agents appropriate to your cluster.

Configure Fencing Devices

• Configure the IPMI fence device for pcmk-1:

pcs stonith create fence_pcmk1_ipmi fence_ipmilan \
  pcmk_host_list="pcmk-1" ipaddr="pcmk-1.ipmi" \
  login="admin" passwd="secret" delay=15 \
  op monitor interval=60s

• Configure the two PDU fence devices for pcmk-1:
  
  Note that we've added power_wait="5" to the second PDU, to tell pacemaker to wait 5 seconds after turning off the second PDU before restoring power. This gives plenty of time for the node's power supplies to completely drain, ensuring that the node loses power.

pcs stonith create fence_pcmk1_psu1 fence_apc_snmp \
  pcmk_host_list="pcmk-1" ipaddr="pdu-1" \
  port="1" op monitor interval="60s"
pcs stonith create fence_pcmk1_psu2 fence_apc_snmp \
  pcmk_host_list="pcmk-1" ipaddr="pdu-2" \
  port="1" power_wait="5" op monitor interval="60s"

• Repeat for pcmk-2:

pcs stonith create fence_pcmk2_ipmi fence_ipmilan \
  pcmk_host_list="pcmk-2" ipaddr="pcmk-2.ipmi" \
  login="admin" passwd="secret" delay=15 \
  op monitor interval=60s
pcs stonith create fence_pcmk2_psu1 fence_apc_snmp \
  pcmk_host_list="pcmk-2" ipaddr="pdu-1" \
  port="2" op monitor interval="60s"
pcs stonith create fence_pcmk2_psu2 fence_apc_snmp \
  pcmk_host_list="pcmk-2" ipaddr="pdu-2" \
  port="2" power_wait="5" op monitor interval="60s"

Configuring fencing_topology

The next step is to tell Pacemaker the order we want the fencing methods to run.

Each fencing level may have one or more fence devices. When fencing is required, Pacemaker will try each level in sequence, stopping at the first level that succeeds. Therefore, separate levels function as a "fallback" mechanism (logical "or"). At any given level, all the devices in that level will be tried in succession, and all must succeed for the level to succeed (logical "and").

For our example, tell Pacemaker that the IPMI-based fence devices are the primary methods to use, and the switched PDUs are the fallback methods:

pcs stonith level add 1 pcmk-1 fence_pcmk1_ipmi
pcs stonith level add 2 pcmk-1 fence_pcmk1_psu1,fence_pcmk1_psu2

pcs stonith level add 1 pcmk-2 fence_pcmk2_ipmi
pcs stonith level add 2 pcmk-2 fence_pcmk2_psu1,fence_pcmk2_psu2

When Pacemaker needs to reboot a node using multiple devices in the same level, it turns them all off, then turns them all on, rather than rebooting each in turn, to ensure the node is completely fenced.

Enable and Test Fencing

Now that fencing is configured, we can enable it:

pcs property set stonith-enabled=true

You can test by unplugging the IPMI interface for pcmk-1 and then crashing it, triggering pcmk-2 to initiate fencing of it. After the IPMI interface times out, you should see PDU 1's port 1 turn off, then PDU 2's port 1 turn off, then the crashed node power down, then PDU 1's port 1 should turn back on, and finally PDU 2's port 1 should turn back on. If you configured your server's BIOS to power on after power loss or to return to last state after power loss, your server should start to power back on.