Be sure that the values you chose do not conflict with any existing
clusters you might have. For this document, I have chosen port '4000'
and used '239.255.1.1' as the multi-cast address.
endif::[]
=== Notes on Multicast Address Assignment ===
There are several subtle points that often deserve consideration when
choosing/assigning multicast addresses for corosync.
footnote:[This information is borrowed from, the now defunct, http://web.archive.org/web/20101211210054/http://29west.com/docs/THPM/multicast-address-assignment.html]
. Avoid '224.0.0.x'
+
Traffic to addresses of the form '224.0.0.x' is often flooded to all
switch ports. This address range is reserved for link-local uses. Many
routing protocols assume that all traffic within this range will be
received by all routers on the network. Hence (at least all Cisco)
switches flood traffic within this range. The flooding behavior
overrides the normal selective forwarding behavior of a
# sed -i.bak "s/.*mcastaddr:.*/mcastaddr:\ $ais_mcast/g" /etc/corosync/corosync.conf
# sed -i.bak "s/.*mcastport:.*/mcastport:\ $ais_port/g" /etc/corosync/corosync.conf
# sed -i.bak "s/.*\tbindnetaddr:.*/bindnetaddr:\ $ais_addr/g" /etc/corosync/corosync.conf
----
Lastly, you'll need to enable quorum
[source,Bash]
-----
cat << END >> /etc/corosync/corosync.conf
quorum {
provider: corosync_votequorum
expected_votes: 2
}
END
-----
endif::[]
The final /etc/corosync.conf configuration on each node should look
something like the sample in Appendix B, Sample Corosync Configuration.
[IMPORTANT]
===========
Pacemaker used to obtain membership and quorum from a custom Corosync plugin.
This plugin also had the capability to start Pacemaker automatically when Corosync was started.
Neither behavior is possible with Corosync 2.0 and beyond as support for plugins was removed.
Instead, Pacemaker must be started as a separate service.
Also, since Pacemaker made use of the plugin for message routing, a node using the plugin (Corosync prior to 2.0) cannot talk to one that isn't (Corosync 2.0+).
Rolling upgrades between these versions are therefor not possible and an alternate strategy footnote:[http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/ap-upgrade.html] must be used.
===========
ifdef::crmsh[]
=== Propagate the Configuration ===
Now we need to copy the changes so far to the other node:
[source,C]
----
# for f in /etc/corosync/corosync.conf /etc/hosts; do scp $f pcmk-2:$f ; done
STONITH is an acronym for Shoot-The-Other-Node-In-The-Head and it
protects your data from being corrupted by rogue nodes or concurrent
access.
Just because a node is unresponsive, this doesn't mean it isn't
accessing your data. The only way to be 100% sure that your data is
safe, is to use STONITH so we can be certain that the node is truly
offline, before allowing the data to be accessed from another node.
STONITH also has a role to play in the event that a clustered service
cannot be stopped. In this case, the cluster uses STONITH to force the
whole node offline, thereby making it safe to start the service
elsewhere.
== What STONITH Device Should You Use ==
It is crucial that the STONITH device can allow the cluster to
differentiate between a node failure and a network one.
The biggest mistake people make in choosing a STONITH device is to
use remote power switch (such as many on-board IMPI controllers) that
shares power with the node it controls. In such cases, the cluster
cannot be sure if the node is really offline, or active and suffering
from a network fault.
Likewise, any device that relies on the machine being active (such as
SSH-based "devices" used during testing) are inappropriate.
== Configuring STONITH ==
ifdef::pcs[]
. Find the correct driver: +pcs stonith list+
. Find the parameters associated with the device: +pcs stonith describe <agent name>+
. Create a local config to make changes to +pcs cluster cib stonith_cfg+
. Create the fencing resource using +pcs -f stonith_cfg stonith create <stonith_id>
<stonith device type> [stonith device options]+
. Set stonith-enable to true. +pcs -f stonith_cfg property set stonith-enabled=true+
endif::[]
ifdef::crmsh[]
. Find the correct driver: +stonith_admin --list-installed+
. Since every device is different, the parameters needed to configure
it will vary. To find out the parameters associated with the device,
run: +stonith_admin --metadata --agent type+
The output should be XML formatted text containing additional
parameter descriptions. We will endevor to make the output more
friendly in a later version.
. Enter the shell crm Create an editable copy of the existing
configuration +cib new stonith+ Create a fencing resource containing a
primitive resource with a class of stonith, a type of type and a
parameter for each of the values returned in step 2: +configure
primitive ...+
endif::[]
. If the device does not know how to fence nodes based on their uname,
you may also need to set the special +pcmk_host_map+ parameter. See
+man stonithd+ for details.
. If the device does not support the list command, you may also need
to set the special +pcmk_host_list+ and/or +pcmk_host_check+
parameters. See +man stonithd+ for details.
. If the device does not expect the victim to be specified with the
port parameter, you may also need to set the special
+pcmk_host_argument+ parameter. See +man stonithd+ for details.
ifdef::crmsh[]
. Upload it into the CIB from the shell: +cib commit stonith+
endif::[]
ifdef::pcs[]
-. Commit the new configuration. +pcs cluster cib-push stonith_cfg+
+. Commit the new configuration. +pcs cluster push cib stonith_cfg+
endif::[]
. Once the stonith resource is running, you can test it by executing:
+stonith_admin --reboot nodename+. Although you might want to stop the
cluster on that machine first.
== Example ==
Assuming we have an chassis containing four nodes and an IPMI device
active on 10.0.0.1, then we would chose the fence_ipmilan driver in step
2 and obtain the following list of parameters
.Obtaining a list of STONITH Parameters
ifdef::pcs[]
[source,C]
----
# pcs stonith describe fence_ipmilan
Stonith options for: fence_ipmilan
auth: IPMI Lan Auth type (md5, password, or none)
ipaddr: IPMI Lan IP to talk to
passwd: Password (if required) to control power on IPMI device
passwd_script: Script to retrieve password (if required)
lanplus: Use Lanplus
login: Username/Login (if required) to control power on IPMI device
action: Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata
timeout: Timeout (sec) for IPMI operation
cipher: Ciphersuite to use (same as ipmitool -C parameter)
method: Method to fence (onoff or cycle)
power_wait: Wait X seconds after on/off operation
delay: Wait X seconds before fencing is started
privlvl: Privilege level on IPMI device
verbose: Verbose mode
----
endif::[]
ifdef::crmsh[]
[source,C]
----
# stonith_admin --metadata -a fence_ipmilan
----
[source,XML]
----
<?xml version="1.0" ?>
<resource-agent name="fence_ipmilan" shortdesc="Fence agent for IPMI over LAN">
<longdesc>
fence_ipmilan is an I/O Fencing agent which can be used with machines controlled by IPMI. This agent calls support software using ipmitool (http://ipmitool.sf.net/).
To use fence_ipmilan with HP iLO 3 you have to enable lanplus option (lanplus / -P) and increase wait after operation to 4 seconds (power_wait=4 / -T 4)</longdesc>
<parameters>
<parameter name="auth" unique="1">
<getopt mixed="-A" />
<content type="string" />
<shortdesc lang="en">IPMI Lan Auth type (md5, password, or none)</shortdesc>
</parameter>
<parameter name="ipaddr" unique="1">
<getopt mixed="-a" />
<content type="string" />
<shortdesc lang="en">IPMI Lan IP to talk to</shortdesc>
</parameter>
<parameter name="passwd" unique="1">
<getopt mixed="-p" />
<content type="string" />
<shortdesc lang="en">Password (if required) to control power on IPMI device</shortdesc>
</parameter>
<parameter name="passwd_script" unique="1">
<getopt mixed="-S" />
<content type="string" />
<shortdesc lang="en">Script to retrieve password (if required)</shortdesc>
</parameter>
<parameter name="lanplus" unique="1">
<getopt mixed="-P" />
<content type="boolean" />
<shortdesc lang="en">Use Lanplus</shortdesc>
</parameter>
<parameter name="login" unique="1">
<getopt mixed="-l" />
<content type="string" />
<shortdesc lang="en">Username/Login (if required) to control power on IPMI device</shortdesc>
</parameter>
<parameter name="action" unique="1">
<getopt mixed="-o" />
<content type="string" default="reboot"/>
<shortdesc lang="en">Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata</shortdesc>
</parameter>
<parameter name="timeout" unique="1">
<getopt mixed="-t" />
<content type="string" />
<shortdesc lang="en">Timeout (sec) for IPMI operation</shortdesc>
</parameter>
<parameter name="cipher" unique="1">
<getopt mixed="-C" />
<content type="string" />
<shortdesc lang="en">Ciphersuite to use (same as ipmitool -C parameter)</shortdesc>
</parameter>
<parameter name="method" unique="1">
<getopt mixed="-M" />
<content type="string" default="onoff"/>
<shortdesc lang="en">Method to fence (onoff or cycle)</shortdesc>
</parameter>
<parameter name="power_wait" unique="1">
<getopt mixed="-T" />
<content type="string" default="2"/>
<shortdesc lang="en">Wait X seconds after on/off operation</shortdesc>
</parameter>
<parameter name="delay" unique="1">
<getopt mixed="-f" />
<content type="string" />
<shortdesc lang="en">Wait X seconds before fencing is started</shortdesc>
</parameter>
<parameter name="verbose" unique="1">
<getopt mixed="-v" />
<content type="boolean" />
<shortdesc lang="en">Verbose mode</shortdesc>
</parameter>
</parameters>
<actions>
<action name="on" />
<action name="off" />
<action name="reboot" />
<action name="status" />
<action name="diag" />
<action name="list" />
<action name="monitor" />
<action name="metadata" />
</actions>
</resource-agent>
----
endif::[]
from which we would create a STONITH resource fragment that might look