diff --git a/doc/Pacemaker_Explained/en-US/Ch-Stonith.txt b/doc/Pacemaker_Explained/en-US/Ch-Stonith.txt index 1025ded06d..1755afb129 100644 --- a/doc/Pacemaker_Explained/en-US/Ch-Stonith.txt +++ b/doc/Pacemaker_Explained/en-US/Ch-Stonith.txt @@ -1,322 +1,499 @@ = Configure STONITH = //// We prefer [[ch-stonith]], but older versions of asciidoc dont deal well with that construct for chapter headings //// anchor:ch-stonith[Chapter 13, STONITH] indexterm:[STONITH, Configuration] == What Is STONITH == STONITH is an acronym for Shoot-The-Other-Node-In-The-Head and it protects your data from being corrupted by rogue nodes or concurrent access. Just because a node is unresponsive, this doesn't mean it isn't accessing your data. The only way to be 100% sure that your data is safe, is to use STONITH so we can be certain that the node is truly offline, before allowing the data to be accessed from another node. STONITH also has a role to play in the event that a clustered service cannot be stopped. In this case, the cluster uses STONITH to force the whole node offline, thereby making it safe to start the service elsewhere. == What STONITH Device Should You Use == It is crucial that the STONITH device can allow the cluster to differentiate between a node failure and a network one. The biggest mistake people make in choosing a STONITH device is to use remote power switch (such as many on-board IMPI controllers) that shares power with the node it controls. In such cases, the cluster cannot be sure if the node is really offline, or active and suffering from a network fault. Likewise, any device that relies on the machine being active (such as SSH-based "devices" used during testing) are inappropriate. == Differences of STONITH Resources == Stonith resources are somewhat special in Pacemaker. In previous versions, only "running" resources could be used by Pacemaker for fencing. This requirement has been relaxed to allow other parts of the cluster (such as resources like DRBD) to reliably initiate fencing. footnote:[Fencing a node while Pacemaker was moving stonith resources around would otherwise fail] Now all nodes have access to their definitions and instantiate them on-the-fly when needed, however preference is given to 'verified' instances which are the ones the cluster has explicitly started. In the case of a cluster split, the partition with a verified instance will have a slight advantage as stonith-ng in the other partition will have to hear from all its current peers before choosing a node to perform the fencing. [NOTE] =========== To disable a fencing device/resource, 'target-role' can be set as you would for a normal resource. =========== [NOTE] =========== To prevent a specific node from using a fencing device, location constraints will work as expected. =========== [IMPORTANT] =========== Currently there is a limitation that fencing resources may only have a one set of meta-attributes and one set of instance-attributes. This can be revisited if it becomes a significant limitation for people. =========== +.Properties of Fencing Devices +[width="95%",cols="1m,1m,1m,5<",options="header",align="center"] +|========================================================= + +|Field +|Type +|Default +|Description + +|stonith-timeout +|time +|60s +|How long to wait for the STONITH action to complete per a stonith device. + Overrides the stonith-timeout cluster property + indexterm:[stonith-timeout,Fencing] + indexterm:[Fencing,Property,stonith-timeout] + +|priority +|integer +|0 +|The priority of the stonith resource. Devices are tried in order of highest priority to lowest. + indexterm:[priority,Fencing] + indexterm:[Fencing,Property,priority] + +|pcmk_host_argument +|string +|port +|Advanced use only: An alternate parameter to supply instead of 'port' + Some devices do not support the standard 'port' parameter or may provide additional ones. Use this to specify an alternate, device-specific, parameter that should indicate the machine to be fenced. A value of 'none' can be used to tell the cluster not to supply any additional parameters. + indexterm:[pcmk_host_argument,Fencing] + indexterm:[Fencing,Property,pcmk_host_argument] + +|pcmk_host_map +|string +| +|A mapping of host names to ports numbers for devices that do not support host names. + Eg. node1:1;node2:2,3 would tell the cluster to use port 1 for node1 and ports 2 and 3 for node2 + indexterm:[pcmk_host_map,Fencing] + indexterm:[Fencing,Property,pcmk_host_map] + +|pcmk_host_list +|string +| +|A list of machines controlled by this device (Optional unless pcmk_host_check=static-list). + indexterm:[pcmk_host_list,Fencing] + indexterm:[Fencing,Property,pcmk_host_list] + +|pcmk_host_check +|string +|dynamic-list +|How to determin which machines are controlled by the device. + Allowed values: dynamic-list (query the device), static-list (check the pcmk_host_list attribute), none (assume every device can fence every machine) + indexterm:[pcmk_host_check,Fencing] + indexterm:[Fencing,Property,pcmk_host_check] + +|pcmk_reboot_action +|string +|reboot +|Advanced use only: An alternate command to run instead of 'reboot' + Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'reboot' action. + indexterm:[pcmk_reboot_action,Fencing] + indexterm:[Fencing,Property,pcmk_reboot_action] + +|pcmk_reboot_timeout +|time +|60s +|Advanced use only: Specify an alternate timeout to use for reboot actions instead of stonith-timeout + Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'reboot' actions. + indexterm:[pcmk_reboot_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_reboot_timeout] + +|pcmk_reboot_retries +|integer +|2 +|Advanced use only: The maximum number of times to retry the 'reboot' command within the timeout period + Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'reboot' actions before giving up. + indexterm:[pcmk_reboot_retries,Fencing] + indexterm:[Fencing,Property,pcmk_reboot_retries] + +|pcmk_off_action +|string +|off +|Advanced use only: An alternate command to run instead of 'off' + Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'off' action. + indexterm:[pcmk_off_action,Fencing] + indexterm:[Fencing,Property,pcmk_off_action] + +|pcmk_off_timeout +|time +|60s +|Advanced use only: Specify an alternate timeout to use for off actions instead of stonith-timeout + Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'off' actions. + indexterm:[pcmk_off_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_off_timeout] + +|pcmk_off_retries +|integer +|2 +|Advanced use only: The maximum number of times to retry the 'off' command within the timeout period + Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'off' actions before giving up. + indexterm:[pcmk_off_retries,Fencing] + indexterm:[Fencing,Property,pcmk_off_retries] + +|pcmk_list_action +|string +|list +|Advanced use only: An alternate command to run instead of 'list' + Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'list' action. + indexterm:[pcmk_list_action,Fencing] + indexterm:[Fencing,Property,pcmk_list_action] + +|pcmk_list_timeout +|time +|60s +|Advanced use only: Specify an alternate timeout to use for list actions instead of stonith-timeout + Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'list' actions. + indexterm:[pcmk_list_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_list_timeout] + +|pcmk_list_retries +|integer +|2 +|Advanced use only: The maximum number of times to retry the 'list' command within the timeout period + Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'list' actions before giving up. + indexterm:[pcmk_list_retries,Fencing] + indexterm:[Fencing,Property,pcmk_list_retries] + +|pcmk_monitor_action +|string +|monitor +|Advanced use only: An alternate command to run instead of 'monitor' + Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'monitor' action. + indexterm:[pcmk_monitor_action,Fencing] + indexterm:[Fencing,Property,pcmk_monitor_action] + +|pcmk_monitor_timeout +|time +|60s +|Advanced use only: Specify an alternate timeout to use for monitor actions instead of stonith-timeout + Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'monitor' actions. + indexterm:[pcmk_monitor_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_monitor_timeout] + +|pcmk_monitor_retries +|integer +|2 +|Advanced use only: The maximum number of times to retry the 'monitor' command within the timeout period + Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'monitor' actions before giving up. + indexterm:[pcmk_monitor_retries,Fencing] + indexterm:[Fencing,Property,pcmk_monitor_retries] + +|pcmk_status_action +|string +|status +|Advanced use only: An alternate command to run instead of 'status' + Some devices do not support the standard commands or may provide additional ones. Use this to specify an alternate, device-specific, command that implements the 'status' action. + indexterm:[pcmk_status_action,Fencing] + indexterm:[Fencing,Property,pcmk_status_action] + +|pcmk_status_timeout +|time +|60s +|Advanced use only: Specify an alternate timeout to use for status actions instead of stonith-timeout + Some devices need much more/less time to complete than normal. Use this to specify an alternate, device-specific, timeout for 'status' actions. + indexterm:[pcmk_status_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_status_timeout] + +|pcmk_status_retries +|integer +|2 +|Advanced use only: The maximum number of times to retry the 'status' command within the timeout period + Some devices do not support multiple connections. Operations may 'fail' if the device is busy with another task so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries 'status' actions before giving up. + indexterm:[pcmk_status_retries,Fencing] + indexterm:[Fencing,Property,pcmk_status_retries] + +|========================================================= + == Configuring STONITH == [NOTE] =========== Both configuration shells include functionality to simplify the process below, particularly the step for deciding which parameters are required. However since this document deals only with core components, you should refer to the Stonith chapter of +Clusters from Scratch+ for those details. =========== . Find the correct driver: +stonith_admin --list-installed+ . Find the required parameters associated with the device: +stonith_admin --metadata --agent + . Create a file called +stonith.xml+ containing a primitive resource with a class of 'stonith', a type of and a parameter for each of the values returned in step 2. . If the device does not know how to fence nodes based on their uname, you may also need to set the special +pcmk_host_map+ parameter. See +man stonithd+ for details. . If the device does not support the list command, you may also need to set the special +pcmk_host_list+ and/or +pcmk_host_check+ parameters. See +man stonithd+ for details. . If the device does not expect the victim to be specified with the port parameter, you may also need to set the special +pcmk_host_argument+ parameter. See +man stonithd+ for details. . Upload it into the CIB using cibadmin: +cibadmin -C -o resources --xml-file stonith.xml+ . Set stonith-enabled to true. +crm_attribute -t crm_config -n stonith-enabled -v true+ . Once the stonith resource is running, you can test it by executing: +stonith_admin --reboot nodename+. Although you might want to stop the cluster on that machine first. === Example === Assuming we have an chassis containing four nodes and an IPMI device active on 10.0.0.1, then we would chose the fence_ipmilan driver in step 2 and obtain the following list of parameters .Obtaining a list of STONITH Parameters [source,C] ---- # stonith_admin --metadata -a fence_ipmilan ---- [source,XML] ---- fence_ipmilan is an I/O Fencing agent which can be used with machines controlled by IPMI. This agent calls support software using ipmitool (http://ipmitool.sf.net/). To use fence_ipmilan with HP iLO 3 you have to enable lanplus option (lanplus / -P) and increase wait after operation to 4 seconds (power_wait=4 / -T 4) IPMI Lan Auth type (md5, password, or none) IPMI Lan IP to talk to Password (if required) to control power on IPMI device Script to retrieve password (if required) Use Lanplus Username/Login (if required) to control power on IPMI device Operation to perform. Valid operations: on, off, reboot, status, list, diag, monitor or metadata Timeout (sec) for IPMI operation Ciphersuite to use (same as ipmitool -C parameter) Method to fence (onoff or cycle) Wait X seconds after on/off operation Wait X seconds before fencing is started Verbose mode ---- from which we would create a STONITH resource fragment that might look like this: .Sample STONITH Resource [source,XML] ---- ---- And finally, since we disabled it earlier, we need to re-enable STONITH. [source,Bash] ---- # crm_attribute -t crm_config -n stonith-enabled -v true ---- == Advanced Fencing Configurations == Some people consider that having one fencing device is a single point of failure footnote:[Not true, since a node or resource must fail before fencing even has a chance to], others prefer removing the node from the storage and network instead of turning it off. Whatever the reason, Pacemaker supports fencing nodes with multiple devices through a feature called fencing topologies. Simply create the individual devices as you normally would and then define one or more fencing levels in the fencing-topology section in the configuration. * Each level is attempted in +ascending index+ order * If a device fails, +processing terminates+ for the current level (optimized boolean logic) and the next is attempted * If the operation succeeds for all the listed devices in a level, the level is deemed to have passed * The operation is finished +when a level has passed+ (success), or all levels have been attempted (failed) * If the operation failed, the next step is determined by the Policy Engine and/or crmd. Some suggested uses of topologies include: * try poison-pill and fail back to power * try disk and network, and fall back to power if either fails * initiate a kdump and then poweroff the node .Properties of Fencing Levels [width="95%",cols="1m,6<",options="header",align="center"] |========================================================= |Field |Description |id |Your name for the level indexterm:[id,fencing-level] indexterm:[Fencing,fencing-level,id] |target |The node to which this level applies indexterm:[target,fencing-level] indexterm:[Fencing,fencing-level,target] |index |The order in which to attempt the levels. Levels are attempted in +ascending index+ order +until one succeeds+. indexterm:[index,fencing-level] indexterm:[Fencing,fencing-level,index] |devices |A comma separated list of devices for which the indexterm:[devices,fencing-level] indexterm:[Fencing,fencing-level,devices] |========================================================= .Example use of Fencing Topologies [source,XML] ---- ... ... ----