diff --git a/doc/sphinx/Clusters_from_Scratch/fencing.rst b/doc/sphinx/Clusters_from_Scratch/fencing.rst index 1e2192f8f4..2495af8466 100644 --- a/doc/sphinx/Clusters_from_Scratch/fencing.rst +++ b/doc/sphinx/Clusters_from_Scratch/fencing.rst @@ -1,240 +1,240 @@ .. index:: fencing Configure Fencing ----------------- What is Fencing? ################ Fencing protects your data from being corrupted, and your application from becoming unavailable, due to unintended concurrent access by rogue nodes. Just because a node is unresponsive doesn't mean it has stopped accessing your data. The only way to be 100% sure that your data is safe, is to use fencing to ensure that the node is truly offline before allowing the data to be accessed from another node. Fencing also has a role to play in the event that a clustered service cannot be stopped. In this case, the cluster uses fencing to force the whole node offline, thereby making it safe to start the service elsewhere. Fencing is also known as STONITH, an acronym for "Shoot The Other Node In The Head", since the most popular form of fencing is cutting a host's power. In order to guarantee the safety of your data [#]_, fencing is enabled by default. .. NOTE:: It is possible to tell the cluster not to use fencing, by setting the **stonith-enabled** cluster property to false: .. code-block:: none [root@pcmk-1 ~]# pcs property set stonith-enabled=false - [root@pcmk-1 ~]# crm_verify -L + [root@pcmk-1 ~]# pcs cluster verify --full However, this is completely inappropriate for a production cluster. It tells the cluster to simply pretend that failed nodes are safely powered off. Some vendors will refuse to support clusters that have fencing disabled. Even disabling it for a test cluster means you won't be able to test real failure scenarios. .. index:: single: fencing; device Choose a Fence Device ##################### The two broad categories of fence device are power fencing, which cuts off power to the target, and fabric fencing, which cuts off the target's access to some critical resource, such as a shared disk or access to the local network. Power fencing devices include: * Intelligent power switches * IPMI * Hardware watchdog device (alone, or in combination with shared storage used as a "poison pill" mechanism) Fabric fencing devices include: * Shared storage that can be cut off for a target host by another host (for example, an external storage device that supports SCSI-3 persistent reservations) * Intelligent network switches Using IPMI as a power fencing device may seem like a good choice. However, if the IPMI shares power and/or network access with the host (such as most onboard IPMI controllers), a power or network failure will cause both the host and its fencing device to fail. The cluster will be unable to recover, and must stop all resources to avoid a possible split-brain situation. Likewise, any device that relies on the machine being active (such as SSH-based "devices" sometimes used during testing) is inappropriate, because fencing will be required when the node is completely unresponsive. (Fence agents like ``fence_ilo_ssh``, which connects via SSH to an HP iLO but not to the cluster node, are fine.) Configure the Cluster for Fencing ################################# #. Install the fence agent(s). To see what packages are available, run ``dnf search fence-``. Be sure to install the package(s) on all cluster nodes. #. Configure the fence device itself to be able to fence your nodes and accept fencing requests. This includes any necessary configuration on the device and on the nodes, and any firewall or SELinux changes needed. Test the communication between the device and your nodes. #. Find the name of the correct fence agent: ``pcs stonith list`` #. Find the parameters associated with the device: ``pcs stonith describe `` #. Create a local copy of the CIB: ``pcs cluster cib stonith_cfg`` #. Create the fencing resource: ``pcs -f stonith_cfg stonith create [STONITH_DEVICE_OPTIONS]`` Any flags that do not take arguments, such as ``--ssl``, should be passed as ``ssl=1``. #. Ensure fencing is enabled in the cluster: ``pcs -f stonith_cfg property set stonith-enabled=true`` #. If the device does not know how to fence nodes based on their cluster node name, you may also need to set the special **pcmk_host_map** parameter. See ``man pacemaker-fenced`` for details. #. If the device does not support the **list** command, you may also need to set the special **pcmk_host_list** and/or **pcmk_host_check** parameters. See ``man pacemaker-fenced`` for details. #. If the device does not expect the victim to be specified with the **port** parameter, you may also need to set the special **pcmk_host_argument** parameter. See ``man pacemaker-fenced`` for details. #. Commit the new configuration: ``pcs cluster cib-push stonith_cfg`` #. Once the fence device resource is running, test it (you might want to stop the cluster on that machine first): - ``stonith_admin --reboot `` + ``pcs stonith fence `` Example ####### For this example, assume we have a chassis containing four nodes and a separately powered IPMI device active on 10.0.0.1. Following the steps above would go something like this: Step 1: Install the **fence-agents-ipmilan** package on both nodes. Step 2: Configure the IP address, authentication credentials, etc. in the IPMI device itself. Step 3: Choose the **fence_ipmilan** STONITH agent. Step 4: Obtain the agent's possible parameters: .. code-block:: none [root@pcmk-1 ~]# pcs stonith describe fence_ipmilan fence_ipmilan - Fence agent for IPMI fence_ipmilan is an I/O Fencing agentwhich can be used with machines controlled by IPMI.This agent calls support software ipmitool (http://ipmitool.sf.net/). WARNING! This fence agent might report success before the node is powered off. You should use -m/method onoff if your fence device works correctly with that option. Stonith options: auth: IPMI Lan Auth type. cipher: Ciphersuite to use (same as ipmitool -C parameter) hexadecimal_kg: Hexadecimal-encoded Kg key for IPMIv2 authentication ip: IP address or hostname of fencing device ipport: TCP/UDP port to use for connection with device lanplus: Use Lanplus to improve security of connection method: Method to fence password: Login password or passphrase password_script: Script to run to retrieve password plug: IP address or hostname of fencing device (together with --port-as-ip) privlvl: Privilege level on IPMI device target: Bridge IPMI requests to the remote target address username: Login name quiet: Disable logging to stderr. Does not affect --verbose or --debug-file or logging to syslog. verbose: Verbose mode. Multiple -v flags can be stacked on the command line (e.g., -vvv) to increase verbosity. verbose_level: Level of debugging detail in output. Defaults to the number of --verbose flags specified on the command line, or to 1 if verbose=1 in a stonith device configuration (i.e., on stdin). debug_file: Write debug information to given file delay: Wait X seconds before fencing is started disable_timeout: Disable timeout (true/false) (default: true when run from Pacemaker 2.0+) ipmitool_path: Path to ipmitool binary login_timeout: Wait X seconds for cmd prompt after login port_as_ip: Make "port/plug" to be an alias to IP address power_timeout: Test X seconds for status change after ON/OFF power_wait: Wait X seconds after issuing ON/OFF shell_timeout: Wait X seconds for cmd prompt after issuing command retry_on: Count of attempts to retry power on use_sudo: Use sudo (without password) when calling 3rd party software sudo_path: Path to sudo binary pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names. Eg. node1:1;node2:2,3 would tell the cluster to use port 1 for node1 and ports 2 and 3 for node2 pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static- list). pcmk_host_check: How to determine which machines are controlled by the device. Allowed values: dynamic- list (query the device via the 'list' command), static-list (check the pcmk_host_list attribute), status (query the device via the 'status' command), none (assume every device can fence every machine) pcmk_delay_max: Enable a random delay for stonith actions and specify the maximum of random delay. This prevents double fencing when using slow devices such as sbd. Use this to enable a random delay for stonith actions. The overall delay is derived from this random delay value adding a static delay so that the sum is kept below the maximum delay. pcmk_delay_base: Enable a base delay for stonith actions and specify base delay value. This prevents double fencing when different delays are configured on the nodes. Use this to enable a static delay for stonith actions. The overall delay is derived from a random delay value adding this static delay so that the sum is kept below the maximum delay. pcmk_action_limit: The maximum number of actions can be performed in parallel on this device Cluster property concurrent-fencing=true needs to be configured first. Then use this to specify the maximum number of actions can be performed in parallel on this device. -1 is unlimited. Default operations: monitor: interval=60s Step 5: ``pcs cluster cib stonith_cfg`` Step 6: Here are example parameters for creating our fence device resource: .. code-block:: none [root@pcmk-1 ~]# pcs -f stonith_cfg stonith create ipmi-fencing fence_ipmilan \ pcmk_host_list="pcmk-1 pcmk-2" ipaddr=10.0.0.1 login=testuser \ passwd=acd123 op monitor interval=60s [root@pcmk-1 ~]# pcs -f stonith_cfg stonith * ipmi-fencing (stonith:fence_ipmilan): Stopped Steps 7-10: Enable fencing in the cluster: .. code-block:: none [root@pcmk-1 ~]# pcs -f stonith_cfg property set stonith-enabled=true [root@pcmk-1 ~]# pcs -f stonith_cfg property Cluster Properties: cluster-infrastructure: corosync cluster-name: mycluster dc-version: 2.0.5-4.el8-ba59be7122 have-watchdog: false stonith-enabled: true Step 11: ``pcs cluster cib-push stonith_cfg --config`` Step 12: Test: .. code-block:: none [root@pcmk-1 ~]# pcs cluster stop pcmk-2 - [root@pcmk-1 ~]# stonith_admin --reboot pcmk-2 + [root@pcmk-1 ~]# pcs stonith fence pcmk-2 After a successful test, login to any rebooted nodes, and start the cluster (with ``pcs cluster start``). .. [#] If the data is corrupt, there is little point in continuing to make it available. diff --git a/doc/sphinx/Clusters_from_Scratch/verification.rst b/doc/sphinx/Clusters_from_Scratch/verification.rst index 28e6036c1a..8181449f66 100644 --- a/doc/sphinx/Clusters_from_Scratch/verification.rst +++ b/doc/sphinx/Clusters_from_Scratch/verification.rst @@ -1,213 +1,216 @@ Start and Verify Cluster ------------------------ Start the Cluster ################# Now that corosync is configured, it is time to start the cluster. The command below will start corosync and pacemaker on both nodes in the cluster. .. code-block:: none [root@pcmk-1 ~]# pcs cluster start --all pcmk-1: Starting Cluster... pcmk-2: Starting Cluster... .. NOTE:: An alternative to using the ``pcs cluster start --all`` command is to issue either of the below command sequences on each node in the cluster separately: .. code-block:: none # pcs cluster start Starting Cluster... or .. code-block:: none # systemctl start corosync.service # systemctl start pacemaker.service .. IMPORTANT:: In this example, we are not enabling the corosync and pacemaker services to start at boot. If a cluster node fails or is rebooted, you will need to run ``pcs cluster start [ | --all]`` to start the cluster on it. While you can enable the services to start at boot (for example, using ``pcs cluster enable [ | --all]``), requiring a manual start of cluster services gives you the opportunity to do a post-mortem investigation of a node failure before returning it to the cluster. Verify Corosync Installation ############################ First, use ``corosync-cfgtool`` to check whether cluster communication is happy: .. code-block:: none [root@pcmk-1 ~]# corosync-cfgtool -s Printing link status. Local node ID 1 LINK ID 0 addr = 192.168.122.101 status: nodeid 1: localhost nodeid 2: connected We can see here that everything appears normal with our fixed IP address (not a 127.0.0.x loopback address) listed as the **addr**, and **localhost** and **connected** for the statuses of nodeid 1 and nodeid 2, respectively. If you see something different, you might want to start by checking the node's network, firewall, and SELinux configurations. Next, check the membership and quorum APIs: .. code-block:: none [root@pcmk-1 ~]# corosync-cmapctl | grep members runtime.members.1.config_version (u64) = 0 runtime.members.1.ip (str) = r(0) ip(192.168.122.101) runtime.members.1.join_count (u32) = 1 runtime.members.1.status (str) = joined runtime.members.2.config_version (u64) = 0 runtime.members.2.ip (str) = r(0) ip(192.168.122.102) runtime.members.2.join_count (u32) = 1 runtime.members.2.status (str) = joined [root@pcmk-1 ~]# pcs status corosync Membership information ---------------------- Nodeid Votes Name 1 1 pcmk-1 (local) 2 1 pcmk-2 You should see both nodes have joined the cluster. Verify Pacemaker Installation ############################# Now that we have confirmed that Corosync is functional, we can check the rest of the stack. Pacemaker has already been started, so verify the necessary processes are running: .. code-block:: none [root@pcmk-1 ~]# ps axf PID TTY STAT TIME COMMAND 2 ? S 0:00 [kthreadd] ...lots of processes... 17121 ? SLsl 0:01 /usr/sbin/corosync -f 17133 ? Ss 0:00 /usr/sbin/pacemakerd 17134 ? Ss 0:00 \_ /usr/libexec/pacemaker/pacemaker-based 17135 ? Ss 0:00 \_ /usr/libexec/pacemaker/pacemaker-fenced 17136 ? Ss 0:00 \_ /usr/libexec/pacemaker/pacemaker-execd 17137 ? Ss 0:00 \_ /usr/libexec/pacemaker/pacemaker-attrd 17138 ? Ss 0:00 \_ /usr/libexec/pacemaker/pacemaker-schedulerd 17139 ? Ss 0:00 \_ /usr/libexec/pacemaker/pacemaker-controld If that looks OK, check the ``pcs status`` output: .. code-block:: none [root@pcmk-1 ~]# pcs status Cluster name: mycluster WARNINGS: No stonith devices and stonith-enabled is not false Cluster Summary: * Stack: corosync * Current DC: pcmk-2 (version 2.0.5-4.el8-ba59be7122) - partition with quorum * Last updated: Wed Jan 20 07:54:02 2021 * Last change: Wed Jan 20 07:48:25 2021 by hacluster via crmd on pcmk-2 * 2 nodes configured * 0 resource instances configured Node List: * Online: [ pcmk-1 pcmk-2 ] Full List of Resources: * No resources Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled Finally, ensure there are no start-up errors from corosync or pacemaker (aside from messages relating to not having STONITH configured, which are OK at this point): .. code-block:: none [root@pcmk-1 ~]# journalctl -b | grep -i error .. NOTE:: Other operating systems may report startup errors in other locations (for example, ``/var/log/messages``). Repeat these checks on the other node. The results should be the same. Explore the Existing Configuration ################################## For those who are not of afraid of XML, you can see the raw cluster configuration and status by using the ``pcs cluster cib`` command. .. topic:: The last XML you'll see in this document .. code-block:: none [root@pcmk-1 ~]# pcs cluster cib .. code-block:: xml Before we make any changes, it's a good idea to check the validity of the configuration. .. code-block:: none - [root@pcmk-1 ~]# crm_verify -L -V - error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined - error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option - error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity - Errors found during check: config not valid + [root@pcmk-1 ~]# pcs cluster verify --full + Error: invalid cib: + (unpack_resources) error: Resource start-up disabled since no STONITH resources have been defined + (unpack_resources) error: Either configure some or disable STONITH with the stonith-enabled option + (unpack_resources) error: NOTE: Clusters with shared data need STONITH to ensure data integrity + crm_verify: Errors found during check: config not valid + + Error: Errors have occurred, therefore pcs is unable to continue As you can see, the tool has found some errors. The cluster will not start any resources until we configure STONITH.