diff --git a/doc/sphinx/Clusters_from_Scratch/intro.rst b/doc/sphinx/Clusters_from_Scratch/intro.rst index 733bdd8c60..eac0035379 100644 --- a/doc/sphinx/Clusters_from_Scratch/intro.rst +++ b/doc/sphinx/Clusters_from_Scratch/intro.rst @@ -1,29 +1,29 @@ -Read-Me-First -------------- +Introduction +------------ The Scope of this Document ########################## Computer clusters can be used to provide highly available services or resources. The redundancy of multiple machines is used to guard against failures of many types. This document will walk through the installation and setup of simple clusters using the |CFS_DISTRO| distribution, version |CFS_DISTRO_VER|. The clusters described here will use Pacemaker and Corosync to provide resource management and messaging. Required packages and modifications to their configuration files are described along with the use of the Pacemaker command line tool for generating the XML used for cluster control. Pacemaker is a central component and provides the resource management required in these systems. This management includes detecting and recovering from the failure of various nodes, resources and services under its control. When more in-depth information is required, and for real-world usage, please refer to the `Pacemaker Explained `_ manual. .. include:: ../shared/pacemaker-intro.rst diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am index d8ece45b3a..1593607c1f 100644 --- a/doc/sphinx/Makefile.am +++ b/doc/sphinx/Makefile.am @@ -1,114 +1,121 @@ # # Copyright 2003-2020 the Pacemaker project contributors # # The version control history for this file may have further details. # # This source code is licensed under the GNU General Public License version 2 # or later (GPLv2+) WITHOUT ANY WARRANTY. # include $(top_srcdir)/mk/common.mk # Things you might want to override on the command line # Books to generate -BOOKS ?= Clusters_from_Scratch \ - Pacemaker_Administration \ +BOOKS ?= Clusters_from_Scratch \ + Pacemaker_Administration \ Pacemaker_Development \ + Pacemaker_Explained \ Pacemaker_Remote # Output formats to generate. Possible values: # html (multiple HTML files) # dirhtml (HTML files named index.html in multiple directories) # singlehtml (a single large HTML file) # text # pdf # epub # latex # linkcheck (not actually a format; check validity of external links) # # The results will end up in /_build/ BOOK_FORMATS ?= html # Set to "a4" or "letter" if building latex format PAPER ?= letter # Additional options for sphinx-build SPHINXFLAGS ?= # toplevel rsync destination for www targets (without trailing slash) RSYNC_DEST ?= root@www.clusterlabs.org:/var/www/html # End of useful overrides EXTRA_DIST = $(wildcard */*.rst) # recursive, preserve symlinks/permissions/times, verbose, compress, # don't cross filesystems, sparse, show progress RSYNC_OPTS = -rlptvzxS --progress BOOK_RSYNC_DEST = $(RSYNC_DEST)/$(PACKAGE)/doc/$(PACKAGE_SERIES) TAG ?= $(shell [ -n "`git tag --points-at HEAD | head -1`" ] \ && ( git tag --points-at HEAD | head -1 ) \ || git log --pretty=format:Pacemaker-2.0.3-%h -n 1 HEAD) BOOK = none +DEPS_Clusters_from_Scratch = shared/pacemaker-intro.rst +DEPS_Pacemaker_Administration = shared/pacemaker-intro.rst +DEPS_Pacemaker_Development = +DEPS_Pacemaker_Explained = shared/pacemaker-intro.rst +DEPS_Pacemaker_Remote = $(wildcard $(srcdir)/Pacemaker_Remote/images/*.png) + if BUILD_SPHINX_DOCS $(BOOKS:%=%/conf.py): conf.py.in $(AM_V_GEN)sed \ -e 's/%VERSION%/$(VERSION)/g' \ -e 's/%BOOK_ID%/$(@:%/conf.py=%)/g' \ -e 's/%BOOK_TITLE%/$(subst _, ,$(@:%/conf.py=%))/g' \ $(<) > "$@" -$(BOOK)/_build: _static/pacemaker.css $(BOOK)/conf.py $(wildcard $(srcdir)/$(BOOK)/*.rst) +$(BOOK)/_build: _static/pacemaker.css $(BOOK)/conf.py $(DEPS_$(BOOK)) $(wildcard $(srcdir)/$(BOOK)/*.rst) @echo 'Building "$(subst _, ,$(BOOK))" because of $?' $(PCMK_quiet) $(AM_V_at)rm -rf "$@" $(AM_V_BOOK)for format in $(BOOK_FORMATS); do \ echo -e "\n * Building $$format" $(PCMK_quiet); \ doctrees="doctrees"; \ real_format="$$format"; \ case "$$format" in \ pdf) real_format="latex" ;; \ gettext) doctrees="gettext-doctrees" ;; \ esac; \ $(SPHINX) -b "$$real_format" -d "$@/$$doctrees" \ -c "$(builddir)/$(BOOK)" \ -D latex_paper_size=$(PAPER) $(SPHINXFLAGS) \ "$(srcdir)/$(BOOK)" "$@/$$format" \ $(PCMK_quiet); \ if [ "$$format" = "pdf" ]; then \ $(MAKE) $(AM_MAKEFLAGS) -C "$@/$$format" \ all-pdf; \ fi; \ done endif .PHONY: books-upload books-upload: all if BUILD_SPHINX_DOCS @echo "Uploading $(PACKAGE_SERIES) documentation set" @for book in $(BOOKS); do \ echo " * $$book"; \ buildfile="$$book/_build/build-$(PACKAGE_SERIES).txt"; \ echo "Generated on `date --utc` from version $(TAG)" \ > "$$buildfile"; \ rsync $(RSYNC_OPTS) "$$buildfile" \ $(BOOK_FORMATS:%=$$book/_build/%) \ "$(BOOK_RSYNC_DEST)/$$book/"; \ done endif all-local: if BUILD_SPHINX_DOCS @for book in $(BOOKS); do \ $(MAKE) $(AM_MAKEFLAGS) BOOK=$$book \ PAPER="$(PAPER)" SPHINXFLAGS="$(SPHINXFLAGS)" \ BOOK_FORMATS="$(BOOK_FORMATS)" $$book/_build; \ done endif clean-local: $(AM_V_at)-rm -rf $(BOOKS:%="$(builddir)/%/_build") $(BOOKS:%="$(builddir)/%/conf.py") diff --git a/doc/sphinx/Pacemaker_Administration/agents.rst b/doc/sphinx/Pacemaker_Administration/agents.rst index ba3645e0b2..18e1c5ce13 100644 --- a/doc/sphinx/Pacemaker_Administration/agents.rst +++ b/doc/sphinx/Pacemaker_Administration/agents.rst @@ -1,380 +1,380 @@ Resource Agents --------------- Resource Agent Actions ###################### If one resource depends on another resource via constraints, the cluster will interpret an expected result as sufficient to continue with dependent actions. This may cause timing issues if the resource agent start returns before the service is not only launched but fully ready to perform its function, or if the resource agent stop returns before the service has fully released all its claims on system resources. At a minimum, the start or stop should not return before a status command would return the expected (started or stopped) result. OCF Resource Agents ################### Location of Custom Scripts __________________________ .. index:: OCF resource agents OCF Resource Agents are found in ``/usr/lib/ocf/resource.d/$PROVIDER`` When creating your own agents, you are encouraged to create a new directory under ``/usr/lib/ocf/resource.d/`` so that they are not confused with (or overwritten by) the agents shipped by existing providers. So, for example, if you choose the provider name of big-corp and want a new resource named big-app, you would create a resource agent called ``/usr/lib/ocf/resource.d/big-corp/big-app`` and define a resource: .. code-block: xml Actions _______ All OCF resource agents are required to implement the following actions. -.. table:: Required Actions for OCF Agents - -+--------------+-------------+------------------------------------------------+ -| Action | Description | Instructions | -+==============+=============+================================================+ -| start | Start the | Return 0 on success and an appropriate | -| | resource | error code otherwise. Must not report | -| | | success until the resource is fully | -| | | active. | -| | | | -| | | .. index:: | -| | | pair: start; OCF action | -| | | pair: start; action | -+--------------+-------------+------------------------------------------------+ -| stop | Stop the | Return 0 on success and an appropriate | -| | resource | error code otherwise. Must not report | -| | | success until the resource is fully | -| | | stopped. | -| | | | -| | | .. index:: | -| | | pair: stop; OCF action | -| | | pair: stop; action | -+--------------+-------------+------------------------------------------------+ -| monitor | Check the | Exit 0 if the resource is running, 7 | -| | resource's | if it is stopped, and any other OCF | -| | state | exit code if it is failed. NOTE: The | -| | | monitor script should test the state | -| | | of the resource on the local machine | -| | | only. | -| | | | -| | | .. index:: | -| | | pair: monitor; OCF action | -| | | pair: monitor; action | -+--------------+-------------+------------------------------------------------+ -| meta-data | Describe | Provide information about this | -| | the | resource in the XML format defined by | -| | resource | the OCF standard. Exit with 0. NOTE: | -| | | This is *not* required to be performed | -| | | as root. | -| | | | -| | | .. index:: | -| | | pair: meta-data; OCF action | -| | | pair: meta-data; action | -+--------------+-------------+------------------------------------------------+ -| validate-all | Verify the | Return 0 if parameters are valid, 2 if | -| | supplied | not valid, and 6 if resource is not | -| | parameters | configured. | -| | | | -| | | .. index:: | -| | | pair: validate-all; OCF action | -| | | pair: validate-all; action | -+--------------+-------------+------------------------------------------------+ +.. table:: **Required Actions for OCF Agents** + + +--------------+-------------+------------------------------------------------+ + | Action | Description | Instructions | + +==============+=============+================================================+ + | start | Start the | Return 0 on success and an appropriate | + | | resource | error code otherwise. Must not report | + | | | success until the resource is fully | + | | | active. | + | | | | + | | | .. index:: | + | | | pair: start; OCF action | + | | | pair: start; action | + +--------------+-------------+------------------------------------------------+ + | stop | Stop the | Return 0 on success and an appropriate | + | | resource | error code otherwise. Must not report | + | | | success until the resource is fully | + | | | stopped. | + | | | | + | | | .. index:: | + | | | pair: stop; OCF action | + | | | pair: stop; action | + +--------------+-------------+------------------------------------------------+ + | monitor | Check the | Exit 0 if the resource is running, 7 | + | | resource's | if it is stopped, and any other OCF | + | | state | exit code if it is failed. NOTE: The | + | | | monitor script should test the state | + | | | of the resource on the local machine | + | | | only. | + | | | | + | | | .. index:: | + | | | pair: monitor; OCF action | + | | | pair: monitor; action | + +--------------+-------------+------------------------------------------------+ + | meta-data | Describe | Provide information about this | + | | the | resource in the XML format defined by | + | | resource | the OCF standard. Exit with 0. NOTE: | + | | | This is *not* required to be performed | + | | | as root. | + | | | | + | | | .. index:: | + | | | pair: meta-data; OCF action | + | | | pair: meta-data; action | + +--------------+-------------+------------------------------------------------+ + | validate-all | Verify the | Return 0 if parameters are valid, 2 if | + | | supplied | not valid, and 6 if resource is not | + | | parameters | configured. | + | | | | + | | | .. index:: | + | | | pair: validate-all; OCF action | + | | | pair: validate-all; action | + +--------------+-------------+------------------------------------------------+ Additional requirements (not part of the OCF specification) are placed on agents that will be used for advanced concepts such as clone resources. -.. table:: Optional Actions for OCF Resource Agents - -+--------------+-------------+------------------------------------------------+ -| Action | Description | Instructions | -+==============+=============+================================================+ -| promote | Promote the | Return 0 on success | -| | local | | -| | instance of | .. index:: | -| | a promotable| pair: promote; OCF action | -| | clone | pair: promote; action | -| | resource to | | -| | the master | | -| | (primary) | | -| | state. | | -+--------------+-------------+------------------------------------------------+ -| demote | Demote the | Return 0 on success | -| | local | | -| | instance of | .. index:: | -| | a promotable| pair: demote; OCF action | -| | clone | pair: demote; action | -| | resource to | | -| | the slave | | -| | (secondary) | | -| | state. | | -+--------------+-------------+------------------------------------------------+ -| notify | Used by the | Must not fail. Must exit with 0 | -| | cluster to | | -| | send | .. index:: | -| | the agent | pair: notify; OCF action | -| | pre- and | pair: notify; action | -| | post- | | -| | notification| | -| | events | | -| | telling the | | -| | resource | | -| | what has | | -| | happened and| | -| | will happen.| | -+--------------+-------------+------------------------------------------------+ +.. table:: **Optional Actions for OCF Resource Agents** + + +--------------+-------------+------------------------------------------------+ + | Action | Description | Instructions | + +==============+=============+================================================+ + | promote | Promote the | Return 0 on success | + | | local | | + | | instance of | .. index:: | + | | a promotable| pair: promote; OCF action | + | | clone | pair: promote; action | + | | resource to | | + | | the master | | + | | (primary) | | + | | state. | | + +--------------+-------------+------------------------------------------------+ + | demote | Demote the | Return 0 on success | + | | local | | + | | instance of | .. index:: | + | | a promotable| pair: demote; OCF action | + | | clone | pair: demote; action | + | | resource to | | + | | the slave | | + | | (secondary) | | + | | state. | | + +--------------+-------------+------------------------------------------------+ + | notify | Used by the | Must not fail. Must exit with 0 | + | | cluster to | | + | | send | .. index:: | + | | the agent | pair: notify; OCF action | + | | pre- and | pair: notify; action | + | | post- | | + | | notification| | + | | events | | + | | telling the | | + | | resource | | + | | what has | | + | | happened and| | + | | will happen.| | + +--------------+-------------+------------------------------------------------+ One action specified in the OCF specs, ``recover``, is not currently used by the cluster. It is intended to be a variant of the ``start`` action that tries to recover a resource locally. .. important:: - If you create a new OCF resource agent, use `ocf-tester` to verify that the - agent complies with the OCF standard properly. + If you create a new OCF resource agent, use `ocf-tester` to verify that the + agent complies with the OCF standard properly. .. index:: ocf-tester How are OCF Return Codes Interpreted? _____________________________________ The first thing the cluster does is to check the return code against the expected result. If the result does not match the expected value, then the operation is considered to have failed, and recovery action is initiated. There are three types of failure recovery: -.. table:: Types of recovery performed by the cluster - -+-------+------------------------------+--------------------------------------+ -| Type | Description | Action Taken by the Cluster | -+=======+==============================+======================================+ -| soft | A transient error occurred | Restart the resource or move it to a | -| | | new location | -| | .. index:: | | -| | pair: soft; OCF error | | -+-------+------------------------------+--------------------------------------+ -| hard | A non-transient error that | Move the resource elsewhere and | -| | may be specific to the | prevent it from being retried on the | -| | current node | current node | -| | | | -| | .. index:: | | -| | pair: hard; OCF error | | -+-------+------------------------------+--------------------------------------+ -| fatal | A non-transient error that | Stop the resource and prevent it | -| | will be common to all | from being started on any cluster | -| | cluster nodes (e.g. a bad | node | -| | configuration was specified) | | -| | | | -| | .. index:: | | -| | pair: fatal; OCF error | | -+-------+------------------------------+--------------------------------------+ +.. table:: **Types of recovery performed by the cluster** + + +-------+------------------------------+--------------------------------------+ + | Type | Description | Action Taken by the Cluster | + +=======+==============================+======================================+ + | soft | A transient error occurred | Restart the resource or move it to a | + | | | new location | + | | .. index:: | | + | | pair: soft; OCF error | | + +-------+------------------------------+--------------------------------------+ + | hard | A non-transient error that | Move the resource elsewhere and | + | | may be specific to the | prevent it from being retried on the | + | | current node | current node | + | | | | + | | .. index:: | | + | | pair: hard; OCF error | | + +-------+------------------------------+--------------------------------------+ + | fatal | A non-transient error that | Stop the resource and prevent it | + | | will be common to all | from being started on any cluster | + | | cluster nodes (e.g. a bad | node | + | | configuration was specified) | | + | | | | + | | .. index:: | | + | | pair: fatal; OCF error | | + +-------+------------------------------+--------------------------------------+ .. _ocf_return_codes: OCF Return Codes ________________ The following table outlines the different OCF return codes and the type of recovery the cluster will initiate when a failure code is received. Although counterintuitive, even actions that return 0 (aka. ``OCF_SUCCESS``) can be considered to have failed, if 0 was not the expected return value. -.. table:: OCF Exit Codes and their Recovery Types - -+-------+-----------------------+---------------------------------------------+----------+ -| Exit | OCF Alias | Description | Recovery | -| Code | | | | -+=======+=======================+=============================================+==========+ -| 0 | OCF_SUCCESS | Success. The command completed successfully.| soft | -| | | This is the expected result for all start, | | -| | | stop, promote and demote commands. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_SUCCESS | | -| | | pair: return code; 0 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 1 | OCF_ERR_GENERIC | Generic "there was a problem" | soft | -| | | error code. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_ERR_GENERIC | | -| | | pair: return code; 1 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 2 | OCF_ERR_ARGS | The resource's configuration is not valid on| hard | -| | | this machine. E.g. it refers to a location | | -| | | not found on the node. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_ERR_ARGS | | -| | | pair: return code; 2 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 3 | OCF_ERR_UNIMPLEMENTED | The requested action is not | hard | -| | | implemented. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_ERR_UNIMPLEMENTED | | -| | | pair: return code; 3 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 4 | OCF_ERR_PERM | The resource agent does not have | hard | -| | | sufficient privileges to complete the task. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_ERR_PERM | | -| | | pair: return code; 4 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 5 | OCF_ERR_INSTALLED | The tools required by the resource are | hard | -| | | not installed on this machine. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_ERR_INSTALLED | | -| | | pair: return code; 5 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 6 | OCF_ERR_CONFIGURED | The resource's configuration is invalid. | fatal | -| | | E.g. required parameters are missing. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_ERR_CONFIGURED | | -| | | pair: return code; 6 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 7 | OCF_NOT_RUNNING | The resource is safely stopped. The cluster | N/A | -| | | will not attempt to stop a resource that | | -| | | returns this for any action. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_NOT_RUNNING | | -| | | pair: return code; 7 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 8 | OCF_RUNNING_MASTER | The resource is running in | soft | -| | | master mode. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_RUNNING_MASTER | | -| | | pair: return code; 8 | | -+-------+-----------------------+---------------------------------------------+----------+ -| 9 | OCF_FAILED_MASTER | The resource is in master mode but has | soft | -| | | failed. The resource will be demoted, | | -| | | stopped and then started (and possibly | | -| | | promoted) again. | | -| | | | | -| | | .. index:: | | -| | | pair: return code; OCF_FAILED_MASTER | | -| | | pair: return code; 9 | | -+-------+-----------------------+---------------------------------------------+----------+ -| other | *none* | Custom error code. | soft | -| | | | | -| | | .. index:: | | -| | | pair: return code; other | | -+-------+-----------------------+---------------------------------------------+----------+ +.. table:: **OCF Exit Codes and their Recovery Types** + + +-------+-----------------------+---------------------------------------------+----------+ + | Exit | OCF Alias | Description | Recovery | + | Code | | | | + +=======+=======================+=============================================+==========+ + | 0 | OCF_SUCCESS | Success. The command completed successfully.| soft | + | | | This is the expected result for all start, | | + | | | stop, promote and demote commands. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_SUCCESS | | + | | | pair: return code; 0 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 1 | OCF_ERR_GENERIC | Generic "there was a problem" | soft | + | | | error code. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_ERR_GENERIC | | + | | | pair: return code; 1 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 2 | OCF_ERR_ARGS | The resource's configuration is not valid on| hard | + | | | this machine. E.g. it refers to a location | | + | | | not found on the node. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_ERR_ARGS | | + | | | pair: return code; 2 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 3 | OCF_ERR_UNIMPLEMENTED | The requested action is not | hard | + | | | implemented. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_ERR_UNIMPLEMENTED | | + | | | pair: return code; 3 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 4 | OCF_ERR_PERM | The resource agent does not have | hard | + | | | sufficient privileges to complete the task. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_ERR_PERM | | + | | | pair: return code; 4 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 5 | OCF_ERR_INSTALLED | The tools required by the resource are | hard | + | | | not installed on this machine. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_ERR_INSTALLED | | + | | | pair: return code; 5 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 6 | OCF_ERR_CONFIGURED | The resource's configuration is invalid. | fatal | + | | | E.g. required parameters are missing. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_ERR_CONFIGURED | | + | | | pair: return code; 6 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 7 | OCF_NOT_RUNNING | The resource is safely stopped. The cluster | N/A | + | | | will not attempt to stop a resource that | | + | | | returns this for any action. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_NOT_RUNNING | | + | | | pair: return code; 7 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 8 | OCF_RUNNING_MASTER | The resource is running in | soft | + | | | master mode. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_RUNNING_MASTER | | + | | | pair: return code; 8 | | + +-------+-----------------------+---------------------------------------------+----------+ + | 9 | OCF_FAILED_MASTER | The resource is in master mode but has | soft | + | | | failed. The resource will be demoted, | | + | | | stopped and then started (and possibly | | + | | | promoted) again. | | + | | | | | + | | | .. index:: | | + | | | pair: return code; OCF_FAILED_MASTER | | + | | | pair: return code; 9 | | + +-------+-----------------------+---------------------------------------------+----------+ + | other | *none* | Custom error code. | soft | + | | | | | + | | | .. index:: | | + | | | pair: return code; other | | + +-------+-----------------------+---------------------------------------------+----------+ Exceptions to the recovery handling described above: * Probes (non-recurring monitor actions) that find a resource active (or in master mode) will not result in recovery action unless it is also found active elsewhere. * The recovery action taken when a resource is found active more than once is determined by the resource's ``multiple-active`` property. * Recurring actions that return ``OCF_ERR_UNIMPLEMENTED`` do not cause any type of recovery. LSB Resource Agents (Init Scripts) ################################## LSB Compliance ______________ The relevant part of the `LSB specifications `_ includes a description of all the return codes listed here. Assuming `some_service` is configured correctly and currently inactive, the following sequence will help you determine if it is LSB-compatible: #. Start (stopped): .. code-block:: none # /etc/init.d/some_service start ; echo "result: $?" * Did the service start? * Did the echo command print ``result: 0`` (in addition to the init script's usual output)? #. Status (running): .. code-block:: none # /etc/init.d/some_service status ; echo "result: $?" * Did the script accept the command? * Did the script indicate the service was running? * Did the echo command print ``result: 0`` (in addition to the init script's usual output)? #. Start (running): .. code-block:: none # /etc/init.d/some_service start ; echo "result: $?" * Is the service still running? * Did the echo command print ``result: 0`` (in addition to the init script's usual output)? #. Stop (running): .. code-block:: none # /etc/init.d/some_service stop ; echo "result: $?" * Was the service stopped? * Did the echo command print ``result: 0`` (in addition to the init script's usual output)? #. Status (stopped): .. code-block:: none # /etc/init.d/some_service status ; echo "result: $?" * Did the script accept the command? * Did the script indicate the service was not running? * Did the echo command print ``result: 3`` (in addition to the init script's usual output)? #. Stop (stopped): .. code-block:: none # /etc/init.d/some_service stop ; echo "result: $?" * Is the service still stopped? * Did the echo command print ``result: 0`` (in addition to the init script's usual output)? #. Status (failed): This step is not readily testable and relies on manual inspection of the script. The script can use one of the error codes (other than 3) listed in the LSB spec to indicate that it is active but failed. This tells the cluster that before moving the resource to another node, it needs to stop it on the existing one first. If the answer to any of the above questions is no, then the script is not LSB-compliant. Your options are then to either fix the script or write an OCF agent based on the existing script. diff --git a/doc/sphinx/Pacemaker_Administration/configuring.rst b/doc/sphinx/Pacemaker_Administration/configuring.rst index 626c396bbb..b0321f1909 100644 --- a/doc/sphinx/Pacemaker_Administration/configuring.rst +++ b/doc/sphinx/Pacemaker_Administration/configuring.rst @@ -1,264 +1,264 @@ Configuring Pacemaker --------------------- Pacemaker's configuration, the CIB, is stored in XML format. Cluster administrators have multiple options for modifying the configuration either via the XML, or at a more abstract (and easier for humans to understand) level. Pacemaker reacts to configuration changes as soon as they are saved. Pacemaker's command-line tools and most higher-level tools provide the ability to batch changes together and commit them at once, rather than make a series of small changes, which could cause avoid unnecessary actions as Pacemaker responds to each change individually. Pacemaker tracks revisions to the configuration and will reject any update older than the current revision. Thus, it is a good idea to serialize all changes to the configuration. Avoid attempting simultaneous changes, whether on the same node or different nodes, and whether manually or using some automated configuration tool. .. note:: It is not necessary to update the configuration on all cluster nodes. Pacemaker immediately synchronizes changes to all active members of the cluster. To reduce bandwidth, the cluster only broadcasts the incremental updates that result from your changes and uses checksums to ensure that each copy is consistent. Configuration Using Higher-level Tools ______________________________________ Most users will benefit from using higher-level tools provided by projects separate from Pacemaker. Some of the most commonly used include the crm shell, hawk, and pcs. [#]_ See those projects' documentation for details on how to configure Pacemaker using them. Configuration Using Pacemaker's Command-Line Tools __________________________________________________ Pacemaker provides lower-level, command-line tools to manage the cluster. Most configuration tasks can be performed with these tools, without needing any XML knowledge. To enable STONITH for example, one could run: .. code-block:: none # crm_attribute --name stonith-enabled --update 1 Or, to check whether **node1** is allowed to run resources, there is: .. code-block:: none # crm_standby --query --node node1 Or, to change the failure threshold of **my-test-rsc**, one can use: .. code-block:: none # crm_resource -r my-test-rsc --set-parameter migration-threshold --parameter-value 3 --meta Examples of using these tools for specific cases will be given throughout this document where appropriate. See the man pages for further details. See :ref:`cibadmin` for how to edit the CIB using XML. See :ref:`crm_shadow` for a way to make a series of changes, then commit them all at once to the live cluster. Working with CIB Properties ########################### Although these fields can be written to by the user, in most cases the cluster will overwrite any values specified by the user with the "correct" ones. To change the ones that can be specified by the user, for example ``admin_epoch``, one should use: .. code-block:: none # cibadmin --modify --xml-text '' A complete set of CIB properties will look something like this: .. topic:: XML attributes set for a cib element .. code-block:: xml Querying and Setting Cluster Options #################################### .. index:: pair: cluster option; querying pair: cluster option; setting Cluster options can be queried and modified using the ``crm_attribute`` tool. To get the current value of ``cluster-delay``, you can run: .. code-block:: none # crm_attribute --query --name cluster-delay which is more simply written as .. code-block:: none # crm_attribute -G -n cluster-delay If a value is found, you'll see a result like this: .. code-block:: none # crm_attribute -G -n cluster-delay scope=crm_config name=cluster-delay value=60s If no value is found, the tool will display an error: .. code-block:: none # crm_attribute -G -n clusta-deway scope=crm_config name=clusta-deway value=(null) Error performing operation: No such device or address To use a different value (for example, 30 seconds), simply run: .. code-block:: none # crm_attribute --name cluster-delay --update 30s To go back to the cluster's default value, you can delete the value, for example: .. code-block:: none # crm_attribute --name cluster-delay --delete Deleted crm_config option: id=cib-bootstrap-options-cluster-delay name=cluster-delay When Options are Listed More Than Once ______________________________________ If you ever see something like the following, it means that the option you're modifying is present more than once. .. topic:: Deleting an option that is listed twice .. code-block:: none # crm_attribute --name batch-limit --delete Multiple attributes match name=batch-limit in crm_config: Value: 50 (set=cib-bootstrap-options, id=cib-bootstrap-options-batch-limit) Value: 100 (set=custom, id=custom-batch-limit) Please choose from one of the matches above and supply the 'id' with --id In such cases, follow the on-screen instructions to perform the requested action. To determine which value is currently being used by the cluster, refer to the "Rules" chapter of *Pacemaker Explained*. .. _remote_connection: Connecting from a Remote Machine ################################ .. index:: pair: cluster; remote connection pair: cluster; remote administration Provided Pacemaker is installed on a machine, it is possible to connect to the cluster even if the machine itself is not in the same cluster. To do this, one simply sets up a number of environment variables and runs the same commands as when working on a cluster node. -.. table:: Environment Variables Used to Connect to Remote Instances of the CIB - -+----------------------+-----------+----------------------------------------------+ -| Environment Variable | Default | Description | -+======================+===========+==============================================+ -| CIB_user | $USER | The user to connect as. Needs to be | -| | | part of the ``haclient`` group on | -| | | the target host. | -| | | | -| | | .. index:: | -| | | pair: environment variable; CIB_user | -+----------------------+-----------+----------------------------------------------+ -| CIB_passwd | | The user's password. Read from the | -| | | command line if unset. | -| | | | -| | | .. index:: | -| | | pair: environment variable; CIB_passwd | -+----------------------+-----------+----------------------------------------------+ -| CIB_server | localhost | The host to contact | -| | | | -| | | .. index:: | -| | | pair: environment variable; CIB_server | -+----------------------+-----------+----------------------------------------------+ -| CIB_port | | The port on which to contact the server; | -| | | required. | -| | | | -| | | .. index:: | -| | | pair: environment variable; CIB_port | -+----------------------+-----------+----------------------------------------------+ -| CIB_encrypted | TRUE | Whether to encrypt network traffic | -| | | | -| | | .. index:: | -| | | pair: environment variable; CIB_encrypted | -+----------------------+-----------+----------------------------------------------+ +.. table:: **Environment Variables Used to Connect to Remote Instances of the CIB** + + +----------------------+-----------+----------------------------------------------+ + | Environment Variable | Default | Description | + +======================+===========+==============================================+ + | CIB_user | $USER | The user to connect as. Needs to be | + | | | part of the ``haclient`` group on | + | | | the target host. | + | | | | + | | | .. index:: | + | | | pair: environment variable; CIB_user | + +----------------------+-----------+----------------------------------------------+ + | CIB_passwd | | The user's password. Read from the | + | | | command line if unset. | + | | | | + | | | .. index:: | + | | | pair: environment variable; CIB_passwd | + +----------------------+-----------+----------------------------------------------+ + | CIB_server | localhost | The host to contact | + | | | | + | | | .. index:: | + | | | pair: environment variable; CIB_server | + +----------------------+-----------+----------------------------------------------+ + | CIB_port | | The port on which to contact the server; | + | | | required. | + | | | | + | | | .. index:: | + | | | pair: environment variable; CIB_port | + +----------------------+-----------+----------------------------------------------+ + | CIB_encrypted | TRUE | Whether to encrypt network traffic | + | | | | + | | | .. index:: | + | | | pair: environment variable; CIB_encrypted | + +----------------------+-----------+----------------------------------------------+ So, if **c001n01** is an active cluster node and is listening on port 1234 for connections, and **someuser** is a member of the **haclient** group, then the following would prompt for **someuser**'s password and return the cluster's current configuration: .. code-block:: none # export CIB_port=1234; export CIB_server=c001n01; export CIB_user=someuser; # cibadmin -Q For security reasons, the cluster does not listen for remote connections by default. If you wish to allow remote access, you need to set the ``remote-tls-port`` (encrypted) or ``remote-clear-port`` (unencrypted) CIB properties (i.e., those kept in the ``cib`` tag, like ``num_updates`` and ``epoch``). -.. table:: Extra top-level CIB properties for remote access - -+----------------------+-----------+------------------------------------------------------+ -| CIB Property | Default | Description | -+======================+===========+======================================================+ -| remote-tls-port | | Listen for encrypted remote connections | -| | | on this port. | -| | | | -| | | .. index:: | -| | | pair: remote connection option; remote-tls-port | -+----------------------+-----------+------------------------------------------------------+ -| remote-clear-port | | Listen for plaintext remote connections | -| | | on this port. | -| | | | -| | | .. index:: | -| | | pair: remote connection option; remote-clear-port | -+----------------------+-----------+------------------------------------------------------+ +.. table:: **Extra top-level CIB properties for remote access** + + +----------------------+-----------+------------------------------------------------------+ + | CIB Property | Default | Description | + +======================+===========+======================================================+ + | remote-tls-port | | Listen for encrypted remote connections | + | | | on this port. | + | | | | + | | | .. index:: | + | | | pair: remote connection option; remote-tls-port | + +----------------------+-----------+------------------------------------------------------+ + | remote-clear-port | | Listen for plaintext remote connections | + | | | on this port. | + | | | | + | | | .. index:: | + | | | pair: remote connection option; remote-clear-port | + +----------------------+-----------+------------------------------------------------------+ .. important:: The Pacemaker version on the administration host must be the same or greater than the version(s) on the cluster nodes. Otherwise, it may not have the schema files necessary to validate the CIB. .. rubric:: Footnotes .. [#] For a list, see "Configuration Tools" at https://clusterlabs.org/components.html diff --git a/doc/sphinx/Pacemaker_Administration/tools.rst b/doc/sphinx/Pacemaker_Administration/tools.rst index 603bfbc101..aff767d9e9 100644 --- a/doc/sphinx/Pacemaker_Administration/tools.rst +++ b/doc/sphinx/Pacemaker_Administration/tools.rst @@ -1,553 +1,553 @@ Using Pacemaker Command-Line Tools ---------------------------------- .. _cmdline_output: Controlling Command Line Output ############################### Some of the pacemaker command line utilities have been converted to a new output system. Among these tools are ``crm_mon`` and ``stonith_admin``. This is an ongoing project, and more tools will be converted over time. This system lets you control the formatting of output with ``--output-as=`` and the destination of output with ``--output-to=``. The available formats vary by tool, but at least plain text and XML are supported by all tools that use the new system. The default format is plain text. The default destination is stdout but can be redirected to any file. Some formats support command line options for changing the style of the output. For instance: .. code-block:: none # crm_mon --help-output Usage: crm_mon [OPTION?] Provides a summary of cluster's current state. Outputs varying levels of detail in a number of different formats. Output Options: --output-as=FORMAT Specify output format as one of: console (default), html, text, xml --output-to=DEST Specify file name for output (or "-" for stdout) --html-cgi Add text needed to use output in a CGI program --html-stylesheet=URI Link to an external CSS stylesheet --html-title=TITLE Page title --text-fancy Use more highly formatted output .. _crm_mon: Monitor a Cluster with crm_mon ############################## .. index:: pair: command-line tool; crm_mon The ``crm_mon`` utility displays the current state of an active cluster. It can show the cluster status organized by node or by resource, and can be used in either single-shot or dynamically updating mode. It can also display operations performed and information about failures. Using this tool, you can examine the state of the cluster for irregularities, and see how it responds when you cause or simulate failures. See the manual page or the output of ``crm_mon --help`` for a full description of its many options. .. topic:: Sample output from crm_mon -1 .. code-block:: none Cluster Summary: * Stack: corosync * Current DC: node2 (version 2.0.0-1) - partition with quorum * Last updated: Mon Jan 29 12:18:42 2018 * Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 * 5 nodes configured * 2 resources configured Node List: * Online: [ node1 node2 node3 node4 node5 ] * Active resources: * Fencing (stonith:fence_xvm): Started node1 * IP (ocf:heartbeat:IPaddr2): Started node2 .. topic:: Sample output from crm_mon -n -1 .. code-block:: none Cluster Summary: * Stack: corosync * Current DC: node2 (version 2.0.0-1) - partition with quorum * Last updated: Mon Jan 29 12:21:48 2018 * Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 * 5 nodes configured * 2 resources configured * Node List: * Node node1: online * Fencing (stonith:fence_xvm): Started * Node node2: online * IP (ocf:heartbeat:IPaddr2): Started * Node node3: online * Node node4: online * Node node5: online As mentioned in an earlier chapter, the DC is the node is where decisions are made. The cluster elects a node to be DC as needed. The only significance of the choice of DC to an administrator is the fact that its logs will have the most information about why decisions were made. .. _crm_mon_css: Styling crm_mon output ______________________ .. index:: pair: crm_mon; CSS Various parts of ``crm_mon``'s HTML output have a CSS class associated with them. Not everything does, but some of the most interesting portions do. In the following example, the status of each node has an ``online`` class and the details of each resource have an ``rsc-ok`` class. .. code-block:: html

Node List

  • Node: cluster01 online
    • ping (ocf::pacemaker:ping): Started
  • Node: cluster02 online
    • ping (ocf::pacemaker:ping): Started
By default, a stylesheet for styling these classes is included in the head of the HTML output. The relevant portions of this stylesheet that would be used in the above example is: .. code-block:: css If you want to override some or all of the styling, simply create your own stylesheet, place it on a web server, and pass ``--html-stylesheet=`` to ``crm_mon``. The link is added after the default stylesheet, so your changes take precedence. You don't need to duplicate the entire default. Only include what you want to change. .. _cibadmin: Edit the CIB XML with cibadmin ############################## .. index:: pair: command-line tool; cibadmin The most flexible tool for modifying the configuration is Pacemaker's ``cibadmin`` command. With ``cibadmin``, you can query, add, remove, update or replace any part of the configuration. All changes take effect immediately, so there is no need to perform a reload-like operation. The simplest way of using ``cibadmin`` is to use it to save the current configuration to a temporary file, edit that file with your favorite text or XML editor, and then upload the revised configuration. .. topic:: Safely using an editor to modify the cluster configuration .. code-block:: none # cibadmin --query > tmp.xml # vi tmp.xml # cibadmin --replace --xml-file tmp.xml Some of the better XML editors can make use of a RELAX NG schema to help make sure any changes you make are valid. The schema describing the configuration can be found in ``pacemaker.rng``, which may be deployed in a location such as ``/usr/share/pacemaker`` depending on your operating system distribution and how you installed the software. If you want to modify just one section of the configuration, you can query and replace just that section to avoid modifying any others. .. topic:: Safely using an editor to modify only the resources section .. code-block:: none # cibadmin --query --scope resources > tmp.xml # vi tmp.xml # cibadmin --replace --scope resources --xml-file tmp.xml To quickly delete a part of the configuration, identify the object you wish to delete by XML tag and id. For example, you might search the CIB for all STONITH-related configuration: .. topic:: Searching for STONITH-related configuration items .. code-block:: none # cibadmin --query | grep stonith If you wanted to delete the ``primitive`` tag with id ``child_DoFencing``, you would run: -.. code-block:: +.. code-block:: none # cibadmin --delete --xml-text '' See the cibadmin man page for more options. .. warning:: Never edit the live ``cib.xml`` file directly. Pacemaker will detect such changes and refuse to use the configuration. .. _crm_shadow: Batch Configuration Changes with crm_shadow ########################################### .. index:: pair: command-line tool; crm_shadow Often, it is desirable to preview the effects of a series of configuration changes before updating the live configuration all at once. For this purpose, ``crm_shadow`` creates a "shadow" copy of the configuration and arranges for all the command-line tools to use it. To begin, simply invoke ``crm_shadow --create`` with a name of your choice, and follow the simple on-screen instructions. Shadow copies are identified with a name to make it possible to have more than one. .. warning:: Read this section and the on-screen instructions carefully; failure to do so could result in destroying the cluster's active configuration! .. topic:: Creating and displaying the active sandbox .. code-block:: none # crm_shadow --create test Setting up shadow instance Type Ctrl-D to exit the crm_shadow shell shadow[test]: shadow[test] # crm_shadow --which test From this point on, all cluster commands will automatically use the shadow copy instead of talking to the cluster's active configuration. Once you have finished experimenting, you can either make the changes active via the ``--commit`` option, or discard them using the ``--delete`` option. Again, be sure to follow the on-screen instructions carefully! For a full list of ``crm_shadow`` options and commands, invoke it with the ``--help`` option. .. topic:: Use sandbox to make multiple changes all at once, discard them, and verify real configuration is untouched .. code-block:: none shadow[test] # crm_failcount -r rsc_c001n01 -G scope=status name=fail-count-rsc_c001n01 value=0 shadow[test] # crm_standby --node c001n02 -v on shadow[test] # crm_standby --node c001n02 -G scope=nodes name=standby value=on shadow[test] # cibadmin --erase --force shadow[test] # cibadmin --query shadow[test] # crm_shadow --delete test --force Now type Ctrl-D to exit the crm_shadow shell shadow[test] # exit # crm_shadow --which No active shadow configuration defined # cibadmin -Q See the next section, :ref:`crm_simulate`, for how to test your changes before committing them to the live cluster. .. _crm_simulate: Simulate Cluster Activity with crm_simulate ########################################### .. index:: pair: command-line tool; crm_simulate The command-line tool `crm_simulate` shows the results of the same logic the cluster itself uses to respond to a particular cluster configuration and status. As always, the man page is the primary documentation, and should be consulted for further details. This section aims for a better conceptual explanation and practical examples. Replaying cluster decision-making logic _______________________________________ At any given time, one node in a Pacemaker cluster will be elected DC, and that node will run Pacemaker's scheduler to make decisions. Each time decisions need to be made (a "transition"), the DC will have log messages like "Calculated transition ... saving inputs in ..." with a file name. You can grab the named file and replay the cluster logic to see why particular decisions were made. The file contains the live cluster configuration at that moment, so you can also look at it directly to see the value of node attributes, etc., at that time. The simplest usage is (replacing $FILENAME with the actual file name): .. topic:: Simulate cluster response to a given CIB .. code-block:: none # crm_simulate --simulate --xml-file $FILENAME That will show the cluster state when the process started, the actions that need to be taken ("Transition Summary"), and the resulting cluster state if the actions succeed. Most actions will have a brief description of why they were required. The transition inputs may be compressed. ``crm_simulate`` can handle these compressed files directly, though if you want to edit the file, you'll need to uncompress it first. You can do the same simulation for the live cluster configuration at the current moment. This is useful mainly when using ``crm_shadow`` to create a sandbox version of the CIB; the ``--live-check`` option will use the shadow CIB if one is in effect. .. topic:: Simulate cluster response to current live CIB or shadow CIB .. code-block:: none # crm_simulate --simulate --live-check Why decisions were made _______________________ To get further insight into the "why", it gets user-unfriendly very quickly. If you add the ``--show-scores`` option, you will also see all the scores that went into the decision-making. The node with the highest cumulative score for a resource will run it. You can look for ``-INFINITY`` scores in particular to see where complete bans came into effect. You can also add ``-VVVV`` to get more detailed messages about what's happening under the hood. You can add up to two more V's even, but that's usually useful only if you're a masochist or tracing through the source code. Visualizing the action sequence _______________________________ Another handy feature is the ability to generate a visual graph of the actions needed, using the ``--dot-file`` option. This relies on the separate Graphviz [#]_ project. .. topic:: Generate a visual graph of cluster actions from a saved CIB .. code-block:: none # crm_simulate --simulate --xml-file $FILENAME --dot-file $FILENAME.dot # dot $FILENAME.dot -Tsvg > $FILENAME.svg ``$FILENAME.dot`` will contain a GraphViz representation of the cluster's response to your changes, including all actions with their ordering dependencies. ``$FILENAME.svg`` will be the same information in a standard graphical format that you can view in your browser or other app of choice. You could, of course, use other ``dot`` options to generate other formats. How to interpret the graphical output: * Bubbles indicate actions, and arrows indicate ordering dependencies * Resource actions have text of the form ``__ `` indicating that the specified action will be executed for the specified resource on the specified node, once if interval is 0 or at specified recurring interval otherwise * Actions with black text will be sent to the executor (that is, the appropriate agent will be invoked) * Actions with orange text are "pseudo" actions that the cluster uses internally for ordering but require no real activity * Actions with a solid green border are part of the transition (that is, the cluster will attempt to execute them in the given order -- though a transition can be interrupted by action failure or new events) * Dashed arrows indicate dependencies that are not present in the transition graph * Actions with a dashed border will not be executed. If the dashed border is blue, the cluster does not feel the action needs to be executed. If the dashed border is red, the cluster would like to execute the action but cannot. Any actions depending on an action with a dashed border will not be able to execute. * Loops should not happen, and should be reported as a bug if found. .. topic:: Small Cluster Transition .. image:: ../../shared/en-US/images/Policy-Engine-small.png :alt: An example transition graph as represented by Graphviz :height: 325 :width: 1161 :scale: 75 % :align: center In the above example, it appears that a new node, ``pcmk-2``, has come online and that the cluster is checking to make sure ``rsc1``, ``rsc2`` and ``rsc3`` are not already running there (indicated by the ``rscN_monitor_0`` entries). Once it did that, and assuming the resources were not active there, it would have liked to stop ``rsc1`` and ``rsc2`` on ``pcmk-1`` and move them to ``pcmk-2``. However, there appears to be some problem and the cluster cannot or is not permitted to perform the stop actions which implies it also cannot perform the start actions. For some reason, the cluster does not want to start ``rsc3`` anywhere. .. topic:: Complex Cluster Transition .. image:: ../../shared/en-US/images/Policy-Engine-big.png :alt: Complex transition graph that you're not expected to be able to read :width: 1455 :height: 1945 :scale: 75 % :align: center What-if scenarios _________________ You can make changes to the saved or shadow CIB and simulate it again, to see how Pacemaker would react differently. You can edit the XML by hand, use command-line tools such as ``cibadmin`` with either a shadow CIB or the ``CIB_file`` environment variable set to the filename, or use higher-level tool support (see the man pages of the specific tool you're using for how to perform actions on a saved CIB file rather than the live CIB). You can also inject node failures and/or action failures into the simulation; see the ``crm_simulate`` man page for more details. This capability is useful when using a shadow CIB to edit the configuration. Before committing the changes to the live cluster with ``crm_shadow --commit``, you can use ``crm_simulate`` to see how the cluster will react to the changes. .. _attrd_updater: .. _crm_attribute: Manage Node Attributes, Cluster Options and Defaults with crm_attribute and attrd_updater ######################################################################################### .. index:: pair: command-line tool; attrd_updater pair: command-line tool; crm_attribute ``crm_attribute`` and ``attrd_updater`` are confusingly similar tools with subtle differences. ``attrd_updater`` can query and update node attributes. ``crm_attribute`` can query and update not only node attributes, but also cluster options, resource defaults, and operation defaults. To understand the differences, it helps to understand the various types of node attribute. -.. table:: Types of Node Attributes - -+-----------+----------+-------------------+------------------+----------------+----------------+ -| Type | Recorded | Recorded in | Survive full | Manageable by | Manageable by | -| | in CIB? | attribute manager | cluster restart? | crm_attribute? | attrd_updater? | -| | | memory? | | | | -+===========+==========+===================+==================+================+================+ -| permanent | yes | no | yes | yes | no | -+-----------+----------+-------------------+------------------+----------------+----------------+ -| transient | yes | yes | no | yes | yes | -+-----------+----------+-------------------+------------------+----------------+----------------+ -| private | no | yes | no | no | yes | -+-----------+----------+-------------------+------------------+----------------+----------------+ +.. table:: **Types of Node Attributes** + + +-----------+----------+-------------------+------------------+----------------+----------------+ + | Type | Recorded | Recorded in | Survive full | Manageable by | Manageable by | + | | in CIB? | attribute manager | cluster restart? | crm_attribute? | attrd_updater? | + | | | memory? | | | | + +===========+==========+===================+==================+================+================+ + | permanent | yes | no | yes | yes | no | + +-----------+----------+-------------------+------------------+----------------+----------------+ + | transient | yes | yes | no | yes | yes | + +-----------+----------+-------------------+------------------+----------------+----------------+ + | private | no | yes | no | no | yes | + +-----------+----------+-------------------+------------------+----------------+----------------+ As you can see from the table above, ``crm_attribute`` can manage permanent and transient node attributes, while ``attrd_updater`` can manage transient and private node attributes. The difference between the two tools lies mainly in *how* they update node attributes: ``attrd_updater`` always contacts the Pacemaker attribute manager directly, while ``crm_attribute`` will contact the attribute manager only for transient node attributes, and will instead modify the CIB directly for permanent node attributes (and for transient node attributes when unable to contact the attribute manager). By contacting the attribute manager directly, ``attrd_updater`` can change an attribute's "dampening" (whether changes are immediately flushed to the CIB or after a specified amount of time, to minimize disk writes for frequent changes), set private node attributes (which are never written to the CIB), and set attributes for nodes that don't yet exist. By modifying the CIB directly, ``crm_attribute`` can set permanent node attributes (which are only in the CIB and not managed by the attribute manager), and can be used with saved CIB files and shadow CIBs. However a transient node attribute is set, it is synchronized between the CIB and the attribute manager, on all nodes. Other Commonly Used Tools ######################### Other command-line tools include: .. index:: pair: command-line tool; crm_failcount pair: command-line tool; crm_node pair: command-line tool; crm_report pair: command-line tool; crm_standby pair: command-line tool; crm_verify pair: command-line tool; stonith_admin * ``crm_failcount``: query or delete resource fail counts * ``crm_node``: manage cluster nodes * ``crm_report``: generate a detailed cluster report for bug submissions * ``crm_resource``: manage cluster resources * ``crm_standby``: manage standby status of nodes * ``crm_verify``: validate a CIB * ``stonith_admin``: manage fencing devices See the manual pages for details. .. rubric:: Footnotes .. [#] Graph visualization software. See http://www.graphviz.org/ for details. diff --git a/doc/sphinx/Pacemaker_Administration/upgrading.rst b/doc/sphinx/Pacemaker_Administration/upgrading.rst index b2f00cc914..1d6f0f68ad 100644 --- a/doc/sphinx/Pacemaker_Administration/upgrading.rst +++ b/doc/sphinx/Pacemaker_Administration/upgrading.rst @@ -1,485 +1,485 @@ Upgrading a Pacemaker Cluster ----------------------------- Pacemaker Versioning #################### Pacemaker has an overall release version, plus separate version numbers for certain internal components. * **Pacemaker release version:** This version consists of three numbers (*x.y.z*). The major version number (the *x* in *x.y.z*) increases when at least some rolling upgrades are not possible from the previous major version. For example, a rolling upgrade from 1.0.8 to 1.1.15 should always be supported, but a rolling upgrade from 1.0.8 to 2.0.0 may not be possible. The minor version (the *y* in *x.y.z*) increases when there are significant changes in cluster default behavior, tool behavior, and/or the API interface (for software that utilizes Pacemaker libraries). The main benefit is to alert you to pay closer attention to the release notes, to see if you might be affected. The release counter (the *z* in *x.y.z*) is increased with all public releases of Pacemaker, which typically include both bug fixes and new features. * **CRM feature set:** This version number applies to the communication between full cluster nodes, and is used to avoid problems in mixed-version clusters. The major version number increases when nodes with different versions would not work (rolling upgrades are not allowed). The minor version number increases when mixed-version clusters are allowed only during rolling upgrades. The minor-minor version number is ignored, but allows resource agents to detect cluster support for various features. [#]_ Pacemaker ensures that the longest-running node is the cluster's DC. This ensures new features are not enabled until all nodes are upgraded to support them. * **Pacemaker Remote protocol version:** This version applies to communication between a Pacemaker Remote node and the cluster. It increases when an older cluster node would have problems hosting the connection to a newer Pacemaker Remote node. To avoid these problems, Pacemaker Remote nodes will accept connections only from cluster nodes with the same or newer Pacemaker Remote protocol version. Unlike with CRM feature set differences between full cluster nodes, mixed Pacemaker Remote protocol versions between Pacemaker Remote nodes and full cluster nodes are fine, as long as the Pacemaker Remote nodes have the older version. This can be useful, for example, to host a legacy application in an older operating system version used as a Pacemaker Remote node. * **XML schema version:** Pacemaker’s configuration syntax — what's allowed in the Configuration Information Base (CIB) — has its own version. This allows the configuration syntax to evolve over time while still allowing clusters with older configurations to work without change. Upgrading Cluster Software ########################## There are three approaches to upgrading a cluster, each with advantages and disadvantages. -.. table:: Upgrade Methods - -+---------------------------------------------------+----------+----------+--------+---------+----------+----------+ -| Method | Available| Can be | Service| Service | Exercises| Allows | -| | between | used with| outage | recovery| failover | change of| -| | all | Pacemaker| during | during | logic | messaging| -| | versions | Remote | upgrade| upgrade | | layer | -| | | nodes | | | | [#]_ | -+===================================================+==========+==========+========+=========+==========+==========+ -| Complete cluster shutdown | yes | yes | always | N/A | no | yes | -| | | | | | | | -| .. index:: | | | | | | | -| pair: cluster; upgrade with shutdown | | | | | | | -| pair: upgrade; upgrade with shutdown | | | | | | | -+---------------------------------------------------+----------+----------+--------+---------+----------+----------+ -| Rolling (node by node) | no | yes | always | yes | yes | no | -| | | | [#]_ | | | | -| | | | | | | | -| .. index:: | | | | | | | -| pair: cluster; rolling upgrade | | | | | | | -| pair: upgrade; rolling upgrade | | | | | | | -+---------------------------------------------------+----------+----------+--------+---------+----------+----------+ -| Detach and reattach | yes | no | only | no | no | yes | -| | | | due to | | | | -| | | | failure| | | | -| | | | | | | | -| .. index:: | | | | | | | -| pair: cluster; upgrade with detach and reattach| | | | | | | -| pair: upgrade; upgrade with detach and reattach| | | | | | | -+---------------------------------------------------+----------+----------+--------+---------+----------+----------+ +.. table:: **Upgrade Methods** + + +---------------------------------------------------+----------+----------+--------+---------+----------+----------+ + | Method | Available| Can be | Service| Service | Exercises| Allows | + | | between | used with| outage | recovery| failover | change of| + | | all | Pacemaker| during | during | logic | messaging| + | | versions | Remote | upgrade| upgrade | | layer | + | | | nodes | | | | [#]_ | + +===================================================+==========+==========+========+=========+==========+==========+ + | Complete cluster shutdown | yes | yes | always | N/A | no | yes | + | | | | | | | | + | .. index:: | | | | | | | + | pair: cluster; upgrade with shutdown | | | | | | | + | pair: upgrade; upgrade with shutdown | | | | | | | + +---------------------------------------------------+----------+----------+--------+---------+----------+----------+ + | Rolling (node by node) | no | yes | always | yes | yes | no | + | | | | [#]_ | | | | + | | | | | | | | + | .. index:: | | | | | | | + | pair: cluster; rolling upgrade | | | | | | | + | pair: upgrade; rolling upgrade | | | | | | | + +---------------------------------------------------+----------+----------+--------+---------+----------+----------+ + | Detach and reattach | yes | no | only | no | no | yes | + | | | | due to | | | | + | | | | failure| | | | + | | | | | | | | + | .. index:: | | | | | | | + | pair: cluster; upgrade with detach and reattach| | | | | | | + | pair: upgrade; upgrade with detach and reattach| | | | | | | + +---------------------------------------------------+----------+----------+--------+---------+----------+----------+ Complete Cluster Shutdown _________________________ In this scenario, one shuts down all cluster nodes and resources, then upgrades all the nodes before restarting the cluster. #. On each node: a. Shutdown the cluster software (pacemaker and the messaging layer). #. Upgrade the Pacemaker software. This may also include upgrading the messaging layer and/or the underlying operating system. #. Check the configuration with the ``crm_verify`` tool. #. On each node: a. Start the cluster software. Currently, only Corosync version 2 and greater is supported as the cluster layer, but if another stack is supported in the future, the stack does not need to be the same one before the upgrade. One variation of this approach is to build a new cluster on new hosts. This allows the new version to be tested beforehand, and minimizes downtime by having the new nodes ready to be placed in production as soon as the old nodes are shut down. Rolling (node by node) ______________________ In this scenario, each node is removed from the cluster, upgraded, and then brought back online, until all nodes are running the newest version. Special considerations when planning a rolling upgrade: * If you plan to upgrade other cluster software -- such as the messaging layer -- at the same time, consult that software's documentation for its compatibility with a rolling upgrade. * If the major version number is changing in the Pacemaker version you are upgrading to, a rolling upgrade may not be possible. Read the new version's release notes (as well the information here) for what limitations may exist. * If the CRM feature set is changing in the Pacemaker version you are upgrading to, you should run a mixed-version cluster only during a small rolling upgrade window. If one of the older nodes drops out of the cluster for any reason, it will not be able to rejoin until it is upgraded. * If the Pacemaker Remote protocol version is changing, all cluster nodes should be upgraded before upgrading any Pacemaker Remote nodes. See the ClusterLabs wiki's `release calendar `_ to figure out whether the CRM feature set and/or Pacemaker Remote protocol version changed between the the Pacemaker release versions in your rolling upgrade. To perform a rolling upgrade, on each node in turn: #. Put the node into standby mode, and wait for any active resources to be moved cleanly to another node. (This step is optional, but allows you to deal with any resource issues before the upgrade.) #. Shutdown the cluster software (pacemaker and the messaging layer) on the node. #. Upgrade the Pacemaker software. This may also include upgrading the messaging layer and/or the underlying operating system. #. If this is the first node to be upgraded, check the configuration with the ``crm_verify`` tool. #. Start the messaging layer. This must be the same messaging layer (currently only Corosync version 2 and greater is supported) that the rest of the cluster is using. .. note:: Even if a rolling upgrade from the current version of the cluster to the newest version is not directly possible, it may be possible to perform a rolling upgrade in multiple steps, by upgrading to an intermediate version first. -.. table:: Version Compatibility Table - -+-------------------------+---------------------------+ -| Version being Installed | Oldest Compatible Version | -+=========================+===========================+ -| Pacemaker 2.y.z | Pacemaker 1.1.11 [#]_ | -+-------------------------+---------------------------+ -| Pacemaker 1.y.z | Pacemaker 1.0.0 | -+-------------------------+---------------------------+ -| Pacemaker 0.7.z | Pacemaker 0.6.z | -+-------------------------+---------------------------+ +.. table:: **Version Compatibility Table** + + +-------------------------+---------------------------+ + | Version being Installed | Oldest Compatible Version | + +=========================+===========================+ + | Pacemaker 2.y.z | Pacemaker 1.1.11 [#]_ | + +-------------------------+---------------------------+ + | Pacemaker 1.y.z | Pacemaker 1.0.0 | + +-------------------------+---------------------------+ + | Pacemaker 0.7.z | Pacemaker 0.6.z | + +-------------------------+---------------------------+ Detach and Reattach ___________________ The reattach method is a variant of a complete cluster shutdown, where the resources are left active and get re-detected when the cluster is restarted. This method may not be used if the cluster contains any Pacemaker Remote nodes. #. Tell the cluster to stop managing services. This is required to allow the services to remain active after the cluster shuts down. .. code-block:: none # crm_attribute --name maintenance-mode --update true #. On each node, shutdown the cluster software (pacemaker and the messaging layer), and upgrade the Pacemaker software. This may also include upgrading the messaging layer. While the underlying operating system may be upgraded at the same time, that will be more likely to cause outages in the detached services (certainly, if a reboot is required). #. Check the configuration with the ``crm_verify`` tool. #. On each node, start the cluster software. Currently, only Corosync version 2 and greater is supported as the cluster layer, but if another stack is supported in the future, the stack does not need to be the same one before the upgrade. #. Verify that the cluster re-detected all resources correctly. #. Allow the cluster to resume managing resources again: .. code-block:: none # crm_attribute --name maintenance-mode --delete .. note:: While the goal of the detach-and-reattach method is to avoid disturbing running services, resources may still move after the upgrade if any resource's location is governed by a rule based on transient node attributes. Transient node attributes are erased when the node leaves the cluster. A common example is using the ``ocf:pacemaker:ping`` resource to set a node attribute used to locate other resources. Upgrading the Configuration ########################### .. index:: pair: upgrading; configuration The CIB schema version can change from one Pacemaker version to another. After cluster software is upgraded, the cluster will continue to use the older schema version that it was previously using. This can be useful, for example, when administrators have written tools that modify the configuration, and are based on the older syntax. [#]_ However, when using an older syntax, new features may be unavailable, and there is a performance impact, since the cluster must do a non-persistent configuration upgrade before each transition. So while using the old syntax is possible, it is not advisable to continue using it indefinitely. Even if you wish to continue using the old syntax, it is a good idea to follow the upgrade procedure outlined below, except for the last step, to ensure that the new software has no problems with your existing configuration (since it will perform much the same task internally). If you are brave, it is sufficient simply to run ``cibadmin --upgrade``. A more cautious approach would proceed like this: #. Create a shadow copy of the configuration. The later commands will automatically operate on this copy, rather than the live configuration. .. code-block:: none # crm_shadow --create shadow #. Verify the configuration is valid with the new software (which may be stricter about syntax mistakes, or may have dropped support for deprecated features): .. index:: pair: verify; configuration .. code-block:: none # crm_verify --live-check #. Fix any errors or warnings. #. Perform the upgrade: .. code-block:: none # cibadmin --upgrade #. If this step fails, there are three main possibilities: a. The configuration was not valid to start with (did you do steps 2 and 3?). #. The transformation failed; `report a bug `_. #. The transformation was successful but produced an invalid result. If the result of the transformation is invalid, you may see a number of errors from the validation library. If these are not helpful, visit the `Validation FAQ wiki page `_ and/or try the manual upgrade procedure described below. #. Check the changes: .. code-block:: none # crm_shadow --diff If at this point there is anything about the upgrade that you wish to fine-tune (for example, to change some of the automatic IDs), now is the time to do so: .. code-block:: none # crm_shadow --edit This will open the configuration in your favorite editor (whichever is specified by the standard ``$EDITOR`` environment variable). #. Preview how the cluster will react: .. code-block:: none # crm_simulate --live-check --save-dotfile shadow.dot -S # dot -Tsvg shadow.dot -o shadow.svg You can then view shadow.svg with any compatible image viewer or web browser. Verify that either no resource actions will occur or that you are happy with any that are scheduled. If the output contains actions you do not expect (possibly due to changes to the score calculations), you may need to make further manual changes. See :ref:`crm_simulate` for further details on how to interpret the output of ``crm_simulate`` and ``dot``. #. Upload the changes: .. code-block:: none # crm_shadow --commit shadow --force In the unlikely event this step fails, please report a bug. .. note:: It is also possible to perform the configuration upgrade steps manually: #. Locate the ``upgrade*.xsl`` conversion scripts provided with the source code. These will often be installed in a location such as ``/usr/share/pacemaker``, or may be obtained from the `source repository `_. #. Run the conversion scripts that apply to your older version, for example: .. code-block:: none # xsltproc /path/to/upgrade06.xsl config06.xml > config10.xml #. Locate the ``pacemaker.rng`` script (from the same location as the xsl files). #. Check the XML validity: .. code-block:: none # xmllint --relaxng /path/to/pacemaker.rng config10.xml The advantage of this method is that it can be performed without the cluster running, and any validation errors are often more informative. What Changed in 2.0 ################### The main goal of the 2.0 release was to remove support for deprecated syntax, along with some small changes in default configuration behavior and tool behavior. Highlights: * Only Corosync version 2 and greater is now supported as the underlying cluster layer. Support for Heartbeat and Corosync 1 (including CMAN) is removed. * The Pacemaker detail log file is now stored in ``/var/log/pacemaker/pacemaker.log`` by default. * The record-pending cluster property now defaults to true, which allows status tools such as crm_mon to show operations that are in progress. * Support for a number of deprecated build options, environment variables, and configuration settings has been removed. * The ``master`` tag has been deprecated in favor of using the ``clone`` tag with the new ``promotable`` meta-attribute set to ``true``. "Master/slave" clone resources are now referred to as "promotable" clone resources, though it will take longer for the full terminology change to be completed. * The public API for Pacemaker libraries that software applications can use has changed significantly. For a detailed list of changes, see the release notes and the `Pacemaker 2.0 Changes `_ page on the ClusterLabs wiki. What Changed in 1.0 ################### New ___ * Failure timeouts. * New section for resource and operation defaults. * Tool for making offline configuration changes. * ``Rules``, ``instance_attributes``, ``meta_attributes`` and sets of operations can be defined once and referenced in multiple places. * The CIB now accepts XPath-based create/modify/delete operations. See ``cibadmin --help``. * Multi-dimensional colocation and ordering constraints. * The ability to connect to the CIB from non-cluster machines. * Allow recurring actions to be triggered at known times. Changed _______ * Syntax * All resource and cluster options now use dashes (-) instead of underscores (_) * ``master_slave`` was renamed to ``master`` * The ``attributes`` container tag was removed * The operation field ``pre-req`` has been renamed ``requires`` * All operations must have an ``interval``, ``start``/``stop`` must have it set to zero * The ``stonith-enabled`` option now defaults to true. * The cluster will refuse to start resources if ``stonith-enabled`` is true (or unset) and no STONITH resources have been defined * The attributes of colocation and ordering constraints were renamed for clarity. * ``resource-failure-stickiness`` has been replaced by ``migration-threshold``. * The parameters for command-line tools have been made consistent * Switched to 'RelaxNG' schema validation and 'libxml2' parser * id fields are now XML IDs which have the following limitations: * id's cannot contain colons (:) * id's cannot begin with a number * id's must be globally unique (not just unique for that tag) * Some fields (such as those in constraints that refer to resources) are IDREFs. This means that they must reference existing resources or objects in order for the configuration to be valid. Removing an object which is referenced elsewhere will therefore fail. * The CIB representation, from which a MD5 digest is calculated to verify CIBs on the nodes, has changed. This means that every CIB update will require a full refresh on any upgraded nodes until the cluster is fully upgraded to 1.0. This will result in significant performance degradation and it is therefore highly inadvisable to run a mixed 1.0/0.6 cluster for any longer than absolutely necessary. * Ping node information no longer needs to be added to ``ha.cf``. Simply include the lists of hosts in your ping resource(s). Removed _______ * Syntax * It is no longer possible to set resource meta options as top-level attributes. Use meta-attributes instead. * Resource and operation defaults are no longer read from ``crm_config``. .. rubric:: Footnotes .. [#] Before CRM feature set 3.1.0 (Pacemaker 2.0.0), the minor-minor version number was treated the same as the minor version. .. [#] Currently, Corosync version 2 and greater is the only supported cluster stack, but other stacks have been supported by past versions, and may be supported by future versions. .. [#] Any active resources will be moved off the node being upgraded, so there will be at least a brief outage unless all resources can be migrated "live". .. [#] Rolling upgrades from Pacemaker 1.1.z to 2.y.z are possible only if the cluster uses corosync version 2 or greater as its messaging layer, and the Cluster Information Base (CIB) uses schema 1.0 or higher in its ``validate-with`` property. .. [#] As of Pacemaker 2.0.0, only schema versions pacemaker-1.0 and higher are supported (excluding pacemaker-1.1, which was an experimental schema now known as pacemaker-next). diff --git a/doc/sphinx/Pacemaker_Explained/acls.rst b/doc/sphinx/Pacemaker_Explained/acls.rst new file mode 100644 index 0000000000..1786768a76 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/acls.rst @@ -0,0 +1,335 @@ +Access Control Lists (ACLs) +--------------------------- + +.. Convert_to_RST: + + anchor:ch-acls[Chapter 13, ACLs] + indexterm:[access control list] + indexterm:[ACL] + + By default, the +root+ user or any user in the +haclient+ group can modify + Pacemaker's CIB without restriction. Pacemaker offers 'access control lists + (ACLs)' to provide more fine-grained authorization. + + == ACL Prerequisites == + + In order to use ACLs: + + * The Pacemaker software must have been compiled with ACL support. If the + output of the command `pacemakerd --features` contains `acls`, your + installation supports ACLs. + + * Desired users must have user accounts in the +haclient+ group on all nodes in + the cluster. + + * If your CIB was created before Pacemaker 1.1.12, it may need to be updated to + the current schema using `cibadmin --upgrade` in order to use the syntax + documented here. + + * The +enable-acl+ <> must be set to true. + + == ACL Configuration == + + ACLs are specified within an +acls+ element of the CIB. The +acls+ element may + contain any number of +acl_role+, +acl_target+, and +acl_group+ elements. + + == ACL Roles == + + An ACL role is a collection of permissions allowing or denying access to + particular portions of the CIB. + + .Properties of an ACL Role + [width="95%",cols="1m,<3",options="header",align="center"] + |==== + + |Attribute + |Description + + |id + |A unique name for the role (required) + indexterm:[id,acl_role] + indexterm:[access control list,acl_role,id] + + |description + |Arbitrary text (not used by Pacemaker) + indexterm:[description,acl_role] + indexterm:[access control list,acl_role,description] + + |==== + + An +acl_role+ element may contain any number of +acl_permission+ elements. + + .Properties of an ACL Permission + [width="95%",cols="1m,<3",options="header",align="center"] + |==== + + |Attribute + |Description + + |id + |A unique name for the permission (required) + indexterm:[id,acl_permission] + indexterm:[access control list,acl_permission,id] + + |description + |Arbitrary text (not used by Pacemaker) + indexterm:[description,acl_permission] + indexterm:[access control list,acl_permission,description] + + |kind + |The access being granted. Allowed values are +read+, +write+, and +deny+. + A value of +write+ grants both read and write access. + indexterm:[kind,acl_permission] + indexterm:[access control list,acl_permission,kind] + + |object-type + |The name of an XML element in the CIB to which the permission applies. + (Exactly one of +object-type+, +xpath+, and +reference+ must be specified for + a permission.) + indexterm:[object-type,acl_permission] + indexterm:[access control list,acl_permission,object-type] + + |attribute + |If specified, the permission applies only to +object-type+ elements that have + this attribute set (to any value). If not specified, the permission applies to + all +object-type+ elements. May only be used with +object-type+. + indexterm:[attribute,acl_permission] + indexterm:[access control list,acl_permission,attribute] + + |reference + |The ID of an XML element in the CIB to which the permission applies. + (Exactly one of +object-type+, +xpath+, and +reference+ must be specified for + a permission.) + indexterm:[reference,acl_permission] + indexterm:[access control list,acl_permission,reference] + + |xpath + |An https://www.w3.org/TR/xpath-10/[XPath] specification selecting an XML + element in the CIB to which the permission applies. Attributes may be + specified in the XPath to select particular elements, but the permissions + apply to the entire element. + (Exactly one of +object-type+, +xpath+, and +reference+ must be specified for + a permission.) + indexterm:[xpath,acl_permission] + indexterm:[access control list,acl_permission,xpath] + + |==== + + [IMPORTANT] + ==== + * Permissions are applied to the selected XML element's entire XML subtree + (all elements enclosed within it). + + * Write permission grants the ability to create, modify, or remove the element + and its subtree, and also the ability to create any "scaffolding" elements + (enclosing elements that do not have attributes other than an ID). + + * Permissions for more specific matches (more deeply nested elements) take + precedence over more general ones. + + * If multiple permissions are configured for the same match (for example, in + different roles applied to the same user), any +deny+ permission takes + precedence, then +write+, then lastly +read+. + ==== + + == ACL Targets and Groups == + + ACL targets correspond to user accounts on the system. + + .Properties of an ACL Target + [width="95%",cols="1m,<3",options="header",align="center"] + |==== + + |Attribute + |Description + + |id + |The name of a user on the system (required) + indexterm:[id,acl_target] + indexterm:[access control list,acl_target,id] + + |==== + + ACL groups may be specified, but are not currently used by Pacemaker. This is + expected to change in a future version. + + .Properties of an ACL Group + [width="95%",cols="1m,<3",options="header",align="center"] + |==== + + |Attribute + |Description + + |id + |The name of a group on the system (required) + indexterm:[id,acl_group] + indexterm:[access control list,acl_group,id] + + |==== + + Each +acl_target+ and +acl_group+ element may contain any number of +role+ + elements. + + .Properties of an ACL Role Reference + [width="95%",cols="1m,<3",options="header",align="center"] + |==== + + |Attribute + |Description + + |id + |The +id+ of an +acl_role+ element that specifies permissions granted to the + enclosing target or group + indexterm:[id,role] + indexterm:[access control list,role,id] + + |==== + + [IMPORTANT] + ==== + The +root+ and +hacluster+ user accounts always have full access to the CIB, + regardless of ACLs. For other user accounts, when +enable-acl+ is true, + permission to all parts of the CIB is denied by default (permissions must be + explicitly granted). + ==== + + == ACL Examples == + + [source,XML] + ---- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ---- + + In the above example, the user +alice+ has the minimal permissions necessary to + run basic Pacemaker CLI tools, including using `crm_mon` to view the cluster + status, without being able to modify anything. The user +bob+ can view the + entire configuration and status of the cluster, but not make any changes. The + user +carol+ can read everything, and change selected cluster properties as + well as resource roles and location constraints. Finally, +dave+ has full read + and write access to the entire CIB. + + Looking at the +minimal+ role in more depth, it is designed to allow read + access to the +cib+ tag itself, while denying access to particular portions of + its subtree (which is the entire CIB). + + This is because the DC node is indicated in the +cib+ tag, so `crm_mon` will + not be able to report the DC otherwise. However, this does change the security + model to allow by default, since any portions of the CIB not explicitly denied + will be readable. The +cib+ read access could be removed and replaced with read + access to just the +crm_config+ and +status+ sections, for a safer approach at + the cost of not seeing the DC in status output. + + For a simpler configuration, the +minimal+ role allows read access to the + entire +crm_config+ section, which contains cluster properties. It would be + possible to allow read access to specific properties instead (such as + +stonith-enabled+, +dc-uuid+, +have-quorum+, and +cluster-name+) to restrict + access further while still allowing status output, but cluster properties are + unlikely to be considered sensitive. diff --git a/doc/sphinx/Pacemaker_Explained/advanced-options.rst b/doc/sphinx/Pacemaker_Explained/advanced-options.rst new file mode 100644 index 0000000000..5ee5c9b5ed --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/advanced-options.rst @@ -0,0 +1,759 @@ +Advanced Configuration +---------------------- + +.. Convert_to_RST: + + [[s-recurring-start]] + == Specifying When Recurring Actions are Performed == + + + By default, recurring actions are scheduled relative to when the + resource started. So if your resource was last started at 14:32 and + you have a backup set to be performed every 24 hours, then the backup + will always run in the middle of the business day -- hardly + desirable. + + To specify a date and time that the operation should be relative to, set + the operation's +interval-origin+. The cluster uses this point to + calculate the correct +start-delay+ such that the operation will occur + at _origin + (interval * N)_. + + So, if the operation's interval is 24h, its interval-origin is set to + 02:00 and it is currently 14:32, then the cluster would initiate + the operation with a start delay of 11 hours and 28 minutes. If the + resource is moved to another node before 2am, then the operation is + cancelled. + + The value specified for +interval+ and +interval-origin+ can be any + date/time conforming to the + http://en.wikipedia.org/wiki/ISO_8601[ISO8601 standard]. By way of + example, to specify an operation that would run on the first Monday of + 2009 and every Monday after that, you would add: + + .Specifying a Base for Recurring Action Intervals + ===== + [source,XML] + + ===== + + [[s-failure-handling]] + == Handling Resource Failure == + + By default, Pacemaker will attempt to recover failed resources by restarting + them. However, failure recovery is highly configurable. + + === Failure Counts === + + Pacemaker tracks resource failures for each combination of node, resource, and + operation (start, stop, monitor, etc.). + + You can query the fail count for a particular node, resource, and/or operation + using the `crm_failcount` command. For example, to see how many times the + 10-second monitor for +myrsc+ has failed on +node1+, run: + + ---- + # crm_failcount --query -r myrsc -N node1 -n monitor -I 10s + ---- + + If you omit the node, `crm_failcount` will use the local node. If you omit the + operation and interval, `crm_failcount` will display the sum of the fail counts + for all operations on the resource. + + You can use `crm_resource --cleanup` or `crm_failcount --delete` to clear + fail counts. For example, to clear the above monitor failures, run: + + ---- + # crm_resource --cleanup -r myrsc -N node1 -n monitor -I 10s + ---- + + If you omit the resource, `crm_resource --cleanup` will clear failures for all + resources. If you omit the node, it will clear failures on all nodes. If you + omit the operation and interval, it will clear the failures for all operations + on the resource. + + [NOTE] + ==== + Even when cleaning up only a single operation, all failed operations will + disappear from the status display. This allows us to trigger a re-check of the + resource's current status. + ==== + + Higher-level tools may provide other commands for querying and clearing + fail counts. + + The `crm_mon` tool shows the current cluster status, including any failed + operations. To see the current fail counts for any failed resources, call + `crm_mon` with the `--failcounts` option. This shows the fail counts per + resource (that is, the sum of any operation fail counts for the resource). + + === Failure Response === + + Normally, if a running resource fails, pacemaker will try to stop it and start + it again. Pacemaker will choose the best location to start it each time, which + may be the same node that it failed on. + + However, if a resource fails repeatedly, it is possible that there is an + underlying problem on that node, and you might desire trying a different node + in such a case. Pacemaker allows you to set your preference via the + +migration-threshold+ resource meta-attribute. + footnote:[ + The naming of this option was perhaps unfortunate as it is easily + confused with live migration, the process of moving a resource from + one node to another without stopping it. Xen virtual guests are the + most common example of resources that can be migrated in this manner. + ] + + If you define +migration-threshold=pass:[N]+ for a + resource, it will be banned from the original node after 'N' failures. + + [NOTE] + ==== + The +migration-threshold+ is per 'resource', even though fail counts are + tracked per 'operation'. The operation fail counts are added together + to compare against the +migration-threshold+. + ==== + + By default, fail counts remain until manually cleared by an administrator + using `crm_resource --cleanup` or `crm_failcount --delete` (hopefully after + first fixing the failure's cause). It is possible to have fail counts expire + automatically by setting the +failure-timeout+ resource meta-attribute. + + [IMPORTANT] + ==== + A successful operation does not clear past failures. If a recurring monitor + operation fails once, succeeds many times, then fails again days later, its + fail count is 2. Fail counts are cleared only by manual intervention or + falure timeout. + ==== + + For example, a setting of +migration-threshold=2+ and +failure-timeout=60s+ + would cause the resource to move to a new node after 2 failures, and + allow it to move back (depending on stickiness and constraint scores) after one + minute. + + [NOTE] + ==== + +failure-timeout+ is measured since the most recent failure. That is, older + failures do not individually time out and lower the fail count. Instead, all + failures are timed out simultaneously (and the fail count is reset to 0) if + there is no new failure for the timeout period. + ==== + + There are two exceptions to the migration threshold concept: + when a resource either fails to start or fails to stop. + + If the cluster property +start-failure-is-fatal+ is set to +true+ (which is the + default), start failures cause the fail count to be set to +INFINITY+ and thus + always cause the resource to move immediately. + + Stop failures are slightly different and crucial. If a resource fails + to stop and STONITH is enabled, then the cluster will fence the node + in order to be able to start the resource elsewhere. If STONITH is + not enabled, then the cluster has no way to continue and will not try + to start the resource elsewhere, but will try to stop it again after + the failure timeout. + + == Moving Resources == + indexterm:[Moving,Resources] + indexterm:[Resource,Moving] + + === Moving Resources Manually === + + There are primarily two occasions when you would want to move a + resource from its current location: when the whole node is under + maintenance, and when a single resource needs to be moved. + + ==== Standby Mode ==== + + Since everything eventually comes down to a score, you could create + constraints for every resource to prevent them from running on one + node. While pacemaker configuration can seem convoluted at times, not even + we would require this of administrators. + + Instead, one can set a special node attribute which tells the cluster + "don't let anything run here". There is even a helpful tool to help + query and set it, called `crm_standby`. To check the standby status + of the current machine, run: + + ---- + # crm_standby -G + ---- + + A value of +on+ indicates that the node is _not_ able to host any + resources, while a value of +off+ says that it _can_. + + You can also check the status of other nodes in the cluster by + specifying the `--node` option: + + ---- + # crm_standby -G --node sles-2 + ---- + + To change the current node's standby status, use `-v` instead of `-G`: + + ---- + # crm_standby -v on + ---- + + Again, you can change another host's value by supplying a hostname with `--node`. + + A cluster node in standby mode will not run resources, but still contributes to + quorum, and may fence or be fenced by nodes. + + ==== Moving One Resource ==== + + When only one resource is required to move, we could do this by creating + location constraints. However, once again we provide a user-friendly + shortcut as part of the `crm_resource` command, which creates and + modifies the extra constraints for you. If +Email+ were running on + +sles-1+ and you wanted it moved to a specific location, the command + would look something like: + + ---- + # crm_resource -M -r Email -H sles-2 + ---- + + Behind the scenes, the tool will create the following location constraint: + + [source,XML] + + + It is important to note that subsequent invocations of `crm_resource + -M` are not cumulative. So, if you ran these commands + + ---- + # crm_resource -M -r Email -H sles-2 + # crm_resource -M -r Email -H sles-3 + ---- + + then it is as if you had never performed the first command. + + To allow the resource to move back again, use: + + ---- + # crm_resource -U -r Email + ---- + + Note the use of the word _allow_. The resource can move back to its + original location but, depending on +resource-stickiness+, it might + stay where it is. To be absolutely certain that it moves back to + +sles-1+, move it there before issuing the call to `crm_resource -U`: + + ---- + # crm_resource -M -r Email -H sles-1 + # crm_resource -U -r Email + ---- + + Alternatively, if you only care that the resource should be moved from + its current location, try: + + ---- + # crm_resource -B -r Email + ---- + + Which will instead create a negative constraint, like + + [source,XML] + + + This will achieve the desired effect, but will also have long-term + consequences. As the tool will warn you, the creation of a + +-INFINITY+ constraint will prevent the resource from running on that + node until `crm_resource -U` is used. This includes the situation + where every other cluster node is no longer available! + + In some cases, such as when +resource-stickiness+ is set to + +INFINITY+, it is possible that you will end up with the problem + described in <>. The tool can detect + some of these cases and deals with them by creating both + positive and negative constraints. E.g. + + +Email+ prefers +sles-1+ with a score of +-INFINITY+ + + +Email+ prefers +sles-2+ with a score of +INFINITY+ + + which has the same long-term consequences as discussed earlier. + + === Moving Resources Due to Connectivity Changes === + + You can configure the cluster to move resources when external connectivity is + lost in two steps. + + ==== Tell Pacemaker to Monitor Connectivity ==== + + First, add an *ocf:pacemaker:ping* resource to the cluster. The + *ping* resource uses the system utility of the same name to a test whether + list of machines (specified by DNS hostname or IPv4/IPv6 address) are + reachable and uses the results to maintain a node attribute called +pingd+ + by default. + footnote:[ + The attribute name is customizable, in order to allow multiple ping groups to be defined. + ] + + [NOTE] + =========== + Older versions of Pacemaker used a different agent *ocf:pacemaker:pingd* which + is now deprecated in favor of *ping*. If your version of Pacemaker does not + contain the *ping* resource agent, download the latest version from + https://github.com/ClusterLabs/pacemaker/tree/master/extra/resources/ping + =========== + + Normally, the ping resource should run on all cluster nodes, which means that + you'll need to create a clone. A template for this can be found below + along with a description of the most interesting parameters. + + .Common Options for a 'ping' Resource + [width="95%",cols="1m,<4",options="header",align="center"] + |========================================================= + + |Field + |Description + + |dampen + |The time to wait (dampening) for further changes to occur. Use this + to prevent a resource from bouncing around the cluster when cluster + nodes notice the loss of connectivity at slightly different times. + indexterm:[dampen,Ping Resource Option] + indexterm:[Ping Resource,Option,dampen] + + |multiplier + |The number of connected ping nodes gets multiplied by this value to + get a score. Useful when there are multiple ping nodes configured. + indexterm:[multiplier,Ping Resource Option] + indexterm:[Ping Resource,Option,multiplier] + + |host_list + |The machines to contact in order to determine the current + connectivity status. Allowed values include resolvable DNS host + names, IPv4 and IPv6 addresses. + indexterm:[host_list,Ping Resource Option] + indexterm:[Ping Resource,Option,host_list] + + |========================================================= + + .An example ping cluster resource that checks node connectivity once every minute + ===== + [source,XML] + ------------ + + + + + + + + + + + + + ------------ + ===== + + [IMPORTANT] + =========== + You're only half done. The next section deals with telling Pacemaker + how to deal with the connectivity status that +ocf:pacemaker:ping+ is + recording. + =========== + + ==== Tell Pacemaker How to Interpret the Connectivity Data ==== + + [IMPORTANT] + ====== + Before attempting the following, make sure you understand + <>. + ====== + + There are a number of ways to use the connectivity data. + + The most common setup is for people to have a single ping + target (e.g. the service network's default gateway), to prevent the cluster + from running a resource on any unconnected node. + + .Don't run a resource on unconnected nodes + ===== + [source,XML] + ------- + + + + + + ------- + ===== + + A more complex setup is to have a number of ping targets configured. + You can require the cluster to only run resources on nodes that can + connect to all (or a minimum subset) of them. + + .Run only on nodes connected to three or more ping targets. + ===== + [source,XML] + ------- + + ... + + ... + + ... + + + + + + ------- + ===== + + Alternatively, you can tell the cluster only to _prefer_ nodes with the best + connectivity. Just be sure to set +multiplier+ to a value higher than + that of +resource-stickiness+ (and don't set either of them to + +INFINITY+). + + .Prefer the node with the most connected ping nodes + ===== + [source,XML] + ------- + + + + + + ------- + ===== + + It is perhaps easier to think of this in terms of the simple + constraints that the cluster translates it into. For example, if + *sles-1* is connected to all five ping nodes but *sles-2* is only + connected to two, then it would be as if you instead had the following + constraints in your configuration: + + .How the cluster translates the above location constraint + ===== + [source,XML] + ------- + + + ------- + ===== + + The advantage is that you don't have to manually update any + constraints whenever your network connectivity changes. + + You can also combine the concepts above into something even more + complex. The example below shows how you can prefer the node with the + most connected ping nodes provided they have connectivity to at least + three (again assuming that +multiplier+ is set to 1000). + + .A more complex example of choosing a location based on connectivity + ===== + [source,XML] + ------- + + + + + + + + + ------- + ===== + + [[s-migrating-resources]] + === Migrating Resources === + + Normally, when the cluster needs to move a resource, it fully restarts + the resource (i.e. stops the resource on the current node + and starts it on the new node). + + However, some types of resources, such as Xen virtual guests, are able to move to + another location without loss of state (often referred to as live migration + or hot migration). In pacemaker, this is called resource migration. + Pacemaker can be configured to migrate a resource when moving it, + rather than restarting it. + + Not all resources are able to migrate; see the Migration Checklist + below, and those that can, won't do so in all situations. + Conceptually, there are two requirements from which the other + prerequisites follow: + + * The resource must be active and healthy at the old location; and + * everything required for the resource to run must be available on + both the old and new locations. + + The cluster is able to accommodate both 'push' and 'pull' migration models + by requiring the resource agent to support two special actions: + +migrate_to+ (performed on the current location) and +migrate_from+ + (performed on the destination). + + In push migration, the process on the current location transfers the + resource to the new location where is it later activated. In this + scenario, most of the work would be done in the +migrate_to+ action + and, if anything, the activation would occur during +migrate_from+. + + Conversely for pull, the +migrate_to+ action is practically empty and + +migrate_from+ does most of the work, extracting the relevant resource + state from the old location and activating it. + + There is no wrong or right way for a resource agent to implement migration, + as long as it works. + + .Migration Checklist + * The resource may not be a clone. + * The resource must use an OCF style agent. + * The resource must not be in a failed or degraded state. + * The resource agent must support +migrate_to+ and + +migrate_from+ actions, and advertise them in its metadata. + * The resource must have the +allow-migrate+ meta-attribute set to + +true+ (which is not the default). + + If an otherwise migratable resource depends on another resource + via an ordering constraint, there are special situations in which it will be + restarted rather than migrated. + + For example, if the resource depends on a clone, and at the time the resource + needs to be moved, the clone has instances that are stopping and instances + that are starting, then the resource will be restarted. The scheduler is not + yet able to model this situation correctly and so takes the safer (if less + optimal) path. + + Also, if a migratable resource depends on a non-migratable resource, and both + need to be moved, the migratable resource will be restarted. + + [[s-node-health]] + == Tracking Node Health == + + A node may be functioning adequately as far as cluster membership is concerned, + and yet be "unhealthy" in some respect that makes it an undesirable location + for resources. For example, a disk drive may be reporting SMART errors, or the + CPU may be highly loaded. + + Pacemaker offers a way to automatically move resources off unhealthy nodes. + + === Node Health Attributes === + + Pacemaker will treat any node attribute whose name starts with +#health+ as an + indicator of node health. Node health attributes may have one of the following + values: + + .Allowed Values for Node Health Attributes + [width="95%",cols="1,<3",options="header",align="center"] + |========================================================= + + |Value + |Intended significance + + |+red+ + |This indicator is unhealthy + indexterm:[Node health,red] + + |+yellow+ + |This indicator is becoming unhealthy + indexterm:[Node health,yellow] + + |+green+ + |This indicator is healthy + indexterm:[Node health,green] + + |'integer' + |A numeric score to apply to all resources on this node + (0 or positive is healthy, negative is unhealthy) + indexterm:[Node health,score] + + |========================================================= + + === Node Health Strategy === + + Pacemaker assigns a node health score to each node, as the sum of the values of + all its node health attributes. This score will be used as a location + constraint applied to this node for all resources. + + The +node-health-strategy+ cluster option controls how Pacemaker responds to + changes in node health attributes, and how it translates +red+, +yellow+, and + +green+ to scores. + + Allowed values are: + + .Node Health Strategies + [width="95%",cols="1m,<3",options="header",align="center"] + |========================================================= + + |Value + |Effect + + |none + |Do not track node health attributes at all. + indexterm:[Node health,none] + + |migrate-on-red + |Assign the value of +-INFINITY+ to +red+, and 0 to +yellow+ and +green+. + This will cause all resources to move off the node if any attribute is +red+. + indexterm:[Node health,migrate-on-red] + + |only-green + |Assign the value of +-INFINITY+ to +red+ and +yellow+, and 0 to +green+. + This will cause all resources to move off the node if any attribute is +red+ + or +yellow+. + indexterm:[Node health,only-green] + + |progressive + |Assign the value of the +node-health-red+ cluster option to +red+, the value + of +node-health-yellow+ to +yellow+, and the value of +node-health-green+ to + +green+. Each node is additionally assigned a score of +node-health-base+ + (this allows resources to start even if some attributes are +yellow+). This + strategy gives the administrator finer control over how important each value + is. + indexterm:[Node health,progressive] + + |custom + |Track node health attributes using the same values as +progressive+ for + +red+, +yellow+, and +green+, but do not take them into account. + The administrator is expected to implement a policy by defining rules + (see <>) referencing node health attributes. + indexterm:[Node health,custom] + + |========================================================= + + === Measuring Node Health === + + Since Pacemaker calculates node health based on node attributes, + any method that sets node attributes may be used to measure node + health. The most common ways are resource agents or separate daemons. + + Pacemaker provides examples that can be used directly or as a basis for + custom code. The +ocf:pacemaker:HealthCPU+ and +ocf:pacemaker:HealthSMART+ + resource agents set node health attributes based on CPU and disk parameters. + The +ipmiservicelogd+ daemon sets node health attributes based on IPMI + values (the +ocf:pacemaker:SystemHealth+ resource agent can be used to manage + the daemon as a cluster resource). + + In order to take advantage of this feature - firstly add the resource to your cluster, preferably as a cloned resource to constantly measure health on all nodes: + + ===== + [source,XML] + ------------ + + + + + + + + + + + + + + ------------ + ===== + + This way attrd_updater will set proper status for each node running this resource. Any attribute matching "#health-[a-zA-z]+" will force cluster to migrate all resources from unhealthy node and place it on other nodes according to all constraints defined in your cluster. + + When the node is no longer faulty you can force the cluster to restart the cloned resource on faulty node and make it available to take resources, in this case since we are using HealthIOWait provider: + + ---- + # attrd_updater -n "#health-iowait" -U "green" --node="" -d "60s" + ---- + + == Reloading Services After a Definition Change == + + The cluster automatically detects changes to the definition of + services it manages. The normal response is to stop the + service (using the old definition) and start it again (with the new + definition). This works well, but some services are smarter and can + be told to use a new set of options without restarting. + + To take advantage of this capability, the resource agent must: + + . Accept the +reload+ operation and perform any required actions. + _The actions here depend completely on your application!_ + + + .The DRBD agent's logic for supporting +reload+ + ===== + [source,Bash] + ------- + case $1 in + start) + drbd_start + ;; + stop) + drbd_stop + ;; + reload) + drbd_reload + ;; + monitor) + drbd_monitor + ;; + *) + drbd_usage + exit $OCF_ERR_UNIMPLEMENTED + ;; + esac + exit $? + ------- + ===== + . Advertise the +reload+ operation in the +actions+ section of its metadata + + + .The DRBD Agent Advertising Support for the +reload+ Operation + ===== + [source,XML] + ------- + + + + 1.1 + + + Master/Slave OCF Resource Agent for DRBD + + + ... + + + + + + + + + + + + + ------- + ===== + . Advertise one or more parameters that can take effect using +reload+. + + + Any parameter with the +unique+ set to 0 is eligible to be used in this way. + + + .Parameter that can be changed using reload + ===== + [source,XML] + ------- + + Full path to the drbd.conf file. + Path to drbd.conf + + + ------- + ===== + + Once these requirements are satisfied, the cluster will automatically + know to reload the resource (instead of restarting) when a non-unique + field changes. + + [NOTE] + ====== + Metadata will not be re-read unless the resource needs to be started. This may + mean that the resource will be restarted the first time, even though you + changed a parameter with +unique=0+. + ====== + + [NOTE] + ====== + If both a unique and non-unique field are changed simultaneously, the + resource will still be restarted. + ====== diff --git a/doc/sphinx/Pacemaker_Explained/advanced-resources.rst b/doc/sphinx/Pacemaker_Explained/advanced-resources.rst new file mode 100644 index 0000000000..9f348e3467 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/advanced-resources.rst @@ -0,0 +1,1480 @@ +Advanced Resource Types +----------------------- + +.. Convert_to_RST: + + [[group-resources]] + == Groups - A Syntactic Shortcut == + indexterm:[Group Resources] + indexterm:[Resource,Groups] + + + One of the most common elements of a cluster is a set of resources + that need to be located together, start sequentially, and stop in the + reverse order. To simplify this configuration, we support the concept + of groups. + + .A group of two primitive resources + ====== + [source,XML] + ------- + + + + + + + + + ------- + ====== + + + Although the example above contains only two resources, there is no + limit to the number of resources a group can contain. The example is + also sufficient to explain the fundamental properties of a group: + + * Resources are started in the order they appear in (+Public-IP+ + first, then +Email+) + * Resources are stopped in the reverse order to which they appear in + (+Email+ first, then +Public-IP+) + + If a resource in the group can't run anywhere, then nothing after that + is allowed to run, too. + + * If +Public-IP+ can't run anywhere, neither can +Email+; + * but if +Email+ can't run anywhere, this does not affect +Public-IP+ + in any way + + The group above is logically equivalent to writing: + + .How the cluster sees a group resource + ====== + [source,XML] + ------- + + + + + + + + + + + + + + + ------- + ====== + + Obviously as the group grows bigger, the reduced configuration effort + can become significant. + + Another (typical) example of a group is a DRBD volume, the filesystem + mount, an IP address, and an application that uses them. + + === Group Properties === + .Properties of a Group Resource + [width="95%",cols="3m,<5",options="header",align="center"] + |========================================================= + + |Field + |Description + + |id + |A unique name for the group + indexterm:[id,Group Resource Property] + indexterm:[Resource,Group Property,id] + + |========================================================= + + === Group Options === + + Groups inherit the +priority+, +target-role+, and +is-managed+ properties + from primitive resources. See <> for information about + those properties. + + === Group Instance Attributes === + + Groups have no instance attributes. However, any that are set for the group + object will be inherited by the group's children. + + === Group Contents === + + Groups may only contain a collection of cluster resources (see + <>). To refer to a child of a group resource, just use + the child's +id+ instead of the group's. + + === Group Constraints === + + Although it is possible to reference a group's children in + constraints, it is usually preferable to reference the group itself. + + .Some constraints involving groups + ====== + [source,XML] + ------- + + + + + + ------- + ====== + + === Group Stickiness === + indexterm:[resource-stickiness,Groups] + + Stickiness, the measure of how much a resource wants to stay where it + is, is additive in groups. Every active resource of the group will + contribute its stickiness value to the group's total. So if the + default +resource-stickiness+ is 100, and a group has seven members, + five of which are active, then the group as a whole will prefer its + current location with a score of 500. + + [[s-resource-clone]] + == Clones - Resources That Can Have Multiple Active Instances == + indexterm:[Clone Resources] + indexterm:[Resource,Clones] + + 'Clone' resources are resources that can have more than one copy active at the + same time. This allows you, for example, to run a copy of a daemon on every + node. You can clone any primitive or group resource. + footnote:[ + Of course, the service must support running multiple instances. + ] + + === Anonymous versus Unique Clones === + + A clone resource is configured to be either 'anonymous' or 'globally unique'. + + Anonymous clones are the simplest. These behave completely identically + everywhere they are running. Because of this, there can be only one instance of + an anonymous clone active per node. + + The instances of globally unique clones are distinct entities. All instances + are launched identically, but one instance of the clone is not identical to any + other instance, whether running on the same node or a different node. As an + example, a cloned IP address can use special kernel functionality such that + each instance handles a subset of requests for the same IP address. + + [[s-resource-promotable]] + === Promotable clones === + + indexterm:[Promotable Clone Resources] + indexterm:[Resource,Promotable] + + If a clone is 'promotable', its instances can perform a special role that + Pacemaker will manage via the +promote+ and +demote+ actions of the resource + agent. + + Services that support such a special role have various terms for the special + role and the default role: primary and secondary, master and replica, + controller and worker, etc. Pacemaker uses the terms 'master' and 'slave', + footnote:[ + These are historical terms that will eventually be replaced, but the extensive + use of them and the need for backward compatibility makes it a long process. + You may see examples using a +master+ tag instead of a +clone+ tag with the + +promotable+ meta-attribute set to +true+; the +master+ tag is supported, but + deprecated, and will be removed in a future version. You may also see such + services referred to as 'multi-state' or 'stateful'; these means the same thing + as 'promotable'. + ] + but is agnostic to what the service calls them or what they do. + + All that Pacemaker cares about is that an instance comes up in the default role + when started, and the resource agent supports the +promote+ and +demote+ actions + to manage entering and exiting the special role. + + === Clone Properties === + + .Properties of a Clone Resource + [width="95%",cols="3m,<5",options="header",align="center"] + |========================================================= + + |Field + |Description + + |id + |A unique name for the clone + indexterm:[id,Clone Property] + indexterm:[Clone,Property,id] + + |========================================================= + + === Clone Options === + + <> inherited from primitive resources: + +priority, target-role, is-managed+ + + .Clone-specific configuration options + [width="95%",cols="1m,1,<3",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |globally-unique + |false + |If +true+, each clone instance performs a distinct function + indexterm:[globally-unique,Clone Option] + indexterm:[Clone,Option,globally-unique] + + |clone-max + |number of nodes in cluster + |The maximum number of clone instances that can be started across the entire + cluster + indexterm:[clone-max,Clone Option] + indexterm:[Clone,Option,clone-max] + + |clone-node-max + |1 + |If +globally-unique+ is +true+, the maximum number of clone instances that can + be started on a single node + indexterm:[clone-node-max,Clone Option] + indexterm:[Clone,Option,clone-node-max] + + |clone-min + |0 + |Require at least this number of clone instances to be runnable before allowing + resources depending on the clone to be runnable. A value of 0 means require + all clone instances to be runnable. + indexterm:[clone-min,Clone Option] + indexterm:[Clone,Option,clone-min] + + |notify + |false + |Call the resource agent's +notify+ action for all active instances, before and + after starting or stopping any clone instance. The resource agent must support + this action. Allowed values: +false+, +true+ + indexterm:[notify,Clone Option] + indexterm:[Clone,Option,notify] + + |ordered + |false + |If +true+, clone instances must be started sequentially instead of in parallel + Allowed values: +false+, +true+ + indexterm:[ordered,Clone Option] + indexterm:[Clone,Option,ordered] + + |interleave + |false + |When this clone is ordered relative to another clone, if this option is + +false+ (the default), the ordering is relative to 'all' instances of the + other clone, whereas if this option is +true+, the ordering is relative only + to instances on the same node. + Allowed values: +false+, +true+ + indexterm:[interleave,Clone Option] + indexterm:[Clone,Option,interleave] + + |promotable + |false + |If +true+, clone instances can perform a special role that Pacemaker will + manage via the resource agent's +promote+ and +demote+ actions. The resource + agent must support these actions. + Allowed values: +false+, +true+ + indexterm:[promotable,Clone Option] + indexterm:[Clone,Option,promotable] + + |promoted-max + |1 + |If +promotable+ is +true+, the number of instances that can be promoted at one + time across the entire cluster + indexterm:[promoted-max,Clone Option] + indexterm:[Clone,Option,promoted-max] + + |promoted-node-max + |1 + |If +promotable+ is +true+ and +globally-unique+ is +false+, the number of + clone instances can be promoted at one time on a single node + indexterm:[promoted-node-max,Clone Option] + indexterm:[Clone,Option,promoted-node-max] + + |========================================================= + + For backward compatibility, +master-max+ and +master-node-max+ are accepted as + aliases for +promoted-max+ and +promoted-node-max+, but are deprecated since + 2.0.0, and support for them will be removed in a future version. + + === Clone Contents === + + Clones must contain exactly one primitive or group resource. + + .A clone that runs a web server on all nodes + ==== + [source,XML] + ---- + + + + + + + + ---- + ==== + + [WARNING] + You should never reference the name of a clone's child (the primitive or group + resource being cloned). If you think you need to do this, you probably need to + re-evaluate your design. + + === Clone Instance Attributes === + + Clones have no instance attributes; however, any that are set here will be + inherited by the clone's child. + + === Clone Constraints === + + In most cases, a clone will have a single instance on each active cluster + node. If this is not the case, you can indicate which nodes the + cluster should preferentially assign copies to with resource location + constraints. These constraints are written no differently from those + for primitive resources except that the clone's +id+ is used. + + .Some constraints involving clones + ====== + [source,XML] + ------- + + + + + + ------- + ====== + + Ordering constraints behave slightly differently for clones. In the + example above, +apache-stats+ will wait until all copies of +apache-clone+ + that need to be started have done so before being started itself. + Only if _no_ copies can be started will +apache-stats+ be prevented + from being active. Additionally, the clone will wait for + +apache-stats+ to be stopped before stopping itself. + + Colocation of a primitive or group resource with a clone means that + the resource can run on any node with an active instance of the clone. + The cluster will choose an instance based on where the clone is running and + the resource's own location preferences. + + Colocation between clones is also possible. If one clone +A+ is colocated + with another clone +B+, the set of allowed locations for +A+ is limited to + nodes on which +B+ is (or will be) active. Placement is then performed + normally. + + ==== Promotable Clone Constraints ==== + + For promotable clone resources, the +first-action+ and/or +then-action+ fields + for ordering constraints may be set to +promote+ or +demote+ to constrain the + master role, and colocation constraints may contain +rsc-role+ and/or + +with-rsc-role+ fields. + + .Additional colocation constraint options for promotable clone resources + [width="95%",cols="1m,1,<3",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |rsc-role + |Started + |An additional attribute of colocation constraints that specifies the + role that +rsc+ must be in. Allowed values: +Started+, +Master+, + +Slave+. + indexterm:[rsc-role,Ordering Constraints] + indexterm:[Constraints,Ordering,rsc-role] + + |with-rsc-role + |Started + |An additional attribute of colocation constraints that specifies the + role that +with-rsc+ must be in. Allowed values: +Started+, + +Master+, +Slave+. + indexterm:[with-rsc-role,Ordering Constraints] + indexterm:[Constraints,Ordering,with-rsc-role] + + |========================================================= + + .Constraints involving promotable clone resources + ====== + [source,XML] + ------- + + + + + + + + ------- + ====== + + In the example above, +myApp+ will wait until one of the database + copies has been started and promoted to master before being started + itself on the same node. Only if no copies can be promoted will +myApp+ be + prevented from being active. Additionally, the cluster will wait for + +myApp+ to be stopped before demoting the database. + + Colocation of a primitive or group resource with a promotable clone + resource means that it can run on any node with an active instance of + the promotable clone resource that has the specified role (+master+ or + +slave+). In the example above, the cluster will choose a location based on + where database is running as a +master+, and if there are multiple + +master+ instances it will also factor in +myApp+'s own location + preferences when deciding which location to choose. + + Colocation with regular clones and other promotable clone resources is also + possible. In such cases, the set of allowed locations for the +rsc+ + clone is (after role filtering) limited to nodes on which the + +with-rsc+ promotable clone resource is (or will be) in the specified role. + Placement is then performed as normal. + + ==== Using Promotable Clone Resources in Colocation Sets ==== + + .Additional colocation set options relevant to promotable clone resources + [width="95%",cols="1m,1,<6",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |role + |Started + |The role that 'all members' of the set must be in. Allowed values: +Started+, +Master+, + +Slave+. + indexterm:[role,Ordering Constraints] + indexterm:[Constraints,Ordering,role] + + |========================================================= + + In the following example +B+'s master must be located on the same node as +A+'s master. + Additionally resources +C+ and +D+ must be located on the same node as +A+'s + and +B+'s masters. + + .Colocate C and D with A's and B's master instances + ====== + [source,XML] + ------- + + + + + + + + + + + + + ------- + ====== + + ==== Using Promotable Clone Resources in Ordered Sets ==== + + .Additional ordered set options relevant to promotable clone resources + [width="95%",cols="1m,1,<3",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |action + |value of +first-action+ + |An additional attribute of ordering constraint sets that specifies the + action that applies to 'all members' of the set. Allowed + values: +start+, +stop+, +promote+, +demote+. + indexterm:[action,Ordering Constraints] + indexterm:[Constraints,Ordering,action] + + |========================================================= + + .Start C and D after first promoting A and B + ====== + [source,XML] + ------- + + + + + + + + + + + + + ------- + ====== + + In the above example, +B+ cannot be promoted to a master role until +A+ has + been promoted. Additionally, resources +C+ and +D+ must wait until +A+ and +B+ + have been promoted before they can start. + + + [[s-clone-stickiness]] + === Clone Stickiness === + + indexterm:[resource-stickiness,Clones] + + To achieve a stable allocation pattern, clones are slightly sticky by + default. If no value for +resource-stickiness+ is provided, the clone + will use a value of 1. Being a small value, it causes minimal + disturbance to the score calculations of other resources but is enough + to prevent Pacemaker from needlessly moving copies around the cluster. + + [NOTE] + ==== + For globally unique clones, this may result in multiple instances of the + clone staying on a single node, even after another eligible node becomes + active (for example, after being put into standby mode then made active again). + If you do not want this behavior, specify a +resource-stickiness+ of 0 + for the clone temporarily and let the cluster adjust, then set it back + to 1 if you want the default behavior to apply again. + ==== + + [IMPORTANT] + ==== + If +resource-stickiness+ is set in the +rsc_defaults+ section, it will + apply to clone instances as well. This means an explicit +resource-stickiness+ + of 0 in +rsc_defaults+ works differently from the implicit default used when + +resource-stickiness+ is not specified. + ==== + + === Clone Resource Agent Requirements === + + Any resource can be used as an anonymous clone, as it requires no + additional support from the resource agent. Whether it makes sense to + do so depends on your resource and its resource agent. + + ==== Resource Agent Requirements for Globally Unique Clones ==== + + Globally unique clones require additional support in the resource agent. In + particular, it must only respond with +$\{OCF_SUCCESS}+ if the node has that + exact instance active. All other probes for instances of the clone should + result in +$\{OCF_NOT_RUNNING}+ (or one of the other OCF error codes if + they are failed). + + Individual instances of a clone are identified by appending a colon and a + numerical offset, e.g. +apache:2+. + + Resource agents can find out how many copies there are by examining + the +OCF_RESKEY_CRM_meta_clone_max+ environment variable and which + instance it is by examining +OCF_RESKEY_CRM_meta_clone+. + + The resource agent must not make any assumptions (based on + +OCF_RESKEY_CRM_meta_clone+) about which numerical instances are active. In + particular, the list of active copies will not always be an unbroken + sequence, nor always start at 0. + + ==== Resource Agent Requirements for Promotable Clones ==== + + Promotable clone resources require two extra actions, +demote+ and +promote+, + which are responsible for changing the state of the resource. Like +start+ and + +stop+, they should return +$\{OCF_SUCCESS}+ if they completed successfully or + a relevant error code if they did not. + + The states can mean whatever you wish, but when the resource is + started, it must come up in the mode called +slave+. From there the + cluster will decide which instances to promote to +master+. + + In addition to the clone requirements for monitor actions, agents must + also _accurately_ report which state they are in. The cluster relies + on the agent to report its status (including role) accurately and does + not indicate to the agent what role it currently believes it to be in. + + .Role implications of OCF return codes + [width="95%",cols="1,<1",options="header",align="center"] + |========================================================= + + |Monitor Return Code + |Description + + |OCF_NOT_RUNNING + |Stopped + indexterm:[Return Code,OCF_NOT_RUNNING] + + |OCF_SUCCESS + |Running (Slave) + indexterm:[Return Code,OCF_SUCCESS] + + |OCF_RUNNING_MASTER + |Running (Master) + indexterm:[Return Code,OCF_RUNNING_MASTER] + + |OCF_FAILED_MASTER + |Failed (Master) + indexterm:[Return Code,OCF_FAILED_MASTER] + + |Other + |Failed (Slave) + + |========================================================= + + ==== Clone Notifications ==== + + If the clone has the +notify+ meta-attribute set to +true+, and the resource + agent supports the +notify+ action, Pacemaker will call the action when + appropriate, passing a number of extra variables which, when combined with + additional context, can be used to calculate the current state of the cluster + and what is about to happen to it. + + .Environment variables supplied with Clone notify actions + [width="95%",cols="5,<3",options="header",align="center"] + |========================================================= + + |Variable + |Description + + |OCF_RESKEY_CRM_meta_notify_type + |Allowed values: +pre+, +post+ + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,type] + indexterm:[type,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_operation + |Allowed values: +start+, +stop+ + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,operation] + indexterm:[operation,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_start_resource + |Resources to be started + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,start_resource] + indexterm:[start_resource,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_stop_resource + |Resources to be stopped + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,stop_resource] + indexterm:[stop_resource,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_active_resource + |Resources that are running + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,active_resource] + indexterm:[active_resource,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_inactive_resource + |Resources that are not running + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,inactive_resource] + indexterm:[inactive_resource,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_start_uname + |Nodes on which resources will be started + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,start_uname] + indexterm:[start_uname,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_stop_uname + |Nodes on which resources will be stopped + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,stop_uname] + indexterm:[stop_uname,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_active_uname + |Nodes on which resources are running + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,active_uname] + indexterm:[active_uname,Notification Environment Variable] + + |========================================================= + + The variables come in pairs, such as + +OCF_RESKEY_CRM_meta_notify_start_resource+ and + +OCF_RESKEY_CRM_meta_notify_start_uname+, and should be treated as an + array of whitespace-separated elements. + + +OCF_RESKEY_CRM_meta_notify_inactive_resource+ is an exception, as the + matching +uname+ variable does not exist since inactive resources + are not running on any node. + + Thus, in order to indicate that +clone:0+ will be started on +sles-1+, + +clone:2+ will be started on +sles-3+, and +clone:3+ will be started + on +sles-2+, the cluster would set: + + .Notification variables + ====== + [source,Bash] + ------- + OCF_RESKEY_CRM_meta_notify_start_resource="clone:0 clone:2 clone:3" + OCF_RESKEY_CRM_meta_notify_start_uname="sles-1 sles-3 sles-2" + ------- + ====== + + [NOTE] + ==== + Pacemaker will log but otherwise ignore failures of notify actions. + ==== + + ==== Interpretation of Notification Variables ==== + + .Pre-notification (stop): + + * Active resources: +$OCF_RESKEY_CRM_meta_notify_active_resource+ + * Inactive resources: +$OCF_RESKEY_CRM_meta_notify_inactive_resource+ + * Resources to be started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources to be stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + + + .Post-notification (stop) / Pre-notification (start): + + * Active resources + ** +$OCF_RESKEY_CRM_meta_notify_active_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * Inactive resources + ** +$OCF_RESKEY_CRM_meta_notify_inactive_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * Resources that were started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources that were stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + + + .Post-notification (start): + + * Active resources: + ** +$OCF_RESKEY_CRM_meta_notify_active_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Inactive resources: + ** +$OCF_RESKEY_CRM_meta_notify_inactive_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources that were started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources that were stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + + ==== Extra Notifications for Promotable Clones ==== + + .Extra environment variables supplied for promotable clones + [width="95%",cols="5,<3",options="header",align="center"] + |========================================================= + + |Variable + |Description + + |OCF_RESKEY_CRM_meta_notify_master_resource + |Resources that are running in +Master+ mode + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,master_resource] + indexterm:[master_resource,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_slave_resource + |Resources that are running in +Slave+ mode + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,slave_resource] + indexterm:[slave_resource,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_promote_resource + |Resources to be promoted + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,promote_resource] + indexterm:[promote_resource,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_demote_resource + |Resources to be demoted + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,demote_resource] + indexterm:[demote_resource,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_promote_uname + |Nodes on which resources will be promoted + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,promote_uname] + indexterm:[promote_uname,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_demote_uname + |Nodes on which resources will be demoted + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,demote_uname] + indexterm:[demote_uname,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_master_uname + |Nodes on which resources are running in +Master+ mode + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,master_uname] + indexterm:[master_uname,Notification Environment Variable] + + |OCF_RESKEY_CRM_meta_notify_slave_uname + |Nodes on which resources are running in +Slave+ mode + indexterm:[Environment Variable,OCF_RESKEY_CRM_meta_notify_,slave_uname] + indexterm:[slave_uname,Notification Environment Variable] + + |========================================================= + + ==== Interpretation of Promotable Notification Variables ==== + + .Pre-notification (demote): + + * +Active+ resources: +$OCF_RESKEY_CRM_meta_notify_active_resource+ + * +Master+ resources: +$OCF_RESKEY_CRM_meta_notify_master_resource+ + * +Slave+ resources: +$OCF_RESKEY_CRM_meta_notify_slave_resource+ + * Inactive resources: +$OCF_RESKEY_CRM_meta_notify_inactive_resource+ + * Resources to be started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources to be promoted: +$OCF_RESKEY_CRM_meta_notify_promote_resource+ + * Resources to be demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * Resources to be stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + + + .Post-notification (demote) / Pre-notification (stop): + + * +Active+ resources: +$OCF_RESKEY_CRM_meta_notify_active_resource+ + * +Master+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_master_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * +Slave+ resources: +$OCF_RESKEY_CRM_meta_notify_slave_resource+ + * Inactive resources: +$OCF_RESKEY_CRM_meta_notify_inactive_resource+ + * Resources to be started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources to be promoted: +$OCF_RESKEY_CRM_meta_notify_promote_resource+ + * Resources to be demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * Resources to be stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * Resources that were demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + + + .Post-notification (stop) / Pre-notification (start) + + * +Active+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_active_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * +Master+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_master_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * +Slave+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_slave_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * Inactive resources: + ** +$OCF_RESKEY_CRM_meta_notify_inactive_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * Resources to be started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources to be promoted: +$OCF_RESKEY_CRM_meta_notify_promote_resource+ + * Resources to be demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * Resources to be stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * Resources that were demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * Resources that were stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + + + .Post-notification (start) / Pre-notification (promote) + + * +Active+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_active_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * +Master+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_master_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * +Slave+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_slave_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Inactive resources: + ** +$OCF_RESKEY_CRM_meta_notify_inactive_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources to be started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources to be promoted: +$OCF_RESKEY_CRM_meta_notify_promote_resource+ + * Resources to be demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * Resources to be stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * Resources that were started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources that were demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * Resources that were stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + + .Post-notification (promote) + + * +Active+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_active_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * +Master+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_master_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_promote_resource+ + * +Slave+ resources: + ** +$OCF_RESKEY_CRM_meta_notify_slave_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_start_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_promote_resource+ + * Inactive resources: + ** +$OCF_RESKEY_CRM_meta_notify_inactive_resource+ + ** plus +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + ** minus +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources to be started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources to be promoted: +$OCF_RESKEY_CRM_meta_notify_promote_resource+ + * Resources to be demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * Resources to be stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + * Resources that were started: +$OCF_RESKEY_CRM_meta_notify_start_resource+ + * Resources that were promoted: +$OCF_RESKEY_CRM_meta_notify_promote_resource+ + * Resources that were demoted: +$OCF_RESKEY_CRM_meta_notify_demote_resource+ + * Resources that were stopped: +$OCF_RESKEY_CRM_meta_notify_stop_resource+ + + === Monitoring Promotable Clone Resources === + + The usual monitor actions are insufficient to monitor a promotable clone + resource, because Pacemaker needs to verify not only that the resource is + active, but also that its actual role matches its intended one. + + Define two monitoring actions: the usual one will cover the slave role, + and an additional one with +role="master"+ will cover the master role. + + .Monitoring both states of a promotable clone resource + ====== + [source,XML] + ------- + + + + + + + + + + + + ------- + ====== + + [IMPORTANT] + =========== + It is crucial that _every_ monitor operation has a different interval! + Pacemaker currently differentiates between operations + only by resource and interval; so if (for example) a promotable clone resource + had the same monitor interval for both roles, Pacemaker would ignore the + role when checking the status -- which would cause unexpected return + codes, and therefore unnecessary complications. + =========== + + [[s-promotion-scores]] + === Determining Which Instance is Promoted === + + Pacemaker can choose a promotable clone instance to be promoted in one of two + ways: + + * Promotion scores: These are node attributes set via the `crm_master` utility, + which generally would be called by the resource agent's start action if it + supports promotable clones. This tool automatically detects both the resource + and host, and should be used to set a preference for being promoted. Based on + this, +promoted-max+, and +promoted-node-max+, the instance(s) with the + highest preference will be promoted. + + * Constraints: Location constraints can indicate which nodes are most preferred + as masters. + + .Explicitly preferring node1 to be promoted to master + ====== + [source,XML] + ------- + + + + + + ------- + ====== + + [[s-resource-bundle]] + == Bundles - Isolated Environments == + indexterm:[Resource,Bundle] + indexterm:[Container,Docker,Bundle] + indexterm:[Container,podman,Bundle] + indexterm:[Container,rkt,Bundle] + + Pacemaker supports a special syntax for launching a + https://en.wikipedia.org/wiki/Operating-system-level_virtualization[container] + with any infrastructure it requires: the 'bundle'. + + Pacemaker bundles support https://www.docker.com/[Docker], + https://podman.io/[podman], and https://coreos.com/rkt/[rkt] + container technologies. + footnote:[Docker is a trademark of Docker, Inc. No endorsement by or + association with Docker, Inc. is implied.] + + .A bundle for a containerized web server + ==== + [source,XML] + ---- + + + + + + + + + + + + + ---- + ==== + + === Bundle Prerequisites === + indexterm:[Resource,Bundle,Prerequisites] + + Before configuring a bundle in Pacemaker, the user must install the appropriate + container launch technology (Docker, podman, or rkt), and supply a fully + configured container image, on every node allowed to run the bundle. + + Pacemaker will create an implicit resource of type +ocf:heartbeat:docker+, + +ocf:heartbeat:podman+, or +ocf:heartbeat:rkt+ to manage a bundle's + container. The user must ensure that the appropriate resource agent is + installed on every node allowed to run the bundle. + + === Bundle Properties === + + indexterm:[XML element,bundle element] + + .XML Attributes of a bundle Element + [width="95%",cols="3m,<5",options="header",align="center"] + |========================================================= + + |Attribute + |Description + + |id + |A unique name for the bundle (required) + indexterm:[XML attribute,id attribute,bundle element] + indexterm:[XML element,bundle element,id attribute] + + |description + |Arbitrary text (not used by Pacemaker) + indexterm:[XML attribute,description attribute,bundle element] + indexterm:[XML element,bundle element,description attribute] + + |========================================================= + + A bundle must contain exactly one +docker+, +podman+, or +rkt+ element. + + === Bundle Container Properties === + indexterm:[XML element,docker element] + indexterm:[XML element,podman element] + indexterm:[XML element,rkt element] + indexterm:[Resource,Bundle,Container] + + .XML attributes of a docker, podman, or rkt Element + [width="95%",cols="3m,4,<5",options="header",align="center"] + |==== + + |Attribute + |Default + |Description + + |image + | + |Container image tag (required) + indexterm:[XML attribute,image attribute,docker element] + indexterm:[XML element,docker element,image attribute] + indexterm:[XML attribute,image attribute,podman element] + indexterm:[XML element,podman element,image attribute] + indexterm:[XML attribute,image attribute,rkt element] + indexterm:[XML element,rkt element,image attribute] + + |replicas + |Value of +promoted-max+ if that is positive, else 1 + |A positive integer specifying the number of container instances to launch + indexterm:[XML attribute,replicas attribute,docker element] + indexterm:[XML element,docker element,replicas attribute] + indexterm:[XML attribute,replicas attribute,podman element] + indexterm:[XML element,podman element,replicas attribute] + indexterm:[XML attribute,replicas attribute,rkt element] + indexterm:[XML element,rkt element,replicas attribute] + + |replicas-per-host + |1 + |A positive integer specifying the number of container instances allowed to run + on a single node + indexterm:[XML attribute,replicas-per-host attribute,docker element] + indexterm:[XML element,docker element,replicas-per-host attribute] + indexterm:[XML attribute,replicas-per-host attribute,podman element] + indexterm:[XML element,podman element,replicas-per-host attribute] + indexterm:[XML attribute,replicas-per-host attribute,rkt element] + indexterm:[XML element,rkt element,replicas-per-host attribute] + + |promoted-max + |0 + |A non-negative integer that, if positive, indicates that the containerized + service should be treated as a promotable service, with this many replicas + allowed to run the service in the master role + indexterm:[XML attribute,promoted-max attribute,docker element] + indexterm:[XML element,docker element,promoted-max attribute] + indexterm:[XML attribute,promoted-max attribute,podman element] + indexterm:[XML element,podman element,promoted-max attribute] + indexterm:[XML attribute,promoted-max attribute,rkt element] + indexterm:[XML element,rkt element,promoted-max attribute] + + |network + | + |If specified, this will be passed to the `docker run`, `podman run`, or + `rkt run` command as the network setting for the container. + indexterm:[XML attribute,network attribute,docker element] + indexterm:[XML element,docker element,network attribute] + indexterm:[XML attribute,network attribute,podman element] + indexterm:[XML element,podman element,network attribute] + indexterm:[XML attribute,network attribute,rkt element] + indexterm:[XML element,rkt element,network attribute] + + |run-command + |`/usr/sbin/pacemaker-remoted` if bundle contains a +primitive+, otherwise none + |This command will be run inside the container when launching it ("PID 1"). If + the bundle contains a +primitive+, this command 'must' start pacemaker-remoted + (but could, for example, be a script that does other stuff, too). + indexterm:[XML attribute,run-command attribute,docker element] + indexterm:[XML element,docker element,run-command attribute] + indexterm:[XML attribute,run-command attribute,podman element] + indexterm:[XML element,podman element,run-command attribute] + indexterm:[XML attribute,run-command attribute,rkt element] + indexterm:[XML element,rkt element,run-command attribute] + + |options + | + |Extra command-line options to pass to the `docker run`, `podman run`, or + `rkt run` command + indexterm:[XML attribute,options attribute,docker element] + indexterm:[XML element,docker element,options attribute] + indexterm:[XML attribute,options attribute,podman element] + indexterm:[XML element,podman element,options attribute] + indexterm:[XML attribute,options attribute,rkt element] + indexterm:[XML element,rkt element,options attribute] + + |==== + + [NOTE] + ==== + Considerations when using cluster configurations or container images from + Pacemaker 1.1: + + - If the container image has a pre-2.0.0 version of Pacemaker, set +run-command+ + to +/usr/sbin/pacemaker_remoted+ (note the underbar instead of dash). + + - +masters+ is accepted as an alias for +promoted-max+, but is deprecated since + 2.0.0, and support for it will be removed in a future version. + ==== + + === Bundle Network Properties === + + A bundle may optionally contain one ++ element. + indexterm:[XML element,network element] + indexterm:[Resource,Bundle,Networking] + + .XML attributes of a network Element + [width="95%",cols="2m,1,<4",options="header",align="center"] + |========================================================= + + |Attribute + |Default + |Description + + |add-host + |TRUE + |If TRUE, and +ip-range-start+ is used, Pacemaker will automatically ensure + that +/etc/hosts+ inside the containers has entries for each + <> and its assigned IP. + indexterm:[XML element,add-host attribute,network element] + indexterm:[XML attribute,network element,add-host attribute] + + |ip-range-start + | + |If specified, Pacemaker will create an implicit +ocf:heartbeat:IPaddr2+ + resource for each container instance, starting with this IP address, + using up to +replicas+ sequential addresses. These addresses can be used + from the host's network to reach the service inside the container, though + it is not visible within the container itself. Only IPv4 addresses are + currently supported. + indexterm:[XML element,ip-range-start attribute,network element] + indexterm:[XML attribute,network element,ip-range-start attribute] + + |host-netmask + |32 + |If +ip-range-start+ is specified, the IP addresses are created with this + CIDR netmask (as a number of bits). + indexterm:[XML element,host-netmask attribute,network element] + indexterm:[XML attribute,network element,host-netmask attribute] + + |host-interface + | + |If +ip-range-start+ is specified, the IP addresses are created on this + host interface (by default, it will be determined from the IP address). + indexterm:[XML element,host-interface attribute,network element] + indexterm:[XML attribute,network element,host-interface attribute] + + |control-port + |3121 + |If the bundle contains a +primitive+, the cluster will use this integer TCP + port for communication with Pacemaker Remote inside the container. Changing + this is useful when the container is unable to listen on the default port, + for example, when the container uses the host's network rather than + +ip-range-start+ (in which case +replicas-per-host+ must be 1), or when the + bundle may run on a Pacemaker Remote node that is already listening on the + default port. Any PCMK_remote_port environment variable set on the host or in + the container is ignored for bundle connections. + indexterm:[XML element,control-port attribute,network element] + indexterm:[XML attribute,network element,control-port attribute] + + |========================================================= + + [[s-resource-bundle-note-replica-names]] + [NOTE] + ==== + Replicas are named by the bundle id plus a dash and an integer counter starting + with zero. For example, if a bundle named +httpd-bundle+ has +replicas=2+, its + containers will be named +httpd-bundle-0+ and +httpd-bundle-1+. + ==== + + Additionally, a +network+ element may optionally contain one or more + +port-mapping+ elements. + indexterm:[XML element,port-mapping] + + .Attributes of a port-mapping Element + [width="95%",cols="2m,1,<4",options="header",align="center"] + |========================================================= + + |Attribute + |Default + |Description + + |id + | + |A unique name for the port mapping (required) + indexterm:[XML attribute,id attribute,port-mapping element] + indexterm:[XML element,port-mapping element,id attribute] + + |port + | + |If this is specified, connections to this TCP port number on the host network + (on the container's assigned IP address, if +ip-range-start+ is specified) + will be forwarded to the container network. Exactly one of +port+ or +range+ + must be specified in a +port-mapping+. + indexterm:[XML attribute,port attribute,port-mapping element] + indexterm:[XML element,port-mapping element,port attribute] + + |internal-port + |value of +port+ + |If +port+ and this are specified, connections to +port+ on the host's network + will be forwarded to this port on the container network. + indexterm:[XML attribute,internal-port attribute,port-mapping element] + indexterm:[XML element,port-mapping element,internal-port attribute] + + |range + | + |If this is specified, connections to these TCP port numbers (expressed as + 'first_port'-'last_port') on the host network (on the container's assigned IP + address, if +ip-range-start+ is specified) will be forwarded to the same ports + in the container network. Exactly one of +port+ or +range+ must be specified + in a +port-mapping+. + indexterm:[XML attribute,range attribute,port-mapping element] + indexterm:[XML element,port-mapping element,range attribute] + + |========================================================= + + [NOTE] + ==== + If the bundle contains a +primitive+, Pacemaker will automatically map the + +control-port+, so it is not necessary to specify that port in a + +port-mapping+. + ==== + + [[s-bundle-storage]] + === Bundle Storage Properties === + + A bundle may optionally contain one +storage+ element. A +storage+ element + has no properties of its own, but may contain one or more +storage-mapping+ + elements. + indexterm:[XML element,storage element] + indexterm:[XML element,storage-mapping element] + indexterm:[Resource,Bundle,Storage] + + .Attributes of a storage-mapping Element + [width="95%",cols="2m,1,<4",options="header",align="center"] + |========================================================= + + |Attribute + |Default + |Description + + |id + | + |A unique name for the storage mapping (required) + indexterm:[XML attribute,id attribute,storage-mapping element] + indexterm:[XML element,storage-mapping element,id attribute] + + |source-dir + | + |The absolute path on the host's filesystem that will be mapped into the + container. Exactly one of +source-dir+ and +source-dir-root+ must be specified + in a +storage-mapping+. + indexterm:[XML attribute,source-dir attribute,storage-mapping element] + indexterm:[XML element,storage-mapping element,source-dir attribute] + + |source-dir-root + | + |The start of a path on the host's filesystem that will be mapped into the + container, using a different subdirectory on the host for each container + instance. The subdirectory will be named the same as the + <>. + Exactly one of +source-dir+ and +source-dir-root+ must be specified in a + +storage-mapping+. + indexterm:[XML attribute,source-dir-root attribute,storage-mapping element] + indexterm:[XML element,storage-mapping element,source-dir-root attribute] + + |target-dir + | + |The path name within the container where the host storage will be mapped + (required) + indexterm:[XML attribute,target-dir attribute,storage-mapping element] + indexterm:[XML element,storage-mapping element,target-dir attribute] + + |options + | + |A comma-separated list of file system mount options to use when mapping the + storage + indexterm:[XML attribute,options attribute,storage-mapping element] + indexterm:[XML element,storage-mapping element,options attribute] + + |========================================================= + + [NOTE] + ==== + Pacemaker does not define the behavior if the source directory does not already + exist on the host. However, it is expected that the container technology and/or + its resource agent will create the source directory in that case. + ==== + + [NOTE] + ==== + If the bundle contains a +primitive+, + Pacemaker will automatically map the equivalent of + +source-dir=/etc/pacemaker/authkey target-dir=/etc/pacemaker/authkey+ + and +source-dir-root=/var/log/pacemaker/bundles target-dir=/var/log+ into the + container, so it is not necessary to specify those paths in a + +storage-mapping+. + ==== + + [IMPORTANT] + ==== + The +PCMK_authkey_location+ environment variable must not be set to anything + other than the default of `/etc/pacemaker/authkey` on any node in the cluster. + ==== + + [IMPORTANT] + ==== + If SELinux is used in enforcing mode on the host, you must ensure the container + is allowed to use any storage you mount into it. For Docker and podman bundles, + adding "Z" to the mount options will create a container-specific label for the + mount that allows the container access. + ==== + + === Bundle Primitive === + + indexterm:[Resource,Bundle,Primitive] + + A bundle may optionally contain one <> + resource. The primitive may have operations, instance attributes, and + meta-attributes defined, as usual. + + If a bundle contains a primitive resource, the container image must include + the Pacemaker Remote daemon, and at least one of +ip-range-start+ or + +control-port+ must be configured in the bundle. Pacemaker will create an + implicit +ocf:pacemaker:remote+ resource for the connection, launch + Pacemaker Remote within the container, and monitor and manage the primitive + resource via Pacemaker Remote. + + If the bundle has more than one container instance (replica), the primitive + resource will function as an implicit <> -- a + <> if the bundle has +masters+ greater + than zero. + + [NOTE] + ==== + If you want to pass environment variables to a bundle's Pacemaker Remote + connection or primitive, you have two options: + + * Environment variables whose value is the same regardless of the underlying host + may be set using the container element's +options+ attribute. + * If you want variables to have host-specific values, you can use the + <> element to map a file on the host as + +/etc/pacemaker/pcmk-init.env+ in the container. Pacemaker Remote will parse + this file as a shell-like format, with variables set as NAME=VALUE, ignoring + blank lines and comments starting with "#". + ==== + + [IMPORTANT] + ==== + When a bundle has a +primitive+, Pacemaker on all cluster nodes must be able to + contact Pacemaker Remote inside the bundle's containers. + + * The containers must have an accessible network (for example, +network+ should + not be set to "none" with a +primitive+). + * The default, using a distinct network space inside the container, works in + combination with +ip-range-start+. Any firewall must allow access from all + cluster nodes to the +control-port+ on the container IPs. + * If the container shares the host's network space (for example, by setting + +network+ to "host"), a unique +control-port+ should be specified for each + bundle. Any firewall must allow access from all cluster nodes to the + +control-port+ on all cluster and remote node IPs. + ==== + + [[s-bundle-attributes]] + === Bundle Node Attributes === + + indexterm:[Resource,Bundle,Node Attributes] + + If the bundle has a +primitive+, the primitive's resource agent may want to set + node attributes such as <>. However, with + containers, it is not apparent which node should get the attribute. + + If the container uses shared storage that is the same no matter which node the + container is hosted on, then it is appropriate to use the promotion score on the + bundle node itself. + + On the other hand, if the container uses storage exported from the underlying host, + then it may be more appropriate to use the promotion score on the underlying host. + + Since this depends on the particular situation, the + +container-attribute-target+ resource meta-attribute allows the user to specify + which approach to use. If it is set to +host+, then user-defined node attributes + will be checked on the underlying host. If it is anything else, the local node + (in this case the bundle node) is used as usual. + + This only applies to user-defined attributes; the cluster will always check the + local node for cluster-defined attributes such as +#uname+. + + If +container-attribute-target+ is +host+, the cluster will pass additional + environment variables to the primitive's resource agent that allow it to set + node attributes appropriately: +CRM_meta_container_attribute_target+ (identical + to the meta-attribute value) and +CRM_meta_physical_host+ (the name of the + underlying host). + + [NOTE] + ==== + When called by a resource agent, the `attrd_updater` and `crm_attribute` + commands will automatically check those environment variables and set + attributes appropriately. + ==== + + === Bundle Meta-Attributes === + + indexterm:[Resource,Bundle,Meta-attributes] + + Any meta-attribute set on a bundle will be inherited by the bundle's + primitive and any resources implicitly created by Pacemaker for the bundle. + + This includes options such as +priority+, +target-role+, and +is-managed+. See + <> for more information. + + === Limitations of Bundles === + + Restarting pacemaker while a bundle is unmanaged or the cluster is in + maintenance mode may cause the bundle to fail. + + Bundles may not be explicitly cloned or included in groups. This includes the + bundle's primitive and any resources implicitly created by Pacemaker for the + bundle. (If +replicas+ is greater than 1, the bundle will behave like a clone + implicitly.) + + Bundles do not have instance attributes, utilization attributes, or operations, + though a bundle's primitive may have them. + + A bundle with a primitive can run on a Pacemaker Remote node only if the bundle + uses a distinct +control-port+. diff --git a/doc/sphinx/Pacemaker_Explained/alerts.rst b/doc/sphinx/Pacemaker_Explained/alerts.rst new file mode 100644 index 0000000000..baa121edc8 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/alerts.rst @@ -0,0 +1,422 @@ +Alerts +------ + +.. Convert_to_RST: + + anchor:ch-alerts[Chapter 7, Alerts] + indexterm:[Resource,Alerts] + + 'Alerts' may be configured to take some external action when a cluster event + occurs (node failure, resource starting or stopping, etc.). + + + == Alert Agents == + + As with resource agents, the cluster calls an external program (an + 'alert agent') to handle alerts. The cluster passes information about the event + to the agent via environment variables. Agents can do anything + desired with this information (send an e-mail, log to a file, + update a monitoring system, etc.). + + + .Simple alert configuration + ===== + [source,XML] + ----- + + + + + + ----- + ===== + + In the example above, the cluster will call +my-script.sh+ for each event. + + Multiple alert agents may be configured; the cluster will call all of them for + each event. + + Alert agents will be called only on cluster nodes. They will be called for + events involving Pacemaker Remote nodes, but they will never be called _on_ + those nodes. + + == Alert Recipients == + + Usually alerts are directed towards a recipient. Thus each alert may be additionally configured with one or more recipients. + The cluster will call the agent separately for each recipient. + + .Alert configuration with recipient + ===== + [source,XML] + ----- + + + + + + + + ----- + ===== + + In the above example, the cluster will call +my-script.sh+ for each event, + passing the recipient +some-address+ as an environment variable. + + The recipient may be anything the alert agent can recognize -- + an IP address, an e-mail address, a file name, whatever the particular + agent supports. + + + == Alert Meta-Attributes == + + As with resource agents, meta-attributes can be configured for alert agents + to affect how Pacemaker calls them. + + .Meta-Attributes of an Alert + + [width="95%",cols="m,1,<2",options="header",align="center"] + |========================================================= + + |Meta-Attribute + |Default + |Description + + |timestamp-format + |%H:%M:%S.%06N + |Format the cluster will use when sending the event's timestamp to the agent. + This is a string as used with the `date(1)` command. + indexterm:[Alert,Option,timestamp-format] + + |timeout + |30s + |If the alert agent does not complete within this amount of time, it will be + terminated. + indexterm:[Alert,Option,timeout] + + |========================================================= + + Meta-attributes can be configured per alert agent and/or per recipient. + + .Alert configuration with meta-attributes + ===== + [source,XML] + ----- + + + + + + + + + + + + + + + + + + + + ----- + ===== + + In the above example, the +my-script.sh+ will get called twice for each event, + with each call using a 15-second timeout. One call will be passed the recipient + +someuser@example.com+ and a timestamp in the format +%D %H:%M+, while the + other call will be passed the recipient +otheruser@example.com+ and a timestamp + in the format +%c+. + + + == Alert Instance Attributes == + + As with resource agents, agent-specific configuration values may be configured + as instance attributes. These will be passed to the agent as additional + environment variables. The number, names and allowed values of these + instance attributes are completely up to the particular agent. + + .Alert configuration with instance attributes + ===== + [source,XML] + ----- + + + + + + + + + + + + + + ----- + ===== + + + == Alert Filters == + + By default, an alert agent will be called for node events, fencing events, and + resource events. An agent may choose to ignore certain types of events, but + there is still the overhead of calling it for those events. To eliminate that + overhead, you may select which types of events the agent should receive. + + .Alert configuration to receive only node events and fencing events + ===== + [source,XML] + ----- + + + + + + + + + ----- + ===== + + The possible options within + + + + + + + + + + + ----- + ===== + + Node attribute alerts are currently considered experimental. Alerts may be + limited to attributes set via attrd_updater, and agents may be called multiple + times with the same attribute value. + + + == Using the Sample Alert Agents == + + Pacemaker provides several sample alert agents, installed in + +/usr/share/pacemaker/alerts+ by default. + + While these sample scripts may be copied and used as-is, they are provided + mainly as templates to be edited to suit your purposes. + See their source code for the full set of instance attributes they support. + + .Sending cluster events as SNMP traps + ===== + [source,XML] + ----- + + + + + + + + + + + + + + ----- + ===== + + .Sending cluster events as e-mails + ===== + [source,XML] + ----- + + + + + + + + + + + ----- + ===== + + + == Writing an Alert Agent == + + .Environment variables passed to alert agents + + [width="95%",cols="m,<2",options="header",align="center"] + |========================================================= + + |Environment Variable + |Description + + |CRM_alert_kind + |The type of alert (+node+, +fencing+, +resource+, or +attribute+) + indexterm:[Environment Variable,CRM_alert_,kind] + + |CRM_alert_version + |The version of Pacemaker sending the alert + indexterm:[Environment Variable,CRM_alert_,version] + + |CRM_alert_recipient + |The configured recipient + indexterm:[Environment Variable,CRM_alert_,recipient] + + |CRM_alert_node_sequence + |A sequence number increased whenever an alert is being issued on the + local node, which can be used to reference the order in which alerts have been + issued by Pacemaker. An alert for an event that happened later in time + reliably has a higher sequence number than alerts for earlier events. + Be aware that this number has no cluster-wide meaning. + indexterm:[Environment Variable,CRM_alert_node_,sequence] + + |CRM_alert_timestamp + |A timestamp created prior to executing the agent, in the format + specified by the +timestamp-format+ meta-attribute. This allows the agent + to have a reliable, high-precision time of when the event occurred, + regardless of when the agent itself was invoked (which could potentially + be delayed due to system load, etc.). + indexterm:[Environment Variable,CRM_alert_,timestamp] + + |CRM_alert_timestamp_epoch + |The same time as +CRM_alert_timestamp+, expressed as the integer number of + seconds since January 1, 1970. This (along with +CRM_alert_timestamp_usec+) + can be useful for alert agents that need to format time in a specific way + rather than let the user configure it. + indexterm:[Environment Variable,CRM_alert_,timestamp_epoch] + + |CRM_alert_timestamp_usec + |The same time as +CRM_alert_timestamp+, expressed as the integer number of + microseconds since +CRM_alert_timestamp_epoch+. + indexterm:[Environment Variable,CRM_alert_,timestamp_usec] + + |CRM_alert_node + |Name of affected node + indexterm:[Environment Variable,CRM_alert_,node] + + |CRM_alert_desc + |Detail about event. For +node+ alerts, this is the node's current state + (+member+ or +lost+). For +fencing+ alerts, this is a summary of the + requested fencing operation, including origin, target, and fencing operation + error code, if any. For +resource+ alerts, this is a readable string + equivalent of +CRM_alert_status+. + indexterm:[Environment Variable,CRM_alert_,desc] + + |CRM_alert_nodeid + |ID of node whose status changed (provided with +node+ alerts only) + indexterm:[Environment Variable,CRM_alert_,nodeid] + + |CRM_alert_task + |The requested fencing or resource operation + (provided with +fencing+ and +resource+ alerts only) + indexterm:[Environment Variable,CRM_alert_,task] + + |CRM_alert_rc + |The numerical return code of the fencing or resource operation + (provided with +fencing+ and +resource+ alerts only) + indexterm:[Environment Variable,CRM_alert_,rc] + + |CRM_alert_rsc + |The name of the affected resource (+resource+ alerts only) + indexterm:[Environment Variable,CRM_alert_,rsc] + + |CRM_alert_interval + |The interval of the resource operation (+resource+ alerts only) + indexterm:[Environment Variable,CRM_alert_,interval] + + |CRM_alert_target_rc + |The expected numerical return code of the operation (+resource+ alerts only) + indexterm:[Environment Variable,CRM_alert_,target_rc] + + |CRM_alert_status + |A numerical code used by Pacemaker to represent the operation result + (+resource+ alerts only) + indexterm:[Environment Variable,CRM_alert_,status] + + |CRM_alert_exec_time + |The (wall-clock) time, in milliseconds, that it took to execute the action. If + the action timed out, +CRM_alert_status+ will be 2, +CRM_alert_desc+ will be + "Timed Out", and this value will be the action timeout. May not be supported + on all platforms. (+resource+ alerts only) + indexterm:[Environment Variable,CRM_alert_,exec_time] + + |CRM_alert_attribute_name + |The name of the node attribute that changed (+attribute+ alerts only) + indexterm:[Environment Variable,CRM_alert_,attribute_name] + + |CRM_alert_attribute_value + |The new value of the node attribute that changed (+attribute+ alerts only) + indexterm:[Environment Variable,CRM_alert_,attribute_value] + + |========================================================= + + Special concerns when writing alert agents: + + * Alert agents may be called with no recipient (if none is configured), + so the agent must be able to handle this situation, even if it + only exits in that case. (Users may modify the configuration in + stages, and add a recipient later.) + + * If more than one recipient is configured for an alert, the alert agent will + be called once per recipient. If an agent is not able to run concurrently, it + should be configured with only a single recipient. The agent is free, + however, to interpret the recipient as a list. + + * When a cluster event occurs, all alerts are fired off at the same time as + separate processes. Depending on how many alerts and recipients are + configured, and on what is done within the alert agents, + a significant load burst may occur. The agent could be written to take + this into consideration, for example by queueing resource-intensive actions + into some other instance, instead of directly executing them. + + * Alert agents are run as the +hacluster+ user, which has a minimal set + of permissions. If an agent requires additional privileges, it is + recommended to configure +sudo+ to allow the agent to run the necessary + commands as another user with the appropriate privileges. + + * As always, take care to validate and sanitize user-configured parameters, + such as CRM_alert_timestamp (whose content is specified by the + user-configured timestamp-format), CRM_alert_recipient, and all instance + attributes. Mostly this is needed simply to protect against configuration + errors, but if some user can modify the CIB without having hacluster-level + access to the cluster nodes, it is a potential security concern as well, to + avoid the possibility of code injection. + + [NOTE] + ===== + The alerts interface is designed to be backward compatible with the external + scripts interface used by the +ocf:pacemaker:ClusterMon+ resource, which is + now deprecated. To preserve this compatibility, the environment variables + passed to alert agents are available prepended with +CRM_notify_+ + as well as +CRM_alert_+. One break in compatibility is that ClusterMon ran + external scripts as the +root+ user, while alert agents are run as the + +hacluster+ user. + ===== diff --git a/doc/sphinx/Pacemaker_Explained/ap-samples.rst b/doc/sphinx/Pacemaker_Explained/ap-samples.rst new file mode 100644 index 0000000000..cbebe404be --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/ap-samples.rst @@ -0,0 +1,156 @@ +Sample Configurations +--------------------- + +.. Convert_to_RST: + + [appendix] + + + === Empty === + + .An Empty Configuration + ======= + [source,XML] + ------- + + + + + + + + + + ------- + ======= + + === Simple === + + .A simple configuration with two nodes, some cluster options and a resource + ======= + [source,XML] + ------- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ------- + ======= + + In the above example, we have one resource (an IP address) that we check + every five minutes and will run on host +c001n01+ until either the + resource fails 10 times or the host shuts down. + + === Advanced Configuration === + + .An advanced configuration with groups, clones and STONITH + ======= + [source,XML] + ------- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ------- + ======= diff --git a/doc/sphinx/Pacemaker_Explained/constraints.rst b/doc/sphinx/Pacemaker_Explained/constraints.rst new file mode 100644 index 0000000000..3bc55d756b --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/constraints.rst @@ -0,0 +1,920 @@ +Resource Constraints +-------------------- + +.. Convert_to_RST: + + anchor:ch-constraints[Chapter 7, Alerts] + indexterm:[Resource,Constraint] + + == Scores == + + indexterm:[Resource,Score] + indexterm:[Node,Score] + Scores of all kinds are integral to how the cluster works. + Practically everything from moving a resource to deciding which + resource to stop in a degraded cluster is achieved by manipulating + scores in some way. + + Scores are calculated per resource and node. Any node with a + negative score for a resource can't run that resource. The cluster + places a resource on the node with the highest score for it. + + === Infinity Math === + + Pacemaker implements +INFINITY+ (or equivalently, ++INFINITY+) internally as a + score of 1,000,000. Addition and subtraction with it follow these three basic + rules: + + * Any value + +INFINITY+ = +INFINITY+ + * Any value - +INFINITY+ = +-INFINITY+ + * +INFINITY+ - +INFINITY+ = +-INFINITY+ + + [NOTE] + ====== + What if you want to use a score higher than 1,000,000? Typically this possibility + arises when someone wants to base the score on some external metric that might + go above 1,000,000. + + The short answer is you can't. + + The long answer is it is sometimes possible work around this limitation + creatively. You may be able to set the score to some computed value based on + the external metric rather than use the metric directly. For nodes, you can + store the metric as a node attribute, and query the attribute when computing + the score (possibly as part of a custom resource agent). + ====== + + == Deciding Which Nodes a Resource Can Run On == + + indexterm:[Constraint,Location Constraint] + 'Location constraints' tell the cluster which nodes a resource can run on. + + There are two alternative strategies. One way is to say that, by default, + resources can run anywhere, and then the location constraints specify nodes + that are not allowed (an 'opt-out' cluster). The other way is to start with + nothing able to run anywhere, and use location constraints to selectively + enable allowed nodes (an 'opt-in' cluster). + + Whether you should choose opt-in or opt-out depends on your + personal preference and the make-up of your cluster. If most of your + resources can run on most of the nodes, then an opt-out arrangement is + likely to result in a simpler configuration. On the other-hand, if + most resources can only run on a small subset of nodes, an opt-in + configuration might be simpler. + + === Location Properties === + + indexterm:[XML element,rsc_location element] + indexterm:[Constraint,Location Constraint,rsc_location element] + + .Attributes of a rsc_location Element + [width="95%",cols="2m,1,<5",options="header",align="center"] + |========================================================= + + |Attribute + |Default + |Description + + |id + | + |A unique name for the constraint (required) + indexterm:[XML attribute,id attribute,rsc_location element] + indexterm:[XML element,rsc_location element,id attribute] + + |rsc + | + |The name of the resource to which this constraint applies. A location + constraint must either have a +rsc+, have a +rsc-pattern+, or contain at least + one resource set. + indexterm:[XML attribute,rsc attribute,rsc_location element] + indexterm:[XML element,rsc_location element,rsc attribute] + + |rsc-pattern + | + |A pattern matching the names of resources to which this constraint applies. + The syntax is the same as + http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04[POSIX] + extended regular expressions, with the addition of an initial '!' indicating + that resources 'not' matching the pattern are selected. If the regular + expression contains submatches, and the constraint is governed by a + <>, the submatches can be referenced as +%0+ through +%9+ in + the rule's +score-attribute+ or a rule expression's +attribute+. A location + constraint must either have a +rsc+, have a +rsc-pattern+, or contain at least + one resource set. + indexterm:[XML attribute,rsc-pattern attribute,rsc_location element] + indexterm:[XML element,rsc_location element,rsc-pattern attribute] + + |node + | + |The name of the node to which this constraint applies. A location constraint + must either have a +node+ and +score+, or contain at least one rule. + indexterm:[XML attribute,node attribute,rsc_location element] + indexterm:[XML element,rsc_location element,node attribute] + + |score + | + |Positive values indicate a preference for running the affected resource(s) on + +node+ -- the higher the value, the stronger the preference. Negative values + indicate the resource(s) should avoid this node (a value of +-INFINITY+ + changes "should" to "must"). A location constraint must either have a +node+ + and +score+, or contain at least one rule. + indexterm:[XML attribute,score attribute,rsc_location element] + indexterm:[XML element,rsc_location element,score attribute] + + |resource-discovery + |always + a|Whether Pacemaker should perform resource discovery (that is, check whether + the resource is already running) for this resource on this node. This should + normally be left as the default, so that rogue instances of a service can be + stopped when they are running where they are not supposed to be. However, + there are two situations where disabling resource discovery is a good idea: + when a service is not installed on a node, discovery might return an error + (properly written OCF agents will not, so this is usually only seen with other + agent types); and when Pacemaker Remote is used to scale a cluster to hundreds + of nodes, limiting resource discovery to allowed nodes can significantly boost + performance. + + * +always:+ Always perform resource discovery for the specified resource on this node. + * +never:+ Never perform resource discovery for the specified resource on this node. + This option should generally be used with a -INFINITY score, although that is not strictly + required. + * +exclusive:+ Perform resource discovery for the specified resource only on + this node (and other nodes similarly marked as +exclusive+). Multiple location + constraints using +exclusive+ discovery for the same resource across + different nodes creates a subset of nodes resource-discovery is exclusive to. + If a resource is marked for +exclusive+ discovery on one or more nodes, that + resource is only allowed to be placed within that subset of nodes. + + indexterm:[XML attribute,resource-discovery attribute,rsc_location element] + indexterm:[XML element,rsc_location element,resource-discovery attribute] + indexterm:[Constraint,Location Constraint,Resource Discovery] + + |========================================================= + + [WARNING] + ========= + Setting resource-discovery to +never+ or +exclusive+ removes Pacemaker's + ability to detect and stop unwanted instances of a service running + where it's not supposed to be. It is up to the system administrator (you!) + to make sure that the service can 'never' be active on nodes without + resource-discovery (such as by leaving the relevant software uninstalled). + ========= + + === Asymmetrical "Opt-In" Clusters === + indexterm:[Asymmetrical Clusters] + indexterm:[Opt-In Clusters] + + To create an opt-in cluster, start by preventing resources from + running anywhere by default: + + ---- + # crm_attribute --name symmetric-cluster --update false + ---- + + Then start enabling nodes. The following fragment says that the web + server prefers *sles-1*, the database prefers *sles-2* and both can + fail over to *sles-3* if their most preferred node fails. + + .Opt-in location constraints for two resources + ====== + [source,XML] + ------- + + + + + + + ------- + ====== + + === Symmetrical "Opt-Out" Clusters === + indexterm:[Symmetrical Clusters] + indexterm:[Opt-Out Clusters] + + To create an opt-out cluster, start by allowing resources to run + anywhere by default: + + ---- + # crm_attribute --name symmetric-cluster --update true + ---- + + Then start disabling nodes. The following fragment is the equivalent + of the above opt-in configuration. + + .Opt-out location constraints for two resources + ====== + [source,XML] + ------- + + + + + + + ------- + ====== + + [[node-score-equal]] + === What if Two Nodes Have the Same Score === + + If two nodes have the same score, then the cluster will choose one. + This choice may seem random and may not be what was intended, however + the cluster was not given enough information to know any better. + + .Constraints where a resource prefers two nodes equally + ====== + [source,XML] + ------- + + + + + + + + ------- + ====== + + In the example above, assuming no other constraints and an inactive + cluster, +Webserver+ would probably be placed on +sles-1+ and +Database+ on + +sles-2+. It would likely have placed +Webserver+ based on the node's + uname and +Database+ based on the desire to spread the resource load + evenly across the cluster. However other factors can also be involved + in more complex configurations. + + [[s-resource-ordering]] + == Specifying the Order in which Resources Should Start/Stop == + + indexterm:[Constraint,Ordering Constraint] + indexterm:[Resource,Start Order] + + 'Ordering constraints' tell the cluster the order in which certain + resource actions should occur. + + [IMPORTANT] + ==== + Ordering constraints affect 'only' the ordering of resource actions; + they do 'not' require that the resources be placed on the + same node. If you want resources to be started on the same node + 'and' in a specific order, you need both an ordering constraint 'and' + a colocation constraint (see <>), or + alternatively, a group (see <>). + ==== + + === Ordering Properties === + + indexterm:[XML element,rsc_order element] + indexterm:[Constraint,Ordering Constraint,rsc_order element] + + .Attributes of a rsc_order Element + [width="95%",cols="1m,1,<4",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |id + | + |A unique name for the constraint + indexterm:[XML attribute,id attribute,rsc_order element] + indexterm:[XML element,rsc_order element,id attribute] + + |first + | + |Name of the resource that the +then+ resource depends on + indexterm:[XML attribute,first attribute,rsc_order element] + indexterm:[XML element,rsc_order element,first attribute] + + |then + | + |Name of the dependent resource + indexterm:[XML attribute,then attribute,rsc_order element] + indexterm:[XML element,rsc_order element,then attribute] + + |first-action + |start + |The action that the +first+ resource must complete before +then-action+ + can be initiated for the +then+ resource. Allowed values: +start+, + +stop+, +promote+, +demote+. + indexterm:[XML attribute,first-action attribute,rsc_order element] + indexterm:[XML element,rsc_order element,first-action attribute] + + |then-action + |value of +first-action+ + |The action that the +then+ resource can execute only after the + +first-action+ on the +first+ resource has completed. Allowed + values: +start+, +stop+, +promote+, +demote+. + indexterm:[XML attribute,then-action attribute,rsc_order element] + indexterm:[XML element,rsc_order element,then-action attribute] + + |kind + |Mandatory + a|How to enforce the constraint. Allowed values: + + * +Mandatory:+ +then-action+ will never be initiated for the +then+ resource + unless and until +first-action+ successfully completes for the +first+ + resource. + * +Optional:+ The constraint applies only if both specified resource actions + are scheduled in the same transition (that is, in response to the same + cluster state). This means that +then-action+ is allowed on the +then+ + resource regardless of the state of the +first+ resource, but if both actions + happen to be scheduled at the same time, they will be ordered. + * +Serialize:+ Ensure that the specified actions are never performed + concurrently for the specified resources. +First-action+ and +then-action+ + can be executed in either order, but one must complete before the other can + be initiated. An example use case is when resource start-up puts a high load + on the host. + + indexterm:[XML attribute,kind attribute,rsc_order element] + indexterm:[XML element,rsc_order element,kind attribute] + + |symmetrical + |TRUE for +Mandatory+ and +Optional+ kinds. FALSE for +Serialize+ kind. + |If true, the reverse of the constraint applies for the opposite action (for + example, if B starts after A starts, then B stops before A stops). + +Serialize+ orders cannot be symmetrical. + indexterm:[XML attribute,symmetrical attribute,rsc_order element] + indexterm:[XML element,rsc_order element,symmetrical attribute] + + |========================================================= + + +Promote+ and +demote+ apply to the master role of + <> resources. + + === Optional and mandatory ordering === + + Here is an example of ordering constraints where +Database+ 'must' start before + +Webserver+, and +IP+ 'should' start before +Webserver+ if they both need to be + started: + + .Optional and mandatory ordering constraints + ====== + [source,XML] + ------- + + + + + ------- + ====== + + Because the above example lets +symmetrical+ default to TRUE, + +Webserver+ must be stopped before +Database+ can be stopped, + and +Webserver+ should be stopped before +IP+ + if they both need to be stopped. + + [[s-resource-colocation]] + == Placing Resources Relative to other Resources == + + indexterm:[Constraint,Colocation Constraint] + indexterm:[Resource,Location Relative to Other Resources] + 'Colocation constraints' tell the cluster that the location of one resource + depends on the location of another one. + + Colocation has an important side-effect: it affects the order in which + resources are assigned to a node. Think about it: You can't place A relative to + B unless you know where B is. + footnote:[ + While the human brain is sophisticated enough to read the constraint + in any order and choose the correct one depending on the situation, + the cluster is not quite so smart. Yet. + ] + + So when you are creating colocation constraints, it is important to + consider whether you should colocate A with B, or B with A. + + Another thing to keep in mind is that, assuming A is colocated with + B, the cluster will take into account A's preferences when + deciding which node to choose for B. + + For a detailed look at exactly how this occurs, see + http://clusterlabs.org/doc/Colocation_Explained.pdf[Colocation Explained]. + + [IMPORTANT] + ==== + Colocation constraints affect 'only' the placement of resources; they do 'not' + require that the resources be started in a particular order. If you want + resources to be started on the same node 'and' in a specific order, you need + both an ordering constraint (see <>) 'and' a colocation + constraint, or alternatively, a group (see <>). + ==== + + === Colocation Properties === + + indexterm:[XML element,rsc_colocation element] + indexterm:[Constraint,Colocation Constraint,rsc_colocation element] + + .Attributes of a rsc_colocation Constraint + [width="95%",cols="1m,1,<4",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |id + | + |A unique name for the constraint (required). + indexterm:[XML attribute,id attribute,rsc_colocation element] + indexterm:[XML element,rsc_colocation element,id attribute] + + |rsc + | + |The name of a resource that should be located relative to +with-rsc+ (required). + indexterm:[XML attribute,rsc attribute,rsc_colocation element] + indexterm:[XML element,rsc_colocation element,rsc attribute] + + |with-rsc + | + |The name of the resource used as the colocation target. The cluster will + decide where to put this resource first and then decide where to put +rsc+ (required). + indexterm:[XML attribute,with-rsc attribute,rsc_colocation element] + indexterm:[XML element,rsc_colocation element,with-rsc attribute] + + |node-attribute + |#uname + |The node attribute that must be the same on the node running +rsc+ and the + node running +with-rsc+ for the constraint to be satisfied. (For details, + see <>.) + indexterm:[XML attribute,node-attribute attribute,rsc_colocation element] + indexterm:[XML element,rsc_colocation element,node-attribute attribute] + + |score + | + |Positive values indicate the resources should run on the same + node. Negative values indicate the resources should run on + different nodes. Values of \+/- +INFINITY+ change "should" to "must". + indexterm:[XML attribute,score attribute,rsc_colocation element] + indexterm:[XML element,rsc_colocation element,score attribute] + + |========================================================= + + === Mandatory Placement === + + Mandatory placement occurs when the constraint's score is + ++INFINITY+ or +-INFINITY+. In such cases, if the constraint can't be + satisfied, then the +rsc+ resource is not permitted to run. For + +score=INFINITY+, this includes cases where the +with-rsc+ resource is + not active. + + If you need resource +A+ to always run on the same machine as + resource +B+, you would add the following constraint: + + .Mandatory colocation constraint for two resources + ==== + [source,XML] + + ==== + + Remember, because +INFINITY+ was used, if +B+ can't run on any + of the cluster nodes (for whatever reason) then +A+ will not + be allowed to run. Whether +A+ is running or not has no effect on +B+. + + Alternatively, you may want the opposite -- that +A+ 'cannot' + run on the same machine as +B+. In this case, use + +score="-INFINITY"+. + + .Mandatory anti-colocation constraint for two resources + ==== + [source,XML] + + ==== + + Again, by specifying +-INFINITY+, the constraint is binding. So if the + only place left to run is where +B+ already is, then + +A+ may not run anywhere. + + As with +INFINITY+, +B+ can run even if +A+ is stopped. + However, in this case +A+ also can run if +B+ is stopped, because it still + meets the constraint of +A+ and +B+ not running on the same node. + + === Advisory Placement === + + If mandatory placement is about "must" and "must not", then advisory + placement is the "I'd prefer if" alternative. For constraints with + scores greater than +-INFINITY+ and less than +INFINITY+, the cluster + will try to accommodate your wishes but may ignore them if the + alternative is to stop some of the cluster resources. + + As in life, where if enough people prefer something it effectively + becomes mandatory, advisory colocation constraints can combine with + other elements of the configuration to behave as if they were + mandatory. + + .Advisory colocation constraint for two resources + ==== + [source,XML] + + ==== + + [[s-coloc-attribute]] + === Colocation by Node Attribute === + + The +node-attribute+ property of a colocation constraints allows you to express + the requirement, "these resources must be on similar nodes". + + As an example, imagine that you have two Storage Area Networks (SANs) that are + not controlled by the cluster, and each node is connected to one or the other. + You may have two resources +r1+ and +r2+ such that +r2+ needs to use the same + SAN as +r1+, but doesn't necessarily have to be on the same exact node. + In such a case, you could define a <> named + +san+, with the value +san1+ or +san2+ on each node as appropriate. Then, you + could colocate +r2+ with +r1+ using +node-attribute+ set to +san+. + + [[s-resource-sets]] + == Resource Sets == + + 'Resource sets' allow multiple resources to be affected by a single constraint. + indexterm:[Constraint,Resource Set] + indexterm:[Resource,Resource Set] + + .A set of 3 resources + ==== + [source,XML] + ---- + + + + + + ---- + ==== + + Resource sets are valid inside +rsc_location+, + +rsc_order+ (see <>), + +rsc_colocation+ (see <>), + and +rsc_ticket+ (see <>) constraints. + + A resource set has a number of properties that can be set, + though not all have an effect in all contexts. + + .Attributes of a resource_set Element + [width="95%",cols="2m,1,<5",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |id + | + |A unique name for the set + indexterm:[XML attribute,id attribute,resource_set element] + indexterm:[XML element,resource_set element,id attribute] + + |sequential + |true + |Whether the members of the set must be acted on in order. + Meaningful within +rsc_order+ and +rsc_colocation+. + indexterm:[XML attribute,sequential attribute,resource_set element] + indexterm:[XML element,resource_set element,sequential attribute] + + |require-all + |true + |Whether all members of the set must be active before continuing. + With the current implementation, the cluster may continue even if only one + member of the set is started, but if more than one member of the set is + starting at the same time, the cluster will still wait until all of those have + started before continuing (this may change in future versions). + Meaningful within +rsc_order+. + indexterm:[XML attribute,require-all attribute,resource_set element] + indexterm:[XML element,resource_set element,require-all attribute] + + |role + | + |Limit the effect of the constraint to the specified role. + Meaningful within +rsc_location+, +rsc_colocation+ and +rsc_ticket+. + indexterm:[XML attribute,role attribute,resource_set element] + indexterm:[XML element,resource_set element,role attribute] + + |action + | + |Limit the effect of the constraint to the specified action. + Meaningful within +rsc_order+. + indexterm:[XML attribute,action attribute,resource_set element] + indexterm:[XML element,resource_set element,action attribute] + + |score + | + |'Advanced use only.' Use a specific score for this set within the constraint. + indexterm:[XML attribute,score attribute,resource_set element] + indexterm:[XML element,resource_set element,score attribute] + + |========================================================= + + [[s-resource-sets-ordering]] + == Ordering Sets of Resources == + + A common situation is for an administrator to create a chain of + ordered resources, such as: + + .A chain of ordered resources + ====== + [source,XML] + ------- + + + + + + ------- + ====== + + .Visual representation of the four resources' start order for the above constraints + image::images/resource-set.png["Ordered set",width="16cm",height="2.5cm",align="center"] + + === Ordered Set === + + To simplify this situation, resource sets (see <>) can be used + within ordering constraints: + + .A chain of ordered resources expressed as a set + ====== + [source,XML] + ------- + + + + + + + + + + + ------- + ====== + + While the set-based format is not less verbose, it is significantly + easier to get right and maintain. + + [IMPORTANT] + ========= + If you use a higher-level tool, pay attention to how it exposes this + functionality. Depending on the tool, creating a set +A B+ may be equivalent to + +A then B+, or +B then A+. + ========= + + === Ordering Multiple Sets === + + The syntax can be expanded to allow sets of resources to be ordered relative to + each other, where the members of each individual set may be ordered or + unordered (controlled by the +sequential+ property). In the example below, +A+ + and +B+ can both start in parallel, as can +C+ and +D+, however +C+ and +D+ can + only start once _both_ +A+ _and_ +B+ are active. + + .Ordered sets of unordered resources + ====== + [source,XML] + ------- + + + + + + + + + + + + + ------- + ====== + + .Visual representation of the start order for two ordered sets of unordered resources + image::images/two-sets.png["Two ordered sets",width="13cm",height="7.5cm",align="center"] + + Of course either set -- or both sets -- of resources can also be + internally ordered (by setting +sequential="true"+) and there is no + limit to the number of sets that can be specified. + + .Advanced use of set ordering - Three ordered sets, two of which are internally unordered + ====== + [source,XML] + ------- + + + + + + + + + + + + + + + + + ------- + ====== + + .Visual representation of the start order for the three sets defined above + image::images/three-sets.png["Three ordered sets",width="16cm",height="7.5cm",align="center"] + + [IMPORTANT] + ==== + An ordered set with +sequential=false+ makes sense only if there is another + set in the constraint. Otherwise, the constraint has no effect. + ==== + + === Resource Set OR Logic === + + The unordered set logic discussed so far has all been "AND" logic. + To illustrate this take the 3 resource set figure in the previous section. + Those sets can be expressed, +(A and B) then \(C) then (D) then (E and F)+. + + Say for example we want to change the first set, +(A and B)+, to use "OR" logic + so the sets look like this: +(A or B) then \(C) then (D) then (E and F)+. + This functionality can be achieved through the use of the +require-all+ + option. This option defaults to TRUE which is why the + "AND" logic is used by default. Setting +require-all=false+ means only one + resource in the set needs to be started before continuing on to the next set. + + .Resource Set "OR" logic: Three ordered sets, where the first set is internally unordered with "OR" logic + ====== + [source,XML] + ------- + + + + + + + + + + + + + + + + + ------- + ====== + + [IMPORTANT] + ==== + An ordered set with +require-all=false+ makes sense only in conjunction with + +sequential=false+. Think of it like this: +sequential=false+ modifies the set + to be an unordered set using "AND" logic by default, and adding + +require-all=false+ flips the unordered set's "AND" logic to "OR" logic. + ==== + + [[s-resource-sets-colocation]] + == Colocating Sets of Resources == + + Another common situation is for an administrator to create a set of + colocated resources. + + The simplest way to do this is to define a resource group (see + <>), but that cannot always accurately express the desired + relationships. For example, maybe the resources do not need to be ordered. + + Another way would be to define each relationship as an individual constraint, + but that causes a difficult-to-follow constraint explosion as the number of + resources and combinations grow. + + .Colocation chain as individual constraints, where A is placed first, then B, then C, then D + ====== + [source,XML] + ------- + + + + + + ------- + ====== + + To express complicated relationships with a simplified syntax + footnote:[which is not the same as saying easy to follow], + <> can be used within colocation constraints. + + .Equivalent colocation chain expressed using +resource_set+ + ====== + [source,XML] + ------- + + + + + + + + + + + ------- + ====== + + [NOTE] + ==== + Within a +resource_set+, the resources are listed in the order they are + _placed_, which is the reverse of the order in which they are _colocated_. + In the above example, resource +A+ is placed before resource +B+, which is + the same as saying resource +B+ is colocated with resource +A+. + ==== + + As with individual constraints, a resource that can't be active prevents any + resource that must be colocated with it from being active. In both of the two + previous examples, if +B+ is unable to run, then both +C+ and by inference +D+ + must remain stopped. + + [IMPORTANT] + ========= + If you use a higher-level tool, pay attention to how it exposes this + functionality. Depending on the tool, creating a set +A B+ may be equivalent to + +A with B+, or +B with A+. + ========= + + Resource sets can also be used to tell the cluster that entire _sets_ of + resources must be colocated relative to each other, while the individual + members within any one set may or may not be colocated relative to each other + (determined by the set's +sequential+ property). + + In the following example, resources +B+, +C+, and +D+ will each be colocated + with +A+ (which will be placed first). +A+ must be able to run in order for any + of the resources to run, but any of +B+, +C+, or +D+ may be stopped without + affecting any of the others. + + .Using colocated sets to specify a shared dependency + ====== + [source,XML] + ------- + + + + + + + + + + + + + ------- + ====== + + [NOTE] + ==== + Pay close attention to the order in which resources and sets are listed. + While the members of any one sequential set are placed first to last (i.e., the + colocation dependency is last with first), multiple sets are placed last to + first (i.e. the colocation dependency is first with last). + ==== + + [IMPORTANT] + ==== + A colocated set with +sequential="false"+ makes sense only if there is + another set in the constraint. Otherwise, the constraint has no effect. + ==== + + There is no inherent limit to the number and size of the sets used. + The only thing that matters is that in order for any member of one set + in the constraint to be active, all members of sets listed after it must also + be active (and naturally on the same node); and if a set has +sequential="true"+, + then in order for one member of that set to be active, all members listed + before it must also be active. + + If desired, you can restrict the dependency to instances of promotable clone + resources that are in a specific role, using the set's +role+ property. + + .Colocation in which the members of the middle set have no interdependencies, and the last set listed applies only to instances in the master role + ====== + [source,XML] + ------- + + + + + + + + + + + + + + + + + + ------- + ====== + + .Visual representation of the above example (resources are placed from left to right) + image::images/pcmk-colocated-sets.png["Colocation chain",width="960px",height="431px",align="center"] + + [NOTE] + ==== + Unlike ordered sets, colocated sets do not use the +require-all+ option. + ==== diff --git a/doc/sphinx/Pacemaker_Explained/fencing.rst b/doc/sphinx/Pacemaker_Explained/fencing.rst new file mode 100644 index 0000000000..d9b8f21d72 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/fencing.rst @@ -0,0 +1,1026 @@ +Fencing +------- + +.. Convert_to_RST: + + anchor:ch-fencing[Chapter 6, Fencing] + indexterm:[Fencing, Configuration] + indexterm:[STONITH, Configuration] + + == What Is Fencing? == + + 'Fencing' is the ability to make a node unable to run resources, even when that + node is unresponsive to cluster commands. + + Fencing is also known as 'STONITH', an acronym for "Shoot The Other Node In The + Head", since the most common fencing method is cutting power to the node. + Another method is "fabric fencing", cutting the node's access to some + capability required to run resources (such as network access or a shared disk). + + == Why Is Fencing Necessary? == + + Fencing protects your data from being corrupted by malfunctioning nodes or + unintentional concurrent access to shared resources. + + Fencing protects against the "split brain" failure scenario, where cluster + nodes have lost the ability to reliably communicate with each other but are + still able to run resources. If the cluster just assumed that uncommunicative + nodes were down, then multiple instances of a resource could be started on + different nodes. + + The effect of split brain depends on the resource type. For example, an IP + address brought up on two hosts on a network will cause packets to randomly be + sent to one or the other host, rendering the IP useless. For a database or + clustered file system, the effect could be much more severe, causing data + corruption or divergence. + + Fencing also is used when a resource cannot otherwise be stopped. If a failed + resource fails to stop, it cannot be recovered elsewhere. Fencing the + resource's node is the only way to ensure the resource is recoverable. + + Users may also configure the +on-fail+ property of any resource operation to + +fencing+, in which case the cluster will fence the resource's node if the + operation fails. + + == Fence Devices == + + A 'fence device' (or 'fencing device') is a special type of resource that + provides the means to fence a node. + + Examples of fencing devices include intelligent power switches and IPMI devices + that accept SNMP commands to cut power to a node, and iSCSI controllers that + allow SCSI reservations to be used to cut a node's access to a shared disk. + + Since fencing devices will be used to recover from loss of networking + connectivity to other nodes, it is essential that they do not rely on the same + network as the cluster itself, otherwise that network becomes a single point of + failure. + + Since loss of a node due to power outage is indistinguishable from loss of + network connectivity to that node, it is also essential that at least one fence + device for a node does not share power with that node. For example, an on-board + IPMI controller that shares power with its host should not be used as the sole + fencing device for that host. + + Since fencing is used to isolate malfunctioning nodes, no fence device should + rely on its target functioning properly. This includes, for example, devices + that ssh into a node and issue a shutdown command (such devices might be + suitable for testing, but never for production). + + == Fence Agents == + + A 'fence agent' (or 'fencing agent') is a +stonith+-class resource agent. + + The fence agent standard provides commands (such as +off+ and +reboot+) that + the cluster can use to fence nodes. As with other resource agent classes, + this allows a layer of abstraction so that Pacemaker doesn't need any knowledge + about specific fencing technologies -- that knowledge is isolated in the agent. + + == When a Fence Device Can Be Used == + + Fencing devices do not actually "run" like most services. Typically, they just + provide an interface for sending commands to an external device. + + Additionally, fencing may be initiated by Pacemaker, by other cluster-aware software + such as DRBD or DLM, or manually by an administrator, at any point in the + cluster life cycle, including before any resources have been started. + + To accommodate this, Pacemaker does not require the fence device resource to be + "started" in order to be used. Whether a fence device is started or not + determines whether a node runs any recurring monitor for the device, and gives + the node a slight preference for being chosen to execute fencing using that + device. + + By default, any node can execute any fencing device. If a fence device is + disabled by setting its +target-role+ to Stopped, then no node can use that + device. If mandatory location constraints prevent a specific node from + "running" a fence device, then that node will never be chosen to execute + fencing using the device. A node may fence itself, but the cluster will choose + that only if no other nodes can do the fencing. + + A common configuration scenario is to have one fence device per target node. + In such a case, users often configure anti-location constraints so that + the target node does not monitor its own device. The best practice is to make + the constraint optional (i.e. a finite negative score rather than +-INFINITY+), + so that the node can fence itself if no other nodes can. + + == Limitations of Fencing Resources == + + Fencing resources have certain limitations that other resource classes don't: + + * They may have only one set of meta-attributes and one set of instance + attributes. + * If <> are used to determine fencing resource options, these + may only be evaluated when first read, meaning that later changes to the + rules will have no effect. Therefore, it is better to avoid confusion and not + use rules at all with fencing resources. + + These limitations could be revisited if there is significant user demand. + + == Special Options for Fencing Resources == + + The table below lists special instance attributes that may be set for any + fencing resource ('not' meta-attributes, even though they are interpreted by + pacemaker rather than the fence agent). These are also listed in the man page + for +pacemaker-fenced+. + + .Additional Properties of Fencing Resources + [width="95%",cols="8m,3,6,<12",options="header",align="center"] + |========================================================= + + |Field + |Type + |Default + |Description + + |stonith-timeout + |NA + |NA + a|Older versions used this to override the default period to wait for a STONITH (reboot, on, off) action to complete for this device. + It has been replaced by the +pcmk_reboot_timeout+ and +pcmk_off_timeout+ properties. + indexterm:[stonith-timeout,Fencing] + indexterm:[Fencing,Property,stonith-timeout] + + //// + (not yet implemented) + priority + integer + 0 + The priority of the STONITH resource. Devices are tried in order of highest priority to lowest. + indexterm priority,Fencing + indexterm Fencing,Property,priority + //// + + |provides + |string + | + |Any special capability provided by the fence device. Currently, only one such + capability is meaningful: +unfencing+ (see <>). + indexterm:[provides,Fencing] + indexterm:[Fencing,Property,provides] + + |pcmk_host_map + |string + | + |A mapping of host names to ports numbers for devices that do not support host names. + Example: +node1:1;node2:2,3+ tells the cluster to use port 1 for + *node1* and ports 2 and 3 for *node2*. If +pcmk_host_check+ is explicitly set + to +static-list+, either this or +pcmk_host_list+ must be set. + indexterm:[pcmk_host_map,Fencing] + indexterm:[Fencing,Property,pcmk_host_map] + + |pcmk_host_list + |string + | + |A list of machines controlled by this device. If +pcmk_host_check+ is + explicitly set to +static-list+, either this or +pcmk_host_map+ must be set. + indexterm:[pcmk_host_list,Fencing] + indexterm:[Fencing,Property,pcmk_host_list] + + |pcmk_host_check + |string + |A value appropriate to other configuration options and + device capabilities (see note below) + a|How to determine which machines are controlled by the device. + Allowed values: + + * +dynamic-list:+ query the device via the "list" command + * +static-list:+ check the +pcmk_host_list+ or +pcmk_host_map+ attribute + * +status:+ query the device via the "status" command + * +none:+ assume every device can fence every machine + + indexterm:[pcmk_host_check,Fencing] + indexterm:[Fencing,Property,pcmk_host_check] + + |pcmk_delay_max + |time + |0s + |Enable a random delay of up to the time specified before executing fencing + actions. This is sometimes used in two-node clusters to ensure that the + nodes don't fence each other at the same time. The overall delay introduced + by pacemaker is derived from this random delay value adding a static delay so + that the sum is kept below the maximum delay. + + indexterm:[pcmk_delay_max,Fencing] + indexterm:[Fencing,Property,pcmk_delay_max] + + |pcmk_delay_base + |time + |0s + |Enable a static delay before executing fencing actions. This can be used + e.g. in two-node clusters to ensure that the nodes don't fence each other, + by having separate fencing resources with different values. The node that is + fenced with the shorter delay will lose a fencing race. The overall delay + introduced by pacemaker is derived from this value plus a random delay such + that the sum is kept below the maximum delay. + + indexterm:[pcmk_delay_base,Fencing] + indexterm:[Fencing,Property,pcmk_delay_base] + + |pcmk_action_limit + |integer + |1 + |The maximum number of actions that can be performed in parallel on this + device, if the cluster option +concurrent-fencing+ is +true+. -1 is unlimited. + + indexterm:[pcmk_action_limit,Fencing] + indexterm:[Fencing,Property,pcmk_action_limit] + + |pcmk_host_argument + |string + |+port+ otherwise +plug+ if supported according to the metadata of the fence agent + |'Advanced use only.' Which parameter should be supplied to the fence agent to + identify the node to be fenced. Some devices support neither the standard +plug+ + nor the deprecated +port+ parameter, or may provide additional ones. Use this to + specify an alternate, device-specific parameter. A value of +none+ tells the + cluster not to supply any additional parameters. + indexterm:[pcmk_host_argument,Fencing] + indexterm:[Fencing,Property,pcmk_host_argument] + + |pcmk_reboot_action + |string + |reboot + |'Advanced use only.' The command to send to the resource agent in order to + reboot a node. Some devices do not support the standard commands or may provide + additional ones. Use this to specify an alternate, device-specific command. + indexterm:[pcmk_reboot_action,Fencing] + indexterm:[Fencing,Property,pcmk_reboot_action] + + |pcmk_reboot_timeout + |time + |60s + |'Advanced use only.' Specify an alternate timeout to use for `reboot` actions + instead of the value of +stonith-timeout+. Some devices need much more or less + time to complete than normal. Use this to specify an alternate, device-specific + timeout. + indexterm:[pcmk_reboot_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_reboot_timeout] + indexterm:[stonith-timeout,Fencing] + indexterm:[Fencing,Property,stonith-timeout] + + |pcmk_reboot_retries + |integer + |2 + |'Advanced use only.' The maximum number of times to retry the `reboot` command + within the timeout period. Some devices do not support multiple connections, and + operations may fail if the device is busy with another task, so Pacemaker will + automatically retry the operation, if there is time remaining. Use this option + to alter the number of times Pacemaker retries before giving up. + indexterm:[pcmk_reboot_retries,Fencing] + indexterm:[Fencing,Property,pcmk_reboot_retries] + + |pcmk_off_action + |string + |off + |'Advanced use only.' The command to send to the resource agent in order to + shut down a node. Some devices do not support the standard commands or may provide + additional ones. Use this to specify an alternate, device-specific command. + indexterm:[pcmk_off_action,Fencing] + indexterm:[Fencing,Property,pcmk_off_action] + + |pcmk_off_timeout + |time + |60s + |'Advanced use only.' Specify an alternate timeout to use for `off` actions + instead of the value of +stonith-timeout+. Some devices need much more or less + time to complete than normal. Use this to specify an alternate, device-specific + timeout. + indexterm:[pcmk_off_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_off_timeout] + indexterm:[stonith-timeout,Fencing] + indexterm:[Fencing,Property,stonith-timeout] + + |pcmk_off_retries + |integer + |2 + |'Advanced use only.' The maximum number of times to retry the `off` command + within the timeout period. Some devices do not support multiple connections, and + operations may fail if the device is busy with another task, so Pacemaker will + automatically retry the operation, if there is time remaining. Use this option + to alter the number of times Pacemaker retries before giving up. + indexterm:[pcmk_off_retries,Fencing] + indexterm:[Fencing,Property,pcmk_off_retries] + + |pcmk_list_action + |string + |list + |'Advanced use only.' The command to send to the resource agent in order to + list nodes. Some devices do not support the standard commands or may provide + additional ones. Use this to specify an alternate, device-specific command. + indexterm:[pcmk_list_action,Fencing] + indexterm:[Fencing,Property,pcmk_list_action] + + |pcmk_list_timeout + |time + |60s + |'Advanced use only.' Specify an alternate timeout to use for `list` actions + instead of the value of +stonith-timeout+. Some devices need much more or less + time to complete than normal. Use this to specify an alternate, device-specific + timeout. + indexterm:[pcmk_list_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_list_timeout] + + |pcmk_list_retries + |integer + |2 + |'Advanced use only.' The maximum number of times to retry the `list` command + within the timeout period. Some devices do not support multiple connections, and + operations may fail if the device is busy with another task, so Pacemaker will + automatically retry the operation, if there is time remaining. Use this option + to alter the number of times Pacemaker retries before giving up. + indexterm:[pcmk_list_retries,Fencing] + indexterm:[Fencing,Property,pcmk_list_retries] + + |pcmk_monitor_action + |string + |monitor + |'Advanced use only.' The command to send to the resource agent in order to + report extended status. Some devices do not support the standard commands or may provide + additional ones. Use this to specify an alternate, device-specific command. + indexterm:[pcmk_monitor_action,Fencing] + indexterm:[Fencing,Property,pcmk_monitor_action] + + |pcmk_monitor_timeout + |time + |60s + |'Advanced use only.' Specify an alternate timeout to use for `monitor` actions + instead of the value of +stonith-timeout+. Some devices need much more or less + time to complete than normal. Use this to specify an alternate, device-specific + timeout. + indexterm:[pcmk_monitor_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_monitor_timeout] + + |pcmk_monitor_retries + |integer + |2 + |'Advanced use only.' The maximum number of times to retry the `monitor` command + within the timeout period. Some devices do not support multiple connections, and + operations may fail if the device is busy with another task, so Pacemaker will + automatically retry the operation, if there is time remaining. Use this option + to alter the number of times Pacemaker retries before giving up. + indexterm:[pcmk_monitor_retries,Fencing] + indexterm:[Fencing,Property,pcmk_monitor_retries] + + |pcmk_status_action + |string + |status + |'Advanced use only.' The command to send to the resource agent in order to + report status. Some devices do not support the standard commands or may provide + additional ones. Use this to specify an alternate, device-specific command. + indexterm:[pcmk_status_action,Fencing] + indexterm:[Fencing,Property,pcmk_status_action] + + |pcmk_status_timeout + |time + |60s + |'Advanced use only.' Specify an alternate timeout to use for `status` actions + instead of the value of +stonith-timeout+. Some devices need much more or less + time to complete than normal. Use this to specify an alternate, device-specific + timeout. + indexterm:[pcmk_status_timeout,Fencing] + indexterm:[Fencing,Property,pcmk_status_timeout] + + |pcmk_status_retries + |integer + |2 + |'Advanced use only.' The maximum number of times to retry the `status` command + within the timeout period. Some devices do not support multiple connections, and + operations may fail if the device is busy with another task, so Pacemaker will + automatically retry the operation, if there is time remaining. Use this option + to alter the number of times Pacemaker retries before giving up. + indexterm:[pcmk_status_retries,Fencing] + indexterm:[Fencing,Property,pcmk_status_retries] + + |========================================================= + + [NOTE] + ==== + The default value for +pcmk_host_check+ is +static-list+ if either + +pcmk_host_list+ or +pcmk_host_map+ is configured. If neither of those are + configured, the default is +dynamic-list+ if the fence device supports the list + action, or +status+ if the fence device supports the status action but not the + list action. If none of those conditions apply, the default is +none+. + ==== + + [[s-unfencing]] + == Unfencing == + + With fabric fencing (such as cutting network or shared disk access rather than + power), it is expected that the cluster will fence the node, and + then a system administrator must manually investigate what went wrong, correct + any issues found, then reboot (or restart the cluster services on) the node. + + Once the node reboots and rejoins the cluster, some fabric fencing devices + require an explicit command to restore the node's access. This capability is + called 'unfencing' and is typically implemented as the fence agent's +on+ + command. + + If any cluster resource has +requires+ set to +unfencing+, then that resource + will not be probed or started on a node until that node has been unfenced. + + == Fence Devices Dependent on Other Resources == + + In some cases, a fence device may require some other cluster resource (such as + an IP address) to be active in order to function properly. + + This is obviously undesirable in general: fencing may be required when the + depended-on resource is not active, or fencing may be required because the node + running the depended-on resource is no longer responding. + + However, this may be acceptable under certain conditions: + + * The dependent fence device should not be able to target any node that is + allowed to run the depended-on resource. + + * The depended-on resource should not be disabled during production operation. + + * The +concurrent-fencing+ cluster property should be set to +true+. Otherwise, + if both the node running the depended-on resource and some node targeted by + the dependent fence device need to be fenced, the fencing of the node + running the depended-on resource might be ordered first, making the second + fencing impossible and blocking further recovery. With concurrent fencing, + the dependent fence device might fail at first due to the depended-on + resource being unavailable, but it will be retried and eventually succeed + once the resource is brought back up. + + Even under those conditions, there is one unlikely problem scenario. The DC + always schedules fencing of itself after any other fencing needed, to avoid + unnecessary repeated DC elections. If the dependent fence device targets the + DC, and both the DC and a different node running the depended-on resource need + to be fenced, the DC fencing will always fail and block further recovery. Note, + however, that losing a DC node entirely causes some other node to become DC and + schedule the fencing, so this is only a risk when a stop or other operation + with +on-fail+ set to +fencing+ fails on the DC. + + == Configuring Fencing == + + . Find the correct driver: + + + ---- + # stonith_admin --list-installed + ---- + + . Find the required parameters associated with the device + (replacing $AGENT_NAME with the name obtained from the previous step): + + + ---- + # stonith_admin --metadata --agent $AGENT_NAME + ---- + + . Create a file called +stonith.xml+ containing a primitive resource + with a class of +stonith+, a type equal to the agent name obtained earlier, + and a parameter for each of the values returned in the previous step. + + . If the device does not know how to fence nodes based on their uname, + you may also need to set the special +pcmk_host_map+ parameter. See + `man pacemaker-fenced` for details. + + . If the device does not support the `list` command, you may also need + to set the special +pcmk_host_list+ and/or +pcmk_host_check+ + parameters. See `man pacemaker-fenced` for details. + + . If the device does not expect the victim to be specified with the + `port` parameter, you may also need to set the special + +pcmk_host_argument+ parameter. See `man pacemaker-fenced` for details. + + . Upload it into the CIB using cibadmin: + + + ---- + # cibadmin -C -o resources --xml-file stonith.xml + ---- + + . Set +stonith-enabled+ to true: + + + ---- + # crm_attribute -t crm_config -n stonith-enabled -v true + ---- + + . Once the stonith resource is running, you can test it by executing the + following (although you might want to stop the cluster on that machine + first): + + + ---- + # stonith_admin --reboot nodename + ---- + + === Example Fencing Configuration === + + Assume we have a chassis containing four nodes and an IPMI device + active on 192.0.2.1. We would choose the `fence_ipmilan` driver, + and obtain the following list of parameters: + + .Obtaining a list of Fence Agent Parameters + ==== + ---- + # stonith_admin --metadata -a fence_ipmilan + ---- + + [source,XML] + ---- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ---- + ==== + + Based on that, we would create a fencing resource fragment that might look + like this: + + .An IPMI-based Fencing Resource + ==== + [source,XML] + ---- + + + + + + + + + + + + ---- + ==== + + Finally, we need to enable fencing: + ---- + # crm_attribute -t crm_config -n stonith-enabled -v true + ---- + + == Fencing Topologies == + + Pacemaker supports fencing nodes with multiple devices through a feature called + 'fencing topologies'. Fencing topologies may be used to provide alternative + devices in case one fails, or to require multiple devices to all be executed + successfully in order to consider the node successfully fenced, or even a + combination of the two. + + Create the individual devices as you normally would, then define one or more + +fencing-level+ entries in the +fencing-topology+ section of the configuration. + + * Each fencing level is attempted in order of ascending +index+. Allowed + values are 1 through 9. + * If a device fails, processing terminates for the current level. + No further devices in that level are exercised, and the next level is attempted instead. + * If the operation succeeds for all the listed devices in a level, the level is deemed to have passed. + * The operation is finished when a level has passed (success), or all levels have been attempted (failed). + * If the operation failed, the next step is determined by the scheduler + and/or the controller. + + Some possible uses of topologies include: + + * Try on-board IPMI, then an intelligent power switch if that fails + * Try fabric fencing of both disk and network, then fall back to power fencing + if either fails + * Wait up to a certain time for a kernel dump to complete, then cut power to + the node + + .Properties of Fencing Levels + [width="95%",cols="1m,<3",options="header",align="center"] + |========================================================= + + |Field + |Description + + |id + |A unique name for the level + indexterm:[id,fencing-level] + indexterm:[Fencing,fencing-level,id] + + |target + |The name of a single node to which this level applies + indexterm:[target,fencing-level] + indexterm:[Fencing,fencing-level,target] + + |target-pattern + |An extended regular expression (as defined in + http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04[POSIX]) + matching the names of nodes to which this level applies + indexterm:[target-pattern,fencing-level] + indexterm:[Fencing,fencing-level,target-pattern] + + |target-attribute + |The name of a node attribute that is set (to +target-value+) for nodes to + which this level applies + indexterm:[target-attribute,fencing-level] + indexterm:[Fencing,fencing-level,target-attribute] + + |target-value + |The node attribute value (of +target-attribute+) that is set for nodes to + which this level applies + indexterm:[target-attribute,fencing-level] + indexterm:[Fencing,fencing-level,target-attribute] + + |index + |The order in which to attempt the levels. + Levels are attempted in ascending order 'until one succeeds'. + Valid values are 1 through 9. + indexterm:[index,fencing-level] + indexterm:[Fencing,fencing-level,index] + + |devices + |A comma-separated list of devices that must all be tried for this level + indexterm:[devices,fencing-level] + indexterm:[Fencing,fencing-level,devices] + + |========================================================= + + .Fencing topology with different devices for different nodes + ==== + [source,XML] + ---- + + + ... + + + + + + + + + + ... + + + + ---- + ==== + + === Example Dual-Layer, Dual-Device Fencing Topologies === + + The following example illustrates an advanced use of +fencing-topology+ in a cluster with the following properties: + + * 3 nodes (2 active prod-mysql nodes, 1 prod_mysql-rep in standby for quorum purposes) + * the active nodes have an IPMI-controlled power board reached at 192.0.2.1 and 192.0.2.2 + * the active nodes also have two independent PSUs (Power Supply Units) + connected to two independent PDUs (Power Distribution Units) reached at + 198.51.100.1 (port 10 and port 11) and 203.0.113.1 (port 10 and port 11) + * the first fencing method uses the `fence_ipmi` agent + * the second fencing method uses the `fence_apc_snmp` agent targetting 2 fencing devices (one per PSU, either port 10 or 11) + * fencing is only implemented for the active nodes and has location constraints + * fencing topology is set to try IPMI fencing first then default to a "sure-kill" dual PDU fencing + + In a normal failure scenario, STONITH will first select +fence_ipmi+ to try to kill the faulty node. + Using a fencing topology, if that first method fails, STONITH will then move on to selecting +fence_apc_snmp+ twice: + + * once for the first PDU + * again for the second PDU + + The fence action is considered successful only if both PDUs report the required status. If any of them fails, STONITH loops back to the first fencing method, +fence_ipmi+, and so on until the node is fenced or fencing action is cancelled. + + .First fencing method: single IPMI device + + Each cluster node has it own dedicated IPMI channel that can be called for fencing using the following primitives: + [source,XML] + ---- + + + + + + + + + + + + + + + + + + + + + + + ---- + + .Second fencing method: dual PDU devices + + Each cluster node also has two distinct power channels controlled by two + distinct PDUs. That means a total of 4 fencing devices configured as follows: + + - Node 1, PDU 1, PSU 1 @ port 10 + - Node 1, PDU 2, PSU 2 @ port 10 + - Node 2, PDU 1, PSU 1 @ port 11 + - Node 2, PDU 2, PSU 2 @ port 11 + + The matching fencing agents are configured as follows: + [source,XML] + ---- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ---- + + .Location Constraints + + To prevent STONITH from trying to run a fencing agent on the same node it is + supposed to fence, constraints are placed on all the fencing primitives: + [source,XML] + ---- + + + + + + + + + ---- + + .Fencing topology + + Now that all the fencing resources are defined, it's time to create the right topology. + We want to first fence using IPMI and if that does not work, fence both PDUs to effectively and surely kill the node. + [source,XML] + ---- + + + + + + + ---- + Please note, in +fencing-topology+, the lowest +index+ value determines the priority of the first fencing method. + + .Final configuration + + Put together, the configuration looks like this: + [source,XML] + ---- + + + + + + + + ... + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ... + + + ---- + + == Remapping Reboots == + + When the cluster needs to reboot a node, whether because +stonith-action+ is +reboot+ or because + a reboot was manually requested (such as by `stonith_admin --reboot`), it will remap that to + other commands in two cases: + + . If the chosen fencing device does not support the +reboot+ command, the cluster + will ask it to perform +off+ instead. + + . If a fencing topology level with multiple devices must be executed, the cluster + will ask all the devices to perform +off+, then ask the devices to perform +on+. + + To understand the second case, consider the example of a node with redundant + power supplies connected to intelligent power switches. Rebooting one switch + and then the other would have no effect on the node. Turning both switches off, + and then on, actually reboots the node. + + In such a case, the fencing operation will be treated as successful as long as + the +off+ commands succeed, because then it is safe for the cluster to recover + any resources that were on the node. Timeouts and errors in the +on+ phase will + be logged but ignored. + + When a reboot operation is remapped, any action-specific timeout for the + remapped action will be used (for example, +pcmk_off_timeout+ will be used when + executing the +off+ command, not +pcmk_reboot_timeout+). diff --git a/doc/sphinx/Pacemaker_Explained/index.rst b/doc/sphinx/Pacemaker_Explained/index.rst new file mode 100644 index 0000000000..de2ddd9434 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/index.rst @@ -0,0 +1,41 @@ +Pacemaker Explained +=================== + +*Configuring Pacemaker Clusters* + + +Abstract +-------- +This document definitively explains Pacemaker's features and capabilities, +particularly the XML syntax used in Pacemaker's Cluster Information Base (CIB). + + +Table of Contents +----------------- + +.. toctree:: + :maxdepth: 3 + :numbered: + + intro + options + nodes + resources + constraints + fencing + alerts + rules + advanced-options + advanced-resources + reusing-configuration + utilization + acls + status + multi-site-clusters + ap-samples + +Index +----- + +* :ref:`genindex` +* :ref:`search` diff --git a/doc/sphinx/Pacemaker_Explained/intro.rst b/doc/sphinx/Pacemaker_Explained/intro.rst new file mode 100644 index 0000000000..3c3805b931 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/intro.rst @@ -0,0 +1,22 @@ +Introduction +------------ + +The Scope of this Document +########################## + +This document is intended to be an exhaustive reference for configuring +Pacemaker. To achieve this, it focuses on the XML syntax used to configure the +CIB. + +For those that are allergic to XML, multiple higher-level front-ends +(both command-line and GUI) are available. These tools will not be covered +in this document, though the concepts explained here should make the +functionality of these tools more easily understood. + +Users may be interested in other parts of the +`Pacemaker documentation set `_, +such as 'Clusters from Scratch', a step-by-step guide to setting up an +example cluster, and 'Pacemaker Administration', a guide to maintaining a +cluster. + +.. include:: ../shared/pacemaker-intro.rst diff --git a/doc/sphinx/Pacemaker_Explained/multi-site-clusters.rst b/doc/sphinx/Pacemaker_Explained/multi-site-clusters.rst new file mode 100644 index 0000000000..5a7554c4b9 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/multi-site-clusters.rst @@ -0,0 +1,342 @@ +Multi-Site Clusters and Tickets +------------------------------- + +.. Convert_to_RST: + + Apart from local clusters, Pacemaker also supports multi-site clusters. + That means you can have multiple, geographically dispersed sites, each with a + local cluster. Failover between these clusters can be coordinated + manually by the administrator, or automatically by a higher-level entity called + a 'Cluster Ticket Registry (CTR)'. + + == Challenges for Multi-Site Clusters == + + Typically, multi-site environments are too far apart to support + synchronous communication and data replication between the sites. + That leads to significant challenges: + + - How do we make sure that a cluster site is up and running? + + - How do we make sure that resources are only started once? + + - How do we make sure that quorum can be reached between the different + sites and a split-brain scenario avoided? + + - How do we manage failover between sites? + + - How do we deal with high latency in case of resources that need to be + stopped? + + In the following sections, learn how to meet these challenges. + + == Conceptual Overview == + + Multi-site clusters can be considered as “overlay” clusters where + each cluster site corresponds to a cluster node in a traditional cluster. + The overlay cluster can be managed by a CTR in order to + guarantee that any cluster resource will be active + on no more than one cluster site. This is achieved by using + 'tickets' that are treated as failover domain between cluster + sites, in case a site should be down. + + The following sections explain the individual components and mechanisms + that were introduced for multi-site clusters in more detail. + + === Ticket === + + Tickets are, essentially, cluster-wide attributes. A ticket grants the + right to run certain resources on a specific cluster site. Resources can + be bound to a certain ticket by +rsc_ticket+ constraints. Only if the + ticket is available at a site can the respective resources be started there. + Vice versa, if the ticket is revoked, the resources depending on that + ticket must be stopped. + + The ticket thus is similar to a 'site quorum', i.e. the permission to + manage/own resources associated with that site. (One can also think of the + current +have-quorum+ flag as a special, cluster-wide ticket that is granted in + case of node majority.) + + Tickets can be granted and revoked either manually by administrators + (which could be the default for classic enterprise clusters), or via + the automated CTR mechanism described below. + + A ticket can only be owned by one site at a time. Initially, none + of the sites has a ticket. Each ticket must be granted once by the cluster + administrator. + + The presence or absence of tickets for a site is stored in the CIB as a + cluster status. With regards to a certain ticket, there are only two states + for a site: +true+ (the site has the ticket) or +false+ (the site does + not have the ticket). The absence of a certain ticket (during the initial + state of the multi-site cluster) is the same as the value +false+. + + === Dead Man Dependency === + + A site can only activate resources safely if it can be sure that the + other site has deactivated them. However after a ticket is revoked, it can + take a long time until all resources depending on that ticket are stopped + "cleanly", especially in case of cascaded resources. To cut that process + short, the concept of a 'Dead Man Dependency' was introduced. + + If a dead man dependency is in force, if a ticket is revoked from a site, the + nodes that are hosting dependent resources are fenced. This considerably speeds + up the recovery process of the cluster and makes sure that resources can be + migrated more quickly. + + This can be configured by specifying a +loss-policy="fence"+ in + +rsc_ticket+ constraints. + + === Cluster Ticket Registry === + + A CTR is a coordinated group of network daemons that automatically handles + granting, revoking, and timing out tickets (instead of the administrator + revoking the ticket somewhere, waiting for everything to stop, and then + granting it on the desired site). + + Pacemaker does not implement its own CTR, but interoperates with external + software designed for that purpose (similar to how resource and fencing agents + are not directly part of pacemaker). + + Participating clusters run the CTR daemons, which connect to each other, exchange + information about their connectivity, and vote on which sites gets which + tickets. + + A ticket is granted to a site only once the CTR is sure that the ticket + has been relinquished by the previous owner, implemented via a timer in most + scenarios. If a site loses connection to its peers, its tickets time out and + recovery occurs. After the connection timeout plus the recovery timeout has + passed, the other sites are allowed to re-acquire the ticket and start the + resources again. + + This can also be thought of as a "quorum server", except that it is not + a single quorum ticket, but several. + + === Configuration Replication === + + As usual, the CIB is synchronized within each cluster, but it is 'not' synchronized + across cluster sites of a multi-site cluster. You have to configure the resources + that will be highly available across the multi-site cluster for every site + accordingly. + + + [[s-ticket-constraints]] + == Configuring Ticket Dependencies == + + The `rsc_ticket` constraint lets you specify the resources depending on a certain + ticket. Together with the constraint, you can set a `loss-policy` that defines + what should happen to the respective resources if the ticket is revoked. + + The attribute `loss-policy` can have the following values: + + * +fence:+ Fence the nodes that are running the relevant resources. + + * +stop:+ Stop the relevant resources. + + * +freeze:+ Do nothing to the relevant resources. + + * +demote:+ Demote relevant resources that are running in master mode to slave mode. + + + .Constraint that fences node if +ticketA+ is revoked + ==== + [source,XML] + ------- + + ------- + ==== + + The example above creates a constraint with the ID +rsc1-req-ticketA+. It + defines that the resource +rsc1+ depends on +ticketA+ and that the node running + the resource should be fenced if +ticketA+ is revoked. + + If resource +rsc1+ were a promotable resource (i.e. it could run in master or + slave mode), you might want to configure that only master mode + depends on +ticketA+. With the following configuration, +rsc1+ will be + demoted to slave mode if +ticketA+ is revoked: + + .Constraint that demotes +rsc1+ if +ticketA+ is revoked + ==== + [source,XML] + ------- + + ------- + ==== + + You can create multiple `rsc_ticket` constraints to let multiple resources + depend on the same ticket. However, `rsc_ticket` also supports resource sets + (see <>), + so one can easily list all the resources in one `rsc_ticket` constraint instead. + + .Ticket constraint for multiple resources + ==== + [source,XML] + ------- + + + + + + + + + + + ------- + ==== + + In the example above, there are two resource sets, so we can list resources + with different roles in a single +rsc_ticket+ constraint. There's no dependency + between the two resource sets, and there's no dependency among the + resources within a resource set. Each of the resources just depends on + +ticketA+. + + Referencing resource templates in +rsc_ticket+ constraints, and even + referencing them within resource sets, is also supported. + + If you want other resources to depend on further tickets, create as many + constraints as necessary with +rsc_ticket+. + + + == Managing Multi-Site Clusters == + + === Granting and Revoking Tickets Manually === + + You can grant tickets to sites or revoke them from sites manually. + If you want to re-distribute a ticket, you should wait for + the dependent resources to stop cleanly at the previous site before you + grant the ticket to the new site. + + Use the `crm_ticket` command line tool to grant and revoke tickets. + + //// + These commands will actually just print a message telling the user that they + require '--force'. That is probably a good exercise rather than letting novice + users cut and paste '--force' here. + //// + + To grant a ticket to this site: + ------- + # crm_ticket --ticket ticketA --grant + ------- + + To revoke a ticket from this site: + ------- + # crm_ticket --ticket ticketA --revoke + ------- + + [IMPORTANT] + ==== + If you are managing tickets manually, use the `crm_ticket` command with + great care, because it cannot check whether the same ticket is already + granted elsewhere. + ==== + + + === Granting and Revoking Tickets via a Cluster Ticket Registry === + + We will use https://github.com/ClusterLabs/booth[Booth] here as an example of + software that can be used with pacemaker as a Cluster Ticket Registry. Booth + implements the + http://en.wikipedia.org/wiki/Raft_%28computer_science%29[Raft] + algorithm to guarantee the distributed consensus among different + cluster sites, and manages the ticket distribution (and thus the failover + process between sites). + + Each of the participating clusters and 'arbitrators' runs the Booth daemon + `boothd`. + + An 'arbitrator' is the multi-site equivalent of a quorum-only node in a local + cluster. If you have a setup with an even number of sites, + you need an additional instance to reach consensus about decisions such + as failover of resources across sites. In this case, add one or more + arbitrators running at additional sites. Arbitrators are single machines + that run a booth instance in a special mode. An arbitrator is especially + important for a two-site scenario, otherwise there is no way for one site + to distinguish between a network failure between it and the other site, and + a failure of the other site. + + The most common multi-site scenario is probably a multi-site cluster with two + sites and a single arbitrator on a third site. However, technically, there are + no limitations with regards to the number of sites and the number of + arbitrators involved. + + `Boothd` at each site connects to its peers running at the other sites and + exchanges connectivity details. Once a ticket is granted to a site, the + booth mechanism will manage the ticket automatically: If the site which + holds the ticket is out of service, the booth daemons will vote which + of the other sites will get the ticket. To protect against brief + connection failures, sites that lose the vote (either explicitly or + implicitly by being disconnected from the voting body) need to + relinquish the ticket after a time-out. Thus, it is made sure that a + ticket will only be re-distributed after it has been relinquished by the + previous site. The resources that depend on that ticket will fail over + to the new site holding the ticket. The nodes that have run the + resources before will be treated according to the `loss-policy` you set + within the `rsc_ticket` constraint. + + Before the booth can manage a certain ticket within the multi-site cluster, + you initially need to grant it to a site manually via the `booth` command-line + tool. After you have initially granted a ticket to a site, `boothd` + will take over and manage the ticket automatically. + + [IMPORTANT] + ==== + The `booth` command-line tool can be used to grant, list, or + revoke tickets and can be run on any machine where `boothd` is running. + If you are managing tickets via Booth, use only `booth` for manual + intervention, not `crm_ticket`. That ensures the same ticket + will only be owned by one cluster site at a time. + ==== + + ==== Booth Requirements ==== + + * All clusters that will be part of the multi-site cluster must be based on + Pacemaker. + + * Booth must be installed on all cluster nodes and on all arbitrators that will + be part of the multi-site cluster. + + * Nodes belonging to the same cluster site should be synchronized via NTP. However, + time synchronization is not required between the individual cluster sites. + + === General Management of Tickets === + + Display the information of tickets: + ------- + # crm_ticket --info + ------- + + Or you can monitor them with: + ------- + # crm_mon --tickets + ------- + + Display the +rsc_ticket+ constraints that apply to a ticket: + ------- + # crm_ticket --ticket ticketA --constraints + ------- + + When you want to do maintenance or manual switch-over of a ticket, + revoking the ticket would trigger the loss policies. If + +loss-policy="fence"+, the dependent resources could not be gracefully + stopped/demoted, and other unrelated resources could even be affected. + + The proper way is making the ticket 'standby' first with: + ------- + # crm_ticket --ticket ticketA --standby + ------- + + Then the dependent resources will be stopped or demoted gracefully without + triggering the loss policies. + + If you have finished the maintenance and want to activate the ticket again, + you can run: + ------- + # crm_ticket --ticket ticketA --activate + ------- + + == For more information == + + * https://www.suse.com/documentation/sle-ha-geo-12/art_ha_geo_quick/data/art_ha_geo_quick.html[SUSE's Geo Clustering quick start] + + * https://github.com/ClusterLabs/booth[Booth] diff --git a/doc/sphinx/Pacemaker_Explained/nodes.rst b/doc/sphinx/Pacemaker_Explained/nodes.rst new file mode 100644 index 0000000000..994f54d4ea --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/nodes.rst @@ -0,0 +1,208 @@ +Cluster Nodes +------------- + +.. Convert_to_RST: + + == Defining a Cluster Node == + + Each node in the cluster will have an entry in the nodes section + containing its UUID, uname, and type. + + .Example Corosync cluster node entry + ====== + [source,XML] + + ====== + + In normal circumstances, the admin should let the cluster populate + this information automatically from the communications and membership + data. + + [[s-node-name]] + == Where Pacemaker Gets the Node Name == + + Traditionally, Pacemaker required nodes to be referred to by the value + returned by `uname -n`. This can be problematic for services that + require the `uname -n` to be a specific value (e.g. for a licence + file). + + This requirement has been relaxed for clusters using Corosync 2.0 or later. + The name Pacemaker uses is: + + . The value stored in +corosync.conf+ under *ring0_addr* in the *nodelist*, if it does not contain an IP address; otherwise + . The value stored in +corosync.conf+ under *name* in the *nodelist*; otherwise + . The value of `uname -n` + + Pacemaker provides the `crm_node -n` command which displays the name + used by a running cluster. + + If a Corosync *nodelist* is used, `crm_node --name-for-id` pass:[number] is also + available to display the name used by the node with the corosync + *nodeid* of pass:[number], for example: `crm_node --name-for-id 2`. + + [[s-node-attributes]] + == Node Attributes == + + indexterm:[Node,attribute] + Pacemaker allows node-specific values to be specified using 'node attributes'. + A node attribute has a name, and may have a distinct value for each node. + + While certain node attributes have specific meanings to the cluster, they are + mainly intended to allow administrators and resource agents to track any + information desired. + + For example, an administrator might choose to define node attributes for how + much RAM and disk space each node has, which OS each uses, or which server room + rack each node is in. + + Users can configure <> that use node attributes to affect + where resources are placed. + + === Setting and querying node attributes === + + Node attributes can be set and queried using the `crm_attribute` and + `attrd_updater` commands, so that the user does not have to deal with XML + configuration directly. + + Here is an example of what XML configuration would be generated if an + administrator ran this command: + + .Result of using crm_attribute to specify which kernel pcmk-1 is running + ====== + ------- + # crm_attribute --type nodes --node pcmk-1 --name kernel --update $(uname -r) + ------- + [source,XML] + ------- + + + + + + ------- + ====== + + To read back the value that was just set: + ---- + # crm_attribute --type nodes --node pcmk-1 --name kernel --query + scope=nodes name=kernel value=3.10.0-862.14.4.el7.x86_64 + ---- + + By specifying `--type nodes` the admin tells the cluster that this + attribute is persistent across reboots. There are also transient attributes + which are kept in the status section and are "forgotten" whenever the node + leaves the cluster. Administrators can use this section by specifying + `--type status`. + + === Special node attributes === + + Certain node attributes have special meaning to the cluster. + + Node attribute names beginning with # are considered reserved for these + special attributes. Some special attributes do not start with #, for + historical reasons. + + Certain special attributes are set automatically by the cluster, should never + be modified directly, and can be used only within <>; + these are listed under <>. + + For true/false values, the cluster considers a value of "1", "y", "yes", "on", + or "true" (case-insensitively) to be true, "0", "n", "no", "off", "false", or + unset to be false, and anything else to be an error. + + .Node attributes with special significance + [width="95%",cols="2m,<5",options="header",align="center"] + |==== + |Name |Description + + | fail-count-* + | Attributes whose names start with +fail-count-+ are managed by the cluster + to track how many times particular resource operations have failed on this + node. These should be queried and cleared via the `crm_failcount` or + `crm_resource --cleanup` commands rather than directly. + indexterm:[Node,attribute,fail-count-] + indexterm:[fail-count-,Node attribute] + + | last-failure-* + | Attributes whose names start with +last-failure-+ are managed by the cluster + to track when particular resource operations have most recently failed on + this node. These should be cleared via the `crm_failcount` or + `crm_resource --cleanup` commands rather than directly. + indexterm:[Node,attribute,last-failure-] + indexterm:[last-failure-,Node attribute] + + | maintenance + | Similar to the +maintenance-mode+ <>, but for + a single node. If true, resources will not be started or stopped on the node, + resources and individual clone instances running on the node will become + unmanaged, and any recurring operations for those will be cancelled. + indexterm:[Node,attribute,maintenance] + indexterm:[maintenance,Node attribute] + + | probe_complete + | This is managed by the cluster to detect when nodes need to be reprobed, and + should never be used directly. + indexterm:[Node,attribute,probe_complete] + indexterm:[probe_complete,Node attribute] + + | resource-discovery-enabled + | If the node is a remote node, fencing is enabled, and this attribute is + explicitly set to false (unset means true in this case), resource discovery + (probes) will not be done on this node. This is highly discouraged; the + +resource-discovery+ location constraint property is preferred for this + purpose. + indexterm:[Node,attribute,resource-discovery-enabled] + indexterm:[resource-discovery-enabled,Node attribute] + + | shutdown + | This is managed by the cluster to orchestrate the shutdown of a node, + and should never be used directly. + indexterm:[Node,attribute,shutdown] + indexterm:[shutdown,Node attribute] + + | site-name + | If set, this will be used as the value of the +#site-name+ node attribute + used in rules. (If not set, the value of the +cluster-name+ cluster option + will be used as +#site-name+ instead.) + indexterm:[Node,attribute,site-name] + indexterm:[site-name,Node attribute] + + | standby + | If true, the node is in standby mode. This is typically set and queried via + the `crm_standby` command rather than directly. + indexterm:[Node,attribute,standby] + indexterm:[standby,Node attribute] + + | terminate + | If the value is true or begins with any nonzero number, the node will be + fenced. This is typically set by tools rather than directly. + indexterm:[Node,attribute,terminate] + indexterm:[terminate,Node attribute] + + | #digests-* + | Attributes whose names start with +#digests-+ are managed by the cluster to + detect when <> needs to be redone, and should never be + used directly. + indexterm:[Node,attribute,#digests-] + indexterm:[#digests-,Node attribute] + + | #node-unfenced + | When the node was last unfenced (as seconds since the epoch). This is managed + by the cluster and should never be used directly. + indexterm:[Node,attribute,#node-unfenced] + indexterm:[#node-unfenced,Node attribute] + + |==== + + [WARNING] + ==== + Restarting pacemaker on a node that is in single-node maintenance mode will + likely lead to undesirable effects. If +maintenance+ is set as a transient + attribute, it will be erased when pacemaker is stopped, which will immediately + take the node out of maintenance mode and likely get it fenced. Even if + permanent, if pacemaker is restarted, any resources active on the node will + have their local history erased when the node rejoins, so the cluster will no + longer consider them running on the node and thus will consider them managed + again, leading them to be started elsewhere. This behavior might be improved + in a future release. + ==== diff --git a/doc/sphinx/Pacemaker_Explained/options.rst b/doc/sphinx/Pacemaker_Explained/options.rst new file mode 100644 index 0000000000..387ace8845 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/options.rst @@ -0,0 +1,486 @@ +Cluster-Wide Configuration +-------------------------- + +.. Convert_to_RST: + + == Configuration Layout == + + The cluster is defined by the Cluster Information Base (CIB), + which uses XML notation. The simplest CIB, an empty one, looks like this: + + .An empty configuration + ====== + [source,XML] + ------- + + + + + + + + + + ------- + ====== + + The empty configuration above contains the major sections that make up a CIB: + + * +cib+: The entire CIB is enclosed with a +cib+ tag. Certain fundamental settings + are defined as attributes of this tag. + + ** +configuration+: This section -- the primary focus of this document -- + contains traditional configuration information such as what resources the + cluster serves and the relationships among them. + + *** +crm_config+: cluster-wide configuration options + *** +nodes+: the machines that host the cluster + *** +resources+: the services run by the cluster + *** +constraints+: indications of how resources should be placed + + ** +status+: This section contains the history of each resource on each node. + Based on this data, the cluster can construct the complete current + state of the cluster. The authoritative source for this section + is the local executor (pacemaker-execd process) on each cluster node, and + the cluster will occasionally repopulate the entire section. For this + reason, it is never written to disk, and administrators are advised + against modifying it in any way. + + In this document, configuration settings will be described as 'properties' or 'options' + based on how they are defined in the CIB: + + * Properties are XML attributes of an XML element. + * Options are name-value pairs expressed as +nvpair+ child elements of an XML element. + + Normally, you will use command-line tools that abstract the XML, so the + distinction will be unimportant; both properties and options are + cluster settings you can tweak. + + == CIB Properties == + + Certain settings are defined by CIB properties (that is, attributes of the + +cib+ tag) rather than with the rest of the cluster configuration in the + +configuration+ section. + + The reason is simply a matter of parsing. These options are used by the + configuration database which is, by design, mostly ignorant of the content it + holds. So the decision was made to place them in an easy-to-find location. + + .CIB Properties + [width="95%",cols="2m,<5",options="header",align="center"] + |========================================================= + |Field |Description + + | admin_epoch | + indexterm:[Configuration Version,Cluster] + indexterm:[Cluster,Option,Configuration Version] + indexterm:[admin_epoch,Cluster Option] + indexterm:[Cluster,Option,admin_epoch] + When a node joins the cluster, the cluster performs a check to see + which node has the best configuration. It asks the node with the highest + (+admin_epoch+, +epoch+, +num_updates+) tuple to replace the configuration on + all the nodes -- which makes setting them, and setting them correctly, very + important. +admin_epoch+ is never modified by the cluster; you can use this + to make the configurations on any inactive nodes obsolete. _Never set this + value to zero_. In such cases, the cluster cannot tell the difference between + your configuration and the "empty" one used when nothing is found on disk. + + | epoch | + indexterm:[epoch,Cluster Option] + indexterm:[Cluster,Option,epoch] + The cluster increments this every time the configuration is updated (usually by + the administrator). + + | num_updates | + indexterm:[num_updates,Cluster Option] + indexterm:[Cluster,Option,num_updates] + The cluster increments this every time the configuration or status is updated + (usually by the cluster) and resets it to 0 when epoch changes. + + | validate-with | + indexterm:[validate-with,Cluster Option] + indexterm:[Cluster,Option,validate-with] + Determines the type of XML validation that will be done on the configuration. + If set to +none+, the cluster will not verify that updates conform to the + DTD (nor reject ones that don't). This option can be useful when + operating a mixed-version cluster during an upgrade. + + |cib-last-written | + indexterm:[cib-last-written,Cluster Property] + indexterm:[Cluster,Property,cib-last-written] + Indicates when the configuration was last written to disk. Maintained by the + cluster; for informational purposes only. + + |have-quorum | + indexterm:[have-quorum,Cluster Property] + indexterm:[Cluster,Property,have-quorum] + Indicates if the cluster has quorum. If false, this may mean that the + cluster cannot start resources or fence other nodes (see + +no-quorum-policy+ below). Maintained by the cluster. + + |dc-uuid | + indexterm:[dc-uuid,Cluster Property] + indexterm:[Cluster,Property,dc-uuid] + Indicates which cluster node is the current leader. Used by the + cluster when placing resources and determining the order of some + events. Maintained by the cluster. + + |========================================================= + + [[s-cluster-options]] + == Cluster Options == + + Cluster options, as you might expect, control how the cluster behaves + when confronted with certain situations. + + They are grouped into sets within the +crm_config+ section, and, in advanced + configurations, there may be more than one set. (This will be described later + in the section on <> where we will show how to have the cluster use + different sets of options during working hours than during weekends.) For now, + we will describe the simple case where each option is present at most once. + + You can obtain an up-to-date list of cluster options, including + their default values, by running the `man pacemaker-schedulerd` and + `man pacemaker-controld` commands. + + .Cluster Options + [width="95%",cols="5m,2,<11",options="header",align="center"] + |========================================================= + |Option |Default |Description + + | cluster-name | | + indexterm:[cluster-name,Cluster Property] + indexterm:[Cluster,Property,cluster-name] + An (optional) name for the cluster as a whole. This is mostly for users' + convenience for use as desired in administration, but this can be used + in the Pacemaker configuration in <> (as the + +#cluster-name+ <>). It may + also be used by higher-level tools when displaying cluster information, and by + certain resource agents (for example, the +ocf:heartbeat:GFS2+ agent stores the + cluster name in filesystem meta-data). + + | dc-version | | + indexterm:[dc-version,Cluster Property] + indexterm:[Cluster,Property,dc-version] + Version of Pacemaker on the cluster's DC. + Determined automatically by the cluster. + Often includes the hash which identifies the exact Git changeset it was built + from. Used for diagnostic purposes. + + | cluster-infrastructure | | + indexterm:[cluster-infrastructure,Cluster Property] + indexterm:[Cluster,Property,cluster-infrastructure] + The messaging stack on which Pacemaker is currently running. + Determined automatically by the cluster. + Used for informational and diagnostic purposes. + + | no-quorum-policy | stop + a| + indexterm:[no-quorum-policy,Cluster Option] + indexterm:[Cluster,Option,no-quorum-policy] + What to do when the cluster does not have quorum. Allowed values: + + * +ignore:+ continue all resource management + * +freeze:+ continue resource management, but don't recover resources from nodes not in the affected partition + * +stop:+ stop all resources in the affected cluster partition + * +demote:+ demote promotable resources and stop all other resources in the + affected cluster partition + * +suicide:+ fence all nodes in the affected cluster partition + + | batch-limit | 0 | + indexterm:[batch-limit,Cluster Option] + indexterm:[Cluster,Option,batch-limit] + The maximum number of actions that the cluster may execute in parallel across + all nodes. The "correct" value will depend on the speed and load of your + network and cluster nodes. If zero, the cluster will impose a dynamically + calculated limit only when any node has high load. + + | migration-limit | -1 | + indexterm:[migration-limit,Cluster Option] + indexterm:[Cluster,Option,migration-limit] + The number of <> actions that the cluster + is allowed to execute in parallel on a node. A value of -1 means unlimited. + + | symmetric-cluster | TRUE | + indexterm:[symmetric-cluster,Cluster Option] + indexterm:[Cluster,Option,symmetric-cluster] + Can all resources run on any node by default? + + | stop-all-resources | FALSE | + indexterm:[stop-all-resources,Cluster Option] + indexterm:[Cluster,Option,stop-all-resources] + Should the cluster stop all resources? + + | stop-orphan-resources | TRUE | + indexterm:[stop-orphan-resources,Cluster Option] + indexterm:[Cluster,Option,stop-orphan-resources] + Should deleted resources be stopped? This value takes precedence over + +is-managed+ (i.e. even unmanaged resources will be stopped if deleted from + the configuration when this value is TRUE). + + | stop-orphan-actions | TRUE | + indexterm:[stop-orphan-actions,Cluster Option] + indexterm:[Cluster,Option,stop-orphan-actions] + Should deleted actions be cancelled? + + | start-failure-is-fatal | TRUE | + indexterm:[start-failure-is-fatal,Cluster Option] + indexterm:[Cluster,Option,start-failure-is-fatal] + Should a failure to start a resource on a particular node prevent further start + attempts on that node? If FALSE, the cluster will decide whether the same + node is still eligible based on the resource's current failure count + and +migration-threshold+ (see <>). + + | enable-startup-probes | TRUE | + indexterm:[enable-startup-probes,Cluster Option] + indexterm:[Cluster,Option,enable-startup-probes] + Should the cluster check for active resources during startup? + + | maintenance-mode | FALSE | + indexterm:[maintenance-mode,Cluster Option] + indexterm:[Cluster,Option,maintenance-mode] + Should the cluster refrain from monitoring, starting and stopping resources? + + | stonith-enabled | TRUE | + indexterm:[stonith-enabled,Cluster Option] + indexterm:[Cluster,Option,stonith-enabled] + Should failed nodes and nodes with resources that can't be stopped be + shot? If you value your data, set up a STONITH device and enable this. + + If true, or unset, the cluster will refuse to start resources unless + one or more STONITH resources have been configured. + If false, unresponsive nodes are immediately assumed to be running no + resources, and resource takeover to online nodes starts without any + further protection (which means _data loss_ if the unresponsive node + still accesses shared storage, for example). See also the +requires+ + meta-attribute in <>. + + | stonith-action | reboot | + indexterm:[stonith-action,Cluster Option] + indexterm:[Cluster,Option,stonith-action] + Action to send to STONITH device. Allowed values are +reboot+ and +off+. + The value +poweroff+ is also allowed, but is only used for + legacy devices. + + | stonith-timeout | 60s | + indexterm:[stonith-timeout,Cluster Option] + indexterm:[Cluster,Option,stonith-timeout] + How long to wait for STONITH actions (reboot, on, off) to complete + + | stonith-max-attempts | 10 | + indexterm:[stonith-max-attempts,Cluster Option] + indexterm:[Cluster,Option,stonith-max-attempts] + How many times fencing can fail for a target before the cluster will no longer + immediately re-attempt it. + + | stonith-watchdog-timeout | 0 | + indexterm:[stonith-watchdog-timeout,Cluster Option] + indexterm:[Cluster,Option,stonith-watchdog-timeout] + If nonzero, rely on hardware watchdog self-fencing. If positive, assume unseen + nodes self-fence within this much time. If negative, and the + SBD_WATCHDOG_TIMEOUT environment variable is set, use twice that value. + + | concurrent-fencing | FALSE | + indexterm:[concurrent-fencing,Cluster Option] + indexterm:[Cluster,Option,concurrent-fencing] + Is the cluster allowed to initiate multiple fence actions concurrently? + + | fence-reaction | stop | + indexterm:[fence-reaction,Cluster Option] + indexterm:[Cluster,Option,fence-reaction] + How should a cluster node react if notified of its own fencing? A cluster node + may receive notification of its own fencing if fencing is misconfigured, or if + fabric fencing is in use that doesn't cut cluster communication. Allowed values + are +stop+ to attempt to immediately stop pacemaker and stay stopped, or + +panic+ to attempt to immediately reboot the local node, falling back to stop + on failure. The default is likely to be changed to +panic+ in a future release. + '(since 2.0.3)' + + | priority-fencing-delay | 0 | + indexterm:[priority-fencing-delay,Cluster Option] + indexterm:[Cluster,Option,priority-fencing-delay] + Apply specified delay for the fencings that are targeting the lost + nodes with the highest total resource priority in case we don't + have the majority of the nodes in our cluster partition, so that + the more significant nodes potentially win any fencing match, + which is especially meaningful under split-brain of 2-node + cluster. A promoted resource instance takes the base priority + 1 + on calculation if the base priority is not 0. Any static/random + delays that are introduced by `pcmk_delay_base/max` configured + for the corresponding fencing resources will be added to this + delay. This delay should be significantly greater than, safely + twice, the maximum `pcmk_delay_base/max`. By default, priority + fencing delay is disabled. '(since 2.0.4)' + + | cluster-delay | 60s | + indexterm:[cluster-delay,Cluster Option] + indexterm:[Cluster,Option,cluster-delay] + Estimated maximum round-trip delay over the network (excluding action + execution). If the DC requires an action to be executed on another + node, it will consider the action failed if it does not get a response + from the other node in this time (after considering the action's + own timeout). The "correct" value will depend on the speed and load of your + network and cluster nodes. + + | dc-deadtime | 20s | + indexterm:[dc-deadtime,Cluster Option] + indexterm:[Cluster,Option,dc-deadtime] + How long to wait for a response from other nodes during startup. + + The "correct" value will depend on the speed/load of your network and the type of switches used. + + | cluster-ipc-limit | 500 | + indexterm:[cluster-ipc-limit,Cluster Option] + indexterm:[Cluster,Option,cluster-ipc-limit] + The maximum IPC message backlog before one cluster daemon will disconnect + another. This is of use in large clusters, for which a good value is the number + of resources in the cluster multiplied by the number of nodes. The default of + 500 is also the minimum. Raise this if you see "Evicting client" messages for + cluster daemon PIDs in the logs. + + | pe-error-series-max | -1 | + indexterm:[pe-error-series-max,Cluster Option] + indexterm:[Cluster,Option,pe-error-series-max] + The number of PE inputs resulting in ERRORs to save. Used when reporting problems. + A value of -1 means unlimited (report all). + + | pe-warn-series-max | -1 | + indexterm:[pe-warn-series-max,Cluster Option] + indexterm:[Cluster,Option,pe-warn-series-max] + The number of PE inputs resulting in WARNINGs to save. Used when reporting problems. + A value of -1 means unlimited (report all). + + | pe-input-series-max | -1 | + indexterm:[pe-input-series-max,Cluster Option] + indexterm:[Cluster,Option,pe-input-series-max] + The number of "normal" PE inputs to save. Used when reporting problems. + A value of -1 means unlimited (report all). + + | placement-strategy | default | + indexterm:[placement-strategy,Cluster Option] + indexterm:[Cluster,Option,placement-strategy] + How the cluster should allocate resources to nodes (see <>). + Allowed values are +default+, +utilization+, +balanced+, and +minimal+. + + | node-health-strategy | none | + indexterm:[node-health-strategy,Cluster Option] + indexterm:[Cluster,Option,node-health-strategy] + How the cluster should react to node health attributes (see <>). + Allowed values are +none+, +migrate-on-red+, +only-green+, +progressive+, and + +custom+. + + | enable-acl | FALSE | + indexterm:[enable-acl,Cluster Option] + indexterm:[Cluster,Option,enable-acl] + Whether access control lists (ACLs) (see <>) can be used to authorize + modifications to the CIB. + + | node-health-base | 0 | + indexterm:[node-health-base,Cluster Option] + indexterm:[Cluster,Option,node-health-base] + The base health score assigned to a node. Only used when + +node-health-strategy+ is +progressive+. + + | node-health-green | 0 | + indexterm:[node-health-green,Cluster Option] + indexterm:[Cluster,Option,node-health-green] + The score to use for a node health attribute whose value is +green+. + Only used when +node-health-strategy+ is +progressive+ or +custom+. + + | node-health-yellow | 0 | + indexterm:[node-health-yellow,Cluster Option] + indexterm:[Cluster,Option,node-health-yellow] + The score to use for a node health attribute whose value is +yellow+. + Only used when +node-health-strategy+ is +progressive+ or +custom+. + + | node-health-red | 0 | + indexterm:[node-health-red,Cluster Option] + indexterm:[Cluster,Option,node-health-red] + The score to use for a node health attribute whose value is +red+. + Only used when +node-health-strategy+ is +progressive+ or +custom+. + + | cluster-recheck-interval | 15min | + indexterm:[cluster-recheck-interval,Cluster Option] + indexterm:[Cluster,Option,cluster-recheck-interval] + Pacemaker is primarily event-driven, and looks ahead to know when to recheck + the cluster for failure timeouts and most time-based rules. However, it will + also recheck the cluster after this amount of inactivity. This has two goals: + rules with +date_spec+ are only guaranteed to be checked this often, and it + also serves as a fail-safe for certain classes of scheduler bugs. A value of 0 + disables this polling; positive values are a time interval. + + | shutdown-lock | false | + The default of false allows active resources to be recovered elsewhere when + their node is cleanly shut down, which is what the vast majority of users will + want. However, some users prefer to make resources highly available only for + failures, with no recovery for clean shutdowns. If this option is true, + resources active on a node when it is cleanly shut down are kept "locked" to + that node (not allowed to run elsewhere) until they start again on that node + after it rejoins (or for at most shutdown-lock-limit, if set). Stonith + resources and Pacemaker Remote connections are never locked. Clone and bundle + instances and the master role of promotable clones are currently never locked, + though support could be added in a future release. Locks may be manually + cleared using the `--refresh` option of `crm_resource` (both the resource and + node must be specified; this works with remote nodes if their connection + resource's target-role is set to Stopped, but not if Pacemaker Remote is + stopped on the remote node without disabling the connection resource). + '(since 2.0.4)' + indexterm:[shutdown-lock,Cluster Option] + indexterm:[Cluster,Option,shutdown-lock] + + | shutdown-lock-limit | 0 | + If shutdown-lock is true, and this is set to a nonzero time duration, locked + resources will be allowed to start after this much time has passed since the + node shutdown was initiated, even if the node has not rejoined. (This works + with remote nodes only if their connection resource's target-role is set to + Stopped.) '(since 2.0.4)' + indexterm:[shutdown-lock-limit,Cluster Option] + indexterm:[Cluster,Option,shutdown-lock-limit] + + | remove-after-stop | FALSE | + indexterm:[remove-after-stop,Cluster Option] + indexterm:[Cluster,Option,remove-after-stop] + _Advanced Use Only:_ Should the cluster remove resources from the LRM after + they are stopped? Values other than the default are, at best, poorly tested and + potentially dangerous. + + | startup-fencing | TRUE | + indexterm:[startup-fencing,Cluster Option] + indexterm:[Cluster,Option,startup-fencing] + _Advanced Use Only:_ Should the cluster shoot unseen nodes? + Not using the default is very unsafe! + + | election-timeout | 2min | + indexterm:[election-timeout,Cluster Option] + indexterm:[Cluster,Option,election-timeout] + _Advanced Use Only:_ If you need to adjust this value, it probably indicates + the presence of a bug. + + | shutdown-escalation | 20min | + indexterm:[shutdown-escalation,Cluster Option] + indexterm:[Cluster,Option,shutdown-escalation] + _Advanced Use Only:_ If you need to adjust this value, it probably indicates + the presence of a bug. + + | join-integration-timeout | 3min | + indexterm:[join-integration-timeout,Cluster Option] + indexterm:[Cluster,Option,join-integration-timeout] + _Advanced Use Only:_ If you need to adjust this value, it probably indicates + the presence of a bug. + + | join-finalization-timeout | 30min | + indexterm:[join-finalization-timeout,Cluster Option] + indexterm:[Cluster,Option,join-finalization-timeout] + _Advanced Use Only:_ If you need to adjust this value, it probably indicates + the presence of a bug. + + | transition-delay | 0s | + indexterm:[transition-delay,Cluster Option] + indexterm:[Cluster,Option,transition-delay] + _Advanced Use Only:_ Delay cluster recovery for the configured interval to + allow for additional/related events to occur. Useful if your configuration is + sensitive to the order in which ping updates arrive. + Enabling this option will slow down cluster recovery under + all conditions. + + |========================================================= diff --git a/doc/sphinx/Pacemaker_Explained/resources.rst b/doc/sphinx/Pacemaker_Explained/resources.rst new file mode 100644 index 0000000000..9d6055e398 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/resources.rst @@ -0,0 +1,950 @@ +Cluster Resources +----------------- + +.. Convert_to_RST: + + [[s-resource-primitive]] + == What is a Cluster Resource? == + + indexterm:[Resource] + + A resource is a service made highly available by a cluster. + The simplest type of resource, a 'primitive' resource, is described + in this chapter. More complex forms, such as groups and clones, + are described in later chapters. + + Every primitive resource has a 'resource agent'. A resource agent is an + external program that abstracts the service it provides and present a + consistent view to the cluster. + + This allows the cluster to be agnostic about the resources it manages. + The cluster doesn't need to understand how the resource works because + it relies on the resource agent to do the right thing when given a + `start`, `stop` or `monitor` command. For this reason, it is crucial that + resource agents are well-tested. + + Typically, resource agents come in the form of shell scripts. However, + they can be written using any technology (such as C, Python or Perl) + that the author is comfortable with. + + [[s-resource-supported]] + == Resource Classes == + + indexterm:[Resource,class] + + Pacemaker supports several classes of agents: + + * OCF + * LSB + * Upstart + * Systemd + * Service + * Fencing + * Nagios Plugins + + === Open Cluster Framework === + + indexterm:[Resource,OCF] + indexterm:[OCF,Resources] + indexterm:[Open Cluster Framework,Resources] + + The OCF standard + footnote:[See https://github.com/ClusterLabs/OCF-spec/tree/master/ra . The + Pacemaker implementation has been somewhat extended from the OCF specs.] + is basically an extension of the Linux Standard Base conventions for + init scripts to: + + * support parameters, + * make them self-describing, and + * make them extensible + + OCF specs have strict definitions of the exit codes that actions must return. + footnote:[ + The resource-agents source code includes the `ocf-tester` script, which + can be useful in this regard. + ] + + The cluster follows these specifications exactly, and giving the wrong + exit code will cause the cluster to behave in ways you will likely + find puzzling and annoying. In particular, the cluster needs to + distinguish a completely stopped resource from one which is in some + erroneous and indeterminate state. + + Parameters are passed to the resource agent as environment variables, with the + special prefix +OCF_RESKEY_+. So, a parameter which the user thinks + of as +ip+ will be passed to the resource agent as +OCF_RESKEY_ip+. The + number and purpose of the parameters is left to the resource agent; however, + the resource agent should use the `meta-data` command to advertise any that it + supports. + + The OCF class is the most preferred as it is an industry standard, + highly flexible (allowing parameters to be passed to agents in a + non-positional manner) and self-describing. + + For more information, see the + http://www.linux-ha.org/wiki/OCF_Resource_Agents[reference] and + the 'Resource Agents' chapter of 'Pacemaker Administration'. + + === Linux Standard Base === + indexterm:[Resource,LSB] + indexterm:[LSB,Resources] + indexterm:[Linux Standard Base,Resources] + + 'LSB' resource agents are more commonly known as 'init scripts'. If a full path + is not given, they are assumed to be located in +/etc/init.d+. + + Commonly, they are provided by the OS distribution. In order to be used + with a Pacemaker cluster, they must conform to the LSB specification. + footnote:[ + See + http://refspecs.linux-foundation.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html + for the LSB Spec as it relates to init scripts. + ] + + [WARNING] + ==== + Many distributions or particular software packages claim LSB compliance + but ship with broken init scripts. For details on how to check whether + your init script is LSB-compatible, see the 'Resource Agents' chapter of + 'Pacemaker Administration'. Common problematic violations of the LSB + standard include: + + * Not implementing the +status+ operation at all + * Not observing the correct exit status codes for + +start+/+stop+/+status+ actions + * Starting a started resource returns an error + * Stopping a stopped resource returns an error + ==== + + [IMPORTANT] + ==== + Remember to make sure the computer is _not_ configured to start any + services at boot time -- that should be controlled by the cluster. + ==== + + [[s-resource-supported-systemd]] + === Systemd === + indexterm:[Resource,Systemd] + indexterm:[Systemd,Resources] + + Some newer distributions have replaced the old + http://en.wikipedia.org/wiki/Init#SysV-style["SysV"] style of + initialization daemons and scripts with an alternative called + http://www.freedesktop.org/wiki/Software/systemd[Systemd]. + + Pacemaker is able to manage these services _if they are present_. + + Instead of init scripts, systemd has 'unit files'. Generally, the + services (unit files) are provided by the OS distribution, but there + are online guides for converting from init scripts. + footnote:[For example, + http://0pointer.de/blog/projects/systemd-for-admins-3.html] + + [IMPORTANT] + ==== + Remember to make sure the computer is _not_ configured to start any + services at boot time -- that should be controlled by the cluster. + ==== + + === Upstart === + indexterm:[Resource,Upstart] + indexterm:[Upstart,Resources] + + Some newer distributions have replaced the old + http://en.wikipedia.org/wiki/Init#SysV-style["SysV"] style of + initialization daemons (and scripts) with an alternative called + http://upstart.ubuntu.com/[Upstart]. + + Pacemaker is able to manage these services _if they are present_. + + Instead of init scripts, upstart has 'jobs'. Generally, the + services (jobs) are provided by the OS distribution. + + [IMPORTANT] + ==== + Remember to make sure the computer is _not_ configured to start any + services at boot time -- that should be controlled by the cluster. + ==== + + === System Services === + indexterm:[Resource,System Services] + indexterm:[System Service,Resources] + + Since there are various types of system services (+systemd+, + +upstart+, and +lsb+), Pacemaker supports a special +service+ alias which + intelligently figures out which one applies to a given cluster node. + + This is particularly useful when the cluster contains a mix of + +systemd+, +upstart+, and +lsb+. + + In order, Pacemaker will try to find the named service as: + + . an LSB init script + . a Systemd unit file + . an Upstart job + + === STONITH === + indexterm:[Resource,STONITH] + indexterm:[STONITH,Resources] + + The STONITH class is used exclusively for fencing-related resources. This is + discussed later in <>. + + === Nagios Plugins === + indexterm:[Resource,Nagios Plugins] + indexterm:[Nagios Plugins,Resources] + + Nagios Plugins + footnote:[The project has two independent forks, hosted at + https://www.nagios-plugins.org/ and https://www.monitoring-plugins.org/. Output + from both projects' plugins is similar, so plugins from either project can be + used with pacemaker.] + allow us to monitor services on remote hosts. + + Pacemaker is able to do remote monitoring with the plugins _if they are + present_. + + A common use case is to configure them as resources belonging to a resource + container (usually a virtual machine), and the container will be restarted + if any of them has failed. Another use is to configure them as ordinary + resources to be used for monitoring hosts or services via the network. + + The supported parameters are same as the long options of the plugin. + + [[primitive-resource]] + == Resource Properties == + + These values tell the cluster which resource agent to use for the resource, + where to find that resource agent and what standards it conforms to. + + .Properties of a Primitive Resource + [width="95%",cols="1m,<6",options="header",align="center"] + |========================================================= + + |Field + |Description + + |id + |Your name for the resource + indexterm:[id,Resource] + indexterm:[Resource,Property,id] + + |class + + |The standard the resource agent conforms to. Allowed values: + +lsb+, +nagios+, +ocf+, +service+, +stonith+, +systemd+, +upstart+ + indexterm:[class,Resource] + indexterm:[Resource,Property,class] + + |type + |The name of the Resource Agent you wish to use. E.g. +IPaddr+ or +Filesystem+ + indexterm:[type,Resource] + indexterm:[Resource,Property,type] + + |provider + |The OCF spec allows multiple vendors to supply the same + resource agent. To use the OCF resource agents supplied by + the Heartbeat project, you would specify +heartbeat+ here. + indexterm:[provider,Resource] + indexterm:[Resource,Property,provider] + + |========================================================= + + The XML definition of a resource can be queried with the `crm_resource` tool. + For example: + + ---- + # crm_resource --resource Email --query-xml + ---- + + might produce: + + .A system resource definition + ===== + [source,XML] + + ===== + + [NOTE] + ===== + One of the main drawbacks to system services (LSB, systemd or + Upstart) resources is that they do not allow any parameters! + ===== + + //// + See https://tools.ietf.org/html/rfc5737 for choice of example IP address + //// + + .An OCF resource definition + ===== + [source,XML] + ------- + + + + + + ------- + ===== + + [[s-resource-options]] + == Resource Options == + + Resources have two types of options: 'meta-attributes' and 'instance attributes'. + Meta-attributes apply to any type of resource, while instance attributes + are specific to each resource agent. + + === Resource Meta-Attributes === + + Meta-attributes are used by the cluster to decide how a resource should + behave and can be easily set using the `--meta` option of the + `crm_resource` command. + + .Meta-attributes of a Primitive Resource + [width="95%",cols="2m,2,<5",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |priority + |0 + |If not all resources can be active, the cluster will stop lower + priority resources in order to keep higher priority ones active. + indexterm:[priority,Resource Option] + indexterm:[Resource,Option,priority] + + |target-role + |Started + a|What state should the cluster attempt to keep this resource in? Allowed values: + + * +Stopped:+ Force the resource to be stopped + * +Started:+ Allow the resource to be started (and in the case of + <>, promoted to master if + appropriate) + * +Slave:+ Allow the resource to be started, but only in Slave mode if + the resource is <> + * +Master:+ Equivalent to +Started+ + indexterm:[target-role,Resource Option] + indexterm:[Resource,Option,target-role] + + |is-managed + |TRUE + |Is the cluster allowed to start and stop the resource? Allowed + values: +true+, +false+ + indexterm:[is-managed,Resource Option] + indexterm:[Resource,Option,is-managed] + + |maintenance + |FALSE + |Similar to the +maintenance-mode+ <>, but for + a single resource. If true, the resource will not be started, stopped, or + monitored on any node. This differs from +is-managed+ in that monitors will + not be run. Allowed values: +true+, +false+ + indexterm:[maintenance,Resource Option] + indexterm:[Resource,Option,maintenance] + + |resource-stickiness + |1 for individual clone instances, 0 for all other resources + |A score that will be added to the current node when a resource is already + active. This allows running resources to stay where they are, even if + they would be placed elsewhere if they were being started from a stopped + state. + indexterm:[resource-stickiness,Resource Option] + indexterm:[Resource,Option,resource-stickiness] + + |requires + |+quorum+ for resources with a +class+ of +stonith+, + otherwise +unfencing+ if unfencing is active in the cluster, + otherwise +fencing+ if +stonith-enabled+ is true, otherwise +quorum+ + a|Conditions under which the resource can be started + Allowed values: + + * +nothing:+ can always be started + * +quorum:+ The cluster can only start this resource if a majority of + the configured nodes are active + * +fencing:+ The cluster can only start this resource if a majority + of the configured nodes are active _and_ any failed or unknown nodes + have been <> + * +unfencing:+ + The cluster can only start this resource if a majority + of the configured nodes are active _and_ any failed or unknown nodes + have been fenced _and_ only on nodes that have been + <> + + indexterm:[requires,Resource Option] + indexterm:[Resource,Option,requires] + + |migration-threshold + |INFINITY + |How many failures may occur for this resource on a node, before this + node is marked ineligible to host this resource. A value of 0 indicates that + this feature is disabled (the node will never be marked ineligible); by + constrast, the cluster treats INFINITY (the default) as a very large but + finite number. This option has an effect only if the failed operation + specifies +on-fail+ as +restart+ (the default), and additionally for + failed +start+ operations, if the cluster property +start-failure-is-fatal+ + is +false+. + indexterm:[migration-threshold,Resource Option] + indexterm:[Resource,Option,migration-threshold] + + |failure-timeout + |0 + |How many seconds to wait before acting as if the failure had not + occurred, and potentially allowing the resource back to the node on + which it failed. A value of 0 indicates that this feature is disabled. + indexterm:[failure-timeout,Resource Option] + indexterm:[Resource,Option,failure-timeout] + + |multiple-active + |stop_start + a|What should the cluster do if it ever finds the resource active on + more than one node? Allowed values: + + * +block:+ mark the resource as unmanaged + * +stop_only:+ stop all active instances and leave them that way + * +stop_start:+ stop all active instances and start the resource in + one location only + + indexterm:[multiple-active,Resource Option] + indexterm:[Resource,Option,multiple-active] + + |allow-migrate + |TRUE for ocf:pacemaker:remote resources, FALSE otherwise + |Whether the cluster should try to "live migrate" this resource when it needs + to be moved (see <>) + + |container-attribute-target + | + |Specific to bundle resources; see <> + + |remote-node + | + |The name of the Pacemaker Remote guest node this resource is associated with, + if any. If specified, this both enables the resource as a guest node and + defines the unique name used to identify the guest node. The guest must be + configured to run the Pacemaker Remote daemon when it is started. +WARNING:+ + This value cannot overlap with any resource or node IDs. + + |remote-port + |3121 + |If +remote-node+ is specified, the port on the guest used for its + Pacemaker Remote connection. The Pacemaker Remote daemon on the guest must be + configured to listen on this port. + + |remote-addr + |value of +remote-node+ + |If +remote-node+ is specified, the IP address or hostname used to connect to + the guest via Pacemaker Remote. The Pacemaker Remote daemon on the guest + must be configured to accept connections on this address. + + |remote-connect-timeout + |60s + |If +remote-node+ is specified, how long before a pending guest connection will + time out. + + |========================================================= + + As an example of setting resource options, if you performed the following + commands on an LSB Email resource: + + ------- + # crm_resource --meta --resource Email --set-parameter priority --parameter-value 100 + # crm_resource -m -r Email -p multiple-active -v block + ------- + + the resulting resource definition might be: + + .An LSB resource with cluster options + ===== + [source,XML] + ------- + + + + + + + ------- + ===== + + In addition to the cluster-defined meta-attributes described above, you may + also configure arbitrary meta-attributes of your own choosing. Most commonly, + this would be done for use in <>. For example, an IT department + might define a custom meta-attribute to indicate which company department each + resource is intended for. To reduce the chance of name collisions with + cluster-defined meta-attributes added in the future, it is recommended to use + a unique, organization-specific prefix for such attributes. + + [[s-resource-defaults]] + === Setting Global Defaults for Resource Meta-Attributes === + + To set a default value for a resource option, add it to the + +rsc_defaults+ section with `crm_attribute`. For example, + + ---- + # crm_attribute --type rsc_defaults --name is-managed --update false + ---- + + would prevent the cluster from starting or stopping any of the + resources in the configuration (unless of course the individual + resources were specifically enabled by having their +is-managed+ set to + +true+). + + === Resource Instance Attributes === + + The resource agents of some resource classes (lsb, systemd and upstart 'not' among them) + can be given parameters which determine how they behave and which instance + of a service they control. + + If your resource agent supports parameters, you can add them with the + `crm_resource` command. For example, + + ---- + # crm_resource --resource Public-IP --set-parameter ip --parameter-value 192.0.2.2 + ---- + + would create an entry in the resource like this: + + .An example OCF resource with instance attributes + ===== + [source,XML] + ------- + + + + + + ------- + ===== + + For an OCF resource, the result would be an environment variable + called +OCF_RESKEY_ip+ with a value of +192.0.2.2+. + + The list of instance attributes supported by an OCF resource agent can be + found by calling the resource agent with the `meta-data` command. + The output contains an XML description of all the supported + attributes, their purpose and default values. + + .Displaying the metadata for the Dummy resource agent template + ===== + ---- + # export OCF_ROOT=/usr/lib/ocf + # $OCF_ROOT/resource.d/pacemaker/Dummy meta-data + ---- + [source,XML] + ------- + + + + 1.0 + + + This is a Dummy Resource Agent. It does absolutely nothing except + keep track of whether its running or not. + Its purpose in life is for testing and to serve as a template for RA writers. + + NB: Please pay attention to the timeouts specified in the actions + section below. They should be meaningful for the kind of resource + the agent manages. They should be the minimum advised timeouts, + but they shouldn't/cannot cover _all_ possible resource + instances. So, try to be neither overly generous nor too stingy, + but moderate. The minimum timeouts should never be below 10 seconds. + + Example stateless resource agent + + + + + Location to store the resource state in. + + State file + + + + + + Fake attribute that can be changed to cause a reload + + Fake attribute that can be changed to cause a reload + + + + + + Number of seconds to sleep during operations. This can be used to test how + the cluster reacts to operation timeouts. + + Operation sleep duration in seconds. + + + + + + + + + + + + + + + + + ------- + ===== + + == Resource Operations == + + indexterm:[Resource,Action] + + 'Operations' are actions the cluster can perform on a resource by calling the + resource agent. Resource agents must support certain common operations such as + start, stop, and monitor, and may implement any others. + + Operations may be explicitly configured for two purposes: to override defaults + for options (such as timeout) that the cluster will use whenever it initiates + the operation, and to run an operation on a recurring basis (for example, to + monitor the resource for failure). + + .An OCF resource with a non-default start timeout + ===== + [source,XML] + ------- + + + + + + + + + ------- + ===== + + Pacemaker identifies operations by a combination of name and interval, so this + combination must be unique for each resource. That is, you should not configure + two operations for the same resource with the same name and interval. + + [[s-operation-properties]] + === Operation Properties === + + Operation properties may be specified directly in the +op+ element as + XML attributes, or in a separate +meta_attributes+ block as +nvpair+ elements. + XML attributes take precedence over +nvpair+ elements if both are specified. + + .Properties of an Operation + [width="95%",cols="2m,3,<6",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |id + | + |A unique name for the operation. + indexterm:[id,Action Property] + indexterm:[Action,Property,id] + + |name + | + |The action to perform. This can be any action supported by the agent; common + values include +monitor+, +start+, and +stop+. + indexterm:[name,Action Property] + indexterm:[Action,Property,name] + + |interval + |0 + |How frequently (in seconds) to perform the operation. A value of 0 means "when + needed". A positive value defines a 'recurring action', which is typically + used with <>. + indexterm:[interval,Action Property] + indexterm:[Action,Property,interval] + + |timeout + | + |How long to wait before declaring the action has failed + indexterm:[timeout,Action Property] + indexterm:[Action,Property,timeout] + + |on-fail + a|Varies by action: + + * +stop+: +fence+ if +stonith-enabled+ is true or +block+ otherwise + * +demote+: +on-fail+ of the +monitor+ action with +role+ set to +Master+, if + present, enabled, and configured to a value other than +demote+, or +restart+ + otherwise + * all other actions: +restart+ + a|The action to take if this action ever fails. Allowed values: + + * +ignore:+ Pretend the resource did not fail. + * +block:+ Don't perform any further operations on the resource. + * +stop:+ Stop the resource and do not start it elsewhere. + * +demote:+ Demote the resource, without a full restart. This is valid only for + +promote+ actions, and for +monitor+ actions with both a nonzero +interval+ + and +role+ set to +Master+; for any other action, a configuration error will + be logged, and the default behavior will be used. + * +restart:+ Stop the resource and start it again (possibly on a different node). + * +fence:+ STONITH the node on which the resource failed. + * +standby:+ Move _all_ resources away from the node on which the resource failed. + + indexterm:[on-fail,Action Property] + indexterm:[Action,Property,on-fail] + + |enabled + |TRUE + |If +false+, ignore this operation definition. This is typically used to pause + a particular recurring +monitor+ operation; for instance, it can complement + the respective resource being unmanaged (+is-managed=false+), as this alone + will <>. + Disabling the operation does not suppress all actions of the given type. + Allowed values: +true+, +false+. + indexterm:[enabled,Action Property] + indexterm:[Action,Property,enabled] + + |record-pending + |TRUE + |If +true+, the intention to perform the operation is recorded so that + GUIs and CLI tools can indicate that an operation is in progress. + This is best set as an _operation default_ (see <>). + Allowed values: +true+, +false+. + indexterm:[enabled,Action Property] + indexterm:[Action,Property,enabled] + + |role + | + |Run the operation only on node(s) that the cluster thinks should be in + the specified role. This only makes sense for recurring +monitor+ operations. + Allowed (case-sensitive) values: +Stopped+, +Started+, and in the + case of <>, +Slave+ and +Master+. + indexterm:[role,Action Property] + indexterm:[Action,Property,role] + + |========================================================= + + [NOTE] + ==== + When +on-fail+ is set to +demote+, recovery from failure by a successful demote + causes the cluster to recalculate whether and where a new instance should be + promoted. The node with the failure is eligible, so if master scores have not + changed, it will be promoted again. + + There is no direct equivalent of +migration-threshold+ for the master role, but + the same effect can be achieved with a location constraint using a + <> with a node attribute expression for the resource's fail + count. + + For example, to immediately ban the master role from a node with any failed + promote or master monitor: + [source,XML] + ---- + + + + + + + ---- + + This example assumes that there is a promotable clone of the +my_primitive+ + resource (note that the primitive name, not the clone name, is used in the + rule), and that there is a recurring 10-second-interval monitor configured for + the master role (fail count attributes specify the interval in milliseconds). + ==== + + [[s-resource-monitoring]] + === Monitoring Resources for Failure === + + When Pacemaker first starts a resource, it runs one-time +monitor+ operations + (referred to as 'probes') to ensure the resource is running where it's + supposed to be, and not running where it's not supposed to be. (This behavior + can be affected by the +resource-discovery+ location constraint property.) + + Other than those initial probes, Pacemaker will 'not' (by default) check that + the resource continues to stay healthy. + footnote:[Currently, anyway. Automatic monitoring operations may be + added in a future version of Pacemaker.] + You must configure +monitor+ operations explicitly to perform these checks. + + .An OCF resource with a recurring health check + ===== + [source,XML] + ------- + + + + + + + + + + ------- + ===== + + By default, a +monitor+ operation will ensure that the resource is running + where it is supposed to. The +target-role+ property can be used for further + checking. + + For example, if a resource has one +monitor+ operation with + +interval=10 role=Started+ and a second +monitor+ operation with + +interval=11 role=Stopped+, the cluster will run the first monitor on any nodes + it thinks 'should' be running the resource, and the second monitor on any nodes + that it thinks 'should not' be running the resource (for the truly paranoid, + who want to know when an administrator manually starts a service by mistake). + + [NOTE] + ==== + Currently, monitors with +role=Stopped+ are not implemented for + <> resources. + ==== + + [[s-monitoring-unmanaged]] + === Monitoring Resources When Administration is Disabled === + + Recurring +monitor+ operations behave differently under various administrative + settings: + + * When a resource is unmanaged (by setting +is-managed=false+): No monitors + will be stopped. + + + If the unmanaged resource is stopped on a node where the cluster thinks it + should be running, the cluster will detect and report that it is not, but it + will not consider the monitor failed, and will not try to start the resource + until it is managed again. + + + Starting the unmanaged resource on a different node is strongly discouraged + and will at least cause the cluster to consider the resource failed, and + may require the resource's +target-role+ to be set to +Stopped+ then +Started+ + to be recovered. + + * When a node is put into standby: All resources will be moved away from the + node, and all +monitor+ operations will be stopped on the node, except those + specifying +role+ as +Stopped+ (which will be newly initiated if + appropriate). + + * When the cluster is put into maintenance mode: All resources will be marked + as unmanaged. All monitor operations will be stopped, except those + specifying +role+ as +Stopped+ (which will be newly initiated if + appropriate). As with single unmanaged resources, starting + a resource on a node other than where the cluster expects it to be will + cause problems. + + [[s-operation-defaults]] + === Setting Global Defaults for Operations === + + You can change the global default values for operation properties + in a given cluster. These are defined in an +op_defaults+ section + of the CIB's +configuration+ section, and can be set with `crm_attribute`. + For example, + + ---- + # crm_attribute --type op_defaults --name timeout --update 20s + ---- + + would default each operation's +timeout+ to 20 seconds. If an + operation's definition also includes a value for +timeout+, then that + value would be used for that operation instead. + + === When Implicit Operations Take a Long Time === + + The cluster will always perform a number of implicit operations: +start+, + +stop+ and a non-recurring +monitor+ operation used at startup to check + whether the resource is already active. If one of these is taking too long, + then you can create an entry for them and specify a longer timeout. + + .An OCF resource with custom timeouts for its implicit actions + ===== + [source,XML] + ------- + + + + + + + + + + + ------- + ===== + + === Multiple Monitor Operations === + + Provided no two operations (for a single resource) have the same name + and interval, you can have as many +monitor+ operations as you like. + In this way, you can do a superficial health check every minute and + progressively more intense ones at higher intervals. + + To tell the resource agent what kind of check to perform, you need to + provide each monitor with a different value for a common parameter. + The OCF standard creates a special parameter called +OCF_CHECK_LEVEL+ + for this purpose and dictates that it is "made available to the + resource agent without the normal +OCF_RESKEY+ prefix". + + Whatever name you choose, you can specify it by adding an + +instance_attributes+ block to the +op+ tag. It is up to each + resource agent to look for the parameter and decide how to use it. + + .An OCF resource with two recurring health checks, performing different levels of checks specified via +OCF_CHECK_LEVEL+. + ===== + [source,XML] + ------- + + + + + + + + + + + + + + + + + + ------- + ===== + + === Disabling a Monitor Operation === + + The easiest way to stop a recurring monitor is to just delete it. + However, there can be times when you only want to disable it + temporarily. In such cases, simply add +enabled=false+ to the + operation's definition. + + .Example of an OCF resource with a disabled health check + ===== + [source,XML] + ------- + + + + + + + + + ------- + ===== + + This can be achieved from the command line by executing: + + ---- + # cibadmin --modify --xml-text '' + ---- + + Once you've done whatever you needed to do, you can then re-enable it with + ---- + # cibadmin --modify --xml-text '' + ---- diff --git a/doc/sphinx/Pacemaker_Explained/reusing-configuration.rst b/doc/sphinx/Pacemaker_Explained/reusing-configuration.rst new file mode 100644 index 0000000000..45212e8d52 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/reusing-configuration.rst @@ -0,0 +1,378 @@ +Reusing Parts of the Configuration +---------------------------------- + +.. Convert_to_RST: + + Pacemaker provides multiple ways to simplify the configuration XML by reusing + parts of it in multiple places. + + Besides simplifying the XML, this also allows you to manipulate multiple + configuration elements with a single reference. + + == Reusing Resource Definitions == + + If you want to create lots of resources with similar configurations, defining a + 'resource template' simplifies the task. Once defined, it can be referenced in + primitives or in certain types of constraints. + + === Configuring Resources with Templates === + + The primitives referencing the template will inherit all meta-attributes, + instance attributes, utilization attributes and operations defined + in the template. And you can define specific attributes and operations for any + of the primitives. If any of these are defined in both the template and the + primitive, the values defined in the primitive will take precedence over the + ones defined in the template. + + Hence, resource templates help to reduce the amount of configuration work. + If any changes are needed, they can be done to the template definition and + will take effect globally in all resource definitions referencing that + template. + + Resource templates have a syntax similar to that of primitives. + + .Resource template for a migratable Xen virtual machine + ==== + [source,XML] + ---- + + ---- + ==== + + Once you define a resource template, you can use it in primitives by specifying the + +template+ property. + + .Xen primitive resource using a resource template + ==== + [source,XML] + ---- + + + + + + + ---- + ==== + + In the example above, the new primitive +vm1+ will inherit everything from +vm-template+. For + example, the equivalent of the above two examples would be: + + .Equivalent Xen primitive resource not using a resource template + ==== + [source,XML] + ---- + + + + + + + + + + + + + + + + + ---- + ==== + + If you want to overwrite some attributes or operations, add them to the + particular primitive's definition. + + .Xen resource overriding template values + ==== + [source,XML] + ---- + + + + + + + + + + + + + + + + + ---- + ==== + + In the example above, the new primitive +vm2+ has special + attribute values. Its +monitor+ operation has a longer +timeout+ and +interval+, and + the primitive has an additional +stop+ operation. + + To see the resulting definition of a resource, run: + + ---- + # crm_resource --query-xml --resource vm2 + ---- + + To see the raw definition of a resource in the CIB, run: + + ---- + # crm_resource --query-xml-raw --resource vm2 + ---- + + === Using Templates in Constraints === + + A resource template can be referenced in the following types of constraints: + + - +order+ constraints (see <>) + - +colocation+ constraints (see <>) + - +rsc_ticket+ constraints (for multi-site clusters as described in <>) + + Resource templates referenced in constraints stand for all primitives which are + derived from that template. This means, the constraint applies to all primitive + resources referencing the resource template. Referencing resource templates in + constraints is an alternative to resource sets and can simplify the cluster + configuration considerably. + + For example, given the example templates earlier in this chapter: + + [source,XML] + + + would colocate all VMs with +base-rsc+ and is the equivalent of the following constraint configuration: + + [source,XML] + ---- + + + + + + + + + + ---- + + [NOTE] + ====== + In a colocation constraint, only one template may be referenced from either + `rsc` or `with-rsc`; the other reference must be a regular resource. + ====== + + === Using Templates in Resource Sets === + + Resource templates can also be referenced in resource sets. + + For example, given the example templates earlier in this section, then: + + [source,XML] + ---- + + + + + + + + ---- + + is the equivalent of the following constraint using a sequential resource set: + + [source,XML] + ---- + + + + + + + + + ---- + + Or, if the resources referencing the template can run in parallel, then: + + [source,XML] + ---- + + + + + + + + + + + + ---- + + is the equivalent of the following constraint configuration: + + [source,XML] + ---- + + + + + + + + + + + + + ---- + + [[s-reusing-config-elements]] + == Reusing Rules, Options and Sets of Operations == + + Sometimes a number of constraints need to use the same set of rules, + and resources need to set the same options and parameters. To + simplify this situation, you can refer to an existing object using an + +id-ref+ instead of an +id+. + + So if for one resource you have + + [source,XML] + ------ + + + + + + ------ + + Then instead of duplicating the rule for all your other resources, you can instead specify: + + .Referencing rules from other constraints + ===== + [source,XML] + ------- + + + + ------- + ===== + + [IMPORTANT] + =========== + The cluster will insist that the +rule+ exists somewhere. Attempting + to add a reference to a non-existing rule will cause a validation + failure, as will attempting to remove a +rule+ that is referenced + elsewhere. + =========== + + The same principle applies for +meta_attributes+ and + +instance_attributes+ as illustrated in the example below: + + .Referencing attributes, options, and operations from other resources + ===== + [source,XML] + ------- + + + + + + + + + + + + + + + + + + + + + ------- + ===== + + +id-ref+ can similarly be used with +resource_set+ (in any constraint type), + +nvpair+, and +operations+. + + == Tagging Configuration Elements == + + Pacemaker allows you to 'tag' any configuration element that has an XML ID. + + The main purpose of tagging is to support higher-level user interface tools; + Pacemaker itself only uses tags within constraints. Therefore, what you can + do with tags mostly depends on the tools you use. + + === Configuring Tags === + + A tag is simply a named list of XML IDs. + + .Tag referencing three resources + ==== + [source,XML] + ---- + + + + + + + + ---- + ==== + + What you can do with this new tag depends on what your higher-level tools + support. For example, a tool might allow you to enable or disable all of + the tagged resources at once, or show the status of just the tagged + resources. + + A single configuration element can be listed in any number of tags. + + === Using Tags in Constraints and Resource Sets === + + Pacemaker itself only uses tags in constraints. If you supply a tag name + instead of a resource name in any constraint, the constraint will apply to + all resources listed in that tag. + + .Constraint using a tag + ==== + [source,XML] + ---- + + ---- + ==== + + In the example above, assuming the +all-vms+ tag is defined as in the previous + example, the constraint will behave the same as: + + .Equivalent constraints without tags + ==== + [source,XML] + ---- + + + + ---- + ==== + + A tag may be used directly in the constraint, or indirectly by being + listed in a <> used in the constraint. + When used in a resource set, an expanded tag will honor the set's + +sequential+ property. diff --git a/doc/sphinx/Pacemaker_Explained/rules.rst b/doc/sphinx/Pacemaker_Explained/rules.rst new file mode 100644 index 0000000000..49bf4df0dc --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/rules.rst @@ -0,0 +1,925 @@ +Rules +----- + +.. Convert_to_RST: + + anchor:ch-rules[Chapter 8, Rules] + indexterm:[Constraint,Rule] + + Rules can be used to make your configuration more dynamic, allowing values to + change depending on the time or the value of a node attribute. Examples of + things rules are useful for: + + * Set a higher value for <> during + working hours, to minimize downtime, and a lower value on weekends, to allow + resources to move to their most preferred locations when people aren't around + to notice. + + * Automatically place the cluster into maintenance mode during a scheduled + maintenance window. + + * Assign certain nodes and resources to a particular department via custom + node attributes and meta-attributes, and add a single location constraint + that restricts the department's resources to run only on those nodes. + + Each constraint type or property set that supports rules may contain one or more + +rule+ elements specifying conditions under which the constraint or properties + take effect. Examples later in this chapter will make this clearer. + + == Rule Properties == + + indexterm:[XML element,rule element] + + .Attributes of a rule Element + [width="95%",cols="2m,1,<5",options="header",align="center"] + |========================================================= + + |Attribute + |Default + |Description + + |id + | + |A unique name for the rule (required) + indexterm:[XML attribute,id attribute,rule element] + indexterm:[XML element,rule element,id attribute] + + |role + |+Started+ + |The rule is in effect only when the resource is in the specified + role. Allowed values are +Started+, +Slave+, and +Master+. A rule + with +role="Master"+ cannot determine the initial location of a + clone instance and will only affect which of the active instances + will be promoted. + indexterm:[XML attribute,role attribute,rule element] + indexterm:[XML element,rule element,role attribute] + + |score + | + |If this rule is used in a location constraint and evaluates to true, apply + this score to the constraint. Only one of +score+ and +score-attribute+ may be + used. + indexterm:[XML attribute,score attribute,rule element] + indexterm:[XML element,rule element,score attribute] + + |score-attribute + | + |If this rule is used in a location constraint and evaluates to true, use the + value of this node attribute as the score to apply to the constraint. Only one + of +score+ and +score-attribute+ may be used. + indexterm:[XML attribute,score-attribute attribute,rule element] + indexterm:[XML element,rule element,score-attribute attribute] + + |boolean-op + |+and+ + |If this rule contains more than one condition, a value of +and+ specifies that + the rule evaluates to true only if all conditions are true, and a value of + +or+ specifies that the rule evaluates to true if any condition is true. + indexterm:[XML attribute,boolean-op attribute,rule element] + indexterm:[XML element,rule element,boolean-op attribute] + + |========================================================= + + A +rule+ element must contain one or more conditions. A condition may be an + +expression+ element, a +date_expression+ element, or another +rule+ element. + + == Node Attribute Expressions == + + [[node-attribute-expressions]] + indexterm:[Rule,Node Attribute Expression] + indexterm:[XML element,expression element] + + Expressions are rule conditions based on the values of node attributes. + + .Attributes of an expression Element + [width="95%",cols="2m,1,<5",options="header",align="center"] + |========================================================= + + |Field + |Default + |Description + + |id + | + |A unique name for the expression (required) + indexterm:[XML attribute,id attribute,expression element] + indexterm:[XML element,expression element,id attribute] + + |attribute + | + |The node attribute to test (required) + indexterm:[XML attribute,attribute attribute,expression element] + indexterm:[XML element,expression element,attribute attribute] + + |type + |+string+ + |How the node attributes should be compared. Allowed values are + +string+, +integer+, and +version+. + indexterm:[XML attribute,type attribute,expression element] + indexterm:[XML element,expression element,type attribute] + + |operation + | + a|The comparison to perform (required). Allowed values: + + * +lt:+ True if the node attribute value is less than the comparison value + * +gt:+ True if the node attribute value is greater than the comparison value + * +lte:+ True if the node attribute value is less than or equal to the comparison value + * +gte:+ True if the node attribute value is greater than or equal to the comparison value + * +eq:+ True if the node attribute value is equal to the comparison value + * +ne:+ True if the node attribute value is not equal to the comparison value + * +defined:+ True if the node has the named attribute + * +not_defined:+ True if the node does not have the named attribute + indexterm:[XML attribute,operation attribute,expression element] + indexterm:[XML element,expression element,operation attribute] + + |value + | + |User-supplied value for comparison (required for operations other than + +defined+ and +not_defined+) + indexterm:[XML attribute,value attribute,expression element] + indexterm:[XML element,expression element,value attribute] + + |value-source + |+literal+ + a|How the +value+ is derived. Allowed values: + + * +literal+: +value+ is a literal string to compare against + * +param+: +value+ is the name of a resource parameter to compare against (only + valid in location constraints) + * +meta+: +value+ is the name of a resource meta-attribute to compare against + (only valid in location constraints) + indexterm:[XML attribute,value-source attribute,expression element] + indexterm:[XML element,expression element,value-source attribute] + + |========================================================= + + [[node-attribute-expressions-special]] + In addition to custom node attributes defined by the administrator, the cluster + defines special, built-in node attributes for each node that can also be used + in rule expressions. + + .Built-in Node Attributes + [width="95%",cols="1m,<5",options="header",align="center"] + |========================================================= + + |Name + |Value + + |#uname + |Node <> + + |#id + |Node ID + + |#kind + |Node type. Possible values are +cluster+, +remote+, and +container+. Kind is + +remote+ for Pacemaker Remote nodes created with the +ocf:pacemaker:remote+ + resource, and +container+ for Pacemaker Remote guest nodes and bundle nodes + + |#is_dc + |"true" if this node is a Designated Controller (DC), "false" otherwise + + |#cluster-name + |The value of the +cluster-name+ cluster property, if set + + |#site-name + |The value of the +site-name+ node attribute, if set, otherwise identical to + +#cluster-name+ + + |#role + a|The role the relevant promotable clone resource has on this node. Valid only within + a rule for a location constraint for a promotable clone resource. + + //// + // if uncommenting, put a pipe in front of first two lines + #ra-version + The installed version of the resource agent on the node, as defined + by the +version+ attribute of the +resource-agent+ tag in the agent's + metadata. Valid only within rules controlling resource options. This can be + useful during rolling upgrades of a backward-incompatible resource agent. + '(coming in x.x.x)' + //// + + |========================================================= + + == Date/Time Expressions == + + indexterm:[Rule,Date/Time Expression] + indexterm:[XML element,date_expression element] + + Date/time expressions are rule conditions based (as the name suggests) on the + current date and time. + + A +date_expression+ element may optionally contain a +date_spec+ or +duration+ + element depending on the context. + + .Attributes of a date_expression Element + [width="95%",cols="2m,<5",options="header",align="center"] + |========================================================= + |Field + |Description + + |id + |A unique name for the expression (required) + indexterm:[XML attribute,id attribute,date_expression element] + indexterm:[XML element,date_expression element,id attribute] + + |start + |A date/time conforming to the http://en.wikipedia.org/wiki/ISO_8601[ISO8601] + specification. May be used when +operation+ is +in_range+ (in which case at + least one of +start+ or +end+ must be specified) or +gt+ (in which case + +start+ is required). + indexterm:[XML attribute,start attribute,date_expression element] + indexterm:[XML element,date_expression element,start attribute] + + |end + |A date/time conforming to the http://en.wikipedia.org/wiki/ISO_8601[ISO8601] + specification. May be used when +operation+ is +in_range+ (in which case at + least one of +start+ or +end+ must be specified) or +lt+ (in which case + +end+ is required). + indexterm:[XML attribute,end attribute,date_expression element] + indexterm:[XML element,date_expression element,end attribute] + + |operation + a|Compares the current date/time with the start and/or end date, + depending on the context. Allowed values: + + * +gt:+ True if the current date/time is after +start+ + * +lt:+ True if the current date/time is before +end+ + * +in_range:+ True if the current date/time is after +start+ (if specified) + and before either +end+ (if specified) or +start+ plus the value of the + +duration+ element (if one is contained in the +date_expression+) + * +date_spec:+ True if the current date/time matches the specification + given in the contained +date_spec+ element (described below) + indexterm:[XML attribute,operation attribute,date_expression element] + indexterm:[XML element,date_expression element,operation attribute] + + |========================================================= + + [NOTE] + ====== + There is no +eq+, +neq+, +gte+, or +lte+ operation, since they would be valid + only for a single second. + ====== + + === Date Specifications === + indexterm:[Rule,Date/Time Expression,Date Specification] + indexterm:[XML element,date_spec element] + + A +date_spec+ element is used to create a cron-like expression relating + to time. Each field can contain a single number or range. Any field not + supplied is ignored. + + .Attributes of a date_spec Element + [width="95%",cols="2m,<5",options="header",align="center"] + |========================================================= + + |Field + |Description + + |id + |A unique name for the object (required) + indexterm:[XML attribute,id attribute,date_spec element] + indexterm:[XML element,date_spec element,id attribute] + + |hours + |Allowed values: 0-23 (where 0 is midnight and 23 is 11 p.m.) + indexterm:[XML attribute,hours attribute,date_spec element] + indexterm:[XML element,date_spec element,hours attribute] + + |monthdays + |Allowed values: 1-31 (depending on month and year) + indexterm:[XML attribute,monthdays attribute,date_spec element] + indexterm:[XML element,date_spec element,monthdays attribute] + + |weekdays + |Allowed values: 1-7 (where 1 is Monday and 7 is Sunday) + indexterm:[XML attribute,weekdays attribute,date_spec element] + indexterm:[XML element,date_spec element,weekdays attribute] + + |yeardays + |Allowed values: 1-366 (depending on the year) + indexterm:[XML attribute,yeardays attribute,date_spec element] + indexterm:[XML element,date_spec element,yeardays attribute] + + |months + |Allowed values: 1-12 + indexterm:[XML attribute,months attribute,date_spec element] + indexterm:[XML element,date_spec element,months attribute] + + |weeks + |Allowed values: 1-53 (depending on weekyear) + indexterm:[XML attribute,weeks attribute,date_spec element] + indexterm:[XML element,date_spec element,weeks attribute] + + |years + |Year according to the Gregorian calendar + indexterm:[XML attribute,years attribute,date_spec element] + indexterm:[XML element,date_spec element,years attribute] + + |weekyears + |Year in which the week started; for example, 1 January 2005 can be specified + in ISO 8601 as '2005-001 Ordinal', '2005-01-01 Gregorian' or + '2004-W53-6 Weekly' and thus would match +years="2005"+ or +weekyears="2004"+ + indexterm:[XML attribute,weekyears attribute,date_spec element] + indexterm:[XML element,date_spec element,weekyears attribute] + + |moon + |Allowed values are 0-7 (where 0 is the new moon and 4 is full moon). + Seriously, you can use this. This was implemented to demonstrate the ease with + which new comparisons could be added. + indexterm:[XML attribute,moon attribute,date_spec element] + indexterm:[XML element,date_spec element,moon attribute] + + |========================================================= + + For example, +monthdays="1"+ matches the first day of every month, and + +hours="09-17"+ matches the hours between 9 a.m. and 5 p.m. (inclusive). + + At this time, multiple ranges (e.g. +weekdays="1,2"+ or +weekdays="1-2,5-6"+) + are not supported. + + [NOTE] + ==== + Pacemaker can calculate when evaluation of a +date_expression+ with an + +operation+ of +gt+, +lt+, or +in_range+ will next change, and schedule a + cluster re-check for that time. However, it does not do this for +date_spec+. + Instead, it evaluates the +date_spec+ whenever a cluster re-check naturally + happens via a cluster event or the +cluster-recheck-interval+ cluster option. + For example, if you have a +date_spec+ enabling a resource from 9 a.m. to 5 p.m., + and +cluster-recheck-interval+ has been set to 5 minutes, then sometime between + 9 a.m. and 9:05 a.m. the cluster would notice that it needs to start the + resource, and sometime between 5 p.m. and 5:05 p.m. it would realize that it + needs to stop the resource. The timing of the actual start and stop actions + will further depend on factors such as any other actions the cluster may need + to perform first, and the load of the machine. + ==== + + === Durations === + indexterm:[Rule,Date/Time Expression,Duration] + indexterm:[XML element,duration element] + + A +duration+ is used to calculate a value for +end+ when one is not supplied to + +in_range+ operations. It contains one or more attributes each containing a + single number. Any attribute not supplied is ignored. + + .Attributes of a duration Element + [width="95%",cols="2m,<5",options="header",align="center"] + |========================================================= + + |Field + |Description + + |id + |A unique name for this duration element (required) + indexterm:[XML attribute,id attribute,duration element] + indexterm:[XML element,duration element,id attribute] + + |seconds + |This many seconds will be added to the total duration + indexterm:[XML attribute,seconds attribute,duration element] + indexterm:[XML element,duration element,seconds attribute] + + |minutes + |This many minutes will be added to the total duration + indexterm:[XML attribute,minutes attribute,duration element] + indexterm:[XML element,duration element,minutes attribute] + + |hours + |This many hours will be added to the total duration + indexterm:[XML attribute,hours attribute,duration element] + indexterm:[XML element,duration element,hours attribute] + + |weeks + |This many weeks will be added to the total duration + indexterm:[XML attribute,weeks attribute,duration element] + indexterm:[XML element,duration element,weeks attribute] + + |months + |This many months will be added to the total duration + indexterm:[XML attribute,months attribute,duration element] + indexterm:[XML element,duration element,months attribute] + + |years + |This many years will be added to the total duration + indexterm:[XML attribute,years attribute,duration element] + indexterm:[XML element,duration element,years attribute] + + |========================================================= + + === Example Time-Based Expressions === + + A small sample of how time-based expressions can be used: + + .True if now is any time in the year 2005 + ==== + [source,XML] + ---- + + + + + + ---- + ==== + + .Equivalent expression + ==== + [source,XML] + ---- + + + + + + ---- + ==== + + .9am-5pm Monday-Friday + ==== + [source,XML] + ------- + + + + + + ------- + ==== + + Please note that the +16+ matches up to +16:59:59+, as the numeric + value (hour) still matches! + + .9am-6pm Monday through Friday or anytime Saturday + ==== + [source,XML] + ------- + + + + + + + + + ------- + ==== + + .9am-5pm or 9pm-12am Monday through Friday + ==== + [source,XML] + ------- + + + + + + + + + + + + + + ------- + ==== + + .Mondays in March 2005 + ==== + [source,XML] + ------- + + + + + + + ------- + ==== + + [NOTE] + ====== + Because no time is specified with the above dates, 00:00:00 is implied. This + means that the range includes all of 2005-03-01 but none of 2005-04-01. + You may wish to write +end="2005-03-31T23:59:59"+ to avoid confusion. + ====== + + .A full moon on Friday the 13th + ===== + [source,XML] + ------- + + + + + + ------- + ===== + + == Resource Expressions == + + An +rsc_expression+ is a rule condition based on a resource agent's properties. + This rule is only valid within an +rsc_defaults+ or +op_defaults+ context. None + of the matching attributes of +class+, +provider+, and +type+ are required. If + one is omitted, all values of that attribute will match. For instance, omitting + +type+ means every type will match. + + .Attributes of an rsc_expression Element + [width="95%",cols="2m,<5",options="header",align="center"] + |========================================================= + + |Field + |Description + + |id + |A unique name for the expression (required) + indexterm:[XML attribute,id attribute,rsc_expression element] + indexterm:[XML element,rsc_expression element,id attribute] + + |class + |The standard name to be matched against resource agents + indexterm:[XML attribute,class attribute,rsc_expression element] + indexterm:[XML element,rsc_expression element,class attribute] + + |provider + |If given, the vendor to be matched against resource agents. This + only makes sense for agents using the OCF spec. + indexterm:[XML attribute,provider attribute,rsc_expression element] + indexterm:[XML element,rsc_expression element,provider attribute] + + |type + |The name of the resource agent to be matched + indexterm:[XML attribute,type attribute,rsc_expression element] + indexterm:[XML element,rsc_expression element,type attribute] + + |========================================================= + + === Example Resource-Based Expressions === + + A small sample of how resource-based expressions can be used: + + .True for all ocf:heartbeat:IPaddr2 resources + ==== + [source,XML] + ---- + + + + ---- + ==== + + .Provider doesn't apply to non-OCF resources + ==== + [source,XML] + ---- + + + + ---- + ==== + + == Operation Expressions == + + An +op_expression+ is a rule condition based on an action of some resource + agent. This rule is only valid within an +op_defaults+ context. + + .Attributes of an op_expression Element + [width="95%",cols="2m,<5",options="header",align="center"] + |========================================================= + + |Field + |Description + + |id + |A unique name for the expression (required) + indexterm:[XML attribute,id attribute,op_expression element] + indexterm:[XML element,op_expression element,id attribute] + + |name + |The action name to match against. This can be any action supported by + the resource agent; common values include +monitor+, +start+, and +stop+ + (required). + indexterm:[XML attribute,name attribute,op_expression element] + indexterm:[XML element,op_expression element,name attribute] + + |interval + |The interval of the action to match against. If not given, only + the name attribute will be used to match. + indexterm:[XML attribute,interval attribute,op_expression element] + indexterm:[XML element,op_expression element,interval attribute] + + |========================================================= + + === Example Operation-Based Expressions === + + A small sample of how operation-based expressions can be used: + + .True for all monitor actions + ==== + [source,XML] + ---- + + + + ---- + ==== + + .True for all monitor actions with a 10 second interval + ==== + [source,XML] + ---- + + + + ---- + ==== + + == Using Rules to Determine Resource Location == + indexterm:[Rule,Determine Resource Location] + indexterm:[Resource,Location,Determine by Rules] + + A location constraint may contain one or more top-level rules. The cluster + will act as if there is a separate location constraint for each rule that + evaluates as true. + + Consider the following simple location constraint: + + .Prevent resource "webserver" from running on node3 + ===== + [source,XML] + ------- + + ------- + ===== + + The constraint can be more verbosely written using a rule: + + .Prevent resource "webserver" from running on node3 using rule + ===== + [source,XML] + ------- + + + + + + ------- + ===== + + The advantage of using the expanded form is that one could add more expressions + (for example, limiting the constraint to certain days of the week), or activate + the constraint by some node attribute other than node name. + + === Location Rules Based on Other Node Properties === + + The expanded form allows us to match on node properties other than its name. + If we rated each machine's CPU power such that the cluster had the + following nodes section: + + .A sample nodes section for use with score-attribute + ===== + [source,XML] + ------- + + + + + + + + + + + + + ------- + ===== + + then we could prevent resources from running on underpowered machines with this rule: + + [source,XML] + ------- + + + + ------- + + === Using +score-attribute+ Instead of +score+ === + + When using +score-attribute+ instead of +score+, each node matched by + the rule has its score adjusted differently, according to its value + for the named node attribute. Thus, in the previous example, if a + rule used +score-attribute="cpu_mips"+, +c001n01+ would have its + preference to run the resource increased by +1234+ whereas +c001n02+ + would have its preference increased by +5678+. + + == Using Rules to Define Options == + + Rules may be used to control a variety of options: + + * <> (+cluster_property_set+ elements) + * <> (as +instance_attributes+ or + +utilization+ elements inside a +node+ element) + * <> (as +utilization+, +meta_attributes+, + or +instance_attributes+ elements inside a resource definition element or + +op+ , +rsc_defaults+, +op_defaults+, or +template+ element) + * <> (+meta_attributes+ + inside an +op+ or +op_defaults+ element) + + === Using Rules to Control Resource Options === + + Often some cluster nodes will be different from their peers. Sometimes, + these differences -- e.g. the location of a binary or the names of network + interfaces -- require resources to be configured differently depending + on the machine they're hosted on. + + By defining multiple +instance_attributes+ objects for the resource + and adding a rule to each, we can easily handle these special cases. + + In the example below, +mySpecialRsc+ will use eth1 and port 9999 when + run on +node1+, eth2 and port 8888 on +node2+ and default to eth0 and + port 9999 for all other nodes. + + .Defining different resource options based on the node name + ===== + [source,XML] + ------- + + + + + + + + + + + + + + + + + + + + ------- + ===== + + The order in which +instance_attributes+ objects are evaluated is + determined by their score (highest to lowest). If not supplied, score + defaults to zero, and objects with an equal score are processed in + listed order. If the +instance_attributes+ object has no rule + or a +rule+ that evaluates to +true+, then for any parameter the resource does + not yet have a value for, the resource will use the parameter values defined by + the +instance_attributes+. + + For example, given the configuration above, if the resource is placed on node1: + + . +special-node1+ has the highest score (3) and so is evaluated first; + its rule evaluates to +true+, so +interface+ is set to +eth1+. + . +special-node2+ is evaluated next with score 2, but its rule evaluates to +false+, + so it is ignored. + . +defaults+ is evaluated last with score 1, and has no rule, so its values + are examined; +interface+ is already defined, so the value here is not used, + but +port+ is not yet defined, so +port+ is set to +9999+. + + === Using Rules to Control Resource Defaults === + + Rules can be used for resource and operation defaults. The following example + illustrates how to set a different +resource-stickiness+ value during and + outside work hours. This allows resources to automatically move back to their + most preferred hosts, but at a time that (in theory) does not interfere with + business activities. + + .Change +resource-stickiness+ during working hours + ===== + [source,XML] + ------- + + + + + + + + + + + + + + ------- + ===== + + Rules may be used similarly in +instance_attributes+ or +utilization+ blocks. + + Any single block may directly contain only a single rule, but that rule may + itself contain any number of rules. + + +rsc_expression+ and +op_expression+ blocks may additionally be used to set defaults + on either a single resource or across an entire class of resources with a single + rule. +rsc_expression+ may be used to select resource agents within both +rsc_defaults+ + and +op_defaults+, while +op_expression+ may only be used within +op_defaults+. If + multiple rules succeed for a given resource agent, the last one specified will be + the one that takes effect. As with any other rule, boolean operations may be used + to make more complicated expressions. + + .Set all IPaddr2 resources to stopped + ===== + [source,XML] + ------- + + + + + + + + + ------- + ===== + + .Set all monitor action timeouts to 7 seconds + ===== + [source,XML] + ------- + + + + + + + + + ------- + ===== + + .Set the monitor action timeout on all IPaddr2 resources with a given monitor interval to 8 seconds + ===== + [source,XML] + ------- + + + + + + + + + + ------- + ===== + + === Using Rules to Control Cluster Options === + indexterm:[Rule,Controlling Cluster Options] + indexterm:[Cluster,Setting Options with Rules] + + Controlling cluster options is achieved in much the same manner as + specifying different resource options on different nodes. + + The difference is that because they are cluster options, one cannot (or should + not, because they won't work) use attribute-based expressions. The following + example illustrates how to set +maintenance_mode+ during a scheduled + maintenance window. This will keep the cluster running but not monitor, start, + or stop resources during this time. + + .Schedule a maintenance window for 9 to 11 p.m. CDT Sept. 20, 2019 + ===== + [source,XML] + ------- + + + + + + + + + + + + + + + ------- + ===== + + [IMPORTANT] + ==== + The +cluster_property_set+ with an +id+ set to "cib-bootstrap-options" will + 'always' have the highest priority, regardless of any scores. Therefore, + rules in another +cluster_property_set+ can never take effect for any + properties listed in the bootstrap set. + ==== diff --git a/doc/sphinx/Pacemaker_Explained/status.rst b/doc/sphinx/Pacemaker_Explained/status.rst new file mode 100644 index 0000000000..de55642515 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/status.rst @@ -0,0 +1,375 @@ +Status -- Here be dragons +------------------------- + +.. Convert_to_RST: + + Most users never need to understand the contents of the status section + and can be happy with the output from `crm_mon`. + + However for those with a curious inclination, this section attempts to + provide an overview of its contents. + + == Node Status == + + indexterm:[Node,Status] + indexterm:[Status of a Node] + + In addition to the cluster's configuration, the CIB holds an + up-to-date representation of each cluster node in the +status+ section. + + .A bare-bones status entry for a healthy node *cl-virt-1* + ====== + [source,XML] + ----- + + + + + ----- + ====== + + Users are highly recommended _not_ to modify any part of a node's + state _directly_. The cluster will periodically regenerate the entire + section from authoritative sources, so any changes should be done + with the tools appropriate to those sources. + + .Authoritative Sources for State Information + [width="95%",cols="1m,<1",options="header",align="center"] + |========================================================= + + | CIB Object | Authoritative Source + + |node_state|pacemaker-controld + + |transient_attributes|pacemaker-attrd + + |lrm|pacemaker-execd + + |========================================================= + + The fields used in the +node_state+ objects are named as they are + largely for historical reasons and are rooted in Pacemaker's origins + as the resource manager for the older Heartbeat project. They have remained + unchanged to preserve compatibility with older versions. + + .Node Status Fields + [width="95%",cols="1m,<4",options="header",align="center"] + |========================================================= + + |Field |Description + + + | id | + indexterm:[id,Node Status] + indexterm:[Node,Status,id] + Unique identifier for the node. Corosync-based clusters use a numeric counter. + + | uname | + indexterm:[uname,Node Status] + indexterm:[Node,Status,uname] + The node's name as known by the cluster + + | in_ccm | + indexterm:[in_ccm,Node Status] + indexterm:[Node,Status,in_ccm] + Is the node a member at the cluster communication layer? Allowed values: + +true+, +false+. + + | crmd | + indexterm:[crmd,Node Status] + indexterm:[Node,Status,crmd] + Is the node a member at the pacemaker layer? Allowed values: +online+, + +offline+. + + | crm-debug-origin | + indexterm:[crm-debug-origin,Node Status] + indexterm:[Node,Status,crm-debug-origin] + The name of the source function that made the most recent change (for debugging + purposes). + + | join | + indexterm:[join,Node Status] + indexterm:[Node,Status,join] + Does the node participate in hosting resources? Allowed values: +down+, + +pending+, +member+, +banned+. + + | expected | + indexterm:[expected,Node Status] + indexterm:[Node,Status,expected] + Expected value for +join+. + + |========================================================= + + The cluster uses these fields to determine whether, at the node level, the + node is healthy or is in a failed state and needs to be fenced. + + == Transient Node Attributes == + + Like regular <>, the name/value + pairs listed in the +transient_attributes+ section help to describe the + node. However they are forgotten by the cluster when the node goes offline. + This can be useful, for instance, when you want a node to be in standby mode + (not able to run resources) just until the next reboot. + + In addition to any values the administrator sets, the cluster will + also store information about failed resources here. + + .A set of transient node attributes for node *cl-virt-1* + ====== + [source,XML] + ----- + + + + + + + + + ----- + ====== + + In the above example, we can see that a monitor on the +pingd:0+ resource has + failed once, at 09:22:22 UTC 6 April 2009. + footnote:[ + You can use the standard `date` command to print a human-readable version of + any seconds-since-epoch value, for example `date -d @1239009742`. + ] + We also see that the node is connected to three *pingd* peers and that + all known resources have been checked for on this machine (+probe_complete+). + + == Operation History == + indexterm:[Operation History] + + A node's resource history is held in the +lrm_resources+ tag (a child + of the +lrm+ tag). The information stored here includes enough + information for the cluster to stop the resource safely if it is + removed from the +configuration+ section. Specifically, the resource's + +id+, +class+, +type+ and +provider+ are stored. + + .A record of the +apcstonith+ resource + ====== + [source,XML] + + ====== + + Additionally, we store the last job for every combination of + +resource+, +action+ and +interval+. The concatenation of the values in + this tuple are used to create the id of the +lrm_rsc_op+ object. + + .Contents of an +lrm_rsc_op+ job + [width="95%",cols="2m,<5",options="header",align="center"] + |========================================================= + + |Field + |Description + + | id | + indexterm:[id,Action Status] + indexterm:[Action,Status,id] + + Identifier for the job constructed from the resource's +id+, + +operation+ and +interval+. + + | call-id | + indexterm:[call-id,Action Status] + indexterm:[Action,Status,call-id] + + The job's ticket number. Used as a sort key to determine the order in + which the jobs were executed. + + | operation | + indexterm:[operation,Action Status] + indexterm:[Action,Status,operation] + + The action the resource agent was invoked with. + + | interval | + indexterm:[interval,Action Status] + indexterm:[Action,Status,interval] + + The frequency, in milliseconds, at which the operation will be + repeated. A one-off job is indicated by 0. + + | op-status | + indexterm:[op-status,Action Status] + indexterm:[Action,Status,op-status] + + The job's status. Generally this will be either 0 (done) or -1 + (pending). Rarely used in favor of +rc-code+. + + | rc-code | + indexterm:[rc-code,Action Status] + indexterm:[Action,Status,rc-code] + + The job's result. Refer to the 'Resource Agents' chapter of 'Pacemaker + Administration' for details on what the values here mean and how they are + interpreted. + + | last-run | + indexterm:[last-run,Action Status] + indexterm:[Action,Status,last-run] + + Machine-local date/time, in seconds since epoch, + at which the job was executed. For diagnostic purposes. + + | last-rc-change | + indexterm:[last-rc-change,Action Status] + indexterm:[Action,Status,last-rc-change] + + Machine-local date/time, in seconds since epoch, + at which the job first returned the current value of +rc-code+. + For diagnostic purposes. + + | exec-time | + indexterm:[exec-time,Action Status] + indexterm:[Action,Status,exec-time] + + Time, in milliseconds, that the job was running for. + For diagnostic purposes. + + | queue-time | + indexterm:[queue-time,Action Status] + indexterm:[Action,Status,queue-time] + + Time, in seconds, that the job was queued for in the LRMd. + For diagnostic purposes. + + | crm_feature_set | + indexterm:[crm_feature_set,Action Status] + indexterm:[Action,Status,crm_feature_set] + + The version which this job description conforms to. Used when + processing +op-digest+. + + | transition-key | + indexterm:[transition-key,Action Status] + indexterm:[Action,Status,transition-key] + + A concatenation of the job's graph action number, the graph number, + the expected result and the UUID of the controller instance that scheduled + it. This is used to construct +transition-magic+ (below). + + | transition-magic | + indexterm:[transition-magic,Action Status] + indexterm:[Action,Status,transition-magic] + + A concatenation of the job's +op-status+, +rc-code+ and + +transition-key+. Guaranteed to be unique for the life of the cluster + (which ensures it is part of CIB update notifications) and contains + all the information needed for the controller to correctly analyze and + process the completed job. Most importantly, the decomposed elements + tell the controller if the job entry was expected and whether it failed. + + | op-digest | + indexterm:[op-digest,Action Status] + indexterm:[Action,Status,op-digest] + + An MD5 sum representing the parameters passed to the job. Used to + detect changes to the configuration, to restart resources if + necessary. + + | crm-debug-origin | + indexterm:[crm-debug-origin,Action Status] + indexterm:[Action,Status,crm-debug-origin] + + The origin of the current values. + For diagnostic purposes. + + |========================================================= + + === Simple Operation History Example === + + .A monitor operation (determines current state of the +apcstonith+ resource) + ====== + [source,XML] + ----- + + + + ----- + ====== + + In the above example, the job is a non-recurring monitor operation + often referred to as a "probe" for the +apcstonith+ resource. + + The cluster schedules probes for every configured resource on a node when + the node first starts, in order to determine the resource's current state + before it takes any further action. + + From the +transition-key+, we can see that this was the 22nd action of + the 2nd graph produced by this instance of the controller + (2668bbeb-06d5-40f9-936d-24cb7f87006a). + + The third field of the +transition-key+ contains a 7, which indicates + that the job expects to find the resource inactive. By looking at the +rc-code+ + property, we see that this was the case. + + As that is the only job recorded for this node, we can conclude that + the cluster started the resource elsewhere. + + === Complex Operation History Example === + + .Resource history of a +pingd+ clone with multiple jobs + ====== + [source,XML] + ----- + + + + + + + ----- + ====== + + When more than one job record exists, it is important to first sort + them by +call-id+ before interpreting them. + + Once sorted, the above example can be summarized as: + + . A non-recurring monitor operation returning 7 (not running), with a +call-id+ of 3 + . A stop operation returning 0 (success), with a +call-id+ of 32 + . A start operation returning 0 (success), with a +call-id+ of 33 + . A recurring monitor returning 0 (success), with a +call-id+ of 34 + + + The cluster processes each job record to build up a picture of the + resource's state. After the first and second entries, it is + considered stopped, and after the third it considered active. + + Based on the last operation, we can tell that the resource is + currently active. + + Additionally, from the presence of a +stop+ operation with a lower + +call-id+ than that of the +start+ operation, we can conclude that the + resource has been restarted. Specifically this occurred as part of + actions 11 and 31 of transition 11 from the controller instance with the key + +2668bbeb...+. This information can be helpful for locating the + relevant section of the logs when looking for the source of a failure. diff --git a/doc/sphinx/Pacemaker_Explained/utilization.rst b/doc/sphinx/Pacemaker_Explained/utilization.rst new file mode 100644 index 0000000000..4bcd777d62 --- /dev/null +++ b/doc/sphinx/Pacemaker_Explained/utilization.rst @@ -0,0 +1,232 @@ +Utilization and Placement Strategy +---------------------------------- + +.. Convert_to_RST: + + [[s-utilization]] + + Pacemaker decides where to place a resource according to the resource + allocation scores on every node. The resource will be allocated to the + node where the resource has the highest score. + + If the resource allocation scores on all the nodes are equal, by the default + placement strategy, Pacemaker will choose a node with the least number of + allocated resources for balancing the load. If the number of resources on each + node is equal, the first eligible node listed in the CIB will be chosen to run + the resource. + + Often, in real-world situations, different resources use significantly + different proportions of a node's capacities (memory, I/O, etc.). + We cannot balance the load ideally just according to the number of resources + allocated to a node. Besides, if resources are placed such that their combined + requirements exceed the provided capacity, they may fail to start completely or + run with degraded performance. + + To take these factors into account, Pacemaker allows you to configure: + + . The capacity a certain node provides. + . The capacity a certain resource requires. + . An overall strategy for placement of resources. + + == Utilization attributes == + + To configure the capacity that a node provides or a resource requires, + you can use 'utilization attributes' in +node+ and +resource+ objects. + You can name utilization attributes according to your preferences and define as + many name/value pairs as your configuration needs. However, the attributes' + values must be integers. + + .Specifying CPU and RAM capacities of two nodes + ==== + [source,XML] + ---- + + + + + + + + + + + + + ---- + ==== + + .Specifying CPU and RAM consumed by several resources + ==== + [source,XML] + ---- + + + + + + + + + + + + + + + + + + + ---- + ==== + + A node is considered eligible for a resource if it has sufficient free + capacity to satisfy the resource's requirements. The nature of the required + or provided capacities is completely irrelevant to Pacemaker -- it just makes + sure that all capacity requirements of a resource are satisfied before placing + a resource to a node. + + == Placement Strategy == + + After you have configured the capacities your nodes provide and the + capacities your resources require, you need to set the +placement-strategy+ + in the global cluster options, otherwise the capacity configurations have + 'no effect'. + + Four values are available for the +placement-strategy+: + + +default+:: + + Utilization values are not taken into account at all. + Resources are allocated according to allocation scores. If scores are equal, + resources are evenly distributed across nodes. + + +utilization+:: + + Utilization values are taken into account 'only' when deciding whether a node + is considered eligible (i.e. whether it has sufficient free capacity to satisfy + the resource's requirements). Load-balancing is still done based on the + number of resources allocated to a node. + + +balanced+:: + + Utilization values are taken into account when deciding whether a node + is eligible to serve a resource 'and' when load-balancing, so an attempt is + made to spread the resources in a way that optimizes resource performance. + + +minimal+:: + + Utilization values are taken into account 'only' when deciding whether a node + is eligible to serve a resource. For load-balancing, an attempt is made to + concentrate the resources on as few nodes as possible, thereby enabling + possible power savings on the remaining nodes. + + + Set +placement-strategy+ with `crm_attribute`: + ---- + # crm_attribute --name placement-strategy --update balanced + ---- + + Now Pacemaker will ensure the load from your resources will be distributed + evenly throughout the cluster, without the need for convoluted sets of + colocation constraints. + + == Allocation Details == + + === Which node is preferred to get consumed first when allocating resources? === + + - The node with the highest node weight gets consumed first. Node weight + is a score maintained by the cluster to represent node health. + + - If multiple nodes have the same node weight: + * If +placement-strategy+ is +default+ or +utilization+, + the node that has the least number of allocated resources gets consumed first. + ** If their numbers of allocated resources are equal, + the first eligible node listed in the CIB gets consumed first. + + * If +placement-strategy+ is +balanced+, + the node that has the most free capacity gets consumed first. + ** If the free capacities of the nodes are equal, + the node that has the least number of allocated resources gets consumed first. + *** If their numbers of allocated resources are equal, + the first eligible node listed in the CIB gets consumed first. + + * If +placement-strategy+ is +minimal+, + the first eligible node listed in the CIB gets consumed first. + + === Which node has more free capacity? === + + If only one type of utilization attribute has been defined, free capacity + is a simple numeric comparison. + + If multiple types of utilization attributes have been defined, then + the node that is numerically highest in the the most attribute types + has the most free capacity. For example: + + - If +nodeA+ has more free +cpus+, and +nodeB+ has more free +memory+, + then their free capacities are equal. + + - If +nodeA+ has more free +cpus+, while +nodeB+ has more free +memory+ and +storage+, + then +nodeB+ has more free capacity. + + === Which resource is preferred to be assigned first? === + + - The resource that has the highest +priority+ (see <>) gets allocated first. + + - If their priorities are equal, check whether they are already running. The + resource that has the highest score on the node where it's running gets allocated + first, to prevent resource shuffling. + + - If the scores above are equal or the resources are not running, the resource has + the highest score on the preferred node gets allocated first. + + - If the scores above are equal, the first runnable resource listed in the CIB + gets allocated first. + + + == Limitations and Workarounds == + + The type of problem Pacemaker is dealing with here is known as the + http://en.wikipedia.org/wiki/Knapsack_problem[knapsack problem] and falls into + the http://en.wikipedia.org/wiki/NP-complete[NP-complete] category of computer + science problems -- a fancy way of saying "it takes a really long time + to solve". + + Clearly in a HA cluster, it's not acceptable to spend minutes, let alone hours + or days, finding an optimal solution while services remain unavailable. + + So instead of trying to solve the problem completely, Pacemaker uses a + 'best effort' algorithm for determining which node should host a particular + service. This means it arrives at a solution much faster than traditional + linear programming algorithms, but by doing so at the price of leaving some + services stopped. + + In the contrived example at the start of this chapter: + + - +rsc-small+ would be allocated to +node1+ + - +rsc-medium+ would be allocated to +node2+ + - +rsc-large+ would remain inactive + + Which is not ideal. + + There are various approaches to dealing with the limitations of + pacemaker's placement strategy: + + Ensure you have sufficient physical capacity.:: + + It might sound obvious, but if the physical capacity of your nodes is (close to) + maxed out by the cluster under normal conditions, then failover isn't going to + go well. Even without the utilization feature, you'll start hitting timeouts and + getting secondary failures. + + Build some buffer into the capabilities advertised by the nodes.:: + + Advertise slightly more resources than we physically have, on the (usually valid) + assumption that a resource will not use 100% of the configured amount of + CPU, memory and so forth 'all' the time. This practice is sometimes called 'overcommit'. + + Specify resource priorities.:: + + If the cluster is going to sacrifice services, it should be the ones you care + about (comparatively) the least. Ensure that resource priorities are properly set + so that your most important resources are scheduled first. diff --git a/doc/sphinx/Pacemaker_Remote/options.rst b/doc/sphinx/Pacemaker_Remote/options.rst index a62fe91883..86b632446b 100644 --- a/doc/sphinx/Pacemaker_Remote/options.rst +++ b/doc/sphinx/Pacemaker_Remote/options.rst @@ -1,134 +1,134 @@ Configuration Explained ----------------------- The walk-through examples use some of these options, but don't explain exactly what they mean or do. This section is meant to be the go-to resource for all the options available for configuring pacemaker_remote-based nodes. .. index:: single: configuration Resource Meta-Attributes for Guest Nodes ######################################## When configuring a virtual machine as a guest node, the virtual machine is created using one of the usual resource agents for that purpose (for example, ocf:heartbeat:VirtualDomain or ocf:heartbeat:Xen), with additional metadata parameters. No restrictions are enforced on what agents may be used to create a guest node, but obviously the agent must create a distinct environment capable of running the pacemaker_remote daemon and cluster resources. An additional requirement is that fencing the host running the guest node resource must be sufficient for ensuring the guest node is stopped. This means, for example, that not all hypervisors supported by VirtualDomain may be used to create guest nodes; if the guest can survive the hypervisor being fenced, it may not be used as a guest node. Below are the metadata options available to enable a resource as a guest node and define its connection parameters. -.. table:: Meta-attributes for configuring VM resources as guest nodes +.. table:: **Meta-attributes for configuring VM resources as guest nodes** +------------------------+-----------------+-----------------------------------------------------------+ | Option | Default | Description | +========================+=================+===========================================================+ | remote-node | none | The node name of the guest node this resource defines. | | | | This both enables the resource as a guest node and | | | | defines the unique name used to identify the guest node. | | | | If no other parameters are set, this value will also be | | | | assumed as the hostname to use when connecting to | | | | pacemaker_remote on the VM. This value **must not** | | | | overlap with any resource or node IDs. | +------------------------+-----------------+-----------------------------------------------------------+ | remote-port | 3121 | The port on the virtual machine that the cluster will | | | | use to connect to pacemaker_remote. | +------------------------+-----------------+-----------------------------------------------------------+ | remote-addr | 'value of' | The IP address or hostname to use when connecting to | | | ``remote-node`` | pacemaker_remote on the VM. | +------------------------+-----------------+-----------------------------------------------------------+ | remote-connect-timeout | 60s | How long before a pending guest connection will time out. | +------------------------+-----------------+-----------------------------------------------------------+ Connection Resources for Remote Nodes ##################################### A remote node is defined by a connection resource. That connection resource has instance attributes that define where the remote node is located on the network and how to communicate with it. Descriptions of these instance attributes can be retrieved using the following ``pcs`` command: .. code-block:: none # pcs resource describe remote ocf:pacemaker:remote - remote resource agent Resource options: server: Server location to connect to. This can be an ip address or hostname. port: tcp port to connect to. reconnect_interval: Interval in seconds at which Pacemaker will attempt to reconnect to a remote node after an active connection to the remote node has been severed. When this value is nonzero, Pacemaker will retry the connection indefinitely, at the specified interval. When defining a remote node's connection resource, it is common and recommended to name the connection resource the same as the remote node's hostname. By default, if no **server** option is provided, the cluster will attempt to contact the remote node using the resource name as the hostname. Example defining a remote node with the hostname **remote1**: .. code-block:: none # pcs resource create remote1 remote Example defining a remote node to connect to a specific IP address and port: .. code-block:: none # pcs resource create remote1 remote server=192.168.122.200 port=8938 Environment Variables for Daemon Start-up ######################################### Authentication and encryption of the connection between cluster nodes and nodes running pacemaker_remote is achieved using with `TLS-PSK `_ encryption/authentication over TCP (port 3121 by default). This means that both the cluster node and remote node must share the same private key. By default, this key is placed at ``/etc/pacemaker/authkey`` on each node. You can change the default port and/or key location for Pacemaker and pacemaker_remote via environment variables. How these variables are set varies by OS, but usually they are set in the ``/etc/sysconfig/pacemaker`` or ``/etc/default/pacemaker`` file. .. code-block:: none #==#==# Pacemaker Remote # Use a custom directory for finding the authkey. PCMK_authkey_location=/etc/pacemaker/authkey # # Specify a custom port for Pacemaker Remote connections PCMK_remote_port=3121 Removing Remote Nodes and Guest Nodes ##################################### If the resource creating a guest node, or the **ocf:pacemaker:remote** resource creating a connection to a remote node, is removed from the configuration, the affected node will continue to show up in output as an offline node. If you want to get rid of that output, run (replacing $NODE_NAME appropriately): .. code-block:: none # crm_node --force --remove $NODE_NAME .. WARNING:: Be absolutely sure that there are no references to the node's resource in the configuration before running the above command.