diff --git a/doc/sphinx/Clusters_from_Scratch/cluster-setup.rst b/doc/sphinx/Clusters_from_Scratch/cluster-setup.rst index 0a7a7a5d28..437b5f8556 100644 --- a/doc/sphinx/Clusters_from_Scratch/cluster-setup.rst +++ b/doc/sphinx/Clusters_from_Scratch/cluster-setup.rst @@ -1,313 +1,313 @@ Set up a Cluster ---------------- Simplify Administration With a Cluster Shell ############################################ In the dark past, configuring Pacemaker required the administrator to read and write XML. In true UNIX style, there were also a number of different commands that specialized in different aspects of querying and updating the cluster. In addition, the various components of the cluster stack (Corosync, Pacemaker, etc.) had to be configured separately, with different configuration tools and formats. All of that has been greatly simplified with the creation of higher-level tools, whether command-line or GUIs, that hide all the mess underneath. Command-line cluster shells take all the individual aspects required for managing and configuring a cluster, and pack them into one simple-to-use command-line tool. They even allow you to queue up several changes at once and commit them all at once. Two popular command-line shells are ``pcs`` and ``crmsh``. Clusters from Scratch is based on ``pcs`` because it comes with |CFS_DISTRO|, but both have similar functionality. Choosing a shell or GUI is a matter of personal preference and what comes with (and perhaps is supported by) your choice of operating system. Install the Cluster Software ############################ Fire up a shell on both nodes and run the following to activate the High Availability repo. .. code-block:: console # dnf config-manager --set-enabled highavailability .. IMPORTANT:: This document will show commands that need to be executed on both nodes with a simple ``#`` prompt. Be sure to run them on each node individually. Now, we'll install ``pacemaker``, ``pcs``, and some other command-line tools that will make our lives easier: .. code-block:: console # dnf install -y pacemaker pcs psmisc policycoreutils-python3 .. NOTE:: This document uses ``pcs`` for cluster management. Other alternatives, such as ``crmsh``, are available, but their syntax will differ from the examples used here. Configure the Cluster Software ############################## .. index:: single: firewall Allow cluster services through firewall _______________________________________ On each node, allow cluster-related services through the local firewall: .. code-block:: console # firewall-cmd --permanent --add-service=high-availability success # firewall-cmd --reload success .. NOTE :: If you are using ``iptables`` directly, or some other firewall solution besides ``firewalld``, simply open the following ports, which can be used by various clustering components: TCP ports 2224, 3121, and 21064, and UDP port 5405. If you run into any problems during testing, you might want to disable the firewall and SELinux entirely until you have everything working. This may create significant security issues and should not be performed on machines that will be exposed to the outside world, but may be appropriate during development and testing on a protected host. To disable security measures: .. code-block:: console [root@pcmk-1 ~]# setenforce 0 [root@pcmk-1 ~]# sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config [root@pcmk-1 ~]# systemctl mask firewalld.service [root@pcmk-1 ~]# systemctl stop firewalld.service [root@pcmk-1 ~]# iptables --flush Enable ``pcs`` Daemon _____________________ Before the cluster can be configured, the ``pcs`` daemon must be started and enabled to start at boot time on each node. This daemon works with the ``pcs`` command-line interface to manage synchronizing the Corosync configuration across all nodes in the cluster, among other functions. Start and enable the daemon by issuing the following commands on each node: .. code-block:: console # systemctl start pcsd.service # systemctl enable pcsd.service Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service. -The installed packages will create an ``hacluster`` user with a disabled password. -While this is fine for running ``pcs`` commands locally, +The installed packages will create an |CRM_DAEMON_USER| user with a disabled +password. While this is fine for running ``pcs`` commands locally, the account needs a login password in order to perform such tasks as syncing the Corosync configuration, or starting and stopping the cluster on other nodes. This tutorial will make use of such commands, -so now we will set a password for the ``hacluster`` user, using the same password -on both nodes: +so now we will set a password for the |CRM_DAEMON_USER| user, using the same +password on both nodes: .. code-block:: console # passwd hacluster Changing password for user hacluster. New password: Retype new password: passwd: all authentication tokens updated successfully. .. NOTE:: Alternatively, to script this process or set the password on a different machine from the one you're logged into, you can use the ``--stdin`` option for ``passwd``: .. code-block:: console [root@pcmk-1 ~]# ssh pcmk-2 -- 'echo mysupersecretpassword | passwd --stdin hacluster' Configure Corosync __________________ On either node, use ``pcs host auth`` to authenticate as the ``hacluster`` user: .. code-block:: console [root@pcmk-1 ~]# pcs host auth pcmk-1 pcmk-2 Username: hacluster Password: pcmk-2: Authorized pcmk-1: Authorized Next, use ``pcs cluster setup`` on the same node to generate and synchronize the Corosync configuration: .. code-block:: console [root@pcmk-1 ~]# pcs cluster setup mycluster pcmk-1 pcmk-2 No addresses specified for host 'pcmk-1', using 'pcmk-1' No addresses specified for host 'pcmk-2', using 'pcmk-2' Destroying cluster on hosts: 'pcmk-1', 'pcmk-2'... pcmk-2: Successfully destroyed cluster pcmk-1: Successfully destroyed cluster Requesting remove 'pcsd settings' from 'pcmk-1', 'pcmk-2' pcmk-1: successful removal of the file 'pcsd settings' pcmk-2: successful removal of the file 'pcsd settings' Sending 'corosync authkey', 'pacemaker authkey' to 'pcmk-1', 'pcmk-2' pcmk-1: successful distribution of the file 'corosync authkey' pcmk-1: successful distribution of the file 'pacemaker authkey' pcmk-2: successful distribution of the file 'corosync authkey' pcmk-2: successful distribution of the file 'pacemaker authkey' Sending 'corosync.conf' to 'pcmk-1', 'pcmk-2' pcmk-1: successful distribution of the file 'corosync.conf' pcmk-2: successful distribution of the file 'corosync.conf' Cluster has been successfully set up. .. NOTE:: If you'd like, you can specify an ``addr`` option for each node in the ``pcs cluster setup`` command. This will create an explicit name-to-address mapping for each node in ``/etc/corosync/corosync.conf``, eliminating the need for hostname resolution via DNS, ``/etc/hosts``, and the like. .. code-block:: console [root@pcmk-1 ~]# pcs cluster setup mycluster \ pcmk-1 addr=192.168.122.101 pcmk-2 addr=192.168.122.102 If you received an authorization error for either of those commands, make sure you configured the ``hacluster`` user account on each node with the same password. The final ``corosync.conf`` configuration on each node should look something like the sample in :ref:`sample-corosync-configuration`. Explore pcs ########### Start by taking some time to familiarize yourself with what ``pcs`` can do. .. code-block:: console [root@pcmk-1 ~]# pcs Usage: pcs [-f file] [-h] [commands]... Control and configure pacemaker and corosync. Options: -h, --help Display usage and exit. -f file Perform actions on file instead of active CIB. Commands supporting the option use the initial state of the specified file as their input and then overwrite the file with the state reflecting the requested operation(s). A few commands only use the specified file in read-only mode since their effect is not a CIB modification. --debug Print all network traffic and external commands run. --version Print pcs version information. List pcs capabilities if --full is specified. --request-timeout Timeout for each outgoing request to another node in seconds. Default is 60s. --force Override checks and errors, the exact behavior depends on the command. WARNING: Using the --force option is strongly discouraged unless you know what you are doing. Commands: cluster Configure cluster options and nodes. resource Manage cluster resources. stonith Manage fence devices. constraint Manage resource constraints. property Manage pacemaker properties. acl Manage pacemaker access control lists. qdevice Manage quorum device provider on the local host. quorum Manage cluster quorum settings. booth Manage booth (cluster ticket manager). status View cluster status. config View and manage cluster configuration. pcsd Manage pcs daemon. host Manage hosts known to pcs/pcsd. node Manage cluster nodes. alert Manage pacemaker alerts. client Manage pcsd client configuration. dr Manage disaster recovery configuration. tag Manage pacemaker tags. As you can see, the different aspects of cluster management are separated into categories. To discover the functionality available in each of these categories, one can issue the command ``pcs help``. Below is an example of all the options available under the status category. .. code-block:: console [root@pcmk-1 ~]# pcs status help Usage: pcs status [commands]... View current cluster and resource status Commands: [status] [--full] [--hide-inactive] View all information about the cluster and resources (--full provides more details, --hide-inactive hides inactive resources). resources [] [node=] [--hide-inactive] Show status of all currently configured resources. If --hide-inactive is specified, only show active resources. If a resource or tag id is specified, only show status of the specified resource or resources in the specified tag. If node is specified, only show status of resources configured for the specified node. cluster View current cluster status. corosync View current membership information as seen by corosync. quorum View current quorum status. qdevice [--full] [] Show runtime status of specified model of quorum device provider. Using --full will give more detailed output. If is specified, only information about the specified cluster will be displayed. booth Print current status of booth on the local node. nodes [corosync | both | config] View current status of nodes from pacemaker. If 'corosync' is specified, view current status of nodes from corosync instead. If 'both' is specified, view current status of nodes from both corosync & pacemaker. If 'config' is specified, print nodes from corosync & pacemaker configuration. pcsd []... Show current status of pcsd on nodes specified, or on all nodes configured in the local cluster if no nodes are specified. xml View xml version of status (output from crm_mon -r -1 -X). Additionally, if you are interested in the version and supported cluster stack(s) available with your Pacemaker installation, run: .. code-block:: console [root@pcmk-1 ~]# pacemakerd --features Pacemaker 2.1.2-4.el9 (Build: ada5c3b36e2) Supporting v3.13.0: agent-manpages cibsecrets corosync-ge-2 default-concurrent-fencing default-resource-stickiness default-sbd-sync generated-manpages monotonic nagios ncurses remote systemd diff --git a/doc/sphinx/Pacemaker_Administration/alerts.rst b/doc/sphinx/Pacemaker_Administration/alerts.rst index c0f54c628a..42efc8d266 100644 --- a/doc/sphinx/Pacemaker_Administration/alerts.rst +++ b/doc/sphinx/Pacemaker_Administration/alerts.rst @@ -1,311 +1,311 @@ .. index:: single: alert; agents Alert Agents ------------ .. index:: single: alert; sample agents Using the Sample Alert Agents ############################# Pacemaker provides several sample alert agents, installed in ``/usr/share/pacemaker/alerts`` by default. While these sample scripts may be copied and used as-is, they are provided mainly as templates to be edited to suit your purposes. See their source code for the full set of instance attributes they support. .. topic:: Sending cluster events as SNMP v2c traps .. code-block:: xml .. note:: **SNMP alert agent attributes** The ``timestamp-format`` meta-attribute should always be set to ``%Y-%m-%d,%H:%M:%S.%01N`` when using the SNMP agent, to match the SNMP standard. The SNMP agent provides a number of instance attributes in addition to the one used in the example above. The most useful are ``trap_version``, which defaults to ``2c``, and ``trap_community``, which defaults to ``public``. See the source code for more details. .. topic:: Sending cluster events as SNMP v3 traps .. code-block:: xml .. note:: **SNMP v3 trap configuration** To use SNMP v3, ``trap_version`` must be set to ``3``. ``trap_community`` will be ignored. The example above uses the ``trap_options`` instance attribute to override the security level, authentication protocol, authentication user, and authentication password from snmp.conf. These will be passed to the snmptrap command. Passing the password on the command line is considered insecure; specify authentication and privacy options suitable for your environment. .. topic:: Sending cluster events as e-mails .. code-block:: xml .. index:: single: alert; agent development Writing an Alert Agent ###################### .. index:: single: alert; environment variables single: environment variable; alert agents .. table:: **Environment variables passed to alert agents** :class: longtable :widths: 1 3 +---------------------------+----------------------------------------------------------------+ | Environment Variable | Description | +===========================+================================================================+ | CRM_alert_kind | .. index:: | | | single:environment variable; CRM_alert_kind | | | single:CRM_alert_kind | | | | | | The type of alert (``node``, ``fencing``, ``resource``, or | | | ``attribute``) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_node | .. index:: | | | single:environment variable; CRM_alert_node | | | single:CRM_alert_node | | | | | | Name of affected node | +---------------------------+----------------------------------------------------------------+ | CRM_alert_node_sequence | .. index:: | | | single:environment variable; CRM_alert_sequence | | | single:CRM_alert_sequence | | | | | | A sequence number increased whenever an alert is being issued | | | on the local node, which can be used to reference the order in | | | which alerts have been issued by Pacemaker. An alert for an | | | event that happened later in time reliably has a higher | | | sequence number than alerts for earlier events. | | | | | | Be aware that this number has no cluster-wide meaning. | +---------------------------+----------------------------------------------------------------+ | CRM_alert_recipient | .. index:: | | | single:environment variable; CRM_alert_recipient | | | single:CRM_alert_recipient | | | | | | The configured recipient | +---------------------------+----------------------------------------------------------------+ | CRM_alert_timestamp | .. index:: | | | single:environment variable; CRM_alert_timestamp | | | single:CRM_alert_timestamp | | | | | | A timestamp created prior to executing the agent, in the | | | format specified by the ``timestamp-format`` meta-attribute. | | | This allows the agent to have a reliable, high-precision time | | | of when the event occurred, regardless of when the agent | | | itself was invoked (which could potentially be delayed due to | | | system load, etc.). | +---------------------------+----------------------------------------------------------------+ | CRM_alert_timestamp_epoch | .. index:: | | | single:environment variable; CRM_alert_timestamp_epoch | | | single:CRM_alert_timestamp_epoch | | | | | | The same time as ``CRM_alert_timestamp``, expressed as the | | | integer number of seconds since January 1, 1970. This (along | | | with ``CRM_alert_timestamp_usec``) can be useful for alert | | | agents that need to format time in a specific way rather than | | | let the user configure it. | +---------------------------+----------------------------------------------------------------+ | CRM_alert_timestamp_usec | .. index:: | | | single:environment variable; CRM_alert_timestamp_usec | | | single:CRM_alert_timestamp_usec | | | | | | The same time as ``CRM_alert_timestamp``, expressed as the | | | integer number of microseconds since | | | ``CRM_alert_timestamp_epoch``. | +---------------------------+----------------------------------------------------------------+ | CRM_alert_version | .. index:: | | | single:environment variable; CRM_alert_version | | | single:CRM_alert_version | | | | | | The version of Pacemaker sending the alert | +---------------------------+----------------------------------------------------------------+ | CRM_alert_desc | .. index:: | | | single:environment variable; CRM_alert_desc | | | single:CRM_alert_desc | | | | | | Detail about event. For ``node`` alerts, this is the node's | | | current state (``member`` or ``lost``). For ``fencing`` | | | alerts, this is a summary of the requested fencing operation, | | | including origin, target, and fencing operation error code, if | | | any. For ``resource`` alerts, this is a readable string | | | equivalent of ``CRM_alert_status``. | +---------------------------+----------------------------------------------------------------+ | CRM_alert_nodeid | .. index:: | | | single:environment variable; CRM_alert_nodeid | | | single:CRM_alert_nodeid | | | | | | ID of node whose status changed (provided with ``node`` alerts | | | only) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_rc | .. index:: | | | single:environment variable; CRM_alert_rc | | | single:CRM_alert_rc | | | | | | The numerical return code of the fencing or resource operation | | | (provided with ``fencing`` and ``resource`` alerts only) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_task | .. index:: | | | single:environment variable; CRM_alert_task | | | single:CRM_alert_task | | | | | | The requested fencing or resource operation (provided with | | | ``fencing`` and ``resource`` alerts only) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_exec_time | .. index:: | | | single:environment variable; CRM_alert_exec_time | | | single:CRM_alert_exec_time | | | | | | The (wall-clock) time, in milliseconds, that it took to | | | execute the action. If the action timed out, | | | ``CRM_alert_status`` will be 2, ``CRM_alert_desc`` will be | | | "Timed Out", and this value will be the action timeout. May | | | not be supported on all platforms. (``resource`` alerts only) | | | *(since 2.0.1)* | +---------------------------+----------------------------------------------------------------+ | CRM_alert_interval | .. index:: | | | single:environment variable; CRM_alert_interval | | | single:CRM_alert_interval | | | | | | The interval of the resource operation (``resource`` alerts | | | only) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_rsc | .. index:: | | | single:environment variable; CRM_alert_rsc | | | single:CRM_alert_rsc | | | | | | The name of the affected resource (``resource`` alerts only) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_status | .. index:: | | | single:environment variable; CRM_alert_status | | | single:CRM_alert_status | | | | | | A numerical code used by Pacemaker to represent the operation | | | result (``resource`` alerts only) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_target_rc | .. index:: | | | single:environment variable; CRM_alert_target_rc | | | single:CRM_alert_target_rc | | | | | | The expected numerical return code of the operation | | | (``resource`` alerts only) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_attribute_name | .. index:: | | | single:environment variable; CRM_alert_attribute_name | | | single:CRM_alert_attribute_name | | | | | | The name of the node attribute that changed (``attribute`` | | | alerts only) | +---------------------------+----------------------------------------------------------------+ | CRM_alert_attribute_value | .. index:: | | | single:environment variable; CRM_alert_attribute_value | | | single:CRM_alert_attribute_value | | | | | | The new value of the node attribute that changed | | | (``attribute`` alerts only) | +---------------------------+----------------------------------------------------------------+ Special concerns when writing alert agents: * Alert agents may be called with no recipient (if none is configured), so the agent must be able to handle this situation, even if it only exits in that case. (Users may modify the configuration in stages, and add a recipient later.) * If more than one recipient is configured for an alert, the alert agent will be called once per recipient. If an agent is not able to run concurrently, it should be configured with only a single recipient. The agent is free, however, to interpret the recipient as a list. * When a cluster event occurs, all alerts are fired off at the same time as separate processes. Depending on how many alerts and recipients are configured, and on what is done within the alert agents, a significant load burst may occur. The agent could be written to take this into consideration, for example by queueing resource-intensive actions into some other instance, instead of directly executing them. -* Alert agents are run as the ``hacluster`` user, which has a minimal set +* Alert agents are run as the |CRM_DAEMON_USER| user, which has a minimal set of permissions. If an agent requires additional privileges, it is recommended to configure ``sudo`` to allow the agent to run the necessary commands as another user with the appropriate privileges. * As always, take care to validate and sanitize user-configured parameters, such as ``CRM_alert_timestamp`` (whose content is specified by the user-configured ``timestamp-format``), ``CRM_alert_recipient,`` and all instance attributes. Mostly this is needed simply to protect against configuration errors, but if some user can modify the CIB without having - ``hacluster``-level access to the cluster nodes, it is a potential security + |CRM_DAEMON_USER| access to the cluster nodes, it is a potential security concern as well, to avoid the possibility of code injection. .. note:: **ocf:pacemaker:ClusterMon compatibility** The alerts interface is designed to be backward compatible with the external scripts interface used by the ``ocf:pacemaker:ClusterMon`` resource, which is now deprecated. To preserve this compatibility, the environment variables passed to alert agents are available prepended with ``CRM_notify_`` as well as ``CRM_alert_``. One break in compatibility is that ``ClusterMon`` ran external scripts as the ``root`` user, while alert agents are run as the - ``hacluster`` user. + |CRM_DAEMON_USER| user. diff --git a/doc/sphinx/Pacemaker_Explained/acls.rst b/doc/sphinx/Pacemaker_Explained/acls.rst index 67d5d15c39..52473c50a4 100644 --- a/doc/sphinx/Pacemaker_Explained/acls.rst +++ b/doc/sphinx/Pacemaker_Explained/acls.rst @@ -1,460 +1,460 @@ .. index:: single: Access Control List (ACL) .. _acl: Access Control Lists (ACLs) --------------------------- By default, the ``root`` user or any user in the ``haclient`` group can modify Pacemaker's CIB without restriction. Pacemaker offers *access control lists (ACLs)* to provide more fine-grained authorization. .. important:: Being able to modify the CIB's resource section allows a user to run any executable file as root, by configuring it as an LSB resource with a full path. ACL Prerequisites ################# In order to use ACLs: * The ``enable-acl`` :ref:`cluster option ` must be set to true. * Desired users must have user accounts in the ``haclient`` group on all cluster nodes in the cluster. * If your CIB was created before Pacemaker 1.1.12, it might need to be updated to the current schema (using ``cibadmin --upgrade`` or a higher-level tool equivalent) in order to use the syntax documented here. * Prior to the 2.1.0 release, the Pacemaker software had to have been built with ACL support. If you are using an older release, your installation supports ACLs only if the output of the command ``pacemakerd --features`` contains ``acls``. In newer versions, ACLs are always enabled. .. index:: single: Access Control List (ACL); acls pair: acls; XML element ACL Configuration ################# ACLs are specified within an ``acls`` element of the CIB. The ``acls`` element may contain any number of ``acl_role``, ``acl_target``, and ``acl_group`` elements. .. index:: single: Access Control List (ACL); acl_role pair: acl_role; XML element ACL Roles ######### An ACL *role* is a collection of permissions allowing or denying access to particular portions of the CIB. A role is configured with an ``acl_role`` element in the CIB ``acls`` section. .. table:: **Properties of an acl_role element** :widths: 1 3 +------------------+-----------------------------------------------------------+ | Attribute | Description | +==================+===========================================================+ | id | .. index:: | | | single: acl_role; id (attribute) | | | single: id; acl_role attribute | | | single: attribute; id (acl_role) | | | | | | A unique name for the role *(required)* | +------------------+-----------------------------------------------------------+ | description | .. index:: | | | single: acl_role; description (attribute) | | | single: description; acl_role attribute | | | single: attribute; description (acl_role) | | | | | | Arbitrary text (not used by Pacemaker) | +------------------+-----------------------------------------------------------+ An ``acl_role`` element may contain any number of ``acl_permission`` elements. .. index:: single: Access Control List (ACL); acl_permission pair: acl_permission; XML element .. table:: **Properties of an acl_permission element** :widths: 1 3 +------------------+-----------------------------------------------------------+ | Attribute | Description | +==================+===========================================================+ | id | .. index:: | | | single: acl_permission; id (attribute) | | | single: id; acl_permission attribute | | | single: attribute; id (acl_permission) | | | | | | A unique name for the permission *(required)* | +------------------+-----------------------------------------------------------+ | description | .. index:: | | | single: acl_permission; description (attribute) | | | single: description; acl_permission attribute | | | single: attribute; description (acl_permission) | | | | | | Arbitrary text (not used by Pacemaker) | +------------------+-----------------------------------------------------------+ | kind | .. index:: | | | single: acl_permission; kind (attribute) | | | single: kind; acl_permission attribute | | | single: attribute; kind (acl_permission) | | | | | | The access being granted. Allowed values are ``read``, | | | ``write``, and ``deny``. A value of ``write`` grants both | | | read and write access. | +------------------+-----------------------------------------------------------+ | object-type | .. index:: | | | single: acl_permission; object-type (attribute) | | | single: object-type; acl_permission attribute | | | single: attribute; object-type (acl_permission) | | | | | | The name of an XML element in the CIB to which the | | | permission applies. (Exactly one of ``object-type``, | | | ``xpath``, and ``reference`` must be specified for a | | | permission.) | +------------------+-----------------------------------------------------------+ | attribute | .. index:: | | | single: acl_permission; attribute (attribute) | | | single: attribute; acl_permission attribute | | | single: attribute; attribute (acl_permission) | | | | | | If specified, the permission applies only to | | | ``object-type`` elements that have this attribute set (to | | | any value). If not specified, the permission applies to | | | all ``object-type`` elements. May only be used with | | | ``object-type``. | +------------------+-----------------------------------------------------------+ | reference | .. index:: | | | single: acl_permission; reference (attribute) | | | single: reference; acl_permission attribute | | | single: attribute; reference (acl_permission) | | | | | | The ID of an XML element in the CIB to which the | | | permission applies. (Exactly one of ``object-type``, | | | ``xpath``, and ``reference`` must be specified for a | | | permission.) | +------------------+-----------------------------------------------------------+ | xpath | .. index:: | | | single: acl_permission; xpath (attribute) | | | single: xpath; acl_permission attribute | | | single: attribute; xpath (acl_permission) | | | | | | An `XPath `_ | | | specification selecting an XML element in the CIB to | | | which the permission applies. Attributes may be specified | | | in the XPath to select particular elements, but the | | | permissions apply to the entire element. (Exactly one of | | | ``object-type``, ``xpath``, and ``reference`` must be | | | specified for a permission.) | +------------------+-----------------------------------------------------------+ .. important:: * Permissions are applied to the selected XML element's entire XML subtree (all elements enclosed within it). * Write permission grants the ability to create, modify, or remove the element and its subtree, and also the ability to create any "scaffolding" elements (enclosing elements that do not have attributes other than an ID). * Permissions for more specific matches (more deeply nested elements) take precedence over more general ones. * If multiple permissions are configured for the same match (for example, in different roles applied to the same user), any ``deny`` permission takes precedence, then ``write``, then lastly ``read``. ACL Targets and Groups ###################### ACL targets correspond to user accounts on the system. .. index:: single: Access Control List (ACL); acl_target pair: acl_target; XML element .. table:: **Properties of an acl_target element** :widths: 1 3 +------------------+-----------------------------------------------------------+ | Attribute | Description | +==================+===========================================================+ | id | .. index:: | | | single: acl_target; id (attribute) | | | single: id; acl_target attribute | | | single: attribute; id (acl_target) | | | | | | A unique identifier for the target (if ``name`` is not | | | specified, this must be the name of the user account) | | | *(required)* | +------------------+-----------------------------------------------------------+ | name | .. index:: | | | single: acl_target; name (attribute) | | | single: name; acl_target attribute | | | single: attribute; name (acl_target) | | | | | | If specified, the user account name (this allows you to | | | specify a user name that is already used as the ``id`` | | | for some other configuration element) *(since 2.1.5)* | +------------------+-----------------------------------------------------------+ ACL groups correspond to groups on the system. Any role configured for these groups apply to all users in that group *(since 2.1.5)*. .. index:: single: Access Control List (ACL); acl_group pair: acl_group; XML element .. table:: **Properties of an acl_group element** :widths: 1 3 +------------------+-----------------------------------------------------------+ | Attribute | Description | +==================+===========================================================+ | id | .. index:: | | | single: acl_group; id (attribute) | | | single: id; acl_group attribute | | | single: attribute; id (acl_group) | | | | | | A unique identifier for the group (if ``name`` is not | | | specified, this must be the group name) *(required)* | +------------------+-----------------------------------------------------------+ | name | .. index:: | | | single: acl_group; name (attribute) | | | single: name; acl_group attribute | | | single: attribute; name (acl_group) | | | | | | If specified, the group name (this allows you to specify | | | a group name that is already used as the ``id`` for some | | | other configuration element) | +------------------+-----------------------------------------------------------+ Each ``acl_target`` and ``acl_group`` element may contain any number of ``role`` elements. .. note:: If the system users and groups are defined by some network service (such as LDAP), the cluster itself will be unaffected by outages in the service, but affected users and groups will not be able to make changes to the CIB. .. index:: single: Access Control List (ACL); role pair: role; XML element .. table:: **Properties of a role element** :widths: 1 3 +------------------+-----------------------------------------------------------+ | Attribute | Description | +==================+===========================================================+ | id | .. index:: | | | single: role; id (attribute) | | | single: id; role attribute | | | single: attribute; id (role) | | | | | | The ``id`` of an ``acl_role`` element that specifies | | | permissions granted to the enclosing target or group. | +------------------+-----------------------------------------------------------+ .. important:: - The ``root`` and ``hacluster`` user accounts always have full access to the - CIB, regardless of ACLs. For all other user accounts, when ``enable-acl`` is - true, permission to all parts of the CIB is denied by default (permissions + The ``root`` and |CRM_DAEMON_USER| user accounts always have full access to + the CIB, regardless of ACLs. For all other user accounts, when ``enable-acl`` + is true, permission to all parts of the CIB is denied by default (permissions must be explicitly granted). ACL Examples ############ .. code-block:: xml In the above example, the user ``alice`` has the minimal permissions necessary to run basic Pacemaker CLI tools, including using ``crm_mon`` to view the cluster status, without being able to modify anything. The user ``bob`` can view the entire configuration and status of the cluster, but not make any changes. The user ``carol`` can read everything, and change selected cluster properties as well as resource roles and location constraints. Finally, ``dave`` has full read and write access to the entire CIB. Looking at the ``minimal`` role in more depth, it is designed to allow read access to the ``cib`` tag itself, while denying access to particular portions of its subtree (which is the entire CIB). This is because the DC node is indicated in the ``cib`` tag, so ``crm_mon`` will not be able to report the DC otherwise. However, this does change the security model to allow by default, since any portions of the CIB not explicitly denied will be readable. The ``cib`` read access could be removed and replaced with read access to just the ``crm_config`` and ``status`` sections, for a safer approach at the cost of not seeing the DC in status output. For a simpler configuration, the ``minimal`` role allows read access to the entire ``crm_config`` section, which contains cluster properties. It would be possible to allow read access to specific properties instead (such as ``stonith-enabled``, ``dc-uuid``, ``have-quorum``, and ``cluster-name``) to restrict access further while still allowing status output, but cluster properties are unlikely to be considered sensitive. ACL Limitations ############### Actions performed via IPC rather than the CIB _____________________________________________ ACLs apply *only* to the CIB. That means ACLs apply to command-line tools that operate by reading or writing the CIB, such as ``crm_attribute`` when managing permanent node attributes, ``crm_mon``, and ``cibadmin``. However, command-line tools that communicate directly with Pacemaker daemons via IPC are not affected by ACLs. For example, users in the ``haclient`` group may still do the following, regardless of ACLs: * Query transient node attribute values using ``crm_attribute`` and ``attrd_updater``. * Query basic node information using ``crm_node``. * Erase resource operation history using ``crm_resource``. * Query fencing configuration information, and execute fencing against nodes, using ``stonith_admin``. ACLs and Pacemaker Remote _________________________ ACLs apply to commands run on Pacemaker Remote nodes using the Pacemaker Remote node's name as the ACL user name. The idea is that Pacemaker Remote nodes (especially virtual machines and containers) are likely to be purpose-built and have different user accounts from full cluster nodes. diff --git a/doc/sphinx/Pacemaker_Remote/baremetal-tutorial.rst b/doc/sphinx/Pacemaker_Remote/baremetal-tutorial.rst index a3c0fbeb19..7c23bd601c 100644 --- a/doc/sphinx/Pacemaker_Remote/baremetal-tutorial.rst +++ b/doc/sphinx/Pacemaker_Remote/baremetal-tutorial.rst @@ -1,288 +1,288 @@ .. index:: single: remote node; walk-through Remote Node Walk-through ------------------------ **What this tutorial is:** An in-depth walk-through of how to get Pacemaker to integrate a remote node into the cluster as a node capable of running cluster resources. **What this tutorial is not:** A realistic deployment scenario. The steps shown here are meant to get users familiar with the concept of remote nodes as quickly as possible. Configure Cluster Nodes ####################### This walk-through assumes you already have a Pacemaker cluster configured. For examples, we will use a cluster with two cluster nodes named pcmk-1 and pcmk-2. You can substitute whatever your node names are, for however many nodes you have. If you are not familiar with setting up basic Pacemaker clusters, follow the walk-through in the Clusters From Scratch document before attempting this one. Configure Remote Node ##################### .. index:: single: remote node; firewall Configure Firewall on Remote Node _________________________________ Allow cluster-related services through the local firewall: .. code-block:: none # firewall-cmd --permanent --add-service=high-availability success # firewall-cmd --reload success .. NOTE:: If you are using some other firewall solution besides firewalld, simply open the following ports, which can be used by various clustering components: TCP ports 2224, 3121, and 21064. If you run into any problems during testing, you might want to disable the firewall and SELinux entirely until you have everything working. This may create significant security issues and should not be performed on machines that will be exposed to the outside world, but may be appropriate during development and testing on a protected host. To disable security measures: .. code-block:: none # setenforce 0 # sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" \ /etc/selinux/config # systemctl mask firewalld.service # systemctl stop firewalld.service Configure ``/etc/hosts`` ________________________ You will need to add the remote node's hostname (we're using **remote1** in this tutorial) to the cluster nodes' ``/etc/hosts`` files if you haven't already. This is required unless you have DNS set up in a way where remote1's address can be discovered. For each remote node, execute the following on each cluster node and on the remote nodes, replacing the IP address with the actual IP address of the remote node. .. code-block:: none # cat << END >> /etc/hosts 192.168.122.10 remote1 END Also add entries for each cluster node to the ``/etc/hosts`` file on each remote node. For example: .. code-block:: none # cat << END >> /etc/hosts 192.168.122.101 pcmk-1 192.168.122.102 pcmk-2 END Configure pacemaker_remote on Remote Node _________________________________________ Install the pacemaker_remote daemon on the remote node. .. code-block:: none [root@remote1 ~]# dnf config-manager --set-enabled highavailability [root@remote1 ~]# dnf install -y pacemaker-remote resource-agents pcs Prepare ``pcsd`` ________________ Now we need to prepare ``pcsd`` on the remote node so that we can use ``pcs`` commands to communicate with it. Start and enable the ``pcsd`` daemon on the remote node. .. code-block:: none [root@remote1 ~]# systemctl start pcsd [root@remote1 ~]# systemctl enable pcsd Created symlink /etc/systemd/system/multi-user.target.wants/pcsd.service → /usr/lib/systemd/system/pcsd.service. -Next, set a password for the ``hacluster`` user on the remote node +Next, set a password for the |CRM_DAEMON_USER| user on the remote node .. code-block:: none [root@remote ~]# echo MyPassword | passwd --stdin hacluster Changing password for user hacluster. passwd: all authentication tokens updated successfully. Now authenticate the existing cluster nodes to ``pcsd`` on the remote node. The below command only needs to be run from one cluster node. .. code-block:: none [root@pcmk-1 ~]# pcs host auth remote1 -u hacluster Password: remote1: Authorized Integrate Remote Node into Cluster __________________________________ Integrating a remote node into the cluster is achieved through the creation of a remote node connection resource. The remote node connection resource both establishes the connection to the remote node and defines that the remote node exists. Note that this resource is actually internal to Pacemaker's controller. The metadata for this resource can be found in the ``/usr/lib/ocf/resource.d/pacemaker/remote`` file. The metadata in this file describes what options are available, but there is no actual **ocf:pacemaker:remote** resource agent script that performs any work. Define the remote node connection resource to our remote node, **remote1**, using the following command on any cluster node. This command creates the ocf:pacemaker:remote resource; creates the authkey if it does not exist already and distributes it to the remote node; and starts and enables ``pacemaker-remoted`` on the remote node. .. code-block:: none [root@pcmk-1 ~]# pcs cluster node add-remote remote1 No addresses specified for host 'remote1', using 'remote1' Sending 'pacemaker authkey' to 'remote1' remote1: successful distribution of the file 'pacemaker authkey' Requesting 'pacemaker_remote enable', 'pacemaker_remote start' on 'remote1' remote1: successful run of 'pacemaker_remote enable' remote1: successful run of 'pacemaker_remote start' That's it. After a moment you should see the remote node come online. The final ``pcs status`` output should look something like this, and you can see that it created the ocf:pacemaker:remote resource: .. code-block:: none [root@pcmk-1 ~]# pcs status Cluster name: mycluster Cluster Summary: * Stack: corosync * Current DC: pcmk-1 (version 2.1.2-4.el9-ada5c3b36e2) - partition with quorum * Last updated: Wed Aug 10 05:17:28 2022 * Last change: Wed Aug 10 05:17:26 2022 by root via cibadmin on pcmk-1 * 3 nodes configured * 2 resource instances configured Node List: * Online: [ pcmk-1 pcmk-2 ] * RemoteOnline: [ remote1 ] Full List of Resources: * xvm (stonith:fence_xvm): Started pcmk-1 * remote1 (ocf:pacemaker:remote): Started pcmk-1 Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled How pcs Configures the Remote ############################# Let's take a closer look at what the ``pcs cluster node add-remote`` command is doing. There is no need to run any of the commands in this section. First, ``pcs`` copies the Pacemaker authkey file to the VM that will become the guest. If an authkey is not already present on the cluster nodes, this command creates one and distributes it to the existing nodes and to the guest. If you want to do this manually, you can run a command like the following to generate an authkey in ``/etc/pacemaker/authkey``, and then distribute the key to the rest of the nodes and to the new guest. .. code-block:: none [root@pcmk-1 ~]# dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1 Then ``pcs`` starts and enables the ``pacemaker_remote`` service on the guest. If you want to do this manually, run the following commands. .. code-block:: none [root@guest1 ~]# systemctl start pacemaker_remote [root@guest1 ~]# systemctl enable pacemaker_remote Starting Resources on Remote Node ################################# Once the remote node is integrated into the cluster, starting and managing resources on a remote node is the exact same as on cluster nodes. Refer to the `Clusters from Scratch `_ document for examples of resource creation. .. WARNING:: Never involve a remote node connection resource in a resource group, colocation constraint, or order constraint. .. index:: single: remote node; fencing Fencing Remote Nodes #################### Remote nodes are fenced the same way as cluster nodes. No special considerations are required. Configure fencing resources for use with remote nodes the same as you would with cluster nodes. Note, however, that remote nodes can never 'initiate' a fencing action. Only cluster nodes are capable of actually executing a fencing operation against another node. Accessing Cluster Tools from a Remote Node ########################################## Besides allowing the cluster to manage resources on a remote node, pacemaker_remote has one other trick. The pacemaker_remote daemon allows nearly all the pacemaker tools (``crm_resource``, ``crm_mon``, ``crm_attribute``, etc.) to work on remote nodes natively. Try it: Run ``crm_mon`` on the remote node after pacemaker has integrated it into the cluster. These tools just work. These means resource agents such as promotable resources (which need access to tools like ``crm_attribute``) work seamlessly on the remote nodes. Higher-level command shells such as ``pcs`` may have partial support on remote nodes, but it is recommended to run them from a cluster node. Troubleshooting a Remote Connection ################################### If connectivity issues occur, it's worth verifying that the cluster nodes can communicate with the remote node on TCP port 3121. We can use the ``nc`` command to test the connection. On the cluster nodes, install the package that provides the ``nc`` command. The package name may vary by distribution; on |REMOTE_DISTRO| |REMOTE_DISTRO_VER| it's ``nmap-ncat``. Now connect using ``nc`` from each of the cluster nodes to the remote node and run a ``/bin/true`` command that does nothing except return success. No output indicates that the cluster node is able to communicate with the remote node on TCP port 3121. An error indicates that the connection failed. This could be due to a network issue or because ``pacemaker-remoted`` is not currently running on the remote node. Example of success: .. code-block:: none [root@pcmk-1 ~]# nc remote1 3121 --sh-exec /bin/true [root@pcmk-1 ~]# Examples of failure: .. code-block:: none [root@pcmk-1 ~]# nc remote1 3121 --sh-exec /bin/true Ncat: Connection refused. [root@pcmk-1 ~]# nc remote1 3121 --sh-exec /bin/true Ncat: No route to host. diff --git a/doc/sphinx/Pacemaker_Remote/kvm-tutorial.rst b/doc/sphinx/Pacemaker_Remote/kvm-tutorial.rst index 253149e6e4..ef098821e9 100644 --- a/doc/sphinx/Pacemaker_Remote/kvm-tutorial.rst +++ b/doc/sphinx/Pacemaker_Remote/kvm-tutorial.rst @@ -1,584 +1,584 @@ .. index:: single: guest node; walk-through Guest Node Walk-through ----------------------- **What this tutorial is:** An in-depth walk-through of how to get Pacemaker to manage a KVM guest instance and integrate that guest into the cluster as a guest node. **What this tutorial is not:** A realistic deployment scenario. The steps shown here are meant to get users familiar with the concept of guest nodes as quickly as possible. Configure Cluster Nodes ####################### This walk-through assumes you already have a Pacemaker cluster configured. For examples, we will use a cluster with two cluster nodes named pcmk-1 and pcmk-2. You can substitute whatever your node names are, for however many nodes you have. If you are not familiar with setting up basic Pacemaker clusters, follow the walk-through in the Clusters From Scratch document before attempting this one. Install Virtualization Software _______________________________ On each node within your cluster, install virt-install, libvirt, and qemu-kvm. Start and enable ``virtnetworkd``. .. code-block:: none # dnf install -y virt-install libvirt qemu-kvm # systemctl start virtnetworkd # systemctl enable virtnetworkd Reboot the host. .. NOTE:: While KVM is used in this example, any virtualization platform with a Pacemaker resource agent can be used to create a guest node. The resource agent needs only to support usual commands (start, stop, etc.); Pacemaker implements the **remote-node** meta-attribute, independent of the agent. Configure the KVM guest ####################### Create Guest ____________ Create a KVM guest to use as a guest node. Be sure to configure the guest with a hostname and a static IP address (as an example here, we will use guest1 and 192.168.122.10). Here's an example way to create a guest: * Download an .iso file from the |REMOTE_DISTRO| |REMOTE_DISTRO_VER| `mirrors list `_ into a directory on your cluster node. * Run the following command, using your own path for the **location** flag: .. code-block:: none [root@pcmk-1 ~]# virt-install \ --name vm-guest1 \ --memory 1536 \ --disk path=/var/lib/libvirt/images/vm-guest1.qcow2,size=4 \ --vcpus 2 \ --os-variant almalinux9 \ --network bridge=virbr0 \ --graphics none \ --console pty,target_type=serial \ --location /tmp/AlmaLinux-9-latest-x86_64-dvd.iso \ --extra-args 'console=ttyS0,115200n8' .. NOTE:: See the Clusters from Scratch document for more details about installing |REMOTE_DISTRO| |REMOTE_DISTRO_VER|. The above command will perform a text-based installation by default, but feel free to do a graphical installation, which exposes more options. .. index:: single: guest node; firewall Configure Firewall on Guest ___________________________ On each guest, allow cluster-related services through the local firewall. If you're using ``firewalld``, run the following commands. .. code-block:: none [root@guest1 ~]# firewall-cmd --permanent --add-service=high-availability success [root@guest1 ~]# firewall-cmd --reload success .. NOTE:: If you are using some other firewall solution besides firewalld, simply open the following ports, which can be used by various clustering components: TCP ports 2224, 3121, and 21064. If you run into any problems during testing, you might want to disable the firewall and SELinux entirely until you have everything working. This may create significant security issues and should not be performed on machines that will be exposed to the outside world, but may be appropriate during development and testing on a protected host. To disable security measures: .. code-block:: none [root@guest1 ~]# setenforce 0 [root@guest1 ~]# sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" \ /etc/selinux/config [root@guest1 ~]# systemctl mask firewalld.service [root@guest1 ~]# systemctl stop firewalld.service Configure ``/etc/hosts`` ________________________ You will need to add the remote node's hostname (we're using **guest1** in this tutorial) to the cluster nodes' ``/etc/hosts`` files if you haven't already. This is required unless you have DNS set up in a way where guest1's address can be discovered. For each guest, execute the following on each cluster node and on the guests, replacing the IP address with the actual IP address of the guest node. .. code-block:: none # cat << END >> /etc/hosts 192.168.122.10 guest1 END Also add entries for each cluster node to the ``/etc/hosts`` file on each guest. For example: .. code-block:: none # cat << END >> /etc/hosts 192.168.122.101 pcmk-1 192.168.122.102 pcmk-2 END Verify Connectivity ___________________ At this point, you should be able to ping and ssh into guests from hosts, and vice versa. Depending on your installation method, you may have to perform an additional step to make SSH work. The simplest approach is to open the ``/etc/ssh/sshd_config`` file and set ``PermitRootLogin yes``. Then to make the change take effect, run the following command. .. code-block:: none [root@guest1 ~]# systemctl restart sshd Configure pacemaker_remote on Guest Node ________________________________________ Install the pacemaker_remote daemon on the guest node. We'll also install the ``pacemaker`` package. It isn't required for a guest node to run, but it provides the ``crm_attribute`` tool, which many resource agents use. .. code-block:: none [root@guest1 ~]# dnf config-manager --set-enabled highavailability [root@guest1 ~]# dnf install -y pacemaker-remote resource-agents pcs \ pacemaker Integrate Guest into Cluster ############################ Now the fun part, integrating the virtual machine you've just created into the cluster. It is incredibly simple. Start the Cluster _________________ On the host, start Pacemaker if it's not already running. .. code-block:: none # pcs cluster start Create a ``VirtualDomain`` Resource for the Guest VM ____________________________________________________ For this simple walk-through, we have created the VM and made its disk available only on node ``pcmk-1``, so that's the only node where the VM is capable of running. In a more realistic scenario, you'll probably want to have multiple nodes that are capable of running the VM. Next we'll assign an attribute to node 1 that denotes its eligibility to host ``vm-guest1``. If other nodes are capable of hosting your guest VM, then add the attribute to each of those nodes as well. .. code-block:: none [root@pcmk-1 ~]# pcs node attribute pcmk-1 can-host-vm-guest1=1 Then we'll create a ``VirtualDomain`` resource so that Pacemaker can manage ``vm-guest1``. Be sure to replace the XML file path below with your own if it differs. We'll also create a rule to prevent Pacemaker from trying to start the resource or probe its status on any node that isn't capable of running the VM. We'll save the CIB to a file, make both of these edits, and push them simultaneously. .. code-block:: none [root@pcmk-1 ~]# pcs cluster cib vm_cfg [root@pcmk-1 ~]# pcs -f vm_cfg resource create vm-guest1 VirtualDomain \ hypervisor="qemu:///system" config="/etc/libvirt/qemu/vm-guest1.xml" Assumed agent name 'ocf:heartbeat:VirtualDomain' (deduced from 'VirtualDomain') [root@pcmk-1 ~]# pcs -f vm_cfg constraint location vm-guest1 rule \ resource-discovery=never score=-INFINITY can-host-vm-guest1 ne 1 [root@pcmk-1 ~]# pcs cluster cib-push --config vm_cfg --wait .. NOTE:: If all nodes in your cluster are capable of hosting the VM that you've created, then you can skip the ``pcs node attribute`` and ``pcs constraint location`` commands. .. NOTE:: The ID of the resource managing the virtual machine (``vm-guest1`` in the above example) **must** be different from the virtual machine's node name (``guest1`` in the above example). Pacemaker will create an implicit internal resource for the Pacemaker Remote connection to the guest. This implicit resource will be named with the value of the ``VirtualDomain`` resource's ``remote-node`` meta attribute, which will be set by ``pcs`` to the guest node's node name. Therefore, that value cannot be used as the name of any other resource. Now we can confirm that the ``VirtualDomain`` resource is running on ``pcmk-1``. .. code-block:: none [root@pcmk-1 ~]# pcs resource status * vm-guest1 (ocf:heartbeat:VirtualDomain): Started pcmk-1 Prepare ``pcsd`` ________________ Now we need to prepare ``pcsd`` on the guest so that we can use ``pcs`` commands to communicate with it. Start and enable the ``pcsd`` daemon on the guest. .. code-block:: none [root@guest1 ~]# systemctl start pcsd [root@guest1 ~]# systemctl enable pcsd Created symlink /etc/systemd/system/multi-user.target.wants/pcsd.service → /usr/lib/systemd/system/pcsd.service. -Next, set a password for the ``hacluster`` user on the guest. +Next, set a password for the |CRM_DAEMON_USER| user on the guest. .. code-block:: none [root@guest1 ~]# echo MyPassword | passwd --stdin hacluster Changing password for user hacluster. passwd: all authentication tokens updated successfully. Now authenticate the existing cluster nodes to ``pcsd`` on the guest. The below command only needs to be run from one cluster node. .. code-block:: none [root@pcmk-1 ~]# pcs host auth guest1 -u hacluster Password: guest1: Authorized Integrate Guest Node into Cluster _________________________________ We're finally ready to integrate the VM into the cluster as a guest node. Run the following command, which will create a guest node from the ``VirtualDomain`` resource and take care of all the remaining steps. Note that the format is ``pcs cluster node add-guest ``. .. code-block:: none [root@pcmk-1 ~]# pcs cluster node add-guest guest1 vm-guest1 No addresses specified for host 'guest1', using 'guest1' Sending 'pacemaker authkey' to 'guest1' guest1: successful distribution of the file 'pacemaker authkey' Requesting 'pacemaker_remote enable', 'pacemaker_remote start' on 'guest1' guest1: successful run of 'pacemaker_remote enable' guest1: successful run of 'pacemaker_remote start' You should soon see ``guest1`` appear in the ``pcs status`` output as a node. The output should look something like this: .. code-block:: none [root@pcmk-1 ~]# pcs status Cluster name: mycluster Cluster Summary: * Stack: corosync * Current DC: pcmk-1 (version 2.1.2-4.el9-ada5c3b36e2) - partition with quorum * Last updated: Wed Aug 10 00:08:58 2022 * Last change: Wed Aug 10 00:02:37 2022 by root via cibadmin on pcmk-1 * 3 nodes configured * 3 resource instances configured Node List: * Online: [ pcmk-1 pcmk-2 ] * GuestOnline: [ guest1@pcmk-1 ] Full List of Resources: * xvm (stonith:fence_xvm): Started pcmk-1 * vm-guest1 (ocf:heartbeat:VirtualDomain): Started pcmk-1 Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled The resulting configuration should look something like the following: .. code-block:: none [root@pcmk-1 ~]# pcs resource config Resource: vm-guest1 (class=ocf provider=heartbeat type=VirtualDomain) Attributes: config=/etc/libvirt/qemu/vm-guest1.xml hypervisor=qemu:///system Meta Attrs: remote-addr=guest1 remote-node=guest1 Operations: migrate_from interval=0s timeout=60s (vm-guest1-migrate_from-interval-0s) migrate_to interval=0s timeout=120s (vm-guest1-migrate_to-interval-0s) monitor interval=10s timeout=30s (vm-guest1-monitor-interval-10s) start interval=0s timeout=90s (vm-guest1-start-interval-0s) stop interval=0s timeout=90s (vm-guest1-stop-interval-0s) How pcs Configures the Guest ____________________________ Let's take a closer look at what the ``pcs cluster node add-guest`` command is doing. There is no need to run any of the commands in this section. First, ``pcs`` copies the Pacemaker authkey file to the VM that will become the guest. If an authkey is not already present on the cluster nodes, this command creates one and distributes it to the existing nodes and to the guest. If you want to do this manually, you can run a command like the following to generate an authkey in ``/etc/pacemaker/authkey``, and then distribute the key to the rest of the nodes and to the new guest. .. code-block:: none [root@pcmk-1 ~]# dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1 Then ``pcs`` starts and enables the ``pacemaker_remote`` service on the guest. If you want to do this manually, run the following commands. .. code-block:: none [root@guest1 ~]# systemctl start pacemaker_remote [root@guest1 ~]# systemctl enable pacemaker_remote Finally, ``pcs`` creates a guest node from the ``VirtualDomain`` resource by adding ``remote-addr`` and ``remote-node`` meta attributes to the resource. If you want to do this manually, you can run the following command if you're using ``pcs``. Alternativately, run an equivalent command if you're using another cluster shell, or edit the CIB manually. .. code-block:: none [root@pcmk-1 ~]# pcs resource update vm-guest1 meta remote-addr='guest1' \ remote-node='guest1' --force Starting Resources on KVM Guest ############################### The following example demonstrates that resources can be run on the guest node in the exact same way as on the cluster nodes. Create a few ``Dummy`` resources. A ``Dummy`` resource is a real resource that actually executes operations on its assigned node. However, these operations are trivial (creating, deleting, or checking the existence of an empty or small file), so ``Dummy`` resources are ideal for testing purposes. ``Dummy`` resources use the ``ocf:heartbeat:Dummy`` or ``ocf:pacemaker:Dummy`` resource agent. .. code-block:: none # for i in {1..5}; do pcs resource create FAKE${i} ocf:heartbeat:Dummy; done Now run ``pcs resource status``. You should see something like the following, where some of the resources are started on the cluster nodes, and some are started on the guest node. .. code-block:: none [root@pcmk-1 ~]# pcs resource status * vm-guest1 (ocf:heartbeat:VirtualDomain): Started pcmk-1 * FAKE1 (ocf:heartbeat:Dummy): Started guest1 * FAKE2 (ocf:heartbeat:Dummy): Started pcmk-2 * FAKE3 (ocf:heartbeat:Dummy): Started guest1 * FAKE4 (ocf:heartbeat:Dummy): Started pcmk-2 * FAKE5 (ocf:heartbeat:Dummy): Started guest1 The guest node, ``guest1``, behaves just like any other node in the cluster with respect to resources. For example, choose a resource that is running on one of your cluster nodes. We'll choose ``FAKE2`` from the output above. It's currently running on ``pcmk-2``. We can force ``FAKE2`` to run on ``guest1`` in the exact same way as we could force it to run on any particular cluster node. We do this by creating a location constraint: .. code-block:: none # pcs constraint location FAKE2 prefers guest1 Now the ``pcs resource status`` output shows that ``FAKE2`` is on ``guest1``. .. code-block:: none [root@pcmk-1 ~]# pcs resource status * vm-guest1 (ocf:heartbeat:VirtualDomain): Started pcmk-1 * FAKE1 (ocf:heartbeat:Dummy): Started guest1 * FAKE2 (ocf:heartbeat:Dummy): Started guest1 * FAKE3 (ocf:heartbeat:Dummy): Started guest1 * FAKE4 (ocf:heartbeat:Dummy): Started pcmk-2 * FAKE5 (ocf:heartbeat:Dummy): Started guest1 Testing Recovery and Fencing ############################ Pacemaker's scheduler is smart enough to know fencing guest nodes associated with a virtual machine means shutting off/rebooting the virtual machine. No special configuration is necessary to make this happen. If you are interested in testing this functionality out, trying stopping the guest's pacemaker_remote daemon. This would be equivalent of abruptly terminating a cluster node's corosync membership without properly shutting it down. ssh into the guest and run this command. .. code-block:: none [root@guest1 ~]# kill -9 $(pidof pacemaker-remoted) Within a few seconds, your ``pcs status`` output will show a monitor failure, and the **guest1** node will not be shown while it is being recovered. .. code-block:: none [root@pcmk-1 ~]# pcs status Cluster name: mycluster Cluster Summary: * Stack: corosync * Current DC: pcmk-1 (version 2.1.2-4.el9-ada5c3b36e2) - partition with quorum * Last updated: Wed Aug 10 01:39:40 2022 * Last change: Wed Aug 10 01:34:55 2022 by root via cibadmin on pcmk-1 * 3 nodes configured * 8 resource instances configured Node List: * Online: [ pcmk-1 pcmk-2 ] Full List of Resources: * xvm (stonith:fence_xvm): Started pcmk-1 * vm-guest1 (ocf:heartbeat:VirtualDomain): FAILED pcmk-1 * FAKE1 (ocf:heartbeat:Dummy): FAILED guest1 * FAKE2 (ocf:heartbeat:Dummy): FAILED guest1 * FAKE3 (ocf:heartbeat:Dummy): FAILED guest1 * FAKE4 (ocf:heartbeat:Dummy): Started pcmk-2 * FAKE5 (ocf:heartbeat:Dummy): FAILED guest1 Failed Resource Actions: * guest1 30s-interval monitor on pcmk-1 could not be executed (Error) because 'Lost connection to remote executor' at Wed Aug 10 01:39:38 2022 Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled .. NOTE:: A guest node involves two resources: an explicitly configured resource that you create, which manages the virtual machine (the ``VirtualDomain`` resource in our example); and an implicit resource that Pacemaker creates, which manages the ``pacemaker-remoted`` connection to the guest. The implicit resource's name is the value of the explicit resource's ``remote-node`` meta attribute. When we killed ``pacemaker-remoted``, the **implicit** resource is what failed. That's why the failed action starts with ``guest1`` and not ``vm-guest1``. Once recovery of the guest is complete, you'll see it automatically get re-integrated into the cluster. The final ``pcs status`` output should look something like this. .. code-block:: none [root@pcmk-1 ~]# pcs status Cluster name: mycluster Cluster Summary: * Stack: corosync * Current DC: pcmk-1 (version 2.1.2-4.el9-ada5c3b36e2) - partition with quorum * Last updated: Wed Aug 10 01:40:05 2022 * Last change: Wed Aug 10 01:34:55 2022 by root via cibadmin on pcmk-1 * 3 nodes configured * 8 resource instances configured Node List: * Online: [ pcmk-1 pcmk-2 ] * GuestOnline: [ guest1@pcmk-1 ] Full List of Resources: * xvm (stonith:fence_xvm): Started pcmk-1 * vm-guest1 (ocf:heartbeat:VirtualDomain): Started pcmk-1 * FAKE1 (ocf:heartbeat:Dummy): Started guest1 * FAKE2 (ocf:heartbeat:Dummy): Started guest1 * FAKE3 (ocf:heartbeat:Dummy): Started pcmk-2 * FAKE4 (ocf:heartbeat:Dummy): Started pcmk-2 * FAKE5 (ocf:heartbeat:Dummy): Started guest1 Failed Resource Actions: * guest1 30s-interval monitor on pcmk-1 could not be executed (Error) because 'Lost connection to remote executor' at Wed Aug 10 01:39:38 2022 Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled Normally, once you've investigated and addressed a failed action, you can clear the failure. However Pacemaker does not yet support cleanup for the implicitly created connection resource while the explicit resource is active. If you want to clear the failed action from the status output, stop the guest resource before clearing it. For example: .. code-block:: none # pcs resource disable vm-guest1 --wait # pcs resource cleanup guest1 # pcs resource enable vm-guest1 Accessing Cluster Tools from Guest Node ####################################### Besides allowing the cluster to manage resources on a guest node, pacemaker_remote has one other trick. The pacemaker_remote daemon allows nearly all the pacemaker tools (``crm_resource``, ``crm_mon``, ``crm_attribute``, etc.) to work on guest nodes natively. Try it: Run ``crm_mon`` on the guest after pacemaker has integrated the guest node into the cluster. These tools just work. This means resource agents such as promotable resources (which need access to tools like ``crm_attribute``) work seamlessly on the guest nodes. Higher-level command shells such as ``pcs`` may have partial support on guest nodes, but it is recommended to run them from a cluster node. Troubleshooting a Remote Connection ################################### If connectivity issues occur, it's worth verifying that the cluster nodes can communicate with the guest node on TCP port 3121. We can use the ``nc`` command to test the connection. On the cluster nodes, install the package that provides the ``nc`` command. The package name may vary by distribution; on |REMOTE_DISTRO| |REMOTE_DISTRO_VER| it's ``nmap-ncat``. Now connect using ``nc`` from each of the cluster nodes to the guest and run a ``/bin/true`` command that does nothing except return success. No output indicates that the cluster node is able to communicate with the guest on TCP port 3121. An error indicates that the connection failed. This could be due to a network issue or because ``pacemaker-remoted`` is not currently running on the guest node. Example of success: .. code-block:: none [root@pcmk-1 ~]# nc guest1 3121 --sh-exec /bin/true [root@pcmk-1 ~]# Examples of failure: .. code-block:: none [root@pcmk-1 ~]# nc guest1 3121 --sh-exec /bin/true Ncat: Connection refused. [root@pcmk-1 ~]# nc guest1 3121 --sh-exec /bin/true Ncat: No route to host.