diff --git a/doc/Pacemaker_Explained/en-US/Ap-Debug.xml b/doc/Pacemaker_Explained/en-US/Ap-Debug.xml index bfe3f171e6..165ff1a30a 100644 --- a/doc/Pacemaker_Explained/en-US/Ap-Debug.xml +++ b/doc/Pacemaker_Explained/en-US/Ap-Debug.xml @@ -1,128 +1,128 @@ Debugging Cluster Startup
- OpenAIS + Corosync
Prerequisites Minimum logging configuration # /etc/init.d/openais start logging { to_syslog: yes syslog_facility: daemon } Whatever other logging you have, these two lines are required for Pacemaker clusters
- Confirm OpenAIS Started + Confirm Corosync Started
Expected output when starting openais # /etc/init.d/openais start - Starting OpenAIS daemon (aisexec): starting... rc=0: OK + Starting Corosync daemon (aisexec): starting... rc=0: OK
Expected log messages - startup # grep -e "openais.*network interface" -e "AIS Executive Service" /var/log/messages Aug 27 16:23:37 test1 openais[26337]: [MAIN ] AIS Executive Service RELEASE 'subrev 1152 version 0.80' Aug 27 16:23:38 test1 openais[26337]: [MAIN ] AIS Executive Service: started and ready to provide service. Aug 27 16:23:38 test1 openais[26337]: [TOTEM] The network interface [192.168.9.41] is now up. - The versions may differ, but you should see OpenAIS indicate it started and sucessfully attached to the machine's network interface + The versions may differ, but you should see Corosync indicate it started and sucessfully attached to the machine's network interface
Expected log messages - membership # grep CLM /var/log/messages Aug 27 16:53:15 test1 openais[2166]: [CLM ] CLM CONFIGURATION CHANGE Aug 27 16:53:15 test1 openais[2166]: [CLM ] New Configuration: Aug 27 16:53:15 test1 openais[2166]: [CLM ] Members Left: Aug 27 16:53:15 test1 openais[2166]: [CLM ] Members Joined: Aug 27 16:53:15 test1 openais[2166]: [CLM ] CLM CONFIGURATION CHANGE Aug 27 16:53:15 test1 openais[2166]: [CLM ] New Configuration: Aug 27 16:53:15 test1 openais[2166]: [CLM ] r(0) ip(192.168.9.41) Aug 27 16:53:15 test1 openais[2166]: [CLM ] Members Left: Aug 27 16:53:15 test1 openais[2166]: [CLM ] Members Joined: Aug 27 16:53:15 test1 openais[2166]: [CLM ] r(0) ip(192.168.9.41) Aug 27 16:53:15 test1 openais[2166]: [CLM ] got nodejoin message 192.168.9.41 The exact messages will differ, but you should see a new membership formed with the real IP address of your node
Checking Pacemaker - Now that we have confirmed that OpenAIS is functional we can check the rest of the stack. + Now that we have confirmed that Corosync is functional we can check the rest of the stack.
- Expected Pacemaker startup logging for OpenAIS + Expected Pacemaker startup logging for Corosync # grep pcmk_plugin_init /var/log/messages Aug 27 16:53:15 test1 openais[2166]: [pcmk ] info: pcmk_plugin_init: CRM: Initialized Aug 27 16:53:15 test1 openais[2166]: [pcmk ] Logging: Initialized pcmk_plugin_init Aug 27 16:53:15 test1 openais[2166]: [pcmk ] info: pcmk_plugin_init: Service: 9 Aug 27 16:53:15 test1 openais[2166]: [pcmk ] info: pcmk_plugin_init: Local hostname: test1 If you don't see these messages, or some like them, there is likely a problem finding or loading the pacemaker plugin.
Expected process listing on a 64-bit machine # ps axf 3718 ? Ssl 0:05 /usr/sbin/aisexec 3723 ? SLs 0:00 \_ /usr/lib64/heartbeat/stonithd 3724 ? S 0:05 \_ /usr/lib64/heartbeat/cib 3725 ? S 0:21 \_ /usr/lib64/heartbeat/lrmd 3726 ? S 0:01 \_ /usr/lib64/heartbeat/attrd 3727 ? S 0:00 \_ /usr/lib64/heartbeat/pengine 3728 ? S 0:01 \_ /usr/lib64/heartbeat/crmd On 32-bit systems the exact path may differ, but all the above processes should be listed.
diff --git a/doc/Pacemaker_Explained/en-US/Ap-FAQ.xml b/doc/Pacemaker_Explained/en-US/Ap-FAQ.xml index 1f12328c18..4b1e58b2c9 100644 --- a/doc/Pacemaker_Explained/en-US/Ap-FAQ.xml +++ b/doc/Pacemaker_Explained/en-US/Ap-FAQ.xml @@ -1,96 +1,96 @@ FAQ History Why is the Project Called Pacemaker? First of all, the reason its not called the CRM is because of the abundance of terms that are commonly abbreviated to those three letters. The Pacemaker name came from Kham, a good friend of mine, and was originally used by a Java GUI that I was prototyping in early 2007. Alas other commitments have prevented the GUI from progressing much and, when it came time to choose a name for this project, Lars suggested it was an even better fit for an independent CRM. The idea stems from the analogy between the role of this software and that of the little device that keeps the human heart pumping. Pacemaker monitors the cluster and intervenes when necessary to ensure the smooth operation of the services it provides. There were a number of other names (and acronyms) tossed around, but suffice to say "Pacemaker" was the best Why was the Pacemaker Project Created? The decision was made to spin-off the CRM into its own project after the 2.1.3 Heartbeat release in order to - support both the OpenAIS and Heartbeat cluster stacks equally + support both the Corosync and Heartbeat cluster stacks equally decouple the release cycles of two projects at very different stages of their life-cycles foster the clearer package boundaries, thus leading to better and more stable interfaces Setup What Messaging Layers are Supported? - OpenAIS () + Corosync () Heartbeat () Can I Choose which Messaging Layer to use at Run Time? Yes. The CRM will automatically detect who started it and behave accordingly. - Can I Have a Mixed Heartbeat-OpenAIS Cluster? + Can I Have a Mixed Heartbeat-Corosync Cluster? No. Which Messaging Layer Should I Choose? This is discussed in . Where Can I Get Pre-built Packages? Official packages for most major .rpm and based distributions are available from: For Debian packages, building from source and details on using the above repositories, see our installation page. What Versions of Pacemaker Are Supported? Please refer to the Releases page for an up-to-date list of versions supported directly by the project. When seeking assistance, please try to ensure you have one of these versions. diff --git a/doc/Pacemaker_Explained/en-US/Ap-Install.xml b/doc/Pacemaker_Explained/en-US/Ap-Install.xml index 23696c04f7..f6fe4578f0 100644 --- a/doc/Pacemaker_Explained/en-US/Ap-Install.xml +++ b/doc/Pacemaker_Explained/en-US/Ap-Install.xml @@ -1,108 +1,108 @@ Installation
Choosing a Cluster Stack Ultimately the choice of cluster stack is a personal decision that must be made in the context of you or your company's needs and strategic direction. Pacemaker currently functions equally well with both stacks. Here are some factors that may influence the decision - SUSE/Novell, Red Hat and Oracle are all putting their collective weight behind the OpenAIS cluster stack. + SUSE/Novell, Red Hat and Oracle are all putting their collective weight behind the Corosync cluster stack. - OpenAIS is an OSI Certified implementation of an industry standard (the Service Availability Forum Application Interface Specification). + Corosync is an OSI Certified implementation of an industry standard (the Service Availability Forum Application Interface Specification). - Using OpenAIS gives your applications access to the following additional cluster services + Using Corosync gives your applications access to the following additional cluster services checkpoint service distributed locking service extended virtual synchrony service cluster closed process group service It is likely that Pacemaker, at some point in the future, will make use of some of these additional services not provided by Heartbeat - To date, Pacemaker has received less real-world testing on OpenAIS than it has on Heartbeat. + To date, Pacemaker has received less real-world testing on Corosync than it has on Heartbeat.
Enabling Pacemaker
- For OpenAIS - The OpenAIS configuration is normally located in /etc/ais/openais.conf and an example for a machine with an address of 1.2.3.4 in a cluster communicating on port 1234 (without peer authentication and message encryption) is shown below. + For Corosync + The Corosync configuration is normally located in /etc/corosync/corosync.conf and an example for a machine with an address of 1.2.3.4 in a cluster communicating on port 1234 (without peer authentication and message encryption) is shown below. - An example OpenAIS configuration file + An example Corosync configuration file totem { version: 2 secauth: off threads: 0 interface { ringnumber: 0 bindnetaddr: 1.2.3.4 mcastaddr: 226.94.1.1 mcastport: 1234 } } logging { fileline: off to_syslog: yes syslog_facility: daemon } amf { mode: disabled } The logging should be mostly obvious and the amf section refers to the Availability Management Framework and is not covered in this document. The interesting part of the configuration is the totem section. This is where we define the how the node can communicate with the rest of the cluster and what protocol version and options (including encryption - Please consult the OpenAIS website and documentation for details on enabling encryption and peer authentication for the cluster. + Please consult the Corosync website and documentation for details on enabling encryption and peer authentication for the cluster. ) it should use. Beginners are encouraged to use the values shown and modify the interface section based on their network. - It is also possible to configure OpenAIS for an IPv6 based environment. Simply configure bindnetaddr and mcastaddr with their IPv6 equivalents. Eg + It is also possible to configure Corosync for an IPv6 based environment. Simply configure bindnetaddr and mcastaddr with their IPv6 equivalents. Eg Example options for an IPv6 environment bindnetaddr: fec0::1:a800:4ff:fe00:20 mcastaddr: ff05::1 - To tell OpenAIS to use the Pacemaker cluster manager, add the following fragment to a functional OpenAIS configuration and restart the cluster. + To tell Corosync to use the Pacemaker cluster manager, add the following fragment to a functional Corosync configuration and restart the cluster. - Configuration fragment for enabling Pacemaker under OpenAIS + Configuration fragment for enabling Pacemaker under Corosync aisexec { user: root group: root } service { name: pacemaker ver: 0 } The cluster needs to be run as root so that its child processes (the lrmd in particular) have sufficient privileges to perform the actions requested of it. After-all, a cluster manager that can't add an IP address or start apache is of little use. The second directive is the one that actually instructs the cluster to run Pacemaker.
For Heartbeat Add the following to a functional ha.cf configuration file and restart Heartbeat Configuration fragment for enabling Pacemaker under Heartbeat crm respawn
diff --git a/doc/Pacemaker_Explained/en-US/Ap-Upgrade.xml b/doc/Pacemaker_Explained/en-US/Ap-Upgrade.xml index 241dfef849..6e399d6a0e 100644 --- a/doc/Pacemaker_Explained/en-US/Ap-Upgrade.xml +++ b/doc/Pacemaker_Explained/en-US/Ap-Upgrade.xml @@ -1,249 +1,249 @@ Upgrading Cluster Software
Version Compatibility When releasing newer versions we take care to make sure we are backwardly compatible with older versions. While you will always be able to upgrade from version x to x+1, in order to continue to produce high quality software it may occasionally be necessary to drop compatibility with older versions. There will always be an upgrade path from any series-2 release to any other series-2 release. There are three approaches to upgrading your cluster software Complete Cluster Shutdown Rolling (node by node) Disconnect and Reattach Each method has advantages and disadvantages, some of which are listed in the table below, and you should chose the one most appropriate to your needs. Summary of Upgrade Methodologies Type Available between all software versions Service Outage During Upgrade Service Recovery During Upgrade Exercises Failover Logic/Configuration Allows change of cluster stack type - For example, switching from Heartbeat to OpenAIS. - Consult the Heartbeat or OpenAIS documentation to see if upgrading them to a newer version is also supported + For example, switching from Heartbeat to Corosync. + Consult the Heartbeat or Corosync documentation to see if upgrading them to a newer version is also supported Shutdown yes always N/A no yes Rolling no always yes yes no Reattach yes only due to failure no no yes
Complete Cluster Shutdown In this scenario one shuts down all cluster nodes and resources and upgrades all the nodes before restarting the cluster.
Procedure On each node: - Shutdown the cluster stack (Heartbeat or OpenAIS) + Shutdown the cluster stack (Heartbeat or Corosync) Upgrade the Pacemaker software. This may also include upgrading the cluster stack and/or the underlying operating system.. Check the configuration manually or with the crm_verify tool if available. On each node: Start the cluster stack. - This can be either OpenAIS or Heartbeat and does not need to be the same as the previous cluster stack. + This can be either Corosync or Heartbeat and does not need to be the same as the previous cluster stack.
Rolling (node by node) In this scenario each node is removed from the cluster, upgraded and then brought back online until all nodes are running the newest version. This method is currently broken between Pacemaker 0.6.x and 1.0.x Measures have been put into place to ensure rolling upgrades always work for versions after 1.0.0 If there is sufficient demand, the work to repair 0.6 -> 1.0 compatibility will be carried out. Otherwise, please try one of the other upgrade strategies. Detach/Reattach is a particularly good option for most people.
Procedure On each node: - Shutdown the cluster stack (Heartbeat or OpenAIS) + Shutdown the cluster stack (Heartbeat or Corosync) Upgrade the Pacemaker software. This may also include upgrading the cluster stack and/or the underlying operating system. On the first node, check the configuration manually or with the crm_verify tool if available. Start the cluster stack. - This must be the same type of cluster stack (OpenAIS or Heartbeat) that the rest of the cluster is using. - Upgrading OpenAIS/Heartbeat may also be possible, please consult the documentation for those projects to see if the two versions will be compatible. + This must be the same type of cluster stack (Corosync or Heartbeat) that the rest of the cluster is using. + Upgrading Corosync/Heartbeat may also be possible, please consult the documentation for those projects to see if the two versions will be compatible. Repeat for each node in the cluster
Version Compatibility Version Compatibility Table Version being Installed Oldest Compatible Version Pacemaker 1.0.x Pacemaker 1.0.0 Pacemaker 0.7.x Pacemaker 0.6 or Heartbeat 2.1.3 Pacemaker 0.6.x Heartbeat 2.0.8 Heartbeat 2.1.3 (or less) Heartbeat 2.0.4 Heartbeat 2.0.4 (or less) Heartbeat 2.0.0 Heartbeat 2.0.0 None. Use an alternate upgrade strategy.
Crossing Compatibility Boundaries Rolling upgrades that cross compatibility boundaries must be preformed in multiple steps. For example, to perform a rolling update from Heartbeat 2.0.1 to Pacemaker 0.6.6 one must: Perform a rolling upgrade from Heartbeat 2.0.1 to Heartbeat 2.0.4 Perform a rolling upgrade from Heartbeat 2.0.4 to Heartbeat 2.1.3 Perform a rolling upgrade from Heartbeat 2.1.3 to Pacemaker 0.6.6
Disconnect and Reattach A variant of a complete cluster shutdown, but the resources are left active and re-detected when the cluster is restarted.
Procedure Tell the cluster to stop managing services. This is required to allow the services to remain active after the cluster shuts down. crm_attribute -t crm_config -n is-managed-default -v false For any resource that has a value for is-managed, make sure it is set to false (so that the cluster will not stop it) crm_resource -t primitive -r <rsc_id> -p is-managed -v false On each node: - Shutdown the cluster stack (Heartbeat or OpenAIS) + Shutdown the cluster stack (Heartbeat or Corosync) Upgrade the cluster stack program - This may also include upgrading the underlying operating system. Check the configuration manually or with the crm_verify tool if available. On each node: Start the cluster stack. - This can be either OpenAIS or Heartbeat and does not need to be the same as the previous cluster stack. + This can be either Corosync or Heartbeat and does not need to be the same as the previous cluster stack. Verify the cluster re-detected all resources correctly Allow the cluster to resume managing resources again crm_attribute -t crm_config -n is-managed-default -v true For any resource that has a value for is-managed reset it to true (so the cluster can recover the service if it fails) if desired crm_resource -t primitive -r <rsc_id> -p is-managed -v false
Notes Always check your existing configuration is still compatible with the version you are installing before starting the cluster. The oldest version of the CRM to support this upgrade type was in Heartbeat 2.0.4
diff --git a/doc/Pacemaker_Explained/en-US/Ch-Intro.xml b/doc/Pacemaker_Explained/en-US/Ch-Intro.xml index e6972209a4..c2734a7832 100644 --- a/doc/Pacemaker_Explained/en-US/Ch-Intro.xml +++ b/doc/Pacemaker_Explained/en-US/Ch-Intro.xml @@ -1,175 +1,175 @@ Read-Me-First
The Scope of this Document The purpose of this document is to definitively explain the concepts used to configure Pacemaker. To achieve this best, it will focus exclusively on the XML syntax used to configure the CIB. For those that are allergic to XML, Pacemaker comes with a cluster shell and a Python based GUI exists, however these tools will not be covered at all in this document It is hoped however, that having understood the concepts explained here, that the functionality of these tools will also be more readily understood. , precisely because they hide the XML. Additionally, this document is NOT a step-by-step how-to guide for configuring a specific clustering scenario. Although such guides exist, the purpose of this document is to provide an understanding of the building blocks that can be used to construct any type of Pacemaker cluster.
What Is Pacemaker? Pacemaker is a cluster resource manager. - It achieves maximum availability for your cluster services (aka. resources) by detecting and recovering from node and resource-level failures by making use of the messaging and membership capabilities provided by your preferred cluster infrastructure (either OpenAIS or Heartbeat). + It achieves maximum availability for your cluster services (aka. resources) by detecting and recovering from node and resource-level failures by making use of the messaging and membership capabilities provided by your preferred cluster infrastructure (either Corosync or Heartbeat). Pacemaker's key features include: Detection and recovery of node and service-level failures Storage agnostic, no requirement for shared storage Resource agnostic, anything that can be scripted can be clustered Supports STONITH for ensuring data integrity Supports large and small clusters Supports both quorate and resource driven clusters Supports practically any redundancy configuration Automatically replicated configuration that can be updated from any node Ability to specify cluster-wide service ordering, colocation and anti-colocation Support for advanced services type Clones: for services which need to be active on multiple nodes Multi-state: for services with multiple modes (eg. master/slave, primary/secondary) Unified, scriptable, cluster shell
Types of Pacemaker Clusters Pacemaker makes no assumptions about your environment, this allows it to support practically any redundancy configuration including Active/Active, Active/Passive, N+1, N+M, N-to-1 and N-to-N.
Active/Passive Redundancy Two-node Active/Passive clusters using Pacemaker and DRBD are a cost-effective solution for many High Availability situations.
Shared Failover By supporting many nodes, Pacemaker can dramatically reduce hardware costs by allowing several active/passive clusters to be combined and share a common backup node
N to N Redundancy When shared storage is available, every node can potentially be used for failover. Pacemaker can even run multiple copies of services to spread out the workload.
Pacemaker Architecture At the highest level, the cluster is made up of three pieces: Core cluster infrastructure providing messaging and membership functionality (illustrated in red) Non-cluster aware components (illustrated in blue). In a Pacemaker cluster, these pieces include not only the scripts that knows how to start, stop and monitor resources, but also a local daemon that masks the differences between the different standards these scripts implement. A brain (illustrated in green) that processes and reacts to events from the cluster (nodes leaving or joining) and resources (eg. monitor failures) as well as configuration changes from the administrator. In response to all of these events, Pacemaker will compute the ideal state of the cluster and plot a path to achieve it. This may include moving resources, stopping nodes and even forcing them offline with remote power switches.
Conceptual Stack Overview Conceptual overview of the cluster stack
- When combined with OpenAIS, Pacemaker also supports popular open source cluster filesystems + When combined with Corosync, Pacemaker also supports popular open source cluster filesystems - Even though Pacemaker also supports Heartbeat, the filesystems need to use the stack for messaging and membership and OpenAIS seems to be what they're standardizing on. + Even though Pacemaker also supports Heartbeat, the filesystems need to use the stack for messaging and membership and Corosync seems to be what they're standardizing on. Technically it would be possible for them to support Heartbeat as well, however there seems little interest in this. - Due to recent standardization within the cluster filesystem community, they make use of a common distributed lock manager which makes use of OpenAIS for its messaging capabilities and Pacemaker for its membership (which nodes are up/down) and fencing services. + Due to recent standardization within the cluster filesystem community, they make use of a common distributed lock manager which makes use of Corosync for its messaging capabilities and Pacemaker for its membership (which nodes are up/down) and fencing services.
The Pacemaker Stack - The Pacemaker stack when running on OpenAIS + The Pacemaker stack when running on Corosync
Internal Components Pacemaker itself is composed of four key components (illustrated below in the same color scheme as the previous diagram): CIB (aka. Cluster Information Base) CRMd (aka. Cluster Resource Management daemon) PEngine (aka. PE or Policy Engine) STONITHd
Internal Components - Subsystems of a Pacemaker cluster running on OpenAIS + Subsystems of a Pacemaker cluster running on Corosync
The CIB uses XML to represent both the cluster's configuration and current state of all resources in the cluster. The contents of the CIB are automatically kept in sync across the entire cluster and are used by the PEngine to compute the ideal state of the cluster and how it should be achieved. This list of instructions is then fed to the DC (Designated Co-ordinator). Pacemaker centralizes all cluster decision making by electing one of the CRMd instances to act as a master. Should the elected CRMd process, or the node it is on, fail... a new one is quickly established. The DC carries out the PEngine's instructions in the required order by passing them to either the LRMd (Local Resource Management daemon) or CRMd peers on other nodes via the cluster messaging infrastructure (which in turn passes them on to their LRMd process). The peer nodes all report the results of their operations back to the DC and based on the expected and actual results, will either execute any actions that needed to wait for the previous one to complete, or abort processing and ask the PEngine to recalculate the ideal cluster state based on the unexpected results. In some cases, it may be necessary to power off nodes in order to protect shared data or complete resource recovery. For this Pacemaker comes with STONITHd. STONITH is an acronym for Shoot-The-Other-Node-In-The-Head and is usually implemented with a remote power switch. In Pacemaker, STONITH devices are modeled as resources (and configured in the CIB) to enable them to be easily monitored for failure, however STONITHd takes care of understanding the STONITH topology such that its clients simply request a node be fenced and it does the rest.
diff --git a/doc/Pacemaker_Explained/en-US/Ch-Nodes.xml b/doc/Pacemaker_Explained/en-US/Ch-Nodes.xml index 8deea89aa0..e374449b43 100644 --- a/doc/Pacemaker_Explained/en-US/Ch-Nodes.xml +++ b/doc/Pacemaker_Explained/en-US/Ch-Nodes.xml @@ -1,132 +1,132 @@ Cluster Nodes
Defining a Cluster Node Each node in the cluster will have an entry in the nodes section containing its UUID, uname and type. Example cluster node entry ]]> In normal circumstances, the admin should let the cluster populate this information automatically from the communications and membership data. However one can use the crm_uuid tool to read an existing UUID or define a value before the cluster starts.
Describing a Cluster Node Beyond the basic definition of a node, the administrator can also describe the node's attributes, such as how much RAM, disk, what OS or kernel version it has, perhaps even its physical location. This information can then be used by the cluster when deciding where to place resources. For more information on the use of node attributes, see the section on . Node attributes can be specified ahead of time or populated later, when the cluster is running, using crm_attribute. Below is what the node's definition would look like if the admin ran the command:
The result of using crm_attribute to specify which kernel pcmk-1 is running crm_attribute --type nodes --node-uname pcmk-1 --attr-name kernel --attr-value `uname -r` ]]>
A simpler way to determine the current value of an attribute is to use crm_attribute command again: crm_attribute --type nodes --node-uname pcmk-1 --attr-name kernel --get-value By specifying --type nodes the admin tells the cluster that this attribute is persistent. There are also transient attributes which are kept in the status section which are "forgotten" whenever the node rejoins the cluster. The cluster uses this area to store a record of how many times a resource has failed on that node but administrators can also read and write to this section by specifying --type status.
Adding a New Cluster Node
- OpenAIS - Adding a new is as simple as installing OpenAIS and Pacemaker, and copying /etc/ais/openais.conf and /etc/ais/authkey (if it exists) from an existing node. + Corosync + Adding a new is as simple as installing Corosync and Pacemaker, and copying /etc/corosync/corosync.conf and /etc/ais/authkey (if it exists) from an existing node. You may need to modify the mcastaddr option to match the new node's IP address. - If a log message containing "Invalid digest" appears from OpenAIS, the keys are not consistent between the machines. + If a log message containing "Invalid digest" appears from Corosync, the keys are not consistent between the machines.
Heartbeat Provided you specified autojoin any in ha.cf, adding a new is as simple as installing heartbeat and copying ha.cf and authkeys from an existing node. If not, then after setting up ha.cf and authkeys, you must use the hb_addnode command before starting the new node.
Removing a Cluster Node
- OpenAIS + Corosync Because the messaging and membership layers are the authoritative source for cluster nodes, deleting them from the CIB is not a reliable solution. First one must arrange for heartbeat to forget about the node (pcmk-1 in the example below). On the host to be removed: - Find and record the node's OpenAIS id: crm_node -i + Find and record the node's Corosync id: crm_node -i - Stop the cluster: /etc/init.d/openais stop + Stop the cluster: /etc/init.d/corosync stop Next, from one of the remaining active cluster nodes: - Tell the cluster to forget about the removed host: crm_node -R OPENAIS_ID + Tell the cluster to forget about the removed host: crm_node -R COROSYNC_ID Only now is it safe to delete the node from the CIB with: cibadmin --delete --obj_type nodes --crm_xml '<node uname="pcmk-1"/>' cibadmin --delete --obj_type status --crm_xml '<node_state uname="pcmk-1"/>'
Heartbeat Because the messaging and membership layers are the authoritative source for cluster nodes, deleting them from the CIB is not a reliable solution. First one must arrange for heartbeat to forget about the node (pcmk-1 in the example below). To do this, shut down heartbeat on the node and then, from one of the remaining active cluster nodes, run: hb_delnode pcmk-1 Only then is it safe to delete the node from the CIB with: cibadmin --delete --obj_type nodes --crm_xml '<node uname="pcmk-1"/>' cibadmin --delete --obj_type status --crm_xml '<node_state uname="pcmk-1"/>'
Replacing a Cluster Node
- OpenAIS + Corosync The five-step guide to replacing an existing cluster node: Make sure the old node is completely stopped Give the new machine the same hostname and IP address as the old one Install the cluster software :-) - Copy /etc/ais/openais.conf and /etc/ais/authkey (if it exists) to the new node + Copy /etc/corosync/corosync.conf and /etc/ais/authkey (if it exists) to the new node Start the new cluster node - If a log message containing "Invalid digest" appears from OpenAIS, the keys are not consistent between the machines. + If a log message containing "Invalid digest" appears from Corosync, the keys are not consistent between the machines.
Heartbeat The seven-step guide to replacing an existing cluster node: Make sure the old node is completely stopped Give the new machine the same hostname as the old one Go to an active cluster node and look up the UUID for the old node in /var/lib/heartbeat/hostcache Install the cluster software Copy ha.cf and authkeys to the new node On the new node, populate it's UUID using crm_uuid -w and the UUID from step 2 Start the new cluster node
diff --git a/doc/Pacemaker_Explained/en-US/Ch-Status.xml b/doc/Pacemaker_Explained/en-US/Ch-Status.xml index 2d8eec2541..59469362ef 100644 --- a/doc/Pacemaker_Explained/en-US/Ch-Status.xml +++ b/doc/Pacemaker_Explained/en-US/Ch-Status.xml @@ -1,298 +1,298 @@ Status - Here be dragons Most users never need understand the contents of the status section and can be content with the output from crm_mon. However for those with a curious inclination, the following attempts to proved an overview of its contents.
Node Status In addition to the cluster's configuration, the CIB holds an up-to-date representation of each cluster node in the status section.
A bare-bones status entry for a healthy node called cl-virt-1 ]]>
Users are highly recommended not to modify any part of a node's state directly. The cluster will periodically regenerate the entire section from authoritative sources. So any changes should be with the tools for those subsystems. Authoritative Sources for State Information Dataset Authoritative Source node_state fields crmd transient_attributes tag attrd lrm tag lrmd
The fields used in the node_state objects are named as they are largely for historical reasons and are rooted in Pacemaker's origins as the Heartbeat resource manager. They have remained unchanged to preserve compatibility with older versions. Node Status Fields Field Description id - Unique identifier for the node. OpenAIS based clusters use the same value as uname, Heartbeat cluster use a human-readable (but annoying) UUID. + Unique identifier for the node. Corosync based clusters use the same value as uname, Heartbeat cluster use a human-readable (but annoying) UUID. uname The node's machine name (output from uname -n) ha Is the cluster software active on the node. Allowed values: active, dead in_ccm Is the node part of the cluster's membership. Allowed values: true, false crmd Is the crmd process active on the node. Allowed values: online, offline join Is the node participating in hosting resources. Allowed values: down, pending, member, banned expected Expected value for join crm-debug-origin Diagnostic indicator. The origin of the most recent change(s).
The cluster uses these fields to determine if, at the node level, the node is healthy or is in a failed state and needs to be fenced.
Transient Node Attributes Like regular node attributes, the name/value pairs listed here also help describe the node. However they are forgotten by the cluster when the node goes offline. This can be useful, for instance, when you only want a node to be in standby mode (not able to run resources) until the next reboot. In addition to any values the administrator sets, the cluster will also store information about failed resources here.
Example set of transient node attributes for node "cl-virt-1" ]]>
In the above example, we can see that the pingd:0 resource has failed once, at Mon Apr 6 11:22:22 2009. You can use the following Perl one-liner to print a human readable of any seconds-since-epoch value: perl -e 'print scalar(localtime($seconds))."\n"' We also see that the node is connected to three "pingd" peers and that all known resources have been checked for on this machine (probe_complete).
Operation History A node's resource history is held in the lrm_resources tag (a child of the lrm tag). The information stored here includes enough information for the cluster to stop the resource safely if it is removed from the configuration section. Specifically we store the resource's id, class, type and provider.
A record of the apcstonith resource <lrm_resource id="apcstonith" type="apcmastersnmp" class="stonith">
Additionally, we store the last job for every combination of resource, action and interval. The concatenation of the values in this tuple are used to create the id of the lrm_rsc_op object. Contents of an lrm_rsc_op job. Field Description id Identifier for the job constructed from the resource id, operation and interval. call-id The job's ticket number. Used as a sort key to determine the order in which the jobs were executed. operation The action the resource agent was invoked with. interval The frequency, in milliseconds, at which the operation will be repeated. 0 indicates a one-off job. op-status The job's status. Generally this will be either 0 (done) or -1 (pending). Rarely used in favor of rc-code. rc-code The job's result. Refer to for details on what the values here mean and how they are interpreted. last-run Diagnostic indicator. Machine local date/time, in seconds since epoch, at which the job was executed. last-rc-change Diagnostic indicator. Machine local date/time, in seconds since epoch, at which the job first returned the current value of rc-code exec-time Diagnostic indicator. Time, in seconds, that the job was running for queue-time Diagnostic indicator. Time, in seconds, that the job was queued for in the LRMd crm_feature_set The version which this job description conforms to. Used when processing op-digest transition-key A concatenation of the job's graph action number, the graph number, the expected result and the UUID of the crmd instance that scheduled it. This is used to construct transition-magic (below). transition-magic A concatenation of the job's op-status, rc-code and transition-key. Guaranteed to be unique for the life of the cluster (which ensures it is part of CIB update notifications) and contains all the information needed for the crmd to correctly analyze and process the completed job. Most importantly, the decomposed elements tell the crmd if the job entry was expected and whether it failed. op-digest An MD5 sum representing the parameters passed to the job. Used to detect changes to the configuration and restart resources if necessary. crm-debug-origin Diagnostic indicator. The origin of the current values.
Simple Example
A monitor operation performed by the cluster to determine the current state of the apcstonith resource ]]>
In the above example, the job is a non-recurring monitor often referred to as a "probe" for the apcstonith resource. The cluster schedules probes for every configured resource on when a new node starts, in order to determine the resource's current state before it takes further any further action. From the transition-key, we can see that this was the 22nd action of the 2nd graph produced by this instance of the crmd (2668bbeb-06d5-40f9-936d-24cb7f87006a). The third field of the transition-key contains a 7, this indicates that the job expects to find the resource inactive. By now looking at the rc-code property, we see that this was the case. Evidently, the cluster started the resource elsewhere as that is the only job recorded for this node.
Complex Resource History Example
Resource history of a pingd clone with multiple jobs ]]>
When more than one job record exists, it is important to first sort them by call-id before interpret them. Once sorted, the above example can be summarized as: A non-recurring monitor operation returning 7 (not running), with a call-id of 3 A stop operation returning 0 (success), with a call-id of 32 A start operation returning 0 (success), with a call-id of 33 A recurring monitor returning 0 (success), with a call-id of 34 The cluster processes each job record to build up a picture of the resource's state. After the first and second entries, it is considered stopped and after the third it considered active. Based on the last operation, we can tell that the resource is currently active. Additionally, from the presence of a stop operation with a lower call-id than that of the start operation, we can conclude that the resource has been restarted. Specifically this occurred as part of actions 11 and 31 of transition 11 from the crmd instance with the key 2668bbeb-06d5-40f9-936d-24cb7f87006a. This information can be helpful for locating the relevant section of the logs when looking for the source of a failure.
diff --git a/doc/Pacemaker_Explained/en-US/Pacemaker_Explained.xml b/doc/Pacemaker_Explained/en-US/Pacemaker_Explained.xml index d019bdcc71..68c7feda97 100644 --- a/doc/Pacemaker_Explained/en-US/Pacemaker_Explained.xml +++ b/doc/Pacemaker_Explained/en-US/Pacemaker_Explained.xml @@ -1,48 +1,48 @@ Receiving Notification of Cluster Events
Configuring Email Notifications
Configuring SNMP Notifications
 Further Reading Project Website: and Documentation Cluster Commands A comprehensive guide to cluster commands has been written by Novell and can be found at: Heartbeat configuration: - OpenAIS Configuration: + Corosync Configuration: