diff --git a/doc/Pacemaker_Administration/en-US/Ch-Configuring.txt b/doc/Pacemaker_Administration/en-US/Ch-Configuring.txt index 38d77b19f4..2f21c088fd 100644 --- a/doc/Pacemaker_Administration/en-US/Ch-Configuring.txt +++ b/doc/Pacemaker_Administration/en-US/Ch-Configuring.txt @@ -1,461 +1,257 @@ :compat-mode: legacy = Configuring Pacemaker = -== How Should the Configuration be Updated? == - -=== General Guidelines === - -Pacemaker's configuration (the CIB) is stored in XML format. Cluster +Pacemaker's configuration, the CIB, is stored in XML format. Cluster administrators have multiple options for modifying the configuration either via the XML, or at a more abstract (and easier for humans to understand) level. -Pacemaker reacts to configuration changes as soon as they are saved. Most tools -provide the ability to batch changes together and commit them at once, rather -than make a series of small changes, which could cause avoid unnecessary -actions as Pacemaker responds to each change individually. +Pacemaker reacts to configuration changes as soon as they are saved. +Pacemaker's command-line tools and most higher-level tools provide the ability +to batch changes together and commit them at once, rather than make a series of +small changes, which could cause avoid unnecessary actions as Pacemaker +responds to each change individually. -Pacemaker keeps track of revisions to the configuration and will reject any -update that is older than the current revision. Thus, it is a good idea to -serialize all changes to the configuration. Avoid attempting simultaneous -changes, whether on the same node or different nodes, and whether manually or -using some automated configuration tool. +Pacemaker tracks revisions to the configuration and will reject any update +older than the current revision. Thus, it is a good idea to serialize all +changes to the configuration. Avoid attempting simultaneous changes, whether on +the same node or different nodes, and whether manually or using some automated +configuration tool. [NOTE] ==== -It is not necessary to update the configuration on all cluster nodes. All -changes are immediately synchronized to all active members of the cluster. To +It is not necessary to update the configuration on all cluster nodes. Pacemaker +immediately synchronizes changes to all active members of the cluster. To reduce bandwidth, the cluster only broadcasts the incremental updates that -result from your changes and uses MD5 checksums to ensure that each copy is -completely consistent. +result from your changes and uses checksums to ensure that each copy is +consistent. ==== -=== Higher-level Tools === -Most users will benefit from using higher-level tools that are provided by +=== Configuration Using Higher-level Tools === + +Most users will benefit from using higher-level tools provided by projects separate from Pacemaker. Some of the most commonly used include the crm shell, hawk, and pcs. footnote:[For a list, see "Configuration Tools" at https://clusterlabs.org/components.html] See those projects' documentation for details on how to configure Pacemaker using them. -=== Pacemaker's Command-Line Tools === +=== Configuration Using Pacemaker's Command-Line Tools === -Most configuration tasks can be performed without needing any XML knowledge, -using one of the lower-level command-line tools provided with Pacemaker. +Pacemaker provides lower-level, command-line tools to manage the cluster. Most +configuration tasks can be performed with these tools, without needing any XML +knowledge. To enable STONITH for example, one could run: ---- # crm_attribute --name stonith-enabled --update 1 ---- Or, to check whether *node1* is allowed to run resources, there is: ---- # crm_standby --query --node node1 ---- Or, to change the failure threshold of *my-test-rsc*, one can use: ---- # crm_resource -r my-test-rsc --set-parameter migration-threshold --parameter-value 3 --meta ---- Examples of using these tools for specific cases will be given throughout this document where appropriate. See the man pages for further details. -=== Editing the CIB Using XML === - -The most flexible tool for modifying the configuration is Pacemaker's -+cibadmin+ command. With +cibadmin+, you can query, add, remove, update -or replace any part of the configuration. All changes take effect immediately, -so there is no need to perform a reload-like operation. - -The simplest way of using `cibadmin` is to use it to save the current -configuration to a temporary file, edit that file with your favorite -text or XML editor, and then upload the revised configuration. - -.Safely using an editor to modify the cluster configuration -====== --------- -# cibadmin --query > tmp.xml -# vi tmp.xml -# cibadmin --replace --xml-file tmp.xml --------- -====== - -Some of the better XML editors can make use of a Relax NG schema to -help make sure any changes you make are valid. The schema describing -the configuration can be found in +pacemaker.rng+, which may be -deployed in a location such as +/usr/share/pacemaker+ depending on your -operating system distribution and how you installed the software. +See <> for how to edit the CIB using XML. -If you want to modify just one section of the configuration, you can -query and replace just that section to avoid modifying any others. - -.Safely using an editor to modify only the resources section -====== --------- -# cibadmin --query --scope resources > tmp.xml -# vi tmp.xml -# cibadmin --replace --scope resources --xml-file tmp.xml --------- -====== +See <> for a way to make a series of changes, then commit them +all at once to the live cluster. -To quickly delete a part of the configuration, identify the object you wish to -delete by XML tag and id. For example, you might search the CIB for all -STONITH-related configuration: - -.Searching for STONITH-related configuration items -====== ----- -# cibadmin --query | grep stonith - - - - - - - - - - ----- -====== - -If you wanted to delete the +primitive+ tag with id +child_DoFencing+, -you would run: - ----- -# cibadmin --delete --xml-text '' ----- - -See the cibadmin man page for more options. - -[IMPORTANT] -==== -Never edit the live +cib.xml+ file directly. Pacemaker will detect such changes -and refuse to use the configuration. -==== - -[[s-config-sandboxes]] -== Making Configuration Changes in a Sandbox == - -Often it is desirable to preview the effects of a series of changes -before updating the configuration all at once. For this purpose, we -have created `crm_shadow` which creates a -"shadow" copy of the configuration and arranges for all the command -line tools to use it. - -To begin, simply invoke `crm_shadow --create` with -the name of a configuration to create footnote:[Shadow copies are -identified with a name, making it possible to have more than one.], -and follow the simple on-screen instructions. - -[WARNING] -==== -Read this section and the on-screen instructions carefully; failure to do so could -result in destroying the cluster's active configuration! -==== - -.Creating and displaying the active sandbox -====== ----- -# crm_shadow --create test -Setting up shadow instance -Type Ctrl-D to exit the crm_shadow shell -shadow[test]: -shadow[test] # crm_shadow --which -test ----- -====== - -From this point on, all cluster commands will automatically use the -shadow copy instead of talking to the cluster's active configuration. -Once you have finished experimenting, you can either make the -changes active via the `--commit` option, or discard them using the `--delete` -option. Again, be sure to follow the on-screen instructions carefully! - -For a full list of `crm_shadow` options and -commands, invoke it with the `--help` option. - -.Use sandbox to make multiple changes all at once, discard them, and verify real configuration is untouched -====== ----- - shadow[test] # crm_failcount -r rsc_c001n01 -G - scope=status name=fail-count-rsc_c001n01 value=0 - shadow[test] # crm_standby --node c001n02 -v on - shadow[test] # crm_standby --node c001n02 -G - scope=nodes name=standby value=on - - shadow[test] # cibadmin --erase --force - shadow[test] # cibadmin --query - - - - - - - - - - shadow[test] # crm_shadow --delete test --force - Now type Ctrl-D to exit the crm_shadow shell - shadow[test] # exit - # crm_shadow --which - No active shadow configuration defined - # cibadmin -Q - - - - - - ----- -====== - -[[s-config-testing-changes]] -== Testing Your Configuration Changes == - -We saw previously how to make a series of changes to a "shadow" copy -of the configuration. Before loading the changes back into the -cluster (e.g. `crm_shadow --commit mytest --force`), it is often -advisable to simulate the effect of the changes with +crm_simulate+. -For example: - ----- -# crm_simulate --live-check -VVVVV --save-graph tmp.graph --save-dotfile tmp.dot ----- - -This tool uses the same library as the live cluster to show what it -would have done given the supplied input. Its output, in addition to -a significant amount of logging, is stored in two files +tmp.graph+ -and +tmp.dot+. Both files are representations of the same thing: the -cluster's response to your changes. - -The graph file stores the complete transition from the existing cluster state -to your desired new state, containing a list of all the actions, their -parameters and their pre-requisites. Because the transition graph is not -terribly easy to read, the tool also generates a Graphviz -footnote:[Graph visualization software. See http://www.graphviz.org/ for details.] -dot-file representing the same information. - -For information on the options supported by `crm_simulate`, use -its `--help` option. - -.Interpreting the Graphviz output - * Arrows indicate ordering dependencies - * Dashed arrows indicate dependencies that are not present in the transition graph - * Actions with a dashed border of any color do not form part of the transition graph - * Actions with a green border form part of the transition graph - * Actions with a red border are ones the cluster would like to execute but cannot run - * Actions with a blue border are ones the cluster does not feel need to be executed - * Actions with orange text are pseudo/pretend actions that the cluster uses to simplify the graph - * Actions with black text are sent to the executor - * Resource actions have text of the form pass:[rsc]_pass:[action]_pass:[interval] pass:[node] - * Any action depending on an action with a red border will not be able to execute. - * Loops are _really_ bad. Please report them to the development team. - -=== Small Cluster Transition === - -image::images/Policy-Engine-small.png["An example transition graph as represented by Graphviz",width="16cm",height="6cm",align="center"] - -In the above example, it appears that a new node, *pcmk-2*, has come -online and that the cluster is checking to make sure *rsc1*, *rsc2* -and *rsc3* are not already running there (Indicated by the -*rscN_monitor_0* entries). Once it did that, and assuming the resources -were not active there, it would have liked to stop *rsc1* and *rsc2* -on *pcmk-1* and move them to *pcmk-2*. However, there appears to be -some problem and the cluster cannot or is not permitted to perform the -stop actions which implies it also cannot perform the start actions. -For some reason the cluster does not want to start *rsc3* anywhere. - -=== Complex Cluster Transition === - -image::images/Policy-Engine-big.png["Another, slightly more complex, transition graph that you're not expected to be able to read",width="16cm",height="20cm",align="center"] == Working with CIB Properties == Although these fields can be written to by the user, in most cases the cluster will overwrite any values specified by the user with the "correct" ones. To change the ones that can be specified by the user, for example +admin_epoch+, one should use: ---- # cibadmin --modify --xml-text '' ---- A complete set of CIB properties will look something like this: -.Attributes set for a cib object +.XML attributes set for a cib element ====== [source,XML] ------- ------- ====== + == Querying and Setting Cluster Options == indexterm:[Querying,Cluster Option] indexterm:[Setting,Cluster Option] indexterm:[Cluster,Querying Options] indexterm:[Cluster,Setting Options] Cluster options can be queried and modified using the `crm_attribute` tool. To get the current value of +cluster-delay+, you can run: ---- # crm_attribute --query --name cluster-delay ---- which is more simply written as ---- # crm_attribute -G -n cluster-delay ---- If a value is found, you'll see a result like this: ---- # crm_attribute -G -n cluster-delay scope=crm_config name=cluster-delay value=60s ---- If no value is found, the tool will display an error: ---- # crm_attribute -G -n clusta-deway scope=crm_config name=clusta-deway value=(null) Error performing operation: No such device or address ---- To use a different value (for example, 30 seconds), simply run: ---- # crm_attribute --name cluster-delay --update 30s ---- To go back to the cluster's default value, you can delete the value, for example: ---- # crm_attribute --name cluster-delay --delete Deleted crm_config option: id=cib-bootstrap-options-cluster-delay name=cluster-delay ---- === When Options are Listed More Than Once === If you ever see something like the following, it means that the option you're modifying is present more than once. .Deleting an option that is listed twice ======= ------ # crm_attribute --name batch-limit --delete Multiple attributes match name=batch-limit in crm_config: Value: 50 (set=cib-bootstrap-options, id=cib-bootstrap-options-batch-limit) Value: 100 (set=custom, id=custom-batch-limit) Please choose from one of the matches above and supply the 'id' with --id ------- ======= In such cases, follow the on-screen instructions to perform the requested action. To determine which value is currently being used by the cluster, refer to the 'Rules' chapter of 'Pacemaker Explained'. [[s-remote-connection]] == Connecting from a Remote Machine == indexterm:[Cluster,Remote connection] indexterm:[Cluster,Remote administration] Provided Pacemaker is installed on a machine, it is possible to connect to the cluster even if the machine itself is not in the same cluster. To do this, one simply sets up a number of environment variables and runs the same commands as when working on a cluster node. .Environment Variables Used to Connect to Remote Instances of the CIB [width="95%",cols="1m,1,<3",options="header",align="center"] |========================================================= |Environment Variable |Default |Description |CIB_user |$USER |The user to connect as. Needs to be part of the +haclient+ group on the target host. indexterm:[Environment Variable,CIB_user] |CIB_passwd | |The user's password. Read from the command line if unset. indexterm:[Environment Variable,CIB_passwd] |CIB_server |localhost |The host to contact indexterm:[Environment Variable,CIB_server] |CIB_port | |The port on which to contact the server; required. indexterm:[Environment Variable,CIB_port] |CIB_encrypted |TRUE |Whether to encrypt network traffic indexterm:[Environment Variable,CIB_encrypted] |========================================================= So, if *c001n01* is an active cluster node and is listening on port 1234 for connections, and *someuser* is a member of the *haclient* group, then the following would prompt for *someuser*'s password and return the cluster's current configuration: ---- # export CIB_port=1234; export CIB_server=c001n01; export CIB_user=someuser; # cibadmin -Q ---- For security reasons, the cluster does not listen for remote connections by default. If you wish to allow remote access, you need to set the +remote-tls-port+ (encrypted) or +remote-clear-port+ (unencrypted) CIB properties (i.e., those kept in the +cib+ tag, like +num_updates+ and +epoch+). .Extra top-level CIB properties for remote access [width="95%",cols="1m,1,<3",options="header",align="center"] |========================================================= |Field |Default |Description |remote-tls-port |_none_ |Listen for encrypted remote connections on this port. indexterm:[remote-tls-port,Remote Connection Option] indexterm:[Remote Connection,Option,remote-tls-port] |remote-clear-port |_none_ |Listen for plaintext remote connections on this port. indexterm:[remote-clear-port,Remote Connection Option] indexterm:[Remote Connection,Option,remote-clear-port] |========================================================= [IMPORTANT] ==== The Pacemaker version on the administration host must be the same or greater than the version(s) on the cluster nodes. Otherwise, it may not have the schema files necessary to validate the CIB. ==== diff --git a/doc/Pacemaker_Administration/en-US/Ch-Monitoring.txt b/doc/Pacemaker_Administration/en-US/Ch-Monitoring.txt deleted file mode 100644 index 9792d5ceff..0000000000 --- a/doc/Pacemaker_Administration/en-US/Ch-Monitoring.txt +++ /dev/null @@ -1,61 +0,0 @@ -:compat-mode: legacy -= Monitoring a Pacemaker Cluster = - -== Using crm_mon == - -The `crm_mon` utility displays the current state of an active cluster. It can -show the cluster status organized by node or by resource, and can be used in -either single-shot or dynamically updating mode. It can also display operations -performed and information about failures. - -Using this tool, you can examine the state of the cluster for irregularities, -and see how it responds when you cause or simulate failures. - -See the manual page or the output of `crm_mon --help` for a full description of -its many options. - -.Sample output from crm_mon -1 -====== -------- -Stack: corosync -Current DC: node2 (version 2.0.0-1) - partition with quorum -Last updated: Mon Jan 29 12:18:42 2018 -Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 - -5 nodes configured -2 resources configured - -Online: [ node1 node2 node3 node4 node5 ] - -Active resources: - -Fencing (stonith:fence_xvm): Started node1 -IP (ocf:heartbeat:IPaddr2): Started node2 -------- -====== - -.Sample output from crm_mon -n -1 -====== -------- -Stack: corosync -Current DC: node2 (version 2.0.0-1) - partition with quorum -Last updated: Mon Jan 29 12:21:48 2018 -Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 - -5 nodes configured -2 resources configured - -Node node1: online - Fencing (stonith:fence_xvm): Started -Node node2: online - IP (ocf:heartbeat:IPaddr2): Started -Node node3: online -Node node4: online -Node node5: online -------- -====== - -As mentioned in an earlier chapter, the DC is the node is where decisions are -made. The cluster elects a node to be DC as needed. The only significance of -the choice of DC to an administrator is the fact that its logs will have the -most information about why decisions were made. diff --git a/doc/Pacemaker_Administration/en-US/Ch-Tools.txt b/doc/Pacemaker_Administration/en-US/Ch-Tools.txt new file mode 100644 index 0000000000..f1728bea7b --- /dev/null +++ b/doc/Pacemaker_Administration/en-US/Ch-Tools.txt @@ -0,0 +1,464 @@ +:compat-mode: legacy += Using Pacemaker Command-Line Tools = + +[[s-crm_mon]] +== Monitor a Cluster with crm_mon == +indexterm:[Command-line tool,crm_mon] + +The `crm_mon` utility displays the current state of an active cluster. It can +show the cluster status organized by node or by resource, and can be used in +either single-shot or dynamically updating mode. It can also display operations +performed and information about failures. + +Using this tool, you can examine the state of the cluster for irregularities, +and see how it responds when you cause or simulate failures. + +See the manual page or the output of `crm_mon --help` for a full description of +its many options. + +.Sample output from crm_mon -1 +====== +------- +Stack: corosync +Current DC: node2 (version 2.0.0-1) - partition with quorum +Last updated: Mon Jan 29 12:18:42 2018 +Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 + +5 nodes configured +2 resources configured + +Online: [ node1 node2 node3 node4 node5 ] + +Active resources: + +Fencing (stonith:fence_xvm): Started node1 +IP (ocf:heartbeat:IPaddr2): Started node2 +------- +====== + +.Sample output from crm_mon -n -1 +====== +------- +Stack: corosync +Current DC: node2 (version 2.0.0-1) - partition with quorum +Last updated: Mon Jan 29 12:21:48 2018 +Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 + +5 nodes configured +2 resources configured + +Node node1: online + Fencing (stonith:fence_xvm): Started +Node node2: online + IP (ocf:heartbeat:IPaddr2): Started +Node node3: online +Node node4: online +Node node5: online +------- +====== + +As mentioned in an earlier chapter, the DC is the node is where decisions are +made. The cluster elects a node to be DC as needed. The only significance of +the choice of DC to an administrator is the fact that its logs will have the +most information about why decisions were made. + + +[[s-cibadmin]] +== Edit the CIB XML with cibadmin == +indexterm:[Command-line tool,cibadmin] + +The most flexible tool for modifying the configuration is Pacemaker's +`cibadmin` command. With `cibadmin`, you can query, add, remove, update +or replace any part of the configuration. All changes take effect immediately, +so there is no need to perform a reload-like operation. + +The simplest way of using `cibadmin` is to use it to save the current +configuration to a temporary file, edit that file with your favorite +text or XML editor, and then upload the revised configuration. + +.Safely using an editor to modify the cluster configuration +====== +-------- +# cibadmin --query > tmp.xml +# vi tmp.xml +# cibadmin --replace --xml-file tmp.xml +-------- +====== + +Some of the better XML editors can make use of a RELAX NG schema to +help make sure any changes you make are valid. The schema describing +the configuration can be found in +pacemaker.rng+, which may be +deployed in a location such as +/usr/share/pacemaker+ depending on your +operating system distribution and how you installed the software. + +If you want to modify just one section of the configuration, you can +query and replace just that section to avoid modifying any others. + +.Safely using an editor to modify only the resources section +====== +-------- +# cibadmin --query --scope resources > tmp.xml +# vi tmp.xml +# cibadmin --replace --scope resources --xml-file tmp.xml +-------- +====== + +To quickly delete a part of the configuration, identify the object you wish to +delete by XML tag and id. For example, you might search the CIB for all +STONITH-related configuration: + +.Searching for STONITH-related configuration items +====== +---- +# cibadmin --query | grep stonith + + + + + + + + + + +---- +====== + +If you wanted to delete the +primitive+ tag with id +child_DoFencing+, +you would run: + +---- +# cibadmin --delete --xml-text '' +---- + +See the cibadmin man page for more options. + +[IMPORTANT] +==== +Never edit the live +cib.xml+ file directly. Pacemaker will detect such changes +and refuse to use the configuration. +==== + + +[[s-crm_shadow]] +== Batch Configuration Changes with crm_shadow == +indexterm:[Command-line tool,crm_shadow] + +Often, it is desirable to preview the effects of a series of configuration +changes before updating the live configuration all at once. For this purpose, +`crm_shadow` creates a "shadow" copy of the configuration and arranges for all +the command-line tools to use it. + +To begin, simply invoke `crm_shadow --create` with a name of your choice, +and follow the simple on-screen instructions. Shadow copies are identified with +a name to make it possible to have more than one. + +[WARNING] +==== +Read this section and the on-screen instructions carefully; failure to do so could +result in destroying the cluster's active configuration! +==== + +.Creating and displaying the active sandbox +====== +---- +# crm_shadow --create test +Setting up shadow instance +Type Ctrl-D to exit the crm_shadow shell +shadow[test]: +shadow[test] # crm_shadow --which +test +---- +====== + +From this point on, all cluster commands will automatically use the shadow copy +instead of talking to the cluster's active configuration. Once you have +finished experimenting, you can either make the changes active via the +`--commit` option, or discard them using the `--delete` option. Again, be sure +to follow the on-screen instructions carefully! + +For a full list of `crm_shadow` options and commands, invoke it with the +`--help` option. + +.Use sandbox to make multiple changes all at once, discard them, and verify real configuration is untouched +====== +---- + shadow[test] # crm_failcount -r rsc_c001n01 -G + scope=status name=fail-count-rsc_c001n01 value=0 + shadow[test] # crm_standby --node c001n02 -v on + shadow[test] # crm_standby --node c001n02 -G + scope=nodes name=standby value=on + + shadow[test] # cibadmin --erase --force + shadow[test] # cibadmin --query + + + + + + + + + + shadow[test] # crm_shadow --delete test --force + Now type Ctrl-D to exit the crm_shadow shell + shadow[test] # exit + # crm_shadow --which + No active shadow configuration defined + # cibadmin -Q + + + + + + +---- +====== + +See the next section, <>, for how to test your changes before +committing them to the live cluster. + + +[[s-crm_simulate]] +== Simulate Cluster Activity with crm_simulate == +indexterm:[Command-line tool,crm_simulate] + +The command-line tool `crm_simulate` shows the results of the same logic +the cluster itself uses to respond to a particular cluster configuration and +status. + +As always, the man page is the primary documentation, and should be consulted +for further details. This section aims for a better conceptual explanation and +practical examples. + +=== Replaying cluster decision-making logic === + +At any given time, one node in a Pacemaker cluster will be elected DC, and that +node will run Pacemaker's scheduler to make decisions. + +Each time decisions need to be made (a "transition"), the DC will have log +messages like "Calculated transition ... saving inputs in ..." with a file +name. You can grab the named file and replay the cluster logic to see why +particular decisions were made. The file contains the live cluster +configuration at that moment, so you can also look at it directly to see the +value of node attributes, etc., at that time. + +The simplest usage is (replacing $FILENAME with the actual file name): + +.Simulate cluster response to a given CIB +==== +---- +crm_simulate --simulate --xml-file $FILENAME +---- +==== + +That will show the cluster state when the process started, the actions that +need to be taken ("Transition Summary"), and the resulting cluster state if the +actions succeed. Most actions will have a brief description of why they were +required. + +The transition inputs may be compressed. `crm_simulate` can handle these +compressed files directly, though if you want to edit the file, you'll need to +uncompress it first. + +You can do the same simulation for the live cluster configuration at the +current moment. This is useful mainly when using `crm_shadow` to create a +sandbox version of the CIB; the `--live-check` option will use the shadow CIB +if one is in effect. + +.Simulate cluster response to current live CIB or shadow CIB +==== +---- +crm_simulate --simulate --live-check +---- +==== + + +=== Why decisions were made === + +To get further insight into the "why", it gets user-unfriendly very quickly. If +you add the `--show-scores` option, you will also see all the scores that went +into the decision-making. The node with the highest cumulative score for a +resource will run it. You can look for +-INFINITY+ scores in particular to see +where complete bans came into effect. + +You can also add `-VVVV` to get more detailed messages about what's happening +under the hood. You can add up to two more V's even, but that's usually useful +only if you're a masochist or tracing through the source code. + +=== Visualizing the action sequence === + +Another handy feature is the ability to generate a visual graph of the actions +needed, using the `--dot-file` option. This relies on the separate Graphviz +footnote:[Graph visualization software. See http://www.graphviz.org/ for details.] +project. + +.Generate a visual graph of cluster actions from a saved CIB +==== +---- +crm_simulate --simulate --xml-file $FILENAME --dot-file $FILENAME.dot +dot $FILENAME.dot -Tsvg > $FILENAME.svg +---- +==== + ++$FILENAME.dot+ will contain a GraphViz representation of the cluster's +response to your changes, including all actions with their ordering +dependencies. + ++$FILENAME.svg+ will be the same information in a standard graphical format +that you can view in your browser or other app of choice. You could, of course, +use other `dot` options to generate other formats. + +How to interpret the graphical output: + + * Bubbles indicate actions, and arrows indicate ordering dependencies + * Resource actions have text of the form + pass:[resource]_pass:[action]_pass:[interval] + pass:[node] indicating that the specified action + will be executed for the specified resource on the specified node, once if + interval is 0 or at specified recurring milliseconds interval otherwise + * Actions with black text will be sent to the executor (that is, the + appropriate agent will be invoked) + * Actions with orange text are "pseudo" actions that the cluster uses + internally for ordering but require no real activity + * Actions with a solid green border are part of the transition (that is, the + cluster will attempt to execute them in the given order -- though a + transition can be interrupted by action failure or new events) + * Dashed arrows indicate dependencies that are not present in the transition + graph + * Actions with a dashed border will not be executed. If the dashed border is + blue, the cluster does not feel the action needs to be executed. If the + dashed border is red, the cluster would like to execute the action but + cannot. Any actions depending on an action with a dashed border will not be + able to execute. + * Loops should not happen, and should be reported as a bug if found. + +.Small Cluster Transition +==== +image::images/Policy-Engine-small.png["An example transition graph as represented by Graphviz",width="1161",height="325",align="center"] +==== + +In the above example, it appears that a new node, *pcmk-2*, has come online and +that the cluster is checking to make sure *rsc1*, *rsc2* and *rsc3* are not +already running there (indicated by the *rscN_monitor_0* entries). Once it did +that, and assuming the resources were not active there, it would have liked to +stop *rsc1* and *rsc2* on *pcmk-1* and move them to *pcmk-2*. However, there +appears to be some problem and the cluster cannot or is not permitted to +perform the stop actions which implies it also cannot perform the start +actions. For some reason, the cluster does not want to start *rsc3* anywhere. + +.Complex Cluster Transition +==== +image::images/Policy-Engine-big.png["Complex transition graph that you're not expected to be able to read",width="1455",height="1945",align="center"] +==== + +=== What-if scenarios === + +You can make changes to the saved or shadow CIB and simulate it again, to see +how Pacemaker would react differently. You can edit the XML by hand, use +command-line tools such as `cibadmin` with either a shadow CIB or the ++CIB_file+ environment variable set to the filename, or use higher-level tool +support (see the man pages of the specific tool you're using for how to perform +actions on a saved CIB file rather than the live CIB). + +You can also inject node failures and/or action failures into the simulation; +see the `crm_simulate` man page for more details. + +This capability is useful when using a shadow CIB to edit the configuration. +Before committing the changes to the live cluster with `crm_shadow --commit`, +you can use `crm_simulate` to see how the cluster will react to the changes. + +[[s-attrd_updater]] +[[s-crm_attribute]] +== Manage Node Attributes, Cluster Options and Defaults with crm_attribute and attrd_updater == +indexterm:[Command-line tool,attrd_updater] +indexterm:[Command-line tool,crm_attribute] + +`crm_attribute` and `attrd_updater` are confusingly similar tools with subtle +differences. + +`attrd_updater` can query and update node attributes. `crm_attribute` can query +and update not only node attributes, but also cluster options, +resource defaults, and operation defaults. + +To understand the differences, it helps to understand the various types of node +attribute. + +.Types of Node Attributes +[width="95%",cols="1,1,1,1,1,1",options="header",align="center"] +|==== + +|Type +|Recorded in CIB? +|Recorded in attribute manager memory? +|Survive full cluster restart? +|Manageable by crm_attribute? +|Manageable by attrd_updater? + +|permanent +|yes +|no +|yes +|yes +|no + +|transient +|yes +|yes +|no +|yes +|yes + +|private +|no +|yes +|no +|no +|yes + +|==== + +As you can see from the table above, `crm_attribute` can manage permanent and +transient node attributes, while `attrd_updater` can manage transient and +private node attributes. + +The difference between the two tools lies mainly in 'how' they update node +attributes: `attrd_updater` always contacts the Pacemaker attribute manager +directly, while `crm_attribute` will contact the attribute manager only for +transient node attributes, and will instead modify the CIB directly for +permanent node attributes (and for transient node attributes when unable to +contact the attribute manager). + +By contacting the attribute manager directly, `attrd_updater` can change +an attribute's "dampening" (whether changes are immediately flushed to the CIB +or after a specified amount of time, to minimize disk writes for frequent +changes), set private node attributes (which are never written to the CIB), and +set attributes for nodes that don't yet exist. + +By modifying the CIB directly, `crm_attribute` can set permanent node +attributes (which are only in the CIB and not managed by the attribute +manager), and can be used with saved CIB files and shadow CIBs. + +However a transient node attribute is set, it is synchronized between the CIB +and the attribute manager, on all nodes. + + +== Other Commonly Used Tools == + +Other command-line tools include: +indexterm:[Command-line tool,crm_failcount] +indexterm:[Command-line tool,crm_node] +indexterm:[Command-line tool,crm_report] +indexterm:[Command-line tool,crm_standby] +indexterm:[Command-line tool,crm_verify] +indexterm:[Command-line tool,stonith_admin] + +* `crm_failcount`: query or delete resource fail counts +* `crm_node`: manage cluster nodes +* `crm_report`: generate a detailed cluster report for bug submissions +* `crm_resource`: manage cluster resources +* `crm_standby`: manage standby status of nodes +* `crm_verify`: validate a CIB +* `stonith_admin`: manage fencing devices + +See the manual pages for details. diff --git a/doc/Pacemaker_Administration/en-US/Ch-Troubleshooting.txt b/doc/Pacemaker_Administration/en-US/Ch-Troubleshooting.txt index f01d2f04cf..371836a374 100644 --- a/doc/Pacemaker_Administration/en-US/Ch-Troubleshooting.txt +++ b/doc/Pacemaker_Administration/en-US/Ch-Troubleshooting.txt @@ -1,64 +1,64 @@ :compat-mode: legacy = Troubleshooting Cluster Problems = == Logging == Pacemaker by default logs messages of notice severity and higher to the system log, and messages of info severity and higher to the detail log, which by default is /var/log/pacemaker/pacemaker.log. Logging options can be controlled via environment variables at Pacemaker start-up. Where these are set varies by operating system (often +/etc/sysconfig/pacemaker+ or +/etc/default/pacemaker+). Because cluster problems are often highly complex, involving multiple machines, cluster daemons, and managed services, Pacemaker logs rather verbosely to provide as much context as possible. It is an ongoing priority to make these logs more user-friendly, but by necessity there is a lot of obscure, low-level information that can make them difficult to follow. The default log rotation configuration shipped with Pacemaker (typically installed in /etc/logrotate.d/pacemaker) rotates the log when it reaches 100MB in size, or weekly, whichever comes first. If you configure debug or (Heaven forbid) trace-level logging, the logs can grow enormous quite quickly. Because rotated logs are by default named with the year, month, and day only, this can cause name collisions if your logs exceed 100MB in a single day. You can add +dateformat -%Y%m%d-%H+ to the rotation configuration to avoid this. == Transitions == A key concept in understanding how a Pacemaker cluster functions is a 'transition'. A transition is a set of actions that need to be taken to bring the cluster from its current state to the desired state (as expressed by the configuration). Whenever a relevant event happens (a node joining or leaving the cluster, a resource failing, etc.), the controller will ask the scheduler to recalculate the status of the cluster, which generates a new transition. The controller then performs the actions in the transition in the proper order. Each transition can be identified in the logs by a line like: ---- Nov 30 20:28:16 rhel7-1 pacemaker-schedulerd[36417] (process_pe_message) notice: Calculated transition 19, saving inputs in /var/lib/pacemaker/pengine/pe-input-1463.bz2 ---- The file listed as the "inputs" is a snapshot of the cluster configuration and state at that moment (the CIB). This file can help determine why particular actions were scheduled. The `crm_simulate` command, described in -<>, can be used to replay the file. +<>, can be used to replay the file. == Further Information About Troubleshooting == Andrew Beekhof wrote a series of articles about troubleshooting in his blog, http://blog.clusterlabs.org/[The Cluster Guy]: * http://blog.clusterlabs.org/blog/2013/debugging-pacemaker[Debugging Pacemaker] * http://blog.clusterlabs.org/blog/2013/debugging-pengine[Debugging the Policy Engine] * http://blog.clusterlabs.org/blog/2013/pacemaker-logging[Pacemaker Logging] The articles were written for an earlier version of Pacemaker, so many of the specific names and log messages to look for have changed, but the concepts are still valid. diff --git a/doc/Pacemaker_Administration/en-US/Ch-Upgrading.txt b/doc/Pacemaker_Administration/en-US/Ch-Upgrading.txt index a607209e8c..a85c14edb7 100644 --- a/doc/Pacemaker_Administration/en-US/Ch-Upgrading.txt +++ b/doc/Pacemaker_Administration/en-US/Ch-Upgrading.txt @@ -1,457 +1,456 @@ :compat-mode: legacy = Upgrading a Pacemaker Cluster = == Pacemaker Versioning == Pacemaker has an overall release version, plus separate version numbers for certain internal components. * *Pacemaker release version:* This version consists of three numbers (_x.y.z_). + The major version number (the _x_ in _x.y.z_) increases when at least some rolling upgrades are not possible from the previous major version. For example, a rolling upgrade from 1.0.8 to 1.1.15 should always be supported, but a rolling upgrade from 1.0.8 to 2.0.0 may not be possible. + The minor version (the _y_ in _x.y.z_) increases when there are significant changes in cluster default behavior, tool behavior, and/or the API interface (for software that utilizes Pacemaker libraries). The main benefit is to alert you to pay closer attention to the release notes, to see if you might be affected. + The release counter (the _z_ in _x.y.z_) is increased with all public releases of Pacemaker, which typically include both bug fixes and new features. * *CRM feature set:* This version number applies to the communication between full cluster nodes, and is used to avoid problems in mixed-version clusters. + The major version number increases when nodes with different versions would not work (rolling upgrades are not allowed). The minor version number increases when mixed-version clusters are allowed only during rolling upgrades. The minor-minor version number is ignored, but allows resource agents to detect cluster support for various features. footnote:[ Before CRM feature set 3.1.0 (Pacemaker 2.0.0), the minor-minor version number was treated the same as the minor version. ] + Pacemaker ensures that the longest-running node is the cluster's DC. This ensures new features are not enabled until all nodes are upgraded to support them. * *Pacemaker Remote protocol version:* This version applies to communication between a Pacemaker Remote node and the cluster. It increases when an older cluster node would have problems hosting the connection to a newer Pacemaker Remote node. To avoid these problems, Pacemaker Remote nodes will accept connections only from cluster nodes with the same or newer Pacemaker Remote protocol version. + Unlike with CRM feature set differences between full cluster nodes, mixed Pacemaker Remote protocol versions between Pacemaker Remote nodes and full cluster nodes are fine, as long as the Pacemaker Remote nodes have the older version. This can be useful, for example, to host a legacy application in an older operating system version used as a Pacemaker Remote node. * *XML schema version:* Pacemaker’s configuration syntax — what's allowed in the Configuration Information Base (CIB) — has its own version. This allows the configuration syntax to evolve over time while still allowing clusters with older configurations to work without change. == Upgrading Cluster Software == There are three approaches to upgrading a cluster, each with advantages and disadvantages. .Upgrade Methods [width="95%",cols="s,6*",options="header",align="center"] |========================================================= |Method |Available between all versions |Can be used with Pacemaker Remote nodes |Service outage during upgrade |Service recovery during upgrade |Exercises failover logic |Allows change of messaging layer indexterm:[Cluster,switching between stacks] indexterm:[Changing cluster stack] footnote:[Currently, Corosync version 2 and greater is the only supported cluster stack, but other stacks have been supported by past versions, and may be supported by future versions.] |Complete cluster shutdown indexterm:[upgrade,shutdown] indexterm:[shutdown upgrade] |yes |yes |always |N/A |no |yes |Rolling (node by node) indexterm:[upgrade,rolling] indexterm:[rolling upgrade] |no |yes |always footnote:[Any active resources will be moved off the node being upgraded, so there will be at least a brief outage unless all resources can be migrated "live".] |yes |yes |no |Detach and reattach indexterm:[upgrade,reattach] indexterm:[reattach upgrade] |yes |no |only due to failure |no |no |yes |========================================================= === Complete Cluster Shutdown === In this scenario, one shuts down all cluster nodes and resources, then upgrades all the nodes before restarting the cluster. . On each node: .. Shutdown the cluster software (pacemaker and the messaging layer). .. Upgrade the Pacemaker software. This may also include upgrading the messaging layer and/or the underlying operating system. .. Check the configuration with the `crm_verify` tool. . On each node: .. Start the cluster software. Currently, only Corosync version 2 and greater is supported as the cluster layer, but if another stack is supported in the future, the stack does not need to be the same one before the upgrade. One variation of this approach is to build a new cluster on new hosts. This allows the new version to be tested beforehand, and minimizes downtime by having the new nodes ready to be placed in production as soon as the old nodes are shut down. === Rolling (node by node) === In this scenario, each node is removed from the cluster, upgraded, and then brought back online, until all nodes are running the newest version. Special considerations when planning a rolling upgrade: * If you plan to upgrade other cluster software -- such as the messaging layer -- at the same time, consult that software's documentation for its compatibility with a rolling upgrade. * If the major version number is changing in the Pacemaker version you are upgrading to, a rolling upgrade may not be possible. Read the new version's release notes (as well the information here) for what limitations may exist. * If the CRM feature set is changing in the Pacemaker version you are upgrading to, you should run a mixed-version cluster only during a small rolling upgrade window. If one of the older nodes drops out of the cluster for any reason, it will not be able to rejoin until it is upgraded. * If the Pacemaker Remote protocol version is changing, all cluster nodes should be upgraded before upgrading any Pacemaker Remote nodes. See the ClusterLabs wiki's http://clusterlabs.org/wiki/ReleaseCalendar[Release Calendar] to figure out whether the CRM feature set and/or Pacemaker Remote protocol version changed between the the Pacemaker release versions in your rolling upgrade. To perform a rolling upgrade, on each node in turn: . Put the node into standby mode, and wait for any active resources to be moved cleanly to another node. (This step is optional, but allows you to deal with any resource issues before the upgrade.) . Shutdown the cluster software (pacemaker and the messaging layer) on the node. . Upgrade the Pacemaker software. This may also include upgrading the messaging layer and/or the underlying operating system. . If this is the first node to be upgraded, check the configuration with the `crm_verify` tool. . Start the messaging layer. This must be the same messaging layer (currently only Corosync version 2 and greater is supported) that the rest of the cluster is using. [NOTE] ==== Even if a rolling upgrade from the current version of the cluster to the newest version is not directly possible, it may be possible to perform a rolling upgrade in multiple steps, by upgrading to an intermediate version first. .Version Compatibility Table [width="95%",cols="2*",options="header",align="center"] |========================================================= |Version being Installed |Oldest Compatible Version |Pacemaker 2.y.z |Pacemaker 1.1.11 footnote:[Rolling upgrades from Pacemaker 1.1.z to 2.y.z are possible only if the cluster uses corosync version 2 or greater as its messaging layer, and the Cluster Information Base (CIB) uses schema 1.0 or higher in its validate-with property.] |Pacemaker 1.y.z |Pacemaker 1.0.0 |Pacemaker 0.7.z |Pacemaker 0.6.z |========================================================= ==== === Detach and Reattach === The reattach method is a variant of a complete cluster shutdown, where the resources are left active and get re-detected when the cluster is restarted. This method may not be used if the cluster contains any Pacemaker Remote nodes. . Tell the cluster to stop managing services. This is required to allow the services to remain active after the cluster shuts down. + ---- # crm_attribute --name maintenance-mode --update true ---- . On each node, shutdown the cluster software (pacemaker and the messaging layer), and upgrade the Pacemaker software. This may also include upgrading the messaging layer. While the underlying operating system may be upgraded at the same time, that will be more likely to cause outages in the detached services (certainly, if a reboot is required). . Check the configuration with the `crm_verify` tool. . On each node, start the cluster software. Currently, only Corosync version 2 and greater is supported as the cluster layer, but if another stack is supported in the future, the stack does not need to be the same one before the upgrade. . Verify that the cluster re-detected all resources correctly. . Allow the cluster to resume managing resources again: + ---- # crm_attribute --name maintenance-mode --delete ---- == Upgrading the Configuration == indexterm:[upgrade,Configuration] indexterm:[Configuration,upgrading] The CIB schema version can change from one Pacemaker version to another. After cluster software is upgraded, the cluster will continue to use the older schema version that it was previously using. This can be useful, for example, when administrators have written tools that modify the configuration, and are based on the older syntax. footnote:[As of Pacemaker 2.0.0, only schema versions pacemaker-1.0 and higher are supported (excluding pacemaker-1.1, which was an experimental schema now known as pacemaker-next).] However, when using an older syntax, new features may be unavailable, and there is a performance impact, since the cluster must do a non-persistent configuration upgrade before each transition. So while using the old syntax is possible, it is not advisable to continue using it indefinitely. Even if you wish to continue using the old syntax, it is a good idea to follow the upgrade procedure outlined below, except for the last step, to ensure that the new software has no problems with your existing configuration (since it will perform much the same task internally). If you are brave, it is sufficient simply to run `cibadmin --upgrade`. A more cautious approach would proceed like this: . Create a shadow copy of the configuration. The later commands will automatically operate on this copy, rather than the live configuration. + ----- # crm_shadow --create shadow ----- . Verify the configuration is valid with the new software (which may be stricter about syntax mistakes, or may have dropped support for deprecated features): indexterm:[Configuration,verify] indexterm:[verify,Configuration] + ----- # crm_verify --live-check ----- . Fix any errors or warnings. . Perform the upgrade: + ----- # cibadmin --upgrade ----- . If this step fails, there are three main possibilities: .. The configuration was not valid to start with (did you do steps 2 and 3?). .. The transformation failed - http://bugs.clusterlabs.org/[report a bug] or mailto:users@clusterlabs.org?subject=Transformation%20failed%20during%20upgrade[email the project]. .. The transformation was successful but produced an invalid result. + If the result of the transformation is invalid, you may see a number of errors from the validation library. If these are not helpful, visit the http://clusterlabs.org/wiki/Validation_FAQ[Validation FAQ wiki page] and/or try the manual upgrade procedure described below. + . Check the changes: + ----- # crm_shadow --diff ----- + If at this point there is anything about the upgrade that you wish to fine-tune (for example, to change some of the automatic IDs), now is the time to do so: + ----- # crm_shadow --edit ----- + This will open the configuration in your favorite editor (whichever is specified by the standard *$EDITOR* environment variable). + . Preview how the cluster will react: + ------ # crm_simulate --live-check --save-dotfile shadow.dot -S # dot -Tsvg shadow.dot -o shadow.svg ------ + You can then view shadow.svg with any compatible image viewer or web browser. Verify that either no resource actions will occur or that you are happy with any that are scheduled. If the output contains actions you do not expect (possibly due to changes to the score calculations), you -may need to make further manual changes. See -<> for further details on how to interpret -the output of `crm_simulate` and `dot`. +may need to make further manual changes. See <> for further +details on how to interpret the output of `crm_simulate` and `dot`. + . Upload the changes: + ----- # crm_shadow --commit shadow --force ----- + In the unlikely event this step fails, please report a bug. [NOTE] ==== indexterm:[Configuration,upgrade manually] It is also possible to perform the configuration upgrade steps manually: . Locate the +upgrade*.xsl+ conversion scripts provided with the source code. These will often be installed in a location such as +/usr/share/pacemaker+, or may be obtained from the https://github.com/ClusterLabs/pacemaker/tree/master/xml[source repository]. . Run the conversion scripts that apply to your older version, for example: indexterm:[XML,convert] + ----- # xsltproc /path/to/upgrade06.xsl config06.xml > config10.xml ----- + . Locate the +pacemaker.rng+ script (from the same location as the xsl files). . Check the XML validity: indexterm:[validate configuration]indexterm:[Configuration,validate XML] + ---- # xmllint --relaxng /path/to/pacemaker.rng config10.xml ---- The advantage of this method is that it can be performed without the cluster running, and any validation errors are often more informative. ==== == What Changed in 2.0 == The main goal of the 2.0 release was to remove support for deprecated syntax, along with some small changes in default configuration behavior and tool behavior. Highlights: * Only Corosync version 2 and greater is now supported as the underlying cluster layer. Support for Heartbeat and Corosync 1 (including CMAN) is removed. * The Pacemaker detail log file is now stored in /var/log/pacemaker/pacemaker.log by default. * The record-pending cluster property now defaults to true, which allows status tools such as crm_mon to show operations that are in progress. * Support for a number of deprecated build options, environment variables, and configuration settings has been removed. * The +master+ tag has been deprecated in favor of using a +clone+ tag with the new +promotable+ meta-attribute set to +true+. "Master/slave" clone resources are now referred to as "promotable" clone resources, though it will take longer for the full terminology change to be completed. * The public API for Pacemaker libraries that software applications can use has changed significantly. For a detailed list of changes, see the release notes and the https://wiki.clusterlabs.org/wiki/Pacemaker_2.0_Changes[Pacemaker 2.0 Changes] page on the ClusterLabs wiki. == What Changed in 1.0 == === New === * Failure timeouts. * New section for resource and operation defaults. * Tool for making offline configuration changes. * +Rules, instance_attributes, meta_attributes+ and sets of operations can be defined once and referenced in multiple places. * The CIB now accepts XPath-based create/modify/delete operations. See the pass:[cibadmin] help text. * Multi-dimensional colocation and ordering constraints. * The ability to connect to the CIB from non-cluster machines. * Allow recurring actions to be triggered at known times. === Changed === * Syntax ** All resource and cluster options now use dashes (-) instead of underscores (_) ** +master_slave+ was renamed to +master+ ** The +attributes+ container tag was removed ** The operation field +pre-req+ has been renamed +requires+ ** All operations must have an +interval+, +start+/+stop+ must have it set to zero * The +stonith-enabled+ option now defaults to true. * The cluster will refuse to start resources if +stonith-enabled+ is true (or unset) and no STONITH resources have been defined * The attributes of colocation and ordering constraints were renamed for clarity. * +resource-failure-stickiness+ has been replaced by +migration-threshold+. * The parameters for command-line tools have been made consistent * Switched to 'RelaxNG' schema validation and 'libxml2' parser ** id fields are now XML IDs which have the following limitations: *** id's cannot contain colons (:) *** id's cannot begin with a number *** id's must be globally unique (not just unique for that tag) ** Some fields (such as those in constraints that refer to resources) are IDREFs. + This means that they must reference existing resources or objects in order for the configuration to be valid. Removing an object which is referenced elsewhere will therefore fail. + ** The CIB representation, from which a MD5 digest is calculated to verify CIBs on the nodes, has changed. + This means that every CIB update will require a full refresh on any upgraded nodes until the cluster is fully upgraded to 1.0. This will result in significant performance degradation and it is therefore highly inadvisable to run a mixed 1.0/0.6 cluster for any longer than absolutely necessary. + * Ping node information no longer needs to be added to _ha.cf_. + Simply include the lists of hosts in your ping resource(s). === Removed === * Syntax ** It is no longer possible to set resource meta options as top-level attributes. Use meta attributes instead. ** Resource and operation defaults are no longer read from +crm_config+. diff --git a/doc/Pacemaker_Administration/en-US/Pacemaker_Administration.xml b/doc/Pacemaker_Administration/en-US/Pacemaker_Administration.xml index 03ce6bcbc0..5765d60b70 100644 --- a/doc/Pacemaker_Administration/en-US/Pacemaker_Administration.xml +++ b/doc/Pacemaker_Administration/en-US/Pacemaker_Administration.xml @@ -1,19 +1,19 @@ %BOOK_ENTITIES; ]> - +