diff --git a/doc/sphinx/Pacemaker_Administration/upgrading.rst b/doc/sphinx/Pacemaker_Administration/upgrading.rst
index c06151fb51..b23c65ea89 100644
--- a/doc/sphinx/Pacemaker_Administration/upgrading.rst
+++ b/doc/sphinx/Pacemaker_Administration/upgrading.rst
@@ -1,565 +1,565 @@
.. index:: upgrade
Upgrading a Pacemaker Cluster
-----------------------------
.. index:: version
Pacemaker Versioning
####################
Pacemaker has an overall release version, plus separate version numbers for
certain internal components.
.. index::
single: version; release
* **Pacemaker release version:** This version consists of three numbers
(*x.y.z*).
The major version number (the *x* in *x.y.z*) increases when at least some
rolling upgrades are not possible from the previous major version. For example,
a rolling upgrade from 1.0.8 to 1.1.15 should always be supported, but a
rolling upgrade from 1.0.8 to 2.0.0 may not be possible.
The minor version (the *y* in *x.y.z*) increases when there are significant
changes in cluster default behavior, tool behavior, and/or the API interface
(for software that utilizes Pacemaker libraries). The main benefit is to alert
you to pay closer attention to the release notes, to see if you might be
affected.
The release counter (the *z* in *x.y.z*) is increased with all public releases
of Pacemaker, which typically include both bug fixes and new features.
.. index::
single: feature set
single: version; feature set
* **CRM feature set:** This version number applies to the communication between
full cluster nodes, and is used to avoid problems in mixed-version clusters.
The major version number increases when nodes with different versions would not
work (rolling upgrades are not allowed). The minor version number increases
when mixed-version clusters are allowed only during rolling upgrades. The
minor-minor version number is ignored, but allows resource agents to detect
cluster support for various features. [#]_
Pacemaker ensures that the longest-running node is the cluster's DC. This
ensures new features are not enabled until all nodes are upgraded to support
them.
.. index::
single: version; Pacemaker Remote protocol
* **Pacemaker Remote protocol version:** This version applies to communication
between a Pacemaker Remote node and the cluster. It increases when an older
cluster node would have problems hosting the connection to a newer
Pacemaker Remote node. To avoid these problems, Pacemaker Remote nodes will
accept connections only from cluster nodes with the same or newer
Pacemaker Remote protocol version.
Unlike with CRM feature set differences between full cluster nodes,
mixed Pacemaker Remote protocol versions between Pacemaker Remote nodes and
full cluster nodes are fine, as long as the Pacemaker Remote nodes have the
older version. This can be useful, for example, to host a legacy application
in an older operating system version used as a Pacemaker Remote node.
.. index::
single: version; XML schema
* **XML schema version:** Pacemaker’s configuration syntax — what's allowed in
the Configuration Information Base (CIB) — has its own version. This allows
the configuration syntax to evolve over time while still allowing clusters
with older configurations to work without change.
.. index::
single: upgrade; methods
Upgrading Cluster Software
##########################
There are three approaches to upgrading a cluster, each with advantages and
disadvantages.
.. table:: **Upgrade Methods**
+---------------------------------------------------+----------+----------+--------+---------+----------+----------+
| Method | Available| Can be | Service| Service | Exercises| Allows |
| | between | used with| outage | recovery| failover | change of|
| | all | Pacemaker| during | during | logic | messaging|
| | versions | Remote | upgrade| upgrade | | layer |
| | | nodes | | | | [#]_ |
+===================================================+==========+==========+========+=========+==========+==========+
| Complete cluster shutdown | yes | yes | always | N/A | no | yes |
+---------------------------------------------------+----------+----------+--------+---------+----------+----------+
| Rolling (node by node) | no | yes | always | yes | yes | no |
| | | | [#]_ | | | |
+---------------------------------------------------+----------+----------+--------+---------+----------+----------+
| Detach and reattach | yes | no | only | no | no | yes |
| | | | due to | | | |
| | | | failure| | | |
+---------------------------------------------------+----------+----------+--------+---------+----------+----------+
.. index::
single: upgrade; shutdown
Complete Cluster Shutdown
_________________________
In this scenario, one shuts down all cluster nodes and resources,
then upgrades all the nodes before restarting the cluster.
#. On each node:
a. Shutdown the cluster software (pacemaker and the messaging layer).
#. Upgrade the Pacemaker software. This may also include upgrading the
messaging layer and/or the underlying operating system.
#. Check the configuration with the ``crm_verify`` tool.
#. On each node:
a. Start the cluster software.
Currently, only Corosync version 2 and greater is supported as the cluster
layer, but if another stack is supported in the future, the stack does not
need to be the same one before the upgrade.
One variation of this approach is to build a new cluster on new hosts.
This allows the new version to be tested beforehand, and minimizes downtime by
having the new nodes ready to be placed in production as soon as the old nodes
are shut down.
.. index::
single: upgrade; rolling upgrade
Rolling (node by node)
______________________
In this scenario, each node is removed from the cluster, upgraded, and then
brought back online, until all nodes are running the newest version.
Special considerations when planning a rolling upgrade:
* If you plan to upgrade other cluster software -- such as the messaging layer --
at the same time, consult that software's documentation for its compatibility
with a rolling upgrade.
* If the major version number is changing in the Pacemaker version you are
upgrading to, a rolling upgrade may not be possible. Read the new version's
release notes (as well the information here) for what limitations may exist.
* If the CRM feature set is changing in the Pacemaker version you are upgrading
to, you should run a mixed-version cluster only during a small rolling
upgrade window. If one of the older nodes drops out of the cluster for any
reason, it will not be able to rejoin until it is upgraded.
* If the Pacemaker Remote protocol version is changing, all cluster nodes
should be upgraded before upgrading any Pacemaker Remote nodes.
See the
`Pacemaker release calendar
`_
on the ClusterLabs wiki to figure out whether the CRM feature set and/or
Pacemaker Remote protocol version changed between the Pacemaker release versions
in your rolling upgrade.
To perform a rolling upgrade, on each node in turn:
#. Put the node into standby mode, and wait for any active resources
to be moved cleanly to another node. (This step is optional, but
allows you to deal with any resource issues before the upgrade.)
#. Shut down Pacemaker or ``pacemaker-remoted``.
#. If a cluster node, shut down the messaging layer.
#. Upgrade the Pacemaker software. This may also include upgrading the
messaging layer and/or the underlying operating system.
#. If this is the first node to be upgraded, check the configuration
with the ``crm_verify`` tool.
#. If a cluster node, start the messaging layer.
This must be the same messaging layer (currently only Corosync version 2 and
greater is supported) that the rest of the cluster is using.
#. Start Pacemaker or ``pacemaker-remoted``.
.. note::
Even if a rolling upgrade from the current version of the cluster to the
newest version is not directly possible, it may be possible to perform a
rolling upgrade in multiple steps, by upgrading to an intermediate version
first.
The following table lists compatible versions for all other nodes in the cluster
when upgrading a cluster node.
.. list-table:: **Version Compatibility for Cluster Nodes**
:class: longtable
:widths: 1 1
:header-rows: 1
* - Version Being Installed
- Minimum Compatible Version
* - Pacemaker 3.y.z
- Pacemaker 2.0.0
* - Pacemaker 2.y.z
- Pacemaker 1.1.11 [#]_
* - Pacemaker 1.y.z
- Pacemaker 1.0.0
* - Pacemaker 0.6.z to 0.7.z
- - Pacemaker 0.6.z
+ - Pacemaker 0.6.0
When upgrading a Pacemaker Remote node, all cluster nodes must be running at
least the minimum version listed in the table below.
.. list-table:: **Cluster Node Version Compatibility for Pacemaker Remote Nodes**
:class: longtable
:widths: 1 1
:header-rows: 1
* - Pacemaker Remote Version
- Minimum Cluster Node Version
* - Pacemaker 3.y.z
- Pacemaker 2.0.0
* - Pacemaker 1.1.9 to 2.1.z
- Pacemaker 1.1.9 [#]_
.. index::
single: upgrade; detach and reattach
Detach and Reattach
___________________
The reattach method is a variant of a complete cluster shutdown, where the
resources are left active and get re-detected when the cluster is restarted.
This method may not be used if the cluster contains any Pacemaker Remote nodes.
#. Tell the cluster to stop managing services. This is required to allow the
services to remain active after the cluster shuts down.
.. code-block:: none
# crm_attribute --name maintenance-mode --update true
#. On each node, shutdown the cluster software (pacemaker and the messaging
layer), and upgrade the Pacemaker software. This may also include upgrading
the messaging layer. While the underlying operating system may be upgraded
at the same time, that will be more likely to cause outages in the detached
services (certainly, if a reboot is required).
#. Check the configuration with the ``crm_verify`` tool.
#. On each node, start the cluster software.
Currently, only Corosync version 2 and greater is supported as the cluster
layer, but if another stack is supported in the future, the stack does not
need to be the same one before the upgrade.
#. Verify that the cluster re-detected all resources correctly.
#. Allow the cluster to resume managing resources again:
.. code-block:: none
# crm_attribute --name maintenance-mode --delete
.. note::
While the goal of the detach-and-reattach method is to avoid disturbing
running services, resources may still move after the upgrade if any
resource's location is governed by a rule based on transient node
attributes. Transient node attributes are erased when the node leaves the
cluster. A common example is using the ``ocf:pacemaker:ping`` resource to
set a node attribute used to locate other resources.
.. index::
pair: upgrade; CIB
Upgrading the Configuration
###########################
The CIB schema version can change from one Pacemaker version to another.
After cluster software is upgraded, the cluster will continue to use the older
schema version that it was previously using. This can be useful, for example,
when administrators have written tools that modify the configuration, and are
based on the older syntax. [#]_
However, when using an older syntax, new features may be unavailable, and there
is a performance impact, since the cluster must do a non-persistent
configuration upgrade before each transition. So while using the old syntax is
possible, it is not advisable to continue using it indefinitely.
Even if you wish to continue using the old syntax, it is a good idea to
follow the upgrade procedure outlined below, except for the last step, to ensure
that the new software has no problems with your existing configuration (since it
will perform much the same task internally).
If you are brave, it is sufficient simply to run ``cibadmin --upgrade``.
A more cautious approach would proceed like this:
#. Create a shadow copy of the configuration. The later commands will
automatically operate on this copy, rather than the live configuration.
.. code-block:: none
# crm_shadow --create shadow
.. index::
single: configuration; verify
#. Verify the configuration is valid with the new software (which may be
stricter about syntax mistakes, or may have dropped support for deprecated
features):
.. code-block:: none
# crm_verify --live-check
#. Fix any errors or warnings.
#. Perform the upgrade:
.. code-block:: none
# cibadmin --upgrade
#. If this step fails, there are three main possibilities:
a. The configuration was not valid to start with (did you do steps 2 and
3?).
#. The transformation failed; `report a bug `_.
#. The transformation was successful but produced an invalid result.
If the result of the transformation is invalid, you may see a number of
errors from the validation library. If these are not helpful, try the manual
upgrade procedure described below.
#. Check the changes:
.. code-block:: none
# crm_shadow --diff
If at this point there is anything about the upgrade that you wish to
fine-tune (for example, to change some of the automatic IDs), now is the
time to do so:
.. code-block:: none
# crm_shadow --edit
This will open the configuration in your favorite editor (whichever is
specified by the standard ``$EDITOR`` environment variable).
#. Preview how the cluster will react:
.. code-block:: none
# crm_simulate --live-check --save-dotfile shadow.dot -S
# dot -Tsvg shadow.dot -o shadow.svg
You can then view shadow.svg with any compatible image viewer or web
browser. Verify that either no resource actions will occur or that you are
happy with any that are scheduled. If the output contains actions you do
not expect (possibly due to changes to the score calculations), you may need
to make further manual changes. See :ref:`crm_simulate` for further details
on how to interpret the output of ``crm_simulate`` and ``dot``.
#. Upload the changes:
.. code-block:: none
# crm_shadow --commit shadow --force
In the unlikely event this step fails, please report a bug.
.. note::
It is also possible to perform the configuration upgrade steps manually:
#. Locate the ``upgrade*.xsl`` conversion scripts provided with the source
code. These will often be installed in a location such as
``/usr/share/pacemaker``, or may be obtained from the
`source repository `_.
#. Run the conversion scripts that apply to your older version, for example:
.. code-block:: none
# xsltproc /path/to/upgrade06.xsl config06.xml > config10.xml
#. Locate the ``pacemaker.rng`` script (from the same location as the xsl
files).
#. Check the XML validity:
.. code-block:: none
# xmllint --relaxng /path/to/pacemaker.rng config10.xml
The advantage of this method is that it can be performed without the cluster
running, and any validation errors are often more informative.
What Changed in 2.1
###################
The Pacemaker 2.1 release is fully backward-compatible in both the CIB XML and
the C API. Highlights:
* Pacemaker now supports the **OCF Resource Agent API version 1.1**.
Most notably, the ``Master`` and ``Slave`` role names have been renamed to
``Promoted`` and ``Unpromoted``.
* Pacemaker now supports colocations where the dependent resource does not
affect the primary resource's placement (via a new ``influence`` colocation
constraint option and ``critical`` resource meta-attribute). This is intended
for cases where a less-important resource must be colocated with an essential
resource, but it is preferred to leave the less-important resource stopped if
it fails, rather than move both resources.
* If Pacemaker is built with libqb 2.0 or later, the detail log will use
**millisecond-resolution timestamps**.
* In addition to crm_mon and stonith_admin, the crmadmin, crm_resource,
crm_simulate, and crm_verify commands now support the ``--output-as`` and
``--output-to`` options, including **XML output** (which scripts and
higher-level tools are strongly recommended to use instead of trying to parse
the text output, which may change from release to release).
For a detailed list of changes, see the release notes and
`Pacemaker 2.1 Changes
`_
on the ClusterLabs wiki.
What Changed in 2.0
###################
The main goal of the 2.0 release was to remove support for deprecated syntax,
along with some small changes in default configuration behavior and tool
behavior. Highlights:
* Only Corosync version 2 and greater is now supported as the underlying
cluster layer. Support for Heartbeat and Corosync 1 (including CMAN) is
removed.
* The Pacemaker detail log file is now stored in
``/var/log/pacemaker/pacemaker.log`` by default.
* The record-pending cluster property now defaults to true, which
allows status tools such as crm_mon to show operations that are in
progress.
* Support for a number of deprecated build options, environment variables,
and configuration settings has been removed.
* The ``master`` tag has been deprecated in favor of using the ``clone`` tag
with the new ``promotable`` meta-attribute set to ``true``. "Master/slave"
clone resources are now referred to as "promotable" clone resources.
* The public API for Pacemaker libraries that software applications can use
has changed significantly.
For a detailed list of changes, see the release notes and
`Pacemaker 2.0 Changes
`_
on the ClusterLabs wiki.
What Changed in 1.0
###################
New
___
* Failure timeouts.
* New section for resource and operation defaults.
* Tool for making offline configuration changes.
* ``Rules``, ``instance_attributes``, ``meta_attributes`` and sets of
operations can be defined once and referenced in multiple places.
* The CIB now accepts XPath-based create/modify/delete operations. See
``cibadmin --help``.
* Multi-dimensional colocation and ordering constraints.
* The ability to connect to the CIB from non-cluster machines.
* Allow recurring actions to be triggered at known times.
Changed
_______
* Syntax
* All resource and cluster options now use dashes (-) instead of underscores
(_)
* ``master_slave`` was renamed to ``master``
* The ``attributes`` container tag was removed
* The operation field ``pre-req`` has been renamed ``requires``
* All operations must have an ``interval``, ``start``/``stop`` must have it
set to zero
* The ``stonith-enabled`` option now defaults to true.
* The cluster will refuse to start resources if ``stonith-enabled`` is true (or
unset) and no STONITH resources have been defined
* The attributes of colocation and ordering constraints were renamed for
clarity.
* ``resource-failure-stickiness`` has been replaced by ``migration-threshold``.
* The parameters for command-line tools have been made consistent
* Switched to 'RelaxNG' schema validation and 'libxml2' parser
* id fields are now XML IDs which have the following limitations:
* id's cannot contain colons (:)
* id's cannot begin with a number
* id's must be globally unique (not just unique for that tag)
* Some fields (such as those in constraints that refer to resources) are
IDREFs.
This means that they must reference existing resources or objects in
order for the configuration to be valid. Removing an object which is
referenced elsewhere will therefore fail.
* The CIB representation, from which a MD5 digest is calculated to verify
CIBs on the nodes, has changed.
This means that every CIB update will require a full refresh on any
upgraded nodes until the cluster is fully upgraded to 1.0. This will result
in significant performance degradation and it is therefore highly
inadvisable to run a mixed 1.0/0.6 cluster for any longer than absolutely
necessary.
* Ping node information no longer needs to be added to ``ha.cf``. Simply
include the lists of hosts in your ping resource(s).
Removed
_______
* Syntax
* It is no longer possible to set resource meta options as top-level
attributes. Use meta-attributes instead.
* Resource and operation defaults are no longer read from ``crm_config``.
.. rubric:: Footnotes
.. [#] Before CRM feature set 3.1.0 (Pacemaker 2.0.0), the minor-minor version
number was treated the same as the minor version.
.. [#] Currently, Corosync version 2 and greater is the only supported cluster
stack, but other stacks have been supported by past versions, and may be
supported by future versions.
.. [#] Any active resources will be moved off the node being upgraded, so there
will be at least a brief outage unless all resources can be migrated
"live".
.. [#] Rolling upgrades from Pacemaker 1.1.z to 2.y.z are possible only if the
cluster uses corosync version 2 or greater as its messaging layer, and
the Cluster Information Base (CIB) uses schema 1.0 or higher in its
``validate-with`` property.
.. [#] Pacemaker Remote versions 1.1.15 through 1.1.17 require cluster nodes to
be at least version 1.1.15. Version 1.1.15 introduced an accidental
remote protocol version bump, breaking rolling upgrade compatibility with
older versions. This was fixed in 1.1.18.
.. [#] As of Pacemaker 2.0.0, only schema versions pacemaker-1.0 and higher
are supported (excluding pacemaker-1.1, which was a special case).
diff --git a/doc/sphinx/Pacemaker_Development/c.rst b/doc/sphinx/Pacemaker_Development/c.rst
index 91efe7cddb..3f3c9746fb 100644
--- a/doc/sphinx/Pacemaker_Development/c.rst
+++ b/doc/sphinx/Pacemaker_Development/c.rst
@@ -1,1129 +1,1129 @@
.. index::
single: C
pair: C; guidelines
C Coding Guidelines
-------------------
Pacemaker is a large project accepting contributions from developers with a
wide range of skill levels and organizational affiliations, and maintained by
multiple people over long periods of time. Following consistent guidelines
makes reading, writing, and reviewing code easier, and helps avoid common
mistakes.
Some existing Pacemaker code does not follow these guidelines, for historical
reasons and API backward compatibility, but new code should.
Code Organization
#################
Pacemaker's C code is organized as follows:
+-----------------+-----------------------------------------------------------+
| Directory | Contents |
+=================+===========================================================+
| daemons | the Pacemaker daemons (pacemakerd, pacemaker-based, etc.) |
+-----------------+-----------------------------------------------------------+
| include | header files for library APIs |
+-----------------+-----------------------------------------------------------+
| lib | libraries |
+-----------------+-----------------------------------------------------------+
| tools | command-line tools |
+-----------------+-----------------------------------------------------------+
Source file names should be unique across the entire project, to allow for
individual tracing via ``PCMK_trace_files``.
.. index::
single: C; library
single: C library
Pacemaker Libraries
###################
+---------------+---------+---------------+---------------------------+-------------------------------------+
| Library | Symbol | Source | API Headers | Description |
| | prefix | location | | |
+===============+=========+===============+===========================+=====================================+
| libcib | cib | lib/cib | | include/crm/cib.h | .. index:: |
| | | | | include/crm/cib/* | single: C library; libcib |
| | | | | single: libcib |
| | | | | |
| | | | | API for pacemaker-based IPC and |
| | | | | the CIB |
+---------------+---------+---------------+---------------------------+-------------------------------------+
| libcrmcluster | pcmk | lib/cluster | | include/crm/cluster.h | .. index:: |
| | | | | include/crm/cluster/* | single: C library; libcrmcluster |
| | | | | single: libcrmcluster |
| | | | | |
| | | | | Abstract interface to underlying |
| | | | | cluster layer |
+---------------+---------+---------------+---------------------------+-------------------------------------+
| libcrmcommon | pcmk | lib/common | | include/crm/common/* | .. index:: |
| | | | | some of include/crm/* | single: C library; libcrmcommon |
| | | | | single: libcrmcommon |
| | | | | |
| | | | | Everything else |
+---------------+---------+---------------+---------------------------+-------------------------------------+
| libcrmservice | svc | lib/services | | include/crm/services.h | .. index:: |
| | | | | single: C library; libcrmservice |
| | | | | single: libcrmservice |
| | | | | |
| | | | | Abstract interface to supported |
| | | | | resource types (OCF, LSB, etc.) |
+---------------+---------+---------------+---------------------------+-------------------------------------+
| liblrmd | lrmd | lib/lrmd | | include/crm/lrmd*.h | .. index:: |
| | | | | single: C library; liblrmd |
| | | | | single: liblrmd |
| | | | | |
| | | | | API for pacemaker-execd IPC |
+---------------+---------+---------------+---------------------------+-------------------------------------+
| libpacemaker | pcmk | lib/pacemaker | | include/pacemaker*.h | .. index:: |
| | | | | include/pcmki/* | single: C library; libpacemaker |
| | | | | single: libpacemaker |
| | | | | |
| | | | | High-level APIs equivalent to |
| | | | | command-line tool capabilities |
| | | | | (and high-level internal APIs) |
+---------------+---------+---------------+---------------------------+-------------------------------------+
| libpe_rules | pe | lib/pengine | | include/crm/pengine/* | .. index:: |
| | | | | single: C library; libpe_rules |
| | | | | single: libpe_rules |
| | | | | |
| | | | | Scheduler functionality related |
| | | | | to evaluating rules |
+---------------+---------+---------------+---------------------------+-------------------------------------+
| libpe_status | pe | lib/pengine | | include/crm/pengine/* | .. index:: |
| | | | | single: C library; libpe_status |
| | | | | single: libpe_status |
| | | | | |
| | | | | Low-level scheduler functionality |
+---------------+---------+---------------+---------------------------+-------------------------------------+
| libstonithd | stonith | lib/fencing | | include/crm/stonith-ng.h| .. index:: |
| | | | | include/crm/fencing/* | single: C library; libstonithd |
| | | | | single: libstonithd |
| | | | | |
| | | | | API for pacemaker-fenced IPC |
+---------------+---------+---------------+---------------------------+-------------------------------------+
Public versus Internal APIs
___________________________
Pacemaker libraries have both internal and public APIs. Internal APIs are those
used only within Pacemaker; public APIs are those offered (via header files and
documentation) for external code to use.
Generic functionality needed by Pacemaker itself, such as string processing or
XML processing, should remain internal, while functions providing useful
high-level access to Pacemaker capabilities should be public. When in doubt,
keep APIs internal, because it's easier to expose a previously internal API
than hide a previously public API.
Internal APIs can be changed as needed.
The public API/ABI should maintain a degree of stability so that external
applications using it do not need to be rewritten or rebuilt frequently. Many
OSes/distributions avoid breaking API/ABI compatibility within a major release,
so if Pacemaker breaks compatibility, that significantly delays when OSes
can package the new version. Therefore, changes to public APIs should be
backward-compatible (as detailed throughout this chapter), unless we are doing
a (rare) release where we specifically intend to break compatibility.
External applications known to use Pacemaker's public C API include
`sbd `_ and dlm_controld.
.. index::
pair: C; naming
API Symbol Naming
_________________
Exposed API symbols (non-``static`` function names, ``struct`` and ``typedef``
names in header files, etc.) must begin with the prefix appropriate to the
library (shown in the table at the beginning of this section). This reduces the
chance of naming collisions when external software links against the library.
The prefix is usually lowercase but may be all-caps for some defined constants
and macros.
Public API symbols should follow the library prefix with a single underbar
(for example, ``pcmk_something``), and internal API symbols with a double
underbar (for example, ``pcmk__other_thing``).
File-local symbols (such as static functions) and non-library code do not
require a prefix, though a unique prefix indicating an executable (controld,
crm_mon, etc.) can be helpful when symbols are shared between multiple
source files for the executable.
API Header File Naming
______________________
* Internal API headers should be named ending in ``_internal.h``, in the same
location as public headers, with the exception of libpacemaker, which for
historical reasons keeps internal headers in ``include/pcmki/pcmki_*.h``).
* If a library needs to share symbols just within the library, header files for
these should be named ending in ``_private.h`` and located in the library
source directory (not ``include``). Such functions should be declared as
``G_GNUC_INTERNAL``, to aid compiler efficiency (glib defines this
symbol appropriately for the compiler).
Header files that are not library API are kept in the same directory as the
source code they're included from.
The easiest way to tell what kind of API a symbol is, is to see where it's
declared. If it's in a public header, it's public API; if it's in an internal
header, it's internal API; if it's in a library-private header, it's
library-private API; otherwise, it's not an API.
.. index::
pair: C; API documentation
single: Doxygen
API Documentation
_________________
Pacemaker uses `Doxygen `_
to automatically generate its
`online API documentation `_,
so all public API (header files, functions, structs, enums, etc.) should be
documented with Doxygen comment blocks. Other code may be documented in the
same way if desired, with an ``\internal`` tag in the Doxygen comment.
Simple example of an internal function with a Doxygen comment block:
.. code-block:: c
/*!
* \internal
* \brief Return string length plus 1
*
* Return the number of characters in a given string, plus one.
*
* \param[in] s A string (must not be NULL)
*
* \return The length of \p s plus 1.
*/
static int
f(const char *s)
{
return strlen(s) + 1;
}
Function arguments are marked as ``[in]`` for input only, ``[out]`` for output
only, or ``[in,out]`` for both input and output.
``[in,out]`` should be used for struct pointer arguments if the function can
change any data accessed via the pointer. For example, if the struct contains
a ``GHashTable *`` member, the argument should be marked as ``[in,out]`` if the
function inserts data into the table, even if the struct members themselves are
not changed. However, an argument is not ``[in,out]`` if something reachable
via the argument is modified via a separate argument. For example, both
``pcmk_resource_t`` and ``pcmk_node_t`` contain pointers to their
``pcmk_scheduler_t`` and thus indirectly to each other, but if the function
modifies the resource via the resource argument, the node argument does not
have to be ``[in,out]``.
Public API Deprecation
______________________
Public APIs may not be removed in most Pacemaker releases, but they may be
deprecated.
When a public API is deprecated, it is moved to a header whose name ends in
``compat.h``. The original header includes the compatibility header only if the
``PCMK_ALLOW_DEPRECATED`` symbol is undefined or defined to 1. This allows
external code to continue using the deprecated APIs, but internal code is
prevented from using them because the ``crm_internal.h`` header defines the
symbol to 0.
.. index::
pair: C; boilerplate
pair: license; C
pair: copyright; C
C Boilerplate
#############
Every C file should start with a short copyright and license notice:
.. code-block:: c
/*
* Copyright the Pacemaker project contributors
*
* The version control history for this file may have further details.
*
* This source code is licensed under WITHOUT ANY WARRANTY.
*/
** should follow the policy set forth in the
`COPYING `_ file,
generally one of "GNU General Public License version 2 or later (GPLv2+)"
or "GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)".
Header files should additionally protect against multiple inclusion by defining
a unique symbol of the form ``PCMK____H``, and declare
C compatibility for inclusion by C++. For example:
.. code-block:: c
#ifndef PCMK__MY_HEADER__H
#define PCMK__MY_HEADER__H
// put #include directives here
#ifdef __cplusplus
extern "C" {
#endif
// put header code here
#ifdef __cplusplus
}
#endif
#endif // PCMK__MY_HEADER__H
Public API header files should give a Doxygen file description at the top of
the header code. For example:
.. code-block:: c
/*!
* \file
* \brief My brief description here
* \ingroup core
*/
.. index::
pair: C; whitespace
Line Formatting
###############
* Indentation must be 4 spaces, no tabs.
* Do not leave trailing whitespace.
* Lines should be no longer than 80 characters unless limiting line length
hurts readability.
.. index::
pair: C; comment
Comments
########
.. code-block:: c
/* Single-line comments may look like this */
// ... or this
/* Multi-line comments should start immediately after the comment opening.
* Subsequent lines should start with an aligned asterisk. The comment
* closing should be aligned and on a line by itself.
*/
.. index::
pair: C; operator
Operators
#########
.. code-block:: c
// Operators have spaces on both sides
x = a;
/* (1) Do not rely on operator precedence; use parentheses when mixing
* operators with different priority, for readability.
* (2) No space is used after an opening parenthesis or before a closing
* parenthesis.
*/
x = a + b - (c * d);
.. index::
single: C; if
single: C; else
single: C; while
single: C; for
single: C; switch
Control Statements (if, else, while, for, switch)
#################################################
.. code-block:: c
/*
* (1) The control keyword is followed by a space, a left parenthesis
* without a space, the condition, a right parenthesis, a space, and the
* opening bracket on the same line.
* (2) Always use braces around control statement blocks, even if they only
* contain one line. This makes code review diffs smaller if a line gets
* added in the future, and avoids the chance of bad indenting making a
* line incorrectly appear to be part of the block.
* (3) The closing bracket is on a line by itself.
*/
if (v < 0) {
return 0;
}
/* "else" and "else if" are on the same line with the previous ending brace
* and next opening brace, separated by a space. Blank lines may be used
* between blocks to help readability.
*/
if (v > 0) {
return 0;
} else if (a == 0) {
return 1;
} else {
return 2;
}
/* Do not use assignments in conditions. This ensures that the developer's
* intent is always clear, makes code reviews easier, and reduces the chance
* of using assignment where comparison is intended.
*/
// Do this ...
a = f();
if (a) {
return 0;
}
// ... NOT this
if (a = f()) {
return 0;
}
/* It helps readability to use the "!" operator only in boolean
* comparisons, and explicitly compare numeric values against 0,
* pointers against NULL, etc. This helps remind the reader of the
* type being compared.
*/
int i = 0;
char *s = NULL;
bool cond = false;
if (!cond) {
return 0;
}
if (i == 0) {
return 0;
}
if (s == NULL) {
return 0;
}
/* In a "switch" statement, indent "case" one level, and indent the body of
* each "case" another level.
*/
switch (expression) {
case 0:
command1;
break;
case 1:
command2;
break;
default:
command3;
break;
}
.. index::
pair: C; macro
Macros
######
Macros are a powerful but easily misused feature of the C preprocessor, and
Pacemaker uses a lot of obscure macro features. If you need to brush up, the
`GCC documentation for macros
`_ is excellent.
Some common issues:
* Beware of side effects in macro arguments that may be evaluated more than
once
* Always parenthesize macro arguments used in the macro body to avoid
precedence issues if the argument is an expression
* Multi-statement macro bodies should be enclosed in do...while(0) to make them
behave more like a single statement and avoid control flow issues
Often, a static inline function defined in a header is preferable to a macro,
to avoid the numerous issues that plague macros and gain the benefit of
argument and return value type checking.
.. index::
pair: C; memory
Memory Management
#################
* Always use ``calloc()`` rather than ``malloc()``. It has no additional cost on
modern operating systems, and reduces the severity and security risks of
uninitialized memory usage bugs.
* Ensure that all dynamically allocated memory is freed when no longer needed,
and not used after it is freed. This can be challenging in the more
event-driven, callback-oriented sections of code.
* Free dynamically allocated memory using the free function corresponding to
how it was allocated. For example, use ``free()`` with ``calloc()``, and
``g_free()`` with most glib functions that allocate objects.
.. index::
single: C; struct
Structures
##########
Changes to structures defined in public API headers (adding or removing
members, or changing member types) are generally not possible without breaking
API compatibility. However, there are exceptions:
* Public API structures can be designed such that they can be allocated only
via API functions, not declared directly or allocated with standard memory
functions using ``sizeof``.
* This can be enforced simply by documentating the limitation, in which case
new ``struct`` members can be added to the end of the structure without
breaking compatibility.
* Alternatively, the structure definition can be kept in an internal header,
with only a pointer type definition kept in a public header, in which case
the structure definition can be changed however needed.
.. index::
single: C; variable
Variables
#########
.. index::
single: C; pointer
Pointers
________
.. code-block:: c
/* (1) The asterisk goes by the variable name, not the type;
* (2) Avoid leaving pointers uninitialized, to lessen the impact of
* use-before-assignment bugs
*/
char *my_string = NULL;
// Use space before asterisk and after closing parenthesis in a cast
char *foo = (char *) bar;
.. index::
single: C; global variable
Globals
_______
Global variables should be avoided in libraries when possible. State
information should instead be passed as function arguments (often as a
structure). This is not for thread safety -- Pacemaker's use of forking
ensures it will never be threaded -- but it does minimize overhead,
improve readability, and avoid obscure side effects.
Variable Naming
_______________
Time intervals are sometimes represented in Pacemaker code as user-defined
text specifications (for example, "10s"), other times as an integer number of
seconds or milliseconds, and still other times as a string representation
of an integer number. Variables for these should be named with an indication
of which is being used (for example, use ``interval_spec``, ``interval_ms``,
or ``interval_ms_s`` instead of ``interval``).
.. index::
pair: C; booleans
pair: C; bool
pair: C; gboolean
Booleans
________
Booleans in C can be represented by an integer type, ``bool``, or ``gboolean``.
Integers are sometimes useful for storing booleans when they must be converted
to and from a string, such as an XML attribute value (for which
``crm_element_value_int()`` can be used). Integer booleans use 0 for false and
nonzero (usually 1) for true.
``gboolean`` should be used with glib APIs that specify it. ``gboolean`` should
always be used with glib's ``TRUE`` and ``FALSE`` constants.
Otherwise, ``bool`` should be preferred. ``bool`` should be used with the
``true`` and ``false`` constants from the ``stdbool.h`` header.
Do not use equality operators when testing booleans. For example:
.. code-block:: c
// Do this
if (bool1) {
fn();
}
if (!bool2) {
fn2();
}
// Not this
if (bool1 == true) {
fn();
}
if (bool2 == false) {
fn2();
}
// Otherwise there's no logical end ...
if ((bool1 == false) == true) {
fn();
}
.. index::
pair: C; strings
String Handling
###############
Define Constants for Magic Strings
__________________________________
A "magic" string is one used for control purposes rather than human reading,
and which must be exactly the same every time it is used. Examples would be
configuration option names, XML attribute names, or environment variable names.
These should always be defined constants, rather than using the string literal
everywhere. If someone mistypes a defined constant, the code won't compile, but
if they mistype a literal, it could go unnoticed until a user runs into a
problem.
String-Related Library Functions
________________________________
Pacemaker's libcrmcommon has a large number of functions to assist in string
handling. The most commonly used ones are:
* ``pcmk__str_eq()`` tests string equality (similar to ``strcmp()``), but can
handle NULL, and takes options for case-insensitive, whether NULL should be
considered a match, etc.
* ``crm_strdup_printf()`` takes ``printf()``-style arguments and creates a
string from them (dynamically allocated, so it must be freed with
``free()``). It asserts on memory failure, so the return value is always
non-NULL.
String handling functions should almost always be internal API, since Pacemaker
isn't intended to be used as a general-purpose library. Most are declared in
``include/crm/common/strings_internal.h``. ``util.h`` has some older ones that
are public API (for now, but will eventually be made internal).
char*, gchar*, and GString
__________________________
When using dynamically allocated strings, be careful to always use the
appropriate free function.
* ``char*`` strings allocated with something like ``calloc()`` must be freed
with ``free()``. Most Pacemaker library functions that allocate strings use
this implementation.
* glib functions often use ``gchar*`` instead, which must be freed with
``g_free()``.
* Occasionally, it's convenient to use glib's flexible ``GString*`` type, which
must be freed with ``g_string_free()``.
.. index::
pair: C; regular expression
Regular Expressions
___________________
- Use ``REG_NOSUB`` with ``regcomp()`` whenever possible, for efficiency.
- Be sure to use ``regfree()`` appropriately.
.. index::
single: C; enum
Enumerations
############
* Enumerations should not have a ``typedef``, and do not require any naming
convention beyond what applies to all exposed symbols.
* New values should usually be added to the end of public API enumerations,
because the compiler will define the values to 0, 1, etc., in the order
given, and inserting a value in the middle would change the numerical values
of all later values, breaking code compiled with the old values. However, if
enum numerical values are explicitly specified rather than left to the
compiler, new values can be added anywhere.
* When defining constant integer values, enum should be preferred over
``#define`` or ``const`` when possible. This allows type checking without
consuming memory.
Flag groups
___________
Pacemaker often uses flag groups (also called bit fields or bitmasks) for a
collection of boolean options (flags/bits).
This is more efficient for storage and manipulation than individual booleans,
but its main advantage is when used in public APIs, because using another bit
in a bitmask is backward compatible, whereas adding a new function argument (or
sometimes even a structure member) is not.
.. code-block:: c
#include
/* (1) Define an enumeration to name the individual flags, for readability.
* An enumeration is preferred to a series of "#define" constants
* because it is typed, and logically groups the related names.
* (2) Define the values using left-shifting, which is more readable and
* less error-prone than hexadecimal literals (0x0001, 0x0002, 0x0004,
* etc.).
* (3) Using a comma after the last entry makes diffs smaller for reviewing
* if a new value needs to be added or removed later.
*/
enum pcmk__some_bitmask_type {
pcmk__some_value = (1 << 0),
pcmk__other_value = (1 << 1),
pcmk__another_value = (1 << 2),
};
/* The flag group itself should be an unsigned type from stdint.h (not
* the enum type, since it will be a mask of the enum values and not just
* one of them). uint32_t is the most common, since we rarely need more than
* 32 flags, but a smaller or larger type could be appropriate in some
* cases.
*/
uint32_t flags = pcmk__some_value|pcmk__other_value;
/* If the values will be used only with uint64_t, define them accordingly,
* to make compilers happier.
*/
enum pcmk__something_else {
pcmk__whatever = (UINT64_C(1) << 0),
};
We have convenience functions for checking flags (see ``pcmk_any_flags_set()``,
``pcmk_all_flags_set()``, and ``pcmk_is_set()``) as well as setting and
clearing them (see ``pcmk__set_flags_as()`` and ``pcmk__clear_flags_as()``,
usually used via wrapper macros defined for specific flag groups). These
convenience functions should be preferred to direct bitwise arithmetic, for
readability and logging consistency.
.. index::
pair: C; function
Functions
#########
Function Naming
_______________
Function names should be unique across the entire project, to allow for
individual tracing via ``PCMK_trace_functions``, and make it easier to search
code and follow detail logs.
.. _sort_func:
Sorting
^^^^^^^
A function that sorts an entire list should have ``sort`` in its name. It sorts
elements using a :ref:`comparison ` function, which may be either
hard-coded or passed as an argument.
.. _compare_func:
Comparison
^^^^^^^^^^
A comparison function for :ref:`sorting ` should have ``cmp`` in its
name and should *not* have ``sort`` in its name.
.. _constructor_func:
Constructors
^^^^^^^^^^^^
A constructor creates a new dynamically allocated object. It may perform some
initialization procedure on the new object.
* If the constructor always creates an independent object instance, its name
should include ``new``.
* If the constructor may add the new object to some existing object, its name
should include ``create``.
Function Definitions
____________________
.. code-block:: c
/*
* (1) The return type goes on its own line
* (2) The opening brace goes by itself on a line
* (3) Use "const" with pointer arguments whenever appropriate, to allow the
* function to be used by more callers.
*/
int
my_func1(const char *s)
{
return 0;
}
/* Functions with no arguments must explicitly list them as void,
* for compatibility with strict compilers
*/
int
my_func2(void)
{
return 0;
}
/*
* (1) For functions with enough arguments that they must break to the next
* line, align arguments with the first argument.
* (2) When a function argument is a function itself, use the pointer form.
* (3) Declare functions and file-global variables as ``static`` whenever
* appropriate. This gains a slight efficiency in shared libraries, and
* helps the reader know that it is not used outside the one file.
*/
static int
my_func3(int bar, const char *a, const char *b, const char *c,
void (*callback)())
{
return 0;
}
Return Values
_____________
Functions that need to indicate success or failure should follow one of the
following guidelines. More details, including functions for using them in user
messages and converting from one to another, can be found in
``include/crm/common/results.h``.
* A **standard Pacemaker return code** is one of the ``pcmk_rc_*`` enum values
or a system errno code, as an ``int``.
* ``crm_exit_t`` (the ``CRM_EX_*`` enum values) is a system-independent code
suitable for the exit status of a process, or for interchange between nodes.
* Other special-purpose status codes exist, such as ``enum ocf_exitcode`` for
the possible exit statuses of OCF resource agents (along with some
Pacemaker-specific extensions). It is usually obvious when the context calls
for such.
* Some older Pacemaker APIs use the now-deprecated "legacy" return values of
``pcmk_ok`` or the positive or negative value of one of the ``pcmk_err_*``
constants or system errno codes.
* Functions registered with external libraries (as callbacks for example)
should use the appropriate signature defined by those libraries, rather than
follow Pacemaker guidelines.
Of course, functions may have return values that aren't success/failure
indicators, such as a pointer, integer count, or bool.
:ref:`Comparison ` functions should return
* a negative integer if the first argument should sort first
* 0 if its arguments are equal for sorting purposes
* a positive integer is the second argument should sort first
Public API Functions
____________________
Unless we are doing a (rare) release where we break public API compatibility,
new public API functions can be added, but existing function signatures (return
type, name, and argument types) should not be changed. To work around this, an
existing function can become a wrapper for a new function.
.. index::
pair: C; logging
pair: C; output
Logging and Output
##################
Logging Vs. Output
__________________
Log messages and output messages are logically similar but distinct.
Oversimplifying a bit, daemons log, and tools output.
Log messages are intended to help with troubleshooting and debugging.
They may have a high level of technical detail, and are usually filtered by
severity -- for example, the system log by default gets messages of notice
level and higher.
Output is intended to let the user know what a tool is doing, and is generally
terser and less technical, and may even be parsed by scripts. Output might have
"verbose" and "quiet" modes, but it is not filtered by severity.
Common Guidelines for All Messages
__________________________________
* When format strings are used for derived data types whose implementation may
vary across platforms (``pid_t``, ``time_t``, etc.), the safest approach is
to use ``%lld`` in the format string, and cast the value to ``long long``.
* Do not rely on ``%s`` handling ``NULL`` values properly. While the standard
library functions might, not all functions using printf-style formatting
does, and it's safest to get in the habit of always ensuring format values
are non-NULL. If a value can be NULL, the ``pcmk__s()`` function is a
convenient way to say "this string if not NULL otherwise this default".
* The convenience macros ``pcmk__plural_s()`` and ``pcmk__plural_alt()`` are
handy when logging a word that may be singular or plural.
Log Levels
__________
When to use each log level:
* **critical:** fatal error (usually something that would make a daemon exit)
* **error:** failure of something that affects the cluster (such as a resource
action, fencing action, etc.) or daemon operation
* **warning:** minor, potential, or recoverable failures (such as something
only affecting a daemon client, or invalid configuration that can be left to
default)
* **notice:** important successful events (such as a node joining or leaving,
resource action results, or configuration changes)
* **info:** events that would be helpful with troubleshooting (such as status
section updates or elections)
* **debug:** information that would be helpful for debugging code or complex
problems
* **trace:** like debug but for very noisy or low-level stuff
By default, critical through notice are logged to the system log and detail
log, info is logged to the detail log only, and debug and trace are not logged
(if enabled, they go to the detail log only).
Logging
_______
Pacemaker uses libqb for logging, but wraps it with a higher level of
functionality (see ``include/crm/common/logging*h``).
A few macros ``crm_err()``, ``crm_warn()``, etc. do most of the heavy lifting.
By default, Pacemaker sends logs at notice level and higher to the system log,
and logs at info level and higher to the detail log (typically
``/var/log/pacemaker/pacemaker.log``). The intent is that most users will only
ever need the system log, but for deeper troubleshooting and developer
debugging, the detail log may be helpful, at the cost of being more technical
and difficult to follow.
The same message can have more detail in the detail log than in the system log,
using libqb's "extended logging" feature:
.. code-block:: c
/* The following will log a simple message in the system log, like:
warning: Action failed: Node not found
with extra detail in the detail log, like:
warning: Action failed: Node not found | rc=-1005 id=hgjjg-51006
*/
crm_warn("Action failed: %s " QB_XS " rc=%d id=%s",
pcmk_rc_str(rc), rc, id);
Assertion Logging
_________________
``pcmk__assert(expr)``
- If ``expr`` is false, this will call crm_err() with a "Triggered
- fatal assertion" message (with details), then abort execution. This should be
+ If ``expr`` is false, this will call ``crm_err()`` with a "Triggered fatal
+ assertion" message (with details), then abort execution. This should be
used for logic errors that should be impossible (such as a NULL function
argument where not accepted) and environmental errors that can't be handled
gracefully (for example, memory allocation failures, though returning
``ENOMEM`` is often better).
``CRM_LOG_ASSERT(expr)``
If ``expr`` is false, this will generally log a message without aborting. If
the log level is below trace, it just calls ``crm_err()`` with a "Triggered
assert" message (with details). If the log level is trace, and the caller is
a daemon, then it will fork a child process in which to dump core, as well as
logging the message. If the log level is trace, and the caller is not a
daemon, then it will behave like ``pcmk__assert()`` (i.e. log and abort).
This should be used for logic or protocol errors that require no special
handling.
``CRM_CHECK(expr, failed_action)``
If ``expr`` is false, behave like ``CRM_LOG_ASSERT(expr)`` (that is, log a
message and dump core if requested) then perform ``failed_action`` (which
must not contain ``continue``, ``break``, or ``errno``). This should be used
for logic or protocol errors that can be handled, usually by returning an
error status.
Output
______
Pacemaker has a somewhat complicated system for tool output. The main benefit
is that the user can select the output format with the ``--output-as`` option
(usually "text" for human-friendly output or "xml" for reliably script-parsable
output, though ``crm_mon`` additionally supports "console" and "html").
A custom message can be defined with a unique string identifier, plus
implementation functions for each supported format. The caller invokes the
message using the identifier. The user selects the output format via
``--output-as``, and the output code automatically calls the appropriate
implementation function. Custom messages are useful when you want to output
messages that are more complex than a one-line error or informational message,
reproducible, and automatically handled by the output formatting system.
Custom messages can contain other custom messages.
Custom message functions are implemented as follows: Start with the macro
``PCMK__OUTPUT_ARGS``, whose arguments are the message name, followed by the
arguments to the message. Then there is the function declaration, for which the
arguments are the pointer to the current output object, then a variable argument
list.
To output a custom message, you first need to create, i.e. register, the custom
message that you want to output. Either call ``register_message``, which
registers a custom message at runtime, or make use of the collection of
predefined custom messages in ``fmt_functions``, which is defined in
``lib/pacemaker/pcmk_output.c``. Once you have the message to be outputted,
output it by calling ``message``.
Note: The ``fmt_functions`` functions accommodate all of the output formats;
the default implementation accommodates any format that isn't explicitly
accommodated. The default output provides valid output for any output format,
but you may still want to implement a specific output, i.e. xml, text, or html.
The ``message`` function automatically knows which implementation to use,
because the ``pcmk__output_s`` contains this information.
The interface (most importantly ``pcmk__output_t``) is declared in
``include/crm/common/output*h``. See the API comments and existing tools for
examples.
Some of its important member functions are ``err``, which formats error messages
and ``info``, which formats informational messages. Also, ``list_item``,
which formats list items, ``begin_list``, which starts lists, and ``end_list``,
which ends lists, are important because lists can be useful, yet differently
handled by the different output types.
.. index::
pair: C; XML
XML
###
External Libraries
__________________
Pacemaker uses `libxml2 `_ and
`libxslt `_ to process XML. These
libraries implement only version 1.0 of the XML, XPath, and XSLT specifications.
Naming
______
Names of functions, constants, and enum values related to XML should contain
substrings indicating the type of object they're used with, according to the
following convention:
* ``xml``: XML subtree, or XML generically
* ``xe``: XML element node, including the attributes belonging to an element
* ``xa``: XML attribute node
* ``xc``: XML comment node
Private Data
____________
Libxml2 data structures such as ``xmlNode`` and ``xmlDoc`` contain a
``void *_private`` member for application-specific data. Pacemaker uses this
field to store internal bookkeeping data, such as changes relative to another
XML tree, or ACLs.
XML documents, elements, attributes, and comments have private data. The private
data field must be allocated immediately after the node is created and freed
immediately before the node is freed.
Wrapper Functions
_________________
Pacemaker provides wrappers for a variety of libxml2 and libxslt functions. They
should be used whenever possible. Some are merely for convenience. However, many
perform additional, Pacemaker-specific tasks, such as change tracking, ACL
checking, and allocation/deallocation of XML documents and private data.
Pacemaker assumes that every XML node is part of a document and has private data
allocated. If libxml2 APIs are used directly instead of the wrapper functions,
Pacemaker may crash with a segmentation fault, or change tracking and ACL
checking may be incorrectly disabled.
.. index::
single: Makefile.am
Makefiles
#########
Pacemaker uses
`automake `_
for building, so the Makefile.am in each directory should be edited rather than
Makefile.in or Makefile, which are automatically generated.
* Public API headers are installed (by adding them to a ``HEADERS`` variable in
``Makefile.am``), but internal API headers are not (by adding them to
``noinst_HEADERS``).
.. index::
pair: C; vim settings
vim Settings
############
Developers who use ``vim`` to edit source code can add the following settings
to their ``~/.vimrc`` file to follow Pacemaker C coding guidelines:
.. code-block:: none
" follow Pacemaker coding guidelines when editing C source code files
filetype plugin indent on
au FileType c setlocal expandtab tabstop=4 softtabstop=4 shiftwidth=4 textwidth=80
autocmd BufNewFile,BufRead *.h set filetype=c
let c_space_errors = 1
diff --git a/doc/sphinx/Pacemaker_Development/helpers.rst b/doc/sphinx/Pacemaker_Development/helpers.rst
index 2fa9b4d9b6..5a7da9bae0 100644
--- a/doc/sphinx/Pacemaker_Development/helpers.rst
+++ b/doc/sphinx/Pacemaker_Development/helpers.rst
@@ -1,641 +1,642 @@
C Development Helpers
---------------------
.. index::
single: unit testing
Refactoring
###########
Pacemaker uses an optional tool called `coccinelle `_
to do automatic refactoring. coccinelle is a very complicated tool that can be
difficult to understand, and the existing documentation makes it pretty tough
to get started. Much of the documentation is either aimed at kernel developers
or takes the form of grammars.
However, it can apply very complex transformations across an entire source tree.
This is useful for tasks like code refactoring, changing APIs (number or type of
arguments, etc.), catching functions that should not be called, and changing
existing patterns.
coccinelle is driven by input scripts called `semantic patches `_
written in its own language. These scripts bear a passing resemblance to source
code patches and tell coccinelle how to match and modify a piece of source
code. They are stored in ``devel/coccinelle`` and each script either contains
a single source transformation or several related transformations. In general,
we try to keep these as simple as possible.
In Pacemaker development, we use a couple targets in ``devel/Makefile.am`` to
control coccinelle. The ``cocci`` target tries to apply each script to every
Pacemaker source file, printing out any changes it would make to the console.
The ``cocci-inplace`` target does the same but also makes those changes to the
source files. A variety of warnings might also be printed. If you aren't working
on a new script, these can usually be ignored.
If you are working on a new coccinelle script, it can be useful (and faster) to
skip everything else and only run the new script. The ``COCCI_FILES`` variable
can be used for this:
.. code-block:: none
$ make -C devel COCCI_FILES=coccinelle/new-file.cocci cocci
This variable is also used for preventing some coccinelle scripts in the Pacemaker
source tree from running. Some scripts are disabled because they are not currently
fully working or because they are there as templates. When adding a new script,
remember to add it to this variable if it should always be run.
One complication when writing coccinelle scripts is that certain Pacemaker source
files may not use private functions (those whose name starts with ``pcmk__``).
Handling this requires work in both the Makefile and in the coccinelle scripts.
The Makefile deals with this by maintaining two lists of source files: those that
may use private functions and those that may not. For those that may, a special
argument (``-D internal``) is added to the coccinelle command line. This creates
a virtual dependency named ``internal``.
In the coccinelle scripts, those transformations that modify source code to use
a private function also have a dependency on ``internal``. If that dependency
was given on the command line, the transformation will be run. Otherwise, it will
be skipped.
This means that not all instances of an older style of code will be changed after
running a given transformation. Some developer intervention is still necessary
to know whether a source code block should have been changed or not.
Probably the easiest way to learn how to use coccinelle is by following other
people's scripts. In addition to the ones in the Pacemaker source directory,
there's several others on the `coccinelle website `_.
Sanitizers
##########
gcc supports a variety of run-time checks called sanitizers. These can be used to
catch programming errors with memory, race conditions, various undefined behavior
conditions, and more. Because these are run-time checks, they should only be used
during development and not in compiled packages or production code.
Certain sanitizers cannot be combined with others because their run-time checks
cause interfere. Instead of trying to figure out which combinations work, it is
simplest to just enable one at a time.
Each supported sanitizer requires an installed libray. In addition to just
enabling the sanitizer, their use can be configured with environment variables.
For example:
.. code-block:: none
$ ASAN_OPTIONS=verbosity=1:replace_str=true crm_mon -1R
Pacemaker supports the following subset of gcc's sanitizers:
+--------------------+-------------------------+----------+----------------------+
| Sanitizer | Configure Option | Library | Environment Variable |
+====================+=========================+==========+======================+
| Address | --with-sanitizers=asan | libasan | ASAN_OPTIONS |
+--------------------+-------------------------+----------+----------------------+
| Threads | --with-sanitizers=tsan | libtsan | TSAN_OPTIONS |
+--------------------+-------------------------+----------+----------------------+
| Undefined behavior | --with-sanitizers=ubsan | libubsan | UBSAN_OPTIONS |
+--------------------+-------------------------+----------+----------------------+
The undefined behavior sanitizer further supports suboptions that need to be
given as CFLAGS when configuring pacemaker:
.. code-block:: none
$ CFLAGS=-fsanitize=integer-divide-by-zero ./configure --with-sanitizers=ubsan
For more information, see the `gcc documentation `_
which also provides links to more information on each sanitizer.
Unit Testing
############
Where possible, changes to the C side of Pacemaker should be accompanied by unit
tests. Much of Pacemaker cannot effectively be unit tested (and there are other
testing systems used for those parts), but the ``lib`` subdirectory is pretty easy
to write tests for.
Pacemaker uses the `cmocka unit testing framework `_ which looks
a lot like other unit testing frameworks for C and should be fairly familiar. In
addition to regular unit tests, cmocka also gives us the ability to use
`mock functions `_ for unit testing
functions that would otherwise be difficult to test.
Organization
____________
Pay close attention to the organization and naming of test cases to ensure the
unit tests continue to work as they should.
Tests are spread throughout the source tree, alongside the source code they test.
For instance, all the tests for the source code in ``lib/common/`` are in the
``lib/common/tests`` directory. If there is no ``tests`` subdirectory, there are no
tests for that library yet.
Under that directory, there is a ``Makefile.am`` and additional subdirectories. Each
subdirectory contains the tests for a single library source file. For instance,
all the tests for ``lib/common/strings.c`` are in the ``lib/common/tests/strings``
directory. Note that the test subdirectory does not have a ``.c`` suffix. If there
is no test subdirectory, there are no tests for that file yet.
Finally, under that directory, there is a ``Makefile.am`` and then various source
files. Each of these source files tests the single function that it is named
after. For instance, ``lib/common/tests/strings/pcmk__btoa_test.c`` tests the
``pcmk__btoa()`` function in ``lib/common/strings.c``. If there is no test
source file, there are no tests for that function yet.
The ``_test`` suffix on the test source file is important. All tests have this
suffix, which means all the compiled test cases will also end with this suffix.
That lets us ignore all the compiled tests with a single line in ``.gitignore``:
.. code-block:: none
/lib/*/tests/*/*_test
Adding a test
_____________
Testing a new function in an already testable source file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Follow these steps if you want to test a function in a source file where there
are already other tested functions. For the purposes of this example, we will
add a test for the ``pcmk__scan_port()`` function in ``lib/common/strings.c``. As
you can see, there are already tests for other functions in this same file in
the ``lib/common/tests/strings`` directory.
* cd into ``lib/common/tests/strings``
* Add the new file to the ``check_PROGRAMS`` variable in ``Makefile.am``, making
it something like this:
.. code-block:: none
check_PROGRAMS = \
pcmk__add_word_test \
pcmk__btoa_test \
pcmk__scan_port_test
* Create a new ``pcmk__scan_port_test.c`` file, copying the copyright and include
boilerplate from another file in the same directory.
* Continue with the steps in `Writing the test`_.
Testing a function in a source file without tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Follow these steps if you want to test a function in a source file where there
are not already other tested functions, but there are tests for other files in
the same library. For the purposes of this example, we will add a test for the
``pcmk_acl_required()`` function in ``lib/common/acls.c``. At the time of this
documentation being written, no tests existed for that source file, so there
is no ``lib/common/tests/acls`` directory.
* Add to ``AC_CONFIG_FILES`` in the top-level ``configure.ac`` file so the build
process knows to use directory we're about to create. That variable would
now look something like:
.. code-block:: none
dnl Other files we output
AC_CONFIG_FILES(Makefile \
...
lib/common/tests/Makefile \
lib/common/tests/acls/Makefile \
lib/common/tests/agents/Makefile \
...
)
* cd into ``lib/common/tests``
* Add to the ``SUBDIRS`` variable in ``Makefile.am``, making it something like:
.. code-block:: none
SUBDIRS = agents acls cmdline flags operations strings utils xpath results
* Create a new ``acls`` directory, copying the ``Makefile.am`` from some other
directory. At this time, each ``Makefile.am`` is largely boilerplate with
very little that needs to change from directory to directory.
* cd into ``acls``
* Get rid of any existing values for ``check_PROGRAMS`` and set it to
``pcmk_acl_required_test`` like so:
.. code-block:: none
check_PROGRAMS = pcmk_acl_required_test
* Double check that the following includes are at the top of ``Makefile.am``:
.. code-block:: none
include $(top_srcdir)/mk/common.mk
include $(top_srcdir)/mk/tap.mk
include $(top_srcdir)/mk/unittest.mk
* If necessary, settings from those includes can be overridden like so:
.. code-block:: none
AM_TESTS_ENVIRONMENT += PCMK_CTS_CLI_DIR=$(top_srcdir)/cts/cli
AM_CPPFLAGS += -I$(top_srcdir)
LDADD += $(top_builddir)/lib/pengine/libpe_status_test.la
* Follow the steps in `Testing a new function in an already testable source file`_
to create the new ``pcmk_acl_required_test.c`` file.
Testing a function in a library without tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Adding a test case for a function in a library that doesn't have any test cases
to begin with is only slightly more complicated. In general, the steps are the
same as for the previous section, except with an additional layer of directory
creation.
For the purposes of this example, we will add a test case for the
``lrmd_send_resource_alert()`` function in ``lib/lrmd/lrmd_alerts.c``. Note that this
may not be a very good function or even library to write actual unit tests for.
* Add to ``AC_CONFIG_FILES`` in the top-level ``configure.ac`` file so the build
process knows to use directory we're about to create. That variable would
now look something like:
.. code-block:: none
dnl Other files we output
AC_CONFIG_FILES(Makefile \
...
lib/lrmd/Makefile \
lib/lrmd/tests/Makefile \
lib/services/Makefile \
...
)
* cd into ``lib/lrmd``
* Create a ``SUBDIRS`` variable in ``Makefile.am`` if it doesn't already exist.
Most libraries should not have this variable already.
.. code-block:: none
SUBDIRS = tests
* Create a new ``tests`` directory and add a ``Makefile.am`` with the following
contents:
.. code-block:: none
SUBDIRS = lrmd_alerts
* Follow the steps in `Testing a function in a source file without tests`_ to create
the rest of the new directory structure.
* Follow the steps in `Testing a new function in an already testable source file`_
to create the new ``lrmd_send_resource_alert_test.c`` file.
Adding to an existing test case
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If all you need to do is add additional test cases to an existing file, none of
the above work is necessary. All you need to do is find the test source file
with the name matching your function and add to it and then follow the
instructions in `Writing the test`_.
Writing the test
________________
A test case file contains a fair amount of boilerplate. For this reason, it's
usually easiest to just copy an existing file and adapt it to your needs. However,
here's the basic structure:
.. code-block:: c
/*
* Copyright 2021 the Pacemaker project contributors
*
* The version control history for this file may have further details.
*
* This source code is licensed under the GNU Lesser General Public License
* version 2.1 or later (LGPLv2.1+) WITHOUT ANY WARRANTY.
*/
#include
#include
/* Put your test-specific includes here */
/* Put your test functions here */
PCMK__UNIT_TEST(NULL, NULL,
/* Register your test functions here */)
Each test-specific function should test one aspect of the library function,
though it can include many assertions if there are many ways of testing that
one aspect. For instance, there might be multiple ways of testing regular
expression matching:
.. code-block:: c
static void
regex(void **state) {
const char *s1 = "abcd";
const char *s2 = "ABCD";
assert_true(pcmk__strcmp(NULL, "a..d", pcmk__str_regex) < 0);
assert_true(pcmk__strcmp(s1, NULL, pcmk__str_regex) > 0);
assert_int_equal(pcmk__strcmp(s1, "a..d", pcmk__str_regex), 0);
}
Each test-specific function must also be registered or it will not be called.
This is done with ``cmocka_unit_test()`` in the ``PCMK__UNIT_TEST`` macro:
.. code-block:: c
PCMK__UNIT_TEST(NULL, NULL,
cmocka_unit_test(regex))
Most unit tests do not require a setup and teardown function to be executed
around the entire group of tests. On occassion, this may be necessary. Simply
pass those functions in as the first two parameters to ``PCMK__UNIT_TEST``
instead of using NULL.
Assertions
__________
-In addition to the `assertions provided by `_,
-``unittest_internal.h`` also provides ``pcmk__assert_asserts``. This macro takes an
-expression and verifies that the expression aborts due to a failed call to
-``pcmk__assert()`` or some other similar function. It can be used like so:
+In addition to the `assertions provided by cmocka
+`_, ``unittest_internal.h``
+also provides ``pcmk__assert_asserts``. This macro takes an expression and
+verifies that the expression aborts due to a failed call to ``pcmk__assert()``
+or some other similar function. It can be used like so:
.. code-block:: c
static void
null_input_variables(void **state)
{
long long start, end;
pcmk__assert_asserts(pcmk__parse_ll_range("1234", NULL, &end));
pcmk__assert_asserts(pcmk__parse_ll_range("1234", &start, NULL));
}
Here, ``pcmk__parse_ll_range`` expects non-NULL for its second and third
arguments. If one of those arguments is NULL, ``pcmk__assert()`` will fail and
the program will abort. ``pcmk__assert_asserts`` checks that the code would
abort and the test passes. If the code does not abort, the test fails.
Running
_______
If you had to create any new files or directories, you will first need to run
``./configure`` from the top level of the source directory. This will regenerate
the Makefiles throughout the tree. If you skip this step, your changes will be
skipped and you'll be left wondering why the output doesn't match what you
expected.
To run the tests, simply run ``make check`` after previously building the source
with ``make``. The test cases in each directory will be built and then run.
This should not take long. If all the tests succeed, you will be back at the
prompt. Scrolling back through the history, you should see lines like the
following:
.. code-block:: none
PASS: pcmk__strcmp_test 1 - same_pointer
PASS: pcmk__strcmp_test 2 - one_is_null
PASS: pcmk__strcmp_test 3 - case_matters
PASS: pcmk__strcmp_test 4 - case_insensitive
PASS: pcmk__strcmp_test 5 - regex
============================================================================
Testsuite summary for pacemaker 2.1.0
============================================================================
# TOTAL: 33
# PASS: 33
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
make[7]: Leaving directory '/home/clumens/src/pacemaker/lib/common/tests/strings'
The testing process will quit on the first failed test, and you will see lines
like these:
.. code-block:: none
PASS: pcmk__scan_double_test 3 - trailing_chars
FAIL: pcmk__scan_double_test 4 - typical_case
PASS: pcmk__scan_double_test 5 - double_overflow
PASS: pcmk__scan_double_test 6 - double_underflow
ERROR: pcmk__scan_double_test - exited with status 1
PASS: pcmk__starts_with_test 1 - bad_input
============================================================================
Testsuite summary for pacemaker 2.1.0
============================================================================
# TOTAL: 56
# PASS: 54
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 1
============================================================================
See lib/common/tests/strings/test-suite.log
Please report to users@clusterlabs.org
============================================================================
make[7]: *** [Makefile:1218: test-suite.log] Error 1
make[7]: Leaving directory '/home/clumens/src/pacemaker/lib/common/tests/strings'
The failure is in ``lib/common/tests/strings/test-suite.log``:
.. code-block:: none
ERROR: pcmk__scan_double_test
=============================
1..6
ok 1 - empty_input_string
PASS: pcmk__scan_double_test 1 - empty_input_string
ok 2 - bad_input_string
PASS: pcmk__scan_double_test 2 - bad_input_string
ok 3 - trailing_chars
PASS: pcmk__scan_double_test 3 - trailing_chars
not ok 4 - typical_case
FAIL: pcmk__scan_double_test 4 - typical_case
# 0.000000 != 3.000000
# pcmk__scan_double_test.c:80: error: Failure!
ok 5 - double_overflow
PASS: pcmk__scan_double_test 5 - double_overflow
ok 6 - double_underflow
PASS: pcmk__scan_double_test 6 - double_underflow
# not ok - tests
ERROR: pcmk__scan_double_test - exited with status 1
At this point, you need to determine whether your test case is incorrect or
whether the code being tested is incorrect. Fix whichever is wrong and continue.
Fuzz Testing
############
Pacemaker is integrated with the
`OSS-Fuzz `_ project. OSS-Fuzz calls
selected Pacemaker APIs with random argument values to catch edge cases that
might be missed by other forms of testing.
The OSS-Fuzz project has a contact address for Pacemaker in
projects/pacemaker/project.yaml that will receive bug reports. The address must
have been used to commit to Pacemaker, and should be tied to a Google account.
Open reports that aren't security-related can be seen at `OSS-Fuzz testcases
`_.
Fuzzers
_______
Each fuzz-tested library has a fuzzers subdirectory (for example,
``lib/common/fuzzers``). That directory has a file for each fuzzed source file,
named the same except ending in ``_fuzzer.c`` (for example,
``lib/common/fuzzers/strings_fuzzer.c`` has fuzzing for
``lib/common/strings.c``). Those files are not built or distributed as part of
Pacemaker but are used by OSS-Fuzz (see ``projects/pacemaker/build.sh`` in the
OSS-Fuzz repository).
By default, fuzzing uses `libFuzzer `_.
Only Pacemaker APIs that accept any input and do not exit can be fuzzed.
Ideally, fuzzed functions will not modify global state or vary code paths by
anything other than the fuzzed input (such as environment variable values,
date/time, etc.).
Local Fuzzing
_____________
You can use OSS-Fuzz locally to run fuzz testing or reproduce issues reported
by OSS-Fuzz.
To prep a test host:
1. If podman is installed, it will conflict with Docker, so remove it first.
Example for RHEL-like OSes:
* ``dnf remove runc``
1. Install and start Docker. Example for RHEL-like OSes:
* ``dnf config-manager --add-repo
https://download.docker.com/linux/rhel/docker-ce.repo``
* ``dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin
docker-compose-plugin``
* ``usermod -a -G docker $USER``
2. Clone the OSS-Fuzz repository:
* ``cd`` to wherever you want to put it
* ``git clone https://github.com/google/oss-fuzz.git``
* ``cd oss-fuzz``
3. Specify the Pacemaker source you want to test:
* Edit ``projects/pacemaker/Dockerfile`` and replace the last ``git clone``
with the source that you want to test. For example, if you have a branch
``my-fuzzing-branch`` that you've pushed to your GitHub account, you could
use: ``git clone -b my-fuzzing-branch --single-branch --depth 1
https://github.com/$USER/pacemaker``.
To fuzz the code:
1. Ensure Docker is running:
* ``systemctl start docker``
2. Build the necessary Docker containers:
* ``python3 infra/helper.py build_image pacemaker``
3. Build the fuzzers. Choose a sanitizer (for example, ``SANITIZER=address``).
There are three possible sanitizers: address, memory, and undefined. The
memory sanitizer requires special preparation and is generally not used. If
you are reproducing an OSS-Fuzz-reported issue, the issue will list the
sanitizer that was used.
* ``python3 infra/helper.py build_fuzzers --sanitizer $SANITIZER pacemaker``
4. Ensure the build succeeded (use the same sanitizer as the previous step):
* ``python3 infra/helper.py check_build --sanitizer $SANITIZER pacemaker``
5. If you want to run fuzzing yourself, choose a fuzzer (for example,
``FUZZER=iso8601_fuzzer``). Create a temporary directory for the fuzzer's
outputs, then run the fuzzing command, which will fuzz for 25 seconds then
time out:
* ``rm -rf /tmp/corpus >/dev/null 2>&/dev/null``
* ``mkdir /tmp/corpus``
* ``python3 infra/helper.py run_fuzzer --corpus-dir=/tmp/corpus pacemaker
$FUZZER``
* This can be repeated with different fuzzers. The ``build_fuzzers`` step
can also be repeated with a different sanitizer, and the fuzzers tested
again.
6. If you want to reproduce an OSS-Fuzz-reported issue, make a note of the
fuzzer that was used (``$FUZZER`` in this example) and download the provided
reproducer test case file (``$TESTCASE`` in this example), then run:
* ``python3 infra/helper.py reproduce pacemaker $FUZZER $TESTCASE``
For details, see the `OSS-Fuzz documentation
`_.
Code Coverage
#############
Figuring out what needs unit tests written is the purpose of a code coverage tool.
The Pacemaker build process uses ``lcov`` and special make targets to generate
an HTML coverage report that can be inspected with any web browser.
To start, you'll need to install the ``lcov`` package which is included in most
distributions. Next, reconfigure the source tree:
.. code-block:: none
$ ./configure --with-coverage
Then run ``make -C devel coverage``. This will do the same thing as ``make check``,
but will generate a bunch of intermediate files as part of the compiler's output.
Essentially, the coverage tools run all the unit tests and make a note if a given
line if code is executed as a part of some test program. This will include not
just things run as part of the tests but anything in the setup and teardown
functions as well.
Afterwards, the HTML report will be in ``coverage/index.html``. You can drill down
into individual source files to see exactly which lines are covered and which are
not, which makes it easy to target new unit tests. Note that sometimes, it is
impossible to achieve 100% coverage for a source file. For instance, how do you
test a function with a return type of void that simply returns on some condition?
Note that Pacemaker's overall code coverage numbers are very low at the moment.
One reason for this is the large amount of code in the ``daemons`` directory that
will be very difficult to write unit tests for. For now, it is best to focus
efforts on increasing the coverage on individual libraries.
Additionally, there is a ``coverage-cts`` target that does the same thing but
instead of testing ``make check``, it tests ``cts/cts-cli``. The idea behind this
target is to see what parts of our command line tools are covered by our regression
tests. It is probably best to clean and rebuild the source tree when switching
between these various targets.
Debugging
#########
gdb
___
If you use ``gdb`` for debugging, some helper functions are defined in
``devel/gdbhelpers``, which can be given to ``gdb`` using the ``-x`` option.
From within the debugger, you can then invoke the ``pcmk`` command that
will describe the helper functions available.
diff --git a/doc/sphinx/Pacemaker_Explained/collective.rst b/doc/sphinx/Pacemaker_Explained/collective.rst
index f327059984..3665557574 100644
--- a/doc/sphinx/Pacemaker_Explained/collective.rst
+++ b/doc/sphinx/Pacemaker_Explained/collective.rst
@@ -1,1199 +1,1199 @@
.. index:
single: collective resource
single: resource; collective
Collective Resources
--------------------
Pacemaker supports several types of *collective* resources, which consist of
multiple, related resource instances.
.. index:
single: group resource
single: resource; group
.. _group-resources:
Groups - A Syntactic Shortcut
#############################
One of the most common elements of a cluster is a set of resources
that need to be located together, start sequentially, and stop in the
reverse order. To simplify this configuration, we support the concept
of groups.
.. topic:: A group of two primitive resources
.. code-block:: xml
Although the example above contains only two resources, there is no
limit to the number of resources a group can contain. The example is
also sufficient to explain the fundamental properties of a group:
* Resources are started in the order they appear in (**Public-IP** first,
then **Email**)
* Resources are stopped in the reverse order to which they appear in
(**Email** first, then **Public-IP**)
If a resource in the group can't run anywhere, then nothing after that
is allowed to run, too.
* If **Public-IP** can't run anywhere, neither can **Email**;
* but if **Email** can't run anywhere, this does not affect **Public-IP**
in any way
The group above is logically equivalent to writing:
.. topic:: How the cluster sees a group resource
.. code-block:: xml
Obviously as the group grows bigger, the reduced configuration effort
can become significant.
Another (typical) example of a group is a DRBD volume, the filesystem
mount, an IP address, and an application that uses them.
.. index::
pair: XML element; group
Group Properties
________________
.. table:: **Properties of a Group Resource**
:widths: 1 4
+-------------+------------------------------------------------------------------+
| Field | Description |
+=============+==================================================================+
| id | .. index:: |
| | single: group; property, id |
| | single: property; id (group) |
| | single: id; group property |
| | |
| | A unique name for the group |
+-------------+------------------------------------------------------------------+
| description | .. index:: |
| | single: group; attribute, description |
| | single: attribute; description (group) |
| | single: description; group attribute |
| | |
| | An optional description of the group, for the user's own |
| | purposes. |
| | E.g. ``resources needed for website`` |
+-------------+------------------------------------------------------------------+
Group Options
_____________
Groups inherit the ``priority``, ``target-role``, and ``is-managed`` properties
from primitive resources. See :ref:`resource_options` for information about
those properties.
Group Instance Attributes
_________________________
Groups have no instance attributes. However, any that are set for the group
object will be inherited by the group's children.
Group Contents
______________
Groups may only contain a collection of cluster resources (see
:ref:`primitive-resource`). To refer to a child of a group resource, just use
the child's ``id`` instead of the group's.
Group Constraints
_________________
Although it is possible to reference a group's children in
constraints, it is usually preferable to reference the group itself.
.. topic:: Some constraints involving groups
.. code-block:: xml
.. index::
pair: resource-stickiness; group
Group Stickiness
________________
Stickiness, the measure of how much a resource wants to stay where it
is, is additive in groups. Every active resource of the group will
contribute its stickiness value to the group's total. So if the
default ``resource-stickiness`` is 100, and a group has seven members,
five of which are active, then the group as a whole will prefer its
current location with a score of 500.
.. index::
single: clone
single: resource; clone
.. _s-resource-clone:
Clones - Resources That Can Have Multiple Active Instances
##########################################################
*Clone* resources are resources that can have more than one copy active at the
same time. This allows you, for example, to run a copy of a daemon on every
node. You can clone any primitive or group resource [#]_.
Anonymous versus Unique Clones
______________________________
A clone resource is configured to be either *anonymous* or *globally unique*.
Anonymous clones are the simplest. These behave completely identically
everywhere they are running. Because of this, there can be only one instance of
an anonymous clone active per node.
The instances of globally unique clones are distinct entities. All instances
are launched identically, but one instance of the clone is not identical to any
other instance, whether running on the same node or a different node. As an
example, a cloned IP address can use special kernel functionality such that
each instance handles a subset of requests for the same IP address.
.. index::
single: promotable clone
single: resource; promotable
.. _s-resource-promotable:
Promotable clones
_________________
If a clone is *promotable*, its instances can perform a special role that
Pacemaker will manage via the ``promote`` and ``demote`` actions of the resource
agent.
Services that support such a special role have various terms for the special
role and the default role: primary and secondary, master and replica,
controller and worker, etc. Pacemaker uses the terms *promoted* and
*unpromoted* to be agnostic to what the service calls them or what they do.
All that Pacemaker cares about is that an instance comes up in the unpromoted role
when started, and the resource agent supports the ``promote`` and ``demote`` actions
to manage entering and exiting the promoted role.
.. index::
pair: XML element; clone
Clone Properties
________________
.. table:: **Properties of a Clone Resource**
:widths: 1 4
+-------------+------------------------------------------------------------------+
| Field | Description |
+=============+==================================================================+
| id | .. index:: |
| | single: clone; property, id |
| | single: property; id (clone) |
| | single: id; clone property |
| | |
| | A unique name for the clone |
+-------------+------------------------------------------------------------------+
| description | .. index:: |
| | single: clone; attribute, description |
| | single: attribute; description (clone) |
| | single: description; clone attribute |
| | |
| | An optional description of the clone, for the user's own |
| | purposes. |
| | E.g. ``IP address for website`` |
+-------------+------------------------------------------------------------------+
.. index::
pair: options; clone
Clone Options
_____________
:ref:`Options ` inherited from primitive resources:
``priority, target-role, is-managed``
.. table:: **Clone-specific configuration options**
:class: longtable
:widths: 1 1 3
+-------------------+-----------------+-------------------------------------------------------+
| Field | Default | Description |
+===================+=================+=======================================================+
- | globally-unique | true if | .. index:: |
+ | globally-unique | **true** if | .. index:: |
| | clone-node-max | single: clone; option, globally-unique |
| | is greater than | single: option; globally-unique (clone) |
| | 1, otherwise | single: globally-unique; clone option |
- | | false | |
+ | | **false** | |
| | | If **true**, each clone instance performs a |
| | | distinct function, such that a single node can run |
| | | more than one instance at the same time |
+-------------------+-----------------+-------------------------------------------------------+
| clone-max | 0 | .. index:: |
| | | single: clone; option, clone-max |
| | | single: option; clone-max (clone) |
| | | single: clone-max; clone option |
| | | |
| | | The maximum number of clone instances that can |
| | | be started across the entire cluster. If 0, the |
| | | number of nodes in the cluster will be used. |
+-------------------+-----------------+-------------------------------------------------------+
| clone-node-max | 1 | .. index:: |
| | | single: clone; option, clone-node-max |
| | | single: option; clone-node-max (clone) |
| | | single: clone-node-max; clone option |
| | | |
| | | If the clone is globally unique, this is the maximum |
| | | number of clone instances that can be started |
| | | on a single node |
+-------------------+-----------------+-------------------------------------------------------+
| clone-min | 0 | .. index:: |
| | | single: clone; option, clone-min |
| | | single: option; clone-min (clone) |
| | | single: clone-min; clone option |
| | | |
| | | Require at least this number of clone instances |
| | | to be runnable before allowing resources |
| | | depending on the clone to be runnable. A value |
| | | of 0 means require all clone instances to be |
| | | runnable. |
+-------------------+-----------------+-------------------------------------------------------+
| notify | false | .. index:: |
| | | single: clone; option, notify |
| | | single: option; notify (clone) |
| | | single: notify; clone option |
| | | |
| | | Call the resource agent's **notify** action for |
| | | all active instances, before and after starting |
| | | or stopping any clone instance. The resource |
| | | agent must support this action. |
| | | Allowed values: **false**, **true** |
+-------------------+-----------------+-------------------------------------------------------+
| ordered | false | .. index:: |
| | | single: clone; option, ordered |
| | | single: option; ordered (clone) |
| | | single: ordered; clone option |
| | | |
| | | If **true**, clone instances must be started |
| | | sequentially instead of in parallel. |
| | | Allowed values: **false**, **true** |
+-------------------+-----------------+-------------------------------------------------------+
| interleave | false | .. index:: |
| | | single: clone; option, interleave |
| | | single: option; interleave (clone) |
| | | single: interleave; clone option |
| | | |
| | | When this clone is ordered relative to another |
| | | clone, if this option is **false** (the default), |
| | | the ordering is relative to *all* instances of |
| | | the other clone, whereas if this option is |
| | | **true**, the ordering is relative only to |
| | | instances on the same node. |
| | | Allowed values: **false**, **true** |
+-------------------+-----------------+-------------------------------------------------------+
| promotable | false | .. index:: |
| | | single: clone; option, promotable |
| | | single: option; promotable (clone) |
| | | single: promotable; clone option |
| | | |
| | | If **true**, clone instances can perform a |
| | | special role that Pacemaker will manage via the |
| | | resource agent's **promote** and **demote** |
| | | actions. The resource agent must support these |
| | | actions. |
| | | Allowed values: **false**, **true** |
+-------------------+-----------------+-------------------------------------------------------+
| promoted-max | 1 | .. index:: |
| | | single: clone; option, promoted-max |
| | | single: option; promoted-max (clone) |
| | | single: promoted-max; clone option |
| | | |
| | | If ``promotable`` is **true**, the number of |
| | | instances that can be promoted at one time |
| | | across the entire cluster |
+-------------------+-----------------+-------------------------------------------------------+
| promoted-node-max | 1 | .. index:: |
| | | single: clone; option, promoted-node-max |
| | | single: option; promoted-node-max (clone) |
| | | single: promoted-node-max; clone option |
| | | |
| | | If the clone is promotable and globally unique, this |
| | | is the number of instances that can be promoted at |
- | | | one time on a single node (up to `clone-node-max`) |
+ | | | one time on a single node (up to ``clone-node-max``) |
+-------------------+-----------------+-------------------------------------------------------+
.. note:: **Deprecated Terminology**
In older documentation and online examples, you may see promotable clones
referred to as *multi-state*, *stateful*, or *master/slave*; these mean the
same thing as *promotable*. Certain syntax is supported for backward
compatibility, but is deprecated and will be removed in a future version:
* Using the ``master-max`` meta-attribute instead of ``promoted-max``
* Using the ``master-node-max`` meta-attribute instead of
``promoted-node-max``
* Using ``Master`` as a role name instead of ``Promoted``
* Using ``Slave`` as a role name instead of ``Unpromoted``
Clone Contents
______________
Clones must contain exactly one primitive or group resource.
.. topic:: A clone that runs a web server on all nodes
.. code-block:: xml
.. warning::
You should never reference the name of a clone's child (the primitive or group
resource being cloned). If you think you need to do this, you probably need to
re-evaluate your design.
Clone Instance Attribute
________________________
Clones have no instance attributes; however, any that are set here will be
inherited by the clone's child.
.. index::
single: clone; constraint
Clone Constraints
_________________
In most cases, a clone will have a single instance on each active cluster
node. If this is not the case, you can indicate which nodes the
cluster should preferentially assign copies to with resource location
constraints. These constraints are written no differently from those
for primitive resources except that the clone's **id** is used.
.. topic:: Some constraints involving clones
.. code-block:: xml
Ordering constraints behave slightly differently for clones. In the
example above, ``apache-stats`` will wait until all copies of ``apache-clone``
that need to be started have done so before being started itself.
Only if *no* copies can be started will ``apache-stats`` be prevented
from being active. Additionally, the clone will wait for
``apache-stats`` to be stopped before stopping itself.
Colocation of a primitive or group resource with a clone means that
the resource can run on any node with an active instance of the clone.
The cluster will choose an instance based on where the clone is running and
the resource's own location preferences.
Colocation between clones is also possible. If one clone **A** is colocated
with another clone **B**, the set of allowed locations for **A** is limited to
nodes on which **B** is (or will be) active. Placement is then performed
normally.
.. index::
single: promotable clone; constraint
.. _promotable-clone-constraints:
Promotable Clone Constraints
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For promotable clone resources, the ``first-action`` and/or ``then-action`` fields
for ordering constraints may be set to ``promote`` or ``demote`` to constrain the
promoted role, and colocation constraints may contain ``rsc-role`` and/or
``with-rsc-role`` fields.
.. topic:: Constraints involving promotable clone resources
.. code-block:: xml
In the example above, **myApp** will wait until one of the database
copies has been started and promoted before being started
itself on the same node. Only if no copies can be promoted will **myApp** be
prevented from being active. Additionally, the cluster will wait for
**myApp** to be stopped before demoting the database.
Colocation of a primitive or group resource with a promotable clone
resource means that it can run on any node with an active instance of
the promotable clone resource that has the specified role (``Promoted`` or
``Unpromoted``). In the example above, the cluster will choose a location
based on where database is running in the promoted role, and if there are
multiple promoted instances it will also factor in **myApp**'s own location
preferences when deciding which location to choose.
Colocation with regular clones and other promotable clone resources is also
possible. In such cases, the set of allowed locations for the **rsc**
clone is (after role filtering) limited to nodes on which the
``with-rsc`` promotable clone resource is (or will be) in the specified role.
Placement is then performed as normal.
Using Promotable Clone Resources in Colocation Sets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When a promotable clone is used in a :ref:`resource set `
inside a colocation constraint, the resource set may take a ``role`` attribute.
In the following example, an instance of **B** may be promoted only on a node
where **A** is in the promoted role. Additionally, resources **C** and **D**
must be located on a node where both **A** and **B** are promoted.
.. topic:: Colocate C and D with A's and B's promoted instances
.. code-block:: xml
Using Promotable Clone Resources in Ordered Sets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When a promotable clone is used in a :ref:`resource set `
inside an ordering constraint, the resource set may take an ``action``
attribute.
.. topic:: Start C and D after first promoting A and B
.. code-block:: xml
In the above example, **B** cannot be promoted until **A** has been promoted.
Additionally, resources **C** and **D** must wait until **A** and **B** have
been promoted before they can start.
.. index::
pair: resource-stickiness; clone
.. _s-clone-stickiness:
Clone Stickiness
________________
To achieve stable assignments, clones are slightly sticky by default. If no
value for ``resource-stickiness`` is provided, the clone will use a value of 1.
Being a small value, it causes minimal disturbance to the score calculations of
other resources but is enough to prevent Pacemaker from needlessly moving
instances around the cluster.
.. note::
For globally unique clones, this may result in multiple instances of the
clone staying on a single node, even after another eligible node becomes
active (for example, after being put into standby mode then made active again).
If you do not want this behavior, specify a ``resource-stickiness`` of 0
for the clone temporarily and let the cluster adjust, then set it back
to 1 if you want the default behavior to apply again.
.. important::
If ``resource-stickiness`` is set in the ``rsc_defaults`` section, it will
apply to clone instances as well. This means an explicit ``resource-stickiness``
of 0 in ``rsc_defaults`` works differently from the implicit default used when
``resource-stickiness`` is not specified.
Monitoring Promotable Clone Resources
_____________________________________
The usual monitor actions are insufficient to monitor a promotable clone
resource, because Pacemaker needs to verify not only that the resource is
active, but also that its actual role matches its intended one.
Define two monitoring actions: the usual one will cover the unpromoted role,
and an additional one with ``role="Promoted"`` will cover the promoted role.
.. topic:: Monitoring both states of a promotable clone resource
.. code-block:: xml
.. important::
It is crucial that *every* monitor operation has a different interval!
Pacemaker currently differentiates between operations
only by resource and interval; so if (for example) a promotable clone resource
had the same monitor interval for both roles, Pacemaker would ignore the
role when checking the status -- which would cause unexpected return
codes, and therefore unnecessary complications.
.. _s-promotion-scores:
Determining Which Instance is Promoted
______________________________________
Pacemaker can choose a promotable clone instance to be promoted in one of two
ways:
* Promotion scores: These are node attributes set via the ``crm_attribute``
command using the ``--promotion`` option, which generally would be called by
the resource agent's start action if it supports promotable clones. This tool
automatically detects both the resource and host, and should be used to set a
preference for being promoted. Based on this, ``promoted-max``, and
``promoted-node-max``, the instance(s) with the highest preference will be
promoted.
* Constraints: Location constraints can indicate which nodes are most preferred
to be promoted.
.. topic:: Explicitly preferring node1 to be promoted
.. code-block:: xml
.. index:
single: bundle
single: resource; bundle
pair: container; Docker
pair: container; podman
.. _s-resource-bundle:
Bundles - Containerized Resources
#################################
Pacemaker supports a special syntax for launching a service inside a
`container `_
with any infrastructure it requires: the *bundle*.
Pacemaker bundles support `Docker `_ and
`podman `_ *(since 2.0.1)* container technologies. [#]_
.. topic:: A bundle for a containerized web server
.. code-block:: xml
Bundle Prerequisites
____________________
Before configuring a bundle in Pacemaker, the user must install the appropriate
container launch technology (Docker or podman), and supply a fully configured
container image, on every node allowed to run the bundle.
Pacemaker will create an implicit resource of type **ocf:heartbeat:docker** or
**ocf:heartbeat:podman** to manage a bundle's container. The user must ensure
that the appropriate resource agent is installed on every node allowed to run
the bundle.
.. index::
pair: XML element; bundle
Bundle Properties
_________________
.. table:: **XML Attributes of a bundle Element**
:widths: 1 4
+-------------+------------------------------------------------------------------+
| Field | Description |
+=============+==================================================================+
| id | .. index:: |
| | single: bundle; attribute, id |
| | single: attribute; id (bundle) |
| | single: id; bundle attribute |
| | |
| | A unique name for the bundle (required) |
+-------------+------------------------------------------------------------------+
| description | .. index:: |
| | single: bundle; attribute, description |
| | single: attribute; description (bundle) |
| | single: description; bundle attribute |
| | |
| | An optional description of the group, for the user's own |
| | purposes. |
| | E.g. ``manages the container that runs the service`` |
+-------------+------------------------------------------------------------------+
A bundle must contain exactly one ``docker`` or ``podman`` element.
.. index::
pair: XML element; docker
pair: XML element; podman
Bundle Container Properties
___________________________
.. table:: **XML attributes of a docker or podman Element**
:class: longtable
:widths: 2 3 4
+-------------------+------------------------------------+---------------------------------------------------+
| Attribute | Default | Description |
+===================+====================================+===================================================+
| image | | .. index:: |
| | | single: docker; attribute, image |
| | | single: attribute; image (docker) |
| | | single: image; docker attribute |
| | | single: podman; attribute, image |
| | | single: attribute; image (podman) |
| | | single: image; podman attribute |
| | | |
| | | Container image tag (required) |
+-------------------+------------------------------------+---------------------------------------------------+
| replicas | Value of ``promoted-max`` | .. index:: |
| | if that is positive, else 1 | single: docker; attribute, replicas |
| | | single: attribute; replicas (docker) |
| | | single: replicas; docker attribute |
| | | single: podman; attribute, replicas |
| | | single: attribute; replicas (podman) |
| | | single: replicas; podman attribute |
| | | |
| | | A positive integer specifying the number of |
| | | container instances to launch |
+-------------------+------------------------------------+---------------------------------------------------+
| replicas-per-host | 1 | .. index:: |
| | | single: docker; attribute, replicas-per-host |
| | | single: attribute; replicas-per-host (docker) |
| | | single: replicas-per-host; docker attribute |
| | | single: podman; attribute, replicas-per-host |
| | | single: attribute; replicas-per-host (podman) |
| | | single: replicas-per-host; podman attribute |
| | | |
| | | A positive integer specifying the number of |
| | | container instances allowed to run on a |
| | | single node |
+-------------------+------------------------------------+---------------------------------------------------+
| promoted-max | 0 | .. index:: |
| | | single: docker; attribute, promoted-max |
| | | single: attribute; promoted-max (docker) |
| | | single: promoted-max; docker attribute |
| | | single: podman; attribute, promoted-max |
| | | single: attribute; promoted-max (podman) |
| | | single: promoted-max; podman attribute |
| | | |
| | | A non-negative integer that, if positive, |
| | | indicates that the containerized service |
| | | should be treated as a promotable service, |
| | | with this many replicas allowed to run the |
| | | service in the promoted role |
+-------------------+------------------------------------+---------------------------------------------------+
| network | | .. index:: |
| | | single: docker; attribute, network |
| | | single: attribute; network (docker) |
| | | single: network; docker attribute |
| | | single: podman; attribute, network |
| | | single: attribute; network (podman) |
| | | single: network; podman attribute |
| | | |
| | | If specified, this will be passed to the |
| | | ``docker run`` or ``podman run`` command as the |
| | | network setting for the container. |
+-------------------+------------------------------------+---------------------------------------------------+
| run-command | ``/usr/sbin/pacemaker-remoted`` if | .. index:: |
| | bundle contains a **primitive**, | single: docker; attribute, run-command |
| | otherwise none | single: attribute; run-command (docker) |
| | | single: run-command; docker attribute |
| | | single: podman; attribute, run-command |
| | | single: attribute; run-command (podman) |
| | | single: run-command; podman attribute |
| | | |
| | | This command will be run inside the container |
| | | when launching it ("PID 1"). If the bundle |
| | | contains a **primitive**, this command *must* |
| | | start ``pacemaker-remoted`` (but could, for |
| | | example, be a script that does other stuff, too). |
+-------------------+------------------------------------+---------------------------------------------------+
| options | | .. index:: |
| | | single: docker; attribute, options |
| | | single: attribute; options (docker) |
| | | single: options; docker attribute |
| | | single: podman; attribute, options |
| | | single: attribute; options (podman) |
| | | single: options; podman attribute |
| | | |
| | | Extra command-line options to pass to the |
| | | ``docker run`` or ``podman run`` command |
+-------------------+------------------------------------+---------------------------------------------------+
.. note::
Considerations when using cluster configurations or container images from
Pacemaker 1.1:
* If the container image has a pre-2.0.0 version of Pacemaker, set ``run-command``
to ``/usr/sbin/pacemaker_remoted`` (note the underbar instead of dash).
* ``masters`` is accepted as an alias for ``promoted-max``, but is deprecated since
2.0.0, and support for it will be removed in a future version.
Bundle Network Properties
_________________________
A bundle may optionally contain one ```` element.
.. index::
pair: XML element; network
single: bundle; network
.. table:: **XML attributes of a network Element**
:widths: 2 1 5
+----------------+---------+------------------------------------------------------------+
| Attribute | Default | Description |
+================+=========+============================================================+
| add-host | TRUE | .. index:: |
| | | single: network; attribute, add-host |
| | | single: attribute; add-host (network) |
| | | single: add-host; network attribute |
| | | |
| | | If TRUE, and ``ip-range-start`` is used, Pacemaker will |
| | | automatically ensure that ``/etc/hosts`` inside the |
| | | containers has entries for each |
| | | :ref:`replica name ` |
| | | and its assigned IP. |
+----------------+---------+------------------------------------------------------------+
| ip-range-start | | .. index:: |
| | | single: network; attribute, ip-range-start |
| | | single: attribute; ip-range-start (network) |
| | | single: ip-range-start; network attribute |
| | | |
| | | If specified, Pacemaker will create an implicit |
| | | ``ocf:heartbeat:IPaddr2`` resource for each container |
| | | instance, starting with this IP address, using up to |
| | | ``replicas`` sequential addresses. These addresses can be |
| | | used from the host's network to reach the service inside |
| | | the container, though it is not visible within the |
| | | container itself. Only IPv4 addresses are currently |
| | | supported. |
+----------------+---------+------------------------------------------------------------+
| host-netmask | 32 | .. index:: |
| | | single: network; attribute; host-netmask |
| | | single: attribute; host-netmask (network) |
| | | single: host-netmask; network attribute |
| | | |
| | | If ``ip-range-start`` is specified, the IP addresses |
| | | are created with this CIDR netmask (as a number of bits). |
+----------------+---------+------------------------------------------------------------+
| host-interface | | .. index:: |
| | | single: network; attribute; host-interface |
| | | single: attribute; host-interface (network) |
| | | single: host-interface; network attribute |
| | | |
| | | If ``ip-range-start`` is specified, the IP addresses are |
| | | created on this host interface (by default, it will be |
| | | determined from the IP address). |
+----------------+---------+------------------------------------------------------------+
| control-port | 3121 | .. index:: |
| | | single: network; attribute; control-port |
| | | single: attribute; control-port (network) |
| | | single: control-port; network attribute |
| | | |
| | | If the bundle contains a ``primitive``, the cluster will |
| | | use this integer TCP port for communication with |
| | | Pacemaker Remote inside the container. Changing this is |
| | | useful when the container is unable to listen on the |
| | | default port, for example, when the container uses the |
| | | host's network rather than ``ip-range-start`` (in which |
| | | case ``replicas-per-host`` must be 1), or when the bundle |
| | | may run on a Pacemaker Remote node that is already |
| | | listening on the default port. Any ``PCMK_remote_port`` |
| | | environment variable set on the host or in the container |
| | | is ignored for bundle connections. |
+----------------+---------+------------------------------------------------------------+
.. _s-resource-bundle-note-replica-names:
.. note::
Replicas are named by the bundle id plus a dash and an integer counter starting
with zero. For example, if a bundle named **httpd-bundle** has **replicas=2**, its
containers will be named **httpd-bundle-0** and **httpd-bundle-1**.
.. index::
pair: XML element; port-mapping
Additionally, a ``network`` element may optionally contain one or more
``port-mapping`` elements.
.. table:: **Attributes of a port-mapping Element**
:widths: 2 1 5
+---------------+-------------------+------------------------------------------------------+
| Attribute | Default | Description |
+===============+===================+======================================================+
| id | | .. index:: |
| | | single: port-mapping; attribute, id |
| | | single: attribute; id (port-mapping) |
| | | single: id; port-mapping attribute |
| | | |
| | | A unique name for the port mapping (required) |
+---------------+-------------------+------------------------------------------------------+
| port | | .. index:: |
| | | single: port-mapping; attribute, port |
| | | single: attribute; port (port-mapping) |
| | | single: port; port-mapping attribute |
| | | |
| | | If this is specified, connections to this TCP port |
| | | number on the host network (on the container's |
| | | assigned IP address, if ``ip-range-start`` is |
| | | specified) will be forwarded to the container |
| | | network. Exactly one of ``port`` or ``range`` |
| | | must be specified in a ``port-mapping``. |
+---------------+-------------------+------------------------------------------------------+
| internal-port | value of ``port`` | .. index:: |
| | | single: port-mapping; attribute, internal-port |
| | | single: attribute; internal-port (port-mapping) |
| | | single: internal-port; port-mapping attribute |
| | | |
| | | If ``port`` and this are specified, connections |
| | | to ``port`` on the host's network will be |
| | | forwarded to this port on the container network. |
+---------------+-------------------+------------------------------------------------------+
| range | | .. index:: |
| | | single: port-mapping; attribute, range |
| | | single: attribute; range (port-mapping) |
| | | single: range; port-mapping attribute |
| | | |
| | | If this is specified, connections to these TCP |
| | | port numbers (expressed as *first_port*-*last_port*) |
| | | on the host network (on the container's assigned IP |
| | | address, if ``ip-range-start`` is specified) will |
| | | be forwarded to the same ports in the container |
| | | network. Exactly one of ``port`` or ``range`` |
| | | must be specified in a ``port-mapping``. |
+---------------+-------------------+------------------------------------------------------+
.. note::
If the bundle contains a ``primitive``, Pacemaker will automatically map the
``control-port``, so it is not necessary to specify that port in a
``port-mapping``.
.. index:
pair: XML element; storage
pair: XML element; storage-mapping
single: bundle; storage
.. _s-bundle-storage:
Bundle Storage Properties
_________________________
A bundle may optionally contain one ``storage`` element. A ``storage`` element
has no properties of its own, but may contain one or more ``storage-mapping``
elements.
.. table:: **Attributes of a storage-mapping Element**
:widths: 2 1 5
+-----------------+---------+-------------------------------------------------------------+
| Attribute | Default | Description |
+=================+=========+=============================================================+
| id | | .. index:: |
| | | single: storage-mapping; attribute, id |
| | | single: attribute; id (storage-mapping) |
| | | single: id; storage-mapping attribute |
| | | |
| | | A unique name for the storage mapping (required) |
+-----------------+---------+-------------------------------------------------------------+
| source-dir | | .. index:: |
| | | single: storage-mapping; attribute, source-dir |
| | | single: attribute; source-dir (storage-mapping) |
| | | single: source-dir; storage-mapping attribute |
| | | |
| | | The absolute path on the host's filesystem that will be |
| | | mapped into the container. Exactly one of ``source-dir`` |
| | | and ``source-dir-root`` must be specified in a |
| | | ``storage-mapping``. |
+-----------------+---------+-------------------------------------------------------------+
| source-dir-root | | .. index:: |
| | | single: storage-mapping; attribute, source-dir-root |
| | | single: attribute; source-dir-root (storage-mapping) |
| | | single: source-dir-root; storage-mapping attribute |
| | | |
| | | The start of a path on the host's filesystem that will |
| | | be mapped into the container, using a different |
| | | subdirectory on the host for each container instance. |
| | | The subdirectory will be named the same as the |
| | | :ref:`replica name `. |
| | | Exactly one of ``source-dir`` and ``source-dir-root`` |
| | | must be specified in a ``storage-mapping``. |
+-----------------+---------+-------------------------------------------------------------+
| target-dir | | .. index:: |
| | | single: storage-mapping; attribute, target-dir |
| | | single: attribute; target-dir (storage-mapping) |
| | | single: target-dir; storage-mapping attribute |
| | | |
| | | The path name within the container where the host |
| | | storage will be mapped (required) |
+-----------------+---------+-------------------------------------------------------------+
| options | | .. index:: |
| | | single: storage-mapping; attribute, options |
| | | single: attribute; options (storage-mapping) |
| | | single: options; storage-mapping attribute |
| | | |
| | | A comma-separated list of file system mount |
| | | options to use when mapping the storage |
+-----------------+---------+-------------------------------------------------------------+
.. note::
Pacemaker does not define the behavior if the source directory does not already
exist on the host. However, it is expected that the container technology and/or
its resource agent will create the source directory in that case.
.. note::
If the bundle contains a ``primitive``,
Pacemaker will automatically map the equivalent of
``source-dir=/etc/pacemaker/authkey target-dir=/etc/pacemaker/authkey``
and ``source-dir-root=/var/log/pacemaker/bundles target-dir=/var/log`` into the
container, so it is not necessary to specify those paths in a
``storage-mapping``.
.. important::
The ``PCMK_authkey_location`` environment variable must not be set to anything
other than the default of ``/etc/pacemaker/authkey`` on any node in the cluster.
.. important::
If SELinux is used in enforcing mode on the host, you must ensure the container
is allowed to use any storage you mount into it. For Docker and podman bundles,
adding "Z" to the mount options will create a container-specific label for the
mount that allows the container access.
.. index::
single: bundle; primitive
Bundle Primitive
________________
A bundle may optionally contain one :ref:`primitive `
resource. The primitive may have operations, instance attributes, and
meta-attributes defined, as usual.
If a bundle contains a primitive resource, the container image must include
the Pacemaker Remote daemon, and at least one of ``ip-range-start`` or
``control-port`` must be configured in the bundle. Pacemaker will create an
implicit **ocf:pacemaker:remote** resource for the connection, launch
Pacemaker Remote within the container, and monitor and manage the primitive
resource via Pacemaker Remote.
If the bundle has more than one container instance (replica), the primitive
resource will function as an implicit :ref:`clone ` -- a
:ref:`promotable clone ` if the bundle has ``promoted-max``
greater than zero.
.. note::
If you want to pass environment variables to a bundle's Pacemaker Remote
connection or primitive, you have two options:
* Environment variables whose value is the same regardless of the underlying host
may be set using the container element's ``options`` attribute.
* If you want variables to have host-specific values, you can use the
:ref:`storage-mapping ` element to map a file on the host as
``/etc/pacemaker/pcmk-init.env`` in the container *(since 2.0.3)*.
Pacemaker Remote will parse this file as a shell-like format, with
variables set as NAME=VALUE, ignoring blank lines and comments starting
with "#".
.. important::
When a bundle has a ``primitive``, Pacemaker on all cluster nodes must be able to
contact Pacemaker Remote inside the bundle's containers.
* The containers must have an accessible network (for example, ``network`` should
not be set to "none" with a ``primitive``).
* The default, using a distinct network space inside the container, works in
combination with ``ip-range-start``. Any firewall must allow access from all
cluster nodes to the ``control-port`` on the container IPs.
* If the container shares the host's network space (for example, by setting
``network`` to "host"), a unique ``control-port`` should be specified for each
bundle. Any firewall must allow access from all cluster nodes to the
``control-port`` on all cluster and remote node IPs.
.. index::
single: bundle; node attributes
.. _s-bundle-attributes:
Bundle Node Attributes
______________________
If the bundle has a ``primitive``, the primitive's resource agent may want to set
node attributes such as :ref:`promotion scores `. However, with
containers, it is not apparent which node should get the attribute.
If the container uses shared storage that is the same no matter which node the
container is hosted on, then it is appropriate to use the promotion score on the
bundle node itself.
On the other hand, if the container uses storage exported from the underlying host,
then it may be more appropriate to use the promotion score on the underlying host.
Since this depends on the particular situation, the
``container-attribute-target`` resource meta-attribute allows the user to specify
which approach to use. If it is set to ``host``, then user-defined node attributes
will be checked on the underlying host. If it is anything else, the local node
(in this case the bundle node) is used as usual.
This only applies to user-defined attributes; the cluster will always check the
local node for cluster-defined attributes such as ``#uname``.
If ``container-attribute-target`` is ``host``, the cluster will pass additional
environment variables to the primitive's resource agent that allow it to set
node attributes appropriately: ``CRM_meta_container_attribute_target`` (identical
to the meta-attribute value) and ``CRM_meta_physical_host`` (the name of the
underlying host).
.. note::
When called by a resource agent, the ``attrd_updater`` and ``crm_attribute``
commands will automatically check those environment variables and set
attributes appropriately.
.. index::
single: bundle; meta-attributes
Bundle Meta-Attributes
______________________
Any meta-attribute set on a bundle will be inherited by the bundle's
primitive and any resources implicitly created by Pacemaker for the bundle.
This includes options such as ``priority``, ``target-role``, and ``is-managed``. See
:ref:`resource_options` for more information.
Bundles support clone meta-attributes including ``notify``, ``ordered``, and
``interleave``.
Limitations of Bundles
______________________
Restarting pacemaker while a bundle is unmanaged or the cluster is in
maintenance mode may cause the bundle to fail.
Bundles may not be explicitly cloned or included in groups. This includes the
bundle's primitive and any resources implicitly created by Pacemaker for the
bundle. (If ``replicas`` is greater than 1, the bundle will behave like a clone
implicitly.)
Bundles do not have instance attributes, utilization attributes, or operations,
though a bundle's primitive may have them.
A bundle with a primitive can run on a Pacemaker Remote node only if the bundle
uses a distinct ``control-port``.
.. [#] Of course, the service must support running multiple instances.
.. [#] Docker is a trademark of Docker, Inc. No endorsement by or association with
Docker, Inc. is implied.
diff --git a/doc/sphinx/Pacemaker_Explained/local-options.rst b/doc/sphinx/Pacemaker_Explained/local-options.rst
index 7731f6c918..fc51491f67 100644
--- a/doc/sphinx/Pacemaker_Explained/local-options.rst
+++ b/doc/sphinx/Pacemaker_Explained/local-options.rst
@@ -1,724 +1,723 @@
Host-Local Configuration
------------------------
.. index::
pair: XML element; configuration
.. note:: Directory and file paths below may differ on your system depending on
your Pacemaker build settings. Check your Pacemaker configuration
file to find the correct paths.
Configuration Value Types
#########################
Throughout this document, configuration values will be designated as having one
of the following types:
.. list-table:: **Configuration Value Types**
:class: longtable
:widths: 1 3
:header-rows: 1
* - Type
- Description
* - .. _boolean:
.. index::
pair: type; boolean
boolean
- Case-insensitive text value where ``1``, ``yes``, ``y``, ``on``,
and ``true`` evaluate as true and ``0``, ``no``, ``n``, ``off``,
``false``, and unset evaluate as false
* - .. _date_time:
.. index::
pair: type; date/time
date/time
- Textual timestamp like ``Sat Dec 21 11:47:45 2013``
* - .. _duration:
.. index::
pair: type; duration
duration
- A nonnegative time duration, specified either like a
:ref:`timeout ` or an
`ISO 8601 duration `_.
A duration may be up to approximately 49 days but is intended for much
smaller time periods.
* - .. _enumeration:
.. index::
pair: type; enumeration
enumeration
- Text that must be one of a set of defined values (which will be listed
in the description)
* - .. _epoch_time:
.. index::
pair: type; epoch_time
epoch_time
- Time as the integer number of seconds since the Unix epoch,
``1970-01-01 00:00:00 +0000 (UTC)``.
* - .. _id:
.. index::
pair: type; id
id
- A text string starting with a letter or underbar, followed by any
combination of letters, numbers, dashes, dots, and/or underbars; when
used for a property named ``id``, the string must be unique across all
``id`` properties in the CIB
* - .. _integer:
.. index::
pair: type; integer
integer
- 32-bit signed integer value (-2,147,483,648 to 2,147,483,647)
* - .. _iso8601:
.. index::
pair: type; iso8601
ISO 8601
- An `ISO 8601 `_ date/time.
* - .. _nonnegative_integer:
.. index::
pair: type; nonnegative integer
nonnegative integer
- 32-bit nonnegative integer value (0 to 2,147,483,647)
* - .. _percentage:
.. index::
pair: type; percentage
percentage
- Floating-point number followed by an optional percent sign ('%')
* - .. _port:
.. index::
pair: type; port
port
- Integer TCP port number (0 to 65535)
* - .. _range:
.. index::
pair: type; range
range
- A range may be a single nonnegative integer or a dash-separated range of
nonnegative integers. Either the first or last value may be omitted to
leave the range open-ended. Examples: ``0``, ``3-``, ``-5``, ``4-6``.
* - .. _score:
.. index::
pair: type; score
score
- A Pacemaker score can be an integer between -1,000,000 and 1,000,000, or
a string alias: ``INFINITY`` or ``+INFINITY`` is equivalent to
1,000,000, ``-INFINITY`` is equivalent to -1,000,000, and ``red``,
``yellow``, and ``green`` are equivalent to integers as described in
:ref:`node-health`.
* - .. _text:
.. index::
pair: type; text
text
- A text string
* - .. _timeout:
.. index::
pair: type; timeout
timeout
- A time duration, specified as a bare number (in which case it is
considered to be in seconds) or a number with a unit (``ms`` or ``msec``
for milliseconds, ``us`` or ``usec`` for microseconds, ``s`` or ``sec``
for seconds, ``m`` or ``min`` for minutes, ``h`` or ``hr`` for hours)
optionally with whitespace before and/or after the number.
* - .. _version:
.. index::
pair: type; version
version
- Version number (any combination of alphanumeric characters, dots, and
dashes, starting with a number).
Scores
______
Scores are integral to how Pacemaker works. Practically everything from moving
a resource to deciding which resource to stop in a degraded cluster is achieved
by manipulating scores in some way.
Scores are calculated per resource and node. Any node with a negative score for
a resource can't run that resource. The cluster places a resource on the node
with the highest score for it.
Score addition and subtraction follow these rules:
* Any value (including ``INFINITY``) - ``INFINITY`` = ``-INFINITY``
* ``INFINITY`` + any value other than ``-INFINITY`` = ``INFINITY``
.. note::
What if you want to use a score higher than 1,000,000? Typically this possibility
arises when someone wants to base the score on some external metric that might
go above 1,000,000.
The short answer is you can't.
The long answer is it is sometimes possible work around this limitation
creatively. You may be able to set the score to some computed value based on
the external metric rather than use the metric directly. For nodes, you can
store the metric as a node attribute, and query the attribute when computing
the score (possibly as part of a custom resource agent).
Local Options
#############
Most Pacemaker configuration is in the cluster-wide CIB, but some host-local
-configuration options either are needed at startup, before the CIB is read, or
+configuration options either are needed at startup (before the CIB is read) or
provide per-host overrides of cluster-wide options.
These options are configured as environment variables set when Pacemaker is
started, in the format ``=""``. These are typically set in a file
whose location varies by OS (most commonly ``/etc/sysconfig/pacemaker`` or
``/etc/default/pacemaker``; this documentation was generated on a system using
|PCMK_CONFIG_FILE|).
.. list-table:: **Local Options**
:class: longtable
:widths: 2 2 2 5
:header-rows: 1
* - Name
- Type
- Default
- Description
* - .. _cib_pam_service:
.. index::
pair: node option; CIB_pam_service
CIB_pam_service
- :ref:`text `
- login
- PAM service to use for remote CIB client authentication (passed to
``pam_start``).
* - .. _pcmk_logfacility:
.. index::
pair: node option; PCMK_logfacility
PCMK_logfacility
- :ref:`enumeration `
- daemon
- Enable logging via the system log or journal, using the specified log
facility. Messages sent here are of value to all Pacemaker
administrators. This can be disabled using ``none``, but that is not
recommended. Allowed values:
* ``none``
* ``daemon``
* ``user``
* ``local0``
* ``local1``
* ``local2``
* ``local3``
* ``local4``
* ``local5``
* ``local6``
* ``local7``
* - .. _pcmk_logpriority:
.. index::
pair: node option; PCMK_logpriority
PCMK_logpriority
- :ref:`enumeration `
- notice
- Unless system logging is disabled using ``PCMK_logfacility=none``,
messages of the specified log severity and higher will be sent to the
system log. The default is appropriate for most installations. Allowed
values:
* ``emerg``
* ``alert``
* ``crit``
* ``error``
* ``warning``
* ``notice``
* ``info``
* ``debug``
* - .. _pcmk_logfile:
.. index::
pair: node option; PCMK_logfile
PCMK_logfile
- :ref:`text `
- |PCMK_LOG_FILE|
- Unless set to ``none``, more detailed log messages will be sent to the
specified file (in addition to the system log, if enabled). These
messages may have extended information, and will include messages of info
severity. This log is of more use to developers and advanced system
administrators, and when reporting problems. Note: The default is
|PCMK_CONTAINER_LOG_FILE| (inside the container) for bundled container
nodes; this would typically be mapped to a different path on the host
running the container.
* - .. _pcmk_logfile_mode:
.. index::
pair: node option; PCMK_logfile_mode
PCMK_logfile_mode
- :ref:`text `
- 0660
- Pacemaker will set the permissions on the detail log to this value (see
``chmod(1)``).
* - .. _pcmk_debug:
.. index::
pair: node option; PCMK_debug
PCMK_debug
- :ref:`enumeration `
- no
- Whether to send debug severity messages to the detail log. This may be
set for all subsystems (``yes`` or ``no``) or for specific (comma-
separated) subsystems. Allowed subsystems are:
* ``pacemakerd``
* ``pacemaker-attrd``
* ``pacemaker-based``
* ``pacemaker-controld``
* ``pacemaker-execd``
* ``pacemaker-fenced``
* ``pacemaker-schedulerd``
Example: ``PCMK_debug="pacemakerd,pacemaker-execd"``
* - .. _pcmk_stderr:
.. index::
pair: node option; PCMK_stderr
PCMK_stderr
- :ref:`boolean `
- no
- *Advanced Use Only:* Whether to send daemon log messages to stderr. This
would be useful only during troubleshooting, when starting Pacemaker
manually on the command line.
Setting this option in the configuration file is pointless, since the
file is not read when starting Pacemaker manually. However, it can be set
directly as an environment variable on the command line.
* - .. _pcmk_trace_functions:
.. index::
pair: node option; PCMK_trace_functions
PCMK_trace_functions
- :ref:`text `
-
- *Advanced Use Only:* Send debug and trace severity messages from these
(comma-separated) source code functions to the detail log.
Example:
``PCMK_trace_functions="func1,func2"``
* - .. _pcmk_trace_files:
.. index::
pair: node option; PCMK_trace_files
PCMK_trace_files
- :ref:`text `
-
- *Advanced Use Only:* Send debug and trace severity messages from all
functions in these (comma-separated) source file names to the detail log.
Example: ``PCMK_trace_files="file1.c,file2.c"``
* - .. _pcmk_trace_formats:
.. index::
pair: node option; PCMK_trace_formats
PCMK_trace_formats
- :ref:`text `
-
- *Advanced Use Only:* Send trace severity messages that are generated by
these (comma-separated) format strings in the source code to the detail
log.
Example: ``PCMK_trace_formats="Error: %s (%d)"``
* - .. _pcmk_trace_tags:
.. index::
pair: node option; PCMK_trace_tags
PCMK_trace_tags
- :ref:`text `
-
- *Advanced Use Only:* Send debug and trace severity messages related to
these (comma-separated) resource IDs to the detail log.
Example: ``PCMK_trace_tags="client-ip,dbfs"``
* - .. _pcmk_blackbox:
.. index::
pair: node option; PCMK_blackbox
PCMK_blackbox
- :ref:`enumeration `
- no
- *Advanced Use Only:* Enable blackbox logging globally (``yes`` or ``no``)
or by subsystem. A blackbox contains a rolling buffer of all logs (of all
severities). Blackboxes are stored under |CRM_BLACKBOX_DIR| by default,
by default, and their contents can be viewed using the ``qb-blackbox(8)``
command.
The blackbox recorder can be enabled at start using this variable, or at
runtime by sending a Pacemaker subsystem daemon process a ``SIGUSR1`` or
``SIGTRAP`` signal, and disabled by sending ``SIGUSR2`` (see
``kill(1)``). The blackbox will be written after a crash, assertion
failure, or ``SIGTRAP`` signal.
See :ref:`PCMK_debug ` for allowed subsystems.
Example:
``PCMK_blackbox="pacemakerd,pacemaker-execd"``
* - .. _pcmk_trace_blackbox:
.. index::
pair: node option; PCMK_trace_blackbox
PCMK_trace_blackbox
- :ref:`enumeration `
-
- *Advanced Use Only:* Write a blackbox whenever the message at the
specified function and line is logged. Multiple entries may be comma-
separated.
Example: ``PCMK_trace_blackbox="remote.c:144,remote.c:149"``
* - .. _pcmk_node_start_state:
.. index::
pair: node option; PCMK_node_start_state
PCMK_node_start_state
- :ref:`enumeration `
- default
- By default, the local host will join the cluster in an online or standby
state when Pacemaker first starts depending on whether it was previously
put into standby mode. If this variable is set to ``standby`` or
``online``, it will force the local host to join in the specified state.
* - .. _pcmk_node_action_limit:
.. index::
pair: node option; PCMK_node_action_limit
PCMK_node_action_limit
- :ref:`nonnegative integer `
-
- If set, this overrides the :ref:`node-action-limit `
cluster option on this node to specify the maximum number of jobs that
can be scheduled on this node (or 0 to use twice the number of CPU
cores).
* - .. _pcmk_fail_fast:
.. index::
pair: node option; PCMK_fail_fast
PCMK_fail_fast
- :ref:`boolean `
- no
- By default, if a Pacemaker subsystem crashes, the main ``pacemakerd``
process will attempt to restart it. If this variable is set to ``yes``,
``pacemakerd`` will panic the local host instead.
* - .. _pcmk_panic_action:
.. index::
pair: node option; PCMK_panic_action
PCMK_panic_action
- :ref:`enumeration `
- reboot
- Pacemaker will panic the local host under certain conditions. By default,
this means rebooting the host. This variable can change that behavior: if
``crash``, trigger a kernel crash (useful if you want a kernel dump to
investigate); if ``sync-reboot`` or ``sync-crash``, synchronize
filesystems before rebooting the host or triggering a kernel crash. The
sync values are more likely to preserve log messages, but with the risk
that the host may be left active if the synchronization hangs.
* - .. _pcmk_authkey_location:
.. index::
pair: node option; PCMK_authkey_location
PCMK_authkey_location
- :ref:`text `
- |PCMK_AUTHKEY_FILE|
- Use the contents of this file as the authorization key to use with
:ref:`Pacemaker Remote ` connections. This file must
be readable by Pacemaker daemons (that is, it must allow read
permissions to either the |CRM_DAEMON_USER| user or the
|CRM_DAEMON_GROUP| group), and its contents must be identical on all
nodes.
* - .. _pcmk_remote_address:
.. index::
pair: node option; PCMK_remote_address
PCMK_remote_address
- :ref:`text `
-
- By default, if the :ref:`Pacemaker Remote ` service is
run on the local node, it will listen for connections on all IP
addresses. This may be set to one address to listen on instead, as a
resolvable hostname or as a numeric IPv4 or IPv6 address. When resolving
names or listening on all addresses, IPv6 will be preferred if
available. When listening on an IPv6 address, IPv4 clients will be
supported via IPv4-mapped IPv6 addresses.
Example: ``PCMK_remote_address="192.0.2.1"``
* - .. _pcmk_remote_port:
.. index::
pair: node option; PCMK_remote_port
PCMK_remote_port
- :ref:`port `
- 3121
- Use this TCP port number for :ref:`Pacemaker Remote `
node connections. This value must be the same on all nodes.
* - .. _pcmk_remote_pid1:
.. index::
pair: node option; PCMK_remote_pid1
PCMK_remote_pid1
- :ref:`enumeration `
- default
- *Advanced Use Only:* When a bundle resource's ``run-command`` option is
left to default, :ref:`Pacemaker Remote ` runs as PID
1 in the bundle's containers. When it does so, it loads environment
variables from the container's |PCMK_INIT_ENV_FILE| and performs the PID
1 responsibility of reaping dead subprocesses.
This option controls whether those actions are performed when Pacemaker
Remote is not running as PID 1. It is intended primarily for developer
testing but can be useful when ``run-command`` is set to a separate,
custom PID 1 process that launches Pacemaker Remote.
* ``full``: Pacemaker Remote loads environment variables from
|PCMK_INIT_ENV_FILE| and reaps dead subprocesses.
* ``vars``: Pacemaker Remote loads environment variables from
|PCMK_INIT_ENV_FILE| but does not reap dead subprocesses.
* ``default``: Pacemaker Remote performs neither action.
If Pacemaker Remote is running as PID 1, this option is ignored, and the
behavior is the same as for ``full``.
* - .. _pcmk_tls_priorities:
.. index::
pair: node option; PCMK_tls_priorities
PCMK_tls_priorities
- :ref:`text `
- |PCMK__GNUTLS_PRIORITIES|
- - *Advanced Use Only:* These GnuTLS cipher priorities will be used for TLS
- connections (whether for :ref:`Pacemaker Remote `
- connections or remote CIB access, when enabled). See:
-
- https://gnutls.org/manual/html_node/Priority-Strings.html
+ - *Advanced Use Only:* These `GnuTLS cipher priorities
+ `_ will be
+ used for TLS connections (whether for :ref:`Pacemaker Remote
+ ` connections or remote CIB access, when enabled).
Pacemaker will append ``":+ANON-DH"`` for remote CIB access and
``":+DHE-PSK:+PSK"`` for Pacemaker Remote connections, as they are
required for the respective functionality.
Example:
``PCMK_tls_priorities="SECURE128:+SECURE192"``
* - .. _pcmk_dh_max_bits:
.. index::
pair: node option; PCMK_dh_max_bits
PCMK_dh_max_bits
- :ref:`nonnegative integer `
- 0 (no maximum)
- *Advanced Use Only:* Set an upper bound on the bit length of the prime
number generated for Diffie-Hellman parameters needed by TLS connections.
The default is no maximum.
The server (:ref:`Pacemaker Remote ` daemon, or CIB
manager configured to accept remote clients) will use this value to
provide a ceiling for the value recommended by the GnuTLS library. The
library will only accept a limited number of specific values, which vary
by library version, so setting these is recommended only when required
for compatibility with specific client versions.
Clients do not use ``PCMK_dh_max_bits``.
* - .. _pcmk_ipc_type:
.. index::
pair: node option; PCMK_ipc_type
PCMK_ipc_type
- :ref:`enumeration `
- shared-mem
- *Advanced Use Only:* Force use of a particular IPC method. Allowed values:
* ``shared-mem``
* ``socket``
* ``posix``
* ``sysv``
* - .. _pcmk_ipc_buffer:
.. index::
pair: node option; PCMK_ipc_buffer
PCMK_ipc_buffer
- :ref:`nonnegative integer `
- 131072
- *Advanced Use Only:* Specify an IPC buffer size in bytes. This can be
useful when connecting to large clusters that result in messages
exceeding the default size (which will also result in log messages
referencing this variable).
* - .. _pcmk_cluster_type:
.. index::
pair: node option; PCMK_cluster_type
PCMK_cluster_type
- :ref:`enumeration `
- corosync
- *Advanced Use Only:* Specify the cluster layer to be used. If unset,
Pacemaker will detect and use a supported cluster layer, if available.
Currently, ``"corosync"`` is the only supported cluster layer. If
multiple layers are supported in the future, this will allow overriding
Pacemaker's automatic detection to select a specific one.
* - .. _pcmk_schema_directory:
.. index::
pair: node option; PCMK_schema_directory
PCMK_schema_directory
- :ref:`text `
- |PCMK_SCHEMA_DIR|
- *Advanced Use Only:* Specify an alternate location for RNG schemas and
XSL transforms.
* - .. _pcmk_remote_schema_directory:
.. index::
pair: node option; PCMK_remote_schema_directory
PCMK_remote_schema_directory
- :ref:`text `
- |PCMK__REMOTE_SCHEMA_DIR|
- *Advanced Use Only:* Specify an alternate location on
:ref:`Pacemaker Remote ` nodes for storing newer RNG
schemas and XSL transforms fetched from the cluster.
* - .. _pcmk_valgrind_enabled:
.. index::
pair: node option; PCMK_valgrind_enabled
PCMK_valgrind_enabled
- :ref:`enumeration `
- no
- *Advanced Use Only:* Whether subsystem daemons should be run under
``valgrind``. Allowed values are the same as for ``PCMK_debug``.
* - .. _pcmk_callgrind_enabled:
.. index::
pair: node option; PCMK_callgrind_enabled
PCMK_callgrind_enabled
- :ref:`enumeration `
- no
- *Advanced Use Only:* Whether subsystem daemons should be run under
``valgrind`` with the ``callgrind`` tool enabled. Allowed values are the
same as for ``PCMK_debug``.
* - .. _sbd_sync_resource_startup:
.. index::
pair: node option; SBD_SYNC_RESOURCE_STARTUP
SBD_SYNC_RESOURCE_STARTUP
- :ref:`boolean `
-
- If true, ``pacemakerd`` waits for a ping from ``sbd`` during startup
before starting other Pacemaker daemons, and during shutdown after
stopping other Pacemaker daemons but before exiting. Default value is set
based on the ``--with-sbd-sync-default`` configure script option.
* - .. _sbd_watchdog_timeout:
.. index::
pair: node option; SBD_WATCHDOG_TIMEOUT
SBD_WATCHDOG_TIMEOUT
- :ref:`duration `
-
- If the ``stonith-watchdog-timeout`` cluster property is set to a negative
or invalid value, use double this value as the default if positive, or
use 0 as the default otherwise. This value must be greater than the value
of ``stonith-watchdog-timeout`` if both are set.
* - .. _valgrind_opts:
.. index::
pair: node option; VALGRIND_OPTS
VALGRIND_OPTS
- :ref:`text `
-
- *Advanced Use Only:* Pass these options to valgrind, when enabled (see
``valgrind(1)``). ``"--vgdb=no"`` should usually be specified because
``pacemaker-execd`` can lower privileges when executing commands, which
would otherwise leave a bunch of unremovable files in ``/tmp``.