[[https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-2.1.0 | Pacemaker 2.1.0]], released in June 2021, is a minor Pacemaker release that preserves backward compatibility for both rolling upgrades and the C API, which means there should be no major concerns for individuals or distributions who upgrade. The minor version bump is intended to draw attention to larger changes than usual, including changes that may affect any custom scripts users have written.
As with Pacemaker 2.0, Corosync 2 or later is supported as the underlying cluster layer, and rolling upgrades from Pacemaker 1.1.11 or later are possible as long as the cluster layer stays at the same Corosync major version (2 or 3).
Since backward compatibility will be maintained, there will be no need to backport future changes to the 2.0 series, and there will be no more 2.0 series releases.
== New Features ==
=== OCF 1.1 Support ===
Pacemaker now supports the recently adopted OCF Resource Agent API 1.1 standard. Related changes are described in the relevant sections of this page.
=== Noncritical resources ===
Colocation constraints may take a new "influence" option that determines whether the dependent resource influences the location of the main resource, if the main resource is already active. The default of true preserves the previous behavior. Setting it to false has the effect of making the dependent resource stop if it reaches its migration-threshold in failures, rather than cause the main resource to move to another node.
Resources may take a new "critical" meta-attribute that serves as a default for all colocation constraints involving the resource as the dependent, as well as all groups involving the resource.
As an example, a large, important database that takes a long time to start and stop may have a less important, occasionally used reporting tool colocated or grouped with it. In this situation, it might be preferable to stop the reporting tool if it fails too many times, rather than cause a database outage to move both resources. In this case, either the colocation constraint could be marked with influence=false, or the reporting tool resource could be give the meta-attribute critical=false, to achieve the desired effect.
=== Sync before panic ===
Pacemaker's existing '''PCMK_panic_action''' environment variable (typically set in /etc/sysconfig/pacemaker, /etc/default/pacemaker, or similar) determines how Pacemaker responds to certain critical errors. Previously, it could be set to "crash" to trigger a kernel crash (generating a kernel dump for investigation) or "reboot" (actually any other value) to trigger a reboot (the default behavior).
Now, it may also take the values "sync-crash" or "sync-reboot", which will attempt to sync the host's disks before crashing or rebooting. This can be helpful to record any log messages cached just before the failure, however it runs the risk of the sync hanging (for example if a disk controller has issues), which would leave the host running after a critical error.
== Build process changes ==
=== Dependency versions ===
The minimum dependency versions required to build Pacemaker have been raised to these (all released 6 or more years ago and available in recent and even many not-so-recent versions of all major Linux distributions):
* Python 3.2 or later (support for Python 2.7 has been removed)
* glib 2.32.0 or later
* libqb 0.17.0 or later
* GnuTLS 2.12.0 or later (to enable Pacemaker Remote support)
* rpm 4.11.0 or later (if building RPMs)
* C library must provide setenv() and unsetenv()
=== configure script: changes in option defaults ===
* The default for '''--enable-legacy-links''' is now "no" instead of "yes". This means that symbolic links at the pacemaker 1.1 daemon names, such as crmd linked to pacemaker-controld, will no longer be installed by default. However, a link to the old pacemaker_remoted name will now always be created, regardless of the value of this option, to ensure that container images with Pacemaker 2 can be used in bundles on cluster nodes running Pacemaker 1. This option is now deprecated, and will be removed in a future release.
* '''--with-ocfdir''' now defaults to the value provided by the resource-agents package if available, or otherwise the previous default of /usr/lib/ocf. This is passed to resource agents as the value of the OCF_ROOT environment variable, which is used to find the ocf-shellfuncs include files.
=== configure script: new options ===
* If set to "yes", '''--enable-compat-2.0''' (which defaults to "no") will revert certain output changes, to maintain compatibility with tools, scripts, and resource agents that rely on the previous output. The exact changes that are reverted will be noted below with the changes themselves.
* '''--with-concurrent-fencing-default''' can be used to set the default for the concurrent-fencing option. The default of "false" preserves the previous behavior. Setting it to "true" replaces the previous method of setting -DDEFAULT_CONCURRENT_FENCING_TRUE in CFLAGS or CPPFLAGS before configuring (which will no longer have any effect).
* '''--with-resource-stickiness-default''' can be used to set the default for the resource-stickiness resource meta-attribute. The default of "0" preserves the previous behavior. If positive, this value will be added as an explicit resource default in new CIBs (which means it will not affect any existing clusters, even in a rolling upgrade). Distributions and users who build from source can set this, if their users expect sticky behavior by default and are regularly surprised by resources moving around by default.
* '''--with-sbd-sync-default''' can be used to set the default for the SBD_SYNC_RESOURCE_STARTUP environment variable. The default of "false" preserves the previous behavior. If set to "true", SBD ''must'' be be built to default similarly, otherwise data corruption could occur. Building both Pacemaker and SBD to default to synchronization improves safety, without requiring higher-level tools to be aware of the setting or requiring users to modify configurations after upgrading to versions that support synchronization.
* '''--with-ocfrainstalldir''' specifies where the ocf:pacemaker resource agents will be installed, defaulting to the value provided by the resource-agents package if available, or otherwise the previous behavior (the value of --with-ocfdir plus "/resource.d").
* '''--with-ocfrapath''' specifies a colon-separated path of directories that will be searched for OCF resource agents (like PATH, for agents), defaulting to the value provided by the resource-agents pacakge if available, or otherwise the previous behavior (searching only the value of --with-ocfdir plus "/resource.d").
* '''--with-gnutls''' can now be used to determine support for Pacemaker Remote and remote-tls-port. The default of "try" preserves the former behavior (enable support if a recent enough GnuTLS is available). "yes" will fail if support cannot be enabled, and "no" will disable support even if it's possible.
=== configure script: removed options ===
* Remove support for the '''--with-acl''' option. Support for ACLs will now always be built. Setting --with-acl to "no" has been broken since the 1.1.16 release, and since no one's even noticed it until now, dropping the option seems like a better choice than fixing it.
* Remove support for the deprecated '''--with-pkgname''' and '''--with-pkg-name''' options, which have no effect
* Remove support for the '''--with-brand''' option, which was only needed for Publican (see "Documentation changes" below)
* Remove support for the '''--enable-ansi''' option, which hasn't worked in a very long time
* Remove support for the '''--enable-no-stack''' option, which builds only the scheduler
=== configure script: other changes ===
* Stack clash protection will now be added by default to compiler hardening flags, when available. Hardening can still be disabled with --disable-hardening, or overridden by setting the various HARDENED environment variables before calling configure.
* The configure script already supports a '''--with-runstatedir''' option that is ignored if the installed autoconf version supports the --runstatedir option. Until recently, only Debian patched its autoconf package to support --runstatedir, but autoconf 2.70 was released in December 2020 with support, so as time goes on, more builds will need to use the standard --runstatedir instead of --with-runstatedir.
=== Removed files ===
* The '''clustermon.sh''' script, which was the pre-1.1.15 way of doing alerts
* The nonfunctional '''ocf:pacemaker:pingd''' resource agent (the separate and widely used ocf:pacemaker:ping agent is unaffected)
* The '''cibmon''' utility, which was a no-longer-useful holdover from the heartbeat days typically installed in /usr/libexec/pacemaker
* The '''showscores.sh''' script, which was in the extra directory (not installed by default) and hasn't worked in several releases
* The '''openstack.md''' documentation file, which was in the doc directory (not installed by default), not of general interest, and outdated
=== RPMs ===
* A new --with/--without '''sbd_sync''' rpmbuild option will set the default for SBD_SYNC_RESOURCE_STARTUP to true/false (see above).
* "subtree" is now the default for '''RPMDEST''' in GNUmakefile, which means that the "make rpm" target will put all artifacts in a dedicated "rpm/" subdirectory rather than a mix of the top-level directory and system defaults. You can run "make RPMDEST=toplevel rpm" to get the previous behavior, though that will be considered deprecated and eventually removed.
* "make clean" will now remove rpmbuild-created directories from the "rpm/" subdirectory when RPMDEST is set to subtree
* RPMs no longer provide the obsolete '''Group:''' tag
* Users can set the '''ocf_root''' variable at the top of the spec file to change the OCF root directory. The default now respects the _prefix variable instead of hardcoding /usr in the path, and the variable will be used when both configuring the build and installing agents.
* Users can set the '''bug_url''' variable at the top of the spec file to change the bug report URL
* Default configure options used when building RPMs may now differ by OS distribution and version, to be more similar to the respective stock packages. In particular, support for CIB secrets will now be enabled by default, except for certain distribution versions.
=== Other ===
* The Pacemaker '''environment variables file''' (typically installed as /etc/sysconfig/pacemaker, /etc/default/pacemaker, etc., depending on OS and distribution) will now be installed when `make install` is run. If the file already exists, the new file will be installed with a `.new` file extension. Previously, the user had to install the file themselves, or use the `make rpm` command which would install the file via the generated RPM packages (which still works).
* Remove support for the deprecated and undocumented '''SUSE_ACL_COMPAT''' compile-time option. (Official SUSE builds have never used this option.)
== Logging changes ==
* If Pacemaker is built with libqb 2.0 or later, the detail log will use millisecond-resolution timestamps. (If for some reason this causes a problem for some users, we could potentially make this behavior optional.)
* Pacemaker now uses the OCF 1.1 role names, "Promoted" and "Unpromoted", in log messages. ''(If Pacemaker is built with the --enable-compat-2.0 configure option, the old names will still be used in some messages.)''
== Cluster Information Base (CIB) changes ==
* Pacemaker now accepts the '''OCF 1.1 standard role names''' ("Promoted" and "Unpromoted") in addition to the legacy terms "Master" and "Slave", which are now deprecated, anywhere roles are specified in Pacemaker configuration (such as the "target-role" resource meta-attribute). Using the new names in "role" in <op>, <rsc_location>, or <resource_set>, or in "rsc-role" or "with-rsc-role" in <rsc_ticket> or <rsc_colocation>, requires CIB schema 3.7 (in other words, "cibadmin --upgrade" or an equivalent must be run on an existing cluster to use the new names).
* The undocumented '''can_fail''' operation meta-attribute, which serves no useful purpose, is now deprecated and will be removed in a future version. A deprecation warning will be logged if it is changed from its default ''(unless Pacemaker is built with the --enable-compat-2.0 configure option)''.
* The obscure and useless '''remove-after-stop''' cluster option is now deprecated and will be removed in a future version. A deprecation warning will be logged if it is changed from its default ''(unless Pacemaker is built with the --enable-compat-2.0 configure option)''.
* Support for '''Upstart''' (both starting Pacemaker via Upstart, and managing Upstart-based resources in the cluster) is officially deprecated, and will be removed in a future release.
* Pacemaker will stop adding the '''last-run''' attribute to '''<lrm_rsc_op>''' entries in the '''<status>''' section. This attribute hasn't been added to recurring operations since version 1.1.5, and is always identical to last-rc-change for non-recurring operations. As a result, last-run will no longer appear in crm_mon XML output either.
== Tool changes ==
=== All ===
* More commands support the '''--output-as''' and '''--output-to''' options, including XML output. These include crmadmin, crm_resource, crm_simulate, and crm_verify (in addition to crm_mon and stonith_admin, which already did). The intent is that the XML output will remain stable across releases as much as possible, for reliable parsing by scripts, while the text output may change from release to release for human-friendliness.
* Some tools currently have (or had) undocumented support for '''long option abbreviation'''; for example, if the long option is "--upgrade", then any shorter abbreviation ("--upgr", "--up", etc.) would also work as long as it is unique for the tool. This support has been removed from many tools and is now considered deprecated for remaining tools, and it will be dropped from those in future releases.
* Most short options that take arguments currently make the space after the short option optional, for example "-X saved.xml" or "-Xsaved.xml". Specifying a '''short option without a space''' is now considered deprecated; a future release will print a warning when this is used, and an even later release will drop support.
* OCF resource agents will be displayed as ocf:PROVIDER:AGENT rather than '''ocf::PROVIDER:AGENT''' in crm_mon and crm_simulate output and pacemaker-schedulerd logs ''unless Pacemaker is built with the --enable-compat-2.0 configure option.''
* The OCF 1.1 role names ("Promoted" and "Unpromoted") will be used in all tool output instead of the legacy names ''unless Pacemaker is built with the --enable-compat-2.0 configure option.''
=== crm_resource ===
* The '''-a/--constraints''' and '''-A/--stack''' options (to display location and colocation constraints related to a given resource) now have considerably different (and hopefully more readable) text output.
* The constraints created by '''crm_resource --ban''' will now use the OCF 1.1 standard role name "Promoted" ''unless Pacemaker is built with the --enable-compat-2.0 configure option.''
* The '''crm_resource --master''' option is now deprecated (but still accepted) and replaced with a new '''--promoted''' option.
=== crm_simulate ===
* Output is now formatted similarly to crm_mon's.
* New '''--show-attrs''' option will display node attribute values in simulation output.
* New '''--show-failcounts''' option will display resource fail counts in simulation output.
* stderr will no longer automatically be redirected to stdout when '''--verbose''' is given ''unless Pacemaker is built with the --enable-compat-2.0 configure option.''
=== crmadmin ===
* The '''-q/--quiet''' option previously printed its usual output to stdout and a briefer version to stderr. For consistency with other tools, it now prints only the brief version, to stdout.
* The '''-t/--timeout''' option, which previously took an integer number of milliseconds, now takes a Pacemaker time specification (bare number of seconds, integer plus units, or ISO 8601 duration) for consistency with the other tools and configuration.
* The text output for the '''-S/--status''' option has been changed to be more readable. (Scripts are recommended to use the new XML output support if parsing the output.)
* The long-broken '''-E/--election''' and '''-K/--kill''' options have been completely removed. (They were only ever useful for debugging during early Pacemaker development.)
=== Other ===
* When showing ban constraints, '''crm_mon --output-as=xml''' (and --as-xml) will now show promoted-only=true/false in addition to master_only=true/false, which is now deprecated.
* The '''crm_master''' command has been deprecated (but still works) and replaced with a new '''crm_attribute --promotion''' option that defaults to --lifetime=reboot. This is typically called from within a resource agent. For example, "crm_master -l reboot -v 10" becomes "crm_attribute --promotion -v 10".
* The output of the '''pacemakerd --features''' command better reflects the build options currently available. (Feature listings are not guaranteed to stay the same from release to release, and are intended only for troubleshooting.)
== Resource agents ==
* Pacemaker now supports OCF 1.1 compliant resource agents (in particular, reloadable parameters and the reload-agent action).
* Pacemaker now provides resource agents with new environment variables (in addition to the existing ones) for promotable clone notifications, with "master" replaced with "promoted" and "slave" replaced with "unpromoted", for example OCF_RESKEY_CRM_meta_notify_unpromoted_resource will be identical to OCF_RESKEY_CRM_meta_notify_slave_resource. The old names are now deprecated.
* The ocf:pacemaker:Dummy, ocf:pacemaker:Stateful, and ocf:pacemaker:remote resource agents now support OCF 1.1 (reloadable parameters, UI parameter hints, and advertising OCF 1.1 role names in monitor action meta-data).
== Documentation changes ==
* The source for the Pacemaker "books" (''Clusters from Scratch'', ''Pacemaker Explained'', etc.) has been switched from using asciidoc source and Publican to using '''reStructuredText source and Sphinx'''. This is mainly because Publican is a dead project and distributions are dropping it, but it will also give us a cleaner, more modern look.
* The '''--with-brand''' configure option has been removed. The books will now be built based on whether the sphinx-build and inkscape commands are available at build time.
* The '''--with doc''' RPM build option remains, to decide whether to enable python3-sphinx and inkscape as build requirements for the pacemaker-doc subpackage (effectively controlling whether the books are built or not). The previous build requirement for publican has been removed. (The build requirement of asciidoc or asciidoctor remains, for generating the deprecated text file documentation, but that may change.)
* The directory structure for the generated books has changed:
| Generated content | 2.0 location | 2.1 location |
| ------------------- | --------- | --------- |
| Books built locally (in source directory) | doc/''TITLE''/publish/''FORMAT''/en-US/ | doc/sphinx/''TITLE''/_build/''FORMAT''/ |
| Installed books (in package documentation directory a.k.a. docdir) | ''TITLE''/''FORMAT''/en-US/ | ''TITLE''/''FORMAT''/ |
| Uploaded books (in RSYNC_DEST directory) | pacemaker/doc/en-US/Pacemaker/2.0/''FORMAT''/''TITLE''/ | pacemaker/doc/2.1/''TITLE''/''FORMAT'' |
* Pacemaker uses the OCF 1.1 role names, "Promoted" and "Unpromoted", in the books, as well as tool help and man pages.
== Development changes ==
=== Public C API changes ===
If you develop code using the Pacemaker C libraries, these could affect you.
* A new constant, '''PCMK_ALLOW_DEPRECATED''', enables control of API availability. It should normally be left undefined, but if you want to check whether your code uses any deprecated Pacemaker APIs, define it to 0 before including any Pacemaker headers.
* The following APIs have been '''deprecated''' and will be removed in a future release (replacements are often not one-to-one, so read their relevant API documentation):
|libcrmcluster |
|------------- |
| Deprecated | Recommended replacement |
| ----------- | -------------------- |
| crm_terminate_member() | Use libstonithd APIs instead |
| crm_terminate_member_no_mainloop() | Use libstonithd APIs instead |
|libcrmcommon|
|----------------|
| Deprecated | Recommended replacement |
| ------------- | ----------------------------- |
| __builtin_expect() | Use only with GNU C compiler |
| __likely() | __builtin_expect() |
| __unlikely() | __builtin_expect() |
| EOS | '\0' |
| MAX_IPC_DELAY | none |
| GListPtr | GList* |
| PCMK_OCF_DEGRADED_MASTER | PCMK_OCF_DEGRADED_PROMOTED |
| PCMK_OCF_FAILED_MASTER | PCMK_OCF_FAILED_PROMOTED |
| PCMK_OCF_RUNNING_MASTER | PCMK_OCF_RUNNING_PROMOTED |
| XML_CIB_TAG_MASTER | PCMK_XE_PROMOTABLE_LEGACY |
| XML_RSC_ATTR_MASTER_MAX | PCMK_XE_PROMOTED_MAX_LEGACY |
| XML_RSC_ATTR_MASTER_NODEMAX | PCMK_XE_PROMOTED_NODE_MAX_LEGACY |
| crm_atoi() | strtoll() |
| crm_build_path() | none |
| crm_config_error | none |
| crm_config_warning | none |
| crm_ftoa() | a printf()-style function |
| crm_hash_table_size() | g_hash_table_size() |
| crm_itoa() | a printf()-style function |
| crm_itoa_stack() | snprintf() |
| crm_log_cli_init() | none |
| crm_parse_int() | strtoll() |
| crm_parse_ll() | strtoll() |
| crm_str_hash() | g_str_hash() |
| crm_str_table_dup() | none |
| crm_str_table_new() | g_hash_table_new_full() |
| crm_strcase_equal() | none |
| crm_strcase_hash() | none |
| crm_strcase_table_new() | g_hash_table_new_full() |
| crm_strip_trailing_newline() | none |
| crm_ttoa() | a printf()-style function |
| g_str_hash_traditional() | g_str_hash() |
| pcmk_format_named_time() | printf()-style function |
| pcmk_format_nvpair() | printf()-style function |
| pcmk_numeric_strcasecmp() | none |
| pcmk_scan_nvpair() | sscanf() |
|libpe_status|
|-----------|
| Deprecated | Recommended replacement |
| ---------- | -------------------- |
| pe_order_implies_first_master | pe_order_promoted_implies_first |
| pe_print_details | none |
| pe_print_dev | none |
| pe_print_html | none |
| pe_print_log | none |
| pe_print_max_details | none |
| pe_print_ncurses | none |
| pe_print_xml | none |
| pe_resource_t parameters member | pe_rsc_params() |
| RSC_ROLE_MASTER | RSC_ROLE_PROMOTED |
| RSC_ROLE_MASTER_S | RSC_ROLE_PROMOTED_LEGACY_S |
| RSC_ROLE_SLAVE | RSC_ROLE_UNPROMOTED |
| RSC_ROLE_SLAVE_S | RSC_ROLE_UNPROMOTED_LEGACY_S |
* The following public APIs have '''changed''' behavior:
** libpe_status: resource_object_functions_t parameter() member now ignores its `create` argument (parameters will always be created if not already available)
* The following public APIs have been '''added''':
** libcrmcommon
*** CRMD_ACTION_RELOAD_AGENT string constant
*** enum ocf_exitcode (moved from libcrmservice)
*** PCMK_OCF_MAJOR_VERSION string constant
*** PCMK_OCF_MINOR_VERSION string constant
*** PCMK_OCF_RUNNING_PROMOTED enum value
*** PCMK_OCF_VERSION string constant
*** PCMK_XE_PROMOTABLE_LEGACY string constant
*** PCMK_XE_PROMOTED_MAX_LEGACY string constant
*** PCMK_XE_PROMOTED_NODE_MAX_LEGACY string constant
** libpacemaker
*** pcmk_controller_status()
*** pcmk_designated_controller()
*** pcmk_list_nodes()
*** pcmk_pacemakerd_status()
*** pcmk_resource_digests()
** libpe_status
*** RSC_ROLE_PROMOTED enum value
*** RSC_ROLE_PROMOTED_LEGACY_S string constant
*** RSC_ROLE_PROMOTED_S string constant
*** RSC_ROLE_UNPROMOTED enum value
*** RSC_ROLE_UNPROMOTED_LEGACY_S string constant
*** RSC_ROLE_UNPROMOTED_S string constant
*** pcmk_working_set_t priv member (for Pacemaker use only)
*** pe_order_promoted_implies_first enum value
*** pe_resource_t parameter_cache member (managed via pe_rsc_params())
*** pe_rsc_params()
=== Source files moved ===
If you make changes to the Pacemaker source code, or build from the master branch, these could affect you.
* Certain '''build artifacts''' are now created in the libltdl subdirectory instead of the top-level directory. This is uninteresting unless you update an existing source checkout and want to rebuild, in which case you first need to run:
```
rm -rf aclocal.m4 autom4te.cache config.guess config.log config.status config.sub \
configure depcomp install-sh libltdl libtool ltmain.sh missing py-compile test-driver
./autogen.sh
```
* Certain files changed locations in the source tree. Their installed locations did not change.
| 2.0 location | 2.1 location |
| ------------- | ---------- |
| version.m4 | m4/version.m4 |
| extra/logrotate/pacemaker.in | etc/logrotate.d/pacemaker.in |
| daemons/pacemakerd/pacemaker.in | etc/init.d/pacemaker.in |
| daemons/pacemakerd/pacemaker.sysconfig | etc/sysconfig/pacemaker |
| tools/crm_mon.sysconfig | etc/sysconfig/crm_mon |
| cts/cts-support.in and related files | cts/support/* |
| cts/CTSlab.py.in, cts/CTS.py.in, and related files | cts/lab/* |