Pacemaker 2.1 Changes

Updated 328 Days AgoPublic

Actions

Pacemaker 2.1.0, released in June 2021, is a minor Pacemaker release that preserves backward compatibility for both rolling upgrades and the C API, which means there should be no major concerns for individuals or distributions who upgrade. The minor version bump is intended to draw attention to larger changes than usual, including changes that may affect any custom scripts users have written.

As with Pacemaker 2.0, Corosync 2 or later is supported as the underlying cluster layer, and rolling upgrades from Pacemaker 1.1.11 or later are possible as long as the cluster layer stays at the same Corosync major version (2 or 3).

Since backward compatibility will be maintained, there will be no need to backport future changes to the 2.0 series, and there will be no more 2.0 series releases.

New Features

OCF 1.1 Support

Pacemaker now supports the recently adopted OCF Resource Agent API 1.1 standard. Related changes are described in the relevant sections of this page.

Noncritical resources

Colocation constraints may take a new "influence" option that determines whether the dependent resource influences the location of the main resource, if the main resource is already active. The default of true preserves the previous behavior. Setting it to false has the effect of making the dependent resource stop if it reaches its migration-threshold in failures, rather than cause the main resource to move to another node.

Resources may take a new "critical" meta-attribute that serves as a default for all colocation constraints involving the resource as the dependent, as well as all groups involving the resource.

As an example, a large, important database that takes a long time to start and stop may have a less important, occasionally used reporting tool colocated or grouped with it. In this situation, it might be preferable to stop the reporting tool if it fails too many times, rather than cause a database outage to move both resources. In this case, either the colocation constraint could be marked with influence=false, or the reporting tool resource could be give the meta-attribute critical=false, to achieve the desired effect.

Sync before panic

Pacemaker's existing PCMK_panic_action environment variable (typically set in /etc/sysconfig/pacemaker, /etc/default/pacemaker, or similar) determines how Pacemaker responds to certain critical errors. Previously, it could be set to "crash" to trigger a kernel crash (generating a kernel dump for investigation) or "reboot" (actually any other value) to trigger a reboot (the default behavior).

Now, it may also take the values "sync-crash" or "sync-reboot", which will attempt to sync the host's disks before crashing or rebooting. This can be helpful to record any log messages cached just before the failure, however it runs the risk of the sync hanging (for example if a disk controller has issues), which would leave the host running after a critical error.

Build process changes

Dependency versions

The minimum dependency versions required to build Pacemaker have been raised to these (all released 6 or more years ago and available in recent and even many not-so-recent versions of all major Linux distributions):

• Python 3.2 or later (support for Python 2.7 has been removed)
• glib 2.32.0 or later
• libqb 0.17.0 or later
• GnuTLS 2.12.0 or later (to enable Pacemaker Remote support)
• rpm 4.11.0 or later (if building RPMs)
• C library must provide setenv() and unsetenv()

configure script: changes in option defaults

• The default for --enable-legacy-links is now "no" instead of "yes". This means that symbolic links at the pacemaker 1.1 daemon names, such as crmd linked to pacemaker-controld, will no longer be installed by default. However, a link to the old pacemaker_remoted name will now always be created, regardless of the value of this option, to ensure that container images with Pacemaker 2 can be used in bundles on cluster nodes running Pacemaker 1. This option is now deprecated, and will be removed in a future release.
• --with-ocfdir now defaults to the value provided by the resource-agents package if available, or otherwise the previous default of /usr/lib/ocf. This is passed to resource agents as the value of the OCF_ROOT environment variable, which is used to find the ocf-shellfuncs include files.

configure script: new options

• If set to "yes", --enable-compat-2.0 (which defaults to "no") will revert certain output changes, to maintain compatibility with tools, scripts, and resource agents that rely on the previous output. The exact changes that are reverted will be noted below with the changes themselves.
• --with-concurrent-fencing-default can be used to set the default for the concurrent-fencing option. The default of "false" preserves the previous behavior. Setting it to "true" replaces the previous method of setting -DDEFAULT_CONCURRENT_FENCING_TRUE in CFLAGS or CPPFLAGS before configuring (which will no longer have any effect).
• --with-resource-stickiness-default can be used to set the default for the resource-stickiness resource meta-attribute. The default of "0" preserves the previous behavior. If positive, this value will be added as an explicit resource default in new CIBs (which means it will not affect any existing clusters, even in a rolling upgrade). Distributions and users who build from source can set this, if their users expect sticky behavior by default and are regularly surprised by resources moving around by default.
• --with-sbd-sync-default can be used to set the default for the SBD_SYNC_RESOURCE_STARTUP environment variable. The default of "false" preserves the previous behavior. If set to "true", SBD must be be built to default similarly, otherwise data corruption could occur. Building both Pacemaker and SBD to default to synchronization improves safety, without requiring higher-level tools to be aware of the setting or requiring users to modify configurations after upgrading to versions that support synchronization.
• --with-ocfrainstalldir specifies where the ocf:pacemaker resource agents will be installed, defaulting to the value provided by the resource-agents package if available, or otherwise the previous behavior (the value of --with-ocfdir plus "/resource.d").
• --with-ocfrapath specifies a colon-separated path of directories that will be searched for OCF resource agents (like PATH, for agents), defaulting to the value provided by the resource-agents pacakge if available, or otherwise the previous behavior (searching only the value of --with-ocfdir plus "/resource.d").
• --with-gnutls can now be used to determine support for Pacemaker Remote and remote-tls-port. The default of "try" preserves the former behavior (enable support if a recent enough GnuTLS is available). "yes" will fail if support cannot be enabled, and "no" will disable support even if it's possible.

configure script: removed options

• Remove support for the --with-acl option. Support for ACLs will now always be built. Setting --with-acl to "no" has been broken since the 1.1.16 release, and since no one's even noticed it until now, dropping the option seems like a better choice than fixing it.
• Remove support for the deprecated --with-pkgname and --with-pkg-name options, which have no effect
• Remove support for the --with-brand option, which was only needed for Publican (see "Documentation changes" below)
• Remove support for the --enable-ansi option, which hasn't worked in a very long time
• Remove support for the --enable-no-stack option, which builds only the scheduler

configure script: other changes

• Stack clash protection will now be added by default to compiler hardening flags, when available. Hardening can still be disabled with --disable-hardening, or overridden by setting the various HARDENED environment variables before calling configure.
• The configure script already supports a --with-runstatedir option that is ignored if the installed autoconf version supports the --runstatedir option. Until recently, only Debian patched its autoconf package to support --runstatedir, but autoconf 2.70 was released in December 2020 with support, so as time goes on, more builds will need to use the standard --runstatedir instead of --with-runstatedir.

Removed files

• The clustermon.sh script, which was the pre-1.1.15 way of doing alerts
• The nonfunctional ocf:pacemaker:pingd resource agent (the separate and widely used ocf:pacemaker:ping agent is unaffected)
• The cibmon utility, which was a no-longer-useful holdover from the heartbeat days typically installed in /usr/libexec/pacemaker
• The showscores.sh script, which was in the extra directory (not installed by default) and hasn't worked in several releases
• The openstack.md documentation file, which was in the doc directory (not installed by default), not of general interest, and outdated

RPMs

• A new --with/--without sbd_sync rpmbuild option will set the default for SBD_SYNC_RESOURCE_STARTUP to true/false (see above).
• "subtree" is now the default for RPMDEST in GNUmakefile, which means that the "make rpm" target will put all artifacts in a dedicated "rpm/" subdirectory rather than a mix of the top-level directory and system defaults. You can run "make RPMDEST=toplevel rpm" to get the previous behavior, though that will be considered deprecated and eventually removed.
• "make clean" will now remove rpmbuild-created directories from the "rpm/" subdirectory when RPMDEST is set to subtree
• RPMs no longer provide the obsolete Group: tag
• Users can set the ocf_root variable at the top of the spec file to change the OCF root directory. The default now respects the _prefix variable instead of hardcoding /usr in the path, and the variable will be used when both configuring the build and installing agents.
• Users can set the bug_url variable at the top of the spec file to change the bug report URL
• Default configure options used when building RPMs may now differ by OS distribution and version, to be more similar to the respective stock packages. In particular, support for CIB secrets will now be enabled by default, except for certain distribution versions.

Other

• The Pacemaker environment variables file (typically installed as /etc/sysconfig/pacemaker, /etc/default/pacemaker, etc., depending on OS and distribution) will now be installed when make install is run. If the file already exists, the new file will be installed with a .new file extension. Previously, the user had to install the file themselves, or use the make rpm command which would install the file via the generated RPM packages (which still works).
• Remove support for the deprecated and undocumented SUSE_ACL_COMPAT compile-time option. (Official SUSE builds have never used this option.)

Logging changes

• If Pacemaker is built with libqb 2.0 or later, the detail log will use millisecond-resolution timestamps. (If for some reason this causes a problem for some users, we could potentially make this behavior optional.)

• Pacemaker now uses the OCF 1.1 role names, "Promoted" and "Unpromoted", in log messages. (If Pacemaker is built with the --enable-compat-2.0 configure option, the old names will still be used in some messages.)

Cluster Information Base (CIB) changes

• Pacemaker now accepts the OCF 1.1 standard role names ("Promoted" and "Unpromoted") in addition to the legacy terms "Master" and "Slave", which are now deprecated, anywhere roles are specified in Pacemaker configuration (such as the "target-role" resource meta-attribute). Using the new names in "role" in <op>, <rsc_location>, or <resource_set>, or in "rsc-role" or "with-rsc-role" in <rsc_ticket> or <rsc_colocation>, requires CIB schema 3.7 (in other words, "cibadmin --upgrade" or an equivalent must be run on an existing cluster to use the new names).

• The undocumented can_fail operation meta-attribute, which serves no useful purpose, is now deprecated and will be removed in a future version. A deprecation warning will be logged if it is changed from its default (unless Pacemaker is built with the --enable-compat-2.0 configure option).

• The obscure and useless remove-after-stop cluster option is now deprecated and will be removed in a future version. A deprecation warning will be logged if it is changed from its default (unless Pacemaker is built with the --enable-compat-2.0 configure option).

• Support for Upstart (both starting Pacemaker via Upstart, and managing Upstart-based resources in the cluster) is officially deprecated, and will be removed in a future release.

• Pacemaker will stop adding the last-run attribute to <lrm_rsc_op> entries in the <status> section. This attribute hasn't been added to recurring operations since version 1.1.5, and is always identical to last-rc-change for non-recurring operations. As a result, last-run will no longer appear in crm_mon XML output either.

Tool changes

All

• More commands support the --output-as and --output-to options, including XML output. These include crmadmin, crm_resource, crm_simulate, and crm_verify (in addition to crm_mon and stonith_admin, which already did). The intent is that the XML output will remain stable across releases as much as possible, for reliable parsing by scripts, while the text output may change from release to release for human-friendliness.

• Some tools currently have (or had) undocumented support for long option abbreviation; for example, if the long option is "--upgrade", then any shorter abbreviation ("--upgr", "--up", etc.) would also work as long as it is unique for the tool. This support has been removed from many tools and is now considered deprecated for remaining tools, and it will be dropped from those in future releases.

• Most short options that take arguments currently make the space after the short option optional, for example "-X saved.xml" or "-Xsaved.xml". Specifying a short option without a space is now considered deprecated; a future release will print a warning when this is used, and an even later release will drop support.

• OCF resource agents will be displayed as ocf:PROVIDER:AGENT rather than ocf::PROVIDER:AGENT in crm_mon and crm_simulate output and pacemaker-schedulerd logs unless Pacemaker is built with the --enable-compat-2.0 configure option.

• The OCF 1.1 role names ("Promoted" and "Unpromoted") will be used in all tool output instead of the legacy names unless Pacemaker is built with the --enable-compat-2.0 configure option.

crm_resource

• The -a/--constraints and -A/--stack options (to display location and colocation constraints related to a given resource) now have considerably different (and hopefully more readable) text output.

• The constraints created by crm_resource --ban will now use the OCF 1.1 standard role name "Promoted" unless Pacemaker is built with the --enable-compat-2.0 configure option.

• The crm_resource --master option is now deprecated (but still accepted) and replaced with a new --promoted option.

crm_simulate

• Output is now formatted similarly to crm_mon's.

• New --show-attrs option will display node attribute values in simulation output.

• New --show-failcounts option will display resource fail counts in simulation output.

• stderr will no longer automatically be redirected to stdout when --verbose is given unless Pacemaker is built with the --enable-compat-2.0 configure option.

crmadmin

• The -q/--quiet option previously printed its usual output to stdout and a briefer version to stderr. For consistency with other tools, it now prints only the brief version, to stdout.

• The -t/--timeout option, which previously took an integer number of milliseconds, now takes a Pacemaker time specification (bare number of seconds, integer plus units, or ISO 8601 duration) for consistency with the other tools and configuration.

• The text output for the -S/--status option has been changed to be more readable. (Scripts are recommended to use the new XML output support if parsing the output.)

• The long-broken -E/--election and -K/--kill options have been completely removed. (They were only ever useful for debugging during early Pacemaker development.)

Other

• When showing ban constraints, crm_mon --output-as=xml (and --as-xml) will now show promoted-only=true/false in addition to master_only=true/false, which is now deprecated.

• The crm_master command has been deprecated (but still works) and replaced with a new crm_attribute --promotion option that defaults to --lifetime=reboot. This is typically called from within a resource agent. For example, "crm_master -l reboot -v 10" becomes "crm_attribute --promotion -v 10".

• The output of the pacemakerd --features command better reflects the build options currently available. (Feature listings are not guaranteed to stay the same from release to release, and are intended only for troubleshooting.)

Resource agents

• Pacemaker now supports OCF 1.1 compliant resource agents (in particular, reloadable parameters and the reload-agent action).

• Pacemaker now provides resource agents with new environment variables (in addition to the existing ones) for promotable clone notifications, with "master" replaced with "promoted" and "slave" replaced with "unpromoted", for example OCF_RESKEY_CRM_meta_notify_unpromoted_resource will be identical to OCF_RESKEY_CRM_meta_notify_slave_resource. The old names are now deprecated.

• The ocf:pacemaker:Dummy, ocf:pacemaker:Stateful, and ocf:pacemaker:remote resource agents now support OCF 1.1 (reloadable parameters, UI parameter hints, and advertising OCF 1.1 role names in monitor action meta-data).

Documentation changes

• The source for the Pacemaker "books" (Clusters from Scratch, Pacemaker Explained, etc.) has been switched from using asciidoc source and Publican to using reStructuredText source and Sphinx. This is mainly because Publican is a dead project and distributions are dropping it, but it will also give us a cleaner, more modern look.

• The --with-brand configure option has been removed. The books will now be built based on whether the sphinx-build and inkscape commands are available at build time.

• The --with doc RPM build option remains, to decide whether to enable python3-sphinx and inkscape as build requirements for the pacemaker-doc subpackage (effectively controlling whether the books are built or not). The previous build requirement for publican has been removed. (The build requirement of asciidoc or asciidoctor remains, for generating the deprecated text file documentation, but that may change.)

• The directory structure for the generated books has changed:

• Pacemaker uses the OCF 1.1 role names, "Promoted" and "Unpromoted", in the books, as well as tool help and man pages.

Development changes

Public C API changes

If you develop code using the Pacemaker C libraries, these could affect you.

• A new constant, PCMK_ALLOW_DEPRECATED, enables control of API availability. It should normally be left undefined, but if you want to check whether your code uses any deprecated Pacemaker APIs, define it to 0 before including any Pacemaker headers.

• The following APIs have been deprecated and will be removed in a future release (replacements are often not one-to-one, so read their relevant API documentation):

• The following public APIs have changed behavior:
  • libpe_status: resource_object_functions_t parameter() member now ignores its create argument (parameters will always be created if not already available)

• The following public APIs have been added:
  • libcrmcommon
    • CRMD_ACTION_RELOAD_AGENT string constant
    • enum ocf_exitcode (moved from libcrmservice)
    • PCMK_OCF_MAJOR_VERSION string constant
    • PCMK_OCF_MINOR_VERSION string constant
    • PCMK_OCF_RUNNING_PROMOTED enum value
    • PCMK_OCF_VERSION string constant
    • PCMK_XE_PROMOTABLE_LEGACY string constant
    • PCMK_XE_PROMOTED_MAX_LEGACY string constant
    • PCMK_XE_PROMOTED_NODE_MAX_LEGACY string constant
  • libpacemaker
    • pcmk_controller_status()
    • pcmk_designated_controller()
    • pcmk_list_nodes()
    • pcmk_pacemakerd_status()
    • pcmk_resource_digests()
  • libpe_status
    • RSC_ROLE_PROMOTED enum value
    • RSC_ROLE_PROMOTED_LEGACY_S string constant
    • RSC_ROLE_PROMOTED_S string constant
    • RSC_ROLE_UNPROMOTED enum value
    • RSC_ROLE_UNPROMOTED_LEGACY_S string constant
    • RSC_ROLE_UNPROMOTED_S string constant
    • pcmk_working_set_t priv member (for Pacemaker use only)
    • pe_order_promoted_implies_first enum value
    • pe_resource_t parameter_cache member (managed via pe_rsc_params())
    • pe_rsc_params()

Source files moved

If you make changes to the Pacemaker source code, or build from the master branch, these could affect you.

• Certain build artifacts are now created in the libltdl subdirectory instead of the top-level directory. This is uninteresting unless you update an existing source checkout and want to rebuild, in which case you first need to run:

rm -rf aclocal.m4 autom4te.cache config.guess config.log config.status config.sub \
       configure depcomp install-sh libltdl libtool ltmain.sh missing py-compile test-driver
./autogen.sh

• Certain files changed locations in the source tree. Their installed locations did not change.