Besides a release version, Pacemaker has a separate "CRM feature set". This version number applies to the communication between cluster nodes, and is used to avoid problems in mixed-version clusters.
Most of the time, CRM feature sets are for Pacemaker's internal use. However, they can also be used by resource agents and other external code to detect support for various features. The table below lists those features.
||| Pacemaker CRM Feature Sets |
|---------------------------|-----|----|
| CRM feature set | Available in Pacemaker release | Features
|---------------------------|-----|----|
| 3.19.0 | 2.1.7 | Pacemaker supports CIB transactions and node-pending-timeout cluster property; crm_node supports --output-as (text, none, or xml) and --output-to arguments |
| 3.17.4 | 2.1.6 | utilization attributes may be set as transient; alerts and alert recipients support enabled meta-attribute in CIB; attrd_updater supports --pattern, --utilization, and --wait arguments; crm_attribute supports --wait argument, and supports --pattern argument with --update and --delete and with permanent node attributes; crm_mon supports --interval argument with text and xml output; crm_resource supports --element option; crm_shadow, pacemaker-based, pacemaker-controld, pacemaker-execd, and pacemaker-fenced support --output-as (text, none, or xml) and --output-to arguments |
| 3.16.1 | 2.1.5 | crm_error supports --output-as (text, none, or xml) and --output-to arguments; crm_attribute supports --pattern with --update, and with --delete and --query when used with --lifetime=reboot; crm_attribute with --query and --node but not --name shows all node attributes on the given node |
| 3.15.1 | 2.1.5 | Supported feature set stored as `#feature-set` node attribute |
| 3.15.0 | 2.1.3 | Pacemaker supports setting multiple-active to stop_unexpected |
| 3.14.0 | 2.1.3 | Pacemaker supports allow-unhealthy-nodes resource meta-attribute |
| 3.13.4 | 2.1.3 | attrd_updater supports --output-as (text, none, or xml) and --output-to options |
| 3.13.3 | 2.1.3 | crm_attribute supports --output-as (text, none, or xml) and --output-to options |
| 3.13.2 | 2.1.3 | crm_rule supports --output-as (text, none, or xml) and --output-to options, and specifying -r option more than once |
| 3.13.1 | 2.1.3 | cibadmin supports --show-access option |
| 3.11.0 | 2.1.2 | fence_watchdog is available (allowing watchdog fencing to be restricted to specific nodes) |
| 3.10.2 | 2.1.1 | crm_resource --validate and --force-check arguments accept an optional argument for OCF check level |
| 3.10.1 | 2.1.1 | pacemakerd accepts --output-as (text, none, or xml) and --output-to options, mainly for use with the --features argument |
| 3.10.0 | 2.1.0 | Pacemaker supports OCF resource agents' reload-agent action |
| 3.9.0 | 2.1.0 | "Promoted" and "Unpromoted" are accepted as role names; crm_attribute accepts the --promotion option; crm_resource accepts the --promoted option; promotable clone notify actions will have notify_promoted_resource, notify_unpromoted_resource, notify_promoted_uname, and notify_unpromoted_uname environment variables; crm_mon XML output for bans includes a "promoted-only" attribute |
| 3.8.0 | 2.1.0 | crm_simulate accepts --show-attrs and --show-failcounts options |
| 3.7.4 | 2.1.0 | crm_simulate accepts --output-as (text, none, or xml) and --output-to options, and its output is formatted more similarly to crm_mon's |
| 3.7.3 | 2.1.0 | CIB_file="-" for standard input is supported by tools that support CIB_file |
| 3.7.2 | 2.1.0 | crm_verify accepts --output-as (text, none, or xml) and --output-to options; crmadmin drops -E/--election and -K/--kill options |
| 3.7.1 | 2.1.0 | support for node type argument to crmadmin --nodes |
| 3.7.0 | 2.1.0 | support for "critical" resource meta-attribute and "influence" colocation option |
| 3.6.4 | 2.1.0 | crm_resource offers --digests option |
| 3.6.3 | 2.1.0 | crm_resource accepts --output-as (text or xml) and --output-to |
| 3.6.2 | 2.1.0 | crmadmin accepts --output-as (text or xml) and --output-to |
| 3.6.1 | 2.0.5 | cibsecret returns standard Pacemaker exit status codes |
| 3.6.0 | 2.0.5 | |
| 3.5.0 | 2.0.5 | "integer" supported as value for "type" in rules, and "number" compares as double-precision floating-point value |
| 3.4.1 | 2.0.5 | crmadmin supports -P/--pacemakerd and -i/--ipc-name options |
| 3.4.0 | 2.0.5 | op_expression and rsc_expression supported in rules; "demote" supported as value for "no-quorum-policy" and "on-fail" options; priority-fencing-delay cluster option supported; stonith_admin supports --delay option; crm_mon supports --node and --resource options; crm_rule --check can now check "date_spec" entries that contain "years" but not "moon" |
| 3.3.0 | 2.0.4 | fence-reaction, shutdown-lock, and shutdown-lock-limit cluster options supported; crm_mon supports --include and --exclude options |
| 3.2.0 | 2.0.3 | new tool crm_rule; crm_mon and stonith_admin support --output-as and --output-to options |
| 3.1.0 | 2.0.0 | |
(Note: in some cases, the specified features were available in the prior Pacemaker release, but the listed feature set version is the first that includes them.)
== How CRM feature sets are numbered ==
The major version increases when nodes with different versions would not work together. Rolling upgrades would not be possible in this case.
The minor version increases when mixed-version clusters are allowed only during rolling upgrades. If one of the older nodes drops out of the cluster for any reason, it will not be able to rejoin until it is upgraded. In this case, you should run a mixed-version cluster only during a small upgrade window.
The minor-minor version number is ignored by Pacemaker. Clusters with different versions should work without problems. This number allows resource agents and other external code to detect cluster support for various features.
Pacemaker ensures that the longest-running node is the cluster’s DC. This ensures new features are not enabled until all nodes are upgraded to support them.
== In code ==
The feature set is defined as `CRM_FEATURE_SET` in `include/crm/crm.h`.