Page MenuHomeClusterLabs Projects

Pacemaker Release Checklist
Updated 9 Days AgoPublic

Policies

  • Pacemaker's main branch gets all new development first. There are release branches for major-minor series (such as 2.1 and 3.0). Only one or two release branches receive any changes; older release branches do not. The latest release series should be released first, then any supported older series.
  • When the latest series starts a new release cycle, the main branch is pulled into its release branch. This means all main-branch development lands in the latest series' next rc1.
  • If a change is desired in the latest series after an rc1 is released, or in an older series, a PR backporting it to the desired release branch must be submitted separately.
  • After rc1 is released, only bug-fix backports should be merged into the release branch (though exceptions can be made in unusual circumstances). After some later rc (typically around rc3), only regression fixes should be merged.

Testing

For the latest release series, this should be done just before rc1, addressing any issues found. Selected parts can be done for later release candidates or older release series as appropriate. Any fixes needed should be merged into the main branch.

  • Locally generate all documentation that will be uploaded to the website, and proof changes since the last release.
    • To see changes: git diff Pacemaker-$LAST_RELEASE.. -- doc/sphinx
    • Ensure that significant new features are documented in Pacemaker Explained (with "(since <version>)").
    • For a new major series, remove any "(since <version>)" for older versions no longer supported in rolling upgrades.
  • Verify OCF agent meta-data is valid (replacing $OCF_RNG_SCHEMA with the path to ra-api.rng from a local checkout of the OCF-spec source repository):
    • make RNG="$OCF_RNG_SCHEMA" -C agents/ocf validate
  • Verify formatted output args using fosa.
  • Perform static analysis. The devel directory has make targets for convenience (cppcheck, clang, and coverity or coverity-corp).
  • Run all regression tests including cts-cli -t agents, cts-attrd, cts-fencing, cts-exec, and cts-exec -R (see cts/README.md).
  • Run a long CTS lab (see cts/README.md) against a several-node cluster using cts-lab or a wrapper (such as cluster_test or your own).
  • Do some valgrind testing on typical cluster behavior. cts-lab can be used for this, but ignore any test errors, just look at the valgrind results.
  • Test creating, starting, recovering, and stopping a bundle resource.
  • Perform a rolling upgrade from any supported previous version.
  • Optionally, compare scheduler timings against the previous release to make sure there are no large unexplained regressions. From a checkout, run the first command below for the old and new builds, then run the second command to compare the two. Minimize variation however you can (use same host, don't do anything else while running it, maybe try to seed the disk read cache). There still is a good bit of variation from run to run so it is unclear at this point what is significant.
    • PCMK_schema_directory=xml tools/crm_simulate --profile cts/scheduler/xml --repeat 1000 > ~/crm_simulate.$VERSION.out
    • tools/pcmk_simtimes ~/crm_simulate.*.out -s 0.1 -p 5
  • Optionally, check the current buildability in COPR (default sort starts with the newest builds, optionally use pacemaker in the filter box, check at least Git branch field on the details page, optionally also git commit encoded in the R> part of package identifier)

rc1 only

All changes should be made in the main branch for the latest release series, or the release branch for older release series.

  • If this is a new major or minor release, create the release branch, and ensure CI gets updated to check it. Also update the supported series in SECURITY.md, and consider whether the Clusters from Scratch document should be redone with the new series (typically after the initial release).
  • Things that don't need to be done every release but are helpful every now and then (only for the latest release series):
    • cts/cts-scheduler --update (only if the tests already all pass!), to update whitespace usage in expected output (which is ignored by the regression tests, but can help manual comparisons)
    • (Do not do this until we can raise the minimum automake dependency to 1.14) make -C maint gnulib-update to update the code we bundle from gnulib
  • Ensure the feature set is correct.
    • For the latest release series, make sure that it has been bumped appropriately for code changes in this release (the minor version if any new features require DC support including most schema changes, and the minor-minor version if any new features at all have been added including command-line tool arguments).
    • For older release series, ensure it did not change in the release branch (and no features were backported that would require it).
  • Update translation files: make -C po update-po
  • Ensure the XML schema version is correct.
    • For the latest release series, update xml/README.md in the main branch for the Pacemaker-to-schema version mapping if needed. To see schema changes since last release: git log --no-merges Pacemaker-$LAST_VERSION.. xml
    • For older release series, ensure that the schema version did not change.
  • Run maint/bumplibs to update shared library (libtool) versions for changes since last point release, which will show all library diffs since the last release and ask whether the changes involve compatible public API additions, incompatible public API additions/removals, or fixes. Verify the script's changes (which it displays at the end).
    • Incompatible changes should only be made at a major or minor series bump.
    • Versions can be updated again for later release candidates if they break the public API (which should be avoided unless absolutely necessary) by running the script with Pacemaker-$PREVIOUS_RC_TAG as the argument.
    • For details about library versioning, see Updating library version information.
  • Update m4/version.m4 to new release version, without -rc suffix.
    • When bumping the major or minor release number, epub_identifier in doc/sphinx/conf.py.in needs to be updated
    • At some point, it might be worthwhile to include the -rc suffix for better differentiation of release candidates, but that will need a lot of testing to see how it affects RPMs, version comparisons, etc.
  • Merge a pull request for any changes. For the latest release series, pull main into the release branch.

All releases

Repeat until final.

  • Update the Changelog:
    • release candidates: make -C maint LAST_RELEASE=Pacemaker-$LAST_VERSION changelog and then edit manually to make sense from a user perspective (in particular, list all regression fixes at the top of each category with the release the regression was introduced in)
    • final: merge the rc entries into a single entry, then update summary with the output of make -C maint LAST_RELEASE=Pacemaker-$LAST_FINAL_VERSION NEXT_RELEASE=Pacemaker-$NEW_VERSION summary
    • Add the ChangeLog commit to the release branch using "Rebase and merge" (instead of "Create a merge commit"), and if it is for the latest series, forward-port it to the main branch.
  • Make a github release:
    • Go to Pacemaker releases on Github and select Draft a new release
    • Choose a tag: Pacemaker-X.Y.Z-rcN or Pacemaker-X.Y.Z
    • Target: release branch
    • Release title: "Pacemaker X.Y.Z - Release Candidate N" or "Pacemaker X.Y.Z - Final"
    • Describe this release: copy relevant part of ChangeLog
    • This is a pre-release: check if this is an rc
    • Can save as draft to preview if desired
  • If this release fixed any regressions, update the descriptions on GitHub of the releases that introduced the regressions
  • If this is an rc, wait for feedback, and address any reported issues. About 1-4 weeks between release candidates is good.

After final release

  • git fetch upstream --tags so the make targets can use the new release tag instead of commit ID.
  • export RSYNC_RSH="ssh -p $PORT" appropriately for the ClusterLabs website host.
  • Run make -C doc www to regenerate most documentation on ClusterLabs.org website (doxygen, indexed source, Clusters From Scratch, Pacemaker Explained, etc.).
  • Run make -C doc LAST_RELEASE=Pacemaker-$PREVIOUS_RELEASE TAG=Pacemaker-$NEW_RELEASE abi-www to generate the ABI compatibility report.
Last Author
kgaillot
Last Edited
Wed, Jul 3, 12:44 PM

Event Timeline

kgaillot added a project: Restricted Project.
kgaillot edited the content of this document. (Show Details)