Page Menu
Home
ClusterLabs Projects
Search
Configure Global Search
Log In
Files
F4638937
No One
Temporary
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Flag For Later
Award Token
Size
190 KB
Referenced Files
None
Subscribers
None
View Options
diff --git a/cts/README.md b/cts/README.md
index ef7b46117c..999131dc26 100644
--- a/cts/README.md
+++ b/cts/README.md
@@ -1,315 +1,315 @@
# Pacemaker Cluster Test Suite (CTS)
The Cluster Test Suite (CTS) refers to all Pacemaker testing code that can be
run in an installed environment. (Pacemaker also has unit tests that must be
run from a source distribution.)
CTS includes:
* Regression tests: These test specific Pacemaker components individually (no
integration tests). The primary front end is cts-regression in this
directory. Run it with the --help option to see its usage.
cts-regression is a wrapper for individual component regression tests also
in this directory (cts-cli, cts-exec, cts-fencing, and cts-scheduler).
The CLI and scheduler regression tests can also be run from a source
distribution. The other regression tests can only run in an installed
environment, and the cluster should not be running on the node running these
tests.
* The CTS lab: This is a cluster exerciser for intensively testing the behavior
of an entire working cluster. It is primarily for developers and packagers of
the Pacemaker source code, but it can be useful for users who wish to see how
their cluster will react to various situations. Most of the lab code is in
the Pacemaker Python module. The front end, cts-lab, is in this directory.
The CTS lab runs a randomized series of predefined tests on the cluster. It
can be run against a pre-existing cluster configuration or overwrite the
existing configuration with a test configuration.
* Helpers: Some of the component regression tests and the CTS lab require
certain helpers to be installed as root. These include a dummy LSB init
script, dummy systemd service, etc. In a source distribution, the source for
these is in cts/support.
The tests will install these as needed and uninstall them when done. This
means that the cluster configuration created by the CTS lab will generate
failures if started manually after the lab exits. However, the helper
installer can be run manually to make the configuration usable, if you want
to do your own further testing with it:
/usr/libexec/pacemaker/cts-support install
As you might expect, you can also remove the helpers with:
/usr/libexec/pacemaker/cts-support uninstall
(The actual directory location may vary depending on how Pacemaker was
built.)
* Cluster benchmark: The benchmark subdirectory of this directory contains some
cluster test environment benchmarking code. It is not particularly useful for
end users.
* Valgrind suppressions: When memory-testing Pacemaker code with valgrind,
various bugs in non-Pacemaker libraries and such can clutter the results. The
valgrind-pcmk.suppressions file in this directory can be used with valgrind's
--suppressions option to eliminate many of these.
## Using the CTS lab
### Requirements
* Three or more machines (one test exerciser and at least two cluster nodes).
* The test cluster nodes should be on the same subnet and have journalling
filesystems (ext4, xfs, etc.) for all of their filesystems other than
/boot. You also need a number of free IP addresses on that subnet if you
intend to test IP address takeover.
* The test exerciser machine doesn't need to be on the same subnet as the test
cluster machines. Minimal demands are made on the exerciser; it just has to
stay up during the tests.
* Tracking problems is easier if all machines' clocks are closely synchronized.
NTP does this automatically, but you can do it by hand if you want.
* The account on the exerciser used to run the CTS lab (which does not need to
be root) must be able to ssh as root to the cluster nodes without a password
challenge. See the Mini-HOWTO at the end of this file for details about how
to configure ssh for this.
* The exerciser needs to be able to resolve all cluster node names, whether by
DNS or /etc/hosts.
* CTS is not guaranteed to run on all platforms that Pacemaker itself does.
It calls commands such as service that may not be provided by all OSes.
### Preparation
* Install Pacemaker, including the testing code, on all machines. The testing
code must be the same version as the rest of Pacemaker, and the Pacemaker
version must be the same on the exerciser and all cluster nodes.
You can install from source, although many distributions package the testing
code (named pacemaker-cts or similar). Typically, everything needed by the
CTS lab is installed in /usr/share/pacemaker/tests/cts.
* Configure the cluster layer (Corosync) on the cluster machines (*not* the
exerciser), and verify it works. Node names used in the cluster configuration
*must* match the hosts' names as returned by `uname -n`; they do not have to
match the machines' fully qualified domain names.
* Optionally, configure the exerciser as a log aggregator, using something like
`rsyslog` log forwarding. If aggregation is detected, the exerciser will look
for new messages locally instead of requesting them repeatedly from cluster
nodes.
* Currently, `/var/log/messages` on the exerciser is the only supported log
destination. Further, if it's specified explicitly on the command line as
the log file, then CTS lab will not check for aggregation.
* CTS lab does not currently detect systemd journal log aggregation.
* Optionally, if the lab nodes use the systemd journal for logs, create
/etc/systemd/journald.conf.d/cts-lab.conf on each with
`RateLimitIntervalSec=0` or `RateLimitBurst=0`, to avoid issues with log
detection.
### Run
The primary interface to the CTS lab is the cts-lab executable:
/usr/share/pacemaker/tests/cts-lab [options] <number-of-tests-to-run>
(The actual directory location may vary depending on how Pacemaker was built.)
As part of the options, specify the cluster nodes with --nodes, for example:
--nodes "pcmk-1 pcmk-2 pcmk-3"
Most people will want to save the output to a file, for example:
--outputfile ~/cts.log
Unless you want to test a pre-existing cluster configuration, you also want
(*warning*: with these options, any existing configuration will be lost):
--clobber-cib
--populate-resources
You can test floating IP addresses (*not* already used by any host), one per
cluster node, by specifying the first, for example:
--test-ip-base 192.168.9.100
Configure some sort of fencing, for example to use fence\_xvm:
--stonith xvm
Putting all the above together, a command line might look like:
/usr/share/pacemaker/tests/cts-lab --nodes "pcmk-1 pcmk-2 pcmk-3" \
--outputfile ~/cts.log --clobber-cib --populate-resources \
--test-ip-base 192.168.9.100 --stonith xvm 50
For more options, run with the --help option.
There are also a couple of wrappers for cts-lab that some users may find more
convenient: cts, which is typically installed in the same place as the rest of
the testing code; and cluster\_test, which is in the source directory and
typically not installed.
To extract the result of a particular test, run:
crm_report -T $test
### Optional: Memory testing
Pacemaker has various options for testing memory management. On cluster nodes,
Pacemaker components use various environment variables to control these
options. How these variables are set varies by OS, but usually they are set in
a file such as /etc/sysconfig/pacemaker or /etc/default/pacemaker.
Valgrind is a program for detecting memory management problems such as
use-after-free errors. If you have valgrind installed, you can enable it by
setting the following environment variables on all cluster nodes:
PCMK_valgrind_enabled=pacemaker-attrd,pacemaker-based,pacemaker-controld,pacemaker-execd,pacemaker-fenced,pacemaker-schedulerd
VALGRIND_OPTS="--leak-check=full --trace-children=no --num-callers=25
--log-file=/var/lib/pacemaker/valgrind-%p
--suppressions=/usr/share/pacemaker/tests/valgrind-pcmk.suppressions
--gen-suppressions=all"
If running the CTS lab with valgrind enabled on the cluster nodes, add these
options to cts-lab:
- --valgrind-tests --valgrind-procs "pacemaker-attrd pacemaker-based pacemaker-controld pacemaker-execd pacemaker-schedulerd pacemaker-fenced"
+ --valgrind-procs "pacemaker-attrd pacemaker-based pacemaker-controld pacemaker-execd pacemaker-schedulerd pacemaker-fenced"
These options should only be set while specifically testing memory management,
because they may slow down the cluster significantly, and they will disable
writes to the CIB. If desired, you can enable valgrind on a subset of pacemaker
components rather than all of them as listed above.
Valgrind will put a text file for each process in the location specified by
valgrind's --log-file option. See
https://www.valgrind.org/docs/manual/mc-manual.html for explanations of the
messages valgrind generates.
Separately, if you are using the GNU C library, the G\_SLICE,
MALLOC\_PERTURB\_, and MALLOC\_CHECK\_ environment variables can be set to
affect the library's memory management functions.
When using valgrind, G\_SLICE should be set to "always-malloc", which helps
valgrind track memory by always using the malloc() and free() routines
directly. When not using valgrind, G\_SLICE can be left unset, or set to
"debug-blocks", which enables the C library to catch many memory errors
but may impact performance.
If the MALLOC\_PERTURB\_ environment variable is set to an 8-bit integer, the C
library will initialize all newly allocated bytes of memory to the integer
value, and will set all newly freed bytes of memory to the bitwise inverse of
the integer value. This helps catch uses of uninitialized or freed memory
blocks that might otherwise go unnoticed. Example:
MALLOC_PERTURB_=221
If the MALLOC\_CHECK\_ environment variable is set, the C library will check for
certain heap corruption errors. The most useful value in testing is 3, which
will cause the library to print a message to stderr and abort execution.
Example:
MALLOC_CHECK_=3
Valgrind should be enabled for either all nodes or none when used with the CTS
lab, but the C library variables may be set differently on different nodes.
### Optional: Remote node testing
If the pacemaker-remoted daemon is installed on all cluster nodes, the CTS lab
will enable remote node tests.
The remote node tests choose a random node, stop the cluster on it, start
pacemaker-remoted on it, and add an ocf:pacemaker:remote resource to turn it
into a remote node. When the test is done, the lab will turn the node back into
a cluster node.
To avoid conflicts, the lab will rename the node, prefixing the original node
name with "remote-". For example, "pcmk-1" will become "remote-pcmk-1". These
names do not need to be resolvable.
The name change may require special fencing configuration, if the fence agent
expects the node name to be the same as its hostname. A common approach is to
specify the "remote-" names in pcmk\_host\_list. If you use
pcmk\_host\_list=all, the lab will expand that to all cluster nodes and their
"remote-" names. You may additionally need a pcmk\_host\_map argument to map
the "remote-" names to the hostnames. Example:
--stonith xvm --stonith-args \
pcmk_host_list=all,pcmk_host_map=remote-pcmk-1:pcmk-1;remote-pcmk-2:pcmk-2
### Optional: Remote node testing with valgrind
When running the remote node tests, the Pacemaker components on the *cluster*
nodes can be run under valgrind as described in the "Memory testing" section.
However, pacemaker-remoted cannot be run under valgrind that way, because it is
started by the OS's regular boot system and not by Pacemaker.
Details vary by system, but the goal is to set the VALGRIND\_OPTS environment
variable and then start pacemaker-remoted by prefixing it with the path to
valgrind.
The init script and systemd service file provided with pacemaker-remoted will
load the pacemaker environment variables from the same location used by other
Pacemaker components, so VALGRIND\_OPTS will be set correctly if using one of
those.
For an OS using systemd, you can override the ExecStart parameter to run
valgrind. For example:
mkdir /etc/systemd/system/pacemaker_remote.service.d
cat >/etc/systemd/system/pacemaker_remote.service.d/valgrind.conf <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/valgrind /usr/sbin/pacemaker-remoted
EOF
### Mini-HOWTO: Allow passwordless remote SSH connections
The CTS lab runs "ssh -l root" so you don't have to do any of your testing
logged in as root on the exerciser. Here is how to allow such connections
without requiring a password to be entered each time:
* On your test exerciser, create an SSH key if you do not already have one.
Most commonly, SSH keys will be in your ~/.ssh directory, with the
private key file not having an extension, and the public key file
named the same with the extension ".pub" (for example, ~/.ssh/id\_rsa.pub).
If you don't already have a key, you can create one with:
ssh-keygen -t rsa
* From your test exerciser, authorize your SSH public key for root on all test
machines (both the exerciser and the cluster test machines):
ssh-copy-id -i ~/.ssh/id_rsa.pub root@$MACHINE
You will probably have to provide your password, and possibly say
"yes" to some questions about accepting the identity of the test machines.
The above assumes you have a RSA SSH key in the specified location;
if you have some other type of key (DSA, ECDSA, etc.), use its file name
in the -i option above.
* To verify, try this command from the exerciser machine for each
of your cluster machines, and for the exerciser machine itself.
ssh -l root $MACHINE
If this works without prompting for a password, you're in business.
If not, look at the documentation for your version of ssh.
diff --git a/cts/cts-scheduler.in b/cts/cts-scheduler.in
index f5d4ed5c38..e505d3e989 100644
--- a/cts/cts-scheduler.in
+++ b/cts/cts-scheduler.in
@@ -1,1737 +1,1737 @@
#!@PYTHON@
"""Regression tests for Pacemaker's scheduler."""
# pylint doesn't like the module name "cts-scheduler" which is an invalid complaint for this file
# but probably something we want to continue warning about elsewhere
# pylint: disable=invalid-name
# pacemaker imports need to come after we modify sys.path, which pylint will complain about.
# pylint: disable=wrong-import-position
__copyright__ = "Copyright 2004-2025 the Pacemaker project contributors"
__license__ = "GNU General Public License version 2 or later (GPLv2+) WITHOUT ANY WARRANTY"
import io
import os
import re
import sys
import stat
import shlex
import shutil
import argparse
import subprocess
import platform
import tempfile
# These imports allow running from a source checkout after running `make`.
# Note that while this doesn't necessarily mean it will successfully run tests,
# but being able to see --help output can be useful.
if os.path.exists("@abs_top_srcdir@/python"):
sys.path.insert(0, "@abs_top_srcdir@/python")
# pylint: disable=comparison-of-constants,comparison-with-itself,condition-evals-to-constant
if os.path.exists("@abs_top_builddir@/python") and "@abs_top_builddir@" != "@abs_top_srcdir@":
sys.path.insert(0, "@abs_top_builddir@/python")
from pacemaker.buildoptions import BuildOptions
from pacemaker.exitstatus import ExitStatus
class SchedulerTest:
"""A single scheduler test."""
def __init__(self, name, desc, args=None):
"""
Create a new SchedulerTest instance.
Arguments:
name -- A unique name for this test.
desc -- A meaningful description for the test.
args -- Additional arguments to pass when running this test
"""
self.name = name
self.desc = desc
if args is None:
self.args = []
else:
self.args = args
class SchedulerTestGroup:
"""Collection of scheduler regression tests."""
def __init__(self, tests):
"""
Create a new SchedulerTestGroup instance.
Arguments:
tests -- A list of SchedulerTest instances to be executed as part of
this group.
"""
self.tests = tests
# Each entry in TESTS is a group of tests, where each test consists of a
# test base name, test description, and additional test arguments.
# Test groups will be separated by newlines in output.
TESTS = [
SchedulerTestGroup([
SchedulerTest("simple1", "Offline"),
SchedulerTest("simple2", "Start"),
SchedulerTest("simple3", "Start 2"),
SchedulerTest("simple4", "Start Failed"),
SchedulerTest("simple6", "Stop Start"),
SchedulerTest("simple7", "Shutdown"),
SchedulerTest("simple11", "Priority (ne)"),
SchedulerTest("simple12", "Priority (eq)"),
SchedulerTest("simple8", "Stickiness"),
]),
SchedulerTestGroup([
SchedulerTest("group1", "Group"),
SchedulerTest("group2", "Group + Native"),
SchedulerTest("group3", "Group + Group"),
SchedulerTest("group4", "Group + Native (nothing)"),
SchedulerTest("group5", "Group + Native (move)"),
SchedulerTest("group6", "Group + Group (move)"),
SchedulerTest("group7", "Group colocation"),
SchedulerTest("group13", "Group colocation (cant run)"),
SchedulerTest("group8", "Group anti-colocation"),
SchedulerTest("group9", "Group recovery"),
SchedulerTest("group10", "Group partial recovery"),
SchedulerTest("group11", "Group target_role"),
SchedulerTest("group14", "Group stop (graph terminated)"),
SchedulerTest("group15", "Negative group colocation"),
SchedulerTest("bug-1573", "Partial stop of a group with two children"),
SchedulerTest("bug-1718", "Mandatory group ordering - Stop group_FUN"),
SchedulerTest("failed-sticky-group", "Move group on last member failure despite infinite stickiness"),
SchedulerTest("failed-sticky-anticolocated-group",
"Move group on last member failure despite infinite stickiness and optional anti-colocation"),
SchedulerTest("bug-lf-2619", "Move group on clone failure"),
SchedulerTest("group-fail", "Ensure stop order is preserved for partially active groups"),
SchedulerTest("group-unmanaged", "No need to restart r115 because r114 is unmanaged"),
SchedulerTest("group-unmanaged-stopped", "Make sure r115 is stopped when r114 fails"),
SchedulerTest("partial-unmanaged-group", "New member in partially unmanaged group"),
SchedulerTest("group-dependents", "Account for the location preferences of things colocated with a group"),
SchedulerTest("group-stop-ordering", "Ensure blocked group member stop does not force other member stops"),
SchedulerTest("colocate-unmanaged-group", "Respect mandatory colocations even if earlier group member is unmanaged"),
SchedulerTest("coloc-with-inner-group-member", "Consider explicit colocations with inner group members"),
SchedulerTest("banned-group-inner-constraints",
"Group banned from current node, inner member constrained"),
]),
SchedulerTestGroup([
SchedulerTest("rsc_dep1", "Must not"),
SchedulerTest("rsc_dep3", "Must"),
SchedulerTest("rsc_dep5", "Must not 3"),
SchedulerTest("rsc_dep7", "Must 3"),
SchedulerTest("rsc_dep10", "Must (but cant)"),
SchedulerTest("rsc_dep2", "Must (running)"),
SchedulerTest("rsc_dep8", "Must (running : alt)"),
SchedulerTest("rsc_dep4", "Must (running + move)"),
SchedulerTest("asymmetric", "Asymmetric - require explicit location constraints"),
]),
SchedulerTestGroup([
SchedulerTest("orphan-0", "Orphan ignore"),
SchedulerTest("orphan-1", "Orphan stop"),
SchedulerTest("orphan-2", "Orphan stop, remove failcount"),
]),
SchedulerTestGroup([
SchedulerTest("params-0", "Params: No change"),
SchedulerTest("params-1", "Params: Changed"),
SchedulerTest("params-2", "Params: Resource definition"),
SchedulerTest("params-3", "Params: Restart instead of reload if start pending"),
SchedulerTest("params-4", "Params: Reload"),
SchedulerTest("params-5", "Params: Restart based on probe digest"),
SchedulerTest("novell-251689", "Resource definition change + target_role=stopped"),
SchedulerTest("bug-lf-2106", "Restart all anonymous clone instances after config change"),
SchedulerTest("params-6", "Params: Detect reload in previously migrated resource"),
SchedulerTest("nvpair-id-ref", "Support id-ref in nvpair with optional name"),
SchedulerTest("not-reschedule-unneeded-monitor",
"Do not reschedule unneeded monitors while resource definitions have changed"),
SchedulerTest("reload-becomes-restart", "Cancel reload if restart becomes required"),
SchedulerTest("restart-with-extra-op-params", "Restart if with extra operation parameters upon changes of any"),
]),
SchedulerTestGroup([
SchedulerTest("target-0", "Target Role : baseline"),
SchedulerTest("target-1", "Target Role : promoted"),
SchedulerTest("target-2", "Target Role : invalid"),
]),
SchedulerTestGroup([
SchedulerTest("base-score", "Set a node's default score for all nodes"),
]),
SchedulerTestGroup([
SchedulerTest("date-1", "Dates", ["-t", "2005-020"]),
SchedulerTest("date-2", "Date Spec - Pass", ["-t", "2005-020T12:30"]),
SchedulerTest("date-3", "Date Spec - Fail", ["-t", "2005-020T11:30"]),
SchedulerTest("origin", "Timing of recurring operations", ["-t", "2014-05-07 00:28:00"]),
SchedulerTest("probe-0", "Probe (anon clone)"),
SchedulerTest("probe-1", "Pending Probe"),
SchedulerTest("probe-2", "Correctly re-probe cloned groups"),
SchedulerTest("probe-3", "Probe (pending node)"),
SchedulerTest("probe-4", "Probe (pending node + stopped resource)"),
SchedulerTest("probe-pending-node", "Probe (pending node + unmanaged resource)"),
SchedulerTest("failed-probe-primitive", "Maskable vs. unmaskable probe failures on primitive resources"),
SchedulerTest("failed-probe-clone", "Maskable vs. unmaskable probe failures on cloned resources"),
SchedulerTest("expired-failed-probe-primitive", "Maskable, expired probe failure on primitive resources"),
SchedulerTest("standby", "Standby"),
SchedulerTest("comments", "Comments"),
]),
SchedulerTestGroup([
SchedulerTest("one-or-more-0", "Everything starts"),
SchedulerTest("one-or-more-1", "Nothing starts because of A"),
SchedulerTest("one-or-more-2", "D can start because of C"),
SchedulerTest("one-or-more-3", "D cannot start because of B and C"),
SchedulerTest("one-or-more-4", "D cannot start because of target-role"),
SchedulerTest("one-or-more-5", "Start A and F even though C and D are stopped"),
SchedulerTest("one-or-more-6", "Leave A running even though B is stopped"),
SchedulerTest("one-or-more-7", "Leave A running even though C is stopped"),
SchedulerTest("bug-5140-require-all-false", "Allow basegrp:0 to stop"),
SchedulerTest("clone-require-all-1", "clone B starts node 3 and 4"),
SchedulerTest("clone-require-all-2", "clone B remains stopped everywhere"),
SchedulerTest("clone-require-all-3", "clone B stops everywhere because A stops everywhere"),
SchedulerTest("clone-require-all-4", "clone B remains on node 3 and 4 with only one instance of A remaining"),
SchedulerTest("clone-require-all-5", "clone B starts on node 1 3 and 4"),
SchedulerTest("clone-require-all-6", "clone B remains active after shutting down instances of A"),
SchedulerTest("clone-require-all-7",
"clone A and B both start at the same time. all instances of A start before B"),
SchedulerTest("clone-require-all-no-interleave-1", "C starts everywhere after A and B"),
SchedulerTest("clone-require-all-no-interleave-2",
"C starts on nodes 1, 2, and 4 with only one active instance of B"),
SchedulerTest("clone-require-all-no-interleave-3",
"C remains active when instance of B is stopped on one node and started on another"),
SchedulerTest("one-or-more-unrunnable-instances", "Avoid dependencies on instances that won't ever be started"),
]),
SchedulerTestGroup([
SchedulerTest("location-date-rules-1", "Use location constraints with ineffective date-based rules"),
SchedulerTest("location-date-rules-2", "Use location constraints with effective date-based rules"),
SchedulerTest("nvpair-date-rules-1", "Use nvpair blocks with a variety of date-based rules"),
SchedulerTest("value-source", "Use location constraints with node attribute expressions using value-source"),
SchedulerTest("rule-dbl-as-auto-number-match",
"Floating-point rule values default to number comparison: match"),
SchedulerTest("rule-dbl-as-auto-number-no-match",
"Floating-point rule values default to number comparison: no match"),
SchedulerTest("rule-dbl-as-integer-match",
"Floating-point rule values set to integer comparison: match"),
SchedulerTest("rule-dbl-as-integer-no-match",
"Floating-point rule values set to integer comparison: no match"),
SchedulerTest("rule-dbl-as-number-match",
"Floating-point rule values set to number comparison: match"),
SchedulerTest("rule-dbl-as-number-no-match",
"Floating-point rule values set to number comparison: no match"),
SchedulerTest("rule-dbl-parse-fail-default-str-match",
"Floating-point rule values fail to parse, default to string comparison: match"),
SchedulerTest("rule-dbl-parse-fail-default-str-no-match",
"Floating-point rule values fail to parse, default to string comparison: no match"),
SchedulerTest("rule-int-as-auto-integer-match",
"Integer rule values default to integer comparison: match"),
SchedulerTest("rule-int-as-auto-integer-no-match",
"Integer rule values default to integer comparison: no match"),
SchedulerTest("rule-int-as-integer-match",
"Integer rule values set to integer comparison: match"),
SchedulerTest("rule-int-as-integer-no-match",
"Integer rule values set to integer comparison: no match"),
SchedulerTest("rule-int-as-number-match",
"Integer rule values set to number comparison: match"),
SchedulerTest("rule-int-as-number-no-match",
"Integer rule values set to number comparison: no match"),
SchedulerTest("rule-int-parse-fail-default-str-match",
"Integer rule values fail to parse, default to string comparison: match"),
SchedulerTest("rule-int-parse-fail-default-str-no-match",
"Integer rule values fail to parse, default to string comparison: no match"),
]),
SchedulerTestGroup([
SchedulerTest("order1", "Order start 1"),
SchedulerTest("order2", "Order start 2"),
SchedulerTest("order3", "Order stop"),
SchedulerTest("order4", "Order (multiple)"),
SchedulerTest("order5", "Order (move)"),
SchedulerTest("order6", "Order (move w/ restart)"),
SchedulerTest("order7", "Order (mandatory)"),
SchedulerTest("order-optional", "Order (score=0)"),
SchedulerTest("order-required", "Order (score=INFINITY)"),
SchedulerTest("bug-lf-2171", "Prevent group start when clone is stopped"),
SchedulerTest("order-clone", "Clone ordering should be able to prevent startup of dependent clones"),
SchedulerTest("order-sets", "Ordering for resource sets"),
SchedulerTest("order-serialize", "Serialize resources without inhibiting migration"),
SchedulerTest("order-serialize-set", "Serialize a set of resources without inhibiting migration"),
SchedulerTest("clone-order-primitive", "Order clone start after a primitive"),
SchedulerTest("clone-order-16instances", "Verify ordering of 16 cloned resources"),
SchedulerTest("order-optional-keyword", "Order (optional keyword)"),
SchedulerTest("order-mandatory", "Order (mandatory keyword)"),
SchedulerTest("bug-lf-2493", "Don't imply colocation requirements when applying ordering constraints with clones"),
SchedulerTest("ordered-set-basic-startup", "Constraint set with default order settings"),
SchedulerTest("ordered-set-natural", "Allow natural set ordering"),
SchedulerTest("order-wrong-kind", "Order (error)"),
]),
SchedulerTestGroup([
SchedulerTest("coloc-loop", "Colocation - loop"),
SchedulerTest("coloc-many-one", "Colocation - many-to-one"),
SchedulerTest("coloc-list", "Colocation - many-to-one with list"),
SchedulerTest("coloc-group", "Colocation - groups"),
SchedulerTest("coloc-unpromoted-anti", "Anti-colocation with unpromoted shouldn't prevent promoted colocation"),
SchedulerTest("coloc-attr", "Colocation based on node attributes"),
SchedulerTest("coloc-negative-group", "Negative colocation with a group"),
SchedulerTest("coloc-intra-set", "Intra-set colocation"),
SchedulerTest("bug-lf-2435", "Colocation sets with a negative score"),
SchedulerTest("coloc-clone-stays-active",
"Ensure clones don't get stopped/demoted because a dependent must stop"),
SchedulerTest("coloc_fp_logic", "Verify floating point calculations in colocation are working"),
SchedulerTest("colo_promoted_w_native",
"cl#5070 - Verify promotion order is affected when colocating promoted with primitive"),
SchedulerTest("colo_unpromoted_w_native",
"cl#5070 - Verify promotion order is affected when colocating unpromoted with primitive"),
SchedulerTest("anti-colocation-order",
"cl#5187 - Prevent resources in an anti-colocation from even temporarily running on a same node"),
SchedulerTest("anti-colocation-promoted", "Organize order of actions for promoted resources in anti-colocations"),
SchedulerTest("anti-colocation-unpromoted", "Organize order of actions for unpromoted resources in anti-colocations"),
SchedulerTest("group-anticolocation", "Group with failed last member anti-colocated with another group"),
SchedulerTest("group-anticolocation-2",
"Group with failed last member anti-colocated with another sticky group"),
SchedulerTest("group-anticolocation-3",
"Group with failed last member mandatorily anti-colocated with another group"),
SchedulerTest("group-anticolocation-4",
"Group with failed last member anti-colocated without influence with another group"),
SchedulerTest("group-anticolocation-5",
"Group with failed last member anti-colocated with another group (third node allowed)"),
SchedulerTest("group-colocation-failure",
"Group with sole member failed, colocated with another group"),
SchedulerTest("enforce-colo1", "Always enforce B with A INFINITY"),
SchedulerTest("complex_enforce_colo", "Always enforce B with A INFINITY. (make sure heat-engine stops)"),
SchedulerTest("coloc-dependee-should-stay", "Stickiness outweighs group colocation"),
SchedulerTest("coloc-dependee-should-move", "Group colocation outweighs stickiness"),
SchedulerTest("colocation-influence", "Respect colocation influence"),
SchedulerTest("colocation-priority-group", "Apply group colocations in order of primary priority"),
SchedulerTest("colocation-vs-stickiness", "Group stickiness outweighs anti-colocation score"),
SchedulerTest("promoted-with-blocked", "Promoted role colocated with a resource with blocked start"),
SchedulerTest("primitive-with-group-with-clone",
"Consider group dependent when colocating with clone"),
SchedulerTest("primitive-with-group-with-promoted",
"Consider group dependent when colocating with promoted role"),
SchedulerTest("primitive-with-unrunnable-group",
"Block primitive colocated with group that can't start"),
SchedulerTest("coloc-cloned-group-promoted-dependent1",
"Cloned group promoted role with primitive (mandatory)"),
SchedulerTest("coloc-cloned-group-promoted-dependent2",
"Cloned group promoted role with primitive (optional)"),
SchedulerTest("coloc-optional-promoted-dependent-moves-1",
"Colocation score less than promotion score difference: move"),
SchedulerTest("coloc-optional-promoted-dependent-moves-2",
"Colocation score greater than promotion score difference: move"),
SchedulerTest("coloc-optional-promoted-dependent-stays-1",
"Colocation score greater than promotion score difference: stay"),
SchedulerTest("coloc-optional-promoted-dependent-stays-2",
"Colocation score less than promotion score difference: stay"),
]),
SchedulerTestGroup([
SchedulerTest("rsc-sets-seq-true", "Resource Sets - sequential=false"),
SchedulerTest("rsc-sets-seq-false", "Resource Sets - sequential=true"),
SchedulerTest("rsc-sets-clone", "Resource Sets - Clone"),
SchedulerTest("rsc-sets-promoted", "Resource Sets - Promoted"),
SchedulerTest("rsc-sets-clone-1", "Resource Sets - Clone (lf#2404)"),
]),
SchedulerTestGroup([
SchedulerTest("attrs1", "string: eq (and)"),
SchedulerTest("attrs2", "string: lt / gt (and)"),
SchedulerTest("attrs3", "string: ne (or)"),
SchedulerTest("attrs4", "string: exists"),
SchedulerTest("attrs5", "string: not_exists"),
SchedulerTest("attrs6", "is_dc: true"),
SchedulerTest("attrs7", "is_dc: false"),
SchedulerTest("attrs8", "score_attribute"),
SchedulerTest("per-node-attrs", "Per node resource parameters"),
]),
SchedulerTestGroup([
SchedulerTest("mon-rsc-1", "Schedule Monitor - start"),
SchedulerTest("mon-rsc-2", "Schedule Monitor - move"),
SchedulerTest("mon-rsc-3", "Schedule Monitor - pending start"),
SchedulerTest("mon-rsc-4", "Schedule Monitor - move/pending start"),
]),
SchedulerTestGroup([
SchedulerTest("rec-rsc-0", "Resource Recover - no start"),
SchedulerTest("rec-rsc-1", "Resource Recover - start"),
SchedulerTest("rec-rsc-2", "Resource Recover - monitor"),
SchedulerTest("rec-rsc-3", "Resource Recover - stop - ignore"),
SchedulerTest("rec-rsc-4", "Resource Recover - stop - block"),
SchedulerTest("rec-rsc-5", "Resource Recover - stop - fence"),
SchedulerTest("rec-rsc-6", "Resource Recover - multiple - restart"),
SchedulerTest("rec-rsc-7", "Resource Recover - multiple - stop"),
SchedulerTest("rec-rsc-8", "Resource Recover - multiple - block"),
SchedulerTest("rec-rsc-9", "Resource Recover - group/group"),
SchedulerTest("stop-unexpected", "Recover multiply active group with stop_unexpected"),
SchedulerTest("stop-unexpected-2", "Resource multiply active primitve with stop_unexpected"),
SchedulerTest("monitor-recovery", "on-fail=block + resource recovery detected by recurring monitor"),
SchedulerTest("stop-failure-no-quorum", "Stop failure without quorum"),
SchedulerTest("stop-failure-no-fencing", "Stop failure without fencing available"),
SchedulerTest("stop-failure-with-fencing", "Stop failure with fencing available"),
SchedulerTest("multiple-active-block-group", "Support of multiple-active=block for resource groups"),
SchedulerTest("multiple-monitor-one-failed",
"Consider resource failed if any of the configured monitor operations failed"),
]),
SchedulerTestGroup([
SchedulerTest("quorum-1", "No quorum - ignore"),
SchedulerTest("quorum-2", "No quorum - freeze"),
SchedulerTest("quorum-3", "No quorum - stop"),
SchedulerTest("quorum-4", "No quorum - start anyway"),
SchedulerTest("quorum-5", "No quorum - start anyway (group)"),
SchedulerTest("quorum-6", "No quorum - start anyway (clone)"),
SchedulerTest("bug-cl-5212", "No promotion with no-quorum-policy=freeze"),
SchedulerTest("suicide-needed-inquorate", "no-quorum-policy=suicide: suicide necessary"),
SchedulerTest("suicide-not-needed-initial-quorum",
"no-quorum-policy=suicide: suicide not necessary at initial quorum"),
SchedulerTest("suicide-not-needed-never-quorate",
"no-quorum-policy=suicide: suicide not necessary if never quorate"),
SchedulerTest("suicide-not-needed-quorate", "no-quorum-policy=suicide: suicide necessary if quorate"),
]),
SchedulerTestGroup([
SchedulerTest("rec-node-1", "Node Recover - Startup - no fence"),
SchedulerTest("rec-node-2", "Node Recover - Startup - fence"),
SchedulerTest("rec-node-3", "Node Recover - HA down - no fence"),
SchedulerTest("rec-node-4", "Node Recover - HA down - fence"),
SchedulerTest("rec-node-5", "Node Recover - CRM down - no fence"),
SchedulerTest("rec-node-6", "Node Recover - CRM down - fence"),
SchedulerTest("rec-node-7", "Node Recover - no quorum - ignore"),
SchedulerTest("rec-node-8", "Node Recover - no quorum - freeze"),
SchedulerTest("rec-node-9", "Node Recover - no quorum - stop"),
SchedulerTest("rec-node-10", "Node Recover - no quorum - stop w/fence"),
SchedulerTest("rec-node-11", "Node Recover - CRM down w/ group - fence"),
SchedulerTest("rec-node-12", "Node Recover - nothing active - fence"),
SchedulerTest("rec-node-13", "Node Recover - failed resource + shutdown - fence"),
SchedulerTest("rec-node-15", "Node Recover - unknown lrm section"),
SchedulerTest("rec-node-14", "Serialize all stonith's"),
]),
SchedulerTestGroup([
SchedulerTest("multi1", "Multiple Active (stop/start)"),
]),
SchedulerTestGroup([
SchedulerTest("migrate-begin", "Normal migration"),
SchedulerTest("migrate-success", "Completed migration"),
SchedulerTest("migrate-partial-1", "Completed migration, missing stop on source"),
SchedulerTest("migrate-partial-2", "Successful migrate_to only"),
SchedulerTest("migrate-partial-3", "Successful migrate_to only, target down"),
SchedulerTest("migrate-partial-4", "Migrate from the correct host after migrate_to+migrate_from"),
SchedulerTest("bug-5186-partial-migrate", "Handle partial migration when src node loses membership"),
SchedulerTest("migrate-fail-2", "Failed migrate_from"),
SchedulerTest("migrate-fail-3", "Failed migrate_from + stop on source"),
SchedulerTest("migrate-fail-4",
"Failed migrate_from + stop on target - ideally we wouldn't need to re-stop on target"),
SchedulerTest("migrate-fail-5", "Failed migrate_from + stop on source and target"),
SchedulerTest("migrate-fail-6", "Failed migrate_to"),
SchedulerTest("migrate-fail-7", "Failed migrate_to + stop on source"),
SchedulerTest("migrate-fail-8",
"Failed migrate_to + stop on target - ideally we wouldn't need to re-stop on target"),
SchedulerTest("migrate-fail-9", "Failed migrate_to + stop on source and target"),
SchedulerTest("migration-ping-pong", "Old migrate_to failure + successful migrate_from on same node"),
SchedulerTest("migrate-stop", "Migration in a stopping stack"),
SchedulerTest("migrate-start", "Migration in a starting stack"),
SchedulerTest("migrate-stop_start", "Migration in a restarting stack"),
SchedulerTest("migrate-stop-complex", "Migration in a complex stopping stack"),
SchedulerTest("migrate-start-complex", "Migration in a complex starting stack"),
SchedulerTest("migrate-stop-start-complex", "Migration in a complex moving stack"),
SchedulerTest("migrate-shutdown", "Order the post-migration 'stop' before node shutdown"),
SchedulerTest("migrate-1", "Migrate (migrate)"),
SchedulerTest("migrate-2", "Migrate (stable)"),
SchedulerTest("migrate-3", "Migrate (failed migrate_to)"),
SchedulerTest("migrate-4", "Migrate (failed migrate_from)"),
SchedulerTest("novell-252693", "Migration in a stopping stack"),
SchedulerTest("novell-252693-2", "Migration in a starting stack"),
SchedulerTest("novell-252693-3", "Non-Migration in a starting and stopping stack"),
SchedulerTest("bug-1820", "Migration in a group"),
SchedulerTest("bug-1820-1", "Non-migration in a group"),
SchedulerTest("migrate-5", "Primitive migration with a clone"),
SchedulerTest("migrate-fencing", "Migration after Fencing"),
SchedulerTest("migrate-both-vms", "Migrate two VMs that have no colocation"),
SchedulerTest("migration-behind-migrating-remote", "Migrate resource behind migrating remote connection"),
SchedulerTest("1-a-then-bm-move-b", "Advanced migrate logic. A then B. migrate B"),
SchedulerTest("2-am-then-b-move-a", "Advanced migrate logic, A then B, migrate A without stopping B"),
SchedulerTest("3-am-then-bm-both-migrate", "Advanced migrate logic. A then B. migrate both"),
SchedulerTest("4-am-then-bm-b-not-migratable", "Advanced migrate logic, A then B, B not migratable"),
SchedulerTest("5-am-then-bm-a-not-migratable", "Advanced migrate logic. A then B. move both, a not migratable"),
SchedulerTest("6-migrate-group", "Advanced migrate logic, migrate a group"),
SchedulerTest("7-migrate-group-one-unmigratable",
"Advanced migrate logic, migrate group mixed with allow-migrate true/false"),
SchedulerTest("8-am-then-bm-a-migrating-b-stopping",
"Advanced migrate logic, A then B, A migrating, B stopping"),
SchedulerTest("9-am-then-bm-b-migrating-a-stopping",
"Advanced migrate logic, A then B, B migrate, A stopping"),
SchedulerTest("10-a-then-bm-b-move-a-clone",
"Advanced migrate logic, A clone then B, migrate B while stopping A"),
SchedulerTest("11-a-then-bm-b-move-a-clone-starting",
"Advanced migrate logic, A clone then B, B moving while A is start/stopping"),
SchedulerTest("a-promote-then-b-migrate", "A promote then B start. migrate B"),
SchedulerTest("a-demote-then-b-migrate", "A demote then B stop. migrate B"),
SchedulerTest("probe-target-of-failed-migrate_to-1", "Failed migrate_to, target rejoins"),
SchedulerTest("probe-target-of-failed-migrate_to-2", "Failed migrate_to, target rejoined and probed"),
SchedulerTest("partial-live-migration-multiple-active", "Prevent running on multiple nodes due to partial live migration"),
SchedulerTest("migration-intermediary-cleaned",
"Probe live-migration intermediary with no history"),
SchedulerTest("bug-lf-2422", "Dependency on partially active group - stop ocfs:*"),
]),
SchedulerTestGroup([
SchedulerTest("clone-anon-probe-1", "Probe the correct (anonymous) clone instance for each node"),
SchedulerTest("clone-anon-probe-2", "Avoid needless re-probing of anonymous clones"),
SchedulerTest("clone-anon-failcount", "Merge failcounts for anonymous clones"),
SchedulerTest("force-anon-clone-max", "Update clone-max properly when forcing a clone to be anonymous"),
SchedulerTest("anon-instance-pending", "Assign anonymous clone instance numbers properly when action pending"),
SchedulerTest("inc0", "Incarnation start"),
SchedulerTest("inc1", "Incarnation start order"),
SchedulerTest("inc2", "Incarnation silent restart, stop, move"),
SchedulerTest("inc3", "Inter-incarnation ordering, silent restart, stop, move"),
SchedulerTest("inc4", "Inter-incarnation ordering, silent restart, stop, move (ordered)"),
SchedulerTest("inc5", "Inter-incarnation ordering, silent restart, stop, move (restart 1)"),
SchedulerTest("inc6", "Inter-incarnation ordering, silent restart, stop, move (restart 2)"),
SchedulerTest("inc7", "Clone colocation"),
SchedulerTest("inc8", "Clone anti-colocation"),
SchedulerTest("inc9", "Non-unique clone"),
SchedulerTest("inc10", "Non-unique clone (stop)"),
SchedulerTest("inc11", "Primitive colocation with clones"),
SchedulerTest("inc12", "Clone shutdown"),
SchedulerTest("cloned-group", "Make sure only the correct number of cloned groups are started"),
SchedulerTest("cloned-group-stop", "Ensure stopping qpidd also stops glance and cinder"),
SchedulerTest("clone-no-shuffle", "Don't prioritize allocation of instances that must be moved"),
SchedulerTest("clone-recover-no-shuffle-1",
"Don't shuffle instances when starting a new primitive instance"),
SchedulerTest("clone-recover-no-shuffle-2",
"Don't shuffle instances when starting a new group instance"),
SchedulerTest("clone-recover-no-shuffle-3",
"Don't shuffle instances when starting a new bundle instance"),
SchedulerTest("clone-recover-no-shuffle-4",
"Don't shuffle instances when starting a new primitive instance with location preference"),
SchedulerTest("clone-recover-no-shuffle-5",
"Don't shuffle instances when starting a new group instance with location preference"),
SchedulerTest("clone-recover-no-shuffle-6",
"Don't shuffle instances when starting a new bundle instance with location preference"),
SchedulerTest("clone-recover-no-shuffle-7",
"Don't shuffle instances when starting a new primitive instance that will be promoted"),
SchedulerTest("clone-recover-no-shuffle-8",
"Don't shuffle instances when starting a new group instance that will be promoted"),
SchedulerTest("clone-recover-no-shuffle-9",
"Don't shuffle instances when starting a new bundle instance that will be promoted"),
SchedulerTest("clone-recover-no-shuffle-10",
"Don't shuffle instances when starting a new primitive instance that won't be promoted"),
SchedulerTest("clone-recover-no-shuffle-11",
"Don't shuffle instances when starting a new group instance that won't be promoted"),
SchedulerTest("clone-recover-no-shuffle-12",
"Don't shuffle instances when starting a new bundle instance that won't be promoted"),
SchedulerTest("clone-max-zero", "Orphan processing with clone-max=0"),
SchedulerTest("clone-anon-dup",
"Bug LF#2087 - Correctly parse the state of anonymous clones that are active more than once per node"),
SchedulerTest("bug-lf-2160", "Don't shuffle clones due to colocation"),
SchedulerTest("bug-lf-2213", "clone-node-max enforcement for cloned groups"),
SchedulerTest("bug-lf-2153", "Clone ordering constraints"),
SchedulerTest("bug-lf-2361", "Ensure clones observe mandatory ordering constraints if the LHS is unrunnable"),
SchedulerTest("bug-lf-2317", "Avoid needless restart of primitive depending on a clone"),
SchedulerTest("bug-lf-2453", "Enforce mandatory clone ordering without colocation"),
SchedulerTest("bug-lf-2508", "Correctly reconstruct the status of anonymous cloned groups"),
SchedulerTest("bug-lf-2544", "Balanced clone placement"),
SchedulerTest("bug-lf-2445", "Redistribute clones with node-max > 1 and stickiness = 0"),
SchedulerTest("bug-lf-2574", "Avoid clone shuffle"),
SchedulerTest("bug-lf-2581", "Avoid group restart due to unrelated clone (re)start"),
SchedulerTest("bug-cl-5168", "Don't shuffle clones"),
SchedulerTest("bug-cl-5170", "Prevent clone from starting with on-fail=block"),
SchedulerTest("clone-fail-block-colocation", "Move colocated group when failed clone has on-fail=block"),
SchedulerTest("clone-interleave-1",
"Clone-3 cannot start on pcmk-1 due to interleaved ordering (no colocation)"),
SchedulerTest("clone-interleave-2", "Clone-3 must stop on pcmk-1 due to interleaved ordering (no colocation)"),
SchedulerTest("clone-interleave-3",
"Clone-3 must be recovered on pcmk-1 due to interleaved ordering (no colocation)"),
SchedulerTest("rebalance-unique-clones", "Rebalance unique clone instances with no stickiness"),
SchedulerTest("clone-requires-quorum-recovery", "Clone with requires=quorum on failed node needing recovery"),
SchedulerTest("clone-requires-quorum",
"Clone with requires=quorum with presumed-inactive instance on failed node"),
]),
SchedulerTestGroup([
SchedulerTest("cloned_start_one", "order first clone then clone... first clone_min=2"),
SchedulerTest("cloned_start_two", "order first clone then clone... first clone_min=2"),
SchedulerTest("cloned_stop_one", "order first clone then clone... first clone_min=2"),
SchedulerTest("cloned_stop_two", "order first clone then clone... first clone_min=2"),
SchedulerTest("clone_min_interleave_start_one",
"order first clone then clone... first clone_min=2 and then has interleave=true"),
SchedulerTest("clone_min_interleave_start_two",
"order first clone then clone... first clone_min=2 and then has interleave=true"),
SchedulerTest("clone_min_interleave_stop_one",
"order first clone then clone... first clone_min=2 and then has interleave=true"),
SchedulerTest("clone_min_interleave_stop_two",
"order first clone then clone... first clone_min=2 and then has interleave=true"),
SchedulerTest("clone_min_start_one", "order first clone then primitive... first clone_min=2"),
SchedulerTest("clone_min_start_two", "order first clone then primitive... first clone_min=2"),
SchedulerTest("clone_min_stop_all", "order first clone then primitive... first clone_min=2"),
SchedulerTest("clone_min_stop_one", "order first clone then primitive... first clone_min=2"),
SchedulerTest("clone_min_stop_two", "order first clone then primitive... first clone_min=2"),
]),
SchedulerTestGroup([
SchedulerTest("unfence-startup", "Clean unfencing"),
SchedulerTest("unfence-definition", "Unfencing when the agent changes"),
SchedulerTest("unfence-parameters", "Unfencing when the agent parameters changes"),
SchedulerTest("unfence-device", "Unfencing when a cluster has only fence devices"),
]),
SchedulerTestGroup([
SchedulerTest("promoted-0", "Stopped -> Unpromoted"),
SchedulerTest("promoted-1", "Stopped -> Promote"),
SchedulerTest("promoted-2", "Stopped -> Promote : notify"),
SchedulerTest("promoted-3", "Stopped -> Promote : promoted location"),
SchedulerTest("promoted-4", "Started -> Promote : promoted location"),
SchedulerTest("promoted-5", "Promoted -> Promoted"),
SchedulerTest("promoted-6", "Promoted -> Promoted (2)"),
SchedulerTest("promoted-7", "Promoted -> Fenced"),
SchedulerTest("promoted-8", "Promoted -> Fenced -> Moved"),
SchedulerTest("promoted-9", "Stopped + Promotable + No quorum"),
SchedulerTest("promoted-10", "Stopped -> Promotable : notify with monitor"),
SchedulerTest("promoted-11", "Stopped -> Promote : colocation"),
SchedulerTest("novell-239082", "Demote/Promote ordering"),
SchedulerTest("novell-239087", "Stable promoted placement"),
SchedulerTest("promoted-12", "Promotion based solely on rsc_location constraints"),
SchedulerTest("promoted-13", "Include preferences of colocated resources when placing promoted"),
SchedulerTest("promoted-demote", "Ordering when actions depends on demoting an unpromoted resource"),
SchedulerTest("promoted-ordering", "Prevent resources from starting that need a promoted"),
SchedulerTest("bug-1765", "Verify promoted-with-promoted colocation does not stop unpromoted instances"),
SchedulerTest("promoted-group", "Promotion of cloned groups"),
SchedulerTest("bug-lf-1852", "Don't shuffle promotable instances unnecessarily"),
SchedulerTest("promoted-failed-demote", "Don't retry failed demote actions"),
SchedulerTest("promoted-failed-demote-2", "Don't retry failed demote actions (notify=false)"),
SchedulerTest("promoted-depend",
"Ensure resources that depend on promoted instance don't get allocated until that does"),
SchedulerTest("promoted-reattach", "Re-attach to a running promoted"),
SchedulerTest("promoted-allow-start", "Don't include promoted score if it would prevent allocation"),
SchedulerTest("promoted-colocation",
"Allow promoted instances placemaker to be influenced by colocation constraints"),
SchedulerTest("promoted-pseudo", "Make sure promote/demote pseudo actions are created correctly"),
SchedulerTest("promoted-role", "Prevent target-role from promoting more than promoted-max instances"),
SchedulerTest("bug-lf-2358", "Anti-colocation of promoted instances"),
SchedulerTest("promoted-promotion-constraint", "Mandatory promoted colocation constraints"),
SchedulerTest("unmanaged-promoted", "Ensure role is preserved for unmanaged resources"),
SchedulerTest("promoted-unmanaged-monitor", "Start correct monitor for unmanaged promoted instances"),
SchedulerTest("promoted-demote-2", "Demote does not clear past failure"),
SchedulerTest("promoted-move", "Move promoted based on failure of colocated group"),
SchedulerTest("promoted-probed-score", "Observe the promotion score of probed resources"),
SchedulerTest("colocation_constraint_stops_promoted",
"cl#5054 - Ensure promoted is demoted when stopped by colocation constraint"),
SchedulerTest("colocation_constraint_stops_unpromoted",
"cl#5054 - Ensure unpromoted is not demoted when stopped by colocation constraint"),
SchedulerTest("order_constraint_stops_promoted",
"cl#5054 - Ensure promoted is demoted when stopped by order constraint"),
SchedulerTest("order_constraint_stops_unpromoted",
"cl#5054 - Ensure unpromoted is not demoted when stopped by order constraint"),
SchedulerTest("promoted_monitor_restart", "cl#5072 - Ensure promoted monitor operation will start after promotion"),
SchedulerTest("bug-rh-880249", "Handle replacement of an m/s resource with a primitive"),
SchedulerTest("bug-5143-ms-shuffle", "Prevent promoted instance shuffling due to promotion score"),
SchedulerTest("promoted-demote-block", "Block promotion if demote fails with on-fail=block"),
SchedulerTest("promoted-dependent-ban",
"Don't stop instances from being active because a dependent is banned from that host"),
SchedulerTest("promoted-stop", "Stop instances due to location constraint with role=Started"),
SchedulerTest("promoted-partially-demoted-group", "Allow partially demoted group to finish demoting"),
SchedulerTest("bug-cl-5213", "Ensure role colocation with -INFINITY is enforced"),
SchedulerTest("bug-cl-5219", "Allow unrelated resources with a common colocation target to remain promoted"),
SchedulerTest("promoted-asymmetrical-order",
"Fix the behaviors of multi-state resources with asymmetrical ordering"),
SchedulerTest("promoted-notify", "Promotion with notifications"),
SchedulerTest("promoted-score-startup", "Use permanent promoted scores without LRM history"),
SchedulerTest("failed-demote-recovery", "Recover resource in unpromoted role after demote fails"),
SchedulerTest("failed-demote-recovery-promoted", "Recover resource in promoted role after demote fails"),
SchedulerTest("on_fail_demote1", "Recovery with on-fail=\"demote\" on healthy cluster, remote, guest, and bundle nodes"),
SchedulerTest("on_fail_demote2", "Recovery with on-fail=\"demote\" with promotion on different node"),
SchedulerTest("on_fail_demote3", "Recovery with on-fail=\"demote\" with no promotion"),
SchedulerTest("on_fail_demote4", "Recovery with on-fail=\"demote\" on failed cluster, remote, guest, and bundle nodes"),
SchedulerTest("no_quorum_demote", "Promotable demotion and primitive stop with no-quorum-policy=\"demote\""),
SchedulerTest("no-promote-on-unrunnable-guest", "Don't select bundle instance for promotion when container can't run"),
SchedulerTest("leftover-pending-monitor", "Prevent a leftover pending monitor from causing unexpected stop of other instances"),
]),
SchedulerTestGroup([
SchedulerTest("history-1", "Correctly parse stateful-1 resource state"),
]),
SchedulerTestGroup([
SchedulerTest("managed-0", "Managed (reference)"),
SchedulerTest("managed-1", "Not managed - down"),
SchedulerTest("managed-2", "Not managed - up"),
SchedulerTest("bug-5028", "Shutdown should block if anything depends on an unmanaged resource"),
SchedulerTest("bug-5028-detach", "Ensure detach still works"),
SchedulerTest("bug-5028-bottom",
"Ensure shutdown still blocks if the blocked resource is at the bottom of the stack"),
SchedulerTest("unmanaged-stop-1",
"cl#5155 - Block the stop of resources if any depending resource is unmanaged"),
SchedulerTest("unmanaged-stop-2",
"cl#5155 - Block the stop of resources if the first resource in a mandatory stop order is unmanaged"),
SchedulerTest("unmanaged-stop-3",
"cl#5155 - Block the stop of resources if any depending resource in a group is unmanaged"),
SchedulerTest("unmanaged-stop-4",
"cl#5155 - Block the stop of resources if any depending resource in the middle of a group is unmanaged"),
SchedulerTest("unmanaged-block-restart",
"Block restart of resources if any dependent resource in a group is unmanaged"),
]),
SchedulerTestGroup([
SchedulerTest("interleave-0", "Interleave (reference)"),
SchedulerTest("interleave-1", "coloc - not interleaved"),
SchedulerTest("interleave-2", "coloc - interleaved"),
SchedulerTest("interleave-3", "coloc - interleaved (2)"),
SchedulerTest("interleave-pseudo-stop", "Interleaved clone during stonith"),
SchedulerTest("interleave-stop", "Interleaved clone during stop"),
SchedulerTest("interleave-restart", "Interleaved clone during dependency restart"),
]),
SchedulerTestGroup([
SchedulerTest("notify-0", "Notify reference"),
SchedulerTest("notify-1", "Notify simple"),
SchedulerTest("notify-2", "Notify simple, confirm"),
SchedulerTest("notify-3", "Notify move, confirm"),
SchedulerTest("novell-239079", "Notification priority"),
SchedulerTest("notifs-for-unrunnable", "Don't schedule notifications for an unrunnable action"),
SchedulerTest("route-remote-notify", "Route remote notify actions through correct cluster node"),
SchedulerTest("notify-behind-stopping-remote", "Don't schedule notifications behind stopped remote"),
]),
SchedulerTestGroup([
SchedulerTest("594", "OSDL #594 - Unrunnable actions scheduled in transition"),
SchedulerTest("662", "OSDL #662 - Two resources start on one node when incarnation_node_max = 1"),
SchedulerTest("696", "OSDL #696 - CRM starts stonith RA without monitor"),
SchedulerTest("726", "OSDL #726 - Attempting to schedule rsc_posic041_monitor_5000 _after_ a stop"),
SchedulerTest("735", "OSDL #735 - Correctly detect that rsc_hadev1 is stopped on hadev3"),
SchedulerTest("764", "OSDL #764 - Missing monitor op for DoFencing:child_DoFencing:1"),
SchedulerTest("797", "OSDL #797 - Assert triggered: task_id_i > max_call_id"),
SchedulerTest("829", "OSDL #829"),
SchedulerTest("994",
"OSDL #994 - Stopping the last resource in a resource group causes the entire group to be restarted"),
SchedulerTest("994-2", "OSDL #994 - with a dependent resource"),
SchedulerTest("1360", "OSDL #1360 - Clone stickiness"),
SchedulerTest("1484", "OSDL #1484 - on_fail=stop"),
SchedulerTest("1494", "OSDL #1494 - Clone stability"),
SchedulerTest("unrunnable-1", "Unrunnable"),
SchedulerTest("unrunnable-2", "Unrunnable 2"),
SchedulerTest("stonith-0", "Stonith loop - 1"),
SchedulerTest("stonith-1", "Stonith loop - 2"),
SchedulerTest("stonith-2", "Stonith loop - 3"),
SchedulerTest("stonith-3", "Stonith startup"),
SchedulerTest("stonith-4", "Stonith node state"),
SchedulerTest("dc-fence-ordering", "DC needs fencing while other nodes are shutting down"),
SchedulerTest("bug-1572-1", "Recovery of groups depending on promotable role"),
SchedulerTest("bug-1572-2", "Recovery of groups depending on promotable role when promoted is not re-promoted"),
SchedulerTest("bug-1685", "Depends-on-promoted ordering"),
SchedulerTest("bug-1822", "Don't promote partially active groups"),
SchedulerTest("bug-pm-11", "New resource added to a m/s group"),
SchedulerTest("bug-pm-12", "Recover only the failed portion of a cloned group"),
SchedulerTest("bug-n-387749", "Don't shuffle clone instances"),
SchedulerTest("bug-n-385265",
"Don't ignore the failure stickiness of group children - resource_idvscommon should stay stopped"),
SchedulerTest("bug-n-385265-2",
"Ensure groups are migrated instead of remaining partially active on the current node"),
SchedulerTest("bug-lf-1920", "Correctly handle probes that find active resources"),
SchedulerTest("bnc-515172", "Location constraint with multiple expressions"),
SchedulerTest("colocate-primitive-with-clone", "Optional colocation with a clone"),
SchedulerTest("use-after-free-merge", "Use-after-free in native_merge_weights"),
SchedulerTest("bug-lf-2551", "STONITH ordering for stop"),
SchedulerTest("bug-lf-2606", "Stonith implies demote"),
SchedulerTest("bug-lf-2474", "Ensure resource op timeout takes precedence over op_defaults"),
SchedulerTest("bug-suse-707150", "Prevent vm-01 from starting due to colocation/ordering"),
SchedulerTest("bug-5014-A-start-B-start", "Verify when A starts B starts using symmetrical=false"),
SchedulerTest("bug-5014-A-stop-B-started",
"Verify when A stops B does not stop if it has already started using symmetric=false"),
SchedulerTest("bug-5014-A-stopped-B-stopped",
"Verify when A is stopped and B has not started, B does not start before A using symmetric=false"),
SchedulerTest("bug-5014-CthenAthenB-C-stopped",
"Verify when C then A is symmetrical=true, A then B is symmetric=false, and C is stopped that nothing starts"),
SchedulerTest("bug-5014-CLONE-A-start-B-start",
"Verify when A starts B starts using clone resources with symmetric=false"),
SchedulerTest("bug-5014-CLONE-A-stop-B-started",
"Verify when A stops B does not stop if it has already started using clone resources with symmetric=false"),
SchedulerTest("bug-5014-GROUP-A-start-B-start",
"Verify when A starts B starts when using group resources with symmetric=false"),
SchedulerTest("bug-5014-GROUP-A-stopped-B-started",
"Verify when A stops B does not stop if it has already started using group resources with symmetric=false"),
SchedulerTest("bug-5014-GROUP-A-stopped-B-stopped",
"Verify when A is stopped and B has not started, B does not start before A using group resources with symmetric=false"),
SchedulerTest("bug-5014-ordered-set-symmetrical-false",
"Verify ordered sets work with symmetrical=false"),
SchedulerTest("bug-5014-ordered-set-symmetrical-true",
"Verify ordered sets work with symmetrical=true"),
SchedulerTest("clbz5007-promotable-colocation",
"Verify use of colocation scores other than INFINITY and -INFINITY work on multi-state resources"),
SchedulerTest("bug-5038", "Prevent restart of anonymous clones when clone-max decreases"),
SchedulerTest("bug-5025-1", "Automatically clean up failcount after resource config change with reload"),
SchedulerTest("bug-5025-2", "Make sure clear failcount action isn't set when config does not change"),
SchedulerTest("bug-5025-3", "Automatically clean up failcount after resource config change with restart"),
SchedulerTest("bug-5025-4", "Clear failcount when last failure is a start op and rsc attributes changed"),
SchedulerTest("failcount", "Ensure failcounts are correctly expired"),
SchedulerTest("failcount-block", "Ensure failcounts are not expired when on-fail=block is present"),
SchedulerTest("per-op-failcount", "Ensure per-operation failcount is handled and not passed to fence agent"),
SchedulerTest("on-fail-ignore", "Ensure on-fail=ignore works even beyond migration-threshold"),
SchedulerTest("monitor-onfail-restart", "bug-5058 - Monitor failure with on-fail set to restart"),
SchedulerTest("monitor-onfail-stop", "bug-5058 - Monitor failure wiht on-fail set to stop"),
SchedulerTest("bug-5059", "No need to restart p_stateful1:*"),
SchedulerTest("bug-5069-op-enabled", "Test on-fail=ignore with failure when monitor is enabled"),
SchedulerTest("bug-5069-op-disabled", "Test on-fail-ignore with failure when monitor is disabled"),
SchedulerTest("obsolete-lrm-resource", "cl#5115 - Do not use obsolete lrm_resource sections"),
SchedulerTest("expire-non-blocked-failure",
"Ignore failure-timeout only if the failed operation has on-fail=block"),
SchedulerTest("asymmetrical-order-move", "Respect asymmetrical ordering when trying to move resources"),
SchedulerTest("asymmetrical-order-restart", "Respect asymmetrical ordering when restarting dependent resource"),
SchedulerTest("start-then-stop-with-unfence", "Avoid graph loop with start-then-stop constraint plus unfencing"),
SchedulerTest("order-expired-failure", "Order failcount cleanup after remote fencing"),
SchedulerTest("expired-stop-1", "Expired stop failure should not block resource"),
SchedulerTest("ignore_stonith_rsc_order1",
"cl#5056- Ignore order constraint between stonith and non-stonith rsc"),
SchedulerTest("ignore_stonith_rsc_order2",
"cl#5056- Ignore order constraint with group rsc containing mixed stonith and non-stonith"),
SchedulerTest("ignore_stonith_rsc_order3", "cl#5056- Ignore order constraint, stonith clone and mixed group"),
SchedulerTest("ignore_stonith_rsc_order4",
"cl#5056- Ignore order constraint, stonith clone and clone with nested mixed group"),
SchedulerTest("honor_stonith_rsc_order1",
"cl#5056- Honor order constraint, stonith clone and pure stonith group(single rsc)"),
SchedulerTest("honor_stonith_rsc_order2",
"cl#5056- Honor order constraint, stonith clone and pure stonith group(multiple rsc)"),
SchedulerTest("honor_stonith_rsc_order3",
"cl#5056- Honor order constraint, stonith clones with nested pure stonith group"),
SchedulerTest("honor_stonith_rsc_order4",
"cl#5056- Honor order constraint, between two native stonith rscs"),
SchedulerTest("multiply-active-stonith", "Multiply active stonith"),
SchedulerTest("probe-timeout", "cl#5099 - Default probe timeout"),
SchedulerTest("order-first-probes",
"cl#5301 - respect order constraints when relevant resources are being probed"),
SchedulerTest("concurrent-fencing", "Allow performing fencing operations in parallel"),
SchedulerTest("priority-fencing-delay", "Delay fencing targeting the more significant node"),
SchedulerTest("pending-node-no-uname", "Do not fence a pending node that doesn't have an uname in node state yet"),
SchedulerTest("node-pending-timeout", "Fence a pending node that has reached `node-pending-timeout`"),
]),
SchedulerTestGroup([
SchedulerTest("systemhealth1", "System Health () #1"),
SchedulerTest("systemhealth2", "System Health () #2"),
SchedulerTest("systemhealth3", "System Health () #3"),
SchedulerTest("systemhealthn1", "System Health (None) #1"),
SchedulerTest("systemhealthn2", "System Health (None) #2"),
SchedulerTest("systemhealthn3", "System Health (None) #3"),
SchedulerTest("systemhealthm1", "System Health (Migrate On Red) #1"),
SchedulerTest("systemhealthm2", "System Health (Migrate On Red) #2"),
SchedulerTest("systemhealthm3", "System Health (Migrate On Red) #3"),
SchedulerTest("systemhealtho1", "System Health (Only Green) #1"),
SchedulerTest("systemhealtho2", "System Health (Only Green) #2"),
SchedulerTest("systemhealtho3", "System Health (Only Green) #3"),
SchedulerTest("systemhealthp1", "System Health (Progessive) #1"),
SchedulerTest("systemhealthp2", "System Health (Progessive) #2"),
SchedulerTest("systemhealthp3", "System Health (Progessive) #3"),
SchedulerTest("allow-unhealthy-nodes", "System Health (migrate-on-red + allow-unhealth-nodes)"),
]),
SchedulerTestGroup([
SchedulerTest("utilization", "Placement Strategy - utilization"),
SchedulerTest("minimal", "Placement Strategy - minimal"),
SchedulerTest("balanced", "Placement Strategy - balanced"),
]),
SchedulerTestGroup([
SchedulerTest("placement-stickiness", "Optimized Placement Strategy - stickiness"),
SchedulerTest("placement-priority", "Optimized Placement Strategy - priority"),
SchedulerTest("placement-location", "Optimized Placement Strategy - location"),
SchedulerTest("placement-capacity", "Optimized Placement Strategy - capacity"),
]),
SchedulerTestGroup([
SchedulerTest("utilization-order1", "Utilization Order - Simple"),
SchedulerTest("utilization-order2", "Utilization Order - Complex"),
SchedulerTest("utilization-order3", "Utilization Order - Migrate"),
SchedulerTest("utilization-order4", "Utilization Order - Live Migration (bnc#695440)"),
SchedulerTest("utilization-complex", "Utilization with complex relationships"),
SchedulerTest("utilization-shuffle",
"Don't displace prmExPostgreSQLDB2 on act2, Start prmExPostgreSQLDB1 on act3"),
SchedulerTest("load-stopped-loop", "Avoid transition loop due to load_stopped (cl#5044)"),
SchedulerTest("load-stopped-loop-2",
"cl#5235 - Prevent graph loops that can be introduced by load_stopped -> migrate_to ordering"),
]),
SchedulerTestGroup([
SchedulerTest("colocated-utilization-primitive-1", "Colocated Utilization - Primitive"),
SchedulerTest("colocated-utilization-primitive-2", "Colocated Utilization - Choose the most capable node"),
SchedulerTest("colocated-utilization-group", "Colocated Utilization - Group"),
SchedulerTest("colocated-utilization-clone", "Colocated Utilization - Clone"),
SchedulerTest("utilization-check-allowed-nodes",
"Only check the capacities of the nodes that can run the resource"),
]),
SchedulerTestGroup([
SchedulerTest("node-maintenance-1", "cl#5128 - Node maintenance"),
SchedulerTest("node-maintenance-2", "cl#5128 - Node maintenance (coming out of maintenance mode)"),
SchedulerTest("shutdown-maintenance-node", "Do not fence a maintenance node if it shuts down cleanly"),
SchedulerTest("rsc-maintenance", "Per-resource maintenance"),
]),
SchedulerTestGroup([
SchedulerTest("not-installed-agent", "The resource agent is missing"),
SchedulerTest("not-installed-tools", "Something the resource agent needs is missing"),
]),
SchedulerTestGroup([
SchedulerTest("stopped-monitor-00", "Stopped Monitor - initial start"),
SchedulerTest("stopped-monitor-01", "Stopped Monitor - failed started"),
SchedulerTest("stopped-monitor-02", "Stopped Monitor - started multi-up"),
SchedulerTest("stopped-monitor-03", "Stopped Monitor - stop started"),
SchedulerTest("stopped-monitor-04", "Stopped Monitor - failed stop"),
SchedulerTest("stopped-monitor-05", "Stopped Monitor - start unmanaged"),
SchedulerTest("stopped-monitor-06", "Stopped Monitor - unmanaged multi-up"),
SchedulerTest("stopped-monitor-07", "Stopped Monitor - start unmanaged multi-up"),
SchedulerTest("stopped-monitor-08", "Stopped Monitor - migrate"),
SchedulerTest("stopped-monitor-09", "Stopped Monitor - unmanage started"),
SchedulerTest("stopped-monitor-10", "Stopped Monitor - unmanaged started multi-up"),
SchedulerTest("stopped-monitor-11", "Stopped Monitor - stop unmanaged started"),
SchedulerTest("stopped-monitor-12", "Stopped Monitor - unmanaged started multi-up (target-role=Stopped)"),
SchedulerTest("stopped-monitor-20", "Stopped Monitor - initial stop"),
SchedulerTest("stopped-monitor-21", "Stopped Monitor - stopped single-up"),
SchedulerTest("stopped-monitor-22", "Stopped Monitor - stopped multi-up"),
SchedulerTest("stopped-monitor-23", "Stopped Monitor - start stopped"),
SchedulerTest("stopped-monitor-24", "Stopped Monitor - unmanage stopped"),
SchedulerTest("stopped-monitor-25", "Stopped Monitor - unmanaged stopped multi-up"),
SchedulerTest("stopped-monitor-26", "Stopped Monitor - start unmanaged stopped"),
SchedulerTest("stopped-monitor-27", "Stopped Monitor - unmanaged stopped multi-up (target-role=Started)"),
SchedulerTest("stopped-monitor-30", "Stopped Monitor - new node started"),
SchedulerTest("stopped-monitor-31", "Stopped Monitor - new node stopped"),
]),
SchedulerTestGroup([
# This is a combo test to check:
# - probe timeout defaults to the minimum-interval monitor's
# - duplicate recurring operations are ignored
# - if timeout spec is bad, the default timeout is used
# - failure is blocked with on-fail=block even if ISO8601 interval is specified
# - started/stopped role monitors are started/stopped on right nodes
SchedulerTest("intervals", "Recurring monitor interval handling"),
]),
SchedulerTestGroup([
SchedulerTest("ticket-primitive-1", "Ticket - Primitive (loss-policy=stop, initial)"),
SchedulerTest("ticket-primitive-2", "Ticket - Primitive (loss-policy=stop, granted)"),
SchedulerTest("ticket-primitive-3", "Ticket - Primitive (loss-policy-stop, revoked)"),
SchedulerTest("ticket-primitive-4", "Ticket - Primitive (loss-policy=demote, initial)"),
SchedulerTest("ticket-primitive-5", "Ticket - Primitive (loss-policy=demote, granted)"),
SchedulerTest("ticket-primitive-6", "Ticket - Primitive (loss-policy=demote, revoked)"),
SchedulerTest("ticket-primitive-7", "Ticket - Primitive (loss-policy=fence, initial)"),
SchedulerTest("ticket-primitive-8", "Ticket - Primitive (loss-policy=fence, granted)"),
SchedulerTest("ticket-primitive-9", "Ticket - Primitive (loss-policy=fence, revoked)"),
SchedulerTest("ticket-primitive-10", "Ticket - Primitive (loss-policy=freeze, initial)"),
SchedulerTest("ticket-primitive-11", "Ticket - Primitive (loss-policy=freeze, granted)"),
SchedulerTest("ticket-primitive-12", "Ticket - Primitive (loss-policy=freeze, revoked)"),
SchedulerTest("ticket-primitive-13", "Ticket - Primitive (loss-policy=stop, standby, granted)"),
SchedulerTest("ticket-primitive-14", "Ticket - Primitive (loss-policy=stop, granted, standby)"),
SchedulerTest("ticket-primitive-15", "Ticket - Primitive (loss-policy=stop, standby, revoked)"),
SchedulerTest("ticket-primitive-16", "Ticket - Primitive (loss-policy=demote, standby, granted)"),
SchedulerTest("ticket-primitive-17", "Ticket - Primitive (loss-policy=demote, granted, standby)"),
SchedulerTest("ticket-primitive-18", "Ticket - Primitive (loss-policy=demote, standby, revoked)"),
SchedulerTest("ticket-primitive-19", "Ticket - Primitive (loss-policy=fence, standby, granted)"),
SchedulerTest("ticket-primitive-20", "Ticket - Primitive (loss-policy=fence, granted, standby)"),
SchedulerTest("ticket-primitive-21", "Ticket - Primitive (loss-policy=fence, standby, revoked)"),
SchedulerTest("ticket-primitive-22", "Ticket - Primitive (loss-policy=freeze, standby, granted)"),
SchedulerTest("ticket-primitive-23", "Ticket - Primitive (loss-policy=freeze, granted, standby)"),
SchedulerTest("ticket-primitive-24", "Ticket - Primitive (loss-policy=freeze, standby, revoked)"),
]),
SchedulerTestGroup([
SchedulerTest("ticket-group-1", "Ticket - Group (loss-policy=stop, initial)"),
SchedulerTest("ticket-group-2", "Ticket - Group (loss-policy=stop, granted)"),
SchedulerTest("ticket-group-3", "Ticket - Group (loss-policy-stop, revoked)"),
SchedulerTest("ticket-group-4", "Ticket - Group (loss-policy=demote, initial)"),
SchedulerTest("ticket-group-5", "Ticket - Group (loss-policy=demote, granted)"),
SchedulerTest("ticket-group-6", "Ticket - Group (loss-policy=demote, revoked)"),
SchedulerTest("ticket-group-7", "Ticket - Group (loss-policy=fence, initial)"),
SchedulerTest("ticket-group-8", "Ticket - Group (loss-policy=fence, granted)"),
SchedulerTest("ticket-group-9", "Ticket - Group (loss-policy=fence, revoked)"),
SchedulerTest("ticket-group-10", "Ticket - Group (loss-policy=freeze, initial)"),
SchedulerTest("ticket-group-11", "Ticket - Group (loss-policy=freeze, granted)"),
SchedulerTest("ticket-group-12", "Ticket - Group (loss-policy=freeze, revoked)"),
SchedulerTest("ticket-group-13", "Ticket - Group (loss-policy=stop, standby, granted)"),
SchedulerTest("ticket-group-14", "Ticket - Group (loss-policy=stop, granted, standby)"),
SchedulerTest("ticket-group-15", "Ticket - Group (loss-policy=stop, standby, revoked)"),
SchedulerTest("ticket-group-16", "Ticket - Group (loss-policy=demote, standby, granted)"),
SchedulerTest("ticket-group-17", "Ticket - Group (loss-policy=demote, granted, standby)"),
SchedulerTest("ticket-group-18", "Ticket - Group (loss-policy=demote, standby, revoked)"),
SchedulerTest("ticket-group-19", "Ticket - Group (loss-policy=fence, standby, granted)"),
SchedulerTest("ticket-group-20", "Ticket - Group (loss-policy=fence, granted, standby)"),
SchedulerTest("ticket-group-21", "Ticket - Group (loss-policy=fence, standby, revoked)"),
SchedulerTest("ticket-group-22", "Ticket - Group (loss-policy=freeze, standby, granted)"),
SchedulerTest("ticket-group-23", "Ticket - Group (loss-policy=freeze, granted, standby)"),
SchedulerTest("ticket-group-24", "Ticket - Group (loss-policy=freeze, standby, revoked)"),
]),
SchedulerTestGroup([
SchedulerTest("ticket-clone-1", "Ticket - Clone (loss-policy=stop, initial)"),
SchedulerTest("ticket-clone-2", "Ticket - Clone (loss-policy=stop, granted)"),
SchedulerTest("ticket-clone-3", "Ticket - Clone (loss-policy-stop, revoked)"),
SchedulerTest("ticket-clone-4", "Ticket - Clone (loss-policy=demote, initial)"),
SchedulerTest("ticket-clone-5", "Ticket - Clone (loss-policy=demote, granted)"),
SchedulerTest("ticket-clone-6", "Ticket - Clone (loss-policy=demote, revoked)"),
SchedulerTest("ticket-clone-7", "Ticket - Clone (loss-policy=fence, initial)"),
SchedulerTest("ticket-clone-8", "Ticket - Clone (loss-policy=fence, granted)"),
SchedulerTest("ticket-clone-9", "Ticket - Clone (loss-policy=fence, revoked)"),
SchedulerTest("ticket-clone-10", "Ticket - Clone (loss-policy=freeze, initial)"),
SchedulerTest("ticket-clone-11", "Ticket - Clone (loss-policy=freeze, granted)"),
SchedulerTest("ticket-clone-12", "Ticket - Clone (loss-policy=freeze, revoked)"),
SchedulerTest("ticket-clone-13", "Ticket - Clone (loss-policy=stop, standby, granted)"),
SchedulerTest("ticket-clone-14", "Ticket - Clone (loss-policy=stop, granted, standby)"),
SchedulerTest("ticket-clone-15", "Ticket - Clone (loss-policy=stop, standby, revoked)"),
SchedulerTest("ticket-clone-16", "Ticket - Clone (loss-policy=demote, standby, granted)"),
SchedulerTest("ticket-clone-17", "Ticket - Clone (loss-policy=demote, granted, standby)"),
SchedulerTest("ticket-clone-18", "Ticket - Clone (loss-policy=demote, standby, revoked)"),
SchedulerTest("ticket-clone-19", "Ticket - Clone (loss-policy=fence, standby, granted)"),
SchedulerTest("ticket-clone-20", "Ticket - Clone (loss-policy=fence, granted, standby)"),
SchedulerTest("ticket-clone-21", "Ticket - Clone (loss-policy=fence, standby, revoked)"),
SchedulerTest("ticket-clone-22", "Ticket - Clone (loss-policy=freeze, standby, granted)"),
SchedulerTest("ticket-clone-23", "Ticket - Clone (loss-policy=freeze, granted, standby)"),
SchedulerTest("ticket-clone-24", "Ticket - Clone (loss-policy=freeze, standby, revoked)"),
]),
SchedulerTestGroup([
SchedulerTest("ticket-promoted-1", "Ticket - Promoted (loss-policy=stop, initial)"),
SchedulerTest("ticket-promoted-2", "Ticket - Promoted (loss-policy=stop, granted)"),
SchedulerTest("ticket-promoted-3", "Ticket - Promoted (loss-policy-stop, revoked)"),
SchedulerTest("ticket-promoted-4", "Ticket - Promoted (loss-policy=demote, initial)"),
SchedulerTest("ticket-promoted-5", "Ticket - Promoted (loss-policy=demote, granted)"),
SchedulerTest("ticket-promoted-6", "Ticket - Promoted (loss-policy=demote, revoked)"),
SchedulerTest("ticket-promoted-7", "Ticket - Promoted (loss-policy=fence, initial)"),
SchedulerTest("ticket-promoted-8", "Ticket - Promoted (loss-policy=fence, granted)"),
SchedulerTest("ticket-promoted-9", "Ticket - Promoted (loss-policy=fence, revoked)"),
SchedulerTest("ticket-promoted-10", "Ticket - Promoted (loss-policy=freeze, initial)"),
SchedulerTest("ticket-promoted-11", "Ticket - Promoted (loss-policy=freeze, granted)"),
SchedulerTest("ticket-promoted-12", "Ticket - Promoted (loss-policy=freeze, revoked)"),
SchedulerTest("ticket-promoted-13", "Ticket - Promoted (loss-policy=stop, standby, granted)"),
SchedulerTest("ticket-promoted-14", "Ticket - Promoted (loss-policy=stop, granted, standby)"),
SchedulerTest("ticket-promoted-15", "Ticket - Promoted (loss-policy=stop, standby, revoked)"),
SchedulerTest("ticket-promoted-16", "Ticket - Promoted (loss-policy=demote, standby, granted)"),
SchedulerTest("ticket-promoted-17", "Ticket - Promoted (loss-policy=demote, granted, standby)"),
SchedulerTest("ticket-promoted-18", "Ticket - Promoted (loss-policy=demote, standby, revoked)"),
SchedulerTest("ticket-promoted-19", "Ticket - Promoted (loss-policy=fence, standby, granted)"),
SchedulerTest("ticket-promoted-20", "Ticket - Promoted (loss-policy=fence, granted, standby)"),
SchedulerTest("ticket-promoted-21", "Ticket - Promoted (loss-policy=fence, standby, revoked)"),
SchedulerTest("ticket-promoted-22", "Ticket - Promoted (loss-policy=freeze, standby, granted)"),
SchedulerTest("ticket-promoted-23", "Ticket - Promoted (loss-policy=freeze, granted, standby)"),
SchedulerTest("ticket-promoted-24", "Ticket - Promoted (loss-policy=freeze, standby, revoked)"),
]),
SchedulerTestGroup([
SchedulerTest("ticket-rsc-sets-1", "Ticket - Resource sets (1 ticket, initial)"),
SchedulerTest("ticket-rsc-sets-2", "Ticket - Resource sets (1 ticket, granted)"),
SchedulerTest("ticket-rsc-sets-3", "Ticket - Resource sets (1 ticket, revoked)"),
SchedulerTest("ticket-rsc-sets-4", "Ticket - Resource sets (2 tickets, initial)"),
SchedulerTest("ticket-rsc-sets-5", "Ticket - Resource sets (2 tickets, granted)"),
SchedulerTest("ticket-rsc-sets-6", "Ticket - Resource sets (2 tickets, granted)"),
SchedulerTest("ticket-rsc-sets-7", "Ticket - Resource sets (2 tickets, revoked)"),
SchedulerTest("ticket-rsc-sets-8", "Ticket - Resource sets (1 ticket, standby, granted)"),
SchedulerTest("ticket-rsc-sets-9", "Ticket - Resource sets (1 ticket, granted, standby)"),
SchedulerTest("ticket-rsc-sets-10", "Ticket - Resource sets (1 ticket, standby, revoked)"),
SchedulerTest("ticket-rsc-sets-11", "Ticket - Resource sets (2 tickets, standby, granted)"),
SchedulerTest("ticket-rsc-sets-12", "Ticket - Resource sets (2 tickets, standby, granted)"),
SchedulerTest("ticket-rsc-sets-13", "Ticket - Resource sets (2 tickets, granted, standby)"),
SchedulerTest("ticket-rsc-sets-14", "Ticket - Resource sets (2 tickets, standby, revoked)"),
SchedulerTest("cluster-specific-params", "Cluster-specific instance attributes based on rules"),
SchedulerTest("site-specific-params", "Site-specific instance attributes based on rules"),
]),
SchedulerTestGroup([
SchedulerTest("template-1", "Template - 1"),
SchedulerTest("template-2", "Template - 2"),
SchedulerTest("template-3", "Template - 3 (merge operations)"),
SchedulerTest("template-coloc-1", "Template - Colocation 1"),
SchedulerTest("template-coloc-2", "Template - Colocation 2"),
SchedulerTest("template-coloc-3", "Template - Colocation 3"),
SchedulerTest("template-order-1", "Template - Order 1"),
SchedulerTest("template-order-2", "Template - Order 2"),
SchedulerTest("template-order-3", "Template - Order 3"),
SchedulerTest("template-ticket", "Template - Ticket"),
SchedulerTest("template-rsc-sets-1", "Template - Resource Sets 1"),
SchedulerTest("template-rsc-sets-2", "Template - Resource Sets 2"),
SchedulerTest("template-rsc-sets-3", "Template - Resource Sets 3"),
SchedulerTest("template-rsc-sets-4", "Template - Resource Sets 4"),
SchedulerTest("template-clone-primitive", "Cloned primitive from template"),
SchedulerTest("template-clone-group", "Cloned group from template"),
SchedulerTest("location-sets-templates", "Resource sets and templates - Location"),
SchedulerTest("tags-coloc-order-1", "Tags - Colocation and Order (Simple)"),
SchedulerTest("tags-coloc-order-2", "Tags - Colocation and Order (Resource Sets with Templates)"),
SchedulerTest("tags-location", "Tags - Location"),
SchedulerTest("tags-ticket", "Tags - Ticket"),
]),
SchedulerTestGroup([
SchedulerTest("container-1", "Container - initial"),
SchedulerTest("container-2", "Container - monitor failed"),
SchedulerTest("container-3", "Container - stop failed"),
SchedulerTest("container-4", "Container - reached migration-threshold"),
SchedulerTest("container-group-1", "Container in group - initial"),
SchedulerTest("container-group-2", "Container in group - monitor failed"),
SchedulerTest("container-group-3", "Container in group - stop failed"),
SchedulerTest("container-group-4", "Container in group - reached migration-threshold"),
SchedulerTest("container-is-remote-node", "Place resource within container when container is remote-node"),
SchedulerTest("bug-rh-1097457", "Kill user defined container/contents ordering"),
SchedulerTest("bug-cl-5247", "Graph loop when recovering m/s resource in a container"),
SchedulerTest("bundle-order-startup", "Bundle startup ordering"),
SchedulerTest("bundle-order-partial-start",
"Bundle startup ordering when some dependencies are already running"),
SchedulerTest("bundle-order-partial-start-2",
"Bundle startup ordering when some dependencies and the container are already running"),
SchedulerTest("bundle-order-stop", "Bundle stop ordering"),
SchedulerTest("bundle-order-partial-stop", "Bundle startup ordering when some dependencies are already stopped"),
SchedulerTest("bundle-order-stop-on-remote", "Stop nested resource after bringing up the connection"),
SchedulerTest("bundle-order-startup-clone", "Prevent startup because bundle isn't promoted"),
SchedulerTest("bundle-order-startup-clone-2", "Bundle startup with clones"),
SchedulerTest("bundle-order-stop-clone", "Stop bundle because clone is stopping"),
SchedulerTest("bundle-interleave-start", "Interleave bundle starts"),
SchedulerTest("bundle-interleave-promote", "Interleave bundle promotes"),
SchedulerTest("bundle-nested-colocation", "Colocation of nested connection resources"),
SchedulerTest("bundle-order-fencing",
"Order pseudo bundle fencing after parent node fencing if both are happening"),
SchedulerTest("bundle-probe-order-1", "order 1"),
SchedulerTest("bundle-probe-order-2", "order 2"),
SchedulerTest("bundle-probe-order-3", "order 3"),
SchedulerTest("bundle-probe-remotes", "Ensure remotes get probed too"),
SchedulerTest("bundle-replicas-change", "Change bundle from 1 replica to multiple"),
SchedulerTest("bundle-connection-with-container", "Don't move a container due to connection preferences"),
SchedulerTest("nested-remote-recovery", "Recover bundle's container hosted on remote node"),
SchedulerTest("bundle-promoted-location-1",
"Promotable bundle, positive location"),
SchedulerTest("bundle-promoted-location-2",
"Promotable bundle, negative location"),
SchedulerTest("bundle-promoted-location-3",
"Promotable bundle, positive location for promoted role"),
SchedulerTest("bundle-promoted-location-4",
"Promotable bundle, negative location for promoted role"),
SchedulerTest("bundle-promoted-location-5",
"Promotable bundle, positive location for unpromoted role"),
SchedulerTest("bundle-promoted-location-6",
"Promotable bundle, negative location for unpromoted role"),
SchedulerTest("bundle-promoted-colocation-1",
"Primary promoted bundle, dependent primitive (mandatory coloc)"),
SchedulerTest("bundle-promoted-colocation-2",
"Primary promoted bundle, dependent primitive (optional coloc)"),
SchedulerTest("bundle-promoted-colocation-3",
"Dependent promoted bundle, primary primitive (mandatory coloc)"),
SchedulerTest("bundle-promoted-colocation-4",
"Dependent promoted bundle, primary primitive (optional coloc)"),
SchedulerTest("bundle-promoted-colocation-5",
"Primary and dependent promoted bundle instances (mandatory coloc)"),
SchedulerTest("bundle-promoted-colocation-6",
"Primary and dependent promoted bundle instances (optional coloc)"),
SchedulerTest("bundle-promoted-anticolocation-1",
"Primary promoted bundle, dependent primitive (mandatory anti)"),
SchedulerTest("bundle-promoted-anticolocation-2",
"Primary promoted bundle, dependent primitive (optional anti)"),
SchedulerTest("bundle-promoted-anticolocation-3",
"Dependent promoted bundle, primary primitive (mandatory anti)"),
SchedulerTest("bundle-promoted-anticolocation-4",
"Dependent promoted bundle, primary primitive (optional anti)"),
SchedulerTest("bundle-promoted-anticolocation-5",
"Primary and dependent promoted bundle instances (mandatory anti)"),
SchedulerTest("bundle-promoted-anticolocation-6",
"Primary and dependent promoted bundle instances (optional anti)"),
]),
SchedulerTestGroup([
SchedulerTest("whitebox-fail1", "Fail whitebox container rsc"),
SchedulerTest("whitebox-fail2", "Fail cluster connection to guest node"),
SchedulerTest("whitebox-fail3", "Failed containers should not run nested on remote nodes"),
SchedulerTest("whitebox-start", "Start whitebox container with resources assigned to it"),
SchedulerTest("whitebox-stop", "Stop whitebox container with resources assigned to it"),
SchedulerTest("whitebox-move", "Move whitebox container with resources assigned to it"),
SchedulerTest("whitebox-asymmetric", "Verify connection rsc opts-in based on container resource"),
SchedulerTest("whitebox-ms-ordering", "Verify promote/demote can not occur before connection is established"),
SchedulerTest("whitebox-ms-ordering-move", "Stop/Start cycle within a moving container"),
SchedulerTest("whitebox-orphaned", "Properly shutdown orphaned whitebox container"),
SchedulerTest("whitebox-orphan-ms", "Properly tear down orphan ms resources on remote-nodes"),
SchedulerTest("whitebox-unexpectedly-running", "Recover container nodes the cluster did not start"),
SchedulerTest("whitebox-migrate1", "Migrate both container and connection resource"),
SchedulerTest("whitebox-imply-stop-on-fence",
"imply stop action on container node rsc when host node is fenced"),
SchedulerTest("whitebox-nested-group", "Verify guest remote-node works nested in a group"),
SchedulerTest("guest-node-host-dies", "Verify guest node is recovered if host goes away"),
SchedulerTest("guest-node-cleanup", "Order guest node connection recovery after container probe"),
SchedulerTest("guest-host-not-fenceable", "Actions on guest node are unrunnable if host is unclean and cannot be fenced"),
]),
SchedulerTestGroup([
SchedulerTest("remote-startup-probes", "Baremetal remote-node startup probes"),
SchedulerTest("remote-startup", "Startup a newly discovered remote-nodes with no status"),
SchedulerTest("remote-fence-unclean", "Fence unclean baremetal remote-node"),
SchedulerTest("remote-fence-unclean2",
"Fence baremetal remote-node after cluster node fails and connection can not be recovered"),
SchedulerTest("remote-fence-unclean-3", "Probe failed remote nodes (triggers fencing)"),
SchedulerTest("remote-move", "Move remote-node connection resource"),
SchedulerTest("remote-disable", "Disable a baremetal remote-node"),
SchedulerTest("remote-probe-disable", "Probe then stop a baremetal remote-node"),
SchedulerTest("remote-orphaned", "Properly shutdown orphaned connection resource"),
SchedulerTest("remote-orphaned2",
"verify we can handle orphaned remote connections with active resources on the remote"),
SchedulerTest("remote-recover", "Recover connection resource after cluster-node fails"),
SchedulerTest("remote-stale-node-entry",
"Make sure we properly handle leftover remote-node entries in the node section"),
SchedulerTest("remote-partial-migrate",
"Make sure partial migrations are handled before ops on the remote node"),
SchedulerTest("remote-partial-migrate2",
"Make sure partial migration target is prefered for remote connection"),
SchedulerTest("remote-recover-fail", "Make sure start failure causes fencing if rsc are active on remote"),
SchedulerTest("remote-start-fail",
"Make sure a start failure does not result in fencing if no active resources are on remote"),
SchedulerTest("remote-unclean2",
"Make monitor failure always results in fencing, even if no rsc are active on remote"),
SchedulerTest("remote-fence-before-reconnect", "Fence before clearing recurring monitor failure"),
SchedulerTest("remote-recovery", "Recover remote connections before attempting demotion"),
SchedulerTest("remote-recover-connection", "Optimistically recovery of only the connection"),
SchedulerTest("remote-recover-all", "Fencing when the connection has no home"),
SchedulerTest("remote-recover-no-resources", "Fencing when the connection has no home and no active resources"),
SchedulerTest("remote-recover-unknown",
"Fencing when the connection has no home and the remote has no operation history"),
SchedulerTest("remote-reconnect-delay", "Waiting for remote reconnect interval to expire"),
SchedulerTest("remote-connection-unrecoverable",
"Remote connection host must be fenced, with connection unrecoverable"),
SchedulerTest("remote-connection-shutdown", "Remote connection shutdown"),
SchedulerTest("cancel-behind-moving-remote",
"Route recurring monitor cancellations through original node of a moving remote connection"),
]),
SchedulerTestGroup([
SchedulerTest("resource-discovery", "Exercises resource-discovery location constraint option"),
SchedulerTest("rsc-discovery-per-node", "Disable resource discovery per node"),
SchedulerTest("shutdown-lock", "Ensure shutdown lock works properly"),
SchedulerTest("shutdown-lock-expiration", "Ensure shutdown lock expiration works properly"),
]),
SchedulerTestGroup([
SchedulerTest("op-defaults", "Test op_defaults conditional expressions"),
SchedulerTest("op-defaults-2", "Test op_defaults AND'ed conditional expressions"),
SchedulerTest("op-defaults-3", "Test op_defaults precedence"),
SchedulerTest("rsc-defaults", "Test rsc_defaults conditional expressions"),
SchedulerTest("rsc-defaults-2", "Test rsc_defaults conditional expressions without type"),
]),
SchedulerTestGroup([
SchedulerTest("stop-all-resources", "Test stop-all-resources=true"),
]),
SchedulerTestGroup([
SchedulerTest("ocf_degraded-remap-ocf_ok", "Test degraded remapped to OK"),
SchedulerTest("ocf_degraded_promoted-remap-ocf_ok", "Test degraded promoted remapped to OK"),
]),
]
TESTS_64BIT = [
SchedulerTestGroup([
SchedulerTest("year-2038", "Check handling of timestamps beyond 2038-01-19 03:14:08 UTC"),
]),
]
def is_executable(path):
"""Check whether a file at a given path is executable."""
try:
return os.stat(path)[stat.ST_MODE] & stat.S_IXUSR
except OSError:
return False
def diff(file1, file2, **kwargs):
"""Call diff on two files."""
return subprocess.call(["diff", "-u", "-N", "--ignore-all-space",
"--ignore-blank-lines", file1, file2], **kwargs)
def sort_file(filename):
"""Sort a file alphabetically."""
with io.open(filename, "rt", encoding="utf-8") as f:
lines = sorted(f)
with io.open(filename, "wt", encoding="utf-8") as f:
f.writelines(lines)
def remove_files(filenames):
"""Remove a list of files."""
for filename in filenames:
try:
os.remove(filename)
except OSError:
pass
def normalize(filename):
"""Remove text from a file that isn't important for comparison."""
if not hasattr(normalize, "patterns"):
normalize.patterns = [
re.compile(r'crm_feature_set="[^"]*"'),
re.compile(r'batch-limit="[0-9]*"')
]
if not os.path.isfile(filename):
return
with io.open(filename, "rt", encoding="utf-8") as f:
lines = f.readlines()
with io.open(filename, "wt", encoding="utf-8") as f:
for line in lines:
for pattern in normalize.patterns:
line = pattern.sub("", line)
f.write(line)
def cat(filename, dest=sys.stdout):
"""Copy a file to a destination file descriptor."""
with io.open(filename, "rt", encoding="utf-8") as f:
shutil.copyfileobj(f, dest)
class CtsScheduler:
"""Regression tests for Pacemaker's scheduler."""
def _parse_args(self, argv):
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(description="Regression tests for Pacemaker's scheduler")
parser.add_argument('-V', '--verbose', action='count',
help='Display any differences from expected output')
parser.add_argument('--run', metavar='TEST',
help=('Run only single specified test (any further '
'arguments will be passed to crm_simulate)'))
parser.add_argument('--update', action='store_true',
help='Update expected results with actual results')
parser.add_argument('-b', '--binary', metavar='PATH',
help='Specify path to crm_simulate')
parser.add_argument('-i', '--io-dir', metavar='PATH',
help='Specify path to regression test data directory')
parser.add_argument('-o', '--out-dir', metavar='PATH',
help='Specify where intermediate and output files should go')
parser.add_argument('-v', '--valgrind', action='store_true',
help='Run all commands under valgrind')
parser.add_argument('--valgrind-dhat', action='store_true',
help='Run all commands under valgrind with heap analyzer')
parser.add_argument('--valgrind-skip-output', action='store_true',
help='If running under valgrind, do not display output')
parser.add_argument('--testcmd-options', metavar='OPTIONS', default='',
help='Additional options for command under test')
# argparse can't handle "everything after --run TEST", so grab that
self.single_test_args = []
narg = 0
for arg in argv:
narg += 1
if arg == '--run':
(argv, self.single_test_args) = (argv[:narg + 1], argv[narg + 1:])
break
self.args = parser.parse_args(argv[1:])
def _error(self, s):
"""Print an error message."""
print(f" * ERROR: {s}")
def _failed(self, s):
"""Print a failure message."""
print(f" * FAILED: {s}")
def _get_valgrind_cmd(self):
"""Return command arguments needed (or not) to run valgrind."""
if self.args.valgrind:
os.environ['G_SLICE'] = "always-malloc"
return [
"valgrind",
"-q",
"--gen-suppressions=all",
"--time-stamp=yes",
"--trace-children=no",
"--show-reachable=no",
"--leak-check=full",
"--num-callers=20",
f"--suppressions={self.test_home}/valgrind-pcmk.suppressions"
]
if self.args.valgrind_dhat:
os.environ['G_SLICE'] = "always-malloc"
return [
"valgrind",
"--tool=exp-dhat",
"--time-stamp=yes",
"--trace-children=no",
"--show-top-n=100",
"--num-callers=4"
]
return []
def _get_simulator_cmd(self):
"""Locate the simulation binary."""
if self.args.binary is None:
# pylint: disable=protected-access
self.args.binary = f"{BuildOptions._BUILD_DIR}/tools/crm_simulate"
if not is_executable(self.args.binary):
self.args.binary = f"{BuildOptions.SBIN_DIR}/crm_simulate"
if not is_executable(self.args.binary):
# @TODO it would be more pythonic to raise an exception
self._error(f"Test binary {self.args.binary} not found")
sys.exit(ExitStatus.NOT_INSTALLED)
return [self.args.binary] + shlex.split(self.args.testcmd_options)
def set_schema_env(self):
"""Ensure schema directory environment variable is set, if possible."""
try:
return os.environ['PCMK_schema_directory']
except KeyError:
# pylint: disable=protected-access
for d in [os.path.join(BuildOptions._BUILD_DIR, "xml"),
BuildOptions.SCHEMA_DIR]:
if not os.path.isdir(d):
continue
os.environ['PCMK_schema_directory'] = d
return d
return None
def __init__(self, argv=sys.argv):
"""Create a new CtsScheduler instance."""
# Ensure all command output is in portable locale for comparison
os.environ['LC_ALL'] = "C"
self._parse_args(argv)
# Where this executable lives
self.test_home = os.path.dirname(os.path.realpath(argv[0]))
# Where test data resides
if self.args.io_dir is None:
self.args.io_dir = os.path.join(self.test_home, "scheduler")
self.xml_input_dir = os.path.join(self.args.io_dir, "xml")
self.expected_dir = os.path.join(self.args.io_dir, "exp")
self.dot_expected_dir = os.path.join(self.args.io_dir, "dot")
self.scores_dir = os.path.join(self.args.io_dir, "scores")
self.summary_dir = os.path.join(self.args.io_dir, "summary")
self.stderr_expected_dir = os.path.join(self.args.io_dir, "stderr")
# Create a temporary directory to store diff file
self.failed_dir = tempfile.mkdtemp(prefix='cts-scheduler_')
# Where to store generated files
if self.args.out_dir is None:
self.args.out_dir = self.args.io_dir
self.failed_filename = os.path.join(self.failed_dir, "test-output.diff")
else:
self.failed_filename = os.path.join(self.args.out_dir, "test-output.diff")
os.environ['CIB_shadow_dir'] = self.args.out_dir
self.failed_file = None
self.outfile_out_dir = os.path.join(self.args.out_dir, "out")
self.dot_out_dir = os.path.join(self.args.out_dir, "dot")
self.scores_out_dir = os.path.join(self.args.out_dir, "scores")
self.summary_out_dir = os.path.join(self.args.out_dir, "summary")
self.stderr_out_dir = os.path.join(self.args.out_dir, "stderr")
self.valgrind_out_dir = os.path.join(self.args.out_dir, "valgrind")
# Single test mode (if requested)
try:
# User can give test base name or file name of a test input
self.args.run = os.path.splitext(os.path.basename(self.args.run))[0]
except (AttributeError, TypeError):
pass # --run was not specified
self.set_schema_env()
# Arguments needed (or not) to run commands
self.valgrind_args = self._get_valgrind_cmd()
self.simulate_args = self._get_simulator_cmd()
# Test counters
self.num_failed = 0
self.num_tests = 0
# Ensure that the main output directory exists
# We don't want to create it with os.makedirs below
if not os.path.isdir(self.args.out_dir):
self._error("Output directory missing; can't create output files")
sys.exit(ExitStatus.CANTCREAT)
# Create output subdirectories if they don't exist
try:
os.makedirs(self.outfile_out_dir, 0o755, True)
os.makedirs(self.dot_out_dir, 0o755, True)
os.makedirs(self.scores_out_dir, 0o755, True)
os.makedirs(self.summary_out_dir, 0o755, True)
os.makedirs(self.stderr_out_dir, 0o755, True)
if self.valgrind_args:
os.makedirs(self.valgrind_out_dir, 0o755, True)
except OSError as ex:
self._error(f"Unable to create output subdirectory: {ex}")
remove_files([
self.outfile_out_dir,
self.dot_out_dir,
self.scores_out_dir,
self.summary_out_dir,
self.stderr_out_dir,
])
sys.exit(ExitStatus.CANTCREAT)
def _compare_files(self, filename1, filename2):
"""Add any file differences to failed results."""
if diff(filename1, filename2, stdout=subprocess.DEVNULL) != 0:
diff(filename1, filename2, stdout=self.failed_file, stderr=subprocess.DEVNULL)
self.failed_file.write("\n")
return True
return False
def _file_missing(self, path):
"""Return True if path does not exist or is empty."""
return not os.path.isfile(path) or os.path.getsize(path) == 0
def run_one(self, test_name, test_desc, test_args):
"""Run one scheduler test."""
# pylint: disable=too-many-locals
print(f" Test {f'{test_name}:':41} {test_desc}")
did_fail = False
self.num_tests += 1
# Test inputs
input_filename = os.path.join(self.xml_input_dir, f"{test_name}.xml")
expected_filename = os.path.join(self.expected_dir, f"{test_name}.exp")
dot_expected_filename = os.path.join(self.dot_expected_dir, f"{test_name}.dot")
scores_filename = os.path.join(self.scores_dir, f"{test_name}.scores")
summary_filename = os.path.join(self.summary_dir, f"{test_name}.summary")
stderr_expected_filename = os.path.join(self.stderr_expected_dir, f"{test_name}.stderr")
# (Intermediate) test outputs
output_filename = os.path.join(self.outfile_out_dir, f"{test_name}.out")
dot_output_filename = os.path.join(self.dot_out_dir, f"{test_name}.dot.pe")
score_output_filename = os.path.join(self.scores_out_dir, f"{test_name}.scores.pe")
summary_output_filename = os.path.join(self.summary_out_dir, f"{test_name}.summary.pe")
stderr_output_filename = os.path.join(self.stderr_out_dir, f"{test_name}.stderr.pe")
valgrind_output_filename = os.path.join(self.valgrind_out_dir, f"{test_name}.valgrind")
# Common arguments for running test
test_cmd = []
if self.valgrind_args:
test_cmd = self.valgrind_args + [f"--log-file={valgrind_output_filename}"]
test_cmd += self.simulate_args
# @TODO It would be more pythonic to raise exceptions for errors,
# then perhaps it would be nice to make a single-test class
# Ensure necessary test inputs exist
if not os.path.isfile(input_filename):
self._error("No input")
self.num_failed += 1
return ExitStatus.NOINPUT
if not self.args.update and not os.path.isfile(expected_filename):
self._error("no stored output")
return ExitStatus.NOINPUT
# Run simulation to generate summary output
test_cmd_full = test_cmd + ['-x', input_filename, '-S'] + test_args
if self.args.run: # Single test mode
print(" ".join(test_cmd_full))
with io.open(summary_output_filename, "wt", encoding="utf-8") as f:
subprocess.run(test_cmd_full, stdout=f, stderr=subprocess.STDOUT,
env=os.environ, check=False)
if self.args.run:
cat(summary_output_filename)
# Re-run simulation to generate dot, graph, and scores
test_cmd_full = test_cmd + ['-x', input_filename,
'-D', dot_output_filename,
'-G', output_filename,
'-sSQ'] + test_args
with io.open(stderr_output_filename, "wt", encoding="utf-8") as f_stderr, \
io.open(score_output_filename, "wt", encoding="utf-8") as f_score:
rc = subprocess.call(test_cmd_full, stdout=f_score, stderr=f_stderr, env=os.environ)
# Check for test command failure
if rc != ExitStatus.OK:
self._failed(f"Test returned: {rc}")
did_fail = True
print(" ".join(test_cmd_full))
# Check for valgrind errors
- if self.valgrind_args and not self.args.valgrind_skip_output:
+ if self.valgrind_args:
if os.path.getsize(valgrind_output_filename) > 0:
self._failed("Valgrind reported errors")
did_fail = True
cat(valgrind_output_filename)
-
- remove_files([valgrind_output_filename])
+ else:
+ remove_files([valgrind_output_filename])
# Check for core dump
if os.path.isfile("core"):
self._failed(f"Core-file detected: core.{test_name}")
did_fail = True
os.rename("core", f"{self.test_home}/core.{test_name}")
# Check any stderr output
if os.path.isfile(stderr_expected_filename):
if self._compare_files(stderr_expected_filename, stderr_output_filename):
self._failed("stderr changed")
did_fail = True
elif os.path.getsize(stderr_output_filename) > 0:
self._failed("Output was written to stderr")
did_fail = True
cat(stderr_output_filename)
remove_files([stderr_output_filename])
# Check whether output graph exists, and normalize it
if self._file_missing(output_filename):
self._error("No graph produced")
did_fail = True
self.num_failed += 1
remove_files([output_filename])
return ExitStatus.ERROR
normalize(output_filename)
# Check whether dot output exists, and sort it
if self._file_missing(dot_output_filename):
self._error("No dot-file summary produced")
did_fail = True
self.num_failed += 1
remove_files([dot_output_filename, output_filename])
return ExitStatus.ERROR
with io.open(dot_output_filename, "rt", encoding="utf-8") as f:
first_line = f.readline() # "digraph" line with opening brace
lines = f.readlines()
last_line = lines[-1] # closing brace
del lines[-1]
lines = sorted(set(lines)) # unique sort
with io.open(dot_output_filename, "wt", encoding="utf-8") as f:
f.write(first_line)
f.writelines(lines)
f.write(last_line)
# Check whether score output exists, and sort it
if self._file_missing(score_output_filename):
self._error("No allocation scores produced")
did_fail = True
self.num_failed += 1
remove_files([score_output_filename, output_filename])
return ExitStatus.ERROR
sort_file(score_output_filename)
if self.args.update:
shutil.copyfile(output_filename, expected_filename)
shutil.copyfile(dot_output_filename, dot_expected_filename)
shutil.copyfile(score_output_filename, scores_filename)
shutil.copyfile(summary_output_filename, summary_filename)
print(" Updated expected outputs")
if self._compare_files(summary_filename, summary_output_filename):
self._failed("summary changed")
did_fail = True
if self._compare_files(dot_expected_filename, dot_output_filename):
self._failed("dot-file summary changed")
did_fail = True
else:
remove_files([dot_output_filename])
if self._compare_files(expected_filename, output_filename):
self._failed("xml-file changed")
did_fail = True
if self._compare_files(scores_filename, score_output_filename):
self._failed("scores-file changed")
did_fail = True
remove_files([output_filename,
dot_output_filename,
score_output_filename,
summary_output_filename])
if did_fail:
self.num_failed += 1
return ExitStatus.ERROR
return ExitStatus.OK
def run_all(self):
"""Run all defined tests."""
if platform.architecture()[0] == "64bit":
TESTS.extend(TESTS_64BIT)
for group in TESTS:
for test in group.tests:
self.run_one(test.name, test.desc, test.args)
print()
def _print_summary(self):
"""Print a summary of parameters for this test run."""
print(f"Test home is:\t{self.test_home}")
print(f"Test binary is:\t{self.args.binary}")
if 'PCMK_schema_directory' in os.environ:
print(f"Schema home is:\t{os.environ['PCMK_schema_directory']}")
if self.valgrind_args:
print("Activating memory testing with valgrind")
print()
def _test_results(self):
"""Report test results."""
if self.num_failed == 0:
shutil.rmtree(self.failed_dir)
return ExitStatus.OK
if self._file_missing(self.failed_filename):
self._error(f"{self.num_failed} (of {self.num_tests}) tests failed (no diff results)")
if os.path.isfile(self.failed_filename):
shutil.rmtree(self.failed_dir)
elif self.args.verbose:
self._error(f"Results of {self.num_failed} failed tests (out of {self.num_tests}):")
cat(self.failed_filename)
else:
self._error(f"Results of {self.num_failed} failed tests (out of {self.num_tests}) "
f"are in {self.failed_filename}")
self._error("Use -V to display them after running the tests")
return ExitStatus.ERROR
def find_test(self, name):
"""Return the SchedulerTest object with the given name."""
if platform.architecture()[0] == "64bit":
TESTS.extend(TESTS_64BIT)
for group in TESTS:
for test in group.tests:
if test.name == name:
return test
return None
def run(self):
"""Run test(s) as specified."""
# Check for pre-existing core so we don't think it's from us
if os.path.exists("core"):
self._failed(f"Can't run with core already present in {self.test_home}")
return ExitStatus.OSFILE
self._print_summary()
# Zero out the error log
# pylint: disable=consider-using-with
self.failed_file = io.open(self.failed_filename, "wt", encoding="utf-8")
if self.args.run is None:
print(f"Performing the following tests from {self.args.io_dir}")
print()
self.run_all()
print()
self.failed_file.close()
rc = self._test_results()
else:
# Find the test we were asked to run
test = self.find_test(self.args.run)
if test is None:
print(f"No test named {self.args.run}")
return ExitStatus.INVALID_PARAM
# If no arguments were given on the command line, default to the ones
# contained in the test
if self.single_test_args:
args = self.single_test_args
else:
args = test.args
rc = self.run_one(test.name, test.desc, args)
self.failed_file.close()
if self.num_failed > 0:
print(f"\nFailures:\nThese have also been written to: {self.failed_filename}\n")
cat(self.failed_filename)
shutil.rmtree(self.failed_dir)
return rc
if __name__ == "__main__":
sys.exit(CtsScheduler().run())
# vim: set filetype=python expandtab tabstop=4 softtabstop=4 shiftwidth=4 textwidth=120:
diff --git a/devel/Makefile.am b/devel/Makefile.am
index 8158734a6a..461424b7b8 100644
--- a/devel/Makefile.am
+++ b/devel/Makefile.am
@@ -1,307 +1,308 @@
#
# Copyright 2020-2025 the Pacemaker project contributors
#
# The version control history for this file may have further details.
#
# This source code is licensed under the GNU General Public License version 2
# or later (GPLv2+) WITHOUT ANY WARRANTY.
#
include $(top_srcdir)/mk/common.mk
include $(top_srcdir)/mk/release.mk
# Coccinelle is a tool that takes special patch-like files (called semantic patches) and
# applies them throughout a source tree. This is useful when refactoring, changing APIs,
# catching dangerous or incorrect code, and other similar tasks. It's not especially
# easy to write a semantic patch but most users should only be concerned about running
# the target and inspecting the results.
#
# Documentation (including examples, which are the most useful):
# https://coccinelle.gitlabpages.inria.fr/website/docs/
#
# Run the "make cocci" target to just output what would be done, or "make cocci-inplace"
# to apply the changes to the source tree.
#
# COCCI_FILES may be set on the command line, if you want to test just a single file
# while it's under development. Otherwise, it is a list of all the files that are ready
# to be run.
#
# ref-passed-variables-inited.cocci seems to be returning some false positives around
# GHashTableIters, so it is disabled for the moment.
COCCI_FILES ?= coccinelle/string-any-of.cocci \
coccinelle/string-empty.cocci \
coccinelle/string-null-matches.cocci \
coccinelle/use-func.cocci
dist_noinst_SCRIPTS = coccinelle/test/testrunner.sh
EXTRA_DIST = README \
gdbhelpers \
$(COCCI_FILES) \
coccinelle/ref-passed-variables-inited.cocci \
coccinelle/rename-fn.cocci \
coccinelle/test/ref-passed-variables-inited.input.c \
coccinelle/test/ref-passed-variables-inited.output
# Any file in this list is allowed to use any of the pcmk__ internal functions.
# Coccinelle can use any transformation that depends on "internal" to rewrite
# code to use the internal functions.
MAY_USE_INTERNAL_FILES = $(shell find .. -path "../lib/*.c" -o -path "../lib/*private.h" -o -path "../tools/*.c" -o -path "../daemons/*.c" -o -path '../include/pcmki/*h' -o -name '*internal.h')
# And then any file in this list is public API, which may not use internal
# functions. Thus, only those transformations that do not depend on "internal"
# may be applied.
OTHER_FILES = $(shell find ../include -name '*h' -a \! -name '*internal.h' -a \! -path '../include/pcmki/*')
.PHONY: cocci
cocci:
-for cf in $(COCCI_FILES); do \
for f in $(MAY_USE_INTERNAL_FILES); do \
spatch $(_SPATCH_FLAGS) -D internal --very-quiet --local-includes --preprocess --sp-file $$cf $$f; \
done ; \
for f in $(OTHER_FILES); do \
spatch $(_SPATCH_FLAGS) --very-quiet --local-includes --preprocess --sp-file $$cf $$f; \
done ; \
done
.PHONY: cocci-inplace
cocci-inplace:
$(MAKE) $(AM_MAKEFLAGS) _SPATCH_FLAGS=--in-place cocci
.PHONY: cocci-test
cocci-test:
for f in coccinelle/test/*.c; do \
coccinelle/test/testrunner.sh $$f; \
done
#
# Static analysis
#
## clang
# See scan-build(1) for possible checkers (leave empty to use default set)
CLANG_checkers ?=
.PHONY: clang
clang:
OUT=$$(cd $(top_builddir) \
&& scan-build $(CLANG_checkers:%=-enable-checker %) \
$(MAKE) $(AM_MAKEFLAGS) CFLAGS="-std=c99 $(CFLAGS)" \
clean all 2>&1); \
REPORT=$$(echo "$$OUT" \
| sed -n -e "s/.*'scan-view \(.*\)'.*/\1/p"); \
[ -z "$$REPORT" ] && echo "$$OUT" || scan-view "$$REPORT"
## coverity
# Aggressiveness (low, medium, or high)
COVLEVEL ?= low
# Generated outputs
COVERITY_DIR = $(abs_top_builddir)/coverity-$(TAG)
COVTAR = $(abs_top_builddir)/$(PACKAGE)-coverity-$(TAG).tgz
COVEMACS = $(abs_top_builddir)/$(TAG).coverity
COVHTML = $(COVERITY_DIR)/output/errors
# Coverity outputs are phony so they get rebuilt every invocation
.PHONY: $(COVERITY_DIR)
$(COVERITY_DIR): coverity-clean
$(MAKE) $(AM_MAKEFLAGS) -C $(top_builddir) init core-clean
$(AM_V_GEN)cd $(top_builddir) \
&& cov-build --dir "$@" $(MAKE) $(AM_MAKEFLAGS) core
# Public coverity instance
.PHONY: $(COVTAR)
$(COVTAR): $(COVERITY_DIR)
$(AM_V_GEN)tar czf "$@" --transform="s@.*$(TAG)@cov-int@" "$<"
.PHONY: coverity
coverity: $(COVTAR)
@echo "Now go to https://scan.coverity.com/users/sign_in and upload:"
@echo " $(COVTAR)"
@echo "then make clean at the top level"
# Licensed coverity instance
#
# The prerequisites are a little hacky; rather than actually required, some
# of them are designed so that things execute in the proper order (which is
# not the same as GNU make's order-only prerequisites).
.PHONY: coverity-analyze
coverity-analyze: $(COVERITY_DIR)
@echo ""
@echo "Analyzing (waiting for coverity license if necessary) ..."
cd $(top_builddir) && cov-analyze --dir "$<" --wait-for-license \
--security --aggressiveness-level "$(COVLEVEL)"
.PHONY: $(COVEMACS)
$(COVEMACS): coverity-analyze
$(AM_V_GEN)cd $(top_builddir) \
&& cov-format-errors --dir "$(COVERITY_DIR)" --emacs-style > "$@"
.PHONY: $(COVHTML)
$(COVHTML): $(COVEMACS)
$(AM_V_GEN)cd $(top_builddir) \
&& cov-format-errors --dir "$(COVERITY_DIR)" --html-output "$@"
.PHONY: coverity-corp
coverity-corp: $(COVHTML)
$(MAKE) $(AM_MAKEFLAGS) -C $(top_builddir) core-clean
@echo "Done. See:"
@echo " file://$(COVHTML)/index.html"
@echo "When no longer needed, make coverity-clean"
# Remove all outputs regardless of tag
.PHONY: coverity-clean
coverity-clean:
- -rm -rf "$(abs_builddir)"/coverity-* \
- "$(abs_builddir)"/$(PACKAGE)-coverity-*.tgz \
- "$(abs_builddir)"/*.coverity
+ -rm -rf "$(abs_top_builddir)"/coverity-* \
+ "$(abs_top_builddir)"/$(PACKAGE)-coverity-*.tgz \
+ "$(abs_top_builddir)"/*.coverity
## cppcheck
GLIB_CFLAGS ?= $(pkg-config --cflags glib-2.0)
GLIB_INCL_DEF_CFLAGS = $(shell echo $(GLIB_CFLAGS) \
| tr ' ' '\n' | grep '^-[IDU]' | paste -d ' ')
# Use CPPCHECK_ARGS to pass extra cppcheck options, e.g.:
# --enable={warning,style,performance,portability,information,all}
# --inconclusive --std=posix
# -DDEFAULT_CONCURRENT_FENCING_TRUE
CPPCHECK_ARGS ?=
CPPCHECK_DIRS = replace lib daemons tools
CPPCHECK_OUT = $(abs_top_builddir)/cppcheck.out
.PHONY: cppcheck
cppcheck:
cppcheck $(CPPCHECK_ARGS) -I $(top_srcdir)/include \
-I $(top_srcdir)/lib/common \
+ --check-level=exhaustive \
--include=/usr/include/qb/qblog.h \
--output-file=$(CPPCHECK_OUT) \
--max-configs=30 --inline-suppr -q \
--library=posix --library=gnu --library=gtk \
$(GLIB_INCL_DEF_CFLAGS) -D__GNUC__ \
$(foreach dir,$(CPPCHECK_DIRS),$(top_srcdir)/$(dir))
@echo "Done: See $(CPPCHECK_OUT)"
@echo "When no longer needed, make cppcheck-clean"
.PHONY: cppcheck-clean
cppcheck-clean:
-rm -f "$(CPPCHECK_OUT)"
#
# Coverage/profiling
#
COVERAGE_DIR = $(abs_top_builddir)/coverage
# Check coverage of unit tests
.PHONY: coverage
coverage: coverage-partial-clean
cd $(top_builddir) \
&& $(MAKE) $(AM_MAKEFLAGS) \
&& lcov --capture --no-external --exclude='*_test.c' --initial \
--directory lib --ignore-errors=unused \
--output-file pacemaker_base.info \
&& $(MAKE) $(AM_MAKEFLAGS) check \
&& lcov --capture --no-external --exclude='*_test.c' \
--directory lib --ignore-errors=unused \
--output-file pacemaker_test.info \
&& lcov --add-tracefile pacemaker_base.info \
--add-tracefile pacemaker_test.info \
--output-file pacemaker_total.info \
&& genhtml --output-directory $(COVERAGE_DIR) \
--show-details --title "Pacemaker library code coverage"\
pacemaker_total.info
# Check coverage of CLI regression tests
.PHONY: coverage-cts
coverage-cts: coverage-partial-clean
cd $(top_builddir) \
&& $(MAKE) $(AM_MAKEFLAGS) \
&& lcov --capture --no-external --initial --directory tools \
--output-file pacemaker_base.info \
&& cts/cts-cli \
&& lcov --capture --no-external --directory tools \
--output-file pacemaker_test.info \
&& lcov --add-tracefile pacemaker_base.info \
--add-tracefile pacemaker_test.info \
--output-file pacemaker_total.info \
&& genhtml --output-directory $(COVERAGE_DIR) \
--show-details --title "Pacemaker tools code coverage" \
pacemaker_total.info
# Remove coverage-related files that aren't needed across runs
.PHONY: coverage-partial-clean
coverage-partial-clean:
-rm -f $(top_builddir)/pacemaker_*.info
-rm -rf $(COVERAGE_DIR)
-find $(top_builddir) -name "*.gcda" -exec rm -f \{\} \;
# This target removes all coverage-related files. It is only to be run when
# done with coverage analysis and you are ready to go back to normal development,
# starting with re-running ./configure. It is not to be run in between
# "make coverage" runs.
#
# In particular, the *.gcno files are generated when the source is built.
# Removing those files will break "make coverage" until the whole source tree
# has been built and the *.gcno files generated again.
.PHONY: coverage-clean
coverage-clean: coverage-partial-clean
-find $(top_builddir) -name "*.gcno" -exec rm -f \{\} \;
# Automatic code formatting - this makes use of clang-format and the .clang-format
# config file. It's based on GNU coding, but heavily modified for our needs
# and to reflect the C coding guidelines documentation.
# Limit clang-format to these directories
INDENT_DIRS ?= .
# Extra options to pass to clang-format
INDENT_OPTS ?=
.PHONY: indent
indent:
find $(INDENT_DIRS) -type f -name "*.[ch]" \
-exec clang-format $(INDENT_OPTS) \{\} \;
#
# Check whether copyrights have been updated appropriately
# (Set COMMIT to desired commit or commit range to check, defaulting to HEAD,
# or set it empty to check uncommitted changes)
#
YEAR = $(shell date +%Y)
MODIFIED_FILES = $(shell case "$(COMMIT)" in \
[0-9a-f]*$(rparen) \
git diff-tree --no-commit-id \
--name-only "$(COMMIT)" -r ;; \
*$(rparen) \
cd "$(top_srcdir)"; \
git ls-files --modified ;; \
esac)
.PHONY: copyright
copyright:
@cd "$(top_srcdir)" && for file in $(MODIFIED_FILES); do \
if ! grep 'opyright .*$(YEAR).* Pacemaker' "$$file" \
>/dev/null 2>&1; then \
echo "$$file"; \
fi; \
done
#
# Scratch file for ad-hoc testing
#
EXTRA_PROGRAMS = scratch
nodist_scratch_SOURCES = scratch.c
scratch_LDADD = $(top_builddir)/lib/common/libcrmcommon.la
.PHONY: clean-local
clean-local: coverage-clean coverity-clean cppcheck-clean
-rm -f $(EXTRA_PROGRAMS)
diff --git a/maint/Makefile.am b/maint/Makefile.am
index fd5373f75d..881f52df72 100644
--- a/maint/Makefile.am
+++ b/maint/Makefile.am
@@ -1,82 +1,82 @@
#
-# Copyright 2019-2024 the Pacemaker project contributors
+# Copyright 2019-2025 the Pacemaker project contributors
#
# The version control history for this file may have further details.
#
# This source code is licensed under the GNU General Public License version 2
# or later (GPLv2+) WITHOUT ANY WARRANTY.
#
# Define release-related variables
include $(top_srcdir)/mk/release.mk
include $(top_srcdir)/mk/common.mk
noinst_SCRIPTS = bumplibs
EXTRA_DIST = README
#
# Change log generation
#
# Count changes in these directories
CHANGELOG_DIRS = ../include \
../lib \
../daemons \
../tools \
../xml
.PHONY: require_last_release
require_last_release:
@if [ -z "$(CHECKOUT)" ]; then \
echo "This target must be run from a git checkout"; \
exit 1; \
elif ! "$(GIT)" rev-parse $(LAST_RELEASE) >/dev/null 2>&1; then \
echo "LAST_RELEASE must be set to a valid git tag"; \
exit 1; \
fi
.PHONY: summary
summary: require_last_release
@printf "# %s (%s)\n* %d commits with%s\n" \
"$(NEXT_RELEASE)" "$$(date +'%d %b %Y')" \
"$$("$(GIT)" log --pretty=oneline --no-merges \
$(LAST_RELEASE)..HEAD | wc -l)" \
"$$("$(GIT)" diff $(LAST_RELEASE)..HEAD --shortstat \
$(CHANGELOG_DIRS))"
.PHONY: changes
changes: summary
@printf "\n## Features added since $(LAST_RELEASE)\n\n"
@"$(GIT)" log --pretty=format:'%s' --no-merges \
--abbrev-commit $(LAST_RELEASE)..HEAD \
| sed -n -e 's/^ *Feature: */* /p' | sort -uf \
| sed -e 's/^\( *[-+*] \)\([^:]*:\)\(.*\)$$/\1**\2**\3/' \
-e 's/\([ (]\)\([A-Za-z0-9]*_[^ ,]*\)/\1`\2`/g'
@printf "\n## Fixes since $(LAST_RELEASE)\n\n"
@"$(GIT)" log --pretty=format:'%s' --no-merges \
--abbrev-commit $(LAST_RELEASE)..HEAD \
- | sed -n -e 's/^ *\(Fix\|High\|Bug\): */* /p' \
+ | sed -n -e 's/^ *\(Fix\|High\|Med\|Low\|Bug\): */* /p' \
| sed -e 's/\(\(pacemaker-\)?based\):/CIB:/' \
-e 's/\(\(pacemaker-\)?execd\):/executor:/' \
-e 's/\(\(pacemaker-\)?controld\):/controller:/' \
-e 's/\(\(pacemaker-\)?fenced\):/fencing:/' \
| sort -uf \
| sed -e 's/^\( *[-+*] \)\([^:]*:\)\(.*\)$$/\1**\2**\3/' \
-e 's/\([ (]\)\([A-Za-z0-9]*_[^ ,]*\)/\1`\2`/g'
@printf "\n## Public API changes since $(LAST_RELEASE)\n\n"
@"$(GIT)" log --pretty=format:'%s' --no-merges \
--abbrev-commit $(LAST_RELEASE)..HEAD \
| sed -n -e 's/^ *API: */* /p' | sort -uf \
| sed -e 's/^\( *[-+*] \)\([^:]*:\)\(.*\)$$/\1**\2**\3/' \
-e 's/\([ (]\)\([A-Za-z0-9]*_[^ ,]*\)/\1`\2`/g'
.PHONY: changelog
changelog: require_last_release
@printf "%s\n\n%s\n" \
"$$($(MAKE) $(AM_MAKEFLAGS) changes \
| grep -v 'make\(\[[0-9]*\]\)\?:')" \
"$$(cat ../ChangeLog.md)" > ../ChangeLog.md
.PHONY: authors
authors: require_last_release
"$(GIT)" log $(LAST_RELEASE)..$(COMMIT) --format='%an' | sort -u
diff --git a/python/pacemaker/_cts/cmcorosync.py b/python/pacemaker/_cts/cmcorosync.py
index d0ab08d70c..b753c36e9a 100644
--- a/python/pacemaker/_cts/cmcorosync.py
+++ b/python/pacemaker/_cts/cmcorosync.py
@@ -1,75 +1,75 @@
"""Corosync-specific class for Pacemaker's Cluster Test Suite (CTS)."""
__all__ = ["Corosync2"]
__copyright__ = "Copyright 2007-2025 the Pacemaker project contributors"
__license__ = "GNU General Public License version 2 or later (GPLv2+) WITHOUT ANY WARRANTY"
from pacemaker._cts.CTS import Process
from pacemaker._cts.clustermanager import ClusterManager
from pacemaker._cts.patterns import PatternSelector
# Throughout this file, pylint has trouble understanding that EnvFactory
# is a singleton instance that can be treated as a subscriptable object.
# Various warnings are disabled because of this. See also a comment about
# self._rsh in environment.py.
# pylint: disable=unsubscriptable-object
class Corosync2(ClusterManager):
"""A subclass of ClusterManager specialized to handle corosync2 and later based clusters."""
def __init__(self):
"""Create a new Corosync2 instance."""
ClusterManager.__init__(self)
self._fullcomplist = {}
self.templates = PatternSelector(self.name)
@property
def components(self):
"""Return a list of patterns that should be ignored for the cluster's components."""
complist = []
if not self._fullcomplist:
common_ignore = self.templates.get_component("common-ignore")
daemons = [
"pacemaker-based",
"pacemaker-controld",
"pacemaker-attrd",
"pacemaker-execd",
"pacemaker-fenced"
]
for c in daemons:
badnews = self.templates.get_component(f"{c}-ignore") + common_ignore
proc = Process(self, c, pats=self.templates.get_component(c),
badnews_ignore=badnews)
self._fullcomplist[c] = proc
# the scheduler uses dc_pats instead of pats
badnews = self.templates.get_component("pacemaker-schedulerd-ignore") + common_ignore
proc = Process(self, "pacemaker-schedulerd",
dc_pats=self.templates.get_component("pacemaker-schedulerd"),
badnews_ignore=badnews)
self._fullcomplist["pacemaker-schedulerd"] = proc
# add (or replace) extra components
badnews = self.templates.get_component("corosync-ignore") + common_ignore
proc = Process(self, "corosync", pats=self.templates.get_component("corosync"),
badnews_ignore=badnews)
self._fullcomplist["corosync"] = proc
# Processes running under valgrind can't be shot with "killall -9 processname",
# so don't include them in the returned list
vgrind = self.env["valgrind-procs"].split()
for (key, val) in self._fullcomplist.items():
- if self.env["valgrind-tests"] and key in vgrind:
+ if key in vgrind:
self.log(f"Filtering {key} from the component list as it is being profiled by valgrind")
continue
if key == "pacemaker-fenced" and not self.env["DoFencing"]:
continue
complist.append(val)
return complist
diff --git a/python/pacemaker/_cts/environment.py b/python/pacemaker/_cts/environment.py
index b7987e45e5..8689463db8 100644
--- a/python/pacemaker/_cts/environment.py
+++ b/python/pacemaker/_cts/environment.py
@@ -1,642 +1,638 @@
"""Test environment classes for Pacemaker's Cluster Test Suite (CTS)."""
__all__ = ["EnvFactory"]
__copyright__ = "Copyright 2014-2025 the Pacemaker project contributors"
__license__ = "GNU General Public License version 2 or later (GPLv2+) WITHOUT ANY WARRANTY"
import argparse
from contextlib import suppress
import os
import random
import socket
import sys
import time
from pacemaker.buildoptions import BuildOptions
from pacemaker._cts.logging import LogFactory
from pacemaker._cts.remote import RemoteFactory
from pacemaker._cts.watcher import LogKind
class Environment:
"""
A class for managing the CTS environment.
This consists largely of processing and storing command line parameters.
"""
# pylint doesn't understand that self._rsh is callable (it stores the
# singleton instance of RemoteExec, as returned by the getInstance method
# of RemoteFactory).
# @TODO See if type annotations fix this.
# I think we could also fix this by getting rid of the getInstance methods,
# but that's a project for another day. For now, just disable the warning.
# pylint: disable=not-callable
def __init__(self, args):
"""
Create a new Environment instance.
This class can be treated kind of like a dictionary due to the presence
of typical dict functions like __contains__, __getitem__, and __setitem__.
However, it is not a dictionary so do not rely on standard dictionary
behavior.
Arguments:
args -- A list of command line parameters, minus the program name.
If None, sys.argv will be used.
"""
self.data = {}
self._nodes = []
# Set some defaults before processing command line arguments. These are
# either not set by any command line parameter, or they need a default
# that can't be set in add_argument.
self["DeadTime"] = 300
self["StartTime"] = 300
self["StableTime"] = 30
self["tests"] = []
self["IPagent"] = "IPaddr2"
self["DoFencing"] = True
self["ClobberCIB"] = False
self["CIBfilename"] = None
self["CIBResource"] = False
self["log_kind"] = None
self["node-limit"] = 0
self["scenario"] = "random"
self.random_gen = random.Random()
self._logger = LogFactory()
self._rsh = RemoteFactory().getInstance()
self._target = "localhost"
self._seed_random()
self._parse_args(args)
if not self["ListTests"]:
self._validate()
self._discover()
def _seed_random(self, seed=None):
"""
Initialize the random number generator.
Arguments:
seed -- Use this to see the random number generator, or use the
current time if None.
"""
if not seed:
seed = int(time.time())
self["RandSeed"] = seed
self.random_gen.seed(str(seed))
def dump(self):
"""Print the current environment."""
keys = []
for key in list(self.data.keys()):
keys.append(key)
keys.sort()
for key in keys:
self._logger.debug(f"{f'Environment[{key}]':35}: {str(self[key])}")
def keys(self):
"""Return a list of all environment keys stored in this instance."""
return list(self.data.keys())
def __contains__(self, key):
"""Return True if the given key exists in the environment."""
if key == "nodes":
return True
return key in self.data
def __getitem__(self, key):
"""Return the given environment key, or None if it does not exist."""
if str(key) == "0":
raise ValueError("Bad call to 'foo in X', should reference 'foo in X.keys()' instead")
if key == "nodes":
return self._nodes
if key == "Name":
return self._get_stack_short()
return self.data.get(key)
def __setitem__(self, key, value):
"""Set the given environment key to the given value, overriding any previous value."""
if key == "Stack":
self._set_stack(value)
elif key == "node-limit":
self.data[key] = value
self._filter_nodes()
elif key == "nodes":
self._nodes = []
for node in value:
# I don't think I need the IP address, etc. but this validates
# the node name against /etc/hosts and/or DNS, so it's a
# GoodThing(tm).
try:
n = node.strip()
# @TODO This only handles IPv4, use getaddrinfo() instead
# (here and in _discover())
socket.gethostbyname_ex(n)
self._nodes.append(n)
except socket.herror:
self._logger.log(f"{node} not found in DNS... aborting")
raise
self._filter_nodes()
else:
self.data[key] = value
def random_node(self):
"""Choose a random node from the cluster."""
return self.random_gen.choice(self["nodes"])
def get(self, key, default=None):
"""Return the value for key if key is in the environment, else default."""
if key == "nodes":
return self._nodes
return self.data.get(key, default)
def _set_stack(self, name):
"""Normalize the given cluster stack name."""
if name in ["corosync", "cs", "mcp"]:
self.data["Stack"] = "corosync 2+"
else:
raise ValueError(f"Unknown stack: {name}")
def _get_stack_short(self):
"""Return the short name for the currently set cluster stack."""
if "Stack" not in self.data:
return "unknown"
if self.data["Stack"] == "corosync 2+":
return "crm-corosync"
LogFactory().log(f"Unknown stack: {self['stack']}")
raise ValueError(f"Unknown stack: {self['stack']}")
def _detect_systemd(self):
"""Detect whether systemd is in use on the target node."""
if "have_systemd" not in self.data:
(rc, _) = self._rsh(self._target, "systemctl list-units", verbose=0)
self["have_systemd"] = rc == 0
def _detect_syslog(self):
"""Detect the syslog variant in use on the target node (if any)."""
if "syslogd" in self.data:
return
if self["have_systemd"]:
# Systemd
(_, lines) = self._rsh(self._target, r"systemctl list-units | grep syslog.*\.service.*active.*running | sed 's:.service.*::'", verbose=1)
else:
# SYS-V
(_, lines) = self._rsh(self._target, "chkconfig --list | grep syslog.*on | awk '{print $1}' | head -n 1", verbose=1)
with suppress(IndexError):
self["syslogd"] = lines[0].strip()
def disable_service(self, node, service):
"""Disable the given service on the given node."""
if self["have_systemd"]:
# Systemd
(rc, _) = self._rsh(node, f"systemctl disable {service}")
return rc
# SYS-V
(rc, _) = self._rsh(node, f"chkconfig {service} off")
return rc
def enable_service(self, node, service):
"""Enable the given service on the given node."""
if self["have_systemd"]:
# Systemd
(rc, _) = self._rsh(node, f"systemctl enable {service}")
return rc
# SYS-V
(rc, _) = self._rsh(node, f"chkconfig {service} on")
return rc
def service_is_enabled(self, node, service):
"""Return True if the given service is enabled on the given node."""
if self["have_systemd"]:
# Systemd
# With "systemctl is-enabled", we should check if the service is
# explicitly "enabled" instead of the return code. For example it returns
# 0 if the service is "static" or "indirect", but they don't really count
# as "enabled".
(rc, _) = self._rsh(node, f"systemctl is-enabled {service} | grep enabled")
return rc == 0
# SYS-V
(rc, _) = self._rsh(node, f"chkconfig --list | grep -e {service}.*on")
return rc == 0
def _detect_at_boot(self):
"""Detect if the cluster starts at boot."""
if "at-boot" not in self.data:
self["at-boot"] = self.service_is_enabled(self._target, "corosync") \
or self.service_is_enabled(self._target, "pacemaker")
def _detect_ip_offset(self):
"""Detect the offset for IPaddr resources."""
if self["CIBResource"] and "IPBase" not in self.data:
(_, lines) = self._rsh(self._target, "ip addr | grep inet | grep -v -e link -e inet6 -e '/32' -e ' lo' | awk '{print $2}'", verbose=0)
network = lines[0].strip()
(_, lines) = self._rsh(self._target, "nmap -sn -n %s | grep 'scan report' | awk '{print $NF}' | sed 's:(::' | sed 's:)::' | sort -V | tail -n 1" % network, verbose=0)
try:
self["IPBase"] = lines[0].strip()
except (IndexError, TypeError):
self["IPBase"] = None
if not self["IPBase"]:
self["IPBase"] = " fe80::1234:56:7890:1000"
self._logger.log("Could not determine an offset for IPaddr resources. Perhaps nmap is not installed on the nodes.")
self._logger.log(f"""Defaulting to '{self["IPBase"]}', use --test-ip-base to override""")
return
# pylint thinks self["IPBase"] is a list, not a string, which causes it
# to error out because a list doesn't have split().
# pylint: disable=no-member
last_part = self["IPBase"].split('.')[3]
if int(last_part) >= 240:
self._logger.log(f"Could not determine an offset for IPaddr resources. Upper bound is too high: {self['IPBase']} {last_part}")
self["IPBase"] = " fe80::1234:56:7890:1000"
self._logger.log(f"""Defaulting to '{self["IPBase"]}', use --test-ip-base to override""")
def _filter_nodes(self):
"""
Filter the list of cluster nodes.
If --limit-nodes is given, keep that many nodes from the front of the
list of cluster nodes and drop the rest.
"""
if self["node-limit"] > 0:
if len(self["nodes"]) > self["node-limit"]:
self._logger.log(f"Limiting the number of nodes configured={len(self['nodes'])} "
f"(max={self['node-limit']})")
while len(self["nodes"]) > self["node-limit"]:
self["nodes"].pop(len(self["nodes"]) - 1)
def _validate(self):
"""Check that we were given all required command line parameters."""
if not self["nodes"]:
raise ValueError("No nodes specified!")
def _discover(self):
"""Probe cluster nodes to figure out how to log and manage services."""
self._target = random.Random().choice(self["nodes"])
exerciser = socket.gethostname()
# Use the IP where possible to avoid name lookup failures
for ip in socket.gethostbyname_ex(exerciser)[2]:
if ip != "127.0.0.1":
exerciser = ip
break
self["cts-exerciser"] = exerciser
self._detect_systemd()
self._detect_syslog()
self._detect_at_boot()
self._detect_ip_offset()
def _parse_args(self, argv):
"""
Parse and validate command line parameters.
Set the appropriate values in the environment dictionary. If argv is
None, use sys.argv instead.
"""
if not argv:
argv = sys.argv[1:]
parser = argparse.ArgumentParser(epilog=f"{sys.argv[0]} -g virt1 -r --stonith ssh --schema pacemaker-2.0 500")
grp1 = parser.add_argument_group("Common options")
grp1.add_argument("-g", "--dsh-group", "--group",
metavar="GROUP", dest="group",
help="Use the nodes listed in the named DSH group (~/.dsh/groups/$name)")
grp1.add_argument("-l", "--limit-nodes",
type=int, default=0,
metavar="MAX",
help="Only use the first MAX cluster nodes supplied with --nodes")
grp1.add_argument("--benchmark",
action="store_true",
help="Add timing information")
grp1.add_argument("--list", "--list-tests",
action="store_true", dest="list_tests",
help="List the valid tests")
grp1.add_argument("--nodes",
metavar="NODES",
help="List of cluster nodes separated by whitespace")
grp1.add_argument("--stack",
default="corosync",
metavar="STACK",
help="Which cluster stack is installed")
grp2 = parser.add_argument_group("Options that CTS will usually auto-detect correctly")
grp2.add_argument("-L", "--logfile",
metavar="PATH",
help="Where to look for logs from cluster nodes (or 'journal' for systemd journal)")
grp2.add_argument("--at-boot", "--cluster-starts-at-boot",
choices=["1", "0", "yes", "no"],
help="Does the cluster software start at boot time?")
grp2.add_argument("--facility", "--syslog-facility",
default="daemon",
metavar="NAME",
help="Which syslog facility to log to")
grp2.add_argument("--ip", "--test-ip-base",
metavar="IP",
help="Offset for generated IP address resources")
grp3 = parser.add_argument_group("Options for release testing")
grp3.add_argument("-r", "--populate-resources",
action="store_true",
help="Generate a sample configuration")
grp3.add_argument("--choose",
metavar="NAME",
help="Run only the named tests, separated by whitespace")
grp3.add_argument("--fencing", "--stonith",
choices=["1", "0", "yes", "no", "lha", "openstack", "rhcs", "rhevm", "scsi", "ssh", "virt", "xvm"],
default="1",
help="What fencing agent to use")
grp3.add_argument("--once",
action="store_true",
help="Run all valid tests once")
grp4 = parser.add_argument_group("Additional (less common) options")
grp4.add_argument("-c", "--clobber-cib",
action="store_true",
help="Erase any existing configuration")
grp4.add_argument("-y", "--yes",
action="store_true", dest="always_continue",
help="Continue to run whenever prompted")
grp4.add_argument("--boot",
action="store_true",
help="")
grp4.add_argument("--cib-filename",
metavar="PATH",
help="Install the given CIB file to the cluster")
grp4.add_argument("--experimental-tests",
action="store_true",
help="Include experimental tests")
grp4.add_argument("--loop-minutes",
type=int, default=60,
help="")
grp4.add_argument("--no-loop-tests",
action="store_true",
help="Don't run looping/time-based tests")
grp4.add_argument("--no-unsafe-tests",
action="store_true",
help="Don't run tests that are unsafe for use with ocfs2/drbd")
grp4.add_argument("--notification-agent",
metavar="PATH",
default="/var/lib/pacemaker/notify.sh",
help="Script to configure for Pacemaker alerts")
grp4.add_argument("--notification-recipient",
metavar="R",
default="/var/lib/pacemaker/notify.log",
help="Recipient to pass to alert script")
grp4.add_argument("--oprofile",
metavar="NODES",
help="List of cluster nodes to run oprofile on")
grp4.add_argument("--outputfile",
metavar="PATH",
help="Location to write logs to")
grp4.add_argument("--qarsh",
action="store_true",
help="Use QARSH to access nodes instead of SSH")
grp4.add_argument("--schema",
metavar="SCHEMA",
default=f"pacemaker-{BuildOptions.CIB_SCHEMA_VERSION}",
help="Create a CIB conforming to the given schema")
grp4.add_argument("--seed",
metavar="SEED",
help="Use the given string as the random number seed")
grp4.add_argument("--set",
action="append",
metavar="ARG",
default=[],
help="Set key=value pairs (can be specified multiple times)")
grp4.add_argument("--stonith-args",
metavar="ARGS",
default="hostlist=all,livedangerously=yes",
help="")
grp4.add_argument("--stonith-type",
metavar="TYPE",
default="external/ssh",
help="")
grp4.add_argument("--trunc",
action="store_true", dest="truncate",
help="Truncate log file before starting")
grp4.add_argument("--valgrind-procs",
metavar="PROCS",
default="pacemaker-attrd pacemaker-based pacemaker-controld pacemaker-execd pacemaker-fenced pacemaker-schedulerd",
help="Run valgrind against the given space-separated list of processes")
- grp4.add_argument("--valgrind-tests",
- action="store_true",
- help="Include tests using valgrind")
grp4.add_argument("--warn-inactive",
action="store_true",
help="Warn if a resource is assigned to an inactive node")
parser.add_argument("iterations",
nargs='?',
type=int, default=1,
help="Number of tests to run")
args = parser.parse_args(args=argv)
# Set values on this object based on what happened with command line
# processing. This has to be done in several blocks.
# These values can always be set. They get a default from the add_argument
# calls, only do one thing, and they do not have any side effects.
self["ClobberCIB"] = args.clobber_cib
self["ListTests"] = args.list_tests
self["Schema"] = args.schema
self["Stack"] = args.stack
self["SyslogFacility"] = args.facility
self["TruncateLog"] = args.truncate
self["at-boot"] = args.at_boot in ["1", "yes"]
self["benchmark"] = args.benchmark
self["continue"] = args.always_continue
self["experimental-tests"] = args.experimental_tests
self["iterations"] = args.iterations
self["loop-minutes"] = args.loop_minutes
self["loop-tests"] = not args.no_loop_tests
self["notification-agent"] = args.notification_agent
self["notification-recipient"] = args.notification_recipient
self["node-limit"] = args.limit_nodes
self["stonith-params"] = args.stonith_args
self["stonith-type"] = args.stonith_type
self["unsafe-tests"] = not args.no_unsafe_tests
self["valgrind-procs"] = args.valgrind_procs
- self["valgrind-tests"] = args.valgrind_tests
self["warn-inactive"] = args.warn_inactive
# Nodes and groups are mutually exclusive, so their defaults cannot be
# set in their add_argument calls. Additionally, groups does more than
# just set a value. Here, set nodes first and then if a group is
# specified, override the previous nodes value.
if args.nodes:
self["nodes"] = args.nodes.split(" ")
else:
self["nodes"] = []
if args.group:
self["OutputFile"] = f"{os.environ['HOME']}/cluster-{args.dsh_group}.log"
LogFactory().add_file(self["OutputFile"], "CTS")
dsh_file = f"{os.environ['HOME']}/.dsh/group/{args.dsh_group}"
if os.path.isfile(dsh_file):
self["nodes"] = []
with open(dsh_file, "r", encoding="utf-8") as f:
for line in f:
stripped = line.strip()
if not stripped.startswith('#'):
self["nodes"].append(stripped)
else:
print(f"Unknown DSH group: {args.dsh_group}")
# Everything else either can't have a default set in an add_argument
# call (likely because we don't want to always have a value set for it)
# or it does something fancier than just set a single value. However,
# order does not matter for these as long as the user doesn't provide
# conflicting arguments on the command line. So just do Everything
# alphabetically.
if args.boot:
self["scenario"] = "boot"
if args.cib_filename:
self["CIBfilename"] = args.cib_filename
else:
self["CIBfilename"] = None
if args.choose:
self["scenario"] = "sequence"
self["tests"].extend(args.choose.split())
self["iterations"] = len(self["tests"])
if args.fencing:
if args.fencing in ["0", "no"]:
self["DoFencing"] = False
else:
self["DoFencing"] = True
if args.fencing in ["rhcs", "virt", "xvm"]:
self["stonith-type"] = "fence_xvm"
elif args.fencing == "scsi":
self["stonith-type"] = "fence_scsi"
elif args.fencing in ["lha", "ssh"]:
self["stonith-params"] = "hostlist=all,livedangerously=yes"
self["stonith-type"] = "external/ssh"
elif args.fencing == "openstack":
self["stonith-type"] = "fence_openstack"
print("Obtaining OpenStack credentials from the current environment")
region = os.environ['OS_REGION_NAME']
tenant = os.environ['OS_TENANT_NAME']
auth = os.environ['OS_AUTH_URL']
user = os.environ['OS_USERNAME']
password = os.environ['OS_PASSWORD']
self["stonith-params"] = f"region={region},tenant={tenant},auth={auth},user={user},password={password}"
elif args.fencing == "rhevm":
self["stonith-type"] = "fence_rhevm"
print("Obtaining RHEV-M credentials from the current environment")
user = os.environ['RHEVM_USERNAME']
password = os.environ['RHEVM_PASSWORD']
server = os.environ['RHEVM_SERVER']
port = os.environ['RHEVM_PORT']
self["stonith-params"] = f"login={user},passwd={password},ipaddr={server},ipport={port},ssl=1,shell_timeout=10"
if args.ip:
self["CIBResource"] = True
self["ClobberCIB"] = True
self["IPBase"] = args.ip
if args.logfile == "journal":
self["LogAuditDisabled"] = True
self["log_kind"] = LogKind.JOURNAL
elif args.logfile:
self["LogAuditDisabled"] = True
self["LogFileName"] = args.logfile
self["log_kind"] = LogKind.REMOTE_FILE
else:
# We can't set this as the default on the parser.add_argument call
# for this option because then args.logfile will be set, which means
# the above branch will be taken and those other values will also be
# set.
self["LogFileName"] = "/var/log/messages"
if args.once:
self["scenario"] = "all-once"
if args.oprofile:
self["oprofile"] = args.oprofile.split(" ")
else:
self["oprofile"] = []
if args.outputfile:
self["OutputFile"] = args.outputfile
LogFactory().add_file(self["OutputFile"])
if args.populate_resources:
self["CIBResource"] = True
self["ClobberCIB"] = True
if args.qarsh:
self._rsh.enable_qarsh()
for kv in args.set:
(name, value) = kv.split("=")
self[name] = value
print(f"Setting {name} = {value}")
class EnvFactory:
"""A class for constructing a singleton instance of an Environment object."""
instance = None
# pylint: disable=invalid-name
def getInstance(self, args=None):
"""
Return the previously created instance of Environment.
If no instance exists, create a new instance and return that.
"""
if not EnvFactory.instance:
EnvFactory.instance = Environment(args)
return EnvFactory.instance
diff --git a/python/pacemaker/_cts/tests/ctstest.py b/python/pacemaker/_cts/tests/ctstest.py
index 976b34a477..16d0f23e7b 100644
--- a/python/pacemaker/_cts/tests/ctstest.py
+++ b/python/pacemaker/_cts/tests/ctstest.py
@@ -1,246 +1,242 @@
"""Base classes for CTS tests."""
__all__ = ["CTSTest"]
__copyright__ = "Copyright 2000-2025 the Pacemaker project contributors"
__license__ = "GNU General Public License version 2 or later (GPLv2+) WITHOUT ANY WARRANTY"
import re
from pacemaker._cts.environment import EnvFactory
from pacemaker._cts.logging import LogFactory
from pacemaker._cts.patterns import PatternSelector
from pacemaker._cts.remote import RemoteFactory
from pacemaker._cts.timer import Timer
from pacemaker._cts.watcher import LogWatcher
# Disable various pylint warnings that occur in so many places throughout this
# file it's easiest to just take care of them globally. This does introduce the
# possibility that we'll miss some other cause of the same warning, but we'll
# just have to be careful.
class CTSTest:
"""
The base class for all cluster tests.
This implements a basic set of properties and behaviors like setup, tear
down, time keeping, and statistics tracking. It is up to specific tests
to implement their own specialized behavior on top of this class.
"""
def __init__(self, cm):
"""
Create a new CTSTest instance.
Arguments:
cm -- A ClusterManager instance
"""
# pylint: disable=invalid-name
self.audits = []
self.name = None
self.templates = PatternSelector(cm["Name"])
self.stats = {
"auditfail": 0,
"calls": 0,
"failure": 0,
"skipped": 0,
"success": 0
}
self._cm = cm
self._env = EnvFactory().getInstance()
self._rsh = RemoteFactory().getInstance()
self._logger = LogFactory()
self._timers = {}
self.benchmark = True # which tests to benchmark
self.failed = False
self.is_experimental = False
self.is_loop = False
self.is_unsafe = False
- self.is_valgrind = False
self.passed = True
def log(self, args):
"""Log a message."""
self._logger.log(args)
def debug(self, args):
"""Log a debug message."""
self._logger.debug(args)
def get_timer(self, key="test"):
"""Get the start time of the given timer."""
try:
return self._timers[key].start_time
except KeyError:
return 0
def set_timer(self, key="test"):
"""Set the start time of the given timer to now, and return that time."""
if key not in self._timers:
self._timers[key] = Timer(self._logger, self.name, key)
self._timers[key].start()
return self._timers[key].start_time
def log_timer(self, key="test"):
"""Log the elapsed time of the given timer."""
if key not in self._timers:
return
elapsed = self._timers[key].elapsed
self.debug(f"{self.name}:{key} runtime: {elapsed:.2f}")
del self._timers[key]
def incr(self, name):
"""Increment the given stats key."""
if name not in self.stats:
self.stats[name] = 0
self.stats[name] += 1
# Reset the test passed boolean
if name == "calls":
self.passed = True
def failure(self, reason="none"):
"""Increment the failure count, with an optional failure reason."""
self.passed = False
self.incr("failure")
self._logger.log(f"{f'Test {self.name}':<35} FAILED: {reason}")
return False
def success(self):
"""Increment the success count."""
self.incr("success")
return True
def skipped(self):
"""Increment the skipped count."""
self.incr("skipped")
return True
def __call__(self, node):
"""Perform this test."""
raise NotImplementedError
def audit(self):
"""Perform all the relevant audits (see ClusterAudit), returning whether or not they all passed."""
passed = True
for audit in self.audits:
if not audit():
self._logger.log(f"Internal {self.name} Audit {audit.name} FAILED.")
self.incr("auditfail")
passed = False
return passed
def setup(self, node):
"""Set up this test."""
# node is used in subclasses
# pylint: disable=unused-argument
return self.success()
def teardown(self, node):
"""Tear down this test."""
# node is used in subclasses
# pylint: disable=unused-argument
return self.success()
def create_watch(self, patterns, timeout, name=None):
"""
Create a new LogWatcher object.
This object can be used to search log files for matching patterns
during this test's run.
Arguments:
patterns -- A list of regular expressions to match against the log
timeout -- Default number of seconds to watch a log file at a time;
this can be overridden by the timeout= parameter to
self.look on an as-needed basis
name -- A unique name to use when logging about this watch
"""
if not name:
name = self.name
return LogWatcher(self._env["LogFileName"], patterns,
self._env["nodes"], self._env["log_kind"], name,
timeout)
def local_badnews(self, prefix, watch, local_ignore=None):
"""
Search through log files for messages.
Arguments:
prefix -- The string to look for at the beginning of lines,
or "LocalBadNews:" if None.
watch -- The LogWatcher object to use for searching.
local_ignore -- A list of regexes that, if found in a line, will
cause that line to be ignored.
Return the number of matches found.
"""
errcount = 0
if not prefix:
prefix = "LocalBadNews:"
ignorelist = [" CTS: ", prefix]
if local_ignore:
ignorelist += local_ignore
while errcount < 100:
match = watch.look(0)
if match:
add_err = True
for ignore in ignorelist:
if add_err and re.search(ignore, match):
add_err = False
if add_err:
self._logger.log(f"{prefix} {match}")
errcount += 1
else:
break
else:
self._logger.log("Too many errors!")
watch.end()
return errcount
def is_applicable(self):
"""
Return True if this test is applicable in the current test configuration.
This method must be implemented by all subclasses.
"""
if self.is_loop and not self._env["loop-tests"]:
return False
if self.is_unsafe and not self._env["unsafe-tests"]:
return False
- if self.is_valgrind and not self._env["valgrind-tests"]:
- return False
-
if self.is_experimental and not self._env["experimental-tests"]:
return False
if self._env["benchmark"] and not self.benchmark:
return False
return True
@property
def errors_to_ignore(self):
"""Return a list of errors which should be ignored."""
return []
diff --git a/python/pylintrc b/python/pylintrc
index 81f63fea95..baea3f713a 100644
--- a/python/pylintrc
+++ b/python/pylintrc
@@ -1,558 +1,559 @@
# NOTE: Any line with CHANGED: describes something that we changed from the
# default pylintrc configuration.
[MAIN]
# Python code to execute, usually for sys.path manipulation such as
# pygtk.require().
#init-hook=
# Files or directories to be skipped. They should be base names, not
# paths.
ignore=CVS
# Add files or directories matching the regex patterns to the ignore-list. The
# regex matches against paths and can be in Posix or Windows format.
ignore-paths=
# Files or directories matching the regex patterns are skipped. The regex
# matches against base names, not paths.
ignore-patterns=^\.#
# Pickle collected data for later comparisons.
persistent=yes
# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
load-plugins=
# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the
# number of processors available to use.
jobs=1
# When enabled, pylint would attempt to guess common misconfiguration and emit
# user-friendly hints instead of false-positive error messages.
suggestion-mode=yes
# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
unsafe-load-any-extension=no
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code
extension-pkg-allow-list=
# Minimum supported python version
# CHANGED
py-version = 3.6
# Control the amount of potential inferred values when inferring a single
# object. This can help the performance when dealing with large functions or
# complex, nested conditions.
limit-inference-results=100
# Specify a score threshold under which the program will exit with error.
fail-under=10.0
# Return non-zero exit code if any of these messages/categories are detected,
# even if score is above --fail-under value. Syntax same as enable. Messages
# specified are enabled, while categories only check already-enabled messages.
fail-on=
# Clear in-memory caches upon conclusion of linting. Useful if running pylint in
# a server-like mode.
clear-cache-post-run=no
[MESSAGES CONTROL]
# Only show warnings with the listed confidence levels. Leave empty to show
# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED
# confidence=
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once). See also the "--disable" option for examples.
enable=
use-symbolic-message-instead,
useless-suppression,
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once).You can also use "--disable=all" to
# disable everything first and then re-enable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use"--disable=all --enable=classes
# --disable=W"
# CHANGED
disable=R0801,
line-too-long,
too-few-public-methods,
too-many-arguments,
too-many-branches,
too-many-instance-attributes,
+ too-many-positional-arguments,
too-many-statements,
unrecognized-option,
useless-option-value
[REPORTS]
# Set the output format. Available formats are text, parseable, colorized, msvs
# (visual studio) and html. You can also give a reporter class, eg
# mypackage.mymodule.MyReporterClass.
output-format=text
# Tells whether to display a full report or only the messages
reports=no
# Python expression which should return a note less than 10 (10 is the highest
# note). You have access to the variables 'fatal', 'error', 'warning', 'refactor', 'convention'
# and 'info', which contain the number of messages in each category, as
# well as 'statement', which is the total number of statements analyzed. This
# score is used by the global evaluation report (RP0004).
evaluation=max(0, 0 if fatal else 10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10))
# Template used to display messages. This is a python new-style format string
# used to format the message information. See doc for all details
#msg-template=
# Activate the evaluation score.
score=yes
[LOGGING]
# Logging modules to check that the string format arguments are in logging
# function parameter format
logging-modules=logging
# The type of string formatting that logging methods do. `old` means using %
# formatting, `new` is for `{}` formatting.
logging-format-style=old
[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
# CHANGED: Don't do anything about FIXME, XXX, or TODO
notes=
# Regular expression of note tags to take in consideration.
#notes-rgx=
[SIMILARITIES]
# Minimum lines number of a similarity.
min-similarity-lines=6
# Ignore comments when computing similarities.
ignore-comments=yes
# Ignore docstrings when computing similarities.
ignore-docstrings=yes
# Ignore imports when computing similarities.
ignore-imports=yes
# Signatures are removed from the similarity computation
ignore-signatures=yes
[VARIABLES]
# Tells whether we should check for unused import in __init__ files.
init-import=no
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins=
# List of strings which can identify a callback function by name. A callback
# name must start or end with one of those strings.
callbacks=cb_,_cb
# Tells whether unused global variables should be treated as a violation.
allow-global-unused-variables=yes
# List of names allowed to shadow builtins
allowed-redefined-builtins=
# List of qualified module names which can have objects that can redefine
# builtins.
redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io
[FORMAT]
# Maximum number of characters on a single line.
max-line-length=100
# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$
# Allow the body of an if to be on the same line as the test if there is no
# else.
single-line-if-stmt=no
# Allow the body of a class to be on the same line as the declaration if body
# contains single statement.
single-line-class-stmt=no
# Maximum number of lines in a module
max-module-lines=2000
# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '
# Number of spaces of indent required inside a hanging or continued line.
indent-after-paren=4
# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
expected-line-ending-format=
[BASIC]
# Good variable names which should always be accepted, separated by a comma
# CHANGED: Single variable names are handled by variable-rgx below, leaving
# _ here as the name for any variable that should be ignored.
good-names=_
# Good variable names regexes, separated by a comma. If names match any regex,
# they will always be accepted
good-names-rgxs=
# Bad variable names which should always be refused, separated by a comma
bad-names=foo,bar,baz,toto,tutu,tata
# Bad variable names regexes, separated by a comma. If names match any regex,
# they will always be refused
bad-names-rgxs=
# Colon-delimited sets of names that determine each other's naming style when
# the name regexes allow several styles.
name-group=
# Include a hint for the correct naming format with invalid-name
include-naming-hint=no
# Naming style matching correct function names.
function-naming-style=snake_case
# Regular expression matching correct function names
function-rgx=[a-z_][a-z0-9_]{2,30}$
# Naming style matching correct variable names.
variable-naming-style=snake_case
# Regular expression matching correct variable names
# CHANGED: One letter variables are fine
variable-rgx=[a-z_][a-z0-9_]{,30}$
# Naming style matching correct constant names.
const-naming-style=UPPER_CASE
# Regular expression matching correct constant names
const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$
# Naming style matching correct attribute names.
attr-naming-style=snake_case
# Regular expression matching correct attribute names
attr-rgx=[a-z_][a-z0-9_]{2,}$
# Naming style matching correct argument names.
argument-naming-style=snake_case
# Regular expression matching correct argument names
argument-rgx=[a-z_][a-z0-9_]{2,30}$
# Naming style matching correct class attribute names.
class-attribute-naming-style=any
# Regular expression matching correct class attribute names
class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$
# Naming style matching correct class constant names.
class-const-naming-style=UPPER_CASE
# Regular expression matching correct class constant names. Overrides class-
# const-naming-style.
#class-const-rgx=
# Naming style matching correct inline iteration names.
inlinevar-naming-style=any
# Regular expression matching correct inline iteration names
inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$
# Naming style matching correct class names.
class-naming-style=PascalCase
# Regular expression matching correct class names
class-rgx=[A-Z_][a-zA-Z0-9]+$
# Naming style matching correct module names.
module-naming-style=snake_case
# Regular expression matching correct module names
module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
# Naming style matching correct method names.
method-naming-style=snake_case
# Regular expression matching correct method names
method-rgx=[a-z_][a-z0-9_]{2,}$
# Regular expression matching correct type variable names
#typevar-rgx=
# Regular expression which should only match function or class names that do
# not require a docstring. Use ^(?!__init__$)_ to also check __init__.
no-docstring-rgx=__.*__
# Minimum line length for functions/classes that require docstrings, shorter
# ones are exempt.
docstring-min-length=-1
# List of decorators that define properties, such as abc.abstractproperty.
property-classes=abc.abstractproperty
[TYPECHECK]
# Regex pattern to define which classes are considered mixins if ignore-mixin-
# members is set to 'yes'
mixin-class-rgx=.*MixIn
# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis). It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=
# List of class names for which member attributes should not be checked (useful
# for classes with dynamically set attributes). This supports the use of
# qualified names.
ignored-classes=SQLObject, optparse.Values, thread._local, _thread._local
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=REQUEST,acl_users,aq_parent,argparse.Namespace
# List of decorators that create context managers from functions, such as
# contextlib.contextmanager.
contextmanager-decorators=contextlib.contextmanager
# Tells whether to warn about missing members when the owner of the attribute
# is inferred to be None.
ignore-none=yes
# This flag controls whether pylint should warn about no-member and similar
# checks whenever an opaque object is returned when inferring. The inference
# can return multiple potential results while evaluating a Python object, but
# some branches might not be evaluated, which results in partial inference. In
# that case, it might be useful to still emit no-member and other checks for
# the rest of the inferred objects.
ignore-on-opaque-inference=yes
# Show a hint with possible names when a member name was not found. The aspect
# of finding the hint is based on edit distance.
missing-member-hint=yes
# The minimum edit distance a name should have in order to be considered a
# similar match for a missing member name.
missing-member-hint-distance=1
# The total number of similar names that should be taken in consideration when
# showing a hint for a missing member.
missing-member-max-choices=1
[SPELLING]
# Spelling dictionary name. Available dictionaries: none. To make it working
# install python-enchant package.
spelling-dict=
# List of comma separated words that should not be checked.
spelling-ignore-words=
# List of comma separated words that should be considered directives if they
# appear and the beginning of a comment and should not be checked.
spelling-ignore-comment-directives=fmt: on,fmt: off,noqa:,noqa,nosec,isort:skip,mypy:,pragma:,# noinspection
# A path to a file that contains private dictionary; one word per line.
spelling-private-dict-file=.pyenchant_pylint_custom_dict.txt
# Tells whether to store unknown words to indicated private dictionary in
# --spelling-private-dict-file option instead of raising a message.
spelling-store-unknown-words=no
# Limits count of emitted suggestions for spelling mistakes.
max-spelling-suggestions=2
[DESIGN]
# Maximum number of arguments for function / method
max-args=10
# Maximum number of locals for function / method body
max-locals=25
# Maximum number of return / yield for function / method body
max-returns=11
# Maximum number of branch for function / method body
max-branches=27
# Maximum number of statements in function / method body
max-statements=100
# Maximum number of parents for a class (see R0901).
max-parents=7
# List of qualified class names to ignore when counting class parents (see R0901).
ignored-parents=
# Maximum number of attributes for a class (see R0902).
max-attributes=11
# Minimum number of public methods for a class (see R0903).
min-public-methods=2
# Maximum number of public methods for a class (see R0904).
max-public-methods=25
# Maximum number of boolean expressions in an if statement (see R0916).
max-bool-expr=5
# Maximum number of statements in a try-block
max-try-statements = 14
# List of regular expressions of class ancestor names to
# ignore when counting public methods (see R0903).
exclude-too-few-public-methods=
[CLASSES]
# List of method names used to declare (i.e. assign) instance attributes.
# CHANGED: Remove setUp and __post_init__, add reset
defining-attr-methods=__init__,__new__,reset
# List of valid names for the first argument in a class method.
valid-classmethod-first-arg=cls
# List of valid names for the first argument in a metaclass class method.
valid-metaclass-classmethod-first-arg=mcs
# List of member names, which should be excluded from the protected access
# warning.
exclude-protected=_asdict,_fields,_replace,_source,_make
# Warn about protected attribute access inside special methods
check-protected-access-in-special-methods=no
[IMPORTS]
# List of modules that can be imported at any level, not just the top level
# one.
allow-any-import-level=
# Allow wildcard imports from modules that define __all__.
allow-wildcard-with-all=no
# Analyse import fallback blocks. This can be used to support both Python 2 and
# 3 compatible code, which means that the block might have code that exists
# only in one or another interpreter, leading to false positives when analysed.
analyse-fallback-blocks=no
# Deprecated modules which should not be used, separated by a comma
deprecated-modules=regsub,TERMIOS,Bastion,rexec
# Create a graph of every (i.e. internal and external) dependencies in the
# given file (report RP0402 must not be disabled)
import-graph=
# Create a graph of external dependencies in the given file (report RP0402 must
# not be disabled)
ext-import-graph=
# Create a graph of internal dependencies in the given file (report RP0402 must
# not be disabled)
int-import-graph=
# Force import order to recognize a module as part of the standard
# compatibility libraries.
known-standard-library=
# Force import order to recognize a module as part of a third party library.
known-third-party=enchant
# Couples of modules and preferred modules, separated by a comma.
preferred-modules=
[EXCEPTIONS]
# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=builtins.Exception
[TYPING]
# Set to ``no`` if the app / library does **NOT** need to support runtime
# introspection of type annotations. If you use type annotations
# **exclusively** for type checking of an application, you're probably fine.
# For libraries, evaluate if some users what to access the type hints at
# runtime first, e.g., through ``typing.get_type_hints``. Applies to Python
# versions 3.7 - 3.9
runtime-typing = no
[DEPRECATED_BUILTINS]
# List of builtins function names that should not be used, separated by a comma
bad-functions=map,input
[REFACTORING]
# Maximum number of nested blocks for function / method body
max-nested-blocks=5
# Complete name of functions that never returns. When checking for
# inconsistent-return-statements if a never returning function is called then
# it will be considered as an explicit return statement and no message will be
# printed.
never-returning-functions=sys.exit,argparse.parse_error
[STRING]
# This flag controls whether inconsistent-quotes generates a warning when the
# character used as a quote delimiter is used inconsistently within a module.
check-quote-consistency=no
# This flag controls whether the implicit-str-concat should generate a warning
# on implicit string concatenation in sequences defined over several lines.
check-str-concat-over-line-jumps=no
[CODE_STYLE]
# Max line length for which to sill emit suggestions. Used to prevent optional
# suggestions which would get split by a code formatter (e.g., black). Will
# default to the setting for ``max-line-length``.
#max-line-length-suggestions=
File Metadata
Details
Attached
Mime Type
text/x-diff
Expires
Thu, Jul 10, 1:44 AM (1 d, 4 h)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
2009569
Default Alt Text
(190 KB)
Attached To
Mode
rP Pacemaker
Attached
Detach File
Event Timeline
Log In to Comment