Seems to be closed by b93f0245 and a1d94f7a, both from Aleksei Burlakov.

crm_resource already uses G_OPTION_REMAINING to collect the remaining arguments into an array. Then after parsing all options, it logs those remaining arguments if they're invalid. However, they're valid if we're running a subcommand that uses options.override_params.

This is the currently expected behavior before pacemaker-attrd has learned a node is remote. Once it does learn it, it will update the attribute cache entry and write it out.

Since this has turned out to be complicated, let's break it down into smaller pieces.

Originated here: https://github.com/ClusterLabs/pacemaker/commit/a27f099

In T817#12112, @nrwahl2 wrote:

From the GitHub discussion:

I believe it was originally for testing the code paths. Not needed anymore

Optionally, we could have a (possibly hidden) --async-call option for developer use, that starts a mainloop and runs a callback function that processes the reply.

From the GitHub discussion:

In T620#11858, @kgaillot wrote:

In T620#11565, @nrwahl2 wrote:

Meta-attribute inheritance is weird. I've just started looking at groups. Very much non-exhaustive:

• is-managed: Any false wins. If is-managed=false for the group and is-managed=true for a primitive, the primitive is unmanaged.
• maintenance: Any true wins. Similar to is-managed but reversed.

These are true in general for all resource parent-child relationships

I'm thinking the simplest approach might be to consider the "inheritable" meta-attributes as separate to each layer. For example there is a group resource-stickiness meta-attribute and a different primitive resource-stickiness meta-attribute. That way each could have its own description of how it behaves. Primitive resource-stickiness keeps the current description from Pacemaker Explained, group resource-stickiness gets something like "Default value to use for members' resource-stickiness if not explicitly set for the member. The group's own stickiness is not this value, but the sum of its members' stickiness." The primitive default could even be described as "Value of resource-stickiness in group meta-attributes if set, otherwise value of resource-stickiness in clone meta-attributes if set, otherwise value of resource-stickiness in bundle meta-attributes if set, otherwise 0, plus 1 for promoted clone instances of the primitive" though that sounds horrible.

In T620#11565, @nrwahl2 wrote:

Meta-attribute inheritance is weird. I've just started looking at groups. Very much non-exhaustive:

• is-managed: Any false wins. If is-managed=false for the group and is-managed=true for a primitive, the primitive is unmanaged.
• maintenance: Any true wins. Similar to is-managed but reversed.

Meta-attribute inheritance is weird. I've just started looking at groups. Very much non-exhaustive:

• is-managed: Any false wins. If is-managed=false for the group and is-managed=true for a primitive, the primitive is unmanaged.
• maintenance: Any true wins. Similar to is-managed but reversed.
• resource-stickiness: Group's stickiness is used as the default for a primitive's stickiness. Primitive's stickiness overrides (does not add to) group's stickiness.
• target-role: Same as resource-stickiness. The group's value is only a default. In particular, if the group has target-role=Stopped and the first member has target-role=Started, the first member is allowed to run.

In T620#11130, @nrwahl2 wrote:

In T620#11098, @kgaillot wrote:

I think it would make sense if we were designing from scratch, but crm_attribute actually is the intended place for this. It has always been used to manage cluster options as well as node attributes.

Okay, that makes sense for cluster options. I'm still not sure it makes sense to put local options there, or to put meta-attributes for alerts, resources, and ops there.

I'm not at that point yet, still finishing up cluster options, but the rest should move faster with the infrastructure in place and the approach settled on.

In T620#11098, @kgaillot wrote:

I think it would make sense if we were designing from scratch, but crm_attribute actually is the intended place for this. It has always been used to manage cluster options as well as node attributes.

In T620#11067, @nrwahl2 wrote:

In T620#10980, @kgaillot wrote:

If it doesn't require anything outside libcrmcommon, I would put the bulk of it there. When we get to the UI (command-line options), the highest-level equivalents of that should be in libpacemaker.
...
The libcrmcommon functions would do all the processing, and the libpacemaker functions would focus on output

That's basically the conundrum. Output is the only thing we're dealing with (there's no meaningful processing otherwise), but the output functions need access to an array that lives in libcrmcommon. There are many ways to approach this that would work. The question is which one's the cleanest and most in line with our existing code.

How do you feel about a new lightweight CLI tool called something like pcmk_option? I think I can get this into crm_attribute relatively easily. It's just that none of these are attributes.

In T620#10980, @kgaillot wrote:

If it doesn't require anything outside libcrmcommon, I would put the bulk of it there. When we get to the UI (command-line options), the highest-level equivalents of that should be in libpacemaker.
...
The libcrmcommon functions would do all the processing, and the libpacemaker functions would focus on output

In T620#10979, @nrwahl2 wrote:

Do we want the "list cluster options" command to go in libpacemaker, or libcrmcommon?

Do we want the "list cluster options" command to go in libpacemaker, or libcrmcommon?

gao-yan posted the following in https://github.com/ClusterLabs/pacemaker/pull/2902#discussion_r1457468975, in response to this task. I'm inclined to agree with their comments.

Fix: https://github.com/ClusterLabs/pacemaker/pull/3331

On Fedora 38/39, the issue starts with commit bc91cc5 (Pacemaker 2.1.0). Previously, mon_cib_connection_destroy() removed all stonith client notifications and disconnected the client, but it didn't delete the client object. As of that commit, it also deletes the client object.

Not reproducible on RHEL 9.3 with the latest packages. Also not reproducible on Fedora 37.

On second thought, although the ref count might be the easiest way to fix this, it would be totally reasonable to say "no function that's registered as a stonith notify callback should ever free the stonith_t object."

I need to verify this, but here's what I think is happening:

1. Mainloop destroys stonith API connection, calling stonith_connection_destroy() on the stonith client.
2. The stonith client sends out T_STONITH_NOTIFY_DISCONNECT notifications via foreach_notify_entry() -> stonith_send_notification().
3. One notification triggers crm_mon.c:mon_st_callback_display() (or event()), which calls mon_cib_connection_destroy().
4. mon_cib_connection_destroy() calls stonith_api_delete(), which calls stonith_api_free() and destroys the entire client object.
5. Control returns to stonith_send_notification() and foreach_notify_entry().
6. One of the further attempts to access members of the stonith client object causes a segfault, since the object and all the members that it owns have been freed.

I've reproduced this on 2.1.0, 2.1.2, and 2.1.7, so it doesn't appear to be a regression. Additionally, it only happens at shutdown, so there's no tangible impact except that it won't reconnect when the cluster starts back... which is not great.

In T620#10902, @nrwahl2 wrote:

However, I'm not sure whether you wanted to include meta-attributes that can be set on a collective for the sole purpose of being inherited by a primitive (sort of the opposite scenario compared to the one above).

• On the one hand: it's cleaner and clearer not to include them, and to include only those meta-attributes that have a direct effect on the collective. We can simply ensure it's documented (maybe it already is) that primitives inherit all of their parents' meta-attributes. That implies that any primitive meta-attribute can be set on a collective for the purpose of inheritance.
• On the other hand: excluding them offloads more work to external tools. For example, if pcs is validating meta-attributes for a bundle, then it needs to check the list of bundle meta-attributes, the list of primitive meta-attributes, and maybe the list of clone meta-attributes. That's not a deal-breaker, but it requires that extra knowledge of Pacemaker behavior be built into external tools.

For the purposes of showing lists, I would think we want to show the meta-attributes that can apply directly to the thing being requested (not just inherited).