Change Details

In `services__execute_systemd()`, we create a timer for the action timeout plus 5 seconds (for setup steps) before invoking the systemd unit. We also create runtime systemd drop-in override files (`systemd_create_override()`) during the start action and remove them (`systemd_remove_override()`) during the stop action, running daemon-reload after both steps to cause the change to take effect. For start and stop actions, it would be nice to do away with the Pacemaker timers and use systemd timers instead. (We currently implement monitor actions by polling, which we may replace with some sort of D-Bus signal monitoring.) The 5 seconds mentioned above is arbitrary. It may not be enough for the setup steps, depending on system load. Or it may be too much and give the resource extra time. What the user really cares about (and is trying to configure via action timeouts) is how much time the resource has to finish its job after all the systemd "bookkeeping" is done. Here is the main obstacle: we support other unit types besides service. Specifically, we support service, mount, path, socket, and timer. * Service units support `TimeoutStartSec=` and `TimeoutStopSec=` options in the `[Service]` section of their unit configuration files. These give us exactly what we want. * Socket units provide only `TimeoutSec=`: `Configures the time to wait for the commands specified in ExecStartPre=, ExecStartPost=, ExecStopPre= and ExecStopPost= to finish.` Notably: * This single timeout applies to both start and stop. This means we would need to reset it to the resource action's stop timeout and perform another daemon-reload before we initiate the stop job. * It doesn't seem to apply to whatever systemd does to actually start or stop the socket. Rather, it applies only to the pre and post scripts. I suspect that systemd doesn't do much of anything except start/stop the service that corresponds to the socket, using whatever timeouts are configured for that service unit; and whatever internal bookkeeping is required as preparation for that. The internal bookkeeping might take an arbitrarily long time under high load. * Mount units provide only `TimeoutSec=`: `Configures the time to wait for the mount command to finish.` This gives us what we want for start actions, but it's unclear whether this same timeout applies to the `umount` command during stop actions. * If it does apply to unmounts/stops, then we'd need to reset it before stops as described above for socket units. * If it does not apply to unmounts/stops, then we have a problem. An unmount can take a long time for various reasons. We need to be able to configure a stop timeout. If systemd's default is infinite, we can hang forever; if its default is finite, then it may not be long enough. * Path and timer units don't provide any kind of timeout option, although their start and stop operations are probably trivial and complete immediately unless the system is under extremely high load. There is a parameter called JobRunningTimeoutSec= that goes in the [Unit] section and can apply to all unit types. It was introduced around systemd v201 if I recall correctly, so we'd have to check on the OS releases that we support and what systemd versions they run by default. This would also mean extra daemon-reloads. As long as this option is in place, it will apply to every job affecting the unit. So we'd want to remove it as soon as the start action is complete and trigger another daemon-reload. We'd also have to set the option again (to the stop action timeout) before a stop job and trigger another daemon-reload. (We already remove the override and trigger a daemon-reload //after// the stop action completes.) Note that a daemon-reload checks ALL configuration files and can take an arbitrary amount of time (though it completes very quickly in the common case), and we have to do it multiple times for each systemd resource. There seems to be no way to avoid daemon-reloads. As far as I can tell, there is no way to override a single option for a single unit (even for a single job for that unit) without reloading the configurations for all units. -- A related issue that may be important... From the `systemd-system.conf(5)` man page: ``` DefaultTimeoutStartSec= and DefaultTimeoutStopSec= default to 45 s in the system manager and 45 s in the user manager. ``` It sounds very plausible that even if we set a resource action timeout of, say, 120 seconds in Pacemaker, systemd may kill the job as "timed out" after 45 seconds. A user could work around this by configuring their timeout manually in the unit file, of course. But I think a reasonable expectation is that the timeouts configured within Pacemaker resources take precedence. Alternatively, we could document that Pacemaker's resource action timeouts are ignored by systemd resources, and that the timeouts must be configured within systemd. Pacemaker's resource action timeouts would then just need to be higher than the systemd timeout, so the executor (or whatever) doesn't declare an action as failed prematurely. Unless the user wants it to do that. -- See also https://github.com/ClusterLabs/pacemaker/pull/3818#issuecomment-2670433567 and the rest of PR #3818 for related work on using D-Bus with systemd start/stop actions.

In `services__execute_systemd()`, we create a timer for the action timeout plus 5 seconds (for setup steps) before invoking the systemd unit. We also create runtime systemd drop-in override files (`systemd_create_override()`) during the start action and remove them (`systemd_remove_override()`) during the stop action, running daemon-reload after both steps to cause the change to take effect. For start and stop actions, it would be nice to do away with the Pacemaker timers and use systemd timers instead. (We currently implement monitor actions by polling, which we may replace with some sort of D-Bus signal monitoring.) The 5 seconds mentioned above is arbitrary. It may not be enough for the setup steps, depending on system load. Or it may be too much and give the resource extra time. What the user really cares about (and is trying to configure via action timeouts) is how much time the resource has to finish its job after all the systemd "bookkeeping" is done. Here is the main obstacle: we support other unit types besides service. Specifically, we support service, mount, path, socket, and timer. * Service units support `TimeoutStartSec=` and `TimeoutStopSec=` options in the `[Service]` section of their unit configuration files. These give us exactly what we want. * Socket units provide only `TimeoutSec=`: `Configures the time to wait for the commands specified in ExecStartPre=, ExecStartPost=, ExecStopPre= and ExecStopPost= to finish.` Notably: * This single timeout applies to both start and stop. This means we would need to reset it to the resource action's stop timeout and perform another daemon-reload before we initiate the stop job. * It doesn't seem to apply to whatever systemd does to actually start or stop the socket. Rather, it applies only to the pre and post scripts. I suspect that systemd doesn't do much of anything except start/stop the service that corresponds to the socket, using whatever timeouts are configured for that service unit; and whatever internal bookkeeping is required as preparation for that. The internal bookkeeping might take an arbitrarily long time under high load. * Mount units provide only `TimeoutSec=`: `Configures the time to wait for the mount command to finish.` This gives us what we want for start actions, but it's unclear whether this same timeout applies to the `umount` command during stop actions. * If it does apply to unmounts/stops, then we'd need to reset it before stops as described above for socket units. * If it does not apply to unmounts/stops, then we have a problem. An unmount can take a long time for various reasons. We need to be able to configure a stop timeout. If systemd's default is infinite, we can hang forever; if its default is finite, then it may not be long enough. * Path and timer units don't provide any kind of timeout option, although their start and stop operations are probably trivial and complete immediately unless the system is under extremely high load. There is a parameter called JobRunningTimeoutSec= that goes in the [Unit] section and can apply to all unit types. It was introduced around systemd v201 if I recall correctly, so we'd have to check on the OS releases that we support and what systemd versions they run by default. This would also mean extra daemon-reloads. As long as this option is in place, it will apply to every job affecting the unit. So we'd want to remove it as soon as the start action is complete and trigger another daemon-reload. We'd also have to set the option again (to the stop action timeout) before a stop job and trigger another daemon-reload. (We already remove the override and trigger a daemon-reload //after// the stop action completes.) Note that a daemon-reload checks ALL configuration files and can take an arbitrary amount of time (though it completes very quickly in the common case), and we have to do it multiple times for each systemd resource. There seems to be no way to avoid daemon-reloads. As far as I can tell, there is no way to override a single option for a single unit (even for a single job for that unit) without reloading the configurations for all units. --- A related issue that may be important... From the `systemd-system.conf(5)` man page: ``` DefaultTimeoutStartSec= and DefaultTimeoutStopSec= default to 45 s in the system manager and 45 s in the user manager. ``` It sounds very plausible that even if we set a resource action timeout of, say, 120 seconds in Pacemaker, systemd may kill the job as "timed out" after 45 seconds. A user could work around this by configuring their timeout manually in the unit file, of course. But I think a reasonable expectation is that the timeouts configured within Pacemaker resources take precedence. Alternatively, we could document that Pacemaker's resource action timeouts are ignored by systemd resources, and that the timeouts must be configured within systemd. Pacemaker's resource action timeouts would then just need to be higher than the systemd timeout, so the executor (or whatever) doesn't declare an action as failed prematurely. Unless the user wants it to do that. --- See also https://github.com/ClusterLabs/pacemaker/pull/3818#issuecomment-2670433567 and the rest of PR #3818 for related work on using D-Bus with systemd start/stop actions.

In `services__execute_systemd()`, we create a timer for the action timeout plus 5 seconds (for setup steps) before invoking the systemd unit. We also create runtime systemd drop-in override files (`systemd_create_override()`) during the start action and remove them (`systemd_remove_override()`) during the stop action, running daemon-reload after both steps to cause the change to take effect. For start and stop actions, it would be nice to do away with the Pacemaker timers and use systemd timers instead. (We currently implement monitor actions by polling, which we may replace with some sort of D-Bus signal monitoring.) The 5 seconds mentioned above is arbitrary. It may not be enough for the setup steps, depending on system load. Or it may be too much and give the resource extra time. What the user really cares about (and is trying to configure via action timeouts) is how much time the resource has to finish its job after all the systemd "bookkeeping" is done. Here is the main obstacle: we support other unit types besides service. Specifically, we support service, mount, path, socket, and timer. * Service units support `TimeoutStartSec=` and `TimeoutStopSec=` options in the `[Service]` section of their unit configuration files. These give us exactly what we want. * Socket units provide only `TimeoutSec=`: `Configures the time to wait for the commands specified in ExecStartPre=, ExecStartPost=, ExecStopPre= and ExecStopPost= to finish.` Notably: * This single timeout applies to both start and stop. This means we would need to reset it to the resource action's stop timeout and perform another daemon-reload before we initiate the stop job. * It doesn't seem to apply to whatever systemd does to actually start or stop the socket. Rather, it applies only to the pre and post scripts. I suspect that systemd doesn't do much of anything except start/stop the service that corresponds to the socket, using whatever timeouts are configured for that service unit; and whatever internal bookkeeping is required as preparation for that. The internal bookkeeping might take an arbitrarily long time under high load. * Mount units provide only `TimeoutSec=`: `Configures the time to wait for the mount command to finish.` This gives us what we want for start actions, but it's unclear whether this same timeout applies to the `umount` command during stop actions. * If it does apply to unmounts/stops, then we'd need to reset it before stops as described above for socket units. * If it does not apply to unmounts/stops, then we have a problem. An unmount can take a long time for various reasons. We need to be able to configure a stop timeout. If systemd's default is infinite, we can hang forever; if its default is finite, then it may not be long enough. * Path and timer units don't provide any kind of timeout option, although their start and stop operations are probably trivial and complete immediately unless the system is under extremely high load. There is a parameter called JobRunningTimeoutSec= that goes in the [Unit] section and can apply to all unit types. It was introduced around systemd v201 if I recall correctly, so we'd have to check on the OS releases that we support and what systemd versions they run by default. This would also mean extra daemon-reloads. As long as this option is in place, it will apply to every job affecting the unit. So we'd want to remove it as soon as the start action is complete and trigger another daemon-reload. We'd also have to set the option again (to the stop action timeout) before a stop job and trigger another daemon-reload. (We already remove the override and trigger a daemon-reload //after// the stop action completes.) Note that a daemon-reload checks ALL configuration files and can take an arbitrary amount of time (though it completes very quickly in the common case), and we have to do it multiple times for each systemd resource. There seems to be no way to avoid daemon-reloads. As far as I can tell, there is no way to override a single option for a single unit (even for a single job for that unit) without reloading the configurations for all units. --- A related issue that may be important... From the `systemd-system.conf(5)` man page: ``` DefaultTimeoutStartSec= and DefaultTimeoutStopSec= default to 45 s in the system manager and 45 s in the user manager. ``` It sounds very plausible that even if we set a resource action timeout of, say, 120 seconds in Pacemaker, systemd may kill the job as "timed out" after 45 seconds. A user could work around this by configuring their timeout manually in the unit file, of course. But I think a reasonable expectation is that the timeouts configured within Pacemaker resources take precedence. Alternatively, we could document that Pacemaker's resource action timeouts are ignored by systemd resources, and that the timeouts must be configured within systemd. Pacemaker's resource action timeouts would then just need to be higher than the systemd timeout, so the executor (or whatever) doesn't declare an action as failed prematurely. Unless the user wants it to do that. --- See also https://github.com/ClusterLabs/pacemaker/pull/3818#issuecomment-2670433567 and the rest of PR #3818 for related work on using D-Bus with systemd start/stop actions.