Currently, we use SSH (via the rsh method abstraction) to run commands from the cts-lab exerciser on the lab cluster nodes. It would be cleaner to run Python code remotely, if possible. This would allow using things like psutil instead of pkill, etc. (See closed T545 -- even though the valgrind problem is basically solved now, it could be cleaner.)

Unfortunately, this will have to wait until we bump dependency versions. Currently we only require Python 3.6 or above.

Some possibilities:

• execnet
  • Seems the simplest -- it needs to be installed on the exerciser but not on the lab cluster nodes.
  • Requires Python >= 3.8, although this requirement probably applies only to the exerciser.
  • Is in maintenance-only mode but may be maintained indefinitely since pytest-xdist depends on it.
  • Docs: https://execnet.readthedocs.io/en/latest/index.html
• Pyro (Python remote objects)
  • Requires creating a Pyro server instance on each lab cluster node and using the exerciser as a client. This would likely mean needing to install pacemaker-cts on the lab nodes, which I don't believe we've required in the past.
  • Requires Python >= 3.7.
  • Actively maintained
  • NOT PACKAGED IN RHEL
  • Docs: https://pyro5.readthedocs.io/en/latest/
• RPyC (Remote Python Calls)
  • I haven't looked into this one much yet. I stumbled across it just before adding this bullet point.
  • Python version requirements keep changing: the latest upstream version supports only Python 3.10+.
  • However, RPyC supported Python 3.6+ until version 5.2.1. So this might be usable without Pacemaker dependency bumps.
    • https://github.com/tomerfiliba-org/rpyc/commit/da62dd7
  • A StackOverflow thread linked below claims that the documentation is not great.
  • Appears to be actively maintained
  • NOT PACKAGED IN RHEL
  • Docs: https://rpyc.readthedocs.io/en/latest/

StackOverflow: What are the pros and cons of PyRo and RPyC python libs?

If this turns out not to be feasible, or if we want to improve on our current system until we can bump dependency versions, then Fabric (Python 3.4+) might be a good option:

• https://www.fabfile.org/index.html

It seems to do basically what we're doing ourselves with RemoteFactory, etc.