diff --git a/README-testing b/README-testing index 3e44627..d647d17 100644 --- a/README-testing +++ b/README-testing @@ -1,194 +1,204 @@ There's a booth-test RPM available that contains two types of tests. It installs the necessary files into `/usr/share/booth/tests`. === Live tests (booth operation) BEWARE: Run this with _test_ clusters only! The live testing utility tests booth operation using the given `booth.conf`: $ /usr/share/booth/tests/test/live_test.sh booth.conf It is possible to run only specific tests. Run the script without arguments to see usage and the list of tests and netem network emulation functions. Example booth.conf: ------------ transport="UDP" port="9929" arbitrator="10.2.12.53" arbitrator="10.2.13.82" site="10.2.12.101" site="10.2.13.101" site="10.121.187.99" ticket="ticket-A" expire = 30 timeout = 3 retries = 3 before-acquire-handler = /usr/share/booth/service-runnable d-src1 ------------ A split brain condition is also tested. For that to work, all sites need `iptables` installed. The supplied script `booth_path` is used to manipulate iptables rules. +==== Pacemaker configuration + +This is a sample pacemaker configuration for a single-node +cluster: + +primitive booth ocf:pacemaker:booth-site +primitive d-src1 ocf:heartbeat:Dummy + +Please adjust to your environment. + ==== Network environment emulation To introduce packet loss or network delays, set the NETEM_ENV environment variable. There are currently three netem network emulation settings supported: - loss: all servers emulate packet loss (30% by default) - single_loss: the first site in the configuration emulates packet loss (30% by default) - net_delay: all servers emulate packet delay (100ms by default with random variation of 10%) The settings can be supplied by adding ':' to the emulator name. For instance: # NETEM_ENV=loss:50 /usr/share/booth/tests/test/live_test.sh booth.conf It is not necessary to run the test script on one of the sites. Just copy the script and make the test `booth.conf` available locally: $ scp testsite:/usr/share/booth/tests/test/live_test.sh . $ scp testsite:/etc/booth/booth.conf . $ sh live_test.sh booth.conf You need at least two sites and one arbitrator. The configuration can contain just one ticket. It is not necessary to configure the `before-acquire-handler`. Notes: - (BEWARE!) the supplied configuration files is copied to /etc/booth/booth.conf to all sites/arbitrators thus overwriting any existing configuration - the utility uses ssh to manage booth at all sites/arbitrators and logs in as user `root` - it is required that ssh public authentication works without providing the passphrase (otherwise it is impractical) - the log file is ./test_booth.log (it is actually a shell trace, with timestamps if you're running bash) - in case one of the tests fail, hb_report is created If you want to open a bug report, please attach all hb_reports and `test_booth.log`. === Simple tests (commandline, config file) Run (as non-root) # python test/runtests.py to run the tests written in python. === Unit tests These use gdb and pexpect to set boothd state to some configured value, injecting some input and looking at the output. # python script/unit-test.py src/boothd unit-tests/ Or, if using the 'booth-test' RPM, # python unit-test.py src/boothd unit-tests/ This must (currently?) be run as a non-root user; another optional argument is the test to start from, eg. '003'. Basically, boothd is started with the config file `unit-tests/booth.conf`, and gdb gets attached to it. Then, some ticket state is set, incoming messages are delivered, and outgoing messages and the state is compared to expected values. `unit-tests/_defaults.txt` has default values for the initial state and message data. Each test file consists of headers and key/value pairs: -------------------- ticket: state ST_STABLE message0: # optional comment for the log file header.cmd OP_ACCEPTING ticket.id "asdga" outgoing0: header.cmd OP_PREPARING last_ack_ballot 42 finally: new_ballot 1234 -------------------- A few details to the the above example: * Ticket states in RAM (`ticket`, `finally`) are written in host-endianness. * Message data (`messageN`, `outgoingN`) are automatically converted via `htonl` resp. `ntohl`. They are delivered/checked in the order defined by the integer `N` component. * Strings are done via `strcpy()` * `ticket` and `messageN` are assignment chunks * `finally` and `outgoingN` are compare chunks * In `outgoingN` you can check _both_ message data (keys with a `.` in them) and ticket state * Symbolic names are useable, GDB translates them for us * The test scripts in `unit-tests/` need to be named with 3 digits, an underscore, some text, and `.txt` * The "fake" `crm_ticket` script gets the current test via `UNIT_TEST`; test scripts can pass additional information via `UNIT_TEST_AUX`. ==== Tips and Hints There's another special header: `gdb__N__`. These lines are sent to GDB after injecting a message, but before waiting for an outgoing line. Values that contain `§` are sent as multiple lines to GDB. This means that a stanza like -------------------- gdb0: watch booth_conf->ticket[0].owner § commands § bt § c § end -------------------- will cause a watchpoint to be set, and when it is triggered a backtrace (`bt`) is written to the log file. This makes it easy to ask for additional data or check for a call-chain when hitting bugs that can be reproduced via such a unit-test. # vim: set ft=asciidoc :