Page MenuHomeClusterLabs Projects
Files F5197326

README-testing
No OneTemporary
Actions

README-testing
View Options

There's a booth-test RPM available that contains two types of tests.
It installs the necessary files into `/usr/share/booth/tests`.
=== Live tests (booth operation)
BEWARE: Run this with _test_ clusters only!
The live testing utility tests booth operation using the given
`booth.conf`:
$ /usr/share/booth/tests/test/live_test.sh booth.conf
It is possible to run only specific tests. Run the script without
arguments to see usage and the list of tests and netem network
emulation functions.
Example booth.conf:
------------
transport="UDP"
port="9929"
arbitrator="10.2.12.53"
arbitrator="10.2.13.82"
site="10.2.12.101"
site="10.2.13.101"
site="10.121.187.99"
ticket="ticket-A"
expire = 30
timeout = 3
retries = 3
before-acquire-handler = /usr/share/booth/service-runnable d-src1
------------
A split brain condition is also tested. For that to work, all
sites need `iptables` installed. The supplied script `booth_path`
is used to manipulate iptables rules.
==== Network environment emulation
To introduce packet loss or network delays, set the NETEM_ENV
environment variable. There are currently three netem network
emulation settings supported:
- loss: all servers emulate packet loss (30% by default)
- single_loss: the first site in the configuration emulates
packet loss (30% by default)
- net_delay: all servers emulate packet delay (100ms by default
with random variation of 10%)
The settings can be supplied by adding ':' to the emulator name.
For instance:
# NETEM_ENV=loss:50 /usr/share/booth/tests/test/live_test.sh booth.conf
It is not necessary to run the test script on one of the sites.
Just copy the script and make the test `booth.conf` available
locally:
$ scp testsite:/usr/share/booth/tests/test/live_test.sh .
$ scp testsite:/etc/booth/booth.conf .
$ sh live_test.sh booth.conf
You need at least two sites and one arbitrator.
The configuration can contain just one ticket.
It is not necessary to configure the `before-acquire-handler`.
Notes:
- (BEWARE!) the supplied configuration files is copied to
/etc/booth/booth.conf to all sites/arbitrators thus overwriting
any existing configuration
- the utility uses ssh to manage booth at all sites/arbitrators
and logs in as user `root`
- it is required that ssh public authentication works without
providing the passphrase (otherwise it is impractical)
- the log file is ./test_booth.log (it is actually a shell trace,
with timestamps if you're running bash)
- in case one of the tests fail, hb_report is created
If you want to open a bug report, please attach all hb_reports
and `test_booth.log`.
=== Simple tests (commandline, config file)
Run (as non-root)
# python test/runtests.py
to run the tests written in python.
=== Unit tests
These use gdb and pexpect to set boothd state to some configured value,
injecting some input and looking at the output.
# python script/unit-test.py src/boothd unit-tests/
Or, if using the 'booth-test' RPM,
# python unit-test.py src/boothd unit-tests/
This must (currently?) be run as a non-root user; another optional argument is
the test to start from, eg. '003'.
Basically, boothd is started with the config file `unit-tests/booth.conf`, and
gdb gets attached to it.
Then, some ticket state is set, incoming messages are delivered, and outgoing
messages and the state is compared to expected values.
`unit-tests/_defaults.txt` has default values for the initial state and
message data.
Each test file consists of headers and key/value pairs:
--------------------
ticket:
state ST_STABLE
message0: # optional comment for the log file
header.cmd OP_ACCEPTING
ticket.id "asdga"
outgoing0:
header.cmd OP_PREPARING
last_ack_ballot 42
finally:
new_ballot 1234
--------------------
A few details to the the above example:
* Ticket states in RAM (`ticket`, `finally`) are written in host-endianness.
* Message data (`messageN`, `outgoingN`) are automatically converted via `htonl` resp. `ntohl`. They are delivered/checked in the order defined by the integer `N` component.
* Strings are done via `strcpy()`
* `ticket` and `messageN` are assignment chunks
* `finally` and `outgoingN` are compare chunks
* In `outgoingN` you can check _both_ message data (keys with a `.` in them) and ticket state
* Symbolic names are useable, GDB translates them for us
* The test scripts in `unit-tests/` need to be named with 3 digits, an underscore, some text, and `.txt`
* The "fake" `crm_ticket` script gets the current test via `UNIT_TEST`; test scripts can pass additional information via `UNIT_TEST_AUX`.
==== Tips and Hints
There's another special header: `gdb__N__`. These lines are sent to GDB after
injecting a message, but before waiting for an outgoing line. Values that
contain `§` are sent as multiple lines to GDB.
This means that a stanza like
--------------------
gdb0:
watch booth_conf->ticket[0].owner § commands § bt § c § end
--------------------
will cause a watchpoint to be set, and when it is triggered a backtrace (`bt`)
is written to the log file.
This makes it easy to ask for additional data or check for a call-chain when
hitting bugs that can be reproduced via such a unit-test.
# vim: set ft=asciidoc :

File Metadata

Mime Type
text/plain
Expires
Thu, Aug 14, 12:37 PM (1 d, 2 h)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
2177434
Default Alt Text
README-testing (5 KB)

Event Timeline