Page MenuHomeClusterLabs Projects
Files F2822360

README-testing
No OneTemporary
Actions

README-testing
View Options

There's a booth-test package which contains two types of tests.
It installs the necessary files into `/usr/share/booth/tests`.
=== Live tests (booth operation)
BEWARE: Run this with _test_ clusters only!
The live testing utility tests booth operation using the given
`booth.conf`:
$ /usr/share/booth/tests/test/live_test.sh booth.conf
It is possible to run only specific tests. Run the script without
arguments to see usage and the list of tests and netem network
emulation functions.
There are some restrictions on how booth.conf is formatted.
There could be several tickets defined, but only the first ticket
is used for testing. This ticket must have expire and timeout
parameters configured.
Example booth.conf:
------------
transport="UDP"
port="9929"
arbitrator="10.2.12.53"
arbitrator="10.2.13.82"
site="10.2.12.101"
site="10.2.13.101"
site="10.121.187.99"
ticket="ticket-A"
expire = 30
timeout = 3
retries = 3
before-acquire-handler = /usr/share/booth/service-runnable d-src1
------------
A split brain condition is also tested. For that to work, all
sites need `iptables` installed. The supplied script `booth_path`
is used to manipulate iptables rules.
==== Pacemaker configuration
This is a sample pacemaker configuration for a single-node
cluster:
primitive booth ocf:pacemaker:booth-site
primitive d-src1 ocf:heartbeat:Dummy
rsc_ticket global-d-src1 ticket-A: d-src1
Additionally, you may also add an ocf:booth:sharedrsc resource to
also check that the ticket is granted always to only one site:
primitive shared ocf:booth:sharedrsc \
params dir="10.2.13.82:/var/tmp/boothtestdir"
rsc_ticket global-shared ticket-A: shared
Please adjust to your environment.
==== Network environment emulation
To introduce packet loss or network delays, set the NETEM_ENV
environment variable. There are currently three netem network
emulation settings supported:
- loss: all servers emulate packet loss (30% by default)
- single_loss: the first site in the configuration emulates
packet loss (30% by default)
- net_delay: all servers emulate packet delay (100ms by default
with random variation of 10%)
The settings can be supplied by adding ':' to the emulator name.
For instance:
# NETEM_ENV=loss:50 /usr/share/booth/tests/test/live_test.sh booth.conf
It is not necessary to run the test script on one of the sites.
Just copy the script and make the test `booth.conf` available
locally:
$ scp testsite:/usr/share/booth/tests/test/live_test.sh .
$ scp testsite:/etc/booth/booth.conf .
$ sh live_test.sh booth.conf
You need at least two sites and one arbitrator.
The configuration can contain just one ticket.
It is not necessary to configure the `before-acquire-handler`.
Notes:
- (BEWARE!) the supplied configuration files is copied to
/etc/booth/booth.conf to all sites/arbitrators thus overwriting
any existing configuration
- the utility uses ssh to manage booth at all sites/arbitrators
and logs in as user `root`
- it is required that ssh public authentication works without
providing the passphrase (otherwise it is impractical)
- the log file is ./test_booth.log (it is actually a shell trace,
with timestamps if you're running bash)
- in case one of the tests fail, hb_report is created
If you want to open a bug report, please attach all hb_reports
and `test_booth.log`.
=== Simple tests (commandline, config file)
Run (as non-root)
# python test/runtests.py
to run the tests written in python.
=== Unit tests
These use gdb and pexpect to set boothd state to some configured value,
injecting some input and looking at the output.
# python script/unit-test.py src/boothd unit-tests/
Or, if using the 'booth-test' RPM,
# python unit-test.py src/boothd unit-tests/
This must (currently?) be run as a non-root user; another optional argument is
the test to start from, eg. '003'.
Basically, boothd is started with the config file `unit-tests/booth.conf`, and
gdb gets attached to it.
Then, some ticket state is set, incoming messages are delivered, and outgoing
messages and the state is compared to expected values.
`unit-tests/_defaults.txt` has default values for the initial state and
message data.
Each test file consists of headers and key/value pairs:
--------------------
ticket:
state ST_STABLE
message0: # optional comment for the log file
header.cmd OP_ACCEPTING
ticket.id "asdga"
outgoing0:
header.cmd OP_PREPARING
last_ack_ballot 42
finally:
new_ballot 1234
--------------------
A few details to the the above example:
* Ticket states in RAM (`ticket`, `finally`) are written in host-endianness.
* Message data (`messageN`, `outgoingN`) are automatically converted via `htonl` resp. `ntohl`. They are delivered/checked in the order defined by the integer `N` component.
* Strings are done via `strcpy()`
* `ticket` and `messageN` are assignment chunks
* `finally` and `outgoingN` are compare chunks
* In `outgoingN` you can check _both_ message data (keys with a `.` in them) and ticket state
* Symbolic names are useable, GDB translates them for us
* The test scripts in `unit-tests/` need to be named with 3 digits, an underscore, some text, and `.txt`
* The "fake" `crm_ticket` script gets the current test via `UNIT_TEST`; test scripts can pass additional information via `UNIT_TEST_AUX`.
==== Tips and Hints
There's another special header: `gdb__N__`. These lines are sent to GDB after
injecting a message, but before waiting for an outgoing line. Values that
contain `§` are sent as multiple lines to GDB.
This means that a stanza like
--------------------
gdb0:
watch booth_conf->ticket[0].owner § commands § bt § c § end
--------------------
will cause a watchpoint to be set, and when it is triggered a backtrace (`bt`)
is written to the log file.
This makes it easy to ask for additional data or check for a call-chain when
hitting bugs that can be reproduced via such a unit-test.
# vim: set ft=asciidoc :

File Metadata

Mime Type
text/plain
Expires
Sat, Jan 25, 5:35 AM (13 h, 57 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
1203461
Default Alt Text
README-testing (5 KB)

Event Timeline