Page MenuHomeClusterLabs Projects
Files F3153120

README
No OneTemporary
Actions

README
View Options

I. Fence_xvm - virtual machine fencing agent
Fence_xvm is an agent which establishes a communications link between
a cluster of virtual machines (VC) and a cluster of domain0/physical
nodes which are hosting the virtual cluster. Its operations are
fairly simple.
(a) Start a listener service.
(b) Send a multicast packet requesting that a VM be fenced.
(c) Authenticate client.
(e) Read response.
(f) Exit with success/failure, depending on the response received.
If any of the above steps fail, the fencing agent exits with a failure
code and fencing is retried by the virtual cluster at a later time.
Because of the simplicty of fence_xvm, it is not necessary that
fence_xvm be run from within a virtualized guest - all it needs is
libnspr and libnss and a shared private key (for authentication; we
would hate to receive a false positive response from a node not in the
cluster!).
II. Fence_xvmd - The virtual machine fencing host
Fence_xvmd is a daemon which runs on physical hosts (e.g. in domain0)
of the cluster hosting the virtual cluster. It listens on a port
for multicast traffic from virtual cluster(s), and takes actions.
Multiple disjoint virtual clusters can coexist on a single physical
host cluster, but this requires multiple instances of fence_xvmd.
NOTE: fence_xvmd *MUST* be run on ALL nodes in a given cluster which
will be hosting virtual machines if fence_xvm is to be used for
fencing!
There are a couple of ways the multicast packet is handled,
depending on the state of the host OS. It might be hosting the VM,
or it might not. Furthermore, the VM might "reside" on a host which
has failed.
In order to be able to guarantee safe fencing of a VM even if the
last- known host is down, we must store the last-known locations of
each virtual machine in some sort of cluster-wide way. For this, we
use the AIS Checkpointing API, which is provided by OpenAIS. Every
few seconds, fence_xvmd queries the hypervisor via libvirt and
stores any local VM states in a checkpoint. In the event of a
physical node failure (which consequently causes the failure of one
or more guests), we can then read the checkpoint section corresponding
to the guest we need to fence to find out the previous
owner. With that information, we can then check with CMAN to see if
the last-known host node has been fenced. If so, then the VM is
clean as well. The physical cluster must, therefore, have fencing
in order for fence_xvmd to work.
Operation of a node hosting a VM which needs to be fenced:
(a) Receive multicast packet
(b) Authenticate multicast packet
(c) Open connection to host contained within multicast
packet.
(d) Authenticate server.
(e) Carry out fencing operation (e.g. call libvirt to destroy or
reboot the VM; there is no "on" method at this point).
(f) If operation succeeds, send success response.
Operation of high-node-ID:
(a) Receive multicast packet
(b) Authenticate multicast packet
(c) Read VM state from checkpoint
(d) Check liveliness of nodeID hosting VM (if alive, do nothing)
(e) Open connection to host contained within multicast
packet.
(f) Check with CMAN to see if last-known host has been fenced.
(If it has not; do nothing -- this is why the physical
cluster also needs fencing!)
(g) Authenticate server & send response.
(h) If last-known host has been fenced, send success response.
NOTE: There is always a possibility that a VM is started again
before the fencing operation and checkpoint update for that VM
occurs. If the VM has booted and rejoined the cluster, fencing will
not be necessary. If it is in the process of booting, but has not
yet joined the cluster, fencing will also not be necessary - because
it will not be using cluster resources yet.
III. Security considerations
While fencing is generally expected to run on a more or less trusted
network, there are cases where it may not be.
* The multicast packet is subject to replay attacks, but because no
fencing action is taken based solely on the information contained
within the packet, this should not allow an attacker to maliciously
fence a VM from outside the cluster, though it may be possible to
cause a DoS of fence_xvmd if enough multicast packets are sent.
* The only currently supported authentication mechanisms are simple
challenge-response based on a shared private key and pseudorandom
number generation.
* An attacker with access to the shared key(s) can easily fence any
known VM, even if they are not on a cluster node.
* Different shared keys should be used for different virtual
clusters on the same subnet (whether in the same physical cluster
or not). Additionally, multiple fence_xvmd instances must be run
(each listening on a different multicast IP + port combination).
IV. Configuration
Generate a random key file. An example of how to generate it is:
dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4096 count=1
Distribute the generated key file to all virtual machines in a
cluster as well as all physical host nodes which will be hosting
that particular cluster of guests. More simply, everything involved
with hosting the virtual cluster as well as the virtual cluster
itself must have the same key file; it acts as a password.
The key should not be placed on shared file systems (because shared
file systems require the cluster, which requires fencing...).
Furthermore, it is considered 'unsupported' to join a host cluster
and a guest cluster in one management domain.
A. Configuring the host (physical) cluster
On the host cluster, you need to add the following tag as a
child of the <cluster> tag in /etc/cluster/cluster.conf:
<fence_xvmd/>
(Do not forget to increment the configuration version number and
run 'ccs_tool update /etc/cluster/cluster.conf' !).
Start fence_xvmd on all host nodes if it isn't already running.
Just run 'fence_xvmd'. The next time the cluster is restarted,
fence_xvmd will start automatically; it is started by the cman
script if you have the above tag in cluster.conf.
B. Configuring the guest (virtual) cluster
On the guest cluster, you need to set up per-node fencing. This
is a fairly simple task as well. First, you need to add a fence
device for 'xvm'. Simply add the following to the <fencedevices/>
tag in the guest cluster's cluster.conf:
<fencedevice name="xvm" agent="fence_xvm"/>
After doing this, each node also needs individual fencing set up.
For each <clusternode/> tag, you will need to add something like
the following:
<fence>
<method name="1">
<device name="xvm" domain="doman-name"/>
</method>
</fence>
For example, if you have a virtual host named 'vm1.test.com' with a
corresponding virtual domain name of 'domU-vm1' in the dom0 cluster,
and a node ID of 1, the <clusternode> tag for that virtual machine
would look like so:
<clusternode name="vm1.test.com" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="xvm" domain="domU-vm1"/>
</method>
</fence>
</clusternode>
C. Advanced configuration
Any advanced configuration parameters (e.g. changing authentication,
hashing, key file, etc.) should be included in the <fence_xvmd/> tag
in the host cluster and the <fencedevice .../> tag in the guest
cluster. For a complete list of advanced parameters, see:
fence_xvmd -h
fence_xvm -h

File Metadata

Mime Type
text/plain
Expires
Tue, Feb 25, 10:35 AM (1 d, 15 h)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
1427480
Default Alt Text
README (7 KB)

Event Timeline