diff --git a/doc/crm_cli.txt b/doc/crm_cli.txt
index a8a0374625..b60dd09c95 100644
--- a/doc/crm_cli.txt
+++ b/doc/crm_cli.txt
@@ -1,2162 +1,2166 @@
 CRM CLI (command line interface) tool
 ======================================
 Dejan_Muhamedagic,_Yan_Gao dejan@suse.de,ygao@novell.com
 v0.94, Februar 18, 2010:
 
 The CRM (a.k.a Pacemaker) is a Cluster Resource Manager which
 implements the cluster configuration provided by the user in CIB
 (Cluster Information Base). The CIB is a set of instructions
 coded in XML. Editing the CIB is a challenge, not only due to its
 complexity and a wide variety of options, but also because XML is
 more computer than user friendly.
 
 .Note
 **************************
 I do understand that there are people capable of
 dealing with XML without an intermediary.
 **************************
 
 There are currently three options to manage the CIB, listed here
 in a decreasing order of user-friendliness:
 
 - the GUI (`hb_gui`)
 - a set of command line tools
 - `cibadmin(8)`
 
 The GUI is very popular and it has recently seen a lot of good
 development. For some it is going to be (or remain) the first
 choice in cluster management.
 
 The command line tools, lead by `crm_resource(8)`, are capable of
 performing almost any kind of CIB transformation. The usage is,
 however, plagued by the notorious weakness common to all UNIX
 tools: a multitude of options, necessary for operation and yet
 very hard to remember. Usage is also inconsistent at times.
 
 The `cibadmin` is the ultimate CIB management tool: it applies
 chunks of XML written by the user or generated by another tool to
 the CIB. Very difficult to use without extensive training. Or
 should I say drill. May be unnerving as well, in particular due
 to sometimes cryptic error messages.
 
 == Design goals
 
 The CLI provides a consistent and unified interface to
 CIB/cluster management. It uses the command line tools where
 possible and may resort to XML and `cibadmin` when there is no
 other option. That is the easiest way to ensure compatibility
 between different management tools.
 
 It may be used either as an interactive shell or for single
 commands directly on the shell's command line. It is also
 possible to feed it a set of commands from standard input or a
 file, thus turning it into a scripting tool. Templates with ready
 made configurations may help people to learn about the cluster
 configuration or facilitate testing procedures.
 
 The CLI may also be used for the CIB description and generation.
 A file containing a set of CLI instructions may be applied to the
 CLI tool to generate a complete CIB.
 
 The new shadow CIB feature may also be put to use. The user may
 work on one of the shadow CIBs and then apply (or commit) it in a
 single step to the cluster.
  
 It should also allow deployment of raw XML which may come either
 from files or network.
 
 Several modes of operation are available to restrict the set of
 features depending on the user's proficiency.
 
 The CLI is line oriented: every command must start and finish
 on the same line. It is possible to use a continuation character
 (`\`) to write one command in two or more lines.
 
 The CLI has to be run on one of the cluster nodes.
 
 .Note
 **************************
 Even though all sensible configurations (and most of those that
 are not) are going to be supported by the CLI, I suspect that it
 may still happen that certain XML constructs may confuse the
 tool. When that happens, please file a bug report.
 
 The CLI will not try to update the objects it does not
 understand. Of course, it is always possible to edit such objects
 in the XML format.
 **************************
 
 == Introduction to the user interface
 
 Arguably the most important aspect of such a program is the user
 interface. We begin with an informal introduction so that the
 reader may get acquainted with it and get a general feeling of
 the tool. It is probably best just to give some examples:
 
 1. Command line (one-shot) use:
 
         # crm resource stop www_app
 
 2. Interactive use:
 
         # crm
         crm(live)# resource
         crm(live) resource# unmanage tetris_1
         crm(live) resource# up
         crm(live)# node standby node4
 
 3. Cluster configuration:
 
     # crm<<EOF
     configure
       erase
       #
       # resources
       #
       primitive disk0 iscsi \
         params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0
       primitive fs0 Filesystem \
         params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3
       primitive internal_ip IPaddr params ip=192.168.1.101
       primitive apache apache \
         params configfile=/disk0/etc/apache2/site0.conf
       primitive apcfence stonith:apcsmart \
         params ttydev=/dev/ttyS0 hostlist="node1 node2" \
         op start timeout=60s
       primitive pingd pingd \
         params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
       #
       # monitor apache and the UPS
       #
       monitor apache 60s:30s
       monitor apcfence 120m:60s
       #
       # cluster layout
       #
       group internal_www \
         disk0 fs0 internal_ip apache
       clone fence apcfence \
         meta globally-unique=false clone-max=2 clone-node-max=1
       clone conn pingd \
         meta globally-unique=false clone-max=2 clone-node-max=1
       location node_pref internal_www \
         rule 50: #uname eq node1 \
         rule pingd: defined pingd
       #
       # cluster properties
       #
       property stonith-enabled=true
       commit
     EOF
 
 If you've ever done a CRM style configuration, you should be able
 to understand the above examples without much difficulties.  The
 CLI should provide a means to manage the cluster efficiently or
 put together a configuration in a concise manner.
 
 The `(live)` string in the prompt signifies that the current CIB
 in use is the cluster live configuration. It is also possible to
 work with the so-called shadow CIBs, i.e. configurations which
 are stored in files and aren't active, but may be applied at any
 time to the cluster.
 
 Since the CIB is hierarchical such is the interface too. There
 are several levels and entering each of them enables the user to
 use a certain set of commands.
 
 === Shadow CIB usage
 
 Shadow CIB is a new feature. Shadow CIBs may be manipulated in
 the same way like the _live_ CIB, but these changes have no
 effect on the cluster resources. The administrator may choose to
 apply any of them to the cluster, thus replacing the running
 configuration with the one which is in the shadow. The `crm`
 prompt always contains the name of the configuration which is
 currently in use. Note that, for obvious reasons, only commands
 at the `configure` level make sense while working with a shadow
 CIB.
 
 No changes take place before the `configure commit` command.
 Sometimes though, the administrator may start working with the
 running configuration, but change mind and instead of committing
 the changes to the cluster save them to a shadow CIB. This
 short `configure` session excerpt shows how:
 ...............
     crm(live)configure# cib new test-2
     INFO: test-2 shadow CIB created
     crm(test-2)configure# commit
 ...............
 
 === Templates
 
 Templates are ready made configurations created by cluster
 experts. They are designed in such a way, so that users may
 generate valid cluster configurations with minimum effort.
 If you are new to Pacemaker/CRM, templates may be the best way to
 start.
 
 We will show here how to create a simple yet functional Apache
 configuration:
 ...............
     # crm configure
     crm(live)configure# template
     crm(live)configure template# list templates
     apache       filesystem   virtual-ip   
     crm(live)configure template# new web <TAB><TAB>
     apache       filesystem   virtual-ip   
     crm(live)configure template# new web apache
     INFO: pulling in template apache
     INFO: pulling in template virtual-ip
     crm(live)configure template# list
     web2-d       web2         vip2         web3         vip          web
 ...............
 
 We enter the `template` level from `configure`. Use the `list`
 command to show templates available on the system.
 The `new` command creates a configuration from the `apache`
 template. You can use tab completion to pick templates. Note that
 the apache template depends on a virtual IP address which is
 automatically pulled along. The `list` command shows the just
 created `web` configuration, among other configurations (I hope
 that you, unlike me, will use more sensible and descriptive names).
 
 The `show` command, which displays the resulting configuration,
 may be used to get an idea about the minimum required changes
 which have to be done. All `ERROR` messages show the line numbers
 in which the respective parameters are to be defined:
 ...............
     crm(live)configure template# show
     ERROR: 23: required parameter ip not set
     ERROR: 61: required parameter id not set
     ERROR: 65: required parameter configfile not set
     crm(live)configure template# edit
 ...............
 
 The `edit` command invokes the preferred text editor with the
 `web` configuration. At the top of the file, the user is advised
 how to make changes. A good template should require from the user
 to specify only parameters. For example, the `web` configuration
 we created above has the following required and optional
 parameters (all parameter lines start with `%%`):
 ...............
     $ grep -n ^%% ~/.crmconf/web
     23:%% ip 
     31:%% netmask 
     35:%% lvs_support 
     61:%% id 
     65:%% configfile 
     71:%% options 
     76:%% envfiles 
 ...............
 
 These lines are the only ones that should be modified. Simply
 append the parameter value at the end of the line. For instance,
 after editing this template, the result could look like this (we
 used tabs instead of spaces to make the values stand out):
 ...............
     $ grep -n ^%% ~/.crmconf/web
     23:%% ip                192.168.1.101
     31:%% netmask 
     35:%% lvs_support 
     61:%% id                websvc
     65:%% configfile        /etc/apache2/httpd.conf
     71:%% options 
     76:%% envfiles 
 ...............
 
 As you can see, the parameter line format is very simple:
 ...............
     %% <name> <value>
 ...............
 
 After editing the file, use `show` again to display the
 configuration:
 ...............
     crm(live)configure template# show
     primitive virtual-ip ocf:heartbeat:IPaddr \
             params ip="192.168.1.101"
     primitive apache ocf:heartbeat:apache \
             params configfile="/etc/apache2/httpd.conf"
     monitor apache 120s:60s
     group websvc \
             apache virtual-ip
 ...............
 
 The target resource of the apache template is a group which we
 named `websvc` in this sample session.
 
 This configuration looks exactly as you could type it at the
 `configure` level. The point of templates is to save you some
 typing. It is important, however, to understand the configuration
 produced.
 
 Finally, the configuration may be applied to the current
 crm configuration (note how the configuration changed slightly,
 though it is still equivalent, after being digested at the
 `configure` level):
 ...............
     crm(live)configure template# apply 
     crm(live)configure template# cd ..
     crm(live)configure# show
     node xen-b
     node xen-c
     primitive apache ocf:heartbeat:apache \
         params configfile="/etc/apache2/httpd.conf" \
         op monitor interval="120s" timeout="60s"
     primitive virtual-ip ocf:heartbeat:IPaddr \
         params ip="192.168.1.101"
     group websvc apache virtual-ip
 ...............
 
 Note that this still does not commit the configuration to the CIB
 which is used in the shell, either the running one (`live`) or
 some shadow CIB. For that you still need to execute the `commit`
 command.
 
 We should also define the preferred node to run the service:
 ...............
     crm(live)configure# location websvc-pref websvc 100: xen-b
 ...............
 
 If you are not happy with some resource names which are provided
 by default, you can rename them now:
 ...............
     crm(live)configure# rename virtual-ip intranet-ip
     crm(live)configure# show
     node xen-b
     node xen-c
     primitive apache ocf:heartbeat:apache \
             params configfile="/etc/apache2/httpd.conf" \
             op monitor interval="120s" timeout="60s"
     primitive intranet-ip ocf:heartbeat:IPaddr \
             params ip="192.168.1.101"
     group websvc apache intranet-ip
     location websvc-pref websvc 100: xen-b
 ...............
 
 To summarize, working with templates typically consists of the
 following steps:
 
 - `new`: create a new configuration from templates
 - `edit`: define parameters, at least the required ones
 - `show`: see if the configuration is valid
 - `apply`: apply the configuration to the `configure` level
 
 === Tab completion
 
 The `crm` makes extensive use of the tab completion of
 `readline`. The completion is both static (i.e. for `crm`
 commands) and dynamic. The latter takes into account the current
 status of the cluster or information from installed resource
 agents. Sometimes, completion may also be used to get short help
 on resource parameters. Here a few examples:
 ...............
     crm(live)# resource <TAB><TAB>
     bye         exit        manage      param       show        unmanage 
     cd          failcount   meta        quit        start       unmigrate 
     cleanup     help        migrate     refresh     status      unmove 
     end         list        move        reprobe     stop        up 
     crm(live)# configure 
     crm(live)configure# primitive fence-1 <TAB><TAB>
     heartbeat:  lsb:        ocf:        stonith:    
     crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB>
     auth=      hostname=  ipaddr=    login=     password=  port=      priv=
     crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB>
     auth* (string)
         The authorization type of the IPMI session ("none", "straight", "md2", or "md5")
     crm(live)configure# primitive fence-1 stonith:ipmilan params auth=
 ...............
 
 [[SEMCHK]]
 === Configuration semantic checks
 
 Resource definitions may be checked against the meta-data
 provided with the resource agents. These checks are currently
 carried out:
 
 - are required parameters set
 - existence of defined parameters
 - timeout values for operations
 
 The parameter checks are obvious and need no further explanation.
 Failures in these checks are treated as configuration errors.
 
 The timeouts for operations should be at least as long as those
 recommended in the meta-data. Too short timeout values are a
 common mistake in cluster configurations and, even worse, they
 often slip through if cluster testing was not thorough. Though
 operation timeouts issues are treated as warnings, make sure that
 the timeouts are usable in your environment. Note also that the
 values given are just _advisory minimum_---your resources may
 require longer timeouts.
 
 User may tune the frequency of checks and the treatment of errors
 by the <<cmdhelp_options_check-frequency,`check-frequency`>> and
 <<cmdhelp_options_check-mode,`check-mode`>> preferences.
 
 Note that if the `check-frequency` is set to `always` and the
 `check-mode` to `strict`, errors are not tolerated and such
 configuration cannot be saved.
 
 == Reference
 
 We define a small and simple language. Most commands consist of
 just a list of simple tokens. The only complex constructs are
 found at the `configure` level.
 
 The syntax is described in a somewhat informal manner: `<>`
 denotes a string, `[]` means that the construct is optional, the
 ellipsis (`...`) signifies that the previous construct may be
 repeated, `|` means pick one of many, and the rest are literals
 (strings, `:`, `=`).
 
 === `status`
 
 Show cluster status. The status is displayed by `crm_mon`. Supply
 additional arguments for more information or different format.
 See `crm_mon(8)` for more details.
 
 Usage:
 ...............
         status [<option> ...]
 
         option :: bynode | inactive | ops | timing | failcounts
 ...............
 
 [[cmdhelp_cib,CIB shadow management]]
 === `cib` (shadow CIBs)
 
 This level is for management of shadow CIBs. It is available both
 at the top level and the `configure` level.
 
 All the commands are implemented using `cib_shadow(8)` and the
 `CIB_shadow` environment variable. The user prompt always
 includes the name of the currently active shadow or the live CIB.
 
 [[cmdhelp_cib_new,create a new shadow CIB]]
 ==== `new`
 
 Create a new shadow CIB. The live cluster configuration and
 status is copied to the shadow CIB. Specify `withstatus` if you
 want to edit the status section of the shadow CIB (see the
 <<cmdhelp_cibstatus,cibstatus section>>). Add `force` to force overwriting the
 existing shadow CIB.
 
+To start with an empty configuration that is not copied from the live
+CIB, specify the `empty` keyword. (This also allows a shadow CIB to be
+created in case no cluster is running.)
+
 Usage:
 ...............
-        new <cib> [withstatus] [force]
+        new <cib> [withstatus] [force] [empty]
 ...............
 
 [[cmdhelp_cib_delete,delete a shadow CIB]]
 ==== `delete`
 
 Delete an existing shadow CIB.
 
 Usage:
 ...............
         delete <cib>
 ...............
 
 [[cmdhelp_cib_reset,copy live cib to a shadow CIB]]
 ==== `reset`
 
 Copy the current cluster configuration into the shadow CIB.
 
 Usage:
 ...............
         reset <cib>
 ...............
 
 [[cmdhelp_cib_commit,copy a shadow CIB to the cluster]]
 ==== `commit`
 
 Apply a shadow CIB to the cluster.
 
 Usage:
 ...............
         commit <cib>
 ...............
 
 [[cmdhelp_cib_use,change working CIB]]
 ==== `use`
 
 Choose a CIB source. If you want to edit the status from the
 shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>).
 Leave out the CIB name to switch to the running CIB.
 
 Usage:
 ...............
         use [<cib>] [withstatus]
 ...............
 
 [[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]]
 ==== `diff`
 
 Print differences between the current cluster configuration and
 the active shadow CIB.
 
 Usage:
 ...............
         diff
 ...............
 
 [[cmdhelp_cib_list,list all shadow CIBs]]
 ==== `list`
 
 List existing shadow CIBs.
 
 Usage:
 ...............
         list
 ...............
 
 [[cmdhelp_cib_import,import a CIB or PE input file to a shadow]]
 ==== `import`
 
 At times it may be useful to create a shadow file from the
 existing CIB. The CIB may be specified as file or as a PE input
 file number. The shell will look up files in the local directory
 first and then in the PE directory (typically `/var/lib/pengine`).
 Once the CIB file is found, it is copied to a shadow and this
 shadow is immediately available for use at both `configure` and
 `cibstatus` levels.
 
 If the shadow name is omitted then the target shadow is named
 after the input CIB file.
 
 Note that there are often more than one PE input file, so you may
 need to specify the full name.
 
 Usage:
 ...............
         import {<file>|<number>} [<shadow>]
 ...............
 Examples:
 ...............
         import pe-warn-2222
         import 2289 issue2
 ...............
 
 [[cmdhelp_cib_cibstatus,CIB status management and editing]]
 ==== `cibstatus`
 
 Enter edit and manage the CIB status section level. See the
 <<cmdhelp_cibstatus,CIB status management section>>.
 
 [[cmdhelp_ra,Resource Agents (RA) lists and documentation]]
 === `ra`
 
 This level contains commands which show various information about
 the installed resource agents. It is available both at the top
 level and at the `configure` level.
 
 [[cmdhelp_ra_classes,list classes and providers]]
 ==== `classes`
 
 Print all resource agents' classes and, where appropriate, a list
 of available providers.
 
 Usage:
 ...............
         classes
 ...............
 
 [[cmdhelp_ra_list,list RA for a class (and provider)]]
 ==== `list`
 
 List available resource agents for the given class. If the class
 is `ocf`, supply a provider to get agents which are available
 only from that provider.
 
 Usage:
 ...............
         list <class> [<provider>]
 ...............
 Example:
 ...............
         list ocf pacemaker
 ...............
 
 [[cmdhelp_ra_meta,show meta data for a RA]]
 ==== `meta` (`info`)
 
 Show the meta-data of a resource agent type. This is where users
 can find information on how to use a resource agent.
 
 Usage:
 ...............
         meta [<class>:[<provider>:]]<type>
         meta <type> <class> [<provider>] (obsolete)
 ...............
 Example:
 ...............
         meta apache
         meta ocf:pacemaker:Dummy
         meta stonith:ipmilan
 ...............
 
 [[cmdhelp_ra_providers,show providers for a RA and a class]]
 ==== `providers`
 
 List providers for a resource agent type. The class parameter
 defaults to `ocf`.
 
 Usage:
 ...............
         providers <type> [<class>]
 ...............
 Example:
 ...............
         providers apache
 ...............
 
 [[cmdhelp_resource,Resource management]]
 === `resource`
 
 At this level resources may be managed.
 
 All (or almost all) commands are implemented with the CRM tools
 such as `crm_resource(8)`.
 
 [[cmdhelp_resource_status,show status of resources]]
 ==== `status` (`show`, `list`)
 
 Print resource status. If the resource parameter is left out
 status of all resources is printed.
 
 Usage:
 ...............
         status [<rsc>]
 ...............
 
 [[cmdhelp_resource_start,start a resource]]
 ==== `start`
 
 Start a resource by setting the `target-role` attribute. If there
 are multiple meta attributes sets, the attribute is set in all of
 them. If the resource is a group or a clone, all `target-role`
 attributes are removed from the children resources.
 
 Usage:
 ...............
         start <rsc>
 ...............
 
 [[cmdhelp_resource_stop,stop a resource]]
 ==== `stop`
 
 Stop a resource using the `target-role` attribute. If there
 are multiple meta attributes sets, the attribute is set in all of
 them. If the resource is a group or a clone, all `target-role`
 attributes are removed from the children resources.
 
 Usage:
 ...............
         stop <rsc>
 ...............
 
 [[cmdhelp_resource_restart,restart a resource]]
 ==== `restart`
 
 Restart a resource. This is essentially a shortcut for resource
 stop followed by a start.
 
 Usage:
 ...............
         restart <rsc>
 ...............
 
 [[cmdhelp_resource_promote,promote a master-slave resource]]
 ==== `promote`
 
 Promote a master-slave resource using the `target-role`
 attribute.
 
 Usage:
 ...............
         promote <rsc>
 ...............
 
 [[cmdhelp_resource_demote,demote a master-slave resource]]
 ==== `demote`
 
 Demote a master-slave resource using the `target-role`
 attribute.
 
 Usage:
 ...............
         demote <rsc>
 ...............
 
 [[cmdhelp_resource_manage,put a resource into managed mode]]
 ==== `manage`
 
 Manage a resource using the `is-managed` attribute. If there
 are multiple meta attributes sets, the attribute is set in all of
 them. If the resource is a group or a clone, all `is-managed`
 attributes are removed from the children resources.
 
 Usage:
 ...............
         manage <rsc>
 ...............
 
 [[cmdhelp_resource_unmanage,put a resource into unmanaged mode]]
 ==== `unmanage`
 
 Unmanage a resource using the `is-managed` attribute. If there
 are multiple meta attributes sets, the attribute is set in all of
 them. If the resource is a group or a clone, all `is-managed`
 attributes are removed from the children resources.
 
 Usage:
 ...............
         unmanage <rsc>
 ...............
 
 [[cmdhelp_resource_migrate,migrate a resource to another node]]
 ==== `migrate` (`move`)
 
 Migrate a resource to a different node. If node is left out, the
 resource is migrated by creating a constraint which prevents it from
 running on the current node. Additionally, you may specify a
 lifetime for the constraint---once it expires, the location
 constraint will no longer be active.
 
 Usage:
 ...............
         migrate <rsc> [<node>] [<lifetime>] [force]
 ...............
 
 [[cmdhelp_resource_unmigrate,unmigrate a resource to another node]]
 ==== `unmigrate` (`unmove`)
 
 Remove the constraint generated by the previous migrate command.
 
 Usage:
 ...............
         unmigrate <rsc>
 ...............
 
 [[cmdhelp_resource_param,manage a parameter of a resource]]
 ==== `param`
 
 Show/edit/delete a parameter of a resource.
 
 Usage:
 ...............
         param <rsc> set <param> <value>
         param <rsc> delete <param>
         param <rsc> show <param>
 ...............
 Example:
 ...............
         param ip_0 show ip
 ...............
 
 [[cmdhelp_resource_meta,manage a meta attribute]]
 ==== `meta`
 
 Show/edit/delete a meta attribute of a resource. Currently, all
 meta attributes of a resource may be managed with other commands
 such as `resource stop`.
 
 Usage:
 ...............
         meta <rsc> set <attr> <value>
         meta <rsc> delete <attr>
         meta <rsc> show <attr>
 ...............
 Example:
 ...............
         meta ip_0 set target-role stopped
 ...............
 
 [[cmdhelp_resource_utilization,manage a utilization attribute]]
 ==== `utilization`
 
 Show/edit/delete a utilization attribute of a resource. These
 attributes describe hardware requirements. By setting the
 `placement-strategy` cluster property appropriately, it is
 possible then to distribute resources based on resource
 requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>.
 
 Usage:
 ...............
         utilization <rsc> set <attr> <value>
         utilization <rsc> delete <attr>
         utilization <rsc> show <attr>
 ...............
 Example:
 ...............
         utilization xen1 set memory 4096
 ...............
 
 [[cmdhelp_resource_failcount,manage failcounts]]
 ==== `failcount`
 
 Show/edit/delete the failcount of a resource.
 
 Usage:
 ...............
         failcount <rsc> set <node> <value>
         failcount <rsc> delete <node>
         failcount <rsc> show <node>
 ...............
 Example:
 ...............
         failcount fs_0 delete node2
 ...............
 
 [[cmdhelp_resource_cleanup,cleanup resource status]]
 ==== `cleanup`
 
 Cleanup resource status. Typically done after the resource has
 temporarily failed. If a node is omitted, cleanup on all nodes.
 If there are many nodes, the command may take a while.
 
 Usage:
 ...............
         cleanup <rsc> [<node>]
 ...............
 
 [[cmdhelp_resource_refresh,refresh CIB from the LRM status]]
 ==== `refresh`
 
 Refresh CIB from the LRM status.
 
 Usage:
 ...............
         refresh [<node>]
 ...............
 
 [[cmdhelp_resource_reprobe,probe for resources not started by the CRM]]
 ==== `reprobe`
 
 Probe for resources not started by the CRM.
 
 Usage:
 ...............
         reprobe [<node>]
 ...............
 
 [[cmdhelp_node,Nodes management]]
 === `node`
 
 Node management and status commands.
 
 [[cmdhelp_node_status,show nodes' status]]
 ==== `status`
 
 Show nodes' status. If the node parameter is omitted then all
 nodes are shown.
 
 Usage:
 ...............
         status [<node>]
 ...............
 
 [[cmdhelp_node_show,show node]]
 ==== `show`
 
 Show a node definition. If the node parameter is omitted then all
 nodes are shown.
 
 Usage:
 ...............
         show [<node>]
 ...............
 
 [[cmdhelp_node_standby,put node into standby]]
 ==== `standby`
 
 Set a node to standby status. The node parameter defaults to the
 node where the command is run. Additionally, you may specify a
 lifetime for the standby---if set to `reboot`, the node will be
 back online once it reboots. `forever` will keep the node in
 standby after reboot.
 
 Usage:
 ...............
         standby [<node>] [<lifetime>]
 
         lifetime :: reboot | forever
 ...............
 
 [[cmdhelp_node_online,set node online]]
 ==== `online`
 
 Set a node to online status. The node parameter
 defaults to the node where the command is run.
 
 Usage:
 ...............
         online [<node>]
 ...............
 
 [[cmdhelp_node_fence,fence node]]
 ==== `fence`
 
 Make CRM fence a node. This functionality depends on stonith
 resources capable of fencing the specified node. No such stonith
 resources, no fencing will happen.
 
 Usage:
 ...............
         fence <node>
 ...............
 
 [[cmdhelp_node_clearstate,Clear node state]]
 ==== `clearnodestate`
 
 Resets and clears the state of the specified node. This node is
 afterwards assumed clean and offline. This command can be used to
 manually confirm that a node has been fenced (e.g., powered off).
 
 Be careful! This can cause data corruption if you confirm that a node is
 down that is, in fact, not cleanly down - the cluster will proceed as if
 the fence had succeeded, possibly starting resources multiple times.
 
 Usage:
 ...............
         clearstate <node>
 ...............
 
 [[cmdhelp_node_delete,delete node]]
 ==== `delete`
 
 Delete a node. This command will remove the node from the CIB
 and, in case the heartbeat stack is running, run hb_delnode too.
 
 Usage:
 ...............
         delete <node>
 ...............
 
 [[cmdhelp_node_attribute,manage attributes]]
 ==== `attribute`
 
 Edit node attributes. This kind of attribute should refer to
 relatively static properties, such as memory size.
 
 Usage:
 ...............
         attribute <node> set <attr> <value>
         attribute <node> delete <attr>
         attribute <node> show <attr>
 ...............
 Example:
 ...............
         attribute node_1 set memory_size 4096
 ...............
 
 [[cmdhelp_node_utilization,manage utilization attributes]]
 ==== `utilization`
 
 Edit node utilization attributes. These attributes describe
 hardware characteristics as integer numbers such as memory size
 or the number of CPUs. By setting the `placement-strategy`
 cluster property appropriately, it is possible then to distribute
 resources based on resource requirements and node size. See also
 <<cmdhelp_resource_utilization,resource utilization attributes>>.
 
 Usage:
 ...............
         utilization <node> set <attr> <value>
         utilization <node> delete <attr>
         utilization <node> show <attr>
 ...............
 Examples:
 ...............
         utilization node_1 set memory 16384
         utilization node_1 show cpu
 ...............
 
 [[cmdhelp_node_status-attr,manage status attributes]]
 ==== `status-attr`
 
 Edit node attributes which are in the CIB status section, i.e.
 attributes which hold properties of a more volatile nature. One
 typical example is attribute generated by the `pingd` utility.
 
 Usage:
 ...............
         status-attr <node> set <attr> <value>
         status-attr <node> delete <attr>
         status-attr <node> show <attr>
 ...............
 Example:
 ...............
         status-attr node_1 show pingd
 ...............
 
 [[cmdhelp_options,user preferences]]
 === `options`
 
 The user may set various options for the CLI program itself.
 
 [[cmdhelp_options_skill-level,set skill level]]
 ==== `skill-level`
 
 Based on the skill-level setting, the user is allowed to use only
 a subset of commands. There are three levels: operator,
 administrator, and expert. The operator level allows only
 commands at the `resource` and `node` levels, but not editing
 or deleting resources. The administrator may do that and may also
 configure the cluster at the `configure` level and manage the
 shadow CIBs. The expert may do all.
 
 Usage:
 ...............
         skill-level <level>
         
         level :: operator | administrator | expert
 ...............
 
 [[cmdhelp_options_user,set the cluster user]]
 ==== `user`
 
 Sufficient privileges are necessary in order to manage a
 cluster: programs such as `crm_verify` or `crm_resource` and,
 ultimately, `cibadmin` have to be run either as `root` or as the
 CRM owner user (typically `hacluster`). You don't have to worry
 about that if you run `crm` as `root`. A more secure way is to
 run the program with your usual privileges, set this option to
 the appropriate user (such as `hacluster`), and setup the
 `sudoers` file.
 
 Usage:
 ...............
         user system-user
 ...............
 Example:
 ...............
         user hacluster
 ...............
 
 [[cmdhelp_options_editor,set preferred editor program]]
 ==== `editor`
 
 The `edit` command invokes an editor. Use this to specify your
 preferred editor program. If not set, it will default to either
 the value of the `EDITOR` environment variable or to one of the
 standard UNIX editors (`vi`,`emacs`,`nano`).
 
 Usage:
 ...............
         editor program
 ...............
 Example:
 ...............
         editor vim
 ...............
 
 [[cmdhelp_options_pager,set preferred pager program]]
 ==== `pager`
 
 The `view` command displays text through a pager. Use this to
 specify your preferred pager program. If not set, it will default
 to either the value of the `PAGER` environment variable or to one
 of the standard UNIX system pagers (`less`,`more`,`pg`).
 
 [[cmdhelp_options_sort-elements,sort CIB elements]]
 ==== `sort-elements`
 
 `crm` by default sorts CIB elements. If you want them appear in
 the order they were created, set this option to `no`.
 
 Usage:
 ...............
         sort-elements {yes|no}
 ...............
 Example:
 ...............
         sort-elements no
 ...............
 
 [[cmdhelp_options_output,set output type]]
 ==== `output`
 
 `crm` can adorn configurations in two ways: in color (similar to
 for instance the `ls --color` command) and by showing keywords in
 upper case. Possible values are `plain`, `color`, and
 'uppercase'. It is possible to combine the latter two in order to
 get an upper case xmass tree. Just set this option to
 `color,uppercase`.
 
 [[cmdhelp_options_colorscheme,set colors for output]]
 ==== `colorscheme`
 
 With `output` set to `color`, a comma separated list of colors
 from this option are used to emphasize:
 
 - keywords
 - object ids
 - attribute names
 - attribute values
 - scores
 - resource references
 
 `crm` can show colors only if there is curses support for python
 installed (usually provided by the `python-curses` package). The
 colors are whatever is available in your terminal. Use `normal`
 if you want to keep the default foreground color.
 
 This user preference defaults to
 `yellow,normal,cyan,red,green,magenta` which is good for
 terminals with dark background. You may want to change the color
 scheme and save it in the preferences file for other color
 setups.
 
 Example:
 ...............
     colorscheme yellow,normal,blue,red,green,magenta
 ...............
 
 [[cmdhelp_options_check-frequency,when to perform semantic check]]
 ==== `check-frequency`
 
 Semantic check of the CIB or elements modified or created may be
 done on every configuration change (`always`), when verifying
 (`on-verify`) or `never`. It is by default set to `always`.
 Experts may want to change the setting to `on-verify`.
 
 The checks require that resource agents are present. If they are
 not installed at the configuration time set this preference to
 `never`.
 
 See <<SEMCHK,Configuration semantic checks>> for more details.
 
 [[cmdhelp_options_check-mode,how to treat semantic errors]]
 ==== `check-mode`
 
 Semantic check of the CIB or elements modified or created may be
 done in the `strict` mode or in the `relaxed` mode. In the former
 certain problems are treated as configuration errors. In the
 `relaxed` mode all are treated as warnings. The default is `strict`.
 
 See <<SEMCHK,Configuration semantic checks>> for more details.
 
 [[cmdhelp_options_show,show current user preference]]
 ==== `show`
 
 Display all current settings.
 
 [[cmdhelp_options_save,save the user preferences to the rc file]]
 ==== `save`
 
 Save current settings to the rc file (`$HOME/.crm.rc`). On
 further `crm` runs, the rc file is automatically read and parsed.
 
 [[cmdhelp_configure,CIB configuration]]
 === `configure`
 
 This level enables all CIB object definition commands.
 
 The configuration may be logically divided into four parts:
 nodes, resources, constraints, and (cluster) properties and
 attributes.  Each of these commands support one or more basic CIB
 objects.
 
 Nodes and attributes describing nodes are managed using the
 `node` command.
 
 Commands for resources are:
 
 - `primitive`
 - `monitor`
 - `group`
 - `clone`
 - `ms`/`master` (master-slave)
 
 There are three types of constraints:
 
 - `location`
 - `colocation`
 - `order`
 
 Finally, there are the cluster properties, resource meta
 attributes defaults, and operations defaults. All are just a set
 of attributes. These attributes are managed by the following
 commands:
 
 - `property`
 - `rsc_defaults`
 - `op_defaults`
 
 The changes applied to the current CIB only on ending the
 configuration session or using the `commit` command.
 
 Comments start with `#` in the first line. The comments are tied
 to the element which follows. If the element moves, its comments
 will follow.
 
 [[cmdhelp_configure_node,define a cluster node]]
 ==== `node`
 
 The node command describes a cluster node. Nodes in the CIB are
 commonly created automatically by the CRM. Hence, you should not
 need to deal with nodes unless you also want to define node
 attributes. Note that it is also possible to manage node
 attributes at the `node` level.
 
 Usage:
 ...............
         node <uname>[:<type>]
           [attributes <param>=<value> [<param>=<value>...]]
           [utilization <param>=<value> [<param>=<value>...]]
 
         type :: normal | member | ping
 ...............
 Example:
 ...............
         node node1
         node big_node attributes memory=64
 ...............
 
 [[cmdhelp_configure_primitive,define a resource]]
 ==== `primitive`
 
 The primitive command describes a resource. It may be referenced
 only once in group, clone, or master-slave objects. If it's not
 referenced, then it is placed as a single resource in the CIB.
 
 Operations may be specified in three ways. "Anonymous" as a
 simple list of "op" specifications. Use that if you don't want to
 reference the set of operations elsewhere. That's by far the most
 common way to define operations. If reusing operation sets is
 desired, use the "operations" keyword along with the id to give
 the operations set a name and the id-ref to reference another set
 of operations.
 
 Operation's attributes which are not recognized are saved as
 instance attributes of that operation. A typical example is
 `OCF_CHECK_LEVEL`.
 
 For multistate resources, roles are specified as `role=<role>`.
 
 Usage:
 ...............
         primitive <rsc> [<class>:[<provider>:]]<type>
           [params attr_list]
           [meta attr_list]
           [utilization attr_list]
           [operations id_spec]
             [op op_type [<attribute>=<value>...] ...]
 
         attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
         id_spec :: $id=<id> | $id-ref=<id>
         op_type :: start | stop | monitor
 ...............
 Example:
 ...............
         primitive apcfence stonith:apcsmart \
           params ttydev=/dev/ttyS0 hostlist="node1 node2" \
           op start timeout=60s \
           op monitor interval=30m timeout=60s
 
         primitive www8 apache \
           params configfile=/etc/apache/www8.conf \
           operations $id-ref=apache_ops
 
         primitive db0 mysql \
           params config=/etc/mysql/db0.conf \
           op monitor interval=60s \
           op monitor interval=300s OCF_CHECK_LEVEL=10
 
         primitive r0 ocf:linbit:drbd \
           params drbd_resource=r0 \
           op monitor role=Master interval=60s \
           op monitor role=Slave interval=300s
 ...............
 
 [[cmdhelp_configure_monitor,add monitor operation to a primitive]]
 ==== `monitor`
 
 Monitor is by far the most common operation. It is possible to
 add it without editing the whole resource. Also, long primitive
 definitions may be a bit uncluttered. In order to make this
 command as concise as possible, less common operation attributes
 are not available. If you need them, then use the `op` part of
 the `primitive` command.
 
 Usage:
 ...............
         monitor <rsc>[:<role>] <interval>[:<timeout>]
 ...............
 Example:
 ...............
         monitor apcfence 60m:60s
 ...............
 
 Note that after executing the command, the monitor operation may
 be shown as part of the primitive definition.
 
 [[cmdhelp_configure_group,define a group]]
 ==== `group`
 
 The `group` command creates a group of resources.
 
 Usage:
 ...............
         group <name> <rsc> [<rsc>...]
           [meta attr_list]
           [params attr_list]
 
         attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
 ...............
 Example:
 ...............
         group internal_www disk0 fs0 internal_ip apache \
           meta target_role=stopped
 ...............
 
 [[cmdhelp_configure_clone,define a clone]]
 ==== `clone`
 
 The `clone` command creates a resource clone. It may contain a
 single primitive resource or one group of resources.
 
 Usage:
 ...............
         clone <name> <rsc>
           [meta attr_list]
           [params attr_list]
 
         attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
 ...............
 Example:
 ...............
         clone cl_fence apc_1 \
           meta clone-node-max=1 globally-unique=false
 ...............
 
 [[cmdhelp_configure_ms,define a master-slave resource]]
 ==== `ms` (`master`)
 
 The `ms` command creates a master/slave resource type. It may contain a
 single primitive resource or one group of resources.
 
 Usage:
 ...............
         ms <name> <rsc>
           [meta attr_list]
           [params attr_list]
 
         attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
 ...............
 Example:
 ...............
         ms disk1 drbd1 \
           meta notify=true globally-unique=false
 ...............
 
 .Note on `id-ref` usage
 ****************************
 Instance or meta attributes (`params` and `meta`) may contain
 a reference to another set of attributes. In that case, no other
 attributes are allowed. Since attribute sets' ids, though they do
 exist, are not shown in the `crm`, it is also possible to
 reference an object instead of an attribute set. `crm` will
 automatically replace such a reference with the right id:
 
 ...............
     crm(live)configure# primitive a2 www-2 meta $id-ref=a1
     crm(live)configure# show a2
     primitive a2 ocf:heartbeat:apache \
         meta $id-ref="a1-meta_attributes"
         [...]
 ...............
 It is advisable to give meaningful names to attribute sets which
 are going to be referenced.
 ****************************
 
 [[cmdhelp_configure_location,a location preference]]
 ==== `location`
 
 `location` defines the preference of nodes for the given
 resource. The location constraints consist of one or more rules
 which specify a score to be awarded if the rule matches.
 
 Usage:
 ...............
         location <id> <rsc> {node_pref|rules}
 
         node_pref :: <score>: <node>
 
         rules ::
           rule [id_spec] [$role=<role>] <score>: <expression>
           [rule [id_spec] [$role=<role>] <score>: <expression> ...]
 
         id_spec :: $id=<id> | $id-ref=<id>
         score :: <number> | <attribute> | [-]inf
         expression :: <simple_exp> [bool_op <simple_exp> ...]
         bool_op :: or | and
         simple_exp :: <attribute> [type:]<binary_op> <value>
                       | <unary_op> <attribute>
                       | date <date_expr>
         type :: string | version | number
         binary_op :: lt | gt | lte | gte | eq | ne
         unary_op :: defined | not_defined
 
         date_expr :: lt <end>
                      | gt <start>
                      | in_range start=<start> end=<end>
                      | in_range start=<start> <duration>
                      | date_spec <date_spec>
         duration|date_spec ::
                      hours=<value>
                      | monthdays=<value>
                      | weekdays=<value>
                      | yearsdays=<value>
                      | months=<value>
                      | weeks=<value>
                      | years=<value>
                      | weekyears=<value>
                      | moon=<value>
 ...............
 Examples:
 ...............
         location conn_1 internal_www 100: node1
 
         location conn_1 internal_www \
           rule 50: #uname eq node1 \
           rule pingd: defined pingd
 
         location conn_2 dummy_float \
           rule -inf: not_defined pingd or pingd number:lte 0
 ...............
 
 [[cmdhelp_configure_colocation,colocate resources]]
 ==== `colocation` (`collocation`)
 
 This constraint expresses the placement relation between two
 or more resources. If there are more than two resources, then the
 constraint is called a resource set. Collocation resource sets have
 an extra attribute to allow for sets of resources which don't depend
 on each other in terms of state. The shell syntax for such sets is
 to put resources in parentheses.
 
 Usage:
 ...............
         colocation <id> <score>: <rsc>[:<role>] <rsc>[:<role>] ...
 ...............
 Example:
 ...............
         colocation dummy_and_apache -inf: apache dummy
         colocation c1 inf: A ( B C )
 ...............
 
 [[cmdhelp_configure_order,order resources]]
 ==== `order`
 
 This constraint expresses the order of actions on two resources
 or more resources. If there are more than two resources, then the
 constraint is called a resource set. Ordered resource sets have an
 extra attribute to allow for sets of resources whose actions may run
 in parallel. The shell syntax for such sets is to put resources in
 parentheses.
 
 Usage:
 ...............
         order <id> score-type: <rsc>[:<action>] <rsc>[:<action>] ...
           [symmetrical=<bool>]
 
         score-type :: advisory | mandatory | <score>
 ...............
 Example:
 ...............
         order c_apache_1 mandatory: apache:start ip_1
         order o1 inf: A ( B C )
 ...............
 
 [[cmdhelp_configure_property,set a cluster property]]
 ==== `property`
 
 Set the cluster (`crm_config`) options.
 
 Usage:
 ...............
         property [$id=<set_id>] <option>=<value> [<option>=<value> ...]
 ...............
 Example:
 ...............
         property stonith-enabled=true
 ...............
 
 [[cmdhelp_configure_rsc_defaults,set resource defaults]]
 ==== `rsc_defaults`
 
 Set defaults for the resource meta attributes.
 
 Usage:
 ...............
         rsc_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...]
 ...............
 Example:
 ...............
         rsc_defaults failure-timeout=3m
 ...............
 
 [[cmdhelp_configure_op_defaults,set resource operations defaults]]
 ==== `op_defaults`
 
 Set defaults for the operations meta attributes.
 
 Usage:
 ...............
         op_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...]
 ...............
 Example:
 ...............
         op_defaults record-pending=true
 ...............
 
 [[cmdhelp_configure_show,display CIB objects]]
 ==== `show`
 
 The `show` command displays objects. It may display all objects
 or a set of objects. The user may also choose to see only objects
 which were changed.
 Optionally, the XML code may be displayed instead of the CLI
 representation.
 
 Usage:
 ...............
         show [xml] [<id> ...]
         show [xml] changed
 ...............
 
 [[cmdhelp_configure_edit,edit CIB objects]]
 ==== `edit`
 
 This command invokes the editor with the object description. As
 with the `show` command, the user may choose to edit all objects
 or a set of objects.
 
 If the user insists, he or she may edit the XML edition of the
 object. If you do that, don't modify any id attributes.
 
 Usage:
 ...............
         edit [xml] [<id> ...]
         edit [xml] changed
 ...............
 
 [[cmdhelp_configure_filter,filter CIB objects]]
 ==== `filter`
 
 This command filters the given CIB elements through an external
 program. The program should accept input on `stdin` and send
 output to `stdout` (the standard UNIX filter conventions). As
 with the `show` command, the user may choose to filter all or
 just a subset of elements.
 
 It is possible to filter the XML representation of objects, but
 probably not as useful as the configuration language. The
 presentation is somewhat different from what would be displayed
 by the `show` command---each element is shown on a single, i.e.
 there are no backslashes and no other embelishments.
 
 Don't forget to put quotes around the filter if it contains
 spaces. Further, if you use this in a single-shot mode (from the
 `sh` command line), protect all quotes with a backslash.
 Otherwise, they won't reach the `crm` shell parser.
 
 Usage:
 ...............
         filter <prog> [xml] [<id> ...]
         filter <prog> [xml] changed
 ...............
 Examples:
 ...............
         filter "sed '/^primitive/s/target-role=[^ ]*//'"
 ...............
 
 [[cmdhelp_configure_delete,delete CIB objects]]
 ==== `delete`
 
 Delete one or more objects. If an object to be deleted belongs to
 a container object, such as a group, and it is the only resource
 in that container, then the container is deleted as well. Any
 related constraints are removed as well.
 
 Usage:
 ...............
         delete <id> [<id>...]
 ...............
 
 [[cmdhelp_configure_default-timeouts,set timeouts for operations to minimums from the meta-data]]
 ==== `default-timeouts`
 
 This command takes the timeouts from the actions section of the
 resource agent meta-data and sets them for the operations of the
 primitive.
 
 .Note
 ****************************
 You may be happy using this, but your applications may not. And
 it will tell you so at the worst possible moment. You have been
 warned.
 ****************************
 
 Usage:
 ...............
         default-timeouts <id> [<id>...]
 ...............
 
 [[cmdhelp_configure_rename,rename a CIB object]]
 ==== `rename`
 
 Rename an object. It is recommended to use this command to rename
 a resource, because it will take care of updating all related
 constraints and a parent resource. Changing ids with the edit
 command won't have the same effect.
 
 If you want to rename a resource, it must be in the stopped state.
 
 Usage:
 ...............
         rename <old_id> <new_id>
 ...............
 
 [[cmdhelp_configure_refresh,refresh from CIB]]
 ==== `refresh`
 
 Refresh the internal structures from the CIB. All changes made
 during this session are lost.
 
 Usage:
 ...............
         refresh
 ...............
 
 [[cmdhelp_configure_erase,erase the CIB]]
 ==== `erase`
 
 The `erase` clears all configuration. Apart from nodes. To remove
 nodes, you have to specify an additional keyword `nodes`.
 
 Note that removing nodes from the live cluster may have some
 strange/interesting/unwelcome effects.
 
 Usage:
 ...............
         erase [nodes]
 ...............
 
 [[cmdhelp_configure_ptest,show cluster actions if changes were committed]]
 ==== `ptest`
 
 Show PE (Policy Engine) motions using `ptest(8)`.
 
 A CIB is constructed using the current user edited configuration
 and the status from the running CIB. The resulting CIB is run
 through `ptest` to show changes which would happen if the
 configuration is committed.
 
 The status section may be loaded from another source and modified
 using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the
 `ptest` command will issue a message informing the user that the
 Policy Engine graph is not calculated based on the current status
 section and therefore won't show what would happen to the
 running but some imaginary cluster.
 
 If you have graphviz installed and X11 session, `dotty(1)` is run
 to display the changes graphically.
 
 Add a string of `v` characters to increase verbosity. `ptest`
 can also show allocation scores. `utilization` turns on
 information about the remaining capacity of nodes. With the
 `actions` option, `ptest` will print all resource actions.
 
 Usage:
 ...............
         ptest [nograph] [v...] [scores] [actions] [utilization]
 ...............
 Examples:
 ...............
         ptest scores
         ptest vvvvv
 ...............
 
 [[cmdhelp_configure_cib,CIB shadow management]]
 === `cib` (shadow CIBs)
 
 This level is for management of shadow CIBs. It is available at
 the `configure` level to enable saving intermediate changes to a
 shadow CIB instead of to the live cluster. This short excerpt
 shows how:
 ...............
     crm(live)configure# cib new test-2
     INFO: test-2 shadow CIB created
     crm(test-2)configure# commit
 ...............
 Note how the current CIB in the prompt changed from `live` to
 `test-2` after issuing the `cib new` command. See also the
 <<cmdhelp_cib,CIB shadow management>> for more information.
 
 [[cmdhelp_configure_cibstatus,CIB status management and editing]]
 ==== `cibstatus`
 
 Enter edit and manage the CIB status section level. See the
 <<cmdhelp_cibstatus,CIB status management section>>.
 
 [[cmdhelp_configure_template,edit and import a configuration from a template]]
 ==== `template`
 
 The specified template is loaded into the editor. It's up to the
 user to make a good CRM configuration out of it. See also the
 <<cmdhelp_template,template section>>.
 
 Usage:
 ...............
         template [xml] url
 ...............
 Example:
 ...............
         template two-apaches.txt
 ...............
 
 [[cmdhelp_configure_commit,commit the changes to the CIB]]
 ==== `commit`
 
 Commit the current configuration to the CIB in use. As noted
 elsewhere, commands in a configure session don't have immediate
 effect on the CIB. All changes are applied at one point in time,
 either using `commit` or when the user leaves the configure
 level. In case the CIB in use changed in the meantime, presumably
 by somebody else, the CLI will refuse to apply the changes. If
 you know that it's fine to still apply them add `force`.
 
 Usage:
 ...............
         commit [force]
 ...............
 
 [[cmdhelp_configure_verify,verify the CIB with crm_verify]]
 ==== `verify`
 
 Verify the contents of the CIB which would be committed.
 
 Usage:
 ...............
         verify
 ...............
 
 [[cmdhelp_configure_upgrade,upgrade the CIB to version 1.0]]
 ==== `upgrade`
 
 If you get the `CIB not supported` error, which typically means
 that the current CIB version is coming from the older release,
 you may try to upgrade it to the latest revision. The command
 to perform the upgrade is:
 ...............
     # cibadmin --upgrade --force
 ...............
 
 If we don't recognize the current CIB as the old one, but you're
 sure that it is, you may force the command.
 
 Usage:
 ...............
         upgrade [force]
 ...............
 
 [[cmdhelp_configure_save,save the CIB to a file]]
 ==== `save`
 
 Save the configuration of the current level to a file.
 Optionally, as XML.
 
 Usage:
 ...............
         save [xml] <file>
 ...............
 Example:
 ...............
         save myfirstcib.txt
 ...............
 
 [[cmdhelp_configure_load,import the CIB from a file]]
 ==== `load`
 
 Load a part of configuration (or all of it) from a local file or
 a network URL. The `replace` method replaces the current
 configuration with the one from the source. The `update` tries to
 import the contents into the current configuration.
 The file may be a CLI file or an XML file.
 
 Usage:
 ...............
         load [xml] <method> URL
 
         method :: replace | update
 ...............
 Example:
 ...............
         load xml update myfirstcib.xml
         load xml replace http://storage.big.com/cibs/bigcib.xml
 ...............
 
 [[cmdhelp_configure_xml,raw xml]]
 ==== `xml`
 
 Even though we promissed no xml, it may happen, but hopefully
 very very seldom, that an element from the CIB cannot be rendered
 in the configuration language. In that case, the element will be
 shown as raw xml, prefixed by this command. That element can then
 be edited like any other. If the shell finds out that after the
 change it can digest it, then it is going to be converted into
 the normal configuration language. Otherwise, there is no need to
 use `xml` for configuration.
 
 Usage:
 ...............
         xml <xml>
 ...............
 
 [[cmdhelp_template,edit and import a configuration from a template]]
 === `template`
 
 User may be assisted in the cluster configuration by templates
 prepared in advance. Templates consist of a typical ready
 configuration which may be edited to suit particular user needs.
 
 This command enters a template level where additional commands
 for configuration/template management are available.
 
 [[cmdhelp_template_new,create a new configuration from templates]]
 ==== `new`
 
 Create a new configuration from one or more templates. Note that
 configurations and templates are kept in different places, so it
 is possible to have a configuration name equal a template name.
 
 If you already know which parameters are required, you can set
 them directly on the command line.
 
 The parameter name `id` is set by default to the name of the
 configuration.
 
 Usage:
 ...............
         new <config> <template> [<template> ...] [params name=value ...]"
 ...............
 Examples:
 ...............
         new vip virtual-ip
         new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs
 ...............
 
 [[cmdhelp_template_load,load a configuration]]
 ==== `load`
 
 Load an existing configuration. Further `edit`, `show`, and
 `apply` commands will refer to this configuration.
 
 Usage:
 ...............
         load <config>
 ...............
 
 [[cmdhelp_template_edit,edit a configuration]]
 ==== `edit`
 
 Edit current or given configuration using your favourite editor.
 
 Usage:
 ...............
         edit [<config>]
 ...............
 
 [[cmdhelp_template_delete,delete a configuration]]
 ==== `delete`
 
 Remove a configuration. The loaded (active) configuration may be
 removed by force.
 
 Usage:
 ...............
         delete <config> [force]
 ...............
 
 [[cmdhelp_template_list,list configurations/templates]]
 ==== `list`
 
 List existing configurations or templates.
 
 Usage:
 ...............
         list [templates]
 ...............
 
 [[cmdhelp_template_apply,process and apply the current configuration to the current CIB]]
 ==== `apply`
 
 Copy the current or given configuration to the current CIB. By
 default, the CIB is replaced, unless the method is set to
 "update".
 
 Usage:
 ...............
         apply [<method>] [<config>]
 
         method :: replace | update
 ...............
 
 [[cmdhelp_template_show,show the processed configuration]]
 ==== `show`
 
 Process the current or given configuration and display the result.
 
 Usage:
 ...............
         show [<config>]
 ...............
 
 [[cmdhelp_cibstatus,CIB status management and editing]]
 === `cibstatus`
 
 The `status` section of the CIB keeps the current status of nodes
 and resources. It is modified _only_ on events, i.e. when some
 resource operation is run or node status changes. For obvious
 reasons, the CRM has no user interface with which it is possible
 to affect the status section. From the user's point of view, the
 status section is essentially a read-only part of the CIB. The
 current status is never even written to disk, though it is
 available in the PE (Policy Engine) input files which represent
 the history of cluster motions. The current status may be read
 using the `cibadmin -Q` command.
 
 It may sometimes be of interest to see how status changes would
 affect the Policy Engine. The set of `cibstatus` level commands
 allow the user to load status sections from various sources and
 then insert or modify resource operations or change nodes' state.
 
 The effect of those changes may then be observed by running the
 <<cmdhelp_configure_ptest,`ptest`>> command at the `configure` level
 or `simulate` and `run` commands at this level. The `ptest`
 runs with the user edited CIB whereas the latter two commands
 run with the CIB which was loaded along with the status section.
 
 The `simulate` and `run` commands as well as all status
 modification commands are implemented using `crm_simulate(8)`.
 
 [[cmdhelp_cibstatus_load,load the CIB status section]]
 ==== `load`
 
 Load a status section from a file, a shadow CIB, or the running
 cluster. By default, the current (`live`) status section is
 modified. Note that if the `live` status section is modified it
 is not going to be updated if the cluster status changes, because
 that would overwrite the user changes. To make `crm` drop changes
 and resume use of the running cluster status, run `load live`.
 
 All CIB shadow configurations contain the status section which is
 a snapshot of the status section taken at the time the shadow was
 created. Obviously, this status section doesn't have much to do
 with the running cluster status, unless the shadow CIB has just
 been created. Therefore, the `ptest` command by default uses the
 running cluster status section.
 
 Usage:
 ...............
         load {<file>|shadow:<cib>|live}
 ...............
 Example:
 ...............
         load bug-12299.xml
         load shadow:test1
 ...............
 
 [[cmdhelp_cibstatus_save,save the CIB status section]]
 ==== `save`
 
 The current internal status section with whatever modifications
 were performed can be saved to a file or shadow CIB.
 
 If the file exists and contains a complete CIB, only the status
 section is going to be replaced and the rest of the CIB will
 remain intact. Otherwise, the current user edited configuration
 is saved along with the status section.
 
 Note that all modifications are saved in the source file as soon
 as they are run.
 
 Usage:
 ...............
         save [<file>|shadow:<cib>]
 ...............
 Example:
 ...............
         save bug-12299.xml
 ...............
 
 [[cmdhelp_cibstatus_origin,display origin of the CIB status section]]
 ==== `origin`
 
 Show the origin of the status section currently in use. This
 essentially shows the latest `load` argument.
 
 Usage:
 ...............
         origin
 ...............
 
 [[cmdhelp_cibstatus_show,show CIB status section]]
 ==== `show`
 
 Show the current status section in the XML format. Brace yourself
 for some unreadable output. Add `changed` option to get a human
 readable output of all changes.
 
 Usage:
 ...............
         show [changed]
 ...............
 
 [[cmdhelp_cibstatus_node,change node status]]
 ==== `node`
 
 Change the node status. It is possible to throw a node out of
 the cluster, make it a member, or set its state to unclean.
 
 `online`:: Set the `node_state` `crmd` attribute to `online`
 and the `expected` and `join` attributes to `member`. The effect
 is that the node becomes a cluster member.
 
 `offline`:: Set the `node_state` `crmd` attribute to `offline`
 and the `expected` attribute to empty. This makes the node
 cleanly removed from the cluster.
 
 `unclean`:: Set the `node_state` `crmd` attribute to `offline`
 and the `expected` attribute to `member`. In this case the node
 has unexpectedly disappeared.
 
 Usage:
 ...............
         node <node> {online|offline|unclean}
 ...............
 Example:
 ...............
         node xen-b unclean
 ...............
 
 [[cmdhelp_cibstatus_op,edit outcome of a resource operation]]
 ==== `op`
 
 Edit the outcome of a resource operation. This way you can
 tell CRM that it ran an operation and that the resource agent
 returned certain exit code. It is also possible to change the
 operation's status. In case the operation status is set to
 something other than `done`, the exit code is effectively
 ignored.
 
 Usage:
 ...............
         op <operation> <resource> <exit_code> [<op_status>] [<node>]
 
         operation :: probe | monitor[:<n>] | start | stop |
            promote | demote | notify | migrate_to | migrate_from
         exit_code :: <rc> | success | generic | args | 
            unimplemented | perm | installed | configured | not_running | 
            master | failed_master
         op_status :: pending | done | cancelled | timeout | notsupported | error
 
         n :: the monitor interval in seconds; if omitted, the first
            recurring operation is referenced
         rc :: numeric exit code in range 0..9
 ...............
 Example:
 ...............
         op start d1 xen-b generic
         op start d1 xen-b 1
         op monitor d1 xen-b not_running
         op stop d1 xen-b 0 timeout
 ...............
 
 [[cmdhelp_cibstatus_quorum,set the quorum]]
 ==== `quorum`
 
 Set the quorum value.
 
 Usage:
 ...............
         quorum <bool>
 ...............
 Example:
 ...............
         quorum false
 ...............
 
 [[cmdhelp_cibstatus_run,run policy engine]]
 ==== `run`
 
 Run the policy engine with the edited status section.
 
 Add a string of `v` characters to increase verbosity. Specify
 `scores` to see allocation scores also. `utilization` turns on
 information about the remaining capacity of nodes.
 
 If you have graphviz installed and X11 session, `dotty(1)` is run
 to display the changes graphically.
 
 Usage:
 ...............
         run [nograph] [v...] [scores] [utilization]
 ...............
 Example:
 ...............
         run
 ...............
 
 [[cmdhelp_cibstatus_simulate,simulate cluster transition]]
 ==== `simulate`
 
 Run the policy engine with the edited status section and simulate
 the transition.
 
 Add a string of `v` characters to increase verbosity. Specify
 `scores` to see allocation scores also. `utilization` turns on
 information about the remaining capacity of nodes.
 
 If you have graphviz installed and X11 session, `dotty(1)` is run
 to display the changes graphically.
 
 Usage:
 ...............
         simulate [nograph] [v...] [scores] [utilization]
 ...............
 Example:
 ...............
         simulate
 ...............
 
 === `end` (`cd`, `up`)
 
 The `end` command ends the current level and the user moves to
 the parent level. This command is available everywhere.
 
 Usage:
 ...............
         end
 ...............
 
 === `help`
 
 The `help` command prints help for the current level or for the
 specified topic (command). This command is available everywhere.
 
 Usage:
 ...............
         help [<topic>]
 ...............
 
 === `quit` (`exit`, `bye`)
 
 Leave the program.
 
 //////////////////////
  vim:ts=4:sw=4:expandtab:
 //////////////////////
diff --git a/shell/modules/ui.py.in b/shell/modules/ui.py.in
index e2ae9c90b0..341fe505cb 100644
--- a/shell/modules/ui.py.in
+++ b/shell/modules/ui.py.in
@@ -1,1682 +1,1685 @@
 # Copyright (C) 2008 Dejan Muhamedagic <dmuhamedagic@suse.de>
 # 
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public
 # License as published by the Free Software Foundation; either
 # version 2.1 of the License, or (at your option) any later version.
 # 
 # This software is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 # General Public License for more details.
 # 
 # You should have received a copy of the GNU General Public
 # License along with this library; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 #
 
 import sys
 import re
 import os
 import shlex
 import time
 import bz2
 
 from help import HelpSystem, cmd_help
 from vars import Vars
 from levels import Levels
 from cibconfig import mkset_obj, CibFactory
 from cibstatus import CibStatus
 from template import LoadTemplate
 from cliformat import nvpairs2list
 from ra import *
 from msg import *
 from utils import *
 from xmlutil import *
 
 def cmd_end(cmd,dir = ".."):
     "Go up one level."
     levels.droplevel()
 def cmd_exit(cmd,rc = 0):
     "Exit the crm program"
     cmd_end(cmd)
     if options.interactive and not options.batch:
         print "bye"
         try:
             from readline import write_history_file
             write_history_file(vars.hist_file)
         except:
             pass
     for f in vars.tmpfiles:
         os.unlink(f)
     sys.exit(rc)
 
 class UserInterface(object):
     '''
     Stuff common to all user interface classes.
     '''
     global_cmd_aliases = {
         "quit": ("bye","exit"),
         "end": ("cd","up"),
     }
     def __init__(self):
         self.cmd_table = odict()
         self.cmd_table["help"] = (self.help,(0,1),0)
         self.cmd_table["quit"] = (self.exit,(0,0),0)
         self.cmd_table["end"] = (self.end,(0,1),0)
         self.cmd_aliases = self.global_cmd_aliases.copy()
     def end_game(self, no_questions_asked = False):
         pass
     def help(self,cmd,topic = ''):
         "usage: help [<topic>]"
         cmd_help(self.help_table,topic)
     def end(self,cmd,dir = ".."):
         "usage: end"
         self.end_game()
         cmd_end(cmd,dir)
     def exit(self,cmd):
         "usage: exit"
         self.end_game()
         cmd_exit(cmd)
 
 class CliOptions(UserInterface):
     '''
     Manage user preferences
     '''
     lvl_name = "options"
     def __init__(self):
         UserInterface.__init__(self)
         self.help_table = help_sys.load_level("options")
         self.cmd_table["skill-level"] = (self.set_skill_level,(1,1),0)
         self.cmd_table["editor"] = (self.set_editor,(1,1),0)
         self.cmd_table["pager"] = (self.set_pager,(1,1),0)
         self.cmd_table["user"] = (self.set_crm_user,(0,1),0)
         self.cmd_table["output"] = (self.set_output,(1,1),0)
         self.cmd_table["colorscheme"] = (self.set_colors,(1,1),0)
         self.cmd_table["check-frequency"] = (self.set_check_frequency,(1,1),0)
         self.cmd_table["check-mode"] = (self.set_check_mode,(1,1),0)
         self.cmd_table["sort-elements"] = (self.set_sort_elements,(1,1),0)
         self.cmd_table["save"] = (self.save_options,(0,0),0)
         self.cmd_table["show"] = (self.show_options,(0,0),0)
         setup_aliases(self)
     def set_skill_level(self,cmd,skill_level):
         """usage: skill-level <level>
         level: operator | administrator | expert"""
         return user_prefs.set_skill_level(skill_level)
     def set_editor(self,cmd,prog):
         "usage: editor <program>"
         return user_prefs.set_editor(prog)
     def set_pager(self,cmd,prog):
         "usage: pager <program>"
         return user_prefs.set_pager(prog)
     def set_crm_user(self,cmd,user = ''):
         "usage: user [<crm_user>]"
         return user_prefs.set_crm_user(user)
     def set_output(self,cmd,otypes):
         "usage: output <type>"
         return user_prefs.set_output(otypes)
     def set_colors(self,cmd,scheme):
         "usage: colorscheme <colors>"
         return user_prefs.set_colors(scheme)
     def set_check_frequency(self,cmd,freq):
         "usage: check-frequence <freq>"
         return user_prefs.set_check_freq(freq)
     def set_check_mode(self,cmd,mode):
         "usage: check-mode <mode>"
         return user_prefs.set_check_mode(mode)
     def set_sort_elements(self,cmd,opt):
         "usage: sort-elements {yes|no}"
         if not verify_boolean(opt):
             common_err("%s: bad boolean option"%opt)
             return True
         return user_prefs.set_sort_elems(opt)
     def show_options(self,cmd):
         "usage: show"
         return user_prefs.write_rc(sys.stdout)
     def save_options(self,cmd):
         "usage: save"
         return user_prefs.save_options(vars.rc_file)
     def end_game(self, no_questions_asked = False):
         if no_questions_asked and not options.interactive:
             self.save_options("save")
 
 class CibShadow(UserInterface):
     '''
     CIB shadow management class
     '''
     lvl_name = "cib"
     extcmd = ">/dev/null </dev/null crm_shadow"
     extcmd_stdout = "</dev/null crm_shadow"
     def __init__(self):
         UserInterface.__init__(self)
         self.help_table = help_sys.load_level("cib")
         self.cmd_table["new"] = (self.new,(1,3),1)
         self.cmd_table["delete"] = (self.delete,(1,1),1)
         self.cmd_table["reset"] = (self.reset,(1,1),1)
         self.cmd_table["commit"] = (self.commit,(1,1),1)
         self.cmd_table["use"] = (self.use,(0,2),1)
         self.cmd_table["diff"] = (self.diff,(0,0),1)
         self.cmd_table["list"] = (self.list,(0,0),1)
         self.cmd_table["import"] = (self.pe_import,(1,2),1)
         self.cmd_table["cibstatus"] = StatusMgmt
         self.chkcmd()
         setup_aliases(self)
     def chkcmd(self):
         try:
             ext_cmd("%s 2>&1" % self.extcmd)
         except os.error:
             no_prog_err(self.extcmd)
             return False
         return True
     def new(self,cmd,name,*args):
-        "usage: new <shadow_cib> [withstatus] [force]"
+        "usage: new <shadow_cib> [withstatus] [force] [empty]"
         if not is_filename_sane(name):
             return False
-        new_cmd = "%s -c '%s'" % (self.extcmd,name)
         for par in args:
-            if not par in ("force","--force","withstatus"):
+            if not par in ("force","--force","withstatus","empty"):
                 syntax_err((cmd,name,par), context = 'new')
                 return False
+        if "empty" in args:
+            new_cmd = "%s -e '%s'" % (self.extcmd,name)
+        else:
+            new_cmd = "%s -c '%s'" % (self.extcmd,name)
         if user_prefs.get_force() or "force" in args or "--force" in args:
             new_cmd = "%s --force" % new_cmd
         if ext_cmd(new_cmd) == 0:
             common_info("%s shadow CIB created"%name)
             self.use("use",name)
             if "withstatus" in args:
                 cib_status.load("shadow:%s" % name)
     def _find_pe(self,infile):
         'Find a pe input'
         for p in ("%s/%s", "%s/%s.bz2", "%s/pe-*-%s.bz2"):
             fl = glob.glob(p % (vars.pe_dir,infile))
             if fl:
                 break
         if not fl:
             common_err("no %s pe input file"%infile)
             return ''
         if len(fl) > 1:
             common_err("more than one %s pe input file: %s" % \
                 (infile,' '.join(fl)))
             return ''
         return fl[0]
     def pe_import(self,cmd,infile,name = None):
         "usage: import {<file>|<number>} [<shadow>]"
         if name and not is_filename_sane(name):
             return False
         # where's the input?
         if not os.access(infile,os.F_OK):
             if "/" in infile:
                 common_err("%s: no such file"%infile)
                 return False
             infile = self._find_pe(infile)
             if not infile:
                 return False
         if not name:
             name = os.path.basename(infile)
         # read input
         try:
             f = open(infile)
         except IOError,msg:
             common_err("open: %s"%msg)
             return
         s = ''.join(f)
         f.close()
         # decompresed and rename shadow if it ends with .bz2
         if infile.endswith(".bz2"):
             name = name.replace(".bz2","")
             s = bz2.decompress(s)
         # copy input to the shadow
         try:
             f = open(shadowfile(name), "w")
         except IOError,msg:
             common_err("open: %s"%msg)
             return
         f.write(s)
         f.close()
         # use the shadow and load the status from there
         return self.use("use",name,"withstatus")
     def delete(self,cmd,name):
         "usage: delete <shadow_cib>"
         if not is_filename_sane(name):
             return False
         if vars.cib_in_use == name:
             common_err("%s shadow CIB is in use"%name)
             return False
         if ext_cmd("%s -D '%s' --force" % (self.extcmd,name)) == 0:
             common_info("%s shadow CIB deleted"%name)
         else:
             common_err("failed to delete %s shadow CIB"%name)
             return False
     def reset(self,cmd,name):
         "usage: reset <shadow_cib>"
         if not is_filename_sane(name):
             return False
         if ext_cmd("%s -r '%s'" % (self.extcmd,name)) == 0:
             common_info("copied live CIB to %s"%name)
         else:
             common_err("failed to copy live CIB to %s"%name)
             return False
     def commit(self,cmd,name):
         "usage: commit <shadow_cib>"
         if not is_filename_sane(name):
             return False
         if ext_cmd("%s -C '%s' --force" % (self.extcmd,name)) == 0:
             common_info("commited '%s' shadow CIB to the cluster"%name)
         else:
             common_err("failed to commit the %s shadow CIB"%name)
             return False
     def diff(self,cmd):
         "usage: diff"
         s = get_stdout(add_sudo("%s -d" % self.extcmd_stdout))
         page_string(s)
     def list(self,cmd):
         "usage: list"
         if options.regression_tests:
             for t in listshadows():
                 print t
         else:
             multicolumn(listshadows())
     def _use(self,name,withstatus):
         # Choose a shadow cib for further changes. If the name
         # provided is empty, then choose the live (cluster) cib.
         # Don't allow ' in shadow names
         if not name or name == "live":
             os.unsetenv(vars.shadow_envvar)
             vars.cib_in_use = ""
             if withstatus:
                 cib_status.load("live")
         else:
             os.putenv(vars.shadow_envvar,name)
             vars.cib_in_use = name
             if withstatus:
                 cib_status.load("shadow:%s" % name)
     def use(self,cmd,name = '', withstatus = ''):
         "usage: use [<shadow_cib>] [withstatus]"
         # check the name argument
         if name and not is_filename_sane(name):
             return False
         if name and name != "live":
             if not os.access(shadowfile(name),os.F_OK):
                 common_err("%s: no such shadow CIB"%name)
                 return False
         if withstatus and withstatus != "withstatus":
             syntax_err((cmd,withstatus), context = 'use')
             return False
         # If invoked from configure
         # take special precautions
         try:
             prev_level = levels.previous().myname()
         except:
             prev_level = ''
         if prev_level != "cibconfig":
             self._use(name,withstatus)
             return True
         if not cib_factory.has_cib_changed():
             self._use(name,withstatus)
             # new CIB: refresh the CIB factory
             cib_factory.refresh()
             return True
         saved_cib = vars.cib_in_use
         self._use(name,'') # don't load the status yet
         if not cib_factory.is_current_cib_equal(silent = True):
             # user made changes and now wants to switch to a
             # different and unequal CIB; we refuse to cooperate
             common_err("the requested CIB is different from the current one")
             if user_prefs.get_force():
                 common_info("CIB overwrite forced")
             elif not ask("All changes will be dropped. Do you want to proceed?"):
                 self._use(saved_cib,'') # revert to the previous CIB
                 return False
         self._use(name,withstatus) # now load the status too
         return True
 
 def check_transition(inp,state,possible_l):
     if not state in possible_l:
         common_err("input (%s) in wrong state %s" % (inp,state))
         return False
     return True
 class Template(UserInterface):
     '''
     Configuration templates.
     '''
     lvl_name = "template"
     def __init__(self):
         UserInterface.__init__(self)
         self.help_table = help_sys.load_level("template")
         self.cmd_table["new"] = (self.new,(2,),1)
         self.cmd_table["load"] = (self.load,(0,1),1)
         self.cmd_table["edit"] = (self.edit,(0,1),1)
         self.cmd_table["delete"] = (self.delete,(1,2),1)
         self.cmd_table["show"] = (self.show,(0,1),0)
         self.cmd_table["apply"] = (self.apply,(0,2),1)
         self.cmd_table["list"] = (self.list,(0,1),0)
         setup_aliases(self)
         self.init_dir()
         self.curr_conf = ''
     def init_dir(self):
         '''Create the conf directory, link to templates'''
         if not os.path.isdir(vars.tmpl_conf_dir):
             try:
                 os.makedirs(vars.tmpl_conf_dir)
             except os.error,msg:
                 common_err("makedirs: %s"%msg)
                 return
     def get_depends(self,tmpl):
         '''return a list of required templates'''
         # Not used. May need it later.
         try:
             tf = open("%s/%s" % (vars.tmpl_dir, tmpl),"r")
         except IOError,msg:
             common_err("open: %s"%msg)
             return
         l = []
         for s in tf:
             a = s.split()
             if len(a) >= 2 and a[0] == '%depends_on':
                 l += a[1:]
         tf.close()
         return l
     def replace_params(self,s,user_data):
         change = False
         for i in range(len(s)):
             word = s[i]
             for p in user_data:
                 # is parameter in the word?
                 pos = word.find('%' + p) 
                 if pos < 0:
                     continue
                 endpos = pos + len('%' + p)
                 # and it isn't part of another word?
                 if re.match("[A-Za-z0-9]", word[endpos:endpos+1]):
                     continue
                 # if the value contains a space or
                 # it is a value of an attribute
                 # put quotes around it
                 if user_data[p].find(' ') >= 0 or word[pos-1:pos] == '=':
                     v = '"' + user_data[p] + '"'
                 else:
                     v = user_data[p]
                 word = word.replace('%' + p, v)
                 change = True # we did replace something
             if change:
                 s[i] = word
         if 'opt' in s:
             if not change:
                 s = []
             else:
                 s.remove('opt')
         return s
     def generate(self,l,user_data):
         '''replace parameters (user_data) and generate output
         '''
         l2 = []
         for piece in l:
             piece2 = []
             for s in piece:
                 s = self.replace_params(s,user_data)
                 if s:
                     piece2.append(' '.join(s))
             if piece2:
                 l2.append(' \\\n\t'.join(piece2))
         return '\n'.join(l2)
     def process(self,config = ''):
         '''Create a cli configuration from the current config'''
         try:
             f = open("%s/%s" % (vars.tmpl_conf_dir, config or self.curr_conf),'r')
         except IOError,msg:
             common_err("open: %s"%msg)
             return ''
         l = []
         piece = []
         user_data = {}
         # states
         START = 0; PFX = 1; DATA = 2; GENERATE = 3
         state = START
         err_buf.start_tmp_lineno()
         rc = True
         for inp in f:
             err_buf.incr_lineno()
             if inp.startswith('#'):
                 continue
             if type(inp) == type(u''):
                 inp = inp.encode('ascii')
             inp = inp.strip()
             try:
                 s = shlex.split(inp)
             except ValueError, msg:
                 common_err(msg)
                 continue
             while '\n' in s:
                 s.remove('\n')
             if not s:
                 if state == GENERATE and piece:
                     l.append(piece)
                     piece = []
             elif s[0] in ("%name","%depends_on","%suggests"):
                 continue
             elif s[0] == "%pfx":
                 if check_transition(inp,state,(START,DATA)) and len(s) == 2:
                     pfx = s[1]
                     state = PFX
             elif s[0] == "%required":
                 if check_transition(inp,state,(PFX,)):
                     state = DATA
                     data_reqd = True
             elif s[0] == "%optional":
                 if check_transition(inp,state,(PFX,DATA)):
                     state = DATA
                     data_reqd = False
             elif s[0] == "%%":
                 if state != DATA:
                     common_warn("user data in wrong state %s" % state)
                 if len(s) < 2:
                     common_warn("parameter name missing")
                 elif len(s) == 2:
                     if data_reqd:
                         common_err("required parameter %s not set" % s[1])
                         rc = False
                 elif len(s) == 3:
                     user_data["%s:%s" % (pfx,s[1])] = s[2]
                 else:
                     common_err("%s: syntax error" % inp)
             elif s[0] == "%generate":
                 if check_transition(inp,state,(DATA,)):
                     state = GENERATE
                     piece = []
             elif state == GENERATE:
                 if s:
                     piece.append(s)
             else:
                 common_err("<%s> unexpected" % inp)
         if piece:
             l.append(piece)
         err_buf.stop_tmp_lineno()
         f.close()
         if not rc:
             return ''
         return self.generate(l,user_data)
     def new(self,cmd,name,*args):
         "usage: new <config> <template> [<template> ...] [params name=value ...]"
         if not is_filename_sane(name):
             return False
         if os.path.isfile("%s/%s" % (vars.tmpl_conf_dir, name)):
             common_err("config %s exists; delete it first" % name)
             return False
         lt = LoadTemplate(name)
         rc = True
         mode = 0
         params = {}
         for s in args:
             if mode == 0 and s == "params":
                 params["id"] = name
                 mode = 1
             elif mode == 1:
                 a = s.split('=')
                 if len(a) != 2:
                     syntax_err(args, context = 'new')
                     rc = False
                 else:
                     params[a[0]] = a[1]
             elif not lt.load_template(s):
                 rc = False
         if rc:
             lt.post_process(params)
         if not rc or not lt.write_config(name):
             return False
         self.curr_conf = name
     def config_exists(self,name):
         if not is_filename_sane(name):
             return False
         if not os.path.isfile("%s/%s" % (vars.tmpl_conf_dir, name)):
             common_err("%s: no such config" % name)
             return False
         return True
     def delete(self,cmd,name,force = ''):
         "usage: delete <config> [force]"
         if force:
             if force != "force" and force != "--force":
                 syntax_err((cmd,force), context = 'delete')
                 return False
         if not self.config_exists(name):
             return False
         if name == self.curr_conf:
             if not force and not user_prefs.get_force() and \
                     not ask("Do you really want to remove config %s which is in use?" % self.curr_conf):
                 return False
             else:
                 self.curr_conf = ''
         os.remove("%s/%s" % (vars.tmpl_conf_dir, name))
     def load(self,cmd,name = ''):
         "usage: load [<config>]"
         if not name:
             self.curr_conf = ''
             return True
         if not self.config_exists(name):
             return False
         self.curr_conf = name
     def edit(self,cmd,name = ''):
         "usage: edit [<config>]"
         if not name and not self.curr_conf:
             common_err("please load a config first")
             return False
         if name:
             if not self.config_exists(name):
                 return False
             edit_file("%s/%s" % (vars.tmpl_conf_dir, name))
         else:
             edit_file("%s/%s" % (vars.tmpl_conf_dir, self.curr_conf))
     def show(self,cmd,name = ''):
         "usage: show [<config>]"
         if not name and not self.curr_conf:
             common_err("please load a config first")
             return False
         if name:
             if not self.config_exists(name):
                 return False
             print self.process(name)
         else:
             print self.process()
     def apply(self,cmd,*args):
         "usage: apply [<method>] [<config>]"
         method = "replace"
         name = ''
         if len(args) > 0:
             i = 0
             if args[0] in ("replace","update"):
                 method = args[0]
                 i += 1
             if len(args) > i:
                 name = args[i]
         if not name and not self.curr_conf:
             common_err("please load a config first")
             return False
         if name:
             if not self.config_exists(name):
                 return False
             s = self.process(name)
         else:
             s = self.process()
         if not s:
             return False
         tmp = str2tmp(s)
         if not tmp:
             return False
         set_obj = mkset_obj("NOOBJ")
         rc = set_obj.import_file(method,tmp)
         try: os.unlink(tmp)
         except: pass
         return rc
     def list(self,cmd,templates = ''):
         "usage: list [templates]"
         if templates == "templates":
             multicolumn(listtemplates())
         else:
             multicolumn(listconfigs())
 
 def manage_attr(cmd,attr_ext_commands,*args):
     if len(args) < 3:
         bad_usage(cmd,' '.join(args))
         return False
     attr_cmd = None
     try:
         attr_cmd = attr_ext_commands[args[1]]
     except KeyError:
         bad_usage(cmd,' '.join(args))
         return False
     if not attr_cmd:
         bad_usage(cmd,' '.join(args))
         return False
     if args[1] == 'set':
         if len(args) == 4:
             if not is_name_sane(args[0]) \
                     or not is_name_sane(args[2]) \
                     or not is_value_sane(args[3]):
                 return False
             return ext_cmd(attr_cmd%(args[0],args[2],args[3])) == 0
         else:
             bad_usage(cmd,' '.join(args))
             return False
     elif args[1] in ('delete','show'):
         if len(args) == 3:
             if not is_name_sane(args[0]) \
                     or not is_name_sane(args[2]):
                 return False
             return ext_cmd(attr_cmd%(args[0],args[2])) == 0
         else:
             bad_usage(cmd,' '.join(args))
             return False
     else:
         bad_usage(cmd,' '.join(args))
         return False
 
 def rm_meta_attribute(node,attr,l):
     '''
     Build a list of nvpair nodes which contain attribute
     (recursively in all children resources)
     '''
     if node.tagName == "group":
         # don't go into groups
         return
     for c in node.childNodes:
         if not is_element(c):
             continue
         if c.tagName == "meta_attributes":
             nvpair = get_attr_in_set(c,attr)
             if nvpair:
                 l.append(nvpair)
         elif is_child_rsc(c):
             rm_meta_attribute(c,attr,l)
 def clean_inferior_attributes(node,attr):
     'Remove attr from all resources below'
     l = []
     for c in node.childNodes:
         if is_child_rsc(c):
             rm_meta_attribute(c,attr,l)
     rmnodes(l)
 def set_deep_meta_attr(attr,value,rsc_id):
     '''
     If the referenced rsc is a primitive that belongs to a group,
     then set its attribute.
     Otherwise, go up to the topmost resource which contains this
     resource and set the attribute there and remove all the
     attributes which may be set in its children.
     '''
     target_node = rsc2node(rsc_id)
     if not target_node:
         common_error("resource %s does not exist" % rsc_id)
         return False
     if not (target_node.tagName == "primitive" and \
         target_node.parentNode.tagName == "group"):
         target_node = get_topmost_rsc(target_node)
     clean_inferior_attributes(target_node,attr)
     for n in get_set_nodes(target_node,"meta_attributes",1):
         set_attr(n,attr,value)
     return commit_rsc(target_node)
 
 def get_max_clone(id):
     v = get_meta_param(id,"clone-max")
     try:
         cnt = int(v)
     except:
         cnt = len(listnodes())
     return cnt
 def cleanup_resource(rsc,node = ''):
     if not is_name_sane(rsc) or not is_name_sane(node):
         return False
     if not node:
         rc = ext_cmd(RscMgmt.rsc_cleanup_all%(rsc)) == 0
     else:
         rc = ext_cmd(RscMgmt.rsc_cleanup%(rsc,node)) == 0
     return rc
 
 class RscMgmt(UserInterface):
     '''
     Resources management class
     '''
     lvl_name = "resource"
     rsc_status_all = "crm_resource -L"
     rsc_status = "crm_resource -W -r '%s'"
     rsc_showxml = "crm_resource -q -r '%s'"
     rsc_setrole = "crm_resource --meta -r '%s' -p target-role -v '%s'"
     rsc_migrate = "crm_resource -M -r '%s' %s"
     rsc_unmigrate = "crm_resource -U -r '%s'"
     rsc_cleanup = "crm_resource -C -r '%s' -H '%s'"
     rsc_cleanup_all = "crm_resource -C -r '%s'"
     rsc_param =  {
         'set': "crm_resource -r '%s' -p '%s' -v '%s'",
         'delete': "crm_resource -r '%s' -d '%s'",
         'show': "crm_resource -r '%s' -g '%s'",
     }
     rsc_meta =  {
         'set': "crm_resource --meta -r '%s' -p '%s' -v '%s'",
         'delete': "crm_resource --meta -r '%s' -d '%s'",
         'show': "crm_resource --meta -r '%s' -g '%s'",
     }
     rsc_failcount = {
         'set': "crm_attribute -t status -n 'fail-count-%s' -N '%s' -v '%s' -d 0",
         'delete': "crm_attribute -t status -n 'fail-count-%s' -N '%s' -D -d 0",
         'show': "crm_attribute -t status -n 'fail-count-%s' -N '%s' -G -d 0",
     }
     rsc_utilization =  {
         'set': "crm_resource -z -r '%s' -p '%s' -v '%s'",
         'delete': "crm_resource -z -r '%s' -d '%s'",
         'show': "crm_resource -z -r '%s' -g '%s'",
     }
     rsc_refresh = "crm_resource -R"
     rsc_refresh_node = "crm_resource -R -H '%s'"
     rsc_reprobe = "crm_resource -P"
     rsc_reprobe_node = "crm_resource -P -H '%s'"
     def __init__(self):
         UserInterface.__init__(self)
         self.help_table = help_sys.load_level("resource")
         self.cmd_table["status"] = (self.status,(0,1),0)
         self.cmd_table["start"] = (self.start,(1,1),0)
         self.cmd_table["stop"] = (self.stop,(1,1),0)
         self.cmd_table["restart"] = (self.restart,(1,1),0)
         self.cmd_table["promote"] = (self.promote,(1,1),0)
         self.cmd_table["demote"] = (self.demote,(1,1),0)
         self.cmd_table["manage"] = (self.manage,(1,1),0)
         self.cmd_table["unmanage"] = (self.unmanage,(1,1),0)
         self.cmd_table["migrate"] = (self.migrate,(1,4),0)
         self.cmd_table["unmigrate"] = (self.unmigrate,(1,1),0)
         self.cmd_table["param"] = (self.param,(3,4),1)
         self.cmd_table["meta"] = (self.meta,(3,4),1)
         self.cmd_table["utilization"] = (self.utilization,(3,4),1)
         self.cmd_table["failcount"] = (self.failcount,(3,4),0)
         self.cmd_table["cleanup"] = (self.cleanup,(1,2),1)
         self.cmd_table["refresh"] = (self.refresh,(0,1),0)
         self.cmd_table["reprobe"] = (self.reprobe,(0,1),0)
         self.cmd_aliases.update({
             "status": ("show","list",),
             "migrate": ("move",),
             "unmigrate": ("unmove",),
         })
         setup_aliases(self)
     def status(self,cmd,rsc = None):
         "usage: status [<rsc>]"
         if rsc:
             if not is_name_sane(rsc):
                 return False
             return ext_cmd(self.rsc_status % rsc) == 0
         else:
             return ext_cmd(self.rsc_status_all) == 0
     def start(self,cmd,rsc):
         "usage: start <rsc>"
         if not is_name_sane(rsc):
             return False
         return set_deep_meta_attr("target-role","Started",rsc)
     def restart(self,cmd,rsc):
         "usage: restart <rsc>"
         if not is_name_sane(rsc):
             return False
         if not self.stop("stop",rsc):
             return False
         # allow the transition to start
         time.sleep(1)
         return self.start("start",rsc)
     def stop(self,cmd,rsc):
         "usage: stop <rsc>"
         if not is_name_sane(rsc):
             return False
         return set_deep_meta_attr("target-role","Stopped",rsc)
     def promote(self,cmd,rsc):
         "usage: promote <rsc>"
         if not is_name_sane(rsc):
             return False
         if not is_rsc_ms(rsc):
             common_err("%s is not a master-slave resource" % rsc)
             return False
         return ext_cmd(self.rsc_setrole%(rsc,"Master")) == 0
     def demote(self,cmd,rsc):
         "usage: demote <rsc>"
         if not is_name_sane(rsc):
             return False
         if not is_rsc_ms(rsc):
             common_err("%s is not a master-slave resource" % rsc)
             return False
         return ext_cmd(self.rsc_setrole%(rsc,"Slave")) == 0
     def manage(self,cmd,rsc):
         "usage: manage <rsc>"
         if not is_name_sane(rsc):
             return False
         return set_deep_meta_attr("is-managed","true",rsc)
     def unmanage(self,cmd,rsc):
         "usage: unmanage <rsc>"
         if not is_name_sane(rsc):
             return False
         return set_deep_meta_attr("is-managed","false",rsc)
     def migrate(self,cmd,*args):
         """usage: migrate <rsc> [<node>] [<lifetime>] [force]"""
         rsc = args[0]
         if not is_name_sane(rsc):
             return False
         node = None
         lifetime = None
         force = False
         if len(args) == 2:
             if not args[1] in listnodes():
                 if args[1] == "force":
                     force = True
                 else:
                     lifetime = args[1]
             else:
                 node = args[1]
         elif len(args) == 3:
             if not args[1] in listnodes():
                 lifetime = args[1]
                 force = True
             else:
                 node = args[1]
                 if args[2] == "force":
                     force = True
                 else:
                     lifetime = args[2]
         elif len(args) == 4:
             if not is_name_sane(args[1]):
                 return False
             node = args[1]
             lifetime = args[2]
             if not args[3] == "force":
                 syntax_err((cmd,force))
                 return False
             force = True
         opts = ''
         if node:
             opts = "--node='%s'" % node
         if lifetime:
             opts = "%s --lifetime='%s'" % (opts,lifetime)
         if force or user_prefs.get_force():
             opts = "%s --force" % opts
         return ext_cmd(self.rsc_migrate % (rsc,opts)) == 0
     def unmigrate(self,cmd,rsc):
         "usage: unmigrate <rsc>"
         if not is_name_sane(rsc):
             return False
         return ext_cmd(self.rsc_unmigrate%rsc) == 0
     def cleanup(self,cmd,*args):
         "usage: cleanup <rsc> [<node>]"
         # Cleanup a resource on a node. Omit node to cleanup on
         # all live nodes.
         if len(args) == 2: # remove
             return cleanup_resource(args[0],args[1])
         else:
             return cleanup_resource(args[0])
     def failcount(self,cmd,*args):
         """usage:
         failcount <rsc> set <node> <value>
         failcount <rsc> delete <node>
         failcount <rsc> show <node>"""
         d = lambda: manage_attr(cmd,self.rsc_failcount,*args)
         return d()
     def param(self,cmd,*args):
         """usage:
         param <rsc> set <param> <value>
         param <rsc> delete <param>
         param <rsc> show <param>"""
         d = lambda: manage_attr(cmd,self.rsc_param,*args)
         return d()
     def meta(self,cmd,*args):
         """usage:
         meta <rsc> set <attr> <value>
         meta <rsc> delete <attr>
         meta <rsc> show <attr>"""
         d = lambda: manage_attr(cmd,self.rsc_meta,*args)
         return d()
     def utilization(self,cmd,*args):
         """usage:
         utilization <rsc> set <attr> <value>
         utilization <rsc> delete <attr>
         utilization <rsc> show <attr>"""
         d = lambda: manage_attr(cmd,self.rsc_utilization,*args)
         return d()
     def refresh(self,cmd,*args):
         'usage: refresh [<node>]'
         if len(args) == 1:
             if not is_name_sane(args[0]):
                 return False
             return ext_cmd(self.rsc_refresh_node%args[0]) == 0
         else:
             return ext_cmd(self.rsc_refresh) == 0
     def reprobe(self,cmd,*args):
         'usage: reprobe [<node>]'
         if len(args) == 1:
             if not is_name_sane(args[0]):
                 return False
             return ext_cmd(self.rsc_reprobe_node%args[0]) == 0
         else:
             return ext_cmd(self.rsc_reprobe) == 0
 
 def print_node(uname,id,node_type,other,inst_attr,offline):
     """
     Try to pretty print a node from the cib. Sth like:
     uname(id): node_type
         attr1: v1
         attr2: v2
     """
     s_offline = offline and "(offline)" or ""
     if uname == id:
         print "%s: %s%s" % (uname,node_type,s_offline)
     else:
         print "%s(%s): %s%s" % (uname,id,node_type,s_offline)
     for a in other:
         print "\t%s: %s" % (a,other[a])
     for a,v in inst_attr:
         print "\t%s: %s" % (a,v)
 
 class NodeMgmt(UserInterface):
     '''
     Nodes management class
     '''
     lvl_name = "node"
     node_standby = "crm_attribute -N '%s' -n standby -v '%s' %s"
     node_delete = "cibadmin -D -o nodes -X '<node uname=\"%s\"/>'"
     node_delete_status = "cibadmin -D -o status -X '<node_state uname=\"%s\"/>'"
     node_clear_state = "cibadmin -M -c -o status --xml-text '<node_state id=\"%s\" uname=\"%s\" ha=\"active\" in_ccm=\"false\" crmd=\"offline\" join=\"member\" expected=\"down\" crm-debug-origin=\"manual_clear\" shutdown=\"0\"/>'"
     hb_delnode = "@datadir@/heartbeat/hb_delnode '%s'"
     crm_node = "crm_node"
     node_fence = "crm_attribute -t status -U '%s' -n terminate -v true"
     dc = "crmadmin -D"
     node_attr = {
         'set': "crm_attribute -t nodes -U '%s' -n '%s' -v '%s'",
         'delete': "crm_attribute -D -t nodes -U '%s' -n '%s'",
         'show': "crm_attribute -G -t nodes -U '%s' -n '%s'",
     }
     node_status = {
         'set': "crm_attribute -t status -U '%s' -n '%s' -v '%s'",
         'delete': "crm_attribute -D -t status -U '%s' -n '%s'",
         'show': "crm_attribute -G -t status -U '%s' -n '%s'",
     }
     node_utilization = {
         'set': "crm_attribute -z -t nodes -U '%s' -n '%s' -v '%s'",
         'delete': "crm_attribute -z -D -t nodes -U '%s' -n '%s'",
         'show': "crm_attribute -z -G -t nodes -U '%s' -n '%s'",
     }
     def __init__(self):
         UserInterface.__init__(self)
         self.help_table = help_sys.load_level("node")
         self.cmd_table["status"] = (self.status,(0,1),0)
         self.cmd_table["show"] = (self.show,(0,1),0)
         self.cmd_table["standby"] = (self.standby,(0,2),0)
         self.cmd_table["online"] = (self.online,(0,1),0)
         self.cmd_table["fence"] = (self.fence,(1,1),0)
         self.cmd_table["delete"] = (self.delete,(1,1),0)
         self.cmd_table["clearstate"] = (self.clearstate,(1,1),0)
         self.cmd_table["attribute"] = (self.attribute,(3,4),0)
         self.cmd_table["utilization"] = (self.utilization,(3,4),0)
         self.cmd_table["status-attr"] = (self.status_attr,(3,4),0)
         self.cmd_aliases.update({
             "show": ("list",),
         })
         setup_aliases(self)
     def status(self,cmd,node = None):
         'usage: status [<node>]'
         return ext_cmd("%s -o nodes"%cib_dump) == 0
     def show(self,cmd,node = None):
         'usage: show [<node>]'
         doc = cibdump2doc()
         if not doc:
             return False
         nodes_node = get_conf_elem(doc, "nodes")
         status = get_conf_elem(doc, "status")
         if not nodes_node:
             return False
         for c in nodes_node.childNodes:
             if not is_element(c) or c.tagName != "node":
                 continue
             if node and c.getAttribute("uname") != node:
                 continue
             type = uname = id = ""
             inst_attr = []
             other = {}
             for attr in c.attributes.keys():
                 v = c.getAttribute(attr)
                 if attr == "type":
                     type = v
                 elif attr == "uname":
                     uname = v
                 elif attr == "id":
                     id = v
                 else:
                     other[attr] = v
             for c2 in c.childNodes:
                 if not is_element(c2):
                     continue
                 if c2.tagName == "instance_attributes":
                     inst_attr += nvpairs2list(c2)
             offline = False
             for c2 in status.getElementsByTagName("node_state"):
                 if uname != c2.getAttribute("uname"):
                     continue
                 offline = c2.getAttribute("crmd") == "offline"
             print_node(uname,id,type,other,inst_attr,offline)
     def standby(self,cmd,*args):
         'usage: standby [<node>] [<lifetime>]'
         node = None
         lifetime = None
         if not args:
             node = vars.this_node
         if len(args) == 1:
             if not args[0] in listnodes():
                 node = vars.this_node
                 lifetime = args[0]
             else:
                 node = args[0]
         elif len(args) == 2:
             node = args[0]
             lifetime = args[1]
         if lifetime not in (None,"reboot","forever"):
             common_err("bad lifetime: %s" % lifetime)
             return False
         if not is_name_sane(node):
             return False
         opts = ''
         if lifetime:
             opts = "--lifetime='%s'" % lifetime
         else:
             opts = "--lifetime='forever'"
         return ext_cmd(self.node_standby%(node,"on",opts)) == 0
     def online(self,cmd,node = None):
         'usage: online [<node>]'
         if not node:
             node = vars.this_node
         if not is_name_sane(node):
             return False
         return ext_cmd(self.node_standby%(node,"off","--lifetime='forever'")) == 0
     def fence(self,cmd,node):
         'usage: fence <node>'
         if not node:
             node = vars.this_node
         if not is_name_sane(node):
             return False
         if not user_prefs.get_force() and \
                 not ask("Do you really want to shoot %s?" % node):
             return False
         return ext_cmd(self.node_fence%(node)) == 0
     def clearstate(self,cmd,node):
         'usage: clearstate <node>'
         if not is_name_sane(node):
             return False
         if not user_prefs.get_force() and \
                 not ask("Do you really want to drop state for node %s?" % node):
             return False
         return ext_cmd(self.node_clear_state%(node,node)) == 0
     def delete(self,cmd,node):
         'usage: delete <node>'
         if not is_name_sane(node):
             return False
         if not node in listnodes():
             common_err("node %s not found in the CIB" % node)
             return False
         rc = True
         if cluster_stack() == "heartbeat":
             rc = ext_cmd(self.hb_delnode%node) == 0
         else:
             node_states = {}
             for s in stdout2list("%s -l" % self.crm_node):
                 a = s.split()
                 if len(a) != 3:
                     common_warn("%s bad format: %s" % (self.crm_node,s))
                     continue
                 # fmt: id uname status
                 # remove only those in state "lost"
                 if a[1] == node:
                     node_states[a[2]] = 1
                     if a[2] == "lost":
                         ec = ext_cmd("%s --force -R %s" % (self.crm_node,a[0]))
                         if ec != 0:
                             common_warn('"%s --force -R %s" failed, rc=%d' % (self.crm_node,a[0],ec))
             if not node_states:
                 common_info("node %s not found by %s" % (node,self.crm_node))
             elif "member" in node_states:
                 common_info("node %s appears to be still active" % node)
                 common_info("check output of %s -l" % self.crm_node)
                 rc = False
             elif not "lost" in node_states:
                 common_err("node %s's state not recognized: %s" % (node,node_states.keys()))
                 common_info("check output of %s -l" % self.crm_node)
                 rc = False
         if rc:
             if ext_cmd(self.node_delete%node) != 0 or \
                     ext_cmd(self.node_delete_status%node) != 0:
                 common_err("failed to remove %s from the CIB" % node)
                 rc = False
         if rc:
             common_info("node %s deleted" % node)
         else:
             common_err('failed to delete node %s' % node)
         return rc
     def attribute(self,cmd,*args):
         """usage:
         attribute <node> set <rsc> <value>
         attribute <node> delete <rsc>
         attribute <node> show <rsc>"""
         d = lambda: manage_attr(cmd,self.node_attr,*args)
         return d()
     def utilization(self,cmd,*args):
         """usage:
         utilization <node> set <rsc> <value>
         utilization <node> delete <rsc>
         utilization <node> show <rsc>"""
         d = lambda: manage_attr(cmd,self.node_utilization,*args)
         return d()
     def status_attr(self,cmd,*args):
         """usage:
         status-attr <node> set <rsc> <value>
         status-attr <node> delete <rsc>
         status-attr <node> show <rsc>"""
         d = lambda: manage_attr(cmd,self.node_status,*args)
         return d()
 
 class RA(UserInterface):
     '''
     CIB shadow management class
     '''
     lvl_name = "ra"
     provider_classes = ["ocf"]
     def __init__(self):
         UserInterface.__init__(self)
         self.help_table = help_sys.load_level("ra")
         self.cmd_table["classes"] = (self.classes,(0,0),0)
         self.cmd_table["list"] = (self.list,(1,2),1)
         self.cmd_table["providers"] = (self.providers,(1,2),1)
         self.cmd_table["meta"] = (self.meta,(1,3),1)
         self.cmd_aliases.update({
             "meta": ("info",),
         })
         setup_aliases(self)
     def classes(self,cmd):
         "usage: classes"
         for c in ra_classes():
             if c in self.provider_classes:
                 print "%s / %s" % (c,' '.join(ra_providers_all(c)))
             else:
                 print "%s" % c
     def providers(self,cmd,ra_type,ra_class = "ocf"):
         "usage: providers <ra> [<class>]"
         print ' '.join(ra_providers(ra_type,ra_class))
     def list(self,cmd,c,p = None):
         "usage: list <class> [<provider>]"
         if not c in ra_classes():
             common_err("class %s does not exist" % c)
             return False
         if p and not p in ra_providers_all(c):
             common_err("there is no provider %s for class %s" % (p,c))
             return False
         if options.regression_tests:
             for t in ra_types(c,p):
                 print t
         else:
             multicolumn(ra_types(c,p))
     def meta(self,cmd,*args):
         "usage: meta [<class>:[<provider>:]]<type>"
         if len(args) > 1: # obsolete syntax
             ra_type = args[0]
             ra_class = args[1]
             if len(args) < 3:
                 ra_provider = "heartbeat"
             else:
                 ra_provider = args[2]
         else:
             if args[0] in vars.meta_progs:
                 ra_class = args[0]
                 ra_provider = ra_type = None
             else:
                 ra_class,ra_provider,ra_type = disambiguate_ra_type(args[0])
         ra = RAInfo(ra_class,ra_type,ra_provider)
         if not ra.mk_ra_node():
             return False
         try:
             page_string(ra.meta_pretty())
         except:
             return False
 
 def ptestlike(simfun,def_verb,cmd,*args):
     verbosity = def_verb  # default verbosity
     nograph = False
     scores = False
     utilization = False
     actions = False
     for p in args:
         if p == "nograph":
             nograph = True
         elif p == "scores":
             scores = True
         elif p == "utilization":
             utilization = True
         elif p == "actions":
             actions = True
         elif re.match("^vv*$", p):
             verbosity = p
         else:
             bad_usage(cmd,' '.join(args))
             return False
     return simfun(nograph, scores, utilization, actions, verbosity)
 
 class StatusMgmt(UserInterface):
     '''
     The CIB status section management user interface class
     '''
     lvl_name = "cibstatus"
     def __init__(self):
         UserInterface.__init__(self)
         self.help_table = help_sys.load_level("cibstatus")
         self.cmd_table["show"] = (self.show,(0,1),1)
         self.cmd_table["save"] = (self.save,(0,1),2)
         self.cmd_table["load"] = (self.load,(1,1),2)
         self.cmd_table["origin"] = (self.origin,(0,0),1)
         self.cmd_table["node"] = (self.edit_node,(2,2),2)
         self.cmd_table["op"] = (self.edit_op,(3,5),2)
         self.cmd_table["run"] = (self.run,(0,3),1)
         self.cmd_table["simulate"] = (self.simulate,(0,3),1)
         self.cmd_table["quorum"] = (self.quorum,(1,1),1)
         setup_aliases(self)
     def myname(self):
         '''Just return some id.'''
         return "cibstatus"
     def load(self,cmd,org):
         "usage: load {<file>|shadow:<cib>|live}"
         return cib_status.load(org)
     def save(self,cmd,dest = None):
         "usage: save [<file>|shadow:<cib>]"
         return cib_status.save(dest)
     def origin(self,cmd):
         "usage: origin"
         state = cib_status.modified and " (modified)" or ""
         print "%s%s" % (cib_status.origin,state)
     def show(self,cmd,changed = ""):
         "usage: show [changed]"
         if changed:
             if changed != "changed":
                 syntax_err((cmd,changed))
                 return False
             else:
                 return cib_status.list_changes()
         return cib_status.show()
     def quorum(self,cmd,opt):
         "usage: quorum <bool>"
         if not verify_boolean(opt):
             common_err("%s: bad boolean option"%opt)
             return False
         return cib_status.set_quorum(is_boolean_true(opt))
     def edit_node(self,cmd,node,state):
         "usage: node <node> {online|offline|unclean}"
         return cib_status.edit_node(node,state)
     def edit_op(self,cmd,op,rsc,rc,op_status = None,node = ''):
         "usage: op <operation> <resource> <exit_code> [<op_status>] [<node>]"
         if rc in vars.lrm_exit_codes:
             num_rc = vars.lrm_exit_codes[rc]
         else:
             num_rc = rc
         if not num_rc.isdigit():
             common_err("%s exit code invalid" % num_rc)
             return False
         num_op_status = op_status
         if op_status:
             if op_status in vars.lrm_status_codes:
                 num_op_status = vars.lrm_status_codes[op_status]
             if not num_op_status.isdigit():
                 common_err("%s operation status invalid" % num_op_status)
                 return False
         return cib_status.edit_op(op,rsc,num_rc,num_op_status,node)
     def run(self,cmd,*args):
         "usage: run [nograph] [v...] [scores] [utilization]"
         return ptestlike(cib_status.run,'',cmd,*args)
     def simulate(self,cmd,*args):
         "usage: simulate [nograph] [v...] [scores] [utilization]"
         return ptestlike(cib_status.simulate,'',cmd,*args)
 
 class CibConfig(UserInterface):
     '''
     The configuration class
     '''
     lvl_name = "configure"
     def __init__(self):
         UserInterface.__init__(self)
         self.help_table = help_sys.load_level("configure")
         self.cmd_table["erase"] = (self.erase,(0,1),1)
         self.cmd_table["verify"] = (self.verify,(0,0),1)
         self.cmd_table["refresh"] = (self.refresh,(0,0),1)
         self.cmd_table["ptest"] = (self.ptest,(0,3),1)
         self.cmd_table["commit"] = (self.commit,(0,1),1)
         self.cmd_table["upgrade"] = (self.upgrade,(0,1),1)
         self.cmd_table["show"] = (self.show,(0,),1)
         self.cmd_table["edit"] = (self.edit,(0,),1)
         self.cmd_table["filter"] = (self.filter,(1,),1)
         self.cmd_table["delete"] = (self.delete,(1,),1)
         self.cmd_table["default-timeouts"] = (self.default_timeouts,(1,),1)
         self.cmd_table["rename"] = (self.rename,(2,2),1)
         self.cmd_table["save"] = (self.save,(1,2),1)
         self.cmd_table["load"] = (self.load,(2,3),1)
         self.cmd_table["node"] = (self.conf_node,(1,),1)
         self.cmd_table["primitive"] = (self.conf_primitive,(2,),1)
         self.cmd_table["group"] = (self.conf_group,(2,),1)
         self.cmd_table["clone"] = (self.conf_clone,(2,),1)
         self.cmd_table["ms"] = (self.conf_ms,(2,),1)
         self.cmd_table["location"] = (self.conf_location,(2,),1)
         self.cmd_table["colocation"] = (self.conf_colocation,(2,),1)
         self.cmd_table["order"] = (self.conf_order,(2,),1)
         self.cmd_table["property"] = (self.conf_property,(1,),1)
         self.cmd_table["rsc_defaults"] = (self.conf_rsc_defaults,(1,),1)
         self.cmd_table["op_defaults"] = (self.conf_op_defaults,(1,),1)
         self.cmd_table["xml"] = (self.conf_xml,(1,),1)
         self.cmd_table["monitor"] = (self.conf_monitor,(2,2),1)
         self.cmd_table["ra"] = RA
         self.cmd_table["cib"] = CibShadow
         self.cmd_table["cibstatus"] = StatusMgmt
         self.cmd_table["template"] = Template
         self.cmd_table["_test"] = (self.check_structure,(0,0),1)
         self.cmd_table["_regtest"] = (self.regression_testing,(1,1),1)
         self.cmd_table["_objects"] = (self.showobjects,(0,0),1)
         self.cmd_aliases.update({
             "colocation": ("collocation",),
             "ms": ("master",),
         })
         setup_aliases(self)
         cib_factory.initialize()
     def myname(self):
         '''Just return some id.'''
         return "cibconfig"
     def check_structure(self,cmd):
         return cib_factory.check_structure()
     def regression_testing(self,cmd,param):
         return cib_factory.regression_testing(param)
     def showobjects(self,cmd):
         cib_factory.showobjects()
     def show(self,cmd,*args):
         "usage: show [xml] [<id>...]"
         if not cib_factory.is_cib_sane():
             return False
         err_buf.buffer() # keep error messages
         set_obj = mkset_obj(*args)
         err_buf.release() # show them, but get an ack from the user
         return set_obj.show()
     def filter(self,cmd,filter,*args):
         "usage: filter <prog> [xml] [<id>...]"
         if not cib_factory.is_cib_sane():
             return False
         err_buf.buffer() # keep error messages
         set_obj = mkset_obj(*args)
         err_buf.release() # show them, but get an ack from the user
         return set_obj.filter(filter)
     def edit(self,cmd,*args):
         "usage: edit [xml] [<id>...]"
         if not cib_factory.is_cib_sane():
             return False
         err_buf.buffer() # keep error messages
         set_obj = mkset_obj(*args)
         err_buf.release() # show them, but get an ack from the user
         return set_obj.edit()
     def _verify(self, set_obj_semantic, set_obj_verify = None):
         if not set_obj_verify:
             set_obj_verify = set_obj_semantic
         rc1 = set_obj_verify.verify()
         if user_prefs.check_frequency != "never":
             rc2 = set_obj_semantic.semantic_check()
         else:
             rc2 = 0
         return rc1 and rc2 <= 1
     def verify(self,cmd):
         "usage: verify"
         if not cib_factory.is_cib_sane():
             return False
         return self._verify(mkset_obj("xml"))
     def save(self,cmd,*args):
         "usage: save [xml] <filename>"
         if not cib_factory.is_cib_sane():
             return False
         if args[0] == "xml":
             f = args[1]
             set_obj = mkset_obj("xml")
         else:
             f = args[0]
             set_obj = mkset_obj()
         return set_obj.save_to_file(f)
     def load(self,cmd,*args):
         "usage: load [xml] {replace|update} {<url>|<path>}"
         if not cib_factory.is_cib_sane():
             return False
         if args[0] == "xml":
             if len(args) != 3:
                 syntax_err(args, context = 'load')
                 return False
             url = args[2]
             method = args[1]
             set_obj = mkset_obj("xml","NOOBJ")
         else:
             if len(args) != 2:
                 syntax_err(args, context = 'load')
                 return False
             url = args[1]
             method = args[0]
             set_obj = mkset_obj("NOOBJ")
         return set_obj.import_file(method,url)
     def delete(self,cmd,*args):
         "usage: delete <id> [<id>...]"
         if not cib_factory.is_cib_sane():
             return False
         return cib_factory.delete(*args)
     def default_timeouts(self,cmd,*args):
         "usage: default-timeouts <id> [<id>...]"
         if not cib_factory.is_cib_sane():
             return False
         return cib_factory.default_timeouts(*args)
     def rename(self,cmd,old_id,new_id):
         "usage: rename <old_id> <new_id>"
         if not cib_factory.is_cib_sane():
             return False
         return cib_factory.rename(old_id,new_id)
     def erase(self,cmd,nodes = None):
         "usage: erase [nodes]"
         if not cib_factory.is_cib_sane():
             return False
         if nodes:
             if nodes == "nodes":
                 return cib_factory.erase_nodes()
             else:
                 syntax_err((cmd,nodes), context = 'erase')
         else:
             return cib_factory.erase()
     def refresh(self,cmd):
         "usage: refresh"
         if not cib_factory.is_cib_sane():
             return False
         if options.interactive and cib_factory.has_cib_changed():
             if not ask("All changes will be dropped. Do you want to proceed?"):
                 return
         cib_factory.refresh()
     def ptest(self,cmd,*args):
         "usage: ptest [nograph] [v...] [scores] [utilization] [actions]"
         if not cib_factory.is_cib_sane():
             return False
         set_obj = mkset_obj("xml")
         return ptestlike(set_obj.ptest,'vv',cmd,*args)
     def commit(self,cmd,force = None):
         "usage: commit [force]"
         if force and force != "force":
             syntax_err((cmd,force))
             return False
         if not cib_factory.is_cib_sane():
             return False
         if not cib_factory.has_cib_changed():
             common_info("apparently there is nothing to commit")
             common_info("try changing something first")
             return
         rc1 = cib_factory.is_current_cib_equal()
         rc2 = cib_factory.is_cib_empty() or \
             self._verify(mkset_obj("xml","changed"),mkset_obj("xml"))
         if rc1 and rc2:
             return cib_factory.commit()
         if force or user_prefs.get_force():
             common_info("commit forced")
             return cib_factory.commit()
         if ask("Do you still want to commit?"):
             return cib_factory.commit()
         return False
     def upgrade(self,cmd,force = None):
         "usage: upgrade [force]"
         if not cib_factory.is_cib_sane():
             return False
         if force and force != "force":
             syntax_err((cmd,force))
             return False
         if user_prefs.get_force() or force:
             return cib_factory.upgrade_cib_06to10(True)
         else:
             return cib_factory.upgrade_cib_06to10()
     def __conf_object(self,cmd,*args):
         "The configure object command."
         if not cib_factory.is_cib_sane():
             return False
         f = lambda: cib_factory.create_object(cmd,*args)
         return f()
     def conf_node(self,cmd,*args):
         """usage: node <uname>[:<type>]
            [attributes <param>=<value> [<param>=<value>...]]
            [utilization <param>=<value> [<param>=<value>...]]"""
         return self.__conf_object(cmd,*args)
     def conf_primitive(self,cmd,*args):
         """usage: primitive <rsc> [<class>:[<provider>:]]<type>
         [params <param>=<value> [<param>=<value>...]]
         [meta <attribute>=<value> [<attribute>=<value>...]]
         [utilization <attribute>=<value> [<attribute>=<value>...]]
         [operations id_spec
             [op op_type [<attribute>=<value>...] ...]]"""
         return self.__conf_object(cmd,*args)
     def conf_group(self,cmd,*args):
         """usage: group <name> <rsc> [<rsc>...]
         [params <param>=<value> [<param>=<value>...]]
         [meta <attribute>=<value> [<attribute>=<value>...]]"""
         return self.__conf_object(cmd,*args)
     def conf_clone(self,cmd,*args):
         """usage: clone <name> <rsc>
         [params <param>=<value> [<param>=<value>...]]
         [meta <attribute>=<value> [<attribute>=<value>...]]"""
         return self.__conf_object(cmd,*args)
     def conf_ms(self,cmd,*args):
         """usage: ms <name> <rsc>
         [params <param>=<value> [<param>=<value>...]]
         [meta <attribute>=<value> [<attribute>=<value>...]]"""
         return self.__conf_object(cmd,*args)
     def conf_location(self,cmd,*args):
         """usage: location <id> <rsc> {node_pref|rules}
 
         node_pref :: <score>: <node>
 
         rules ::
           rule [id_spec] [$role=<role>] <score>: <expression>
           [rule [id_spec] [$role=<role>] <score>: <expression> ...]
 
         id_spec :: $id=<id> | $id-ref=<id>
         score :: <number> | <attribute> | [-]inf
         expression :: <simple_exp> [bool_op <simple_exp> ...]
         bool_op :: or | and
         simple_exp :: <attribute> [type:]<binary_op> <value>
                       | <unary_op> <attribute>
                       | date <date_expr>
         type :: string | version | number
         binary_op :: lt | gt | lte | gte | eq | ne
         unary_op :: defined | not_defined"""
         return self.__conf_object(cmd,*args)
     def conf_colocation(self,cmd,*args):
         """usage: colocation <id> <score>: <rsc>[:<role>] <rsc>[:<role>]
         """
         return self.__conf_object(cmd,*args)
     def conf_order(self,cmd,*args):
         """usage: order <id> score-type: <first-rsc>[:<action>] <then-rsc>[:<action>]
         [symmetrical=<bool>]"""
         return self.__conf_object(cmd,*args)
     def conf_property(self,cmd,*args):
         "usage: property [$id=<set_id>] <option>=<value>"
         return self.__conf_object(cmd,*args)
     def conf_rsc_defaults(self,cmd,*args):
         "usage: rsc_defaults [$id=<set_id>] <option>=<value>"
         return self.__conf_object(cmd,*args)
     def conf_op_defaults(self,cmd,*args):
         "usage: op_defaults [$id=<set_id>] <option>=<value>"
         return self.__conf_object(cmd,*args)
     def conf_xml(self,cmd,*args):
         "usage: xml <xml>"
         return self.__conf_object(cmd,*args)
     def conf_monitor(self,cmd,*args):
         "usage: monitor <rsc>[:<role>] <interval>[:<timeout>]"
         return self.__conf_object(cmd,*args)
     def end_game(self, no_questions_asked = False):
         if cib_factory.has_cib_changed():
             if no_questions_asked or not options.interactive or \
                 ask("There are changes pending. Do you want to commit them?"):
                 self.commit("commit")
         cib_factory.reset()
 
 class TopLevel(UserInterface):
     '''
     The top level.
     '''
     lvl_name = "."
     crm_mon = "crm_mon -1"
     status_opts = {
         "bynode": "-n",
         "inactive": "-r",
         "ops": "-o",
         "timing": "-t",
         "failcounts": "-f",
     }
     help_table = odict()
     help_table["."] = ("","""This is the CRM command line interface program.""")
     help_table["cib"] = ("manage shadow CIBs", """
 A shadow CIB is a regular cluster configuration which is kept in
 a file. The CRM and the CRM tools may manage a shadow CIB in the
 same way as the live CIB (i.e. the current cluster configuration).
 A shadow CIB may be applied to the cluster in one step.
 """)
     help_table["resource"] = ("resources management", """
 Everything related to resources management is available at this
 level. Most commands are implemented using the crm_resource(8)
 program.
 """)
     help_table["node"] = ("nodes management", """
 A few node related tasks such as node standby are implemented
 here.
 """)
     help_table["options"] = ("user preferences", """
 Several user preferences are available. Note that it is possible
 to save the preferences to a startup file.
 """)
     help_table["configure"] = ("CRM cluster configuration", """
 The configuration level.
 
 Note that you can change the working CIB at the cib level. It is
 advisable to configure shadow CIBs and then commit them to the
 cluster.
 """)
     help_table["ra"] = ("resource agents information center", """
 This level contains commands which show various information about
 the installed resource agents. It is available both at the top
 level and at the `configure` level.
 """)
     help_table["status"] = ("show cluster status", """
 Show cluster status. The status is displayed by crm_mon. Supply
 additional arguments for more information or different format.
 See crm_mon(8) for more details.
 
 Usage:
 ...............
         status [<option> ...]
 
         option :: bynode | inactive | ops | timing | failcounts
 ...............
 """)
     help_table["quit"] = ("exit the program", "")
     help_table["help"] = ("show help", "")
     help_table["end"] = ("go back one level", "")
     def __init__(self):
         UserInterface.__init__(self)
         self.cmd_table['cib'] = CibShadow
         self.cmd_table['resource'] = RscMgmt
         self.cmd_table['configure'] = CibConfig
         self.cmd_table['node'] = NodeMgmt
         self.cmd_table['options'] = CliOptions
         self.cmd_table['status'] = (self.status,(0,5),0)
         self.cmd_table['ra'] = RA
         setup_aliases(self)
     def status(self,cmd,*args):
         """usage: status [<option> ...]
             option :: bynode | inactive | ops | timing | failcounts
         """
         status_cmd = self.crm_mon
         for par in args:
             if par in self.status_opts:
                 status_cmd = "%s %s" % (status_cmd, self.status_opts[par])
             else:
                 syntax_err((cmd,par), context = 'status')
                 return False
         return ext_cmd(status_cmd) == 0
 
 help_sys = HelpSystem()
 user_prefs = UserPrefs.getInstance()
 options = Options.getInstance()
 err_buf = ErrorBuffer.getInstance()
 vars = Vars.getInstance()
 levels = Levels.getInstance(TopLevel)
 cib_status = CibStatus.getInstance()
 cib_factory = CibFactory.getInstance()
 
 # vim:ts=4:sw=4:et: