diff --git a/doc/crm_cli.txt b/doc/crm_cli.txt
index dbe26d25e6..aa8f83e986 100644
--- a/doc/crm_cli.txt
+++ b/doc/crm_cli.txt
@@ -1,2004 +1,2008 @@
 CRM CLI (command line interface) tool
 ======================================
 Dejan_Muhamedagic,_Yan_Gao dejan@suse.de,ygao@novell.com
 v0.94, Februar 18, 2010:
 
 The CRM (a.k.a Pacemaker) is a Cluster Resource Manager which
 implements the cluster configuration provided by the user in CIB
 (Cluster Information Base). The CIB is a set of instructions
 coded in XML. Editing the CIB is a challenge, not only due to its
 complexity and a wide variety of options, but also because XML is
 more computer than user friendly.
 
 .Note
 **************************
 I do understand that there are people capable of
 dealing with XML without an intermediary.
 **************************
 
 There are currently three options to manage the CIB, listed here
 in a decreasing order of user-friendliness:
 
 - the GUI (`hb_gui`)
 - a set of command line tools
 - `cibadmin(8)`
 
 The GUI is very popular and it has recently seen a lot of good
 development. For some it is going to be (or remain) the first
 choice in cluster management.
 
 The command line tools, lead by `crm_resource(8)`, are capable of
 performing almost any kind of CIB transformation. The usage is,
 however, plagued by the notorious weakness common to all UNIX
 tools: a multitude of options, necessary for operation and yet
 very hard to remember. Usage is also inconsistent at times.
 
 The `cibadmin` is the ultimate CIB management tool: it applies
 chunks of XML written by the user or generated by another tool to
 the CIB. Very difficult to use without extensive training. Or
 should I say drill. May be unnerving as well, in particular due
 to sometimes cryptic error messages.
 
 == Design goals
 
 The CLI provides a consistent and unified interface to
 CIB/cluster management. It uses the command line tools where
 possible and may resort to XML and `cibadmin` when there is no
 other option. That is the easiest way to ensure compatibility
 between different management tools.
 
 It may be used either as an interactive shell or for single
 commands directly on the shell's command line. It is also
 possible to feed it a set of commands from standard input or a
 file, thus turning it into a scripting tool. Templates with ready
 made configurations may help people to learn about the cluster
 configuration or facilitate testing procedures.
 
 The CLI may also be used for the CIB description and generation.
 A file containing a set of CLI instructions may be applied to the
 CLI tool to generate a complete CIB.
 
 The new shadow CIB feature may also be put to use. The user may
 work on one of the shadow CIBs and then apply (or commit) it in a
 single step to the cluster.
  
 It should also allow deployment of raw XML which may come either
 from files or network.
 
 Several modes of operation are available to restrict the set of
 features depending on the user's proficiency.
 
 The CLI is line oriented: every command must start and finish
 on the same line. It is possible to use a continuation character
 (`\`) to write one command in two or more lines.
 
 The CLI has to be run on one of the cluster nodes.
 
 .Note
 **************************
 Even though all sensible configurations (and most of those that
 are not) are going to be supported by the CLI, I suspect that it
 may still happen that certain XML constructs may confuse the
 tool. When that happens, please file a bug report.
 
 The CLI will not try to update the objects it does not
 understand. Of course, it is always possible to edit such objects
 in the XML format.
 **************************
 
 == Introduction to the user interface
 
 Arguably the most important aspect of such a program is the user
 interface. We begin with an informal introduction so that the
 reader may get acquainted with it and get a general feeling of
 the tool. It is probably best just to give some examples:
 
 1. Command line (one-shot) use:
 
         # crm resource stop www_app
 
 2. Interactive use:
 
         # crm
         crm(live)# resource
         crm(live) resource# unmanage tetris_1
         crm(live) resource# up
         crm(live)# node standby node4
 
 3. Cluster configuration:
 
     # crm<<EOF
     configure
       erase
       #
       # resources
       #
       primitive disk0 iscsi \
         params portal=192.168.2.108:3260 target=iqn.2008-07.com.suse:disk0
       primitive fs0 Filesystem \
         params device=/dev/disk/by-label/disk0 directory=/disk0 fstype=ext3
       primitive internal_ip IPaddr params ip=192.168.1.101
       primitive apache apache \
         params configfile=/disk0/etc/apache2/site0.conf
       primitive apcfence stonith:apcsmart \
         params ttydev=/dev/ttyS0 hostlist="node1 node2" \
         op start timeout=60s
       primitive pingd pingd \
         params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
       #
       # monitor apache and the UPS
       #
       monitor apache 60s:30s
       monitor apcfence 120m:60s
       #
       # cluster layout
       #
       group internal_www \
         disk0 fs0 internal_ip apache
       clone fence apcfence \
         meta globally-unique=false clone-max=2 clone-node-max=1
       clone conn pingd \
         meta globally-unique=false clone-max=2 clone-node-max=1
       location node_pref internal_www \
         rule 50: #uname eq node1 \
         rule pingd: defined pingd
       #
       # cluster properties
       #
       property stonith-enabled=true
       commit
     EOF
 
 If you've ever done a CRM style configuration, you should be able
 to understand the above examples without much difficulties.  The
 CLI should provide a means to manage the cluster efficiently or
 put together a configuration in a concise manner.
 
 The `(live)` string in the prompt signifies that the current CIB
 in use is the cluster live configuration. It is also possible to
 work with the so-called shadow CIBs, i.e. configurations which
 are stored in files and aren't active, but may be applied at any
 time to the cluster.
 
 Since the CIB is hierarchical such is the interface too. There
 are several levels and entering each of them enables the user to
 use a certain set of commands.
 
 === Shadow CIB usage
 
 Shadow CIB is a new feature. Shadow CIBs may be manipulated in
 the same way like the _live_ CIB, but these changes have no
 effect on the cluster resources. The administrator may choose to
 apply any of them to the cluster, thus replacing the running
 configuration with the one which is in the shadow. The `crm`
 prompt always contains the name of the configuration which is
 currently in use. Note that, for obvious reasons, only commands
 at the `configure` level make sense while working with a shadow
 CIB.
 
 No changes take place before the `configure commit` command.
 Sometimes though, the administrator may start working with the
 running configuration, but change mind and instead of committing
 the changes to the cluster save them to a shadow CIB. This
 short `configure` session excerpt shows how:
 ...............
     crm(live)configure# cib new test-2
     INFO: test-2 shadow CIB created
     crm(test-2)configure# commit
 ...............
 
 === Templates
 
 Templates are ready made configurations created by cluster
 experts. They are designed in such a way, so that users may
 generate valid cluster configurations with minimum effort.
 If you are new to Pacemaker/CRM, templates may be the best way to
 start.
 
 We will show here how to create a simple yet functional Apache
 configuration:
 ...............
     # crm configure
     crm(live)configure# template
     crm(live)configure template# list templates
     apache       filesystem   virtual-ip   
     crm(live)configure template# new web <TAB><TAB>
     apache       filesystem   virtual-ip   
     crm(live)configure template# new web apache
     INFO: pulling in template apache
     INFO: pulling in template virtual-ip
     crm(live)configure template# list
     web2-d       web2         vip2         web3         vip          web
 ...............
 
 We enter the `template` level from `configure`. Use the `list`
 command to show templates available on the system.
 The `new` command creates a configuration from the `apache`
 template. You can use tab completion to pick templates. Note that
 the apache template depends on a virtual IP address which is
 automatically pulled along. The `list` command shows the just
 created `web` configuration, among other configurations (I hope
 that you, unlike me, will use more sensible and descriptive names).
 
 The `show` command, which displays the resulting configuration,
 may be used to get an idea about the minimum required changes
 which have to be done. All `ERROR` messages show the line numbers
 in which the respective parameters are to be defined:
 ...............
     crm(live)configure template# show
     ERROR: 23: required parameter ip not set
     ERROR: 61: required parameter id not set
     ERROR: 65: required parameter configfile not set
     crm(live)configure template# edit
 ...............
 
 The `edit` command invokes the preferred text editor with the
 `web` configuration. At the top of the file, the user is advised
 how to make changes. A good template should require from the user
 to specify only parameters. For example, the `web` configuration
 we created above has the following required and optional
 parameters (all parameter lines start with `%%`):
 ...............
     $ grep -n ^%% ~/.crmconf/web
     23:%% ip 
     31:%% netmask 
     35:%% lvs_support 
     61:%% id 
     65:%% configfile 
     71:%% options 
     76:%% envfiles 
 ...............
 
 These lines are the only ones that should be modified. Simply
 append the parameter value at the end of the line. For instance,
 after editing this template, the result could look like this (we
 used tabs instead of spaces to make the values stand out):
 ...............
     $ grep -n ^%% ~/.crmconf/web
     23:%% ip                192.168.1.101
     31:%% netmask 
     35:%% lvs_support 
     61:%% id                websvc
     65:%% configfile        /etc/apache2/httpd.conf
     71:%% options 
     76:%% envfiles 
 ...............
 
 As you can see, the parameter line format is very simple:
 ...............
     %% <name> <value>
 ...............
 
 After editing the file, use `show` again to display the
 configuration:
 ...............
     crm(live)configure template# show
     primitive virtual-ip ocf:heartbeat:IPaddr \
             params ip="192.168.1.101"
     primitive apache ocf:heartbeat:apache \
             params configfile="/etc/apache2/httpd.conf"
     monitor apache 120s:60s
     group websvc \
             apache virtual-ip
 ...............
 
 The target resource of the apache template is a group which we
 named `websvc` in this sample session.
 
 This configuration looks exactly as you could type it at the
 `configure` level. The point of templates is to save you some
 typing. It is important, however, to understand the configuration
 produced.
 
 Finally, the configuration may be applied to the current
 crm configuration (note how the configuration changed slightly,
 though it is still equivalent, after being digested at the
 `configure` level):
 ...............
     crm(live)configure template# apply 
     crm(live)configure template# cd ..
     crm(live)configure# show
     node xen-b
     node xen-c
     primitive apache ocf:heartbeat:apache \
         params configfile="/etc/apache2/httpd.conf" \
         op monitor interval="120s" timeout="60s"
     primitive virtual-ip ocf:heartbeat:IPaddr \
         params ip="192.168.1.101"
     group websvc apache virtual-ip
 ...............
 
 Note that this still does not commit the configuration to the CIB
 which is used in the shell, either the running one (`live`) or
 some shadow CIB. For that you still need to execute the `commit`
 command.
 
 We should also define the preferred node to run the service:
 ...............
     crm(live)configure# location websvc-pref websvc 100: xen-b
 ...............
 
 If you are not happy with some resource names which are provided
 by default, you can rename them now:
 ...............
     crm(live)configure# rename virtual-ip intranet-ip
     crm(live)configure# show
     node xen-b
     node xen-c
     primitive apache ocf:heartbeat:apache \
             params configfile="/etc/apache2/httpd.conf" \
             op monitor interval="120s" timeout="60s"
     primitive intranet-ip ocf:heartbeat:IPaddr \
             params ip="192.168.1.101"
     group websvc apache intranet-ip
     location websvc-pref websvc 100: xen-b
 ...............
 
 To summarize, working with templates typically consists of the
 following steps:
 
 - `new`: create a new configuration from templates
 - `edit`: define parameters, at least the required ones
 - `show`: see if the configuration is valid
 - `apply`: apply the configuration to the `configure` level
 
 === Tab completion
 
 The `crm` makes extensive use of the tab completion of
 `readline`. The completion is both static (i.e. for `crm`
 commands) and dynamic. The latter takes into account the current
 status of the cluster or information from installed resource
 agents. Sometimes, completion may also be used to get short help
 on resource parameters. Here a few examples:
 ...............
     crm(live)# resource <TAB><TAB>
     bye         exit        manage      param       show        unmanage 
     cd          failcount   meta        quit        start       unmigrate 
     cleanup     help        migrate     refresh     status      unmove 
     end         list        move        reprobe     stop        up 
     crm(live)# configure 
     crm(live)configure# primitive fence-1 <TAB><TAB>
     heartbeat:  lsb:        ocf:        stonith:    
     crm(live)configure# primitive fence-1 stonith:ipmilan params <TAB><TAB>
     auth=      hostname=  ipaddr=    login=     password=  port=      priv=
     crm(live)configure# primitive fence-1 stonith:ipmilan params auth=<TAB><TAB>
     auth* (string)
         The authorization type of the IPMI session ("none", "straight", "md2", or "md5")
     crm(live)configure# primitive fence-1 stonith:ipmilan params auth=
 ...............
 
 [[SEMCHK]]
 === Configuration semantic checks
 
 Resource definitions may be checked against the meta-data
 provided with the resource agents. These checks are currently
 carried out:
 
 - are required parameters set
 - existence of defined parameters
 - timeout values for operations
 
 The parameter checks are obvious and need no further explanation.
 Failures in these checks are treated as configuration errors.
 
 The timeouts for operations should be at least as long as those
 recommended in the meta-data. Too short timeout values are a
 common mistake in cluster configurations and, even worse, they
 often slip through if cluster testing was not thorough. Though
 operation timeouts issues are treated as warnings, make sure that
 the timeouts are usable in your environment. Note also that the
 values given are just _advisory minimum_---your resources may
 require longer timeouts.
 
 User may tune the frequency of checks and the treatment of errors
 by the <<cmdhelp_options_check-frequency,`check-frequency`>> and
 <<cmdhelp_options_check-mode,`check-mode`>> preferences.
 
 Note that if the `check-frequency` is set to `always` and the
 `check-mode` to `strict`, errors are not tolerated and such
 configuration cannot be saved.
 
 == Reference
 
 We define a small and simple language. Most commands consist of
 just a list of simple tokens. The only complex constructs are
 found at the `configure` level.
 
 The syntax is described in a somewhat informal manner: `<>`
 denotes a string, `[]` means that the construct is optional, the
 ellipsis (`...`) signifies that the previous construct may be
 repeated, `|` means pick one of many, and the rest are literals
 (strings, `:`, `=`).
 
 === `status`
 
 Show cluster status. The status is displayed by `crm_mon`. Supply
 additional arguments for more information or different format.
 See `crm_mon(8)` for more details.
 
 Usage:
 ...............
         status [<option> ...]
 
         option :: bynode | inactive | ops | timing | failcounts
 ...............
 
 [[cmdhelp_cib,CIB shadow management]]
 === `cib` (shadow CIBs)
 
 This level is for management of shadow CIBs. It is available both
 at the top level and the `configure` level.
 
 All the commands are implemented using `cib_shadow(8)` and the
 `CIB_shadow` environment variable. The user prompt always
 includes the name of the currently active shadow or the live CIB.
 
 [[cmdhelp_cib_new,create a new shadow CIB]]
 ==== `new`
 
 Create a new shadow CIB. The live cluster configuration and
 status is copied to the shadow CIB. Specify `withstatus` if you
 want to edit the status section of the shadow CIB (see the
 <<cmdhelp_cibstatus,cibstatus section>>). Add `force` to force overwriting the
 existing shadow CIB.
 
 Usage:
 ...............
         new <cib> [withstatus] [force]
 ...............
 
 [[cmdhelp_cib_delete,delete a shadow CIB]]
 ==== `delete`
 
 Delete an existing shadow CIB.
 
 Usage:
 ...............
         delete <cib>
 ...............
 
 [[cmdhelp_cib_reset,copy live cib to a shadow CIB]]
 ==== `reset`
 
 Copy the current cluster configuration into the shadow CIB.
 
 Usage:
 ...............
         reset <cib>
 ...............
 
 [[cmdhelp_cib_commit,copy a shadow CIB to the cluster]]
 ==== `commit`
 
 Apply a shadow CIB to the cluster.
 
 Usage:
 ...............
         commit <cib>
 ...............
 
 [[cmdhelp_cib_use,change working CIB]]
 ==== `use`
 
 Choose a CIB source. If you want to edit the status from the
 shadow CIB specify `withstatus` (see <<cmdhelp_cibstatus,`cibstatus`>>).
 Leave out the CIB name to switch to the running CIB.
 
 Usage:
 ...............
         use [<cib>] [withstatus]
 ...............
 
 [[cmdhelp_cib_diff,diff between the shadow CIB and the live CIB]]
 ==== `diff`
 
 Print differences between the current cluster configuration and
 the active shadow CIB.
 
 Usage:
 ...............
         diff
 ...............
 
 [[cmdhelp_cib_list,list all shadow CIBs]]
 ==== `list`
 
 List existing shadow CIBs.
 
 Usage:
 ...............
         list
 ...............
 
 [[cmdhelp_cib_import,import a CIB or PE input file to a shadow]]
 ==== `import`
 
 At times it may be useful to create a shadow file from the
 existing CIB. The CIB may be specified as file or as a PE input
 file number. The shell will look up files in the local directory
 first and then in the PE directory (typically `/var/lib/pengine`).
 Once the CIB file is found, it is copied to a shadow and this
 shadow is immediately available for use at both `configure` and
 `cibstatus` levels.
 
 If the shadow name is omitted then the target shadow is named
 after the input CIB file.
 
 Note that there are often more than one PE input file, so you may
 need to specify the full name.
 
 Usage:
 ...............
         import {<file>|<number>} [<shadow>]
 ...............
 Examples:
 ...............
         import pe-warn-2222
         import 2289 issue2
 ...............
 
 [[cmdhelp_cib_cibstatus,CIB status management and editing]]
 ==== `cibstatus`
 
 Enter edit and manage the CIB status section level. See the
 <<cmdhelp_cibstatus,CIB status management section>>.
 
 [[cmdhelp_ra,Resource Agents (RA) lists and documentation]]
 === `ra`
 
 This level contains commands which show various information about
 the installed resource agents. It is available both at the top
 level and at the `configure` level.
 
 [[cmdhelp_ra_classes,list classes and providers]]
 ==== `classes`
 
 Print all resource agents' classes and, where appropriate, a list
 of available providers.
 
 Usage:
 ...............
         classes
 ...............
 
 [[cmdhelp_ra_list,list RA for a class (and provider)]]
 ==== `list`
 
 List available resource agents for the given class. If the class
 is `ocf`, supply a provider to get agents which are available
 only from that provider.
 
 Usage:
 ...............
         list <class> [<provider>]
 ...............
 Example:
 ...............
         list ocf pacemaker
 ...............
 
 [[cmdhelp_ra_meta,show meta data for a RA]]
 ==== `meta` (`info`)
 
 Show the meta-data of a resource agent type. This is where users
 can find information on how to use a resource agent.
 
 Usage:
 ...............
         meta [<class>:[<provider>:]]<type>
         meta <type> <class> [<provider>] (obsolete)
 ...............
 Example:
 ...............
         meta apache
         meta ocf:pacemaker:Dummy
         meta stonith:ipmilan
 ...............
 
 [[cmdhelp_ra_providers,show providers for a RA]]
 ==== `providers`
 
 List providers for a resource agent type.
 
 Usage:
 ...............
         providers <type>
 ...............
 Example:
 ...............
         providers apache
 ...............
 
 [[cmdhelp_resource,Resource management]]
 === `resource`
 
 At this level resources may be managed.
 
 All (or almost all) commands are implemented with the CRM tools
 such as `crm_resource(8)`.
 
 [[cmdhelp_resource_status,show status of resources]]
 ==== `status` (`show`, `list`)
 
 Print resource status. If the resource parameter is left out
 status of all resources is printed.
 
 Usage:
 ...............
         status [<rsc>]
 ...............
 
 [[cmdhelp_resource_start,start a resource]]
 ==== `start`
 
 Start a resource by setting the `target-role` attribute. If there
 are multiple meta attributes sets, the attribute is set in all of
 them. If the resource is a group or a clone, all `target-role`
 attributes are removed from the children resources.
 
 Usage:
 ...............
         start <rsc>
 ...............
 
 [[cmdhelp_resource_stop,stop a resource]]
 ==== `stop`
 
 Stop a resource using the `target-role` attribute. If there
 are multiple meta attributes sets, the attribute is set in all of
 them. If the resource is a group or a clone, all `target-role`
 attributes are removed from the children resources.
 
 Usage:
 ...............
         stop <rsc>
 ...............
 
 [[cmdhelp_resource_restart,restart a resource]]
 ==== `restart`
 
 Restart a resource. This is essentially a shortcut for resource
 stop followed by a start.
 
 Usage:
 ...............
         restart <rsc>
 ...............
 
 [[cmdhelp_resource_promote,promote a master-slave resource]]
 ==== `promote`
 
 Promote a master-slave resource using the `target-role`
 attribute.
 
 Usage:
 ...............
         promote <rsc>
 ...............
 
 [[cmdhelp_resource_demote,demote a master-slave resource]]
 ==== `demote`
 
 Demote a master-slave resource using the `target-role`
 attribute.
 
 Usage:
 ...............
         demote <rsc>
 ...............
 
 [[cmdhelp_resource_manage,put a resource into managed mode]]
 ==== `manage`
 
 Manage a resource using the `is-managed` attribute. If there
 are multiple meta attributes sets, the attribute is set in all of
 them. If the resource is a group or a clone, all `is-managed`
 attributes are removed from the children resources.
 
 Usage:
 ...............
         manage <rsc>
 ...............
 
 [[cmdhelp_resource_unmanage,put a resource into unmanaged mode]]
 ==== `unmanage`
 
 Unmanage a resource using the `is-managed` attribute. If there
 are multiple meta attributes sets, the attribute is set in all of
 them. If the resource is a group or a clone, all `is-managed`
 attributes are removed from the children resources.
 
 Usage:
 ...............
         unmanage <rsc>
 ...............
 
 [[cmdhelp_resource_migrate,migrate a resource to another node]]
 ==== `migrate` (`move`)
 
 Migrate a resource to a different node. If node is left out, the
 resource is migrated by creating a constraint which prevents it from
 running on the current node. Additionally, you may specify a
 lifetime for the constraint---once it expires, the location
 constraint will no longer be active.
 
 Usage:
 ...............
         migrate <rsc> [<node>] [<lifetime>]
 ...............
 
 [[cmdhelp_resource_unmigrate,unmigrate a resource to another node]]
 ==== `unmigrate` (`unmove`)
 
 Remove the constraint generated by the previous migrate command.
 
 Usage:
 ...............
         unmigrate <rsc>
 ...............
 
 [[cmdhelp_resource_param,manage a parameter of a resource]]
 ==== `param`
 
 Show/edit/delete a parameter of a resource.
 
 Usage:
 ...............
         param <rsc> set <param> <value>
         param <rsc> delete <param>
         param <rsc> show <param>
 ...............
 Example:
 ...............
         param ip_0 show ip
 ...............
 
 [[cmdhelp_resource_meta,manage a meta attribute]]
 ==== `meta`
 
 Show/edit/delete a meta attribute of a resource. Currently, all
 meta attributes of a resource may be managed with other commands
 such as `resource stop`.
 
 Usage:
 ...............
         meta <rsc> set <attr> <value>
         meta <rsc> delete <attr>
         meta <rsc> show <attr>
 ...............
 Example:
 ...............
         meta ip_0 set target-role stopped
 ...............
 
 [[cmdhelp_resource_utilization,manage a utilization attribute]]
 ==== `utilization`
 
 Show/edit/delete a utilization attribute of a resource. These
 attributes describe hardware requirements. By setting the
 `placement-strategy` cluster property appropriately, it is
 possible then to distribute resources based on resource
 requirements and node size. See also <<cmdhelp_node_utilization,node utilization attributes>>.
 
 Usage:
 ...............
         utilization <rsc> set <attr> <value>
         utilization <rsc> delete <attr>
         utilization <rsc> show <attr>
 ...............
 Example:
 ...............
         utilization xen1 set memory 4096
 ...............
 
 [[cmdhelp_resource_failcount,manage failcounts]]
 ==== `failcount`
 
 Show/edit/delete the failcount of a resource.
 
 Usage:
 ...............
         failcount <rsc> set <node> <value>
         failcount <rsc> delete <node>
         failcount <rsc> show <node>
 ...............
 Example:
 ...............
         failcount fs_0 delete node2
 ...............
 
 [[cmdhelp_resource_cleanup,cleanup resource status]]
 ==== `cleanup`
 
 Cleanup resource status. Typically done after the resource has
 temporarily failed. If a node is omitted, cleanup on all nodes.
 If there are many nodes, the command may take a while.
 
 Usage:
 ...............
         cleanup <rsc> [<node>]
 ...............
 
 [[cmdhelp_resource_refresh,refresh CIB from the LRM status]]
 ==== `refresh`
 
 Refresh CIB from the LRM status.
 
 Usage:
 ...............
         refresh [<node>]
 ...............
 
 [[cmdhelp_resource_reprobe,probe for resources not started by the CRM]]
 ==== `reprobe`
 
 Probe for resources not started by the CRM.
 
 Usage:
 ...............
         reprobe [<node>]
 ...............
 
 [[cmdhelp_node,Nodes management]]
 === `node`
 
 Node management and status commands.
 
 [[cmdhelp_node_status,show nodes' status]]
 ==== `status`
 
 Show nodes' status. If the node parameter is omitted then all
 nodes are shown.
 
 Usage:
 ...............
         status [<node>]
 ...............
 
 [[cmdhelp_node_show,show node]]
 ==== `show`
 
 Show a node definition. If the node parameter is omitted then all
 nodes are shown.
 
 Usage:
 ...............
         show [<node>]
 ...............
 
 [[cmdhelp_node_standby,put node into standby]]
 ==== `standby`
 
 Set a node to standby status. The node parameter defaults to the
 node where the command is run. Additionally, you may specify a
 lifetime for the standby---if set to `reboot`, the node will be
 back online once it reboots. `forever` will keep the node in
 standby after reboot.
 
 Usage:
 ...............
         standby [<node>] [<lifetime>]
 
         lifetime :: reboot | forever
 ...............
 
 [[cmdhelp_node_online,set node online]]
 ==== `online`
 
 Set a node to online status. The node parameter
 defaults to the node where the command is run.
 
 Usage:
 ...............
         online [<node>]
 ...............
 
 [[cmdhelp_node_fence,fence node]]
 ==== `fence`
 
 Make CRM fence a node. This functionality depends on stonith
 resources capable of fencing the specified node. No such stonith
 resources, no fencing will happen.
 
 Usage:
 ...............
         fence <node>
 ...............
 
 [[cmdhelp_node_delete,delete node]]
 ==== `delete`
 
 Delete a node. This command will remove the node from the CIB
 and, in case the heartbeat stack is running, run hb_delnode too.
 
 Usage:
 ...............
         delete <node>
 ...............
 
 [[cmdhelp_node_attribute,manage attributes]]
 ==== `attribute`
 
 Edit node attributes. This kind of attribute should refer to
 relatively static properties, such as memory size.
 
 Usage:
 ...............
         attribute <node> set <attr> <value>
         attribute <node> delete <attr>
         attribute <node> show <attr>
 ...............
 Example:
 ...............
         attribute node_1 set memory_size 4096
 ...............
 
 [[cmdhelp_node_utilization,manage utilization attributes]]
 ==== `utilization`
 
 Edit node utilization attributes. These attributes describe
 hardware characteristics as integer numbers such as memory size
 or the number of CPUs. By setting the `placement-strategy`
 cluster property appropriately, it is possible then to distribute
 resources based on resource requirements and node size. See also
 <<cmdhelp_resource_utilization,resource utilization attributes>>.
 
 Usage:
 ...............
         utilization <node> set <attr> <value>
         utilization <node> delete <attr>
         utilization <node> show <attr>
 ...............
 Examples:
 ...............
         utilization node_1 set memory 16384
         utilization node_1 show cpu
 ...............
 
 [[cmdhelp_node_status-attr,manage status attributes]]
 ==== `status-attr`
 
 Edit node attributes which are in the CIB status section, i.e.
 attributes which hold properties of a more volatile nature. One
 typical example is attribute generated by the `pingd` utility.
 
 Usage:
 ...............
         status-attr <node> set <attr> <value>
         status-attr <node> delete <attr>
         status-attr <node> show <attr>
 ...............
 Example:
 ...............
         status-attr node_1 show pingd
 ...............
 
 [[cmdhelp_options,user preferences]]
 === `options`
 
 The user may set various options for the CLI program itself.
 
 [[cmdhelp_options_skill-level,set skill level]]
 ==== `skill-level`
 
 Based on the skill-level setting, the user is allowed to use only
 a subset of commands. There are three levels: operator,
 administrator, and expert. The operator level allows only
 commands at the `resource` and `node` levels, but not editing
 or deleting resources. The administrator may do that and may also
 configure the cluster at the `configure` level and manage the
 shadow CIBs. The expert may do all.
 
 Usage:
 ...............
         skill-level <level>
         
         level :: operator | administrator | expert
 ...............
 
 [[cmdhelp_options_user,set the cluster user]]
 ==== `user`
 
 Sufficient privileges are necessary in order to manage a
 cluster: programs such as `crm_verify` or `crm_resource` and,
 ultimately, `cibadmin` have to be run either as `root` or as the
 CRM owner user (typically `hacluster`). You don't have to worry
 about that if you run `crm` as `root`. A more secure way is to
 run the program with your usual privileges, set this option to
 the appropriate user (such as `hacluster`), and setup the
 `sudoers` file.
 
 Usage:
 ...............
         user system-user
 ...............
 Example:
 ...............
         user hacluster
 ...............
 
 [[cmdhelp_options_editor,set preferred editor program]]
 ==== `editor`
 
 The `edit` command invokes an editor. Use this to specify your
 preferred editor program. If not set, it will default to either
 the value of the `EDITOR` environment variable or to one of the
 standard UNIX editors (`vi`,`emacs`,`nano`).
 
 Usage:
 ...............
         editor program
 ...............
 Example:
 ...............
         editor vim
 ...............
 
 [[cmdhelp_options_pager,set preferred pager program]]
 ==== `pager`
 
 The `view` command displays text through a pager. Use this to
 specify your preferred pager program. If not set, it will default
 to either the value of the `PAGER` environment variable or to one
 of the standard UNIX system pagers (`less`,`more`,`pg`).
 
 [[cmdhelp_options_sort-elements,sort CIB elements]]
 ==== `sort-elements`
 
 `crm` by default sorts CIB elements. If you want them appear in
 the order they were created, set this option to `no`.
 
 Usage:
 ...............
         sort-elements {yes|no}
 ...............
 Example:
 ...............
         sort-elements no
 ...............
 
 [[cmdhelp_options_output,set output type]]
 ==== `output`
 
 `crm` can adorn configurations in two ways: in color (similar to
 for instance the `ls --color` command) and by showing keywords in
 upper case. Possible values are `plain`, `color`, and
 'uppercase'. It is possible to combine the latter two in order to
 get an upper case xmass tree. Just set this option to
 `color,uppercase`.
 
 [[cmdhelp_options_colorscheme,set colors for output]]
 ==== `colorscheme`
 
 With `output` set to `color`, a comma separated list of colors
 from this option are used to emphasize:
 
 - keywords
 - object ids
 - attribute names
 - attribute values
 - scores
 - resource references
 
 `crm` can show colors only if there is curses support for python
 installed (usually provided by the `python-curses` package). The
 colors are whatever is available in your terminal. Use `normal`
 if you want to keep the default foreground color.
 
 This user preference defaults to
 `yellow,normal,cyan,red,green,magenta` which is good for
 terminals with dark background. You may want to change the color
 scheme and save it in the preferences file for other color
 setups.
 
 Example:
 ...............
     colorscheme yellow,normal,blue,red,green,magenta
 ...............
 
 [[cmdhelp_options_check-frequency,when to perform semantic check]]
 ==== `check-frequency`
 
 Semantic check of the CIB or elements modified or created may be
 done on every configuration change (`always`), when verifying
 (`on-verify`) or `never`. It is by default set to `always`.
 Experts may want to change the setting to `on-verify`.
 
 The checks require that resource agents are present. If they are
 not installed at the configuration time set this preference to
 `never`.
 
 See <<SEMCHK,Configuration semantic checks>> for more details.
 
 [[cmdhelp_options_check-mode,how to treat semantic errors]]
 ==== `check-mode`
 
 Semantic check of the CIB or elements modified or created may be
 done in the `strict` mode or in the `relaxed` mode. In the former
 certain problems are treated as configuration errors. In the
 `relaxed` mode all are treated as warnings. The default is `strict`.
 
 See <<SEMCHK,Configuration semantic checks>> for more details.
 
 [[cmdhelp_options_show,show current user preference]]
 ==== `show`
 
 Display all current settings.
 
 [[cmdhelp_options_save,save the user preferences to the rc file]]
 ==== `save`
 
 Save current settings to the rc file (`$HOME/.crm.rc`). On
 further `crm` runs, the rc file is automatically read and parsed.
 
 [[cmdhelp_configure,CIB configuration]]
 === `configure`
 
 This level enables all CIB object definition commands.
 
 The configuration may be logically divided into four parts:
 nodes, resources, constraints, and (cluster) properties and
 attributes.  Each of these commands support one or more basic CIB
 objects.
 
 Nodes and attributes describing nodes are managed using the
 `node` command.
 
 Commands for resources are:
 
 - `primitive`
 - `monitor`
 - `group`
 - `clone`
 - `ms`/`master` (master-slave)
 
 There are three types of constraints:
 
 - `location`
 - `colocation`
 - `order`
 
 Finally, there are the cluster properties, resource meta
 attributes defaults, and operations defaults. All are just a set
 of attributes. These attributes are managed by the following
 commands:
 
 - `property`
 - `rsc_defaults`
 - `op_defaults`
 
 The changes applied to the current CIB only on ending the
 configuration session or using the `commit` command.
 
+Comments start with `#` in the first line. The comments are tied
+to the element which follows. If the element moves, its comments
+will follow.
+
 [[cmdhelp_configure_node,define a cluster node]]
 ==== `node`
 
 The node command describes a cluster node. Nodes in the CIB are
 commonly created automatically by the CRM. Hence, you should not
 need to deal with nodes unless you also want to define node
 attributes. Note that it is also possible to manage node
 attributes at the `node` level.
 
 Usage:
 ...............
         node <uname>[:<type>]
           [attributes <param>=<value> [<param>=<value>...]]
           [utilization <param>=<value> [<param>=<value>...]]
 
         type :: normal | member | ping
 ...............
 Example:
 ...............
         node node1
         node big_node attributes memory=64
 ...............
 
 [[cmdhelp_configure_primitive,define a resource]]
 ==== `primitive`
 
 The primitive command describes a resource. It may be referenced
 only once in group, clone, or master-slave objects. If it's not
 referenced, then it is placed as a single resource in the CIB.
 
 Operations may be specified in three ways. "Anonymous" as a
 simple list of "op" specifications. Use that if you don't want to
 reference the set of operations elsewhere. That's by far the most
 common way to define operations. If reusing operation sets is
 desired, use the "operations" keyword along with the id to give
 the operations set a name and the id-ref to reference another set
 of operations.
 
 Operation's attributes which are not recognized are saved as
 instance attributes of that operation. A typical example is
 `OCF_CHECK_LEVEL`.
 
 Usage:
 ...............
         primitive <rsc> [<class>:[<provider>:]]<type>
           [params attr_list]
           [meta attr_list]
           [utilization attr_list]
           [operations id_spec]
             [op op_type [<attribute>=<value>...] ...]
 
         attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
         id_spec :: $id=<id> | $id-ref=<id>
         op_type :: start | stop | monitor
 ...............
 Example:
 ...............
         primitive apcfence stonith:apcsmart \
           params ttydev=/dev/ttyS0 hostlist="node1 node2" \
           op start timeout=60s \
           op monitor interval=30m timeout=60s
 
         primitive www8 apache \
           params configfile=/etc/apache/www8.conf \
           operations $id-ref=apache_ops
 
         primitive db0 mysql \
           params config=/etc/mysql/db0.conf \
           op monitor interval=60s \
           op monitor interval=300s OCF_CHECK_LEVEL=10
 ...............
 
 [[cmdhelp_configure_monitor,add monitor operation to a primitive]]
 ==== `monitor`
 
 Monitor is by far the most common operation. It is possible to
 add it without editing the whole resource. Also, long primitive
 definitions may be a bit uncluttered. In order to make this
 command as concise as possible, less common operation attributes
 are not available. If you need them, then use the `op` part of
 the `primitive` command.
 
 Usage:
 ...............
         monitor <rsc>[:<role>] <interval>[:<timeout>]
 ...............
 Example:
 ...............
         monitor apcfence 60m:60s
 ...............
 
 Note that after executing the command, the monitor operation may
 be shown as part of the primitive definition.
 
 [[cmdhelp_configure_group,define a group]]
 ==== `group`
 
 The `group` command creates a group of resources.
 
 Usage:
 ...............
         group <name> <rsc> [<rsc>...]
           [meta attr_list]
           [params attr_list]
 
         attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
 ...............
 Example:
 ...............
         group internal_www disk0 fs0 internal_ip apache \
           meta target_role=stopped
 ...............
 
 [[cmdhelp_configure_clone,define a clone]]
 ==== `clone`
 
 The `clone` command creates a resource clone. It may contain a
 single primitive resource or one group of resources.
 
 Usage:
 ...............
         clone <name> <rsc>
           [meta attr_list]
           [params attr_list]
 
         attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
 ...............
 Example:
 ...............
         clone cl_fence apc_1 \
           meta clone-node-max=1 globally-unique=false
 ...............
 
 [[cmdhelp_configure_ms,define a master-slave resource]]
 ==== `ms` (`master`)
 
 The `ms` command creates a master/slave resource type. It may contain a
 single primitive resource or one group of resources.
 
 Usage:
 ...............
         ms <name> <rsc>
           [meta attr_list]
           [params attr_list]
 
         attr_list :: [$id=<id>] <attr>=<val> [<attr>=<val>...] | $id-ref=<id>
 ...............
 Example:
 ...............
         ms disk1 drbd1 \
           meta notify=true globally-unique=false
 ...............
 
 .Note on `id-ref` usage
 ****************************
 Instance or meta attributes (`params` and `meta`) may contain
 a reference to another set of attributes. In that case, no other
 attributes are allowed. Since attribute sets' ids, though they do
 exist, are not shown in the `crm`, it is also possible to
 reference an object instead of an attribute set. `crm` will
 automatically replace such a reference with the right id:
 
 ...............
     crm(live)configure# primitive a2 www-2 meta $id-ref=a1
     crm(live)configure# show a2
     primitive a2 ocf:heartbeat:apache \
         meta $id-ref="a1-meta_attributes"
         [...]
 ...............
 It is advisable to give meaningful names to attribute sets which
 are going to be referenced.
 ****************************
 
 [[cmdhelp_configure_location,a location preference]]
 ==== `location`
 
 `location` defines the preference of nodes for the given
 resource. The location constraints consist of one or more rules
 which specify a score to be awarded if the rule matches.
 
 Usage:
 ...............
         location <id> <rsc> {node_pref|rules}
 
         node_pref :: <score>: <node>
 
         rules ::
           rule [id_spec] [$role=<role>] <score>: <expression>
           [rule [id_spec] [$role=<role>] <score>: <expression> ...]
 
         id_spec :: $id=<id> | $id-ref=<id>
         score :: <number> | <attribute> | [-]inf
         expression :: <simple_exp> [bool_op <simple_exp> ...]
         bool_op :: or | and
         simple_exp :: <attribute> [type:]<binary_op> <value>
                       | <unary_op> <attribute>
                       | date <date_expr>
         type :: string | version | number
         binary_op :: lt | gt | lte | gte | eq | ne
         unary_op :: defined | not_defined
 
         date_expr :: lt <end>
                      | gt <start>
                      | in_range start=<start> end=<end>
                      | in_range start=<start> <duration>
                      | date_spec <date_spec>
         duration|date_spec ::
                      hours=<value>
                      | monthdays=<value>
                      | weekdays=<value>
                      | yearsdays=<value>
                      | months=<value>
                      | weeks=<value>
                      | years=<value>
                      | weekyears=<value>
                      | moon=<value>
 ...............
 Examples:
 ...............
         location conn_1 internal_www 100: node1
 
         location conn_1 internal_www \
           rule 50: #uname eq node1 \
           rule pingd: defined pingd
 
         location conn_2 dummy_float \
           rule -inf: not_defined pingd or pingd number:lte 0
 ...............
 
 [[cmdhelp_configure_colocation,colocate resources]]
 ==== `colocation` (`collocation`)
 
 This constraint expresses the placement relation between two
 or more resources. If there are more than two resources, then the
 constraint is called a resource set. Collocation resource sets have
 an extra attribute to allow for sets of resources which don't depend
 on each other in terms of state. The shell syntax for such sets is
 to put resources in parentheses.
 
 Usage:
 ...............
         colocation <id> <score>: <rsc>[:<role>] <rsc>[:<role>] ...
 ...............
 Example:
 ...............
         colocation dummy_and_apache -inf: apache dummy
         colocation c1 inf: A ( B C )
 ...............
 
 [[cmdhelp_configure_order,order resources]]
 ==== `order`
 
 This constraint expresses the order of actions on two resources
 or more resources. If there are more than two resources, then the
 constraint is called a resource set. Ordered resource sets have an
 extra attribute to allow for sets of resources whose actions may run
 in parallel. The shell syntax for such sets is to put resources in
 parentheses.
 
 Usage:
 ...............
         order <id> score-type: <rsc>[:<action>] <rsc>[:<action>] ...
           [symmetrical=<bool>]
 
         score-type :: advisory | mandatory | <score>
 ...............
 Example:
 ...............
         order c_apache_1 mandatory: apache:start ip_1
         order o1 inf: A ( B C )
 ...............
 
 [[cmdhelp_configure_property,set a cluster property]]
 ==== `property`
 
 Set the cluster (`crm_config`) options.
 
 Usage:
 ...............
         property [$id=<set_id>] <option>=<value> [<option>=<value> ...]
 ...............
 Example:
 ...............
         property stonith-enabled=true
 ...............
 
 [[cmdhelp_configure_rsc_defaults,set resource defaults]]
 ==== `rsc_defaults`
 
 Set defaults for the resource meta attributes.
 
 Usage:
 ...............
         rsc_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...]
 ...............
 Example:
 ...............
         rsc_defaults failure-timeout=3m
 ...............
 
 [[cmdhelp_configure_op_defaults,set resource operations defaults]]
 ==== `op_defaults`
 
 Set defaults for the operations meta attributes.
 
 Usage:
 ...............
         op_defaults [$id=<set_id>] <option>=<value> [<option>=<value> ...]
 ...............
 Example:
 ...............
         op_defaults record-pending=true
 ...............
 
 [[cmdhelp_configure_show,display CIB objects]]
 ==== `show`
 
 The `show` command displays objects. It may display all objects
 or a set of objects. The user may also choose to see only objects
 which were changed.
 Optionally, the XML code may be displayed instead of the CLI
 representation.
 
 Usage:
 ...............
         show [xml] [<id> ...]
         show [xml] changed
 ...............
 
 [[cmdhelp_configure_edit,edit CIB objects]]
 ==== `edit`
 
 This command invokes the editor with the object description. As
 with the `show` command, the user may choose to edit all objects
 or a set of objects.
 
 If the user insists, he or she may edit the XML edition of the
 object. If you do that, don't modify any id attributes.
 
 Usage:
 ...............
         edit [xml] [<id> ...]
         edit [xml] changed
 ...............
 
 [[cmdhelp_configure_delete,delete CIB objects]]
 ==== `delete`
 
 Delete one or more objects. If an object to be deleted belongs to
 a container object, such as a group, and it is the only resource
 in that container, then the container is deleted as well. Any
 related constraints are removed as well.
 
 Usage:
 ...............
         delete <id> [<id>...]
 ...............
 
 [[cmdhelp_configure_rename,rename a CIB object]]
 ==== `rename`
 
 Rename an object. It is recommended to use this command to rename
 a resource, because it will take care of updating all related
 constraints and a parent resource. Changing ids with the edit
 command won't have the same effect.
 
 If you want to rename a resource, it must be in the stopped state.
 
 Usage:
 ...............
         rename <old_id> <new_id>
 ...............
 
 [[cmdhelp_configure_refresh,refresh from CIB]]
 ==== `refresh`
 
 Refresh the internal structures from the CIB. All changes made
 during this session are lost.
 
 Usage:
 ...............
         refresh
 ...............
 
 [[cmdhelp_configure_erase,erase the CIB]]
 ==== `erase`
 
 The `erase` clears all configuration. Apart from nodes. To remove
 nodes, you have to specify an additional keyword `nodes`.
 
 Note that removing nodes from the live cluster may have some
 strange/interesting/unwelcome effects.
 
 Usage:
 ...............
         erase [nodes]
 ...............
 
 [[cmdhelp_configure_ptest,show cluster actions if changes were committed]]
 ==== `ptest`
 
 Show PE (Policy Engine) motions using `ptest(8)`.
 
 A CIB is constructed using the current user edited configuration
 and the status from the running CIB. The resulting CIB is run
 through `ptest` to show changes which would happen if the
 configuration is committed.
 
 The status section may be loaded from another source and modified
 using the <<cmdhelp_cibstatus,`cibstatus`>> level commands. In that case, the
 `ptest` command will issue a message informing the user that the
 Policy Engine graph is not calculated based on the current status
 section and therefore won't show what would happen to the
 running but some imaginary cluster.
 
 If you have graphviz installed and X11 session, `dotty(1)` is run
 to display the changes graphically.
 
 Add a string of `v` characters to increase verbosity. `ptest`
 can also show allocation scores.
 
 Usage:
 ...............
         ptest [nograph] [v...] [scores]
 ...............
 Examples:
 ...............
         ptest scores
         ptest vvvvv
 ...............
 
 [[cmdhelp_configure_cibstatus,CIB status management and editing]]
 ==== `cibstatus`
 
 Enter edit and manage the CIB status section level. See the
 <<cmdhelp_cibstatus,CIB status management section>>.
 
 [[cmdhelp_configure_template,edit and import a configuration from a template]]
 ==== `template`
 
 The specified template is loaded into the editor. It's up to the
 user to make a good CRM configuration out of it. See also the
 <<cmdhelp_template,template section>>.
 
 Usage:
 ...............
         template [xml] url
 ...............
 Example:
 ...............
         template two-apaches.txt
 ...............
 
 [[cmdhelp_configure_commit,commit the changes to the CIB]]
 ==== `commit`
 
 Commit the current configuration to the CIB in use. As noted
 elsewhere, commands in a configure session don't have immediate
 effect on the CIB. All changes are applied at one point in time,
 either using `commit` or when the user leaves the configure
 level. In case the CIB in use changed in the meantime, presumably
 by somebody else, the CLI will refuse to apply the changes. If
 you know that it's fine to still apply them add `force`.
 
 Usage:
 ...............
         commit [force]
 ...............
 
 [[cmdhelp_configure_verify,verify the CIB with crm_verify]]
 ==== `verify`
 
 Verify the contents of the CIB which would be committed.
 
 Usage:
 ...............
         verify
 ...............
 
 [[cmdhelp_configure_upgrade,upgrade the CIB to version 1.0]]
 ==== `upgrade`
 
 If you get the `CIB not supported` error, which typically means
 that the current CIB version is coming from the older release,
 you may try to upgrade it to the latest revision. The command
 to perform the upgrade is:
 ...............
     # cibadmin --upgrade --force
 ...............
 
 If we don't recognize the current CIB as the old one, but you're
 sure that it is, you may force the command.
 
 Usage:
 ...............
         upgrade [force]
 ...............
 
 [[cmdhelp_configure_save,save the CIB to a file]]
 ==== `save`
 
 Save the configuration of the current level to a file.
 Optionally, as XML.
 
 Usage:
 ...............
         save [xml] <file>
 ...............
 Example:
 ...............
         save myfirstcib.txt
 ...............
 
 [[cmdhelp_configure_load,import the CIB from a file]]
 ==== `load`
 
 Load a part of configuration (or all of it) from a local file or
 a network URL. The `replace` method replaces the current
 configuration with the one from the source. The `update` tries to
 import the contents into the current configuration.
 The file may be a CLI file or an XML file.
 
 Usage:
 ...............
         load [xml] <method> URL
 
         method :: replace | update
 ...............
 Example:
 ...............
         load xml update myfirstcib.xml
         load xml replace http://storage.big.com/cibs/bigcib.xml
 ...............
 
 [[cmdhelp_configure_xml,raw xml]]
 ==== `xml`
 
 Even though we promissed no xml, it may happen, but hopefully
 very very seldom, that an element from the CIB cannot be rendered
 in the configuration language. In that case, the element will be
 shown as raw xml, prefixed by this command. That element can then
 be edited like any other. If the shell finds out that after the
 change it can digest it, then it is going to be converted into
 the normal configuration language. Otherwise, there is no need to
 use `xml` for configuration.
 
 Usage:
 ...............
         xml <xml>
 ...............
 
 [[cmdhelp_template,edit and import a configuration from a template]]
 === `template`
 
 User may be assisted in the cluster configuration by templates
 prepared in advance. Templates consist of a typical ready
 configuration which may be edited to suit particular user needs.
 
 This command enters a template level where additional commands
 for configuration/template management are available.
 
 [[cmdhelp_template_new,create a new configuration from templates]]
 ==== `new`
 
 Create a new configuration from one or more templates. Note that
 configurations and templates are kept in different places, so it
 is possible to have a configuration name equal a template name.
 
 If you already know which parameters are required, you can set
 them directly on the command line.
 
 The parameter name `id` is set by default to the name of the
 configuration.
 
 Usage:
 ...............
         new <config> <template> [<template> ...] [params name=value ...]"
 ...............
 Examples:
 ...............
         new vip virtual-ip
         new bigfs ocfs2 params device=/dev/sdx8 directory=/bigfs
 ...............
 
 [[cmdhelp_template_load,load a configuration]]
 ==== `load`
 
 Load an existing configuration. Further `edit`, `show`, and
 `apply` commands will refer to this configuration.
 
 Usage:
 ...............
         load <config>
 ...............
 
 [[cmdhelp_template_edit,edit a configuration]]
 ==== `edit`
 
 Edit current or given configuration using your favourite editor.
 
 Usage:
 ...............
         edit [<config>]
 ...............
 
 [[cmdhelp_template_delete,delete a configuration]]
 ==== `delete`
 
 Remove a configuration. The loaded (active) configuration may be
 removed by force.
 
 Usage:
 ...............
         delete <config> [force]
 ...............
 
 [[cmdhelp_template_list,list configurations/templates]]
 ==== `list`
 
 List existing configurations or templates.
 
 Usage:
 ...............
         list [templates]
 ...............
 
 [[cmdhelp_template_apply,process and apply the current configuration to the current CIB]]
 ==== `apply`
 
 Copy the current or given configuration to the current CIB. By
 default, the CIB is replaced, unless the method is set to
 "update".
 
 Usage:
 ...............
         apply [<method>] [<config>]
 
         method :: replace | update
 ...............
 
 [[cmdhelp_template_show,show the processed configuration]]
 ==== `show`
 
 Process the current or given configuration and display the result.
 
 Usage:
 ...............
         show [<config>]
 ...............
 
 [[cmdhelp_cibstatus,CIB status management and editing]]
 === `cibstatus`
 
 The `cibstatus` section of the CIB keeps the current status of nodes
 and resources. It is modified _only_ on events, i.e. when some
 resource operation is run or node status changes. For obvious
 reasons, the CRM has no user interface with which it is possible
 to affect the status section. From the user's point of view, the
 status section is essentially a read-only part of the CIB. The
 current status is never even written to disk, though it is
 available in the PE (Policy Engine) input files which represent
 the history of cluster motions. The current status may be read
 using the `cibadmin -Q` command.
 
 It may sometimes be of interest to see how status changes would
 affect the Policy Engine. The set of `cibstatus` level commands
 allow the user to load status sections from various sources and
 then mimic resource operations or node changes. The effect of
 those changes may then be observed by running the
 <<cmdhelp_configure_ptest,`ptest`>> command.
 
 [[cmdhelp_cibstatus_load,load the CIB status section]]
 ==== `load`
 
 Load a status section from a file, a shadow CIB, or the running
 cluster. By default, the current (`live`) status section is
 modified. Note that if the `live` status section is modified it
 is not going to be updated if the cluster status changes, because
 that would overwrite the user changes. To make `crm` drop changes
 and resume use of the running cluster status, run `load live`.
 
 All CIB shadow configurations contain the status section which is
 a snapshot of the status section taken at the time the shadow was
 created. Obviously, this status section doesn't have much to do
 with the running cluster status, unless the shadow CIB has just
 been created. Therefore, the `ptest` command by default uses the
 running cluster status section.
 
 Usage:
 ...............
         load {<file>|shadow:<cib>|live}
 ...............
 Example:
 ...............
         load bug-12299.xml
         load shadow:test1
 ...............
 
 [[cmdhelp_cibstatus_save,save the CIB status section]]
 ==== `save`
 
 The current internal status section with whatever modifications
 were performed can be saved to a file or shadow CIB.
 
 If the file exists and contains a complete CIB, only the status
 section is going to be replaced and the rest of the CIB will
 remain intact. Otherwise, the current user edited configuration
 is saved along with the status section.
 
 If the argument is omitted, the status is saved to wherever the
 status section originated. Unless, of course, the origin is the
 running cluster.
 
 Usage:
 ...............
         save [<file>|shadow:<cib>]
 ...............
 Example:
 ...............
         save bug-12299.xml
 ...............
 
 [[cmdhelp_cibstatus_origin,display origin of the CIB status section]]
 ==== `origin`
 
 Show the origin of the status section currently in use. This
 essentially shows the latest `load` argument.
 
 Usage:
 ...............
         origin
 ...............
 
 [[cmdhelp_cibstatus_show,show CIB status section]]
 ==== `show`
 
 Show the current status section in the XML format. Brace yourself
 for some unreadable output. Add `changed` option to get a human
 readable output of all changes.
 
 Usage:
 ...............
         show [changed]
 ...............
 
 [[cmdhelp_cibstatus_node,change node status]]
 ==== `node`
 
 Change the node status. It is possible to throw a node out of
 the cluster, make it a member, or set its state to unclean.
 
 `online`:: Set the `node_state` `crmd` attribute to `online`
 and the `expected` and `join` attributes to `member`. The effect
 is that the node becomes a cluster member.
 
 `offline`:: Set the `node_state` `crmd` attribute to `offline`
 and the `expected` attribute to empty. This makes the node
 cleanly removed from the cluster.
 
 `unclean`:: Set the `node_state` `crmd` attribute to `offline`
 and the `expected` attribute to `member`. In this case the node
 has unexpectedly disappeared.
 
 Usage:
 ...............
         node <node> {online|offline|unclean}
 ...............
 Example:
 ...............
         node xen-b unclean
 ...............
 
 [[cmdhelp_cibstatus_op,edit outcome of a resource operation]]
 ==== `op`
 
 Edit the outcome of a resource operation. This way you can
 tell CRM that it ran an operation and that the resource agent
 returned certain exit code. It is also possible to change the
 operation's status. In case the operation status is set to
 something other than `done`, the exit code is effectively
 ignored.
 
 Usage:
 ...............
         op <operation> <resource> <exit_code> [<op_status>] [<node>]
 
         operation :: probe | monitor[:<n>] | start | stop |
            promote | demote | notify | migrate_to | migrate_from
         exit_code :: <rc> | success | generic | args | 
            unimplemented | perm | installed | configured | not_running | 
            master | failed_master
         op_status :: pending | done | cancelled | timeout | notsupported | error
 
         n :: the monitor interval in seconds; if omitted, the first
            recurring operation is referenced
         rc :: numeric exit code in range 0..9
 ...............
 Example:
 ...............
         op start d1 xen-b generic
         op start d1 xen-b 1
         op monitor d1 xen-b not_running
         op stop d1 xen-b 0 timeout
 ...............
 
 === `end` (`cd`, `up`)
 
 The `end` command ends the current level and the user moves to
 the parent level. This command is available everywhere.
 
 Usage:
 ...............
         end
 ...............
 
 === `help`
 
 The `help` command prints help for the current level or for the
 specified topic (command). This command is available everywhere.
 
 Usage:
 ...............
         help [<topic>]
 ...............
 
 === `quit` (`exit`, `bye`)
 
 Leave the program.
 
 //////////////////////
  vim:ts=4:sw=4:expandtab:
 //////////////////////
diff --git a/shell/modules/cibconfig.py b/shell/modules/cibconfig.py
index a3bd32d3d2..862db5aae2 100644
--- a/shell/modules/cibconfig.py
+++ b/shell/modules/cibconfig.py
@@ -1,2127 +1,2135 @@
 # Copyright (C) 2008 Dejan Muhamedagic <dmuhamedagic@suse.de>
 # 
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public
 # License as published by the Free Software Foundation; either
 # version 2.1 of the License, or (at your option) any later version.
 # 
 # This software is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 # General Public License for more details.
 # 
 # You should have received a copy of the GNU General Public
 # License along with this library; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 #
 
 import sys
 import subprocess
 import copy
 import xml.dom.minidom
 import re
 
 from singletonmixin import Singleton
 from userprefs import Options, UserPrefs
 from vars import Vars
 from cliformat import *
 from utils import *
 from xmlutil import *
 from msg import *
 from parse import CliParser
 from clidisplay import CliDisplay
 from cibstatus import CibStatus
 from idmgmt import IdMgmt
 from ra import RAInfo
 
 def show_unrecognized_elems(doc):
     try:
         conf = doc.getElementsByTagName("configuration")[0]
     except:
         common_warn("CIB has no configuration element")
         return
     for topnode in conf.childNodes:
         if not is_element(topnode):
             continue
         if is_defaults(topnode):
             continue
         if not topnode.tagName in cib_topnodes:
             common_warn("unrecognized CIB element %s" % c.tagName)
             continue
         for c in topnode.childNodes:
             if not is_element(c):
                 continue
             if not c.tagName in cib_object_map:
                 common_warn("unrecognized CIB element %s" % c.tagName)
 
 #
 # object sets (enables operations on sets of elements)
 #
 def mkset_obj(*args):
     if args and args[0] == "xml":
         obj = lambda: CibObjectSetRaw(*args[1:])
     else:
         obj = lambda: CibObjectSetCli(*args)
     return obj()
 
 class CibObjectSet(object):
     '''
     Edit or display a set of cib objects.
     repr() for objects representation and
     save() used to store objects into internal structures
     are defined in subclasses.
     '''
     def __init__(self, *args):
         self.obj_list = []
     def _open_url(self,src):
         import urllib
         try:
             return urllib.urlopen(src)
         except:
             pass
         if src == "-":
             return sys.stdin
         try:
             return open(src)
         except:
             pass
         common_err("could not open %s" % src)
         return False
     def init_aux_lists(self):
         '''
         Before edit, initialize two auxiliary lists which will
         hold a list of objects to be removed and a list of
         objects which were created. Then, we can create a new
         object list which will match the current state of
         affairs, i.e. the object set after the last edit.
         '''
         self.remove_objs = copy.copy(self.obj_list)
         self.add_objs = []
     def recreate_obj_list(self):
         '''
         Recreate obj_list: remove deleted objects and add
         created objects
         '''
         for obj in self.remove_objs:
             self.obj_list.remove(obj)
         self.obj_list += self.add_objs
         rmlist = []
         for obj in self.obj_list:
             if obj.invalid:
                 rmlist.append(obj)
         for obj in rmlist:
             self.obj_list.remove(obj)
     def edit_save(self,s,erase = False):
         '''
         Save string s to a tmp file. Invoke editor to edit it.
         Parse/save the resulting file. In case of syntax error,
         allow user to reedit. If erase is True, erase the CIB
         first.
         If no changes are done, return silently.
         '''
         tmp = str2tmp(s)
         if not tmp:
             return False
         filehash = hash(s)
         rc = False
         while True:
             if edit_file(tmp) != 0:
                 break
             try: f = open(tmp,'r')
             except IOError, msg:
                 common_err(msg)
                 break
             s = ''.join(f)
             f.close()
             if hash(s) == filehash: # file unchanged
                 rc = True
                 break
             if erase:
                 cib_factory.erase()
             if not self.save(s):
                 if ask("Do you want to edit again?"):
                     continue
             rc = True
             break
         try: os.unlink(tmp)
         except: pass
         return rc
     def edit(self):
         if options.batch:
             common_info("edit not allowed in batch mode")
             return False
         cli_display.set_no_pretty()
         s = self.repr()
         cli_display.reset_no_pretty()
         return self.edit_save(s)
     def save_to_file(self,fname):
         if fname == "-":
             f = sys.stdout
         else:
             if not options.batch and os.access(fname,os.F_OK):
                 if not ask("File %s exists. Do you want to overwrite it?"%fname):
                     return False
             try: f = open(fname,"w")
             except IOError, msg:
                 common_err(msg)
                 return False
         rc = True
         cli_display.set_no_pretty()
         s = self.repr()
         cli_display.reset_no_pretty()
         if s:
             f.write(s)
             f.write('\n')
         elif self.obj_list:
             rc = False
         if f != sys.stdout:
             f.close()
         return rc
     def show(self):
         s = self.repr()
         if not s:
             if self.obj_list: # objects could not be displayed
                 return False
             else:
                 return True
         page_string(s)
     def import_file(self,method,fname):
         if not cib_factory.is_cib_sane():
             return False
         if method == "replace":
             if options.interactive and cib_factory.has_cib_changed():
                 if not ask("This operation will erase all changes. Do you want to proceed?"):
                     return False
             cib_factory.erase()
         f = self._open_url(fname)
         if not f:
             return False
         s = ''.join(f)
         if f != sys.stdin:
             f.close()
         return self.save(s, method == "update")
     def repr(self):
         '''
         Return a string with objects's representations (either
         CLI or XML).
         '''
         return ''
     def save(self, s, update = False):
         '''
         For each object:
             - try to find a corresponding object in obj_list
             - if (update and not found) or found:
               replace the object in the obj_list with
               the new object
             - if not found: create new
         See below for specific implementations.
         '''
         pass
     def verify2(self):
         '''
         Test objects for sanity. This is about semantics.
         '''
         rc = 0
         for obj in self.obj_list:
             rc |= obj.check_sanity()
         return rc
     def lookup_cli(self,cli_list):
         for obj in self.obj_list:
             if obj.matchcli(cli_list):
                 return obj
     def lookup(self,xml_obj_type,obj_id):
         for obj in self.obj_list:
             if obj.match(xml_obj_type,obj_id):
                 return obj
     def drop_remaining(self):
         'Any remaining objects in obj_list are deleted.'
         l = [x.obj_id for x in self.remove_objs]
         return cib_factory.delete(*l)
 
 def get_comments(cli_list):
     if not cli_list:
         return []
     last = cli_list[len(cli_list)-1]
     try:
         if last[0] == "comments":
             cli_list.pop()
             return last[1]
     except: pass
     return []
 
 class CibObjectSetCli(CibObjectSet):
     '''
     Edit or display a set of cib objects (using cli notation).
     '''
     def __init__(self, *args):
         CibObjectSet.__init__(self, *args)
         self.obj_list = cib_factory.mkobj_list("cli",*args)
     def repr(self):
         "Return a string containing cli format of all objects."
         if not self.obj_list:
             return ''
         return '\n'.join(obj.repr_cli() \
             for obj in processing_sort_cli(self.obj_list))
     def process(self, cli_list, update = False):
         '''
         Create new objects or update existing ones.
         '''
-        comments = get_comments(cli_list)
         myobj = obj = self.lookup_cli(cli_list)
         if update and not obj:
             obj = cib_factory.find_object_for_cli(cli_list)
         if obj:
             rc = cib_factory.update_from_cli(obj,cli_list) != False
             if myobj:
                 self.remove_objs.remove(myobj)
         else:
             obj = cib_factory.create_from_cli(cli_list)
             rc = obj != None
             if rc:
                 self.add_objs.append(obj)
-        if rc:
-            obj.set_comment(comments)
         return rc
     def save(self, s, update = False):
         '''
         Save a user supplied cli format configuration.
         On errors user is typically asked to review the
         configuration (for instance on editting).
 
         On syntax error (return code 1), no changes are done, but
         on semantic errors (return code 2), some changes did take
         place so object list must be updated properly.
 
         Finally, once syntax check passed, there's no way back,
         all changes are applied to the current configuration.
 
         TODO: Implement undo configuration changes.
         '''
         l = []
         rc = True
         err_buf.start_tmp_lineno()
         cp = CliParser()
         for cli_text in lines2cli(s):
             err_buf.incr_lineno()
             cli_list = cp.parse(cli_text)
             if cli_list:
                 l.append(cli_list)
             elif cli_list == False:
                 rc = False
         err_buf.stop_tmp_lineno()
         # we can't proceed if there was a syntax error, but we
         # can ask the user to fix problems
         if not rc:
             return rc
         self.init_aux_lists()
         if l:
             for cli_list in processing_sort_cli(l):
                 if self.process(cli_list,update) == False:
                     rc = False
         if not self.drop_remaining():
             # this is tricky, we don't know what was removed!
             # it could happen that the user dropped a resource
             # which was running and therefore couldn't be removed
             rc = False
         self.recreate_obj_list()
         return rc
 
 cib_verify = "crm_verify -V -p"
 class CibObjectSetRaw(CibObjectSet):
     '''
     Edit or display one or more CIB objects (XML).
     '''
     def __init__(self, *args):
         CibObjectSet.__init__(self, *args)
         self.obj_list = cib_factory.mkobj_list("xml",*args)
     def repr(self):
         "Return a string containing xml of all objects."
         doc = cib_factory.objlist2doc(self.obj_list)
         s = doc.toprettyxml(user_prefs.xmlindent)
         doc.unlink()
         return s
     def repr_configure(self):
         '''
         Return a string containing xml of configure and its
         children.
         '''
         doc = cib_factory.objlist2doc(self.obj_list)
         conf_node = doc.getElementsByTagName("configuration")[0]
         s = conf_node.toprettyxml(user_prefs.xmlindent)
         doc.unlink()
         return s
     def process(self, node, update = False):
         if not cib_factory.is_cib_sane():
             return False
         myobj = obj = self.lookup(node.tagName,node.getAttribute("id"))
         if update and not obj:
             obj = cib_factory.find_object_for_node(node)
         if obj:
             rc = cib_factory.update_from_node(obj,node)
             if myobj:
                 self.remove_objs.remove(obj)
         else:
             new_obj = cib_factory.create_from_node(node)
             rc = new_obj != None
             if rc:
                 self.add_objs.append(new_obj)
         return rc
     def save(self, s, update = False):
         try:
             doc = xml.dom.minidom.parseString(s)
         except xml.parsers.expat.ExpatError,msg:
             cib_parse_err(msg,s)
             return False
         rc = True
         sanitize_cib(doc)
         show_unrecognized_elems(doc)
         newnodes = get_interesting_nodes(doc,[])
         self.init_aux_lists()
         if newnodes:
             for node in processing_sort(newnodes):
                 if not self.process(node,update):
                     rc = False
         if not self.drop_remaining():
             rc = False
         doc.unlink()
         self.recreate_obj_list()
         return rc
     def verify(self):
         if not self.obj_list:
             return True
         cli_display.set_no_pretty()
         rc = pipe_string(cib_verify,self.repr())
         cli_display.reset_no_pretty()
         return rc in (0,1)
     def ptest(self, nograph, scores, verbosity):
         if not cib_factory.is_cib_sane():
             return False
         ptest = "ptest -X -%s" % verbosity.upper()
         if scores:
             ptest = "%s -s" % ptest
         if user_prefs.dotty and not nograph:
             fd,tmpfile = mkstemp()
             ptest = "%s -D %s" % (ptest,tmpfile)
         else:
             tmpfile = None
         doc = cib_factory.objlist2doc(self.obj_list)
         cib = doc.childNodes[0]
         status = cib_status.get_status()
         if not status:
             common_err("no status section found")
             return False
         cib.appendChild(doc.importNode(status,1))
         pipe_string(ptest,doc.toprettyxml())
         doc.unlink()
         if tmpfile:
             p = subprocess.Popen("%s %s" % (user_prefs.dotty,tmpfile), shell=True, bufsize=0, stdin=None, stdout=None, stderr=None, close_fds=True)
             common_info("starting %s to show transition graph"%user_prefs.dotty)
             vars.tmpfiles.append(tmpfile)
         else:
             if not nograph:
                 common_info("install graphviz to see a transition graph")
         return True
 
 #
 # XML generate utilities
 #
 def set_id(node,oldnode,id_hint,id_required = True):
     '''
     Set the id attribute for the node.
     Procedure:
     - if the node already contains "id", keep it
     - if the old node contains "id", copy that
     - if neither is true, then create a new one using id_hint
       (exception: if not id_required, then no new id is generated)
     Finally, save the new id in id_store.
     '''
     old_id = None
     new_id = node.getAttribute("id")
     if oldnode and oldnode.getAttribute("id"):
         old_id = oldnode.getAttribute("id")
     if not new_id:
         new_id = old_id
     if not new_id:
         if id_required:
             new_id = id_store.new(node,id_hint)
     else:
         id_store.save(new_id)
     if new_id:
         node.setAttribute("id",new_id)
         if oldnode and old_id == new_id:
             set_id_used_attr(oldnode)
 
 def mkxmlsimple(e,oldnode,id_hint):
     '''
     Create an xml node from the (name,dict) pair. The name is the
     name of the element. The dict contains a set of attributes.
     '''
     node = cib_factory.createElement(e[0])
     for n,v in e[1]:
         if n == "$children": # this one's skipped
             continue
         if n == "operation":
             v = v.lower()
         if n.startswith('$'):
             n = n.lstrip('$')
         if (type(v) != type('') and type(v) != type(u'')) \
                 or v: # skip empty strings
             node.setAttribute(n,v)
     id_ref = node.getAttribute("id-ref")
     if id_ref:
         id_ref_2 = cib_factory.resolve_id_ref(e[0],id_ref)
         node.setAttribute("id-ref",id_ref_2)
     else:
         set_id(node,lookup_node(node,oldnode),id_hint)
     return node
 
 def mkxmlnvpairs(e,oldnode,id_hint):
     '''
     Create xml from the (name,dict) pair. The name is the name of
     the element. The dict contains a set of nvpairs. Stuff such
     as instance_attributes.
     NB: Other tags not containing nvpairs are fine if the dict is empty.
     '''
     node = cib_factory.createElement(e[0])
     match_node = lookup_node(node,oldnode)
     #if match_node:
         #print "found nvpairs set:",match_node.tagName,match_node.getAttribute("id")
     id_ref = find_value(e[1],"$id-ref")
     if id_ref:
         id_ref_2 = cib_factory.resolve_id_ref(e[0],id_ref)
         node.setAttribute("id-ref",id_ref_2)
         if e[0] != "operations":
             return node # id_ref is the only attribute (if not operations)
         e[1].remove(["$id-ref",id_ref])
     v = find_value(e[1],"$id")
     if v:
         node.setAttribute("id",v)
         e[1].remove(["$id",v])
     else:
         if e[0] == "operations": # operations don't need no id
             set_id(node,match_node,id_hint,id_required = False)
         else:
             set_id(node,match_node,id_hint)
     try:
         subpfx = vars.subpfx_list[e[0]]
     except: subpfx = ''
     subpfx = subpfx and "%s_%s" % (id_hint,subpfx) or id_hint
     nvpair_pfx = node.getAttribute("id") or subpfx
     for n,v in e[1]:
         nvpair = cib_factory.createElement("nvpair")
         node.appendChild(nvpair)
         nvpair.setAttribute("name",n)
         if v != None:
             nvpair.setAttribute("value",v)
         set_id(nvpair,lookup_node(nvpair,match_node),nvpair_pfx)
     return node
 
 def mkxmlop(e,oldnode,id_hint):
     '''
     Create an operation xml from the (name,dict) pair.
     '''
     node = cib_factory.createElement(e[0])
     inst_attr = []
     for n,v in e[1]:
         if n in olist(vars.req_op_attributes + vars.op_attributes):
             node.setAttribute(n,v)
         else:
             inst_attr.append([n,v])
     tmp = cib_factory.createElement("operations")
     oldops = lookup_node(tmp,oldnode) # first find old operations
     oldop = lookup_node(node,oldops)
     set_id(node,oldop,id_hint)
     if inst_attr:
         e = ["instance_attributes",inst_attr]
         nia = mkxmlnvpairs(e,oldop,node.getAttribute("id"))
         node.appendChild(nia)
     return node
 
 def mkxmldate(e,oldnode,id_hint):
     '''
     Create a date_expression xml from the (name,dict) pair.
     '''
     node = cib_factory.createElement(e[0])
     operation = find_value(e[1],"operation").lower()
     node.setAttribute("operation", operation)
     old_date = lookup_node(node,oldnode) # first find old date element
     set_id(node,old_date,id_hint)
     date_spec_attr = []
     for n,v in e[1]:
         if n in olist(vars.date_ops) or n == "operation":
             continue
         elif n in vars.in_range_attrs:
             node.setAttribute(n,v)
         else:
             date_spec_attr.append([n,v])
     if not date_spec_attr:
         return node
     elem = operation == "date_spec" and "date_spec" or "duration"
     tmp = cib_factory.createElement(elem)
     old_date_spec = lookup_node(tmp,old_date) # first find old date element
     set_id(tmp,old_date_spec,id_hint)
     for n,v in date_spec_attr:
         tmp.setAttribute(n,v)
     node.appendChild(tmp)
     return node
 
 def mkxmlrsc_set(e,oldnode,id_hint):
     '''
     Create a resource_set xml from the (name,dict) pair.
     '''
     node = cib_factory.createElement(e[0])
     old_rsc_set = lookup_node(node,oldnode) # first find old date element
     set_id(node,old_rsc_set,id_hint)
     for ref in e[1]:
         if ref[0] == "resource_ref":
             ref_node = cib_factory.createElement(ref[0])
             ref_node.setAttribute(ref[1][0],ref[1][1])
             node.appendChild(ref_node)
         elif ref[0] in ("sequential", "action", "role"):
             node.setAttribute(ref[0], ref[1])
     return node
 
 conv_list = {
     "params": "instance_attributes",
     "meta": "meta_attributes",
     "property": "cluster_property_set",
     "rsc_defaults": "meta_attributes",
     "op_defaults": "meta_attributes",
     "attributes": "instance_attributes",
     "utilization": "utilization",
     "operations": "operations",
     "op": "op",
 }
 def mkxmlnode(e,oldnode,id_hint):
     '''
     Create xml from the (name,dict) pair. The name is the name of
     the element. The dict contains either a set of nvpairs or a
     set of attributes. The id is either generated or copied if
     found in the provided xml. Stuff such as instance_attributes.
     '''
     if e[0] in conv_list:
         e[0] = conv_list[e[0]]
     if e[0] in ("instance_attributes","meta_attributes","operations","cluster_property_set","utilization"):
         return mkxmlnvpairs(e,oldnode,id_hint)
     elif e[0] == "op":
         return mkxmlop(e,oldnode,id_hint)
     elif e[0] == "date_expression":
         return mkxmldate(e,oldnode,id_hint)
     elif e[0] == "resource_set":
         return mkxmlrsc_set(e,oldnode,id_hint)
     else:
         return mkxmlsimple(e,oldnode,id_hint)
 
 def set_nvpair(set_node,name,value):
     n_id = set_node.getAttribute("id")
     for c in set_node.childNodes:
         if is_element(c) and c.getAttribute("name") == name:
             c.setAttribute("value",value)
             return
     np = cib_factory.createElement("nvpair")
     np.setAttribute("name",name)
     np.setAttribute("value",value)
     new_id = id_store.new(np,n_id)
     np.setAttribute("id",new_id)
     set_node.appendChild(np)
 
 #
 # cib element classes (CibObject the parent class)
 #
 class CibObject(object):
     '''
     The top level object of the CIB. Resources and constraints.
     '''
     state_fmt = "%16s %-8s%-8s%-8s%-8s%-8s%-4s"
     set_names = {}
     def __init__(self,xml_obj_type,obj_id = None):
         if not xml_obj_type in cib_object_map:
             unsupported_err(xml_obj_type)
             return
         self.obj_type = cib_object_map[xml_obj_type][0]
         self.parent_type = cib_object_map[xml_obj_type][2]
         self.xml_obj_type = xml_obj_type
         self.origin = "" # where did it originally come from?
         self.nocli = False # we don't support this one
         self.nocli_warn = True # don't issue warnings all the time
         self.updated = False # was the object updated
         self.invalid = False # the object has been invalidated (removed)
         self.moved = False # the object has been moved (from/to a container)
         self.recreate = False # constraints to be recreated
-        self.comment = '' # comment as text
         self.parent = None # object superior (group/clone/ms)
         self.children = [] # objects inferior
         if obj_id:
             if not self.mknode(obj_id):
                 self = None # won't do :(
         else:
             self.obj_id = None
             self.node = None
     def dump_state(self):
         'Print object status'
         print self.state_fmt % \
             (self.obj_id,self.origin,self.updated,self.moved,self.invalid, \
             self.parent and self.parent.obj_id or "", \
             len(self.children))
     def repr_cli_xml(self,node,format):
         h = cli_display.keyword("xml")
         l = node.toprettyxml('\t').split('\n')
         l = [x for x in l if x] # drop empty lines
         if format > 0:
             return "%s %s" % (h,' \\\n'.join(l))
         else:
             return "%s %s" % (h,''.join(l))
     def repr_cli(self,node = None,format = 1):
         '''
         CLI representation for the node.
         repr_cli_head and repr_cli_child in subclasess.
         '''
         if not node:
             node = self.node
         if self.nocli:
             return self.repr_cli_xml(node,format)
         l = []
         head_s = self.repr_cli_head(node)
         if not head_s: # everybody must have a head
             return None
+        comments = []
         l.append(head_s)
         cli_add_description(node,l)
         for c in node.childNodes:
+            if is_comment(c):
+                comments.append(c.data)
+                continue
             if not is_element(c):
                 continue
             s = self.repr_cli_child(c,format)
             if s:
                 l.append(s)
-        return self.cli_format(l,format)
+        return self.cli_format(l,comments,format)
     def repr_cli_child(self,c,format):
         if c.tagName in self.set_names:
             return "%s %s" % \
                 (cli_display.keyword(self.set_names[c.tagName]), \
                 cli_pairs(nvpairs2list(c)))
     def cli2node(self,cli,oldnode = None):
         '''
         Convert CLI representation to a DOM node.
         Defined in subclasses.
         '''
         cli_list = mk_cli_list(cli)
         if not cli_list:
             return None
         if not oldnode:
             if self.obj_type == "property":
                 oldnode = cib_factory.topnode[cib_object_map[self.xml_obj_type][2]]
             else:
                 oldnode = self.node
-        return self.cli_list2node(cli_list,oldnode)
-    def cli_format(self,l,format):
+        comments = get_comments(cli_list)
+        node = self.cli_list2node(cli_list,oldnode)
+        if comments and node:
+            stuff_comments(node,comments)
+        return node
+    def cli_format(self,l,comments,format):
         '''
         Format and add comment (if any).
         '''
         s = cli_format(l,format)
-        return (self.comment and format >=0) and '\n'.join([self.comment,s]) or s
-    def set_comment(self,l):
-        s = '\n'.join(l)
-        if self.comment != s:
-            self.comment = s
-            self.modified = True
-    def pull_comments(self):
+        cs = '\n'.join(comments)
+        return (comments and format >=0) and '\n'.join([cs,s]) or s
+    def move_comments(self):
         '''
-        Collect comments from within this node.  Remove them from
-        the parent and stuff them in self.comments as an array.
+        Move comments to the top of the node.
         '''
         l = []
-        cnodes = [x for x in self.node.childNodes if is_comment(x)]
-        for n in cnodes:
-            l.append(n.data)
-            n.parentNode.removeChild(n)
-        # convert comments from XML node to text. Multiple comments
-        # are concatenated with '\n'.
-        if not l:
-            self.comment = ''
-            return
-        s = '\n'.join(l)
-        l = s.split('\n')
-        for i in range(len(l)):
-            if not l[i].startswith('#'):
-                l[i] = '#%s' % l[i]
-        self.comment = '\n'.join(l)
+        firstelem = None
+        for n in self.node.childNodes:
+            if is_comment(n):
+                if firstelem:
+                    l.append(n)
+            else:
+                if not firstelem and is_element(n):
+                    firstelem = n
+        for comm_node in l:
+            common_debug("move comm %s" % comm_node.toprettyxml())
+            self.node.insertBefore(comm_node, firstelem)
+        common_debug("obj %s node: %s" % (self.obj_id,self.node.toprettyxml()))
     def mknode(self,obj_id):
         if not cib_factory.is_cib_sane():
             return False
         if id_store.id_in_use(obj_id):
             return False
         if self.xml_obj_type in vars.defaults_tags:
             tag = "meta_attributes"
         else:
             tag = self.xml_obj_type
         self.node = cib_factory.createElement(tag)
         self.obj_id = obj_id
         self.node.setAttribute("id",self.obj_id)
         self.origin = "user"
         return True
     def can_be_renamed(self):
         '''
         Return False if this object can't be renamed.
         '''
         if is_rsc_running(self.obj_id):
             common_err("cannot rename a running resource (%s)" % self.obj_id)
             return False
         if not is_live_cib() and self.node.tagName == "node":
             common_err("cannot rename nodes")
             return False
         return True
     def attr_exists(self,attr):
         if not attr in self.node.attributes.keys():
             no_attribute_err(attr,self.obj_id)
             return False
         return True
     def cli_use_validate(self):
         '''
         Check validity of the object, as we know it. It may
         happen that we don't recognize a construct, but that the
         object is still valid for the CRM. In that case, the
         object is marked as "CLI read only", i.e. we will neither
         convert it to CLI nor try to edit it in that format.
 
         The validation procedure:
         we convert xml to cli and then back to xml. If the two
         xml representations match then we can understand the xml.
         '''
         if not self.node:
             return True
         if not self.attr_exists("id"):
             return False
         cli_display.set_no_pretty()
-        cli_text = self.repr_cli(format = -1)
+        cli_text = self.repr_cli(format = 0)
         cli_display.reset_no_pretty()
         if not cli_text:
             return False
+        common_debug("clitext: %s" % cli_text)
         xml2 = self.cli2node(cli_text)
         if not xml2:
             return False
         rc = xml_cmp(self.node, xml2, show = True)
         xml2.unlink()
         return rc
     def check_sanity(self):
         '''
         Right now, this is only for primitives.
         '''
         return 0
     def matchcli(self,cli_list):
         head = cli_list[0]
         return self.obj_type == head[0] \
             and self.obj_id == find_value(head[1],"id")
     def match(self,xml_obj_type,obj_id):
         return self.xml_obj_type == xml_obj_type and self.obj_id == obj_id
     def obj_string(self):
         return "%s:%s" % (self.obj_type,self.obj_id)
     def reset_updated(self):
         self.updated = False
         self.moved = False
         self.recreate = False
         for child in self.children:
             child.reset_updated()
     def propagate_updated(self):
         if self.parent:
             self.parent.updated = self.updated
             self.parent.propagate_updated()
     def top_parent(self):
         '''Return the top parent or self'''
         if self.parent:
             return self.parent.top_parent()
         else:
             return self
     def find_child_in_node(self,child):
         for c in self.node.childNodes:
             if not is_element(c):
                 continue
             if c.tagName == child.obj_type and \
                     c.getAttribute("id") == child.obj_id:
                 return c
         return None
     def filter(self,*args):
         "Filter objects."
         if not args:
             return True
         if args[0] == "NOOBJ":
             return False
         if args[0] == "changed":
             return self.updated or self.origin == "user"
         return self.obj_id in args
 
 def mk_cli_list(cli):
     'Sometimes we get a string and sometimes a list.'
     if type(cli) == type('') or type(cli) == type(u''):
-        return CliParser().parse(cli)
+        cp = CliParser()
+        # what follows looks strange, but the last string actually matters
+        # the previous ones may be comments and are collected by the parser
+        for s in lines2cli(cli):
+            cli_list = cp.parse(s)
+        return cli_list
     else:
         return cli
 
 class CibNode(CibObject):
     '''
     Node and node's attributes.
     '''
     set_names = {
         "instance_attributes": "attributes",
         "utilization": "utilization",
     }
     def repr_cli_head(self,node):
         obj_type = vars.cib_cli_map[node.tagName]
         node_id = node.getAttribute("id")
         uname = node.getAttribute("uname")
         s = cli_display.keyword(obj_type)
         if node_id != uname:
             s = '%s $id="%s"' % (s, node_id)
         s = '%s %s' % (s, cli_display.id(uname))
         type = node.getAttribute("type")
         if type != vars.node_default_type:
             s = '%s:%s' % (s, type)
         return s
     def cli_list2node(self,cli_list,oldnode):
         head = copy.copy(cli_list[0])
         head[0] = backtrans[head[0]]
         obj_id = find_value(head[1],"$id")
         if not obj_id:
             obj_id = find_value(head[1],"uname")
         if not obj_id:
             return None
         type = find_value(head[1],"type")
         if not type:
             type = vars.node_default_type
             head[1].append(["type",type])
         headnode = mkxmlsimple(head,cib_factory.topnode[cib_object_map[self.xml_obj_type][2]],'node')
         id_hint = headnode.getAttribute("id")
         for e in cli_list[1:]:
             n = mkxmlnode(e,oldnode,id_hint)
             headnode.appendChild(n)
         remove_id_used_attributes(cib_factory.topnode[cib_object_map[self.xml_obj_type][2]])
         return headnode
 
 class CibPrimitive(CibObject):
     '''
     Primitives.
     '''
     set_names = {
         "instance_attributes": "params",
         "meta_attributes": "meta",
         "utilization": "utilization",
     }
     def repr_cli_head(self,node):
         obj_type = vars.cib_cli_map[node.tagName]
         node_id = node.getAttribute("id")
         ra_type = node.getAttribute("type")
         ra_class = node.getAttribute("class")
         ra_provider = node.getAttribute("provider")
         s1 = s2 = ''
         if ra_class:
             s1 = "%s:"%ra_class
         if ra_provider:
             s2 = "%s:"%ra_provider
         s = cli_display.keyword(obj_type)
         id = cli_display.id(node_id)
         return "%s %s %s" % (s, id, ''.join((s1,s2,ra_type)))
     def repr_cli_child(self,c,format):
         if c.tagName in self.set_names:
             return "%s %s" % \
                 (cli_display.keyword(self.set_names[c.tagName]), \
                 cli_pairs(nvpairs2list(c)))
         elif c.tagName == "operations":
             return cli_operations(c,format)
     def cli_list2node(self,cli_list,oldnode):
         '''
         Convert a CLI description to DOM node.
         Try to preserve as many ids as possible in case there's
         an old XML version.
         '''
         head = copy.copy(cli_list[0])
         head[0] = backtrans[head[0]]
         headnode = mkxmlsimple(head,oldnode,'rsc')
         id_hint = headnode.getAttribute("id")
         operations = None
         for e in cli_list[1:]:
             n = mkxmlnode(e,oldnode,id_hint)
             if keyword_cmp(e[0], "operations"):
                 operations = n
             if not keyword_cmp(e[0], "op"):
                 headnode.appendChild(n)
             else:
                 if not operations:
                     operations = mkxmlnode(["operations",{}],oldnode,id_hint)
                     headnode.appendChild(operations)
                 operations.appendChild(n)
         remove_id_used_attributes(oldnode)
         return headnode
     def add_operation(self,cli_list):
         # check if there is already an op with the same interval
+        comments = get_comments(cli_list)
         head = copy.copy(cli_list[0])
         name = find_value(head[1], "name")
         interval = find_value(head[1], "interval")
         if find_operation(self.node,name,interval):
             common_err("%s already has a %s op with interval %s" % \
                 (self.obj_id, name, interval))
             return None
         # drop the rsc attribute
         head[1].remove(["rsc",self.obj_id])
         # create an xml node
         mon_node = mkxmlsimple(head, None, self.obj_id)
         # get the place to append it to
         try:
             op_node = self.node.getElementsByTagName("operations")[0]
         except:
             op_node = cib_factory.createElement("operations")
             self.node.appendChild(op_node)
         op_node.appendChild(mon_node)
+        if comments and self.node:
+            stuff_comments(self.node,comments)
         # the resource is updated
         self.updated = True
         self.propagate_updated()
         return self
     def check_sanity(self):
         '''
         Check operation timeouts and if all required parameters
         are defined.
         '''
         if not self.node:  # eh?
             common_err("%s: no xml (strange)" % self.obj_id)
             return user_prefs.get_check_rc()
         ra_type = self.node.getAttribute("type")
         ra_class = self.node.getAttribute("class")
         ra_provider = self.node.getAttribute("provider")
         ra = RAInfo(ra_class,ra_type,ra_provider)
         if not ra.mk_ra_node():  # no RA found?
             ra.error("no such resource agent")
             return user_prefs.get_check_rc()
         params = []
         for c in self.node.childNodes:
             if not is_element(c):
                 continue
             if c.tagName == "instance_attributes":
                 params += nvpairs2list(c)
         rc1 = ra.sanity_check_params(self.obj_id, params)
         actions = {}
         for c in self.node.childNodes:
             if not is_element(c):
                 continue
             if c.tagName == "operations":
                 for c2 in c.childNodes:
                     if is_element(c2) and c2.tagName == "op":
                         op,pl = op2list(c2)
                         if op:
                             actions[op] = pl
         default_timeout = get_default("default-action-timeout")
         rc2 = ra.sanity_check_ops(self.obj_id, actions, default_timeout)
         return rc1 | rc2
 
 class CibContainer(CibObject):
     '''
     Groups and clones and ms.
     '''
     set_names = {
         "instance_attributes": "params",
         "meta_attributes": "meta",
     }
     def repr_cli_head(self,node):
         try:
             obj_type = vars.cib_cli_map[node.tagName]
         except:
             unsupported_err(node.tagName)
             return None
         node_id = node.getAttribute("id")
         children = []
         for c in node.childNodes:
             if not is_element(c):
                 continue
             if (obj_type == "group" and is_primitive(c)) or \
                     is_child_rsc(c):
                 children.append(cli_display.rscref(c.getAttribute("id")))
             elif obj_type in vars.clonems_tags and is_child_rsc(c):
                 children.append(cli_display.rscref(c.getAttribute("id")))
         s = cli_display.keyword(obj_type)
         id = cli_display.id(node_id)
         return "%s %s %s" % (s, id, ' '.join(children))
     def cli_list2node(self,cli_list,oldnode):
         head = copy.copy(cli_list[0])
         head[0] = backtrans[head[0]]
         headnode = mkxmlsimple(head,oldnode,'grp')
         id_hint = headnode.getAttribute("id")
         for e in cli_list[1:]:
             n = mkxmlnode(e,oldnode,id_hint)
             headnode.appendChild(n)
         v = find_value(head[1],"$children")
         if v:
             for child_id in v:
                 obj = cib_factory.find_object(child_id)
                 if obj:
                     n = obj.node.cloneNode(1)
                     headnode.appendChild(n)
                 else:
                     no_object_err(child_id)
         remove_id_used_attributes(oldnode)
         return headnode
 
 class CibLocation(CibObject):
     '''
     Location constraint.
     '''
     def repr_cli_head(self,node):
         obj_type = vars.cib_cli_map[node.tagName]
         node_id = node.getAttribute("id")
         rsc = cli_display.rscref(node.getAttribute("rsc"))
         s = cli_display.keyword(obj_type)
         id = cli_display.id(node_id)
         s = "%s %s %s"%(s,id,rsc)
         pref_node = node.getAttribute("node")
         score = cli_display.score(get_score(node))
         if pref_node:
             return "%s %s %s" % (s,score,pref_node)
         else:
             return s
     def repr_cli_child(self,c,format):
         if c.tagName == "rule":
             return "%s %s" % \
                 (cli_display.keyword("rule"), cli_rule(c))
     def cli_list2node(self,cli_list,oldnode):
         head = copy.copy(cli_list[0])
         head[0] = backtrans[head[0]]
         headnode = mkxmlsimple(head,oldnode,'location')
         id_hint = headnode.getAttribute("id")
         oldrule = None
         for e in cli_list[1:]:
             if e[0] in ("expression","date_expression"):
                 n = mkxmlnode(e,oldrule,id_hint)
             else:
                 n = mkxmlnode(e,oldnode,id_hint)
             if keyword_cmp(e[0], "rule"):
                 add_missing_attr(n)
                 rule = n
                 headnode.appendChild(n)
                 oldrule = lookup_node(rule,oldnode,location_only=True)
             else:
                 rule.appendChild(n)
         remove_id_used_attributes(oldnode)
         return headnode
 
 class CibSimpleConstraint(CibObject):
     '''
     Colocation and order constraints.
     '''
     def repr_cli_head(self,node):
         obj_type = vars.cib_cli_map[node.tagName]
         node_id = node.getAttribute("id")
         s = cli_display.keyword(obj_type)
         id = cli_display.id(node_id)
         score = cli_display.score(get_score(node))
         if node.getElementsByTagName("resource_set"):
             col = rsc_set_constraint(node,obj_type)
         else:
             col = two_rsc_constraint(node,obj_type)
         if not col:
             return None
         symm = node.getAttribute("symmetrical")
         if symm:
             col.append("symmetrical=%s"%symm)
         return "%s %s %s %s" % (s,id,score,' '.join(col))
     def repr_cli_child(self,c,format):
         pass # no children here
     def cli_list2node(self,cli_list,oldnode):
         head = copy.copy(cli_list[0])
         head[0] = backtrans[head[0]]
         headnode = mkxmlsimple(head,oldnode,'')
         id_hint = headnode.getAttribute("id")
         for e in cli_list[1:]:
             # if more than one element, it's a resource set
             n = mkxmlnode(e,oldnode,id_hint)
             headnode.appendChild(n)
         remove_id_used_attributes(oldnode)
         return headnode
 
 class CibProperty(CibObject):
     '''
     Cluster properties.
     '''
     def repr_cli_head(self,node):
         return '%s $id="%s"' % \
             (cli_display.keyword(self.obj_type), node.getAttribute("id"))
     def repr_cli_child(self,c,format):
         name = c.getAttribute("name")
         if "value" in c.attributes.keys():
             value = c.getAttribute("value")
         else:
             value = None
         return nvpair_format(name,value)
     def cli_list2node(self,cli_list,oldnode):
         head = copy.copy(cli_list[0])
         head[0] = backtrans[head[0]]
         obj_id = find_value(head[1],"$id")
         if not obj_id:
             obj_id = cib_object_map[self.xml_obj_type][3]
         headnode = mkxmlnode(head,oldnode,obj_id)
         remove_id_used_attributes(oldnode)
         return headnode
     def matchcli(self,cli_list):
         head = cli_list[0]
         return self.obj_type == head[0] \
             and self.obj_id == find_value(head[1],"$id")
 #
 ################################################################
 
 #
 # cib factory
 #
 cib_piped = "cibadmin -p"
 
 def get_cib_default(property):
     if cib_factory.is_cib_sane():
         return cib_factory.get_property(property)
 def get_default(property):
     t = get_cib_default(property)
     if t:
         return t
     if not vars.pe_metadata:
         vars.pe_metadata = RAInfo("pengine","metadata")
     if not vars.pe_metadata:
         return 0
     return vars.pe_metadata.param_default(property)
 
 # xml -> cli translations (and classes)
 cib_object_map = {
     "node": ( "node", CibNode, "nodes" ),
     "primitive": ( "primitive", CibPrimitive, "resources" ),
     "group": ( "group", CibContainer, "resources" ),
     "clone": ( "clone", CibContainer, "resources" ),
     "master": ( "ms", CibContainer, "resources" ),
     "rsc_location": ( "location", CibLocation, "constraints" ),
     "rsc_colocation": ( "colocation", CibSimpleConstraint, "constraints" ),
     "rsc_order": ( "order", CibSimpleConstraint, "constraints" ),
     "cluster_property_set": ( "property", CibProperty, "crm_config", "cib-bootstrap-options" ),
     "rsc_defaults": ( "rsc_defaults", CibProperty, "rsc_defaults", "rsc-options" ),
     "op_defaults": ( "op_defaults", CibProperty, "op_defaults", "op-options" ),
 }
 backtrans = odict()  # generate a translation cli -> tag
 for key in cib_object_map:
     backtrans[cib_object_map[key][0]] = key
 cib_topnodes = []  # get a list of parents
 for key in cib_object_map:
     if not cib_object_map[key][2] in cib_topnodes:
         cib_topnodes.append(cib_object_map[key][2])
 
 cib_upgrade = "cibadmin --upgrade --force"
 class CibFactory(Singleton):
     '''
     Juggle with CIB objects.
     See check_structure below for details on the internal cib
     representation.
     '''
     shadowcmd = ">/dev/null </dev/null crm_shadow"
     def __init__(self):
         self.init_vars()
         self.regtest = options.regression_tests
         self.all_committed = True # has commit produced error
         self._no_constraint_rm_msg = False # internal (just not to produce silly messages)
         self.supported_cib_re = "^pacemaker-1[.][012]$"
     def is_cib_sane(self):
         if not self.doc:
             empty_cib_err()
             return False
         return True
     #
     # check internal structures
     #
     def check_topnode(self,obj):
         if not obj.node.parentNode.isSameNode(self.topnode[obj.parent_type]):
             common_err("object %s is not linked to %s"%(obj.obj_id,obj.parent_type))
     def check_parent(self,obj,parent):
         if not obj in parent.children:
             common_err("object %s does not reference its child %s"%(parent.obj_id,obj.obj_id))
             return False
         if not parent.node.isSameNode(obj.node.parentNode):
             common_err("object %s node is not a child of its parent %s, but %s:%s"%(obj.obj_id,parent.obj_id,obj.node.tagName,obj.node.getAttribute("id")))
             return False
     def check_structure(self):
         #print "Checking structure..."
         if not self.doc:
             empty_cib_err()
             return False
         rc = True
         for obj in self.cib_objects:
             #print "Checking %s... (%s)" % (obj.obj_id,obj.nocli)
             if obj.parent:
                 if self.check_parent(obj,obj.parent) == False:
                     rc = False
             else:
                 if self.check_topnode(obj) == False:
                     rc = False
             for child in obj.children:
                 if self.check_parent(child,child.parent) == False:
                     rc = False
         return rc
     def regression_testing(self,param):
         # provide some help for regression testing
         # in particular by trying to provide output which is
         # easier to predict
         if param == "off":
             self.regtest = False
         elif param == "on":
             self.regtest = True
         else:
             common_warn("bad parameter for regtest: %s" % param)
     def createElement(self,tag):
         if self.doc:
             return self.doc.createElement(tag)
         else:
             empty_cib_err()
     def createComment(self,s):
         if self.doc:
             return self.doc.createComment(s)
         else:
             empty_cib_err()
     def replaceNode(self,newnode,oldnode):
         if not self.doc:
             empty_cib_err()
             return None
         if newnode.ownerDocument != self.doc:
             newnode = self.doc.importNode(newnode,1)
         oldnode.parentNode.replaceChild(newnode,oldnode)
         return newnode
     def is_cib_supported(self,cib):
         'Do we support this CIB?'
         req = cib.getAttribute("crm_feature_set")
         validator = cib.getAttribute("validate-with")
         if validator and re.match(self.supported_cib_re,validator):
             return True
         cib_ver_unsupported_err(validator,req)
         return False
     def upgrade_cib_06to10(self,force = False):
         'Upgrade the CIB from 0.6 to 1.0.'
         if not self.doc:
             empty_cib_err()
             return False
         cib = self.doc.getElementsByTagName("cib")
         if not cib:
             common_err("CIB has no cib element")
             return False
         req = cib[0].getAttribute("crm_feature_set")
         validator = cib[0].getAttribute("validate-with")
         if force or not validator or re.match("0[.]6",validator):
             return ext_cmd(cib_upgrade) == 0
     def import_cib(self):
         'Parse the current CIB (from cibadmin -Q).'
         self.doc,cib = read_cib(cibdump2doc)
         if not self.doc:
             return False
         if not cib:
             common_err("CIB has no cib element")
             self.reset()
             return False
         if not self.is_cib_supported(cib):
             self.reset()
             return False
         for attr in cib.attributes.keys():
             self.cib_attrs[attr] = cib.getAttribute(attr)
         for t in cib_topnodes:
             self.topnode[t] = get_conf_elem(self.doc, t)
             if not self.topnode[t]:
                 self.topnode[t] = mk_topnode(self.doc, t)
             if not self.topnode[t]:
                 common_err("could not create %s node; out of memory?" % t)
                 self.reset()
                 return False
         return True
     #
     # create a doc from the list of objects
     # (used by CibObjectSetRaw)
     #
     def regtest_filter(self,cib):
         for attr in ("epoch","admin_epoch"):
             if cib.getAttribute(attr):
                 cib.setAttribute(attr,"0")
         for attr in ("cib-last-written",):
             if cib.getAttribute(attr):
                 cib.removeAttribute(attr)
     def set_cib_attributes(self,cib):
         for attr in self.cib_attrs:
             cib.setAttribute(attr,self.cib_attrs[attr])
         if self.regtest:
             self.regtest_filter(cib)
     def objlist2doc(self,obj_list,obj_filter = None):
         '''
         Return document containing objects in obj_list.
         Must remove all children from the object list, because
         printing xml of parents will include them.
         Optional filter to sieve objects.
         '''
         doc,cib,crm_config,rsc_defaults,op_defaults,nodes,resources,constraints = new_cib()
         # get only top parents for the objects in the list
         # e.g. if we get a primitive which is part of a clone,
         # then the clone gets in, not the primitive
         # dict will weed out duplicates
         d = {}
         for obj in obj_list:
             if obj_filter and not obj_filter(obj):
                 continue
             d[obj.top_parent()] = 1
         for obj in d:
             i_node = doc.importNode(obj.node,1)
-            add_comment(doc,i_node,obj.comment)
             if obj.parent_type == "nodes":
                 nodes.appendChild(i_node)
             elif obj.parent_type == "resources":
                 resources.appendChild(i_node)
             elif obj.parent_type == "constraints":
                 constraints.appendChild(i_node)
             elif obj.parent_type == "crm_config":
                 crm_config.appendChild(i_node)
             elif obj.parent_type == "rsc_defaults":
                 rsc_defaults.appendChild(i_node)
             elif obj.parent_type == "op_defaults":
                 op_defaults.appendChild(i_node)
         self.set_cib_attributes(cib)
         return doc
     #
     # commit changed objects to the CIB
     #
     def attr_match(self,c,a):
         'Does attribute match?'
         try: cib_attr = self.cib_attrs[a]
         except: cib_attr = None
         return c.getAttribute(a) == cib_attr
     def is_current_cib_equal(self, silent = False):
         if self.overwrite:
             return True
         doc,cib = read_cib(cibdump2doc)
         if not doc:
             return False
         if not cib:
             doc.unlink()
             return False
         rc = self.attr_match(cib,'epoch') and \
                 self.attr_match(cib,'admin_epoch')
         if not silent and not rc:
             common_warn("CIB changed in the meantime: won't touch it!")
         doc.unlink()
         return rc
     def state_header(self):
         'Print object status header'
         print CibObject.state_fmt % \
             ("","origin","updated","moved","invalid","parent","children")
     def showobjects(self):
         self.state_header()
         for obj in self.cib_objects:
             obj.dump_state()
         if self.remove_queue:
             print "Remove queue:"
             for obj in self.remove_queue:
                 obj.dump_state()
     def commit(self):
         'Commit the configuration to the CIB.'
         if not self.doc:
             empty_cib_err()
             return False
         # all_committed is updated in the invoked object methods
         self.all_committed = True
         cnt = self.commit_doc()
         if cnt:
             # reload the cib!
             self.reset()
             self.initialize()
         return self.all_committed
     def commit_doc(self):
         try:
             conf_node = self.doc.getElementsByTagName("configuration")[0]
         except:
             common_error("cannot find the configuration node")
             return False
         rc = pipe_string("%s -R" % cib_piped, conf_node.toxml())
         if rc != 0:
             update_err("cib",'-R',conf_node.toprettyxml())
             return False
         return True
     def mk_shadow(self):
         '''
         Create a temporary shadow for commit/apply.
         Unless the user's already working with a shadow CIB.
         '''
         # TODO: split CibShadow into ui and mgmt part, then reuse the mgmt
         if not is_live_cib():
             return True
         self.tmp_shadow = "__crmshell.%d" % os.getpid()
         if ext_cmd("%s -c %s" % (self.shadowcmd,self.tmp_shadow)) != 0:
             common_error("creating tmp shadow %s failed" % self.tmp_shadow)
             self.tmp_shadow = ""
             return False
         os.putenv(vars.shadow_envvar,self.tmp_shadow)
         return True
     def rm_shadow(self):
         '''
         Remove the temporary shadow.
         Unless the user's already working with a shadow CIB.
         '''
         if not is_live_cib() or not self.tmp_shadow:
             return
         if ext_cmd("%s -D '%s' --force" % (self.shadowcmd,self.tmp_shadow)) != 0:
             common_error("deleting tmp shadow %s failed" % self.tmp_shadow)
         self.tmp_shadow = ""
         os.unsetenv(vars.shadow_envvar)
     def apply_shadow(self):
         '''
         Commit the temporary shadow.
         Unless the user's already working with a shadow CIB.
         '''
         if not is_live_cib():
             return True
         if not self.tmp_shadow:
             common_error("cannot commit no shadow")
             return False
         if ext_cmd("%s -C '%s' --force" % (self.shadowcmd,self.tmp_shadow)) != 0:
             common_error("committing tmp shadow %s failed" % self.tmp_shadow)
             return False
         return True
     #
     # initialize cib_objects from CIB
     #
     def save_node(self,node,pnode = None):
         '''
         Need pnode (parent node) acrobacy because cluster
         properties and rsc/op_defaults hold stuff in a
         meta_attributes child.
         '''
         if not pnode:
             pnode = node
         obj = cib_object_map[pnode.tagName][1](pnode.tagName)
         obj.origin = "cib"
         obj.obj_id = node.getAttribute("id")
         obj.node = node
         self.cib_objects.append(obj)
     def populate(self):
         "Walk the cib and collect cib objects."
         all_nodes = get_interesting_nodes(self.doc,[])
         if not all_nodes:
             return
         for node in processing_sort(all_nodes):
             if is_defaults(node):
                 for c in node.childNodes:
                     if not is_element(c) or c.tagName != "meta_attributes":
                         continue
                     self.save_node(c,node)
             else:
                 self.save_node(node)
-        #for obj in self.cib_objects:
-        #    obj.pull_comments()
+        for obj in self.cib_objects:
+            obj.move_comments()
+            fix_comments(obj.node)
         for obj in self.cib_objects:
             if not obj.cli_use_validate():
                 obj.nocli = True
                 obj.nocli_warn = False
                 obj_cli_warn(obj.obj_id)
         for obj in self.cib_objects:
             self.update_links(obj)
     def initialize(self):
         if self.doc:
             return True
         if not self.import_cib():
             return False
         sanitize_cib(self.doc)
         show_unrecognized_elems(self.doc)
         self.populate()
         return self.check_structure()
     def init_vars(self):
         self.doc = None  # the cib
         self.topnode = {}
         for t in cib_topnodes:
             self.topnode[t] = None
         self.cib_attrs = {} # cib version dictionary
         self.cib_objects = [] # a list of cib objects
         self.remove_queue = [] # a list of cib objects to be removed
         self.overwrite = False # update cib unconditionally
     def reset(self):
         if not self.doc:
             return
         self.doc.unlink()
         self.init_vars()
         id_store.clear()
     def find_object(self,obj_id):
         "Find an object for id."
         for obj in self.cib_objects:
             if obj.obj_id == obj_id:
                 return obj
         return None
     #
     # tab completion functions
     #
     def id_list(self):
         "List of ids (for completion)."
         return [x.obj_id for x in self.cib_objects]
     def prim_id_list(self):
         "List of primitives ids (for group completion)."
         return [x.obj_id for x in self.cib_objects if x.obj_type == "primitive"]
     def children_id_list(self):
         "List of child ids (for clone/master completion)."
         return [x.obj_id for x in self.cib_objects if x.obj_type in vars.children_tags]
     def rsc_id_list(self):
         "List of resource ids (for constraint completion)."
         return [x.obj_id for x in self.cib_objects \
             if x.obj_type in vars.resource_tags and not x.parent]
     def node_id_list(self):
         "List of node ids."
         return [x.obj_id for x in self.cib_objects \
             if x.obj_type == "node"]
     def f_prim_id_list(self):
         "List of possible primitives ids (for group completion)."
         return [x.obj_id for x in self.cib_objects \
             if x.obj_type == "primitive" and not x.parent]
     def f_children_id_list(self):
         "List of possible child ids (for clone/master completion)."
         return [x.obj_id for x in self.cib_objects \
             if x.obj_type in vars.children_tags and not x.parent]
     #
     # a few helper functions
     #
     def find_object_for_node(self,node):
         "Find an object which matches a dom node."
         for obj in self.cib_objects:
             if node.getAttribute("id") == obj.obj_id:
                 return obj
         return None
     def find_object_for_cli(self,cli_list):
         "Find an object which matches the cli list."
         for obj in self.cib_objects:
             if obj.matchcli(cli_list):
                 return obj
         return None
 
     def resolve_id_ref(self,attr_list_type,id_ref):
         '''
         User is allowed to specify id_ref either as a an object
         id or as attributes id. Here we try to figure out which
         one, i.e. if the former is the case to find the right
         id to reference.
         '''
         obj= self.find_object(id_ref)
         if obj:
             node_l = obj.node.getElementsByTagName(attr_list_type)
             if node_l:
                 if len(node_l) > 1:
                     common_warn("%s contains more than one %s, using first" % \
                         (obj.obj_id,attr_list_type))
                 id = node_l[0].getAttribute("id")
                 if not id:
                     common_err("%s reference not found" % id_ref)
                     return id_ref # hope that user will fix that
                 return id
         # verify if id_ref exists
         node_l = self.doc.getElementsByTagName(attr_list_type)
         for node in node_l:
             if node.getAttribute("id") == id_ref:
                 return id_ref
         common_err("%s reference not found" % id_ref)
         return id_ref # hope that user will fix that
     def get_property(self,property):
         '''
         Get the value of the given cluster property.
         '''
         for obj in self.cib_objects:
             if obj.obj_type == "property" and obj.node:
                 pl = nvpairs2list(obj.node)
                 v = find_value(pl, property)
                 if v:
                     return v
         return None
     def new_object(self,obj_type,obj_id):
         "Create a new object of type obj_type."
         if id_store.id_in_use(obj_id):
             return None
         for xml_obj_type,v in cib_object_map.items():
             if v[0] == obj_type:
                 obj = v[1](xml_obj_type,obj_id)
                 if obj.obj_id:
                     return obj
                 else:
                     return None
         return None
     def mkobj_list(self,mode,*args):
         obj_list = []
         for obj in self.cib_objects:
             f = lambda: obj.filter(*args)
             if not f():
                 continue
             if mode == "cli" and obj.nocli and obj.nocli_warn:
                 obj.nocli_warn = False
                 obj_cli_warn(obj.obj_id)
             obj_list.append(obj)
         return obj_list
     def has_cib_changed(self):
         return self.mkobj_list("xml","changed") or self.remove_queue
     def verify_constraints(self,node):
         '''
         Check if all resources referenced in a constraint exist
         '''
         rc = True
         constraint_id = node.getAttribute("id")
         for obj_id in referenced_resources(node):
             if not self.find_object(obj_id):
                 constraint_norefobj_err(constraint_id,obj_id)
                 rc = False
         return rc
     def verify_rsc_children(self,node):
         '''
         Check prerequisites:
           a) all children must exist
           b) no child may have other parent than me
           (or should we steal children?)
           c) there may not be duplicate children
         '''
         obj_id = node.getAttribute("id")
         if not obj_id:
             common_err("element %s has no id" % node.tagName)
             return False
         try:
             obj_type = cib_object_map[node.tagName][0]
         except:
             common_err("element %s (%s) not recognized"%(node.tagName,obj_id))
             return False
         c_ids = get_rsc_children_ids(node)
         if not c_ids:
             return True
         rc = True
         c_dict = {}
         for child_id in c_ids:
             if not self.verify_child(child_id,obj_type,obj_id):
                 rc = False
             if child_id in c_dict:
                 common_err("in group %s child %s listed more than once"%(obj_id,child_id))
                 rc = False
             c_dict[child_id] = 1
         return rc
     def verify_child(self,child_id,obj_type,obj_id):
         'Check if child exists and obj_id is (or may become) its parent.'
         child = self.find_object(child_id)
         if not child:
             no_object_err(child_id)
             return False
         if child.parent and child.parent.obj_id != obj_id:
             common_err("%s already in use at %s"%(child_id,child.parent.obj_id))
             return False
         if obj_type == "group" and child.obj_type != "primitive":
             common_err("a group may contain only primitives; %s is %s"%(child_id,child.obj_type))
             return False
         if not child.obj_type in vars.children_tags:
             common_err("%s may contain a primitive or a group; %s is %s"%(obj_type,child_id,child.obj_type))
             return False
         return True
     def verify_element(self,node):
         '''
         Can we create this object given its CLI representation.
         This is not about syntax, we're past that, but about
         semantics.
         Right now we check if the children, if any, are fit for
         the parent. And if this is a constraint, if all
         referenced resources are present.
         '''
         rc = True
         if not self.verify_rsc_children(node):
             rc = False
         if not self.verify_constraints(node):
             rc = False
         return rc
     def create_object(self,*args):
         return self.create_from_cli(CliParser().parse(list(args))) != None
     def set_property_cli(self,cli_list):
+        comments = get_comments(cli_list)
         head_pl = cli_list[0]
         obj_type = head_pl[0].lower()
         pset_id = find_value(head_pl[1],"$id")
         if pset_id:
             head_pl[1].remove(["$id",pset_id])
         else:
             pset_id = cib_object_map[backtrans[obj_type]][3]
         obj = self.find_object(pset_id)
         if not obj:
             if not is_id_valid(pset_id):
                 invalid_id_err(pset_id)
                 return None
             obj = self.new_object(obj_type,pset_id)
             if not obj:
                 return None
             self.topnode[obj.parent_type].appendChild(obj.node)
             obj.origin = "user"
             self.cib_objects.append(obj)
         for n,v in head_pl[1]:
             set_nvpair(obj.node,n,v)
+        if comments and obj.node:
+            stuff_comments(obj.node,comments)
         obj.updated = True
         return obj
     def add_op(self,cli_list):
         '''Add an op to a primitive.'''
         head = cli_list[0]
         # does the referenced primitive exist
         rsc_id = find_value(head[1],"rsc")
         rsc_obj = self.find_object(rsc_id)
         if not rsc_obj:
             no_object_err(rsc_id)
             return None
         if rsc_obj.obj_type != "primitive":
             common_err("%s is not a primitive" % rsc_id)
             return None
         return rsc_obj.add_operation(cli_list)
     def create_from_cli(self,cli):
         'Create a new cib object from the cli representation.'
         cli_list = mk_cli_list(cli)
         if not cli_list:
             return None
         head = cli_list[0]
         obj_type = head[0].lower()
         obj_id = find_value(head[1],"id")
         if obj_id and not is_id_valid(obj_id):
             invalid_id_err(obj_id)
             return None
         if len(cli_list) >= 2 and cli_list[1][0] == "raw":
             doc = xml.dom.minidom.parseString(cli_list[1][1])
             return self.create_from_node(doc.childNodes[0])
         if obj_type in olist(vars.nvset_cli_names):
             return self.set_property_cli(cli_list)
         if obj_type == "op":
             return self.add_op(cli_list)
         if obj_type == "node":
             obj = self.find_object(obj_id)
             # make an exception and allow updating nodes
             if obj:
                 self.merge_from_cli(obj,cli_list)
                 return obj
         obj = self.new_object(obj_type,obj_id)
         if not obj:
             return None
         node = obj.cli2node(cli_list)
         return self.add_element(obj, node)
     def update_from_cli(self,obj,cli_list):
         'Update element from the cli intermediate.'
         id_store.remove_xml(obj.node)
         if len(cli_list) >= 2 and cli_list[1][0] == "raw":
             doc = xml.dom.minidom.parseString(cli_list[1][1])
             id_store.store_xml(doc.childNodes[0])
             return self.update_element(obj,doc.childNodes[0])
         return self.update_element(obj,obj.cli2node(cli_list))
     def update_from_node(self,obj,node):
         'Update element from a doc node.'
         id_store.replace_xml(obj.node,node)
         return self.update_element(obj,node)
     def update_element(self,obj,newnode):
         'Update element from a doc node.'
         if not newnode:
             return False
         if not self.is_cib_sane():
             id_store.replace_xml(newnode,obj.node)
             return False
         oldnode = obj.node
         if xml_cmp(oldnode,newnode):
             newnode.unlink()
             return True # the new and the old versions are equal
         obj.node = newnode
         if not self.test_element(obj,newnode):
             id_store.replace_xml(newnode,oldnode)
             obj.node = oldnode
             newnode.unlink()
             return False
         obj.node = self.replaceNode(newnode,oldnode)
         obj.nocli = False # try again after update
         self.adjust_children(obj)
         if not obj.cli_use_validate():
             obj.nocli_warn = True
             obj.nocli = True
         oldnode.unlink()
         obj.updated = True
         obj.propagate_updated()
         return True
     def merge_from_cli(self,obj,cli_list):
         node = obj.cli2node(cli_list)
         if not node:
             return
         rc = merge_nodes(obj.node, node)
         if rc:
             obj.updated = True
             obj.propagate_updated()
     def update_moved(self,obj):
         'Updated the moved flag. Mark affected constraints.'
         obj.moved = not obj.moved
         if obj.moved:
             for c_obj in self.related_constraints(obj):
                 c_obj.recreate = True
     def adjust_children(self,obj):
         '''
         All stuff children related: manage the nodes of children,
         update the list of children for the parent, update
         parents in the children.
         '''
         new_children_ids = get_rsc_children_ids(obj.node)
         if not new_children_ids:
             return
         old_children = obj.children
         obj.children = [self.find_object(x) for x in new_children_ids]
         self._relink_orphans_to_top(old_children,obj.children)
         self._update_children(obj)
     def _relink_child_to_top(self,obj):
         'Relink a child to the top node.'
         obj.node.parentNode.removeChild(obj.node)
         self.topnode[obj.parent_type].appendChild(obj.node)
         if obj.origin == "cib":
             self.update_moved(obj)
         obj.parent = None
     def _update_children(self,obj):
         '''For composite objects: update all children nodes.
         '''
         # unlink all and find them in the new node
         for child in obj.children:
             oldnode = child.node
             child.node = obj.find_child_in_node(child)
             if child.children: # and children of children
                 self._update_children(child)
             rmnode(oldnode)
             if not child.parent and child.origin == "cib":
                 self.update_moved(child)
             if child.parent and child.parent != obj:
                 child.parent.updated = True # the other parent updated
             child.parent = obj
     def _relink_orphans_to_top(self,old_children,new_children):
         "New orphans move to the top level for the object type."
         for child in old_children:
             if child not in new_children:
                 self._relink_child_to_top(child)
     def test_element(self,obj,node):
         if not node.getAttribute("id"):
             return False
         if not self.verify_element(node):
             return False
         if user_prefs.is_check_always() \
                 and obj.check_sanity() > 1:
             return False
         return True
     def update_links(self,obj):
         '''
         Update the structure links for the object (obj.children,
         obj.parent). Update also the dom nodes, if necessary.
         '''
         obj.children = []
         if obj.obj_type not in vars.container_tags:
             return
         for c in obj.node.childNodes:
             if is_child_rsc(c):
                 child = self.find_object_for_node(c)
                 if not child:
                     missing_obj_err(c)
                     continue
                 child.parent = obj
                 obj.children.append(child)
                 if not c.isSameNode(child.node):
                     rmnode(child.node)
                     child.node = c
     def add_element(self,obj,node):
         obj.node = node
         obj.obj_id = node.getAttribute("id")
         if not self.test_element(obj, node):
             id_store.remove_xml(node)
             node.unlink()
             return None
         common_debug("append child %s to %s" % \
             (obj.obj_id,self.topnode[obj.parent_type].tagName))
         self.topnode[obj.parent_type].appendChild(node)
         self.adjust_children(obj)
         self.redirect_children_constraints(obj)
         if not obj.cli_use_validate():
             self.nocli_warn = True
             obj.nocli = True
         self.update_links(obj)
         obj.origin = "user"
         self.cib_objects.append(obj)
         return obj
     def create_from_node(self,node):
         'Create a new cib object from a document node.'
         if not node:
             return None
         try:
             obj_type = cib_object_map[node.tagName][0]
         except:
             return None
         if is_defaults(node):
             node = get_rscop_defaults_meta_node(node)
             if not node:
                 return None
         if node.ownerDocument != self.doc:
             node = self.doc.importNode(node,1)
         obj = self.new_object(obj_type, node.getAttribute("id"))
         if not obj:
             return None
         if not id_store.store_xml(node):
             return None
         return self.add_element(obj, node)
     def cib_objects_string(self, obj_list = None):
         l = []
         if not obj_list:
             obj_list = self.cib_objects
         for obj in obj_list:
             l.append(obj.obj_string())
         return ' '.join(l)
     def _remove_obj(self,obj):
         "Remove a cib object and its children."
         # remove children first
         # can't remove them here from obj.children!
         common_debug("remove object %s" % obj.obj_string())
         for child in obj.children:
             #self._remove_obj(child)
             # just relink, don't remove children
             self._relink_child_to_top(child)
         if obj.parent: # remove obj from its parent, if any
             obj.parent.children.remove(obj)
         id_store.remove_xml(obj.node)
         rmnode(obj.node)
         obj.invalid = True
         self.add_to_remove_queue(obj)
         self.cib_objects.remove(obj)
         for c_obj in self.related_constraints(obj):
             if is_simpleconstraint(c_obj.node) and obj.children:
                 # the first child inherits constraints
                 rename_rscref(c_obj,obj.obj_id,obj.children[0].obj_id)
             delete_rscref(c_obj,obj.obj_id)
             if silly_constraint(c_obj.node,obj.obj_id):
                 # remove invalid constraints
                 self._remove_obj(c_obj)
                 if not self._no_constraint_rm_msg:
                     err_buf.info("hanging %s deleted" % c_obj.obj_string())
     def related_constraints(self,obj):
         if not is_resource(obj.node):
             return []
         c_list = []
         for obj2 in self.cib_objects:
             if not is_constraint(obj2.node):
                 continue
             if rsc_constraint(obj.obj_id,obj2.node):
                 c_list.append(obj2)
         return c_list
     def redirect_children_constraints(self,obj):
         '''
         Redirect constraints to the new parent
         '''
         for child in obj.children:
             for c_obj in self.related_constraints(child):
                 rename_rscref(c_obj,child.obj_id,obj.obj_id)
         # drop useless constraints which may have been created above
         for c_obj in self.related_constraints(obj):
             if silly_constraint(c_obj.node,obj.obj_id):
                 self._no_constraint_rm_msg = True
                 self._remove_obj(c_obj)
                 self._no_constraint_rm_msg = False
     def add_to_remove_queue(self,obj):
         if obj.origin == "cib":
             self.remove_queue.append(obj)
         #print self.cib_objects_string(self.remove_queue)
     def delete_1(self,obj):
         '''
         Remove an object and its parent in case the object is the
         only child.
         '''
         if obj.parent and len(obj.parent.children) == 1:
             self.delete_1(obj.parent)
         if obj in self.cib_objects: # don't remove parents twice
             self._remove_obj(obj)
     def delete(self,*args):
         'Delete a cib object.'
         if not self.doc:
             empty_cib_err()
             return False
         rc = True
         l = []
         for obj_id in args:
             obj = self.find_object(obj_id)
             if not obj:
                 no_object_err(obj_id)
                 rc = False
                 continue
             if is_rsc_running(obj_id):
                 common_warn("resource %s is running, can't delete it" % obj_id)
             else:
                 l.append(obj)
         if l:
             l = processing_sort_cli(l)
             for obj in reversed(l):
                 self.delete_1(obj)
         return rc
     def rename(self,old_id,new_id):
         '''
         Rename a cib object.
         - check if the resource (if it's a resource) is stopped
         - check if the new id is not taken
         - find the object with old id
         - rename old id to new id in all related objects
           (constraints)
         - if the object came from the CIB, then it must be
           deleted and the one with the new name created
         - rename old id to new id in the object
         '''
         if not self.doc:
             empty_cib_err()
             return False
         if id_store.id_in_use(new_id):
             return False
         obj = self.find_object(old_id)
         if not obj:
             no_object_err(old_id)
             return False
         if not obj.can_be_renamed():
             return False
         for c_obj in self.related_constraints(obj):
             rename_rscref(c_obj,old_id,new_id)
         rename_id(obj.node,old_id,new_id)
         obj.obj_id = new_id
         id_store.rename(old_id,new_id)
         obj.updated = True
         obj.propagate_updated()
     def erase(self):
         "Remove all cib objects."
         # remove only bottom objects and no constraints
         # the rest will automatically follow
         if not self.doc:
             empty_cib_err()
             return False
         erase_ok = True
         l = []
         for obj in [obj for obj in self.cib_objects \
                 if not obj.children and not is_constraint(obj.node) \
                 and obj.obj_type != "node" ]:
             if is_rsc_running(obj.obj_id):
                 common_warn("resource %s is running, can't delete it" % obj.obj_id)
                 erase_ok = False
             else:
                 l.append(obj)
         if not erase_ok:
             common_err("CIB erase aborted (nothing was deleted)")
             return False
         self._no_constraint_rm_msg = True
         for obj in l:
             self.delete(obj.obj_id)
         self._no_constraint_rm_msg = False
         remaining = 0
         for obj in self.cib_objects:
             if obj.obj_type != "node":
                 remaining += 1
         if remaining > 0:
             common_err("strange, but these objects remained:")
             for obj in self.cib_objects:
                 if obj.obj_type != "node":
                     print >> sys.stderr, obj.obj_string()
             self.cib_objects = []
         return True
     def erase_nodes(self):
         "Remove nodes only."
         if not self.doc:
             empty_cib_err()
             return False
         l = [obj for obj in self.cib_objects if obj.obj_type == "node"]
         for obj in l:
             self.delete(obj.obj_id)
     def refresh(self):
         "Refresh from the CIB."
         self.reset()
         self.initialize()
 
 user_prefs = UserPrefs.getInstance()
 options = Options.getInstance()
 err_buf = ErrorBuffer.getInstance()
 vars = Vars.getInstance()
 cib_factory = CibFactory.getInstance()
 cli_display = CliDisplay.getInstance()
 cib_status = CibStatus.getInstance()
 id_store = IdMgmt.getInstance()
 
 # vim:ts=4:sw=4:et:
diff --git a/shell/modules/parse.py b/shell/modules/parse.py
index 681f3c2844..fa43e9398b 100644
--- a/shell/modules/parse.py
+++ b/shell/modules/parse.py
@@ -1,664 +1,664 @@
 # Copyright (C) 2008 Dejan Muhamedagic <dmuhamedagic@suse.de>
 # 
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public
 # License as published by the Free Software Foundation; either
 # version 2.1 of the License, or (at your option) any later version.
 # 
 # This software is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 # General Public License for more details.
 # 
 # You should have received a copy of the GNU General Public
 # License along with this library; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 #
 
 import shlex
 import re
 import xml.dom.minidom
 from utils import *
 from vars import Vars
 from msg import *
 from ra import disambiguate_ra_type, ra_type_validate
 
 #
 # CLI parsing utilities
 # WARNING: ugly code ahead (to be replaced some day by a proper
 # yacc parser, if there's such a thing)
 #
 def cli_parse_rsctype(s, pl):
     '''
     Parse the resource type.
     '''
     ra_class,provider,rsc_type = disambiguate_ra_type(s)
     if not ra_type_validate(s,ra_class,provider,rsc_type):
         return None
     pl.append(["class",ra_class])
     if ra_class == "ocf":
         pl.append(["provider",provider])
     pl.append(["type",rsc_type])
 def is_attribute(p,a):
     return p.startswith(a + '=')
 def cli_parse_attr_strict(s,pl):
     '''
     Parse attributes in the 'p=v' form.
     '''
     if s and '=' in s[0]:
         n,v = s[0].split('=',1)
         if not n:
             return
         pl.append([n,v])
         cli_parse_attr_strict(s[1:],pl)
 def cli_parse_attr(s,pl):
     '''
     Parse attributes in the 'p=v' form.
     Allow also the 'p' form (no value) unless p is one of the
     attr_list_keyw words.
     '''
     attr_lists_keyw = olist(["params","meta","utilization","operations","op","attributes"])
     if s:
         if s[0] in attr_lists_keyw:
             return
         if '=' in s[0]:
             n,v = s[0].split('=',1)
         else:
             n = s[0]; v = None
         if not n:
             return
         pl.append([n,v])
         cli_parse_attr(s[1:],pl)
 def is_only_id(pl,keyw):
     if len(pl) > 1:
         common_err("%s: only single $id or $id-ref attribute is allowed" % keyw)
         return False
     if len(pl) == 1 and pl[0][0] not in ("$id","$id-ref"):
         common_err("%s: only single $id or $id-ref attribute is allowed" % keyw)
         return False
     return True
 def check_operation(pl):
     op_name = find_value(pl,"name")
     if not op_name in olist(vars.op_cli_names):
         common_warn("%s: operation not recognized" % op_name)
     if op_name == "monitor" and not find_value(pl,"interval"):
         common_err("monitor requires interval")
         return False
     return True
 def parse_resource(s):
     el_type = s[0].lower()
     if el_type == "master": # ugly kludge :(
         el_type = "ms"
     attr_lists_keyw = olist(["params","meta","utilization"])
     cli_list = []
     # the head
     head = []
     head.append(["id",s[1]])
     i = 3
     if el_type == "primitive":
         cli_parse_rsctype(s[2],head)
         if not find_value(head,"type"):
             syntax_err(s[2:], context = "primitive")
             return False
     else:
         cl = []
         cl.append(s[2])
         if el_type == "group":
             while i < len(s):
                 if s[i] in attr_lists_keyw:
                     break
                 elif is_attribute(s[i],"description"):
                     break
                 else:
                     cl.append(s[i])
                     i += 1 # skip to the next token
         head.append(["$children",cl])
     try:  # s[i] may be out of range
         if is_attribute(s[i],"description"):
             cli_parse_attr(s[i:i+1],head)
             i += 1 # skip to the next token
     except: pass
     cli_list.append([el_type,head])
     # the rest
     state = 0 # 1: reading operations; 2: operations read
     while len(s) > i+1:
         pl = []
         keyw = s[i].lower()
         if keyw in attr_lists_keyw:
             if state == 1:
                 state = 2
         elif el_type == "primitive" and state == 0 and keyword_cmp(keyw, "operations"):
             state = 1
         elif el_type == "primitive" and state <= 1 and keyword_cmp(keyw, "op"):
             if state == 0:
                 state = 1
             pl.append(["name",s[i+1]])
         else:
             syntax_err(s[i:], context = 'primitive')
             return False
         if keyword_cmp(keyw, "op"):
             if len(s) > i+2:
                 cli_parse_attr(s[i+2:],pl)
             if not check_operation(pl):
                 return False
         else:
             cli_parse_attr(s[i+1:],pl)
             if len(pl) == 0:
                 syntax_err(s[i:], context = 'primitive')
                 return False
         if keyword_cmp(keyw, "operations") and not is_only_id(pl,keyw):
             return False
         i += len(pl)+1
         # interval is obligatory for ops, supply 0 if not there
         if keyword_cmp(keyw, "op") and not find_value(pl,"interval"):
             pl.append(["interval","0"])
         cli_list.append([keyw,pl])
     if len(s) > i:
         syntax_err(s[i:], context = 'primitive')
         return False
     return cli_list
 def parse_op(s):
     if len(s) != 3:
         syntax_err(s, context = s[0])
         return False
     cli_list = []
     head_pl = []
     # this is an op
     cli_list.append(["op",head_pl])
     if not cli_parse_rsc_role(s[1],head_pl):
         return False
     if not cli_parse_op_times(s[2],head_pl):
         return False
     # rename rsc-role to role
     for i in range(len(head_pl)):
         if head_pl[i][0] == "rsc-role":
             head_pl[i][0] = "role"
             break
     # add the operation name
     head_pl.append(["name",s[0]])
     return cli_list
 
 def cli_parse_score(score,pl,noattr = False):
     if score.endswith(':'):
         score = score.rstrip(':')
     else:
         syntax_err(score, context = 'score')
         return False
     if score in vars.score_types:
         pl.append(["score",vars.score_types[score]])
     elif re.match("^[+-]?(inf|infinity|INFINITY|[[0-9]+)$",score):
         score = score.replace("infinity","INFINITY")
         score = score.replace("inf","INFINITY")
         pl.append(["score",score])
     elif score:
         if noattr:
             common_err("attribute not allowed for score in orders")
             return False
         else:
             pl.append(["score-attribute",score])
     return True
 def is_binary_op(s):
     l = s.split(':')
     if len(l) == 2:
         return l[0] in vars.binary_types and l[1] in olist(vars.binary_ops)
     elif len(l) == 1:
         return l[0] in olist(vars.binary_ops)
     else:
         return False
 def cli_parse_binary_op(s,pl):
     l = s.split(':')
     if len(l) == 2:
         pl.append(["type",l[0]])
         pl.append(["operation",l[1]])
     else:
         pl.append(["operation",l[0]])
 def cli_parse_expression(s,pl):
     if len(s) > 1 and s[0] in olist(vars.unary_ops):
         pl.append(["operation",s[0]])
         pl.append(["attribute",s[1]])
     elif len(s) > 2 and is_binary_op(s[1]):
         pl.append(["attribute",s[0]])
         cli_parse_binary_op(s[1],pl)
         pl.append(["value",s[2]])
     else:
         return False
     return True
 def cli_parse_dateexpr(s,pl):
     if len(s) < 3:
         return False
     if s[1] not in olist(vars.date_ops):
         return False
     pl.append(["operation",s[1]])
     if s[1] in olist(vars.simple_date_ops):
         pl.append([keyword_cmp(s[1], 'lt') and "end" or "start",s[2]])
         return True
     cli_parse_attr_strict(s[2:],pl)
     return True
 def parse_rule(s):
     if not keyword_cmp(s[0], "rule"):
         syntax_err(s,context = "rule")
         return 0,None
     rule_list = []
     head_pl = []
     rule_list.append([s[0].lower(),head_pl])
     i = 1
     cli_parse_attr_strict(s[i:],head_pl)
     i += len(head_pl)
     if find_value(head_pl,"$id-ref"):
         return i,rule_list
     if not cli_parse_score(s[i],head_pl):
         return i,None
     i += 1
     bool_op = ''
     while len(s) > i+1:
         pl = []
         if keyword_cmp(s[i], "date"):
             fun = cli_parse_dateexpr
             elem = "date_expression"
         else:
             fun = cli_parse_expression
             elem = "expression"
         if not fun(s[i:],pl):
             syntax_err(s[i:],context = "rule")
             return i,None
         rule_list.append([elem,pl])
         i += len(pl)
         if find_value(pl, "type"):
             i -= 1 # reduce no of tokens by one if there was "type:op"
         if elem == "date_expression":
             i += 1 # increase no of tokens by one if it was date expression
         if len(s) > i and s[i] in olist(vars.boolean_ops):
             if bool_op and not keyword_cmp(bool_op, s[i]):
                 common_err("rule contains different bool operations: %s" % ' '.join(s))
                 return i,None
             else:
                 bool_op = s[i].lower()
                 i += 1
         if len(s) > i and keyword_cmp(s[i], "rule"):
             break
     if bool_op and not keyword_cmp(bool_op, 'and'):
         head_pl.append(["boolean-op",bool_op])
     return i,rule_list
 def parse_location(s):
     cli_list = []
     head_pl = []
     head_pl.append(["id",s[1]])
     head_pl.append(["rsc",s[2]])
     cli_list.append([s[0].lower(),head_pl])
     if len(s) == 5 and not keyword_cmp(s[3], "rule"): # the short node preference form
         if not cli_parse_score(s[3],head_pl):
             return False
         head_pl.append(["node",s[4]])
         return cli_list
     i = 3
     while i < len(s):
         numtoks,l = parse_rule(s[i:])
         if not l:
             return False
         cli_list += l
         i += numtoks
     if len(s) < i:
         syntax_err(s[i:],context = "location")
         return False
     return cli_list
 
 def cli_opt_symmetrical(p,pl):
     if not p:
         return True
     pl1 = []
     cli_parse_attr([p],pl1)
     if len(pl1) != 1 or not find_value(pl1,"symmetrical"):
         syntax_err(p,context = "order")
         return False
     pl += pl1
     return True
 def cli_parse_rsc_role(s,pl,attr_pfx = ''):
     l = s.split(':')
     pl.append([attr_pfx+"rsc",l[0]])
     if len(l) == 2:
         if l[1] not in vars.roles_names:
             bad_def_err("resource role",s)
             return False
         pl.append([attr_pfx+"rsc-role",l[1]])
     elif len(l) > 2:
         bad_def_err("resource role",s)
         return False
     return True
 def cli_parse_op_times(s,pl):
     l = s.split(':')
     pl.append(["interval",l[0]])
     if len(l) == 2:
         pl.append(["timeout",l[1]])
     elif len(l) > 2:
         bad_def_err("op times",s)
         return False
     return True
 
 class ResourceSet(object):
     '''
     Constraint resource set parser. Parses sth like:
     a ( b c:start ) d:Master e ...
     Appends one or more lists to cli_list.
     Lists are in form:
         list :: ["resource_set",set_pl]
         set_pl :: [["sequential","false"], ["action"|"role",action|role],
             ["resource_ref",["id",rsc]], ...]
         (the first two elements of set_pl are optional)
     Action/role change makes a new resource set.
     '''
     def __init__(self,type,s,cli_list):
         self.type = type
         self.valid_q = (type == "order") and vars.actions_names or vars.roles_names
         self.q_attr = (type == "order") and "action" or "role"
         self.tokens = s
         self.cli_list = cli_list
         self.reset_set()
         self.sequential = True
         self.fix_parentheses()
     def fix_parentheses(self):
         newtoks = []
         for p in self.tokens:
             if p.startswith('(') and len(p) > 1:
                 newtoks.append('(')
                 newtoks.append(p[1:])
             elif p.endswith(')') and len(p) > 1:
                 newtoks.append(p[0:len(p)-1])
                 newtoks.append(')')
             else:
                 newtoks.append(p)
         self.tokens = newtoks
     def reset_set(self):
         self.set_pl = []
         self.prev_q = ''  # previous qualifier (action or role)
         self.curr_attr = ''  # attribute (action or role)
     def save_set(self):
         if not self.set_pl:
             return
         if self.curr_attr:
             self.set_pl.insert(0,[self.curr_attr,self.prev_q])
         if not self.sequential:
             self.set_pl.insert(0,["sequential","false"])
         self.cli_list.append(["resource_set",self.set_pl])
         self.reset_set()
     def splitrsc(self,p):
         l = p.split(':')
         return (len(l) == 1) and [p,''] or l
     def parse(self):
         tokpos = -1
         for p in self.tokens:
             tokpos += 1
             if p == "_rsc_set_":
                 continue # a degenerate resource set
             if p == '(':
                 if self.set_pl: # save the set before
                     self.save_set()
                 self.sequential = False
                 continue
             if p == ')':
                 if self.sequential:  # no '('
                     syntax_err(self.tokens[tokpos:],context = self.type)
                     return False
                 if not self.set_pl:  # empty sets not allowed
                     syntax_err(self.tokens[tokpos:],context = self.type)
                     return False
                 self.save_set()
                 self.sequential = True
                 continue
             rsc,q = self.splitrsc(p)
             if q != self.prev_q: # one set can't have different roles/actions
                 self.save_set()
                 self.prev_q = q
             if q:
                 if q not in self.valid_q:
                     common_err("%s: invalid %s in %s" % (q,self.q_attr,self.type))
                     return False
                 if not self.curr_attr:
                     self.curr_attr = self.q_attr
             else:
                 self.curr_attr = ''
             self.set_pl.append(["resource_ref",["id",rsc]])
         if not self.sequential: # no ')'
             syntax_err(self.tokens[tokpos:],context = self.type)
             return False
         if self.set_pl: # save the final set
             self.save_set()
         return True
 
 def parse_colocation(s):
     cli_list = []
     head_pl = []
     type = s[0]
     if type == "collocation": # another ugly :(
         type = "colocation"
     cli_list.append([type,head_pl])
     if len(s) < 5:
         syntax_err(s,context = "colocation")
         return False
     head_pl.append(["id",s[1]])
     if not cli_parse_score(s[2],head_pl):
         return False
     if len(s) == 5:
         if not cli_parse_rsc_role(s[3],head_pl):
             return False
         if not cli_parse_rsc_role(s[4],head_pl,'with-'):
             return False
     else:
         resource_set_obj = ResourceSet(type,s[3:],cli_list)
         if not resource_set_obj.parse():
             return False
     return cli_list
 def cli_parse_rsc_action(s,pl,rsc_pos):
     l = s.split(':')
     pl.append([rsc_pos,l[0]])
     if len(l) == 2:
         if l[1] not in vars.actions_names:
             bad_def_err("resource action",s)
             return False
         pl.append([rsc_pos+"-action",l[1]])
     elif len(l) > 1:
         bad_def_err("resource action",s)
         return False
     return True
 
 def parse_order(s):
     cli_list = []
     head_pl = []
     type = "order"
     cli_list.append([s[0],head_pl])
     if len(s) < 5:
         syntax_err(s,context = "order")
         return False
     head_pl.append(["id",s[1]])
     if not cli_parse_score(s[2],head_pl,noattr = True):
         return False
     # save symmetrical for later (if it exists)
     symm = ""
     if is_attribute(s[len(s)-1],"symmetrical"):
         symm = s.pop()
     if len(s) == 5:
         if not cli_parse_rsc_action(s[3],head_pl,'first'):
             return False
         if not cli_parse_rsc_action(s[4],head_pl,'then'):
             return False
     else:
         resource_set_obj = ResourceSet(type,s[3:],cli_list)
         if not resource_set_obj.parse():
             return False
     if not cli_opt_symmetrical(symm,head_pl):
         return False
     return cli_list
 
 def parse_constraint(s):
     if keyword_cmp(s[0], "location"):
         return parse_location(s)
     elif s[0] in olist(["colocation","collocation"]):
         return parse_colocation(s)
     elif keyword_cmp(s[0], "order"):
         return parse_order(s)
 def parse_property(s):
     cli_list = []
     head_pl = []
     cli_list.append([s[0],head_pl])
     cli_parse_attr(s[1:],head_pl)
     if len(head_pl) < 0 or len(s) > len(head_pl)+1:
         syntax_err(s, context = s[0])
         return False
     return cli_list
 def cli_parse_uname(s, pl):
     l = s.split(':')
     if not l or len(l) > 2:
         return None
     pl.append(["uname",l[0]])
     if len(l) == 2:
         pl.append(["type",l[1]])
 def parse_node(s):
     cli_list = []
     # the head
     head = []
     # optional $id
     id = ''
     opt_id_l = []
     i = 1
     cli_parse_attr_strict(s[i:],opt_id_l)
     if opt_id_l:
         id = find_value(opt_id_l,"$id")
         i += 1
     # uname[:type]
     cli_parse_uname(s[i],head)
     uname = find_value(head,"uname")
     if not uname:
         return False
     head.append(["id",id and id or uname])
     # drop type if default
     type = find_value(head,"type")
     if type == vars.node_default_type:
         head.remove(["type",type])
     cli_list.append([s[0],head])
     if len(s) == i:
         return cli_list
     # the rest
     i += 1
     try:  # s[i] may be out of range
         if is_attribute(s[i],"description"):
             cli_parse_attr(s[i:i+1],head)
             i += 1 # skip to the next token
     except: pass
     while len(s) > i+1:
         if not s[i] in olist(vars.node_attributes_keyw):
             syntax_err(s[i:], context = 'node')
             return False
         pl = []
         cli_parse_attr(s[i+1:],pl)
         if len(pl) == 0:
             syntax_err(s[i:], context = 'node')
             return False
         cli_list.append([s[i],pl])
         i += len(pl)+1
     if len(s) > i:
         syntax_err(s[i:], context = 'node')
         return False
     return cli_list
 def parse_xml(s):
     cli_list = []
     head = []
     try:
         xml_s = ' '.join(s[1:])
     except:
         syntax_err(s, context = 'xml')
         return False
     # strip spaces between elements
     # they produce text elements
     xml_s = re.sub(r">\s+<", "><", xml_s)
     try:
         doc = xml.dom.minidom.parseString(xml_s)
     except xml.parsers.expat.ExpatError, msg:
         common_err("cannot parse xml chunk: %s" % xml_s)
         common_err(msg)
         return False
     try:
         elnode = doc.childNodes[0]
     except:
         common_err("no elements in %s" % xml_s)
         return False
     try:
         el_type = vars.cib_cli_map[elnode.tagName]
     except:
         common_err("element %s not recognized" % elnode.tagName)
         return False
     id = elnode.getAttribute("id")
     head.append(["id",id])
     cli_list.append([el_type,head])
     cli_list.append(["raw",xml_s])
     return cli_list
 
 def xml_lex(s):
     l = lines2cli(s)
     a = []
     for p in l:
         a += p.split()
     return a
 
 class CliParser(object):
     parsers = {
         "primitive": (3,parse_resource),
         "group": (3,parse_resource),
         "clone": (3,parse_resource),
         "ms": (3,parse_resource),
         "master": (3,parse_resource),
         "location": (3,parse_constraint),
         "colocation": (3,parse_constraint),
         "collocation": (3,parse_constraint),
         "order": (3,parse_constraint),
         "monitor": (3,parse_op),
         "node": (2,parse_node),
         "property": (2,parse_property),
         "rsc_defaults": (2,parse_property),
         "op_defaults": (2,parse_property),
         "xml": (3,parse_xml),
     }
     def __init__(self):
         self.comments = []
     def parse(self,s):
         '''
         Input: a list of tokens (or a CLI format string).
         Return: a list of items; each item is a tuple
             with two members: a string (tag) and a nvpairs or
             attributes dict.
         '''
         cli_list = ''
         if type(s) == type(u''):
             s = s.encode('ascii')
         if type(s) == type(''):
             if s and s.startswith('#'):
-                #self.comments.append(s)
+                self.comments.append(s)
                 return None
             if s.startswith('xml'):
                 s = xml_lex(s)
             else:
                 try:
                     s = shlex.split(s)
                 except ValueError, msg:
                     common_err(msg)
                     return False
         # but there shouldn't be any newlines (?)
         while '\n' in s:
             s.remove('\n')
         if not s:
             return None
         if s[0] not in self.parsers.keys():
             syntax_err(s)
             return False
         mintoks,parser_fn = self.parsers[s[0]]
         if len(s) < mintoks:
             syntax_err(s)
             return False
         cli_list = parser_fn(s)
         if not cli_list:
             return False
         if self.comments:
             cli_list.append(["comments",self.comments])
             self.comments = []
         return cli_list
 
 vars = Vars.getInstance()
 # vim:ts=4:sw=4:et:
diff --git a/shell/modules/xmlutil.py b/shell/modules/xmlutil.py
index 2ecaf831ad..6ea94260b9 100644
--- a/shell/modules/xmlutil.py
+++ b/shell/modules/xmlutil.py
@@ -1,709 +1,722 @@
 # Copyright (C) 2008 Dejan Muhamedagic <dmuhamedagic@suse.de>
 # 
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public
 # License as published by the Free Software Foundation; either
 # version 2.1 of the License, or (at your option) any later version.
 # 
 # This software is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 # General Public License for more details.
 # 
 # You should have received a copy of the GNU General Public
 # License along with this library; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 #
 
 import os
 import subprocess
 import xml.dom.minidom
 
 from userprefs import Options, UserPrefs
 from vars import Vars
 from msg import *
 from utils import *
 
 def xmlparse(f):
     try:
         doc = xml.dom.minidom.parse(f)
     except xml.parsers.expat.ExpatError,msg:
         common_err("cannot parse xml: %s" % msg)
         return None
     return doc
 def file2doc(s):
     try: f = open(s,'r')
     except IOError, msg:
         common_err(msg)
         return None
     doc = xmlparse(f)
     f.close()
     return doc
 
 cib_dump = "cibadmin -Ql"
 def cibdump2doc(section = None):
     doc = None
     if section:
         cmd = "%s -o %s" % (cib_dump,section)
     else:
         cmd = cib_dump
     cmd = add_sudo(cmd)
     p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
     try:
         doc = xmlparse(p.stdout)
         p.wait()
     except IOError, msg:
         common_err(msg)
         return None
     return doc
 cib_piped = "cibadmin -p"
 def commit_rsc(node):
     "Replace a resource definition using cibadmin -R"
     rc = pipe_string("%s -R -o %s" % \
         (cib_piped, "resources"), node.toxml())
     return rc == 0
 
 def get_conf_elem(doc, tag):
     try:
         return doc.getElementsByTagName(tag)[0]
     except:
         return None
 def read_cib(fun, params = None):
     doc = fun(params)
     if not doc:
         return doc,None
     cib = doc.childNodes[0]
     if not is_element(cib) or cib.tagName != "cib":
         cib_no_elem_err("cib")
         return doc,None
     return doc,cib
 
 def get_interesting_nodes(node,nodes):
     for c in node.childNodes:
         if is_element(c) and c.tagName in vars.cib_cli_map:
             nodes.append(c)
         get_interesting_nodes(c,nodes)
     return nodes
 
 def resources_xml():
     return cibdump2doc("resources")
 def rsc2node(id):
     doc = resources_xml()
     if not doc:
         return None
     nodes = get_interesting_nodes(doc,[])
     for n in nodes:
         if is_resource(n) and n.getAttribute("id") == id:
             return n
 def get_meta_param(id,param):
     rsc_meta_show = "crm_resource --meta -r '%s' -g '%s'"
     return get_stdout(rsc_meta_show % (id,param), stderr_on = False)
 def is_live_cib():
     '''We working with the live cluster?'''
     return not vars.cib_in_use and not os.getenv("CIB_file")
 def is_rsc_running(id):
     if not is_live_cib():
         return False
     rsc_node = rsc2node(id)
     if not rsc_node:
         return False
     if not is_resource(rsc_node):
         return False
     rsc_status = "crm_resource -W -r '%s'"
     test_id = rsc_clone(id) or id
     outp = get_stdout(rsc_status % test_id, stderr_on = False)
     return outp.find("running") > 0 and outp.find("NOT") == -1
 def is_rsc_clone(rsc_id):
     rsc_node = rsc2node(rsc_id)
     return is_clone(rsc_node)
 def is_rsc_ms(rsc_id):
     rsc_node = rsc2node(rsc_id)
     return is_ms(rsc_node)
 def rsc_clone(rsc_id):
     '''Get a clone of a resource.'''
     rsc_node = rsc2node(rsc_id)
     if not rsc_node or not rsc_node.parentNode:
         return None
     pnode = rsc_node.parentNode
     if is_group(pnode):
         pnode = pnode.parentNode
     if is_clonems(pnode):
         return pnode.getAttribute("id")
 def get_topmost_rsc(node):
     '''
     Return a topmost node which is a resource and contains this resource
     '''
     if is_container(node.parentNode):
         return get_topmost_rsc(node.parentNode)
     return node
 def get_cloned_rsc(rsc_id):
     rsc_node = rsc2node(rsc_id)
     if not rsc_node:
         return ""
     for c in rsc_node.childNodes:
         if is_child_rsc(c):
             return c.getAttribute("id")
     return ""
 attr_defaults_missing = {
 }
 def add_missing_attr(node):
     try:
         for defaults in attr_defaults_missing[node.tagName]:
             if not node.hasAttribute(defaults[0]):
                 node.setAttribute(defaults[0],defaults[1])
     except: pass
 attr_defaults = {
     "rule": (("boolean-op","and"),),
     "expression": (("type","string"),),
 }
 def drop_attr_defaults(node, ts = 0):
     try:
         for defaults in attr_defaults[node.tagName]:
             if node.getAttribute(defaults[0]) == defaults[1]:
                 node.removeAttribute(defaults[0])
     except: pass
 
 def is_element(xmlnode):
     return xmlnode and xmlnode.nodeType == xmlnode.ELEMENT_NODE
 
 def nameandid(xmlnode,level):
     if xmlnode.nodeType == xmlnode.ELEMENT_NODE:
         print level*' ',xmlnode.tagName,xmlnode.getAttribute("id"),xmlnode.getAttribute("name")
 
 def xmltraverse(xmlnode,fun,ts=0):
     for c in xmlnode.childNodes:
         if is_element(c):
             fun(c,ts)
             xmltraverse(c,fun,ts+1)
 
 def xmltraverse_thin(xmlnode,fun,ts=0):
     '''
     Skip elements which may be resources themselves.
     NB: Call this only on resource (or constraint) nodes, but
     never on cib or configuration!
     '''
     for c in xmlnode.childNodes:
         if is_element(c) and not c.tagName in ('primitive','group'):
             xmltraverse_thin(c,fun,ts+1)
     fun(xmlnode,ts)
 
 def xml_processnodes(xmlnode,node_filter,proc):
     '''
     Process with proc all nodes that match filter.
     '''
     node_list = []
     for child in xmlnode.childNodes:
         if node_filter(child):
             node_list.append(child)
         if child.hasChildNodes():
             xml_processnodes(child,node_filter,proc)
     if node_list:
         proc(node_list)
 
 # filter the cib
 def is_whitespace(node):
     return node.nodeType == node.TEXT_NODE and not node.data.strip()
 def is_comment(node):
     return node.nodeType == node.COMMENT_NODE
 def is_status_node(node):
     return is_element(node) and node.tagName == "status"
 
 def is_emptynvpairs(node):
     if is_element(node) and node.tagName in vars.nvpairs_tags:
         for a in vars.precious_attrs:
             if node.getAttribute(a):
                 return False
         for n in node.childNodes:
             if is_element(n):
                 return False
         return True
     else:
         return False
 def is_group(node):
     return is_element(node) \
         and node.tagName == "group"
 def is_ms(node):
     return is_element(node) \
         and node.tagName in ("master","ms")
 def is_clone(node):
     return is_element(node) \
         and node.tagName == "clone"
 def is_clonems(node):
     return is_element(node) \
         and node.tagName in vars.clonems_tags
 def is_container(node):
     return is_element(node) \
         and node.tagName in vars.container_tags
 def is_primitive(node):
     return is_element(node) \
         and node.tagName == "primitive"
 def is_resource(node):
     return is_element(node) \
         and node.tagName in vars.resource_tags
 def is_child_rsc(node):
     return is_element(node) \
         and node.tagName in vars.children_tags
 def is_constraint(node):
     return is_element(node) \
         and node.tagName in vars.constraint_tags
 def is_defaults(node):
     return is_element(node) \
         and node.tagName in vars.defaults_tags
 def rsc_constraint(rsc_id,cons_node):
     if not is_element(cons_node):
         return False
     for attr in cons_node.attributes.keys():
         if attr in vars.constraint_rsc_refs \
                 and rsc_id == cons_node.getAttribute(attr):
             return True
     for rref in cons_node.getElementsByTagName("resource_ref"):
         if rsc_id == rref.getAttribute("id"):
             return True
     return False
 
 def sort_container_children(node_list):
     '''
     Make sure that attributes's nodes are first, followed by the
     elements (primitive/group). The order of elements is not
     disturbed, they are just shifted to end!
     '''
     for node in node_list:
         children = []
         for c in node.childNodes:
             if is_element(c) and c.tagName in vars.children_tags:
                 children.append(c)
         for c in children:
             node.removeChild(c)
         for c in children:
             node.appendChild(c)
 def rmnode(node):
     if node and node.parentNode:
         if node.parentNode:
             node.parentNode.removeChild(node)
         node.unlink()
 def rmnodes(node_list):
     for node in node_list:
         rmnode(node)
 def printid(node_list):
     for node in node_list:
         id = node.getAttribute("id")
         if id: print "node id:",id
 def sanitize_cib(doc):
     xml_processnodes(doc,is_status_node,rmnodes)
     #xml_processnodes(doc,is_element,printid)
     xml_processnodes(doc,is_emptynvpairs,rmnodes)
     xml_processnodes(doc,is_whitespace,rmnodes)
-    xml_processnodes(doc,is_comment,rmnodes)
+    #xml_processnodes(doc,is_comment,rmnodes)
     xml_processnodes(doc,is_container,sort_container_children)
     xmltraverse(doc,drop_attr_defaults)
 
 def is_simpleconstraint(node):
     return len(node.getElementsByTagName("resource_ref")) == 0
 
 match_list = {
     "node": ("uname",),
     "crm_config": (),
     "rsc_defaults": (),
     "op_defaults": (),
     "cluster_property_set": (),
     "instance_attributes": (),
     "meta_attributes": (),
     "utilization": (),
     "operations": (),
     "nvpair": ("name",),
     "op": ("name","interval"),
     "rule": ("score","score-attribute","role"),
     "expression": ("attribute","operation","value"),
 }
-def add_comment(doc,node,s):
+def add_comment(node,s):
     '''
     Add comment s to node from doc.
     '''
-    if not s:
+    if not node or not s:
         return
-    comm_node = doc.createComment(s)
-    if node.hasChildNodes():
-        node.insertBefore(comm_node, node.firstChild)
-    else:
-        node.appendChild(comm_node)
+    comm_node = node.ownerDocument.createComment(s)
+    firstelem = None
+    for n in node.childNodes:
+        if is_element(n):
+            firstelem = n
+            break
+    node.insertBefore(comm_node, firstelem)
+def stuff_comments(node,comments):
+    for s in comments:
+        add_comment(node,s)
+def fix_comments(node):
+    'Make sure that comments start with #'
+    cnodes = [x for x in node.childNodes if is_comment(x)]
+    for n in cnodes:
+        n.data = n.data.strip()
+        if not n.data.startswith("#"):
+            n.data = "# %s" % n.data
+
 def set_id_used_attr(node):
     node.setAttribute("__id_used", "Yes")
 def is_id_used_attr(node):
     return node.getAttribute("__id_used") == "Yes"
 def remove_id_used_attr(node,lvl):
     if is_element(node) and is_id_used_attr(node):
         node.removeAttribute("__id_used")
 def remove_id_used_attributes(node):
     if node:
         xmltraverse(node, remove_id_used_attr)
 def lookup_node(node,oldnode,location_only = False):
     '''
     Find a child of oldnode which matches node.
     '''
     #print "lookup:",node.tagName,node.getAttribute("id")
     if not oldnode:
         return None
     #print "  in:",oldnode.tagName,oldnode.getAttribute("id")
     try:
         attr_list = list(match_list[node.tagName])
     except KeyError:
         attr_list = []
     if node.getAttribute("id") and oldnode.getAttribute("id"):
         attr_list.append("id")
     for c in oldnode.childNodes:
         if not is_element(c):
             continue
         if not location_only and is_id_used_attr(c):
             continue
         #print "  checking:",c.tagName,c.getAttribute("id")
         if node.tagName == c.tagName:
             failed = False
             for a in attr_list:
                 if node.getAttribute(a) != c.getAttribute(a):
                     failed = True
                     break
             if not failed:
                 #print "  found:",c.tagName,c.getAttribute("id")
                 return c
     return None
 
 def find_operation(rsc_node,name,interval):
     op_node_l = rsc_node.getElementsByTagName("operations")
     for ops in op_node_l:
         for c in ops.childNodes:
             if not is_element(c):
                 continue
             if c.tagName != "op":
                 continue
             if c.getAttribute("name") == name \
                     and c.getAttribute("interval") == interval:
                 return c
 
 def filter_on_tag(nl,tag):
     return [node for node in nl if node.tagName == tag]
 def nodes(node_list):
     return filter_on_tag(node_list,"node")
 def primitives(node_list):
     return filter_on_tag(node_list,"primitive")
 def groups(node_list):
     return filter_on_tag(node_list,"group")
 def clones(node_list):
     return filter_on_tag(node_list,"clone")
 def mss(node_list):
     return filter_on_tag(node_list,"master")
 def constraints(node_list):
     return filter_on_tag(node_list,"rsc_location") \
         + filter_on_tag(node_list,"rsc_colocation") \
         + filter_on_tag(node_list,"rsc_order")
 def properties(node_list):
     return filter_on_tag(node_list,"cluster_property_set") \
         + filter_on_tag(node_list,"rsc_defaults") \
         + filter_on_tag(node_list,"op_defaults")
 def processing_sort(nl):
     '''
     It's usually important to process cib objects in this order,
     i.e. simple objects first.
     '''
     return nodes(nl) + primitives(nl) + groups(nl) + mss(nl) + clones(nl) \
         + constraints(nl) + properties(nl)
 
 def obj_cmp(obj1,obj2):
     return cmp(obj1.obj_id,obj2.obj_id)
 def filter_on_type(cl,obj_type):
     if type(cl[0]) == type([]):
         l = [cli_list for cli_list in cl if cli_list[0][0] == obj_type]
         if user_prefs.get_sort_elems():
             l.sort(cmp = cmp)
     else:
         l = [obj for obj in cl if obj.obj_type == obj_type]
         if user_prefs.get_sort_elems():
             l.sort(cmp = obj_cmp)
     return l
 def nodes_cli(cl):
     return filter_on_type(cl,"node")
 def primitives_cli(cl):
     return filter_on_type(cl,"primitive")
 def groups_cli(cl):
     return filter_on_type(cl,"group")
 def clones_cli(cl):
     return filter_on_type(cl,"clone")
 def mss_cli(cl):
     return filter_on_type(cl,"ms") + filter_on_type(cl,"master")
 def constraints_cli(node_list):
     return filter_on_type(node_list,"location") \
         + filter_on_type(node_list,"colocation") \
         + filter_on_type(node_list,"collocation") \
         + filter_on_type(node_list,"order")
 def properties_cli(cl):
     return filter_on_type(cl,"property") \
         + filter_on_type(cl,"rsc_defaults") \
         + filter_on_type(cl,"op_defaults")
 def ops_cli(cl):
     return filter_on_type(cl,"op")
 def processing_sort_cli(cl):
     '''
     Return the given list in this order:
     nodes, primitives, groups, ms, clones, constraints, rest
     Both a list of objects (CibObject) and list of cli
     representations accepted.
     '''
     return nodes_cli(cl) + primitives_cli(cl) + groups_cli(cl) + mss_cli(cl) + clones_cli(cl) \
         + constraints_cli(cl) + properties_cli(cl) + ops_cli(cl)
 
 def is_resource_cli(s):
     return s in olist(vars.resource_cli_names)
 def is_constraint_cli(s):
     return s in olist(vars.constraint_cli_names)
 
 def referenced_resources(node):
     if not is_constraint(node):
         return []
     xml_obj_type = node.tagName
     if xml_obj_type == "rsc_location":
         node_list = node.getElementsByTagName("rsc")
     elif node.getElementsByTagName("resource_ref"): # resource sets
         node_list = node.getElementsByTagName("resource_ref")
     elif xml_obj_type == "rsc_colocation":
         node_list = node.getElementsByTagName("rsc") + \
             node.getElementsByTagName("with-rsc")
     elif xml_obj_type == "rsc_order":
         node_list = node.getElementsByTagName("first") + \
             node.getElementsByTagName("then")
     return [x.getAttribute("id") for x in node_list]
 
 def rename_id(node,old_id,new_id):
     if node.getAttribute("id") == old_id:
         node.setAttribute("id", new_id)
 def rename_rscref_simple(c_obj,old_id,new_id):
     c_modified = False
     for attr in c_obj.node.attributes.keys():
         if attr in vars.constraint_rsc_refs and \
                 c_obj.node.getAttribute(attr) == old_id:
             c_obj.node.setAttribute(attr, new_id)
             c_obj.updated = True
             c_modified = True
     return c_modified
 def delete_rscref_simple(c_obj,rsc_id):
     c_modified = False
     for attr in c_obj.node.attributes.keys():
         if attr in vars.constraint_rsc_refs and \
                 c_obj.node.getAttribute(attr) == rsc_id:
             c_obj.node.removeAttribute(attr)
             c_obj.updated = True
             c_modified = True
     return c_modified
 def rset_uniq(c_obj,d):
     '''
     Drop duplicate resource references.
     '''
     l = []
     for rref in c_obj.node.getElementsByTagName("resource_ref"):
         rsc_id = rref.getAttribute("id")
         if d[rsc_id] > 1: # drop one
             l.append(rref)
             d[rsc_id] -= 1
     rmnodes(l)
 def delete_rscref_rset(c_obj,rsc_id):
     '''
     Drop all reference to rsc_id.
     '''
     c_modified = False
     l = []
     for rref in c_obj.node.getElementsByTagName("resource_ref"):
         if rsc_id == rref.getAttribute("id"):
             l.append(rref)
             c_obj.updated = True
             c_modified = True
     rmnodes(l)
     l = []
     for rset in c_obj.node.getElementsByTagName("resource_set"):
         if len(rset.getElementsByTagName("resource_ref")) == 0:
             l.append(rset)
             c_obj.updated = True
             c_modified = True
     rmnodes(l)
     return c_modified
 def rset_convert(c_obj):
     l = c_obj.node.getElementsByTagName("resource_ref")
     if len(l) != 2:
         return # eh?
     c_obj.modified = True
     cli = c_obj.repr_cli(format = -1)
     newnode = c_obj.cli2node(cli)
     if newnode:
         c_obj.node.parentNode.replaceChild(newnode,c_obj.node)
         c_obj.node.unlink()
 def rename_rscref_rset(c_obj,old_id,new_id):
     c_modified = False
     d = {}
     for rref in c_obj.node.getElementsByTagName("resource_ref"):
         rsc_id = rref.getAttribute("id")
         if rsc_id == old_id:
             rref.setAttribute("id", new_id)
             rsc_id = new_id
             c_obj.updated = True
             c_modified = True
         if not rsc_id in d:
             d[rsc_id] = 0
         else: 
             d[rsc_id] += 1
     rset_uniq(c_obj,d)
     # if only two resource references remained then, to preserve
     # sanity, convert it to a simple constraint (sigh)
     cnt = 0
     for key in d:
         cnt += d[key]
     if cnt == 2:
         rset_convert(c_obj)
     return c_modified
 def rename_rscref(c_obj,old_id,new_id):
     if rename_rscref_simple(c_obj,old_id,new_id) or \
             rename_rscref_rset(c_obj,old_id,new_id):
         err_buf.info("resource references in %s updated" % c_obj.obj_string())
 def delete_rscref(c_obj,rsc_id):
     return delete_rscref_simple(c_obj,rsc_id) or \
         delete_rscref_rset(c_obj,rsc_id)
 def silly_constraint(c_node,rsc_id):
     '''
     Remove a constraint from rsc_id to rsc_id.
     Or an invalid one.
     '''
     if c_node.getElementsByTagName("resource_ref"):
         # it's a resource set
         # the resource sets have already been uniq-ed
         return len(c_node.getElementsByTagName("resource_ref")) <= 1
     cnt = 0  # total count of referenced resources have to be at least two
     rsc_cnt = 0
     for attr in c_node.attributes.keys():
         if attr in vars.constraint_rsc_refs:
             cnt += 1
             if c_node.getAttribute(attr) == rsc_id:
                 rsc_cnt += 1
     if c_node.tagName == "rsc_location":  # locations are never silly
         return cnt < 1
     else:
         return rsc_cnt == 2 or cnt < 2
 
 def get_rsc_children_ids(node):
     return [x.getAttribute("id") \
         for x in node.childNodes if is_child_rsc(x)]
 def get_rscop_defaults_meta_node(node):
     for c in node.childNodes:
         if not is_element(c) or c.tagName != "meta_attributes":
             continue
         return c
     return None
 
 def new_cib():
     doc = xml.dom.minidom.Document()
     cib = doc.createElement("cib")
     doc.appendChild(cib)
     configuration = doc.createElement("configuration")
     cib.appendChild(configuration)
     crm_config = doc.createElement("crm_config")
     configuration.appendChild(crm_config)
     rsc_defaults = doc.createElement("rsc_defaults")
     configuration.appendChild(rsc_defaults)
     op_defaults = doc.createElement("op_defaults")
     configuration.appendChild(op_defaults)
     nodes = doc.createElement("nodes")
     configuration.appendChild(nodes)
     resources = doc.createElement("resources")
     configuration.appendChild(resources)
     constraints = doc.createElement("constraints")
     configuration.appendChild(constraints)
     return doc,cib,crm_config,rsc_defaults,op_defaults,nodes,resources,constraints
 def mk_topnode(doc, tag):
     "Get configuration element or create/append if there's none."
     try:
         e = doc.getElementsByTagName(tag)[0]
     except:
         e = doc.createElement(tag)
         conf = doc.getElementsByTagName("configuration")[0]
         if conf:
             conf.appendChild(e)
         else:
             return None
     return e
 def new_cib_element(node,tagname,id_pfx):
     base_id = node.getAttribute("id")
     newnode = node.ownerDocument.createElement(tagname)
     newnode.setAttribute("id", "%s-%s" % (base_id,id_pfx))
     node.appendChild(newnode)
     return newnode
 def get_attr_in_set(node,attr):
     for c in node.childNodes:
         if not is_element(c):
             continue
         if c.tagName == "nvpair" and c.getAttribute("name") == attr:
             return c
     return None
 def set_attr(node,attr,value):
     '''
     Set an attribute in the attribute set.
     '''
     nvpair = get_attr_in_set(node,attr)
     if not nvpair:
         nvpair = new_cib_element(node,"nvpair",attr)
     nvpair.setAttribute("name",attr)
     nvpair.setAttribute("value",value)
 def get_set_nodes(node,setname,create = 0):
     'Return the attributes set nodes (create one if requested)'
     l = []
     for c in node.childNodes:
         if not is_element(c):
             continue
         if c.tagName == setname:
             l.append(c)
     if l:
         return l
     if create:
         l.append(new_cib_element(node,setname,setname))
     return l
 
 def xml_cmp(n, m, show = False):
     rc = hash(n.toxml()) == hash(m.toxml())
     if not rc and show and user_prefs.get_debug():
         print "original:",n.toprettyxml()
         print "processed:",m.toprettyxml()
     return hash(n.toxml()) == hash(m.toxml())
 
 def merge_nvpairs(dnode,snode):
     rc = False
     add_children = []
     for c in snode.childNodes:
         if c.tagName == "nvpair":
             dc = lookup_node(c,dnode)
             if dc:
                 dc.setAttribute("value",c.getAttribute("value"))
             else:
                 add_children.append(c)
             rc = True
     for c in add_children:
         dnode.appendChild(c)
     return rc
 
 def merge_nodes(dnode,snode):
     '''
     Import elements from snode into dnode.
     If an element is attributes set (vars.nvpairs_tags), then
     merge nvpairs by the name attribute.
     Otherwise, replace the whole element. (TBD)
     '''
     #print "1:",dnode.toprettyxml()
     #print "2:",snode.toprettyxml()
     #vars.nvpairs_tags
     rc = False # any changes done?
     if not dnode or not snode:
         return rc
     for c in snode.childNodes:
         dc = lookup_node(c,dnode)
         if not dc:
             continue
         if dc.tagName in vars.nvpairs_tags:
             rc = rc or merge_nvpairs(dc,c)
     return rc
 
 user_prefs = UserPrefs.getInstance()
 vars = Vars.getInstance()
 # vim:ts=4:sw=4:et: