diff --git a/man/corosync-qdevice.8 b/man/corosync-qdevice.8 index d157679c..d84597cf 100644 --- a/man/corosync-qdevice.8 +++ b/man/corosync-qdevice.8 @@ -1,317 +1,318 @@ .\"/* .\" * Copyright (C) 2016 Red Hat, Inc. .\" * .\" * All rights reserved. .\" * .\" * Author: Jan Friesse .\" * .\" * This software licensed under BSD license, the text of which follows: .\" * .\" * Redistribution and use in source and binary forms, with or without .\" * modification, are permitted provided that the following conditions are met: .\" * .\" * - Redistributions of source code must retain the above copyright notice, .\" * this list of conditions and the following disclaimer. .\" * - Redistributions in binary form must reproduce the above copyright notice, .\" * this list of conditions and the following disclaimer in the documentation .\" * and/or other materials provided with the distribution. .\" * - Neither the name of Red Hat, Inc. nor the names of its .\" * contributors may be used to endorse or promote products derived from this .\" * software without specific prior written permission. .\" * .\" * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" .\" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE .\" * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR .\" * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF .\" * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS .\" * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN .\" * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) .\" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF .\" * THE POSSIBILITY OF SUCH DAMAGE. .\" */ .TH COROSYNC-QDEVICE 8 2016-06-29 .SH NAME corosync-qdevice \- QDevice daemon .SH SYNOPSIS .B "corosync-qdevice [-dfh] [-S option=value[,option2=value2,...]]" .SH DESCRIPTION .B corosync-qdevice is a daemon running on each node of a cluster. It provides a configured number of votes to the quorum subsystem based on a third-party arbitrator's decision. Its primary use is to allow a cluster to sustain more node failures than standard quorum rules allow. It is recommended for clusters with an even number of nodes and highly recommended for 2 node clusters. .SH OPTIONS .TP .B -d Forcefully turn on debug information without the need to change corosync.conf. .TP .B -f Do not daemonize, run in the foreground. .TP .B -h Show short help text .TP .B -S Set advanced settings described in its own section below. This option shouldn't be generally used because most of the options are not safe to change. .SH CONFIGURATION .B corosync-qdevice reads its configuration from corosync.conf file. The main configuration is within .B quorum.device sub-key. Each model also has its own configuration within a similarly named sub-key. .TP .B model Specifies the model to be used. This parameter is required. .B corosync-qdevice is modular and is able to support multiple different models. The model basically defines what type of arbitrator is used. Currently only .I net is supported. .TP .B timeout -Specifies how often should +Specifies how often .B corosync-qdevice should call the votequorum_poll function. It is also used by the net model to adjust its hearbeat timeout. It is recommended that you don't change this value. Default is 10000. .TP .B sync_timeout Specifies how often .B corosync-qdevice should call the votequorum_poll function during a sync phase. It is recommended that you don't change this value. Default is 30000. .TP .B votes The number of votes provided to the cluster by qdevice. Default is (number_of_nodes - 1) or generally sum(votes_per_node) - 1. .PP .B quorum.device.net holds the configuration for model 'net'. .TP .B tls Can be one of .I on, off or required and specifies if tls should be used. .I on means a connection with TLS is attempted first, but if the server doesn't advertise TLS support then non-TLS will be used. .I off is used then TLS is not required and it's then not even tried. This mode is the only one which doesn't need a properly initialized NSS database. .I required means TLS is required and if the server doesn't support TLS, qdevice will exit with error message. Default is on. .TP .B host Specifies the IP address or host name of the qnetd server to be used. This parameter is required. .TP .B port Specifies TCP port of qnetd server. Default is 5403. .TP .B algorithm Decision algorithm. Can be one of the .I ffsplit or .I lms. (actually there are also .I test and .I 2nodelms , both of which are mainly for developers and shouldn't be used for production clusters). For a description of what each algorithm means and how the algorithms differ see their individual sections. Default value is ffsplit. .TP .B tie_breaker can be one of .I lowest, .I highest or valid_node_id (number) values. It's used as a fallback if qdevice has to decide between two or more equal partitions. .I lowest means the partition with the lowest node id is chosen. .I highest means the partition with highest node id is chosen. And valid_node_id means that the partition containing the node with the given node id is chosen. +Default is 'lowest'. .TP .B connect_timeout Timeout when .B corosync-qdevice is trying to connect to .B corosync-qnetd host. Default is 0.8 * quorum.sync_timeout. .TP .B force_ip_version can be one of .I 0|4|6 and forces the software to use the given IP version. .I 0 (default value) means IPv6 is prefered and IPv4 should be used as a fallback. .PP Logging configuration is within the .B logging directive. .B corosync-qdevice parses and supports most of the options with exception of .B to_logfile, .B logfile and .B logfile_priority. The .B logger_subsys sub-directive can be also used if .B subsys is set to QDEVICE. .PP For .B corosync-qdevice to work correctly, the .B nodelist directive has to be used and properly configured. Also the net model requires that .B totem.cluster_name option is set. .SH MODEL NET TLS CONFIGURATION For model net to work using TLS, it's necessary to create the NSS database, import Qnetd CA certificate, and get/distribute a valid client certificate. If pcs is used (recommended) the following steps are not needed because pcs does them automatically. .B corosync-qdevice-net-certutil is the tool to perform required actions semi-automatically. Please consult the help output of it and its man page. For a first time configuration it may make sense to start with the .B -Q option. If TLS is not required just edit corosync.conf file and set .B quorum.device.net.tls to .I off. .SH MODEL NET ALGORITHMS Algorithms are used to change behavior of how .B corosync-qnetd provides votes to a given node/partition. Currently there are two algorithms supported. .TP .B ffsplit This one makes sense only for clusters with even number of nodes. It provides exactly one vote to the partition with the highest number of active nodes. If there are two exactly similar partitions, it provides its vote to the partition that has the most clients connected to the qnetd server. If this number is also equal, then the tie_breaker is used. It is able to transition its vote if the currently active partition becomes partitioned and a non-active partition still has at least 50% of the active nodes. Because of this, a vote is not provided if the qnetd connection is not active. To use this algorithm it's required to set the number of votes per node to 1 (default) and the qdevice number of votes has to be also 1. This is achieved by setting .B quorum.device.votes key in corosync.conf file to 1. .TP .B lms Last-man-standing. If the node is the only one left in the cluster that can see the qnetd server then we return a vote. If more than one node can see the qnetd server but some nodes can't see each other then the cluster is divided up into 'partitions' based on their ring_id and this algorithm returns a vote to the largest active partition or, if there is more than 1 equal partiton, the partition that contains the tie_breaker node (lowest, highest, etc). For LMS to work, the number of qdevice votes has to be set to default (so just delete .B quorum.device.votes key from corosync.conf). .SH ADVANCED SETTINGS Set by using .B -S option. The default value is shown in parentheses) Options beginning with .B net_ prefix are specific to model net. .TP .B lock_file Lock file location. (/var/run/corosync-qdevice/corosync-qdevice.pid) .TP .B local_socket_file Internal IPC socket file location. (/var/run/corosync-qdevice/corosync-qdevice.sock) .TP .B local_socket_backlog Parameter passed to listen syscall. (10) .TP .B max_cs_try_again How many times to retry the call to a corosync function which has returned CS_ERR_TRY_AGAIN. (10) .TP .B votequorum_device_name Name used for qdevice registration. (Qdevice) .TP .B ipc_max_clients Maximum allowed simultaneous IPC clients. (10) .TP .B ipc_max_receive_size Maximum size of a message received by IPC client. (4096) .TP .B ipc_max_send_size Maximum size of a message allowed to be sent to an IPC client. (65536) .TP .B master_wins Force enable/disable master wins. (default is model) .TP .B net_nss_db_dir NSS database directory. (/etc/corosync/qdevice/net/nssdb) .TP .B net_initial_msg_receive_size Initial (used during connection parameters negotiation) maximum size of the receive buffer for message (maximum allowed message size received from qnetd). (32768) .TP .B net_initial_msg_send_size Initial (used during connection parameter negotiation) maximum size of one send buffer (message) to be sent to server. (32768) .TP .B net_min_msg_send_size Minimum required size of one send buffer (message) to be sent to server. (32768) .TP .B net_max_msg_receive_size Maximum allowed size of receive buffer for a message sent by server. (16777216) .TP .B net_max_send_buffers Maximum number of send buffers. (10) .TP .B net_nss_qnetd_cn Canonical name of qnetd server certificate. (Qnetd Server) .TP .B net_nss_client_cert_nickname NSS nickname of qdevice client certificate. (Cluster Cert) .TP .B net_heartbeat_interval_min Minimum heartbeat timeout accepted by client in ms. (1000) .TP .B net_heartbeat_interval_max Maximum heartbeat timeout accepted by client in ms. (120000) .TP .B net_min_connect_timeout Minimum connection timeout accepted by client in ms. (1000) .TP .B net_max_connect_timeout Maximum connection timeout accepted by client in ms. (120000) .TP .B net_test_algorithm_enabled Enable test algorithm. (if built with --enable-debug on, otherwise off) .SH SEE ALSO .BR corosync-qdevice-tool (8) .BR corosync-qdevice-certutil (8) .BR corosync-qnetd (8) .BR corosync.conf (5) .SH AUTHOR Jan Friesse .PP diff --git a/man/corosync.conf.5 b/man/corosync.conf.5 index 226762e7..00a7a382 100644 --- a/man/corosync.conf.5 +++ b/man/corosync.conf.5 @@ -1,692 +1,693 @@ .\"/* .\" * Copyright (c) 2005 MontaVista Software, Inc. .\" * Copyright (c) 2006-2012 Red Hat, Inc. .\" * .\" * All rights reserved. .\" * .\" * Author: Steven Dake (sdake@redhat.com) .\" * .\" * This software licensed under BSD license, the text of which follows: .\" * .\" * Redistribution and use in source and binary forms, with or without .\" * modification, are permitted provided that the following conditions are met: .\" * .\" * - Redistributions of source code must retain the above copyright notice, .\" * this list of conditions and the following disclaimer. .\" * - Redistributions in binary form must reproduce the above copyright notice, .\" * this list of conditions and the following disclaimer in the documentation .\" * and/or other materials provided with the distribution. .\" * - Neither the name of the MontaVista Software, Inc. nor the names of its .\" * contributors may be used to endorse or promote products derived from this .\" * software without specific prior written permission. .\" * .\" * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" .\" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE .\" * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR .\" * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF .\" * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS .\" * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN .\" * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) .\" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF .\" * THE POSSIBILITY OF SUCH DAMAGE. .\" */ .TH COROSYNC_CONF 5 2012-10-10 "corosync Man Page" "Corosync Cluster Engine Programmer's Manual" .SH NAME corosync.conf - corosync executive configuration file .SH SYNOPSIS /etc/corosync/corosync.conf .SH DESCRIPTION The corosync.conf instructs the corosync executive about various parameters needed to control the corosync executive. Empty lines and lines starting with # character are ignored. The configuration file consists of bracketed top level directives. The possible directive choices are: .TP totem { } This top level directive contains configuration options for the totem protocol. .TP logging { } This top level directive contains configuration options for logging. .TP quorum { } This top level directive contains configuration options for quorum. .TP nodelist { } This top level directive contains configuration options for nodes in cluster. .TP qb { } This top level directive contains configuration options related to libqb. .PP .PP Within the .B totem directive, an interface directive is required. There is also one configuration option which is required: .PP .PP Within the .B interface sub-directive of totem there are four parameters which are required. There is one parameter which is optional. .TP ringnumber This specifies the ring number for the interface. When using the redundant ring protocol, each interface should specify separate ring numbers to uniquely identify to the membership protocol which interface to use for which redundant ring. The ringnumber must start at 0. .TP bindnetaddr This specifies the network address the corosync executive should bind to. bindnetaddr should be an IP address configured on the system, or a network address. For example, if the local interface is 192.168.5.92 with netmask 255.255.255.0, you should set bindnetaddr to 192.168.5.92 or 192.168.5.0. If the local interface is 192.168.5.92 with netmask 255.255.255.192, set bindnetaddr to 192.168.5.92 or 192.168.5.64, and so forth. This may also be an IPV6 address, in which case IPV6 networking will be used. In this case, the exact address must be specified and there is no automatic selection of the network interface within a specific subnet as with IPv4. If IPv6 networking is used, the nodeid field in nodelist must be specified. .TP broadcast This is optional and can be set to yes. If it is set to yes, the broadcast address will be used for communication. If this option is set, mcastaddr should not be set. .TP mcastaddr This is the multicast address used by corosync executive. The default should work for most networks, but the network administrator should be queried about a multicast address to use. Avoid 224.x.x.x because this is a "config" multicast address. This may also be an IPV6 multicast address, in which case IPV6 networking will be used. If IPv6 networking is used, the nodeid field in nodelist must be specified. It's not needed to use this option if cluster_name option is used. If both options are used, mcastaddr has higher priority. .TP mcastport This specifies the UDP port number. It is possible to use the same multicast address on a network with the corosync services configured for different UDP ports. Please note corosync uses two UDP ports mcastport (for mcast receives) and mcastport - 1 (for mcast sends). If you have multiple clusters on the same network using the same mcastaddr please configure the mcastports with a gap. .TP ttl This specifies the Time To Live (TTL). If you run your cluster on a routed network then the default of "1" will be too small. This option provides a way to increase this up to 255. The valid range is 0..255. Note that this is only valid on multicast transport types. .PP .PP Within the .B totem directive, there are seven configuration options of which one is required, five are optional, and one is required when IPV6 is configured in the interface subdirective. The required directive controls the version of the totem configuration. The optional option unless using IPV6 directive controls identification of the processor. The optional options control secrecy and authentication, the redundant ring mode of operation and maximum network MTU field. .TP version This specifies the version of the configuration file. Currently the only valid version for this directive is 2. .PP clear_node_high_bit This configuration option is optional and is only relevant when no nodeid is specified. Some corosync clients require a signed 32 bit nodeid that is greater than zero however by default corosync uses all 32 bits of the IPv4 address space when generating a nodeid. Set this option to yes to force the high bit to be zero and therefor ensure the nodeid is a positive signed 32 bit integer. WARNING: The clusters behavior is undefined if this option is enabled on only a subset of the cluster (for example during a rolling upgrade). .TP crypto_hash This specifies which HMAC authentication should be used to authenticate all messages. Valid values are none (no authentication), md5, sha1, sha256, sha384 and sha512. The default is sha1. .TP crypto_cipher This specifies which cipher should be used to encrypt all messages. Valid values are none (no encryption), aes256, aes192, aes128 and 3des. Enabling crypto_cipher, requires also enabling of crypto_hash. The default is aes256. .TP secauth This specifies that HMAC/SHA1 authentication should be used to authenticate all messages. It further specifies that all data should be encrypted with the nss library and aes256 encryption algorithm to protect data from eavesdropping. Enabling this option adds a encryption header to every message sent by totem which reduces total throughput. Also encryption and authentication consume extra CPU cycles in corosync. The default is on. WARNING: This parameter is deprecated. It's recomended to use combination of crypto_cipher and crypto_hash. .TP rrp_mode This specifies the mode of redundant ring, which may be none, active, or passive. Currently only 'passive' is supported or tested (using 'active' is not recommended). Active replication offers slightly lower latency from transmit to delivery in faulty network environments but with less performance. Passive replication may nearly double the speed of the totem protocol if the protocol doesn't become cpu bound. The final option is none, in which case only one network interface will be used to operate the totem protocol. If only one interface directive is specified, none is automatically chosen. If multiple interface directives are specified, only active or passive may be chosen. The maximum number of interface directives that is allowed for either modes (active or passive) is 2. When using multiple interfaces, make sure to use different multicast address/port (port for same address must differ by at least two) pair for each interface (this is checked by parser) to make rrp works. .TP netmtu This specifies the network maximum transmit unit. To set this value beyond 1500, the regular frame MTU, requires ethernet devices that support large, or also called jumbo, frames. If any device in the network doesn't support large frames, the protocol will not operate properly. The hosts must also have their mtu size set from 1500 to whatever frame size is specified here. Please note while some NICs or switches claim large frame support, they support 9000 MTU as the maximum frame size including the IP header. Setting the netmtu and host MTUs to 9000 will cause totem to use the full 9000 bytes of the frame. Then Linux will add a 18 byte header moving the full frame size to 9018. As a result some hardware will not operate properly with this size of data. A netmtu of 8982 seems to work for the few large frame devices that have been tested. Some manufacturers claim large frame support when in fact they support frame sizes of 4500 bytes. When sending multicast traffic, if the network frequently reconfigures, chances are that some device in the network doesn't support large frames. Choose hardware carefully if intending to use large frame support. The default is 1500. .TP transport This directive controls the transport mechanism used. If the interface to which corosync is binding is an RDMA interface such as RoCEE or Infiniband, the "iba" parameter may be specified. To avoid the use of multicast entirely, a unicast transport parameter "udpu" can be specified. This requires specifying the list of members in nodelist directive, that could potentially make up the membership before deployment. The default is udp. The transport type can also be set to udpu or iba. .TP cluster_name This specifies the name of cluster and it's used for automatic generating of multicast address. .TP config_version This specifies version of config file. This is converted to unsigned 64-bit int. By default it's 0. Option is used to prevent joining old nodes with not up-to-date configuration. If value is not 0, and node is going for first time (only for first time, join after split doesn't follow this rules) from single-node membership to multiple nodes membership, other nodes config_versions are collected. If current node config_version is not equal to highest of collected versions, corosync is terminated. .TP ip_version Specifies version of IP to use for communication. Value can be one of ipv4 or ipv6. Default (if unspecified) is ipv4. Within the .B totem directive, there are several configuration options which are used to control the operation of the protocol. It is generally not recommended to change any of these values without proper guidance and sufficient testing. Some networks may require larger values if suffering from frequent reconfigurations. Some applications may require faster failure detection times which can be achieved by reducing the token timeout. .TP token This timeout is used directly or as a base for real token timeout calculation (explained in .B token_coefficient section). Token timeout specifies in milliseconds until a token loss is declared after not receiving a token. This is the time spent detecting a failure of a processor in the current configuration. Reforming a new configuration takes about 50 milliseconds in addition to this timeout. For real token timeout used by totem it's possible to read cmap value of .B runtime.config.token key. The default is 1000 milliseconds. .TP token_coefficient This value is used only when .B nodelist section is specified and contains at least 3 nodes. If so, real token timeout is then computed as token + (number_of_nodes - 2) * token_coefficient. This allows cluster to scale without manually changing token timeout every time new node is added. This value can be set to 0 resulting in effective removal of this feature. The default is 650 milliseconds. .TP token_retransmit This timeout specifies in milliseconds after how long before receiving a token the token is retransmitted. This will be automatically calculated if token is modified. It is not recommended to alter this value without guidance from the corosync community. The default is 238 milliseconds. .TP hold This timeout specifies in milliseconds how long the token should be held by the representative when the protocol is under low utilization. It is not recommended to alter this value without guidance from the corosync community. The default is 180 milliseconds. .TP token_retransmits_before_loss_const This value identifies how many token retransmits should be attempted before forming a new configuration. If this value is set, retransmit and hold will be automatically calculated from retransmits_before_loss and token. The default is 4 retransmissions. .TP join This timeout specifies in milliseconds how long to wait for join messages in the membership protocol. The default is 50 milliseconds. .TP send_join This timeout specifies in milliseconds an upper range between 0 and send_join to wait before sending a join message. For configurations with less then 32 nodes, this parameter is not necessary. For larger rings, this parameter is necessary to ensure the NIC is not overflowed with join messages on formation of a new ring. A reasonable value for large rings (128 nodes) would be 80msec. Other timer values must also change if this value is changed. Seek advice from the corosync mailing list if trying to run larger configurations. The default is 0 milliseconds. .TP consensus This timeout specifies in milliseconds how long to wait for consensus to be achieved before starting a new round of membership configuration. The minimum value for consensus must be 1.2 * token. This value will be automatically calculated at 1.2 * token if the user doesn't specify a consensus value. For two node clusters, a consensus larger then the join timeout but less then token is safe. For three node or larger clusters, consensus should be larger then token. There is an increasing risk of odd membership changes, which stil guarantee virtual synchrony, as node count grows if consensus is less than token. The default is 1200 milliseconds. .TP merge This timeout specifies in milliseconds how long to wait before checking for a partition when no multicast traffic is being sent. If multicast traffic is being sent, the merge detection happens automatically as a function of the protocol. The default is 200 milliseconds. .TP downcheck This timeout specifies in milliseconds how long to wait before checking that a network interface is back up after it has been downed. The default is 1000 millseconds. .TP fail_recv_const This constant specifies how many rotations of the token without receiving any of the messages when messages should be received may occur before a new configuration is formed. The default is 2500 failures to receive a message. .TP seqno_unchanged_const This constant specifies how many rotations of the token without any multicast traffic should occur before the hold timer is started. The default is 30 rotations. .TP heartbeat_failures_allowed [HeartBeating mechanism] Configures the optional HeartBeating mechanism for faster failure detection. Keep in mind that engaging this mechanism in lossy networks could cause faulty loss declaration as the mechanism relies on the network for heartbeating. So as a rule of thumb use this mechanism if you require improved failure in low to medium utilized networks. This constant specifies the number of heartbeat failures the system should tolerate before declaring heartbeat failure e.g 3. Also if this value is not set or is 0 then the heartbeat mechanism is not engaged in the system and token rotation is the method of failure detection The default is 0 (disabled). .TP max_network_delay [HeartBeating mechanism] This constant specifies in milliseconds the approximate delay that your network takes to transport one packet from one machine to another. This value is to be set by system engineers and please don't change if not sure as this effects the failure detection mechanism using heartbeat. The default is 50 milliseconds. .TP window_size This constant specifies the maximum number of messages that may be sent on one token rotation. If all processors perform equally well, this value could be large (300), which would introduce higher latency from origination to delivery for very large rings. To reduce latency in large rings(16+), the defaults are a safe compromise. If 1 or more slow processor(s) are present among fast processors, window_size should be no larger then 256000 / netmtu to avoid overflow of the kernel receive buffers. The user is notified of this by the display of a retransmit list in the notification logs. There is no loss of data, but performance is reduced when these errors occur. The default is 50 messages. .TP max_messages This constant specifies the maximum number of messages that may be sent by one processor on receipt of the token. The max_messages parameter is limited to 256000 / netmtu to prevent overflow of the kernel transmit buffers. The default is 17 messages. .TP miss_count_const This constant defines the maximum number of times on receipt of a token a message is checked for retransmission before a retransmission occurs. This parameter is useful to modify for switches that delay multicast packets compared to unicast packets. The default setting works well for nearly all modern switches. The default is 5 messages. .TP rrp_problem_count_timeout This specifies the time in milliseconds to wait before decrementing the problem count by 1 for a particular ring to ensure a link is not marked faulty for transient network failures. The default is 2000 milliseconds. .TP rrp_problem_count_threshold This specifies the number of times a problem is detected with a link before setting the link faulty. Once a link is set faulty, no more data is transmitted upon it. Also, the problem counter is no longer decremented when the problem count timeout expires. A problem is detected whenever all tokens from the proceeding processor have not been received within the rrp_token_expired_timeout. The rrp_problem_count_threshold * rrp_token_expired_timeout should be atleast 50 milliseconds less then the token timeout, or a complete reconfiguration may occur. The default is 10 problem counts. .TP rrp_problem_count_mcast_threshold This specifies the number of times a problem is detected with multicast before setting the link faulty for passive rrp mode. This variable is unused in active rrp mode. The default is 10 times rrp_problem_count_threshold. .TP rrp_token_expired_timeout This specifies the time in milliseconds to increment the problem counter for the redundant ring protocol after not having received a token from all rings for a particular processor. This value will automatically be calculated from the token timeout and problem_count_threshold but may be overridden. It is not recommended to override this value without guidance from the corosync community. The default is 47 milliseconds. .TP rrp_autorecovery_check_timeout This specifies the time in milliseconds to check if the failed ring can be auto-recovered. The default is 1000 milliseconds. .PP Within the .B logging directive, there are several configuration options which are all optional. .PP The following 3 options are valid only for the top level logging directive: .TP timestamp This specifies that a timestamp is placed on all log messages. The default is off. .TP fileline This specifies that file and line should be printed. The default is off. .TP function_name This specifies that the code function name should be printed. The default is off. .PP The following options are valid both for top level logging directive and they can be overridden in logger_subsys entries. .TP to_stderr .TP to_logfile .TP to_syslog These specify the destination of logging output. Any combination of these options may be specified. Valid options are .B yes and .B no. The default is syslog and stderr. Please note, if you are using to_logfile and want to rotate the file, use logrotate(8) with the option .B copytruncate. eg. .ne 18 .RS .nf .ft CW /var/log/corosync.log { missingok compress notifempty daily rotate 7 copytruncate } .ft .fi .RE .TP logfile If the .B to_logfile directive is set to .B yes , this option specifies the pathname of the log file. No default. .TP logfile_priority This specifies the logfile priority for this particular subsystem. Ignored if debug is on. Possible values are: alert, crit, debug (same as debug = on), emerg, err, info, notice, warning. The default is: info. .TP syslog_facility This specifies the syslog facility type that will be used for any messages sent to syslog. options are daemon, local0, local1, local2, local3, local4, local5, local6 & local7. The default is daemon. .TP syslog_priority This specifies the syslog level for this particular subsystem. Ignored if debug is on. Possible values are: alert, crit, debug (same as debug = on), emerg, err, info, notice, warning. The default is: info. .TP debug This specifies whether debug output is logged for this particular logger. Also can contain value trace, what is highest level of debug information. The default is off. .PP Within the .B logging directive, logger_subsys directives are optional. .PP Within the .B logger_subsys sub-directive, all of the above logging configuration options are valid and can be used to override the default settings. The subsys entry, described below, is mandatory to identify the subsystem. .TP subsys This specifies the subsystem identity (name) for which logging is specified. This is the name used by a service in the log_init () call. E.g. 'CPG'. This directive is required. .PP Within the .B quorum directive it is possible to specify the quorum algorithm to use with the .TP provider directive. At the time of writing only corosync_votequorum is supported. See votequorum(5) for configuration options. .PP Within the .B nodelist directive it is possible to specify specific information about nodes in cluster. Directive can contain only .B node sub-directive, which specifies every node that should be a member of the membership, and where non-default options are needed. Every node must have at least ring0_addr field filled. For UDPU, every node that should be a member of the membership must be specified. Possible options are: .TP ringX_addr This specifies ip address of one of the nodes. X is ring number. .TP nodeid This configuration option is optional when using IPv4 and required when using IPv6. This is a 32 bit value specifying the node identifier delivered to the cluster membership service. If this is not specified with IPv4, the node id will be determined from the 32 bit IP address the system to which the system is bound with ring identifier of 0. The node identifier value of zero is reserved and should not be used. .PP Within the .B qb directive it is possible to specify options for libqb. Possible option is: .TP ipc_type This specifies type of IPC to use. Can be one of native (default), shm and socket. Native means one of shm or socket, depending on what is supported by OS. On systems with support for both, SHM is selected. SHM is generally faster, but need to allocate ring buffer file in /dev/shm. .SH "FILES" .TP /etc/corosync/corosync.conf The corosync executive configuration file. .SH "SEE ALSO" .BR corosync_overview (8), .BR votequorum (5), +.BR corosync-qdevice (8), .BR logrotate (8) .PP diff --git a/man/votequorum.5 b/man/votequorum.5 index 7770fb0d..43d7c284 100644 --- a/man/votequorum.5 +++ b/man/votequorum.5 @@ -1,403 +1,404 @@ .\"/* .\" * Copyright (c) 2012-2014 Red Hat, Inc. .\" * .\" * All rights reserved. .\" * .\" * Authors: Christine Caulfield .\" * Fabio M. Di Nitto .\" * .\" * This software licensed under BSD license, the text of which follows: .\" * .\" * Redistribution and use in source and binary forms, with or without .\" * modification, are permitted provided that the following conditions are met: .\" * .\" * - Redistributions of source code must retain the above copyright notice, .\" * this list of conditions and the following disclaimer. .\" * - Redistributions in binary form must reproduce the above copyright notice, .\" * this list of conditions and the following disclaimer in the documentation .\" * and/or other materials provided with the distribution. .\" * - Neither the name of the MontaVista Software, Inc. nor the names of its .\" * contributors may be used to endorse or promote products derived from this .\" * software without specific prior written permission. .\" * .\" * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" .\" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE .\" * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR .\" * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF .\" * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS .\" * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN .\" * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) .\" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF .\" * THE POSSIBILITY OF SUCH DAMAGE. .\" */ .TH VOTEQUORUM 5 2012-01-24 "corosync Man Page" "Corosync Cluster Engine Programmer's Manual" .SH NAME votequorum \- Votequorum Configuration Overview .SH OVERVIEW The votequorum service is part of the corosync project. This service can be optionally loaded into the nodes of a corosync cluster to avoid split-brain situations. It does this by having a number of votes assigned to each system in the cluster and ensuring that only when a majority of the votes are present, cluster operations are allowed to proceed. The service must be loaded into all nodes or none. If it is loaded into a subset of cluster nodes the results will be unpredictable. .PP The following corosync.conf extract will enable votequorum service within corosync: .PP .nf quorum { provider: corosync_votequorum } .fi .PP votequorum reads its configuration from corosync.conf. Some values can be changed at runtime, others are only read at corosync startup. It is very important that those values are consistent across all the nodes participating in the cluster or votequorum behavior will be unpredictable. .PP votequorum requires an expected_votes value to function, this can be provided in two ways. The number of expected votes will be automatically calculated when the nodelist { } section is present in corosync.conf or expected_votes can be specified in the quorum { } section. Lack of both will disable votequorum. If both are present at the same time, the quorum.expected_votes value will override the one calculated from the nodelist. .PP Example (no nodelist) of an 8 node cluster (each node has 1 vote): .nf quorum { provider: corosync_votequorum expected_votes: 8 } .fi .PP Example (with nodelist) of a 3 node cluster (each node has 1 vote): .nf quorum { provider: corosync_votequorum } nodelist { node { ring0_addr: 192.168.1.1 } node { ring0_addr: 192.168.1.2 } node { ring0_addr: 192.168.1.3 } } .fi .SH SPECIAL FEATURES .PP .B two_node: 1 .PP Enables two node cluster operations (default: 0). .PP The "two node cluster" is a use case that requires special consideration. With a standard two node cluster, each node with a single vote, there are 2 votes in the cluster. Using the simple majority calculation (50% of the votes + 1) to calculate quorum, the quorum would be 2. This means that the both nodes would always have to be alive for the cluster to be quorate and operate. .PP Enabling two_node: 1, quorum is set artificially to 1. .PP Example configuration 1: .nf quorum { provider: corosync_votequorum expected_votes: 2 two_node: 1 } .fi .PP Example configuration 2: .nf quorum { provider: corosync_votequorum two_node: 1 } nodelist { node { ring0_addr: 192.168.1.1 } node { ring0_addr: 192.168.1.2 } } .fi .PP NOTES: enabling two_node: 1 automatically enables wait_for_all. It is still possible to override wait_for_all by explicitly setting it to 0. If more than 2 nodes join the cluster, the two_node option is automatically disabled. .PP .B wait_for_all: 1 .PP Enables Wait For All (WFA) feature (default: 0). .PP The general behaviour of votequorum is to switch a cluster from inquorate to quorate as soon as possible. For example, in an 8 node cluster, where every node has 1 vote, expected_votes is set to 8 and quorum is (50% + 1) 5. As soon as 5 (or more) nodes are visible to each other, the partition of 5 (or more) becomes quorate and can start operating. .PP When WFA is enabled, the cluster will be quorate for the first time only after all nodes have been visible at least once at the same time. .PP This feature has the advantage of avoiding some startup race conditions, with the cost that all nodes need to be up at the same time at least once before the cluster can operate. .PP A common startup race condition based on the above example is that as soon as 5 nodes become quorate, with the other 3 still offline, the remaining 3 nodes will be fenced. .PP It is very useful when combined with last_man_standing (see below). .PP Example configuration: .nf quorum { provider: corosync_votequorum expected_votes: 8 wait_for_all: 1 } .fi .PP .B last_man_standing: 1 / .B last_man_standing_window: 10000 .PP Enables Last Man Standing (LMS) feature (default: 0). Tunable last_man_standing_window (default: 10 seconds, expressed in ms). .PP The general behaviour of votequorum is to set expected_votes and quorum at startup (unless modified by the user at runtime, see below) and use those values during the whole lifetime of the cluster. .PP Using for example an 8 node cluster where each node has 1 vote, expected_votes is set to 8 and quorum to 5. This condition allows a total failure of 3 nodes. If a 4th node fails, the cluster becomes inquorate and it will stop providing services. .PP Enabling LMS allows the cluster to dynamically recalculate expected_votes and quorum under specific circumstances. It is essential to enable WFA when using LMS in High Availability clusters. .PP Using the above 8 node cluster example, with LMS enabled the cluster can retain quorum and continue operating by losing, in a cascade fashion, up to 6 nodes with only 2 remaining active. .PP Example chain of events: .nf 1) cluster is fully operational with 8 nodes. (expected_votes: 8 quorum: 5) 2) 3 nodes die, cluster is quorate with 5 nodes. 3) after last_man_standing_window timer expires, expected_votes and quorum are recalculated. (expected_votes: 5 quorum: 3) 4) at this point, 2 more nodes can die and cluster will still be quorate with 3. 5) once again, after last_man_standing_window timer expires expected_votes and quorum are recalculated. (expected_votes: 3 quorum: 2) 6) at this point, 1 more node can die and cluster will still be quorate with 2. 7) one more last_man_standing_window timer (expected_votes: 2 quorum: 2) .fi .PP NOTES: In order for the cluster to downgrade automatically from 2 nodes to a 1 node cluster, the auto_tie_breaker feature must also be enabled (see below). If auto_tie_breaker is not enabled, and one more failure occours, the remaining node will not be quorate. LMS does not work with asymmetric voting schemes, each node must vote 1. .PP Example configuration 1: .nf quorum { provider: corosync_votequorum expected_votes: 8 last_man_standing: 1 } .fi .PP Example configuration 2 (increase timeout to 20 seconds): .nf quorum { provider: corosync_votequorum expected_votes: 8 last_man_standing: 1 last_man_standing_window: 20000 } .fi .PP .B auto_tie_breaker: 1 .PP Enables Auto Tie Breaker (ATB) feature (default: 0). .PP The general behaviour of votequorum allows a simultaneous node failure up to 50% - 1 node, assuming each node has 1 vote. .PP When ATB is enabled, the cluster can suffer up to 50% of the nodes failing at the same time, in a deterministic fashion. By default the cluster partition, or the set of nodes that are still in contact with the node that has the lowest nodeid will remain quorate. The other nodes will be inquorate. This behaviour can be changed by also specifying .PP .B auto_tie_breaker_node: lowest|highest| .PP \'lowest' is the default, 'highest' is similar in that if the current set of nodes contains the highest nodeid then it will remain quorate. Alternatively it is possible to specify a particular node ID or list of node IDs that will be required to maintain quorum. If a (space-separated) list is given, the nodes are evaluated in order, so if the first node is present then it will be used to determine the quorate partition, if that node is not in either half (ie was not in the cluster before the split) then the second node ID will be checked for and so on. .PP Example configuration 1: .nf quorum { provider: corosync_votequorum expected_votes: 8 auto_tie_breaker: 1 auto_tie_breaker_node: lowest } .fi .PP Example configuration 2: .nf quorum { provider: corosync_votequorum expected_votes: 8 auto_tie_breaker: 1 auto_tie_breaker_node: 1 3 5 } .PP .fi .PP .B allow_downscale: 1 .PP Enables allow downscale (AD) feature (default: 0). .PP THIS FEATURE IS INCOMPLETE AND CURRENTLY UNSUPPORTED. .PP The general behaviour of votequorum is to never decrease expected votes or quorum. .PP When AD is enabled, both expected votes and quorum are recalculated when a node leaves the cluster in a clean state (normal corosync shutdown process) down to configured expected_votes. .PP Example use case: .PP .nf 1) N node cluster (where N is any value higher than 3) 2) expected_votes set to 3 in corosync.conf 3) only 3 nodes are running 4) admin requires to increase processing power and adds 10 nodes 5) internal expected_votes is automatically set to 13 6) minimum expected_votes is 3 (from configuration) - up to this point this is standard votequorum behavior - 7) once the work is done, admin wants to remove nodes from the cluster 8) using an ordered shutdown the admin can reduce the cluster size automatically back to 3, but not below 3, where normal quorum operation will work as usual. .fi .PP Example configuration: .nf quorum { provider: corosync_votequorum expected_votes: 3 allow_downscale: 1 } .fi allow_downscale implicitly enabled EVT (see below). .PP .B expected_votes_tracking: 1 .PP Enables Expected Votes Tracking (EVT) feature (default: 0). .PP Expected Votes Tracking stores the highest-seen value of expected votes on disk and uses that as the minimum value for expected votes in the absence of any higher authority (eg a current quorate cluster). This is useful for when a group of nodes becomes detached from the main cluster and after a restart could have enough votes to provide quorum, which can happen after using allow_downscale. .PP Note that even if the in-memory version of expected_votes is reduced, eg by removing nodes or using corosync-quorumtool, the stored value will still be the highest value seen - it never gets reduced. .PP The value is held in the file /var/lib/corosync/ev_tracking which can be deleted if you really do need to reduce the expected votes for any reason, like the node has been moved to a different cluster. .PP .fi .PP .SH VARIOUS NOTES .PP * WFA / LMS / ATB / AD can be used combined together. .PP * In order to change the default votes for a node there are two options: .nf 1) nodelist: nodelist { node { ring0_addr: 192.168.1.1 quorum_votes: 3 } .... } 2) quorum section (deprecated): quorum { provider: corosync_votequorum expected_votes: 2 votes: 2 } .fi In the event that both nodelist and quorum { votes: } are defined, the value from the nodelist will be used. .PP * Only votes, quorum_votes, expected_votes and two_node can be changed at runtime. Everything else requires a cluster restart. .SH BUGS No known bugs at the time of writing. The authors are from outerspace. Deal with it. .SH "SEE ALSO" .BR corosync (8), .BR corosync.conf (5), .BR corosync-quorumtool (8), +.BR corosync-qdevice (8), .BR votequorum_overview (8) .PP