diff --git a/heartbeat/ethmonitor b/heartbeat/ethmonitor index e2c3efa28..ba8574131 100755 --- a/heartbeat/ethmonitor +++ b/heartbeat/ethmonitor @@ -1,576 +1,577 @@ #!/bin/sh # # OCF Resource Agent compliant script. # Monitor the vitality of a local network interface. # # Based on the work by Robert Euhus and Lars Marowsky-Bree. # # Transfered from Ipaddr2 into ethmonitor by Alexander Krauth # # Copyright (c) 2011 Robert Euhus, Alexander Krauth, Lars Marowsky-Brée # All Rights Reserved. # # This program is free software; you can redistribute it and/or modify # it under the terms of version 2 of the GNU General Public License as # published by the Free Software Foundation. # # This program is distributed in the hope that it would be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # # Further, this software is distributed without any warranty that it is # free of the rightful claim of any third person regarding infringement # or the like. Any license provided herein, whether implied or # otherwise, applies only to this software file. Patent licenses, if # any, provided herein do not apply to combinations of this program with # other software, or any other product whatsoever. # # You should have received a copy of the GNU General Public License # along with this program; if not, write the Free Software Foundation, # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. # # OCF parameters are as below # # OCF_RESKEY_interface # OCF_RESKEY_multiplicator # OCF_RESKEY_name # OCF_RESKEY_repeat_count # OCF_RESKEY_repeat_interval # OCF_RESKEY_pktcnt_timeout # OCF_RESKEY_arping_count # OCF_RESKEY_arping_timeout # OCF_RESKEY_arping_cache_entries # # TODO: Check against IPv6 # ####################################################################### # Initialization: : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat} . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs # Parameter defaults OCF_RESKEY_interface_default="" OCF_RESKEY_name_default="" OCF_RESKEY_multiplier_default="1" OCF_RESKEY_repeat_count_default="5" OCF_RESKEY_repeat_interval_default="10" OCF_RESKEY_pktcnt_timeout_default="5" OCF_RESKEY_arping_count_default="1" OCF_RESKEY_arping_timeout_default="1" OCF_RESKEY_arping_cache_entries_default="5" OCF_RESKEY_link_status_only_default="false" : ${OCF_RESKEY_interface=${OCF_RESKEY_interface_default}} : ${OCF_RESKEY_name=${OCF_RESKEY_name_default}} : ${OCF_RESKEY_multiplier=${OCF_RESKEY_multiplier_default}} : ${OCF_RESKEY_repeat_count=${OCF_RESKEY_repeat_count_default}} : ${OCF_RESKEY_repeat_interval=${OCF_RESKEY_repeat_interval_default}} : ${OCF_RESKEY_pktcnt_timeout=${OCF_RESKEY_pktcnt_timeout_default}} : ${OCF_RESKEY_arping_count=${OCF_RESKEY_arping_count_default}} : ${OCF_RESKEY_arping_timeout=${OCF_RESKEY_arping_timeout_default}} : ${OCF_RESKEY_arping_cache_entries=${OCF_RESKEY_arping_cache_entries_default}} : ${OCF_RESKEY_link_status_only=${OCF_RESKEY_link_status_only_default}} ####################################################################### meta_data() { cat < 1.0 Monitor the vitality of a local network interface. You may set up this RA as a clone resource to monitor the network interfaces on different nodes, with the same interface name. This is not related to the IP address or the network on which a interface is configured. You may use this RA to move resources away from a node, which has a faulty interface or prevent moving resources to such a node. This gives you independent control of the resources, without involving cluster intercommunication. But it requires your nodes to have more than one network interface. The resource configuration requires a monitor operation, because the monitor does the main part of the work. In addition to the resource configuration, you need to configure some location constraints, based on a CIB attribute value. The name of the attribute value is configured in the 'name' option of this RA. Example constraint configuration using crmsh location loc_connected_node my_resource_grp \ rule $id="rule_loc_connected_node" -INF: ethmonitor eq 0 Example constraint configuration using pcs. Only allow 'my_resource' to run on nodes where eth0 ethernet device is available. pcs constraint location my_resource rule score=-INFINITY ethmonitor-eth0 ne 1 The ethmonitor works in 3 different modes to test the interface vitality. 1. call ip to see if the link status is up (if link is down -> error) 2. call ip and watch the RX counter (if packages come around in a certain time -> success) 3. call arping to check whether any of the IPs found in the local ARP cache answers an ARP REQUEST (one answer -> success) 4. return error Monitors network interfaces The name of the network interface which should be monitored (e.g. eth0). Network interface name The name of the CIB attribute to set. This is the name to be used in the constraints. Defaults to "ethmonitor-'interface_name'". Attribute name Multiplier for the value of the CIB attriobute specified in parameter name. Multiplier for result variable Specify how often the interface will be monitored, before the status is set to failed. You need to set the timeout of the monitoring operation to at least repeat_count * repeat_interval Monitor repeat count Specify how long to wait in seconds between the repeat_counts. Monitor repeat interval in seconds Timeout for the RX packet counter. Stop listening for packet counter changes after the given number of seconds. packet counter timeout Number of ARP REQUEST packets to send for every IP. Usually one ARP REQUEST (arping) is send Number of arpings per IP Time in seconds to wait for ARP REQUESTs (all packets of arping_count). This is to limit the time for arp requests, to be able to send requests to more than one node, without running in the monitor operation timeout. Timeout for arpings per IP Maximum number of IPs from ARP cache list to check for ARP REQUEST (arping) answers. Newest entries are tried first. Number of ARP cache entries to try For interfaces that are infiniband devices. infiniband device For infiniband devices, this is the port to monitor. infiniband port Only report success based on link status. Do not perform RX counter or arping related connectivity tests. link status check only END exit $OCF_SUCCESS } # # Return true, if the interface exists # is_interface() { # # List interfaces but exclude FreeS/WAN ipsecN virtual interfaces # local iface=`$IP2UTIL -o -f link addr show | grep -e " $1[:@]" \ | cut -d ' ' -f2 | tr -d ':' | cut -d '@' -f1 | sort -u | grep -v '^ipsec[0-9][0-9]*$'` [ "$iface" != "" ] } infiniband_status() { local device="$OCF_RESKEY_infiniband_device" if [ -n "$OCF_RESKEY_infiniband_port" ]; then device="${OCF_RESKEY_infiniband_device}:${OCF_RESKEY_infiniband_port}" fi case "${OCF_RESKEY_infiniband_device}" in *ib*|*mlx*) ibstatus ${device} | grep -q ACTIVE ;; *hfi*) opainfo | grep -q Active ;; esac } if_init() { local rc if [ X"$OCF_RESKEY_interface" = "X" ]; then ocf_exit_reason "Interface name (the interface parameter) is mandatory" exit $OCF_ERR_CONFIGURED fi NIC="$OCF_RESKEY_interface" if is_interface $NIC then case "$NIC" in *:*) ocf_exit_reason "Do not specify a virtual interface : $OCF_RESKEY_interface" exit $OCF_ERR_CONFIGURED;; *) ;; esac else case $__OCF_ACTION in validate-all) ocf_exit_reason "Interface $NIC does not exist" exit $OCF_ERR_CONFIGURED;; *) ## It might be a bond interface which is temporarily not available, therefore we want to continue here ocf_log warn "Interface $NIC does not exist" ;; esac fi if ! ocf_is_decimal "$OCF_RESKEY_multiplier"; then ocf_exit_reason "Invalid OCF_RESKEY_multiplier [$OCF_RESKEY_multiplier]" exit $OCF_ERR_CONFIGURED fi ATTRNAME=${OCF_RESKEY_name:-"ethmonitor-$NIC"} REP_COUNT=${OCF_RESKEY_repeat_count:-5} if ! ocf_is_decimal "$REP_COUNT" -o [ $REP_COUNT -lt 1 ]; then ocf_exit_reason "Invalid OCF_RESKEY_repeat_count [$REP_COUNT]" exit $OCF_ERR_CONFIGURED fi REP_INTERVAL_S=${OCF_RESKEY_repeat_interval:-10} if ! ocf_is_decimal "$REP_INTERVAL_S"; then ocf_exit_reason "Invalid OCF_RESKEY_repeat_interval [$REP_INTERVAL_S]" exit $OCF_ERR_CONFIGURED fi if ! ocf_is_decimal "$OCF_RESKEY_pktcnt_timeout"; then ocf_exit_reason "Invalid OCF_RESKEY_pktcnt_timeout [$OCF_RESKEY_pktcnt_timeout]" exit $OCF_ERR_CONFIGURED fi if ! ocf_is_decimal "$OCF_RESKEY_arping_count"; then ocf_exit_reason "Invalid OCF_RESKEY_arping_count [$OCF_RESKEY_arping_count]" exit $OCF_ERR_CONFIGURED fi if ! ocf_is_decimal "$OCF_RESKEY_arping_timeout"; then ocf_exit_reason "Invalid OCF_RESKEY_arping_timeout [$OCF_RESKEY_arping_count]" exit $OCF_ERR_CONFIGURED fi if ! ocf_is_decimal "$OCF_RESKEY_arping_cache_entries"; then ocf_exit_reason "Invalid OCF_RESKEY_arping_cache_entries [$OCF_RESKEY_arping_cache_entries]" exit $OCF_ERR_CONFIGURED fi if [ -n "$OCF_RESKEY_infiniband_device" ]; then #ibstatus or opainfo is required if an infiniband_device is provided case "${OCF_RESKEY_infiniband_device}" in *ib*|*mlx*) check_binary ibstatus ;; *hfi*) check_binary opainfo ;; esac fi return $OCF_SUCCESS } # get the link status on $NIC # asks ip about running (up) interfaces, returns the number of matching interface names that are up get_link_status () { $IP2UTIL -o link show up dev "$NIC" | grep -v 'NO-CARRIER' | grep -c "$NIC" } # returns the number of received rx packets on $NIC get_rx_packets () { ocf_log debug "$IP2UTIL -o -s link show dev $NIC" $IP2UTIL -o -s link show dev "$NIC" \ | sed 's/.* RX: [^0-9]*[0-9]* *\([0-9]*\) .*/\1/' # the first number after RX: is the # of bytes , # the second is the # of packets received } # watch for packet counter changes for max. OCF_RESKEY_pktcnt_timeout seconds # returns immedeately with return code 0 if any packets were received # otherwise 1 is returned watch_pkt_counter () { local RX_PACKETS_NEW local RX_PACKETS_OLD RX_PACKETS_OLD="`get_rx_packets`" for n in `seq $(( $OCF_RESKEY_pktcnt_timeout * 10 ))`; do sleep 0.1 RX_PACKETS_NEW="`get_rx_packets`" ocf_log debug "RX_PACKETS_OLD: $RX_PACKETS_OLD RX_PACKETS_NEW: $RX_PACKETS_NEW" if [ "$RX_PACKETS_OLD" -ne "$RX_PACKETS_NEW" ]; then ocf_log debug "we received some packets." return 0 fi done return 1 } # returns list of cached ARP entries for $NIC # sorted by age ("last confirmed") # max. OCF_RESKEY_arping_cache_entries entries get_arp_list () { $IP2UTIL -s neighbour show dev $NIC \ | sort -t/ -k2,2n | cut -d' ' -f1 \ | head -n $OCF_RESKEY_arping_cache_entries # the "used" entries in `ip -s neighbour show` are: # "last used"/"last confirmed"/"last updated" } # arping the IP given as argument $1 on $NIC # until OCF_RESKEY_arping_count answers are received do_arping () { # TODO: add the source IP # TODO: check for diffenrent arping versions out there arping -q -c $OCF_RESKEY_arping_count -w $OCF_RESKEY_arping_timeout -I $NIC $1 # return with the exit code of the arping command return $? } # # Check the interface depending on the level given as parameter: $OCF_RESKEY_check_level # # 09: check for nonempty ARP cache # 10: watch for packet counter changes # # 19: check arping_ip_list # 20: check arping ARP cache entries # # 30: watch for packet counter changes in promiscios mode # # If unsuccessfull in levels 18 and above, # the tests for higher check levels are run. # if_check () { local arp_list # always check link status first link_status="`get_link_status`" ocf_log debug "link_status: $link_status (1=up, 0=down)" if [ $link_status -eq 0 ]; then ocf_log notice "link_status: DOWN" return $OCF_NOT_RUNNING fi # if this is an infiniband device, try ibstatus script if [ -n "$OCF_RESKEY_infiniband_device" ]; then if infiniband_status; then return $OCF_SUCCESS fi ocf_log info "Infiniband device $OCF_RESKEY_infiniband_device is not available, check ibstatus for more information" return $OCF_NOT_RUNNING fi # if using link_status_only, skip RX count and arping related tests if ocf_is_true "$OCF_RESKEY_link_status_only"; then return $OCF_SUCCESS fi # watch for packet counter changes ocf_log debug "watch for packet counter changes" watch_pkt_counter if [ $? -eq 0 ]; then return $OCF_SUCCESS else ocf_log debug "No packets received during packet watch timeout" fi # check arping ARP cache entries ocf_log debug "check arping ARP cache entries" arp_list=`get_arp_list` for ip in `echo $arp_list`; do do_arping $ip && return $OCF_SUCCESS done # if we get here, the ethernet device is considered not running. # provide some logging information if [ -z "$arp_list" ]; then ocf_log info "No ARP cache entries found to arping" fi # watch for packet counter changes in promiscios mode # ocf_log debug "watch for packet counter changes in promiscios mode" # be sure switch off promiscios mode in any case # TODO: check first, wether promisc is already on and leave it untouched. # trap "$IP2UTIL link set dev $NIC promisc off; exit" INT TERM EXIT # $IP2UTIL link set dev $NIC promisc on # watch_pkt_counter && return $OCF_SUCCESS # $IP2UTIL link set dev $NIC promisc off # trap - INT TERM EXIT # looks like it's not working (for whatever reason) return $OCF_NOT_RUNNING } ####################################################################### if_usage() { cat < /dev/null` sleep $sleep_time 2> /dev/null runs=$(($runs + 1)) fi if [ $mon_rc -eq $OCF_SUCCESS -a $runs -ne 0 ]; then ocf_log info "Monitoring of $OCF_RESOURCE_INSTANCE recovered from error" fi done ocf_log debug "Monitoring return code: $mon_rc" if [ $mon_rc -eq $OCF_SUCCESS ]; then set_cib_value 1 attr_rc=$? else ocf_log err "Monitoring of $OCF_RESOURCE_INSTANCE failed." set_cib_value 0 attr_rc=$? fi ## The resource should not fail, if the interface is down. It should fail, if the update of the CIB variable has errors. ## To react on the interface failure you must use constraints based on the CIB variable value, not on the resource itself. exit $attr_rc } if_stop() { attrd_updater -D -n $ATTRNAME ha_pseudo_resource $OCF_RESOURCE_INSTANCE stop } if_start() { local rc ha_pseudo_resource $OCF_RESOURCE_INSTANCE start rc=$? if [ $rc -ne $OCF_SUCCESS ]; then ocf_exit_reason "Failure to create ethmonitor state file" return $rc fi # perform the first monitor during the start operation if_monitor return $? } if_validate() { check_binary $IP2UTIL check_binary arping + check_binary bc if_init } case $__OCF_ACTION in meta-data) meta_data ;; usage|help) if_usage exit $OCF_SUCCESS ;; esac if_validate case $__OCF_ACTION in start) if_start exit $? ;; stop) if_stop exit $? ;; monitor|status) if_monitor exit $? ;; validate-all) exit $? ;; *) if_usage exit $OCF_ERR_UNIMPLEMENTED ;; esac diff --git a/heartbeat/mariadb.in b/heartbeat/mariadb.in index 3b6feb6f8..39ad191bb 100644 --- a/heartbeat/mariadb.in +++ b/heartbeat/mariadb.in @@ -1,1058 +1,1062 @@ #!@BASH_SHELL@ # # # MariaDB # # Description: Manages a MariaDB Master/Slave database as Linux-HA resource # # Authors: Alan Robertson: DB2 Script # Jakub Janczak: rewrite as MySQL # Andrew Beekhof: cleanup and import # Sebastian Reitenbach: add OpenBSD defaults, more cleanup # Narayan Newton: add Gentoo/Debian defaults # Marian Marinov, Florian Haas: add replication capability # Yves Trudeau, Baron Schwartz: add VIP support and improve replication # Nils Carlson: add GTID support and semi-sync support # # Support: users@clusterlabs.org # License: GNU General Public License (GPL) # # (c) 2002-2005 International Business Machines, Inc. # 2005-2010 Linux-HA contributors # # See usage() function below for more details... # # OCF instance parameters: # OCF_RESKEY_binary # OCF_RESKEY_client_binary # OCF_RESKEY_config # OCF_RESKEY_datadir # OCF_RESKEY_user # OCF_RESKEY_group # OCF_RESKEY_node_list # OCF_RESKEY_test_table # OCF_RESKEY_test_user # OCF_RESKEY_test_passwd # OCF_RESKEY_enable_creation # OCF_RESKEY_additional_parameters # OCF_RESKEY_log # OCF_RESKEY_pid # OCF_RESKEY_socket # OCF_RESKEY_replication_user # OCF_RESKEY_replication_passwd # OCF_RESKEY_replication_port ####################################################################### # Initialization: OCF_RESKEY_node_list_default="" : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat} . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs . ${OCF_FUNCTIONS_DIR}/mysql-common.sh ####################################################################### usage() { cat < 1.0 Resource script for MariaDB. Manages a complete master/slave replication setup with GTID, for simpler uses look at the mysql resource agent which supports older replication forms which mysql and mariadb have in common. The resource must be setup to use notifications. Set 'notify=true' in the metadata attributes when defining a MariaDB master/slave instance. The default behavior is to use uname -n values in the change master to command. Other IPs can be specified manually by adding a node attribute \${INSTANCE_ATTR_NAME}_mysql_master_IP giving the IP to use for replication. For example, if the mariadb primitive you are using is p_mariadb, the attribute to set will be p_mariadb_mysql_master_IP. Manages a MariaDB master/slave instance Location of the MariaDB server binary MariaDB server binary Location of the MariaDB client binary MariaDB client binary Configuration file MariaDB config Directory containing databases MariaDB datadir User running MariaDB daemon MariaDB user Group running MariaDB daemon (for logfile and directory permissions) MariaDB group The logfile to be used for mysqld. MariaDB log file All node names of nodes that will execute mariadb. Please separate each node name with a space. This is required for the master selection to function. node list The pidfile to be used for mysqld. MariaDB pid file The socket to be used for mysqld. MariaDB socket Table to be tested in monitor statement (in database.table notation) MariaDB test table MariaDB test user, must have select privilege on test_table MariaDB test user MariaDB test user password MariaDB test user password If the MariaDB database does not exist, it will be created Create the database if it does not exist Additional parameters which are passed to the mysqld on startup. (e.g. --skip-external-locking or --skip-grant-tables) Additional parameters to pass to mysqld MariaDB replication user. This user is used for starting and stopping MariaDB replication, for setting and resetting the master host, and for setting and unsetting read-only mode. Because of that, this user must have SUPER, REPLICATION SLAVE, REPLICATION CLIENT, PROCESS and RELOAD privileges on all nodes within the cluster. Mandatory if you define a master-slave resource. MariaDB replication user MariaDB replication password. Used for replication client and slave. Mandatory if you define a master-slave resource. MariaDB replication user password The port on which the Master MariaDB instance is listening. MariaDB replication port END } # Convenience functions greater_than_equal_long() { # there are values we need to compare in this script # that are too large for shell -gt to process local true=$(echo "$1 > $2" | bc) if [ "$true" -eq "1" ]; then return 0 else return 1 fi } greater_than_gtid() { local gtid1_transaction_id=$(echo $1 | cut -d - -f 3) local gtid2_transaction_id=$(echo $2 | cut -d - -f 3) greater_than_equal_long $gtid1_transaction_id $gtid2_transaction_id return $? } set_gtid() { # Sets the GTID in CIB using attrd_updater for this node. local gtid=$($MYSQL $MYSQL_OPTIONS_REPL \ -s -N -e "show global variables like 'gtid_current_pos'" | cut -f 2) # Ensure that we got somethine like a valid GTID if ! echo $gtid | grep -q '-'; then ocf_exit_reason "Unable to read GTID from MariaDB" ocf_log err "Unable to read GTID from MariaDB" return $OCF_ERR_GENERIC fi ${HA_SBIN_DIR}/attrd_updater -p -n ${OCF_RESOURCE_INSTANCE}-gtid -U $gtid } read_gtid() { local node=$1 local query_result local name local host local value # This produces output of the form 'name="var-name" host="node2" value="val"'. # This should be set at this point, because we have store our own GTID previously. if ! query_result=$(${HA_SBIN_DIR}/attrd_updater -p -N $node -n ${OCF_RESOURCE_INSTANCE}-gtid -Q); then ocf_exit_reason "Unable to read GTID from attrd" ocf_log err "Unable to read GTID from attrd" echo "" return fi # Evaluate the query result to place the variables in the local scope. eval ${query_result} echo ${value} } clear_all_gtid() { for node in $OCF_RESKEY_node_list; do ${HA_SBIN_DIR}/attrd_updater -n ${OCF_RESOURCE_INSTANCE}-gtid -N $node -D done } set_waiting_for_first_master() { ${HA_SBIN_DIR}/attrd_updater -p -n ${OCF_RESOURCE_INSTANCE}-waiting-for-first-master -U true } waiting_for_first_master() { local query_result local name local host local value if ! query_result=$(${HA_SBIN_DIR}/attrd_updater -p -n ${OCF_RESOURCE_INSTANCE}-waiting-for-first-master -Q); then ocf_exit_reason "Unable to read waiting-for-first-master from attrd" ocf_log err "Unable to read waiting-for-first-master from attrd" return 1 fi # Evaluate the query result to place the variables in the local scope. eval ${query_result} if [ "$value" = "true" ]; then return 0 else return 1 fi } clear_waiting_for_first_master() { attrd_updater -n ${OCF_RESOURCE_INSTANCE}-waiting-for-first-master -D } have_master_with_priority() { # Go through each node and validate that at least one has # a set priority. Because we unset the priority on reboot # a lack of priority indicates that we need to select a # new master. for node in $OCF_RESKEY_node_list; do $CRM_MASTER -G -N $node >/dev/null 2>&1 rc=$? if [ $rc -eq 0 ]; then return 0 fi done return 1 } attempt_to_set_master() { ocf_log info "Attempting to set master" local expected_node_count if waiting_for_first_master; then # Wait for all nodes to come online expected_node_count=$OCF_RESKEY_CRM_meta_clone_max else # We accept one node being down. This is not arbitrary, # synchronous replication requires acknowledgement from # at least one host, which means only two nodes must have # the latest GTID. So a set of n - 1 ensures that we do # not lose any writes. expected_node_count=$(($OCF_RESKEY_CRM_meta_clone_max-1)) fi # Set the gtid for this node, making it available to other nodes set_gtid local node_count=0 local highest_gtid=0 local master_candidate="" for node in $OCF_RESKEY_node_list; do local node_gtid=$(read_gtid $node) if [ -z "$node_gtid" ]; then continue fi # Got a valid gtid, increment node count node_count=$(($node_count+1)) # Check if this is a good master candidate if greater_than_gtid $node_gtid $highest_gtid; then master_candidate=$node highest_gtid=$node_gtid fi done # If we managed to query a sufficient number of nodes # then set a master if [ $node_count -ge $expected_node_count ]; then ocf_log info "Promoting $master_candidate to master, highest gtid $highest_gtid, queried $node_count nodes." $CRM_MASTER -v 100 -N $master_candidate else ocf_log info "Not enough nodes ($node_count) contributed to select a master, need $expected_node_count nodes." fi } set_read_only() { # Sets or unsets read-only mode. Accepts one boolean as its # optional argument. If invoked without any arguments, defaults to # enabling read only mode. Should only be set in master/slave # setups. # Returns $OCF_SUCCESS if the operation succeeds, or # $OCF_ERR_GENERIC if it fails. local ro_val if ocf_is_true $1; then ro_val="on" else ro_val="off" fi ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "SET GLOBAL read_only=${ro_val}" } get_read_only() { # Check if read-only is set local read_only_state read_only_state=$($MYSQL $MYSQL_OPTIONS_REPL \ -e "SHOW VARIABLES" | grep -w read_only | awk '{print $2}') if [ "$read_only_state" = "ON" ]; then return 0 else return 1 fi } is_slave() { # Determine whether the machine is currently running as a MariaDB # slave, as determined per SHOW SLAVE STATUS. Returns 1 if SHOW # SLAVE STATUS creates an empty result set, 0 otherwise. local rc # Check whether this machine should be slave if ! get_read_only; then return 1 fi if get_slave_info; then # show slave status is not empty # Is the slave sql thread running, then we are a slave! if [ "$slave_sql" == 'Yes' ]; then return 0 else return 1 fi else # "SHOW SLAVE STATUS" returns an empty set if instance is not a # replication slave return 1 fi } parse_slave_info() { # Extracts field $1 from result of "SHOW SLAVE STATUS\G" from file $2 sed -ne "s/^.* $1: \(.*\)$/\1/p" < $2 } get_slave_info() { if [ "$master_log_file" -a "$master_host" ]; then # variables are already defined, get_slave_info has been run before return $OCF_SUCCESS else local tmpfile=$(mktemp ${HA_RSCTMP}/check_slave.${OCF_RESOURCE_INSTANCE}.XXXXXX) $MYSQL $MYSQL_OPTIONS_REPL \ -e 'SHOW SLAVE STATUS\G' > $tmpfile if [ -s $tmpfile ]; then master_host=$(parse_slave_info Master_Host $tmpfile) master_user=$(parse_slave_info Master_User $tmpfile) master_port=$(parse_slave_info Master_Port $tmpfile) master_using_gtid=$(parse_slave_info Using_Gtid $tmpfile) master_log_file=$(parse_slave_info Master_Log_File $tmpfile) slave_sql=$(parse_slave_info Slave_SQL_Running $tmpfile) slave_io=$(parse_slave_info Slave_IO_Running $tmpfile) last_errno=$(parse_slave_info Last_Errno $tmpfile) last_error=$(parse_slave_info Last_Error $tmpfile) secs_behind=$(parse_slave_info Seconds_Behind_Master $tmpfile) last_io_errno=$(parse_slave_info Last_IO_Errno $tmpfile) last_io_error=$(parse_slave_info Last_IO_Error $tmpfile) ocf_log debug "MariaDB instance running as a replication slave" rm "$tmpfile" else # Instance produced an empty "SHOW SLAVE STATUS" output -- # instance is not a slave rm "$tmpfile" return $OCF_ERR_GENERIC fi return $OCF_SUCCESS fi } check_slave() { # Checks slave status local rc new_master get_slave_info rc=$? if [ $rc -eq 0 ]; then # Check normal errors if [ $last_errno -ne 0 ]; then ocf_exit_reason "MariaDB slave replication has failed ($last_errno): $last_error" exit $OCF_ERR_GENERIC fi # Check IO Errors, ignore 2003 which indicates a connection failure to the master if [ $last_io_errno -ne 0 ] && [ $last_io_errno -ne 2003 ]; then ocf_exit_reason "MariaDB slave io has failed ($last_io_errno): $last_io_error" exit $OCF_ERR_GENERIC fi if [ $last_io_errno -eq 2003 ]; then ocf_log warn "MariaDB master not reachable from slave" fi if [ "$slave_io" != 'Yes' ]; then # Not necessarily a bad thing. The master may have # temporarily shut down, and the slave may just be # reconnecting. A warning can't hurt, though. ocf_log warn "MariaDB Slave IO threads currently not running." # Sanity check, are we at least on the right master new_master=$($CRM_ATTR_REPL_INFO --query -q) if [ "$master_host" != "$new_master" ]; then # Not pointing to the right master, not good, removing the VIPs set_reader_attr 0 exit $OCF_SUCCESS fi fi if [ "$slave_sql" != 'Yes' ]; then # We don't have a replication SQL thread running. Not a # good thing. Try to recoved by restarting the SQL thread # and remove reader vip. Prevent MariaDB restart. ocf_exit_reason "MariaDB Slave SQL threads currently not running." # Remove reader vip set_reader_attr 0 # try to restart slave ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "START SLAVE" # Return success to prevent a restart exit $OCF_SUCCESS fi ocf_log debug "MariaDB instance running as a replication slave" else # Instance produced an empty "SHOW SLAVE STATUS" output -- # instance is not a slave # TODO: Needs to handle when get_slave_info will return too many connections error ocf_exit_reason "check_slave invoked on an instance that is not a replication slave." exit $OCF_ERR_GENERIC fi } set_master() { local new_master=$($CRM_ATTR_REPL_INFO --query -q) # Informs the MariaDB server of the master to replicate # from. Accepts one mandatory argument which must contain the host # name of the new master host. The master must either be unchanged # from the laste master the slave replicated from, or freshly # reset with RESET MASTER. ocf_log info "Changing MariaDB configuration to replicate from $new_master." ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "CHANGE MASTER TO MASTER_HOST='$new_master', \ MASTER_PORT=$OCF_RESKEY_replication_port, \ MASTER_USER='$OCF_RESKEY_replication_user', \ MASTER_PASSWORD='$OCF_RESKEY_replication_passwd', \ MASTER_USE_GTID=current_pos"; } unset_master(){ # Instructs the MariaDB server to stop replicating from a master # host. # If we're currently not configured to be replicating from any # host, then there's nothing to do. But we do log a warning as # no-one but the CRM should be touching the MariaDB master/slave # configuration. if ! is_slave; then ocf_log warn "Attempted to unset the replication master on an instance that is not configured as a replication slave" return $OCF_SUCCESS fi # Stop the slave I/O thread and wait for relay log # processing to complete ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "STOP SLAVE IO_THREAD" if [ $? -gt 0 ]; then ocf_exit_reason "Error stopping slave IO thread" exit $OCF_ERR_GENERIC fi local tmpfile=$(mktemp ${HA_RSCTMP}/threads.${OCF_RESOURCE_INSTANCE}.XXXXXX) while true; do $MYSQL $MYSQL_OPTIONS_REPL \ -e 'SHOW PROCESSLIST\G' > $tmpfile if grep -i 'Has read all relay log' $tmpfile >/dev/null; then ocf_log info "MariaDB slave has finished processing relay log" break fi if ! grep -q 'system user' $tmpfile; then ocf_log info "Slave not runnig - not waiting to finish" break fi ocf_log info "Waiting for MariaDB slave to finish processing relay log" sleep 1 done rm -f $tmpfile # Now, stop all slave activity and unset the master host ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "STOP SLAVE" if [ $? -gt 0 ]; then ocf_exit_reason "Error stopping rest slave threads" exit $OCF_ERR_GENERIC fi ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "RESET SLAVE /*!50516 ALL */;" if [ $? -gt 0 ]; then ocf_exit_reason "Failed to reset slave" exit $OCF_ERR_GENERIC fi } # Start replication as slave start_slave() { ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "START SLAVE" } # Set the attribute controlling the readers VIP set_reader_attr() { local curr_attr_value curr_attr_value=$(get_reader_attr) if [ "$curr_attr_value" -ne "$1" ]; then $CRM_ATTR -l reboot --name ${OCF_RESKEY_reader_attribute} -v $1 fi } # get the attribute controlling the readers VIP get_reader_attr() { local attr_value local rc attr_value=$($CRM_ATTR -l reboot --name ${OCF_RESKEY_reader_attribute} --query -q) rc=$? if [ "$rc" -eq "0" ]; then echo $attr_value else echo -1 fi } # Determines what IP address is attached to the current host. The output of the # crm_attribute command looks like this: # scope=nodes name=IP value=10.2.2.161 # If the ${INSTANCE_ATTR_NAME}_MYSQL_MASTER_IP node attribute is not defined, fallback is to uname -n # The ${INSTANCE_ATTR_NAME}_MYSQL_MASTER_IP is the IP address that will be used for the # change master to command. get_local_ip() { local IP IP=$($CRM_ATTR -l forever -n ${INSTANCE_ATTR_NAME}_mysql_master_IP -q -G 2>/dev/null) if [ ! $? -eq 0 ]; then uname -n else echo $IP fi } ####################################################################### # Functions invoked by resource manager actions mysql_monitor() { local rc local status_loglevel="err" # Set loglevel to info during probe if ocf_is_probe; then status_loglevel="info" fi mysql_common_status $status_loglevel rc=$? # If status returned an error, return that immediately if [ $rc -ne $OCF_SUCCESS ]; then return $rc fi # Check if this instance is configured as a slave, and if so # check slave status if is_slave; then if ! check_slave; then return $OCF_ERR_GENERIC fi fi if [ -n "$OCF_RESKEY_test_table" ]; then # Check for test table ocf_run -q $MYSQL $MYSQL_OPTIONS_TEST \ -e "SELECT COUNT(*) FROM $OCF_RESKEY_test_table" rc=$? if [ $rc -ne 0 ]; then ocf_exit_reason "Failed to select from $test_table"; return $OCF_ERR_GENERIC; fi fi # Check if we are in read-only mode and there is no master # with priority then we attempt to select a master if get_read_only && ! have_master_with_priority; then attempt_to_set_master fi if ! get_read_only; then ocf_log debug "MariaDB monitor succeeded (master)"; return $OCF_RUNNING_MASTER else ocf_log debug "MariaDB monitor succeeded"; return $OCF_SUCCESS fi } mysql_start() { local rc if ! ocf_is_ms; then ocf_exit_reason "Resource is not configured as master/slave" return $OCF_ERR_GENERIC fi # Initialize the ReaderVIP attribute, monitor will enable it set_reader_attr 0 mysql_common_status info if [ $? = $OCF_SUCCESS ]; then ocf_log info "MariaDB already running" return $OCF_SUCCESS fi mysql_common_prepare_dirs mysql_common_start --skip-slave-start --log-slave-updates rc=$? if [ $rc != $OCF_SUCCESS ]; then return $rc fi # Enable semi-sync ocf_run -q $MYSQL $MYSQL_OPTIONS_TEST \ -e "SET GLOBAL rpl_semi_sync_slave_enabled='ON', \ rpl_semi_sync_master_enabled='ON', \ rpl_semi_sync_master_wait_no_slave='OFF', \ rpl_semi_sync_master_wait_point='AFTER_SYNC', \ gtid_strict_mode='ON', \ sync_binlog=1, \ sync_master_info=1, \ innodb_flush_log_at_trx_commit=1;" rc=$? if [ $rc -ne 0 ]; then ocf_exit_reason "Failed to enable semi-sync and set variables"; return $OCF_ERR_GENERIC; fi # We're configured as a stateful resource. We must start as # slave by default. At this point we don't know if the CRM has # already promoted a master. So, we simply start in read only # mode and make sure our old score is invalidated. set_read_only on $CRM_MASTER -D # Now, let's see whether there is a master. We might be a new # node that is just joining the cluster, and the CRM may have # promoted a master before. new_master_host=$(echo $OCF_RESKEY_CRM_meta_notify_master_uname|tr -d " ") if [ "$new_master_host" -a "$new_master_host" != ${NODENAME} ]; then set_master start_slave if [ $? -ne 0 ]; then ocf_exit_reason "Failed to start slave" return $OCF_ERR_GENERIC fi else ocf_log info "No MariaDB master present - clearing replication state, setting gtid in attrd, waiting for first master" unset_master set_waiting_for_first_master fi # Initial monitor action if [ -n "$OCF_RESKEY_test_table" -a -n "$OCF_RESKEY_test_user" -a -n "$OCF_RESKEY_test_passwd" ]; then OCF_CHECK_LEVEL=10 fi mysql_monitor rc=$? if [ $rc != $OCF_SUCCESS -a $rc != $OCF_RUNNING_MASTER ]; then ocf_exit_reason "Failed initial monitor action" return $rc fi ocf_log info "MariaDB started" return $OCF_SUCCESS } mysql_stop() { # clear preference for becoming master $CRM_MASTER -D # Remove VIP capability set_reader_attr 0 mysql_common_stop } mysql_promote() { local master_info if ( ! mysql_common_status err ); then return $OCF_NOT_RUNNING fi ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "STOP SLAVE" set_read_only off || return $OCF_ERR_GENERIC # Force the master to wait for timeout period on slave disconnect ocf_run -q $MYSQL $MYSQL_OPTIONS_TEST \ -e "SET GLOBAL rpl_semi_sync_master_wait_no_slave='ON';" # Set Master Info in CIB, cluster level attribute master_info="$(get_local_ip)" ${CRM_ATTR_REPL_INFO} -v "$master_info" # A master can accept reads set_reader_attr 1 # Clear the gtids in attrd now that there is a master clear_all_gtid return $OCF_SUCCESS } mysql_demote() { if ! mysql_common_status err; then return $OCF_NOT_RUNNING fi # Return to default no wait setting. ocf_run -q $MYSQL $MYSQL_OPTIONS_TEST \ -e "SET GLOBAL rpl_semi_sync_master_wait_no_slave='OFF';" # Return master preference to default, so the cluster manager gets # a chance to select a new master $CRM_MASTER -D } mysql_notify() { local type_op type_op="${OCF_RESKEY_CRM_meta_notify_type}-${OCF_RESKEY_CRM_meta_notify_operation}" ocf_log debug "Received $type_op notification." case "$type_op" in 'pre-promote') # A master is now being promoted, remove the waiting-for-first-master flag clear_waiting_for_first_master ;; 'post-promote') # The master has completed its promotion. Now is a good # time to check whether our replication slave is working # correctly. new_master_host=$(echo $OCF_RESKEY_CRM_meta_notify_promote_uname|tr -d " ") if [ "$new_master_host" = ${NODENAME} ]; then ocf_log info "This will be the new master, ignoring post-promote notification." else ocf_log info "Resetting replication, uname of master: $new_master_host" unset_master if [ $? -ne 0 ]; then return $OCF_ERR_GENERIC fi set_master if [ $? -ne 0 ]; then return $OCF_ERR_GENERIC fi start_slave if [ $? -ne 0 ]; then ocf_exit_reason "Failed to start slave" return $OCF_ERR_GENERIC fi fi return $OCF_SUCCESS ;; 'pre-demote') demote_host=$(echo $OCF_RESKEY_CRM_meta_notify_demote_uname|tr -d " ") if [ $demote_host = ${NODENAME} ]; then ocf_log info "pre-demote notification for $demote_host" set_read_only on if [ $? -ne 0 ]; then ocf_exit_reason "Failed to set read-only"; return $OCF_ERR_GENERIC; fi # Must kill all existing user threads because they are still Read/write # in order for the slaves to complete the read of binlogs local tmpfile=$(mktemp ${HA_RSCTMP}/threads.${OCF_RESOURCE_INSTANCE}.XXXXXX) $MYSQL $MYSQL_OPTIONS_REPL -e "SHOW PROCESSLIST" > $tmpfile for thread in $(awk '$0 !~ /Binlog Dump|system user|event_scheduler|SHOW PROCESSLIST/ && $0 ~ /^[0-9]/ {print $1}' $tmpfile) do ocf_run $MYSQL $MYSQL_OPTIONS_REPL \ -e "KILL ${thread}" done rm -f $tmpfile else ocf_log info "Ignoring post-demote notification execpt for my own demotion." fi return $OCF_SUCCESS ;; 'post-demote') demote_host=$(echo $OCF_RESKEY_CRM_meta_notify_demote_uname|tr -d " ") if [ $demote_host = ${NODENAME} ]; then ocf_log info "Ignoring post-demote notification for my own demotion." return $OCF_SUCCESS fi ocf_log info "post-demote notification for $demote_host." # The former master has just been gracefully demoted. unset_master ;; *) return $OCF_SUCCESS ;; esac } +mysql_validate() { + check_binary bc +} + ####################################################################### ########################################################################## # If DEBUG_LOG is set, make this resource agent easy to debug: set up the # debug log and direct all output to it. Otherwise, redirect to /dev/null. # The log directory must be a directory owned by root, with permissions 0700, # and the log must be writable and not a symlink. ########################################################################## DEBUG_LOG="/tmp/mysql.ocf.ra.debug/log" if [ "${DEBUG_LOG}" -a -w "${DEBUG_LOG}" -a ! -L "${DEBUG_LOG}" ]; then DEBUG_LOG_DIR="${DEBUG_LOG%/*}" if [ -d "${DEBUG_LOG_DIR}" ]; then exec 9>>"$DEBUG_LOG" exec 2>&9 date >&9 echo "$*" >&9 env | grep OCF_ | sort >&9 set -x else exec 9>/dev/null fi fi case "$1" in meta-data) meta_data exit $OCF_SUCCESS;; usage|help) usage exit $OCF_SUCCESS;; esac mysql_common_validate rc=$? LSB_STATUS_STOPPED=3 if [ $rc -ne 0 ]; then case "$1" in stop) ;; monitor) mysql_common_status "info" if [ $? -eq $OCF_SUCCESS ]; then # if validatation fails and pid is active, always treat this as an error ocf_exit_reason "environment validation failed, active pid is in unknown state." exit $OCF_ERR_GENERIC fi # validation failed and pid is not active, it's safe to say this instance is inactive. exit $OCF_NOT_RUNNING;; status) exit $LSB_STATUS_STOPPED;; *) exit $rc;; esac fi # What kind of method was invoked? case "$1" in start) mysql_start;; stop) mysql_stop;; status) mysql_common_status err;; monitor) mysql_monitor;; promote) mysql_promote;; demote) mysql_demote;; notify) mysql_notify;; - validate-all) exit $OCF_SUCCESS;; + validate-all) mysql_validate;; *) usage exit $OCF_ERR_UNIMPLEMENTED;; esac # vi:sw=4:ts=4:et: diff --git a/heartbeat/ovsmonitor b/heartbeat/ovsmonitor index d03902ea8..872ce86eb 100755 --- a/heartbeat/ovsmonitor +++ b/heartbeat/ovsmonitor @@ -1,468 +1,469 @@ #!/bin/sh # # OCF Resource Agent compliant script. # Monitor the vitality of a local OpenVSwitch bond. # # Based on the work by Alexander Krauth. # # Transfered from ethmonitor into ovsmonitor by Mathieu Grzybek. # # Copyright (c) 2017 Robert Euhus, Alexander Krauth, Lars Marowsky-Bré # Mathieu Grzybek # All Rights Reserved. # # This program is free software; you can redistribute it and/or modify # it under the terms of version 2 of the GNU General Public License as # published by the Free Software Foundation. # # This program is distributed in the hope that it would be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # # Further, this software is distributed without any warranty that it is # free of the rightful claim of any third person regarding infringement # or the like. Any license provided herein, whether implied or # otherwise, applies only to this software file. Patent licenses, if # any, provided herein do not apply to combinations of this program with # other software, or any other product whatsoever. # # You should have received a copy of the GNU General Public License # along with this program; if not, write the Free Software Foundation, # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. # # OCF parameters are as below # # OCF_RESKEY_bond # OCF_RESKEY_bridge # OCF_RESKEY_multiplicator # OCF_RESKEY_name # OCF_RESKEY_repeat_count # OCF_RESKEY_repeat_interval # OCF_RESKEY_pktcnt_timeout # ####################################################################### # Initialization: : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat} . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs # Parameter defaults OCF_RESKEY_bond_default="" OCF_RESKEY_bridge_default="" OCF_RESKEY_name_default="" OCF_RESKEY_multiplier_default="1" OCF_RESKEY_repeat_count_default="5" OCF_RESKEY_repeat_interval_default="10" OCF_RESKEY_pktcnt_timeout_default="5" OCF_RESKEY_link_status_only_default="false" : ${OCF_RESKEY_bond=${OCF_RESKEY_bond_default}} : ${OCF_RESKEY_bridge=${OCF_RESKEY_bridge_default}} : ${OCF_RESKEY_name=${OCF_RESKEY_name_default}} : ${OCF_RESKEY_multiplier=${OCF_RESKEY_multiplier_default}} : ${OCF_RESKEY_repeat_count=${OCF_RESKEY_repeat_count_default}} : ${OCF_RESKEY_repeat_interval=${OCF_RESKEY_repeat_interval_default}} : ${OCF_RESKEY_pktcnt_timeout=${OCF_RESKEY_pktcnt_timeout_default}} : ${OCF_RESKEY_link_status_only=${OCF_RESKEY_link_status_only_default}} ####################################################################### meta_data() { cat < 1.0 Monitor the vitality of a local ovs bond. You may set up this RA as a clone resource to monitor the network bonds on different nodes, with the same bond name. This is not related to the IP address or the network on which a bond is configured. You may use this RA to move resources away from a node, which has a faulty bond or prevent moving resources to such a node. This gives you independent control of the resources, without involving cluster intercommunication. But it requires your nodes to have more than one network bond. The resource configuration requires a monitor operation, because the monitor does the main part of the work. In addition to the resource configuration, you need to configure some location constraints, based on a CIB attribute value. The name of the attribute value is configured in the 'name' option of this RA. Example constraint configuration using crmsh location loc_connected_node my_resource_grp \ rule $id="rule_loc_connected_node" -INF: ovsmonitor-bond-public eq 0 Example constraint configuration using pcs. Only allow 'my_resource' to run on nodes where eth0 ethernet device is available. pcs constraint location my_resource rule score=-INFINITY ovsmonitor-bond-public ne 1 The ethmonitor works in 3 different modes to test the bond vitality. 1. call ovs-appctl to see if at least one of the bonding's link status is up (if link is down -> error) 2. call ovs-ofctl and watch the RX counter (if packages come around in a certain time -> success) 3. return error Monitors ovs bonding bonds The name of the network bond which should be monitored (e.g. bond-public). Bond bond name The name of the ovs bridge that contains the bridge. ovs bridge The name of the CIB attribute to set. This is the name to be used in the constraints. Defaults to "ovsmonitor-'bond_name'". Attribute name Multiplier for the value of the CIB attriobute specified in parameter name. Multiplier for result variable Specify how often the bond will be monitored, before the status is set to failed. You need to set the timeout of the monitoring operation to at least repeat_count * repeat_interval Monitor repeat count Specify how long to wait in seconds between the repeat_counts. Monitor repeat interval in seconds Timeout for the RX packet counter. Stop listening for packet counter changes after the given number of seconds. packet counter timeout Only report success based on link status. Do not perform RX counter related connectivity tests. link status check only END exit $OCF_SUCCESS } # # Return true, if the bond exists # is_bond() { # # List bonds but exclude FreeS/WAN ipsecN virtual bonds # ovs-appctl bond/show $OCF_RESKEY_bond 1>/dev/null 2>&1 } # # Return true, if the bridge exists # is_bridge() { # # List bonds but exclude FreeS/WAN ipsecN virtual bonds # #ovs-appctl bond/show $OCF_RESKEY_bond 1>/dev/null 2>&1 ovs-vsctl show|grep Bridge|grep -q $OCF_RESKEY_bridge } if_init() { local rc if [ X"$OCF_RESKEY_bond" = "X" ]; then ocf_exit_reason "Bond name (the bond parameter) is mandatory" exit $OCF_ERR_CONFIGURED fi if [ X"$OCF_RESKEY_bridge" = "X" ]; then ocf_exit_reason "Bridge name (the bridge parameter) is mandatory" exit $OCF_ERR_CONFIGURED fi BOND="$OCF_RESKEY_bond" BRIDGE="$OCF_RESKEY_bridge" if is_bond then if ! is_bridge then ocf_exit_reason "Bridge $OCF_RESKEY_bond does not exist" exit $OCF_ERR_CONFIGURED; fi else ocf_exit_reason "Bond $OCF_RESKEY_bond does not exist" exit $OCF_ERR_CONFIGURED; fi if ! ocf_is_decimal "$OCF_RESKEY_multiplier"; then ocf_exit_reason "Invalid OCF_RESKEY_multiplier [$OCF_RESKEY_multiplier]" exit $OCF_ERR_CONFIGURED fi ATTRNAME=${OCF_RESKEY_name:-"ovsmonitor-$BOND"} REP_COUNT=${OCF_RESKEY_repeat_count} if ! ocf_is_decimal "$REP_COUNT" -o [ $REP_COUNT -lt 1 ]; then ocf_exit_reason "Invalid OCF_RESKEY_repeat_count [$REP_COUNT]" exit $OCF_ERR_CONFIGURED fi REP_INTERVAL_S=${OCF_RESKEY_repeat_interval} if ! ocf_is_decimal "$REP_INTERVAL_S"; then ocf_exit_reason "Invalid OCF_RESKEY_repeat_interval [$REP_INTERVAL_S]" exit $OCF_ERR_CONFIGURED fi if ! ocf_is_decimal "$OCF_RESKEY_pktcnt_timeout"; then ocf_exit_reason "Invalid OCF_RESKEY_pktcnt_timeout [$OCF_RESKEY_pktcnt_timeout]" exit $OCF_ERR_CONFIGURED fi return $OCF_SUCCESS } # get the link status on $BOND # asks ip about running (up) bonds, returns the number of matching bond names that are up get_link_status () { #$IP2UTIL -o link show up dev "$BOND" | grep -v 'NO-CARRIER' | grep -c "$BOND" ovs-appctl bond/show "$BOND"|awk -F: '/^slave/ {print $2}'|grep -c enabled } # returns the number of received rx packets on $BOND get_rx_packets () { ocf_log debug "bond $BOND - bridge $BRIDGE" #$IP2UTIL -o -s link show dev "$BOND" \ # | sed 's/.* RX: [^0-9]*[0-9]* *\([0-9]*\) .*/\1/' local ovs_port for ovs_port in $(ovs-appctl bond/show $BOND|awk '/^slave/ {gsub(":","");print $2}') ; do ovs-ofctl dump-ports $BRIDGE $ovs_port done \ | awk -F, 'BEGIN{total=0} /rx/ {gsub(".*pkts=","");total=total+int($1)} END{print total}' } # watch for packet counter changes for max. OCF_RESKEY_pktcnt_timeout seconds # returns immedeately with return code 0 if any packets were received # otherwise 1 is returned watch_pkt_counter () { local RX_PACKETS_NEW local RX_PACKETS_OLD RX_PACKETS_OLD="`get_rx_packets`" for n in `seq $(( $OCF_RESKEY_pktcnt_timeout * 10 ))`; do sleep 0.1 RX_PACKETS_NEW="`get_rx_packets`" ocf_log debug "RX_PACKETS_OLD: $RX_PACKETS_OLD RX_PACKETS_NEW: $RX_PACKETS_NEW" if [ "$RX_PACKETS_OLD" -ne "$RX_PACKETS_NEW" ]; then ocf_log debug "we received some packets." return 0 fi done return 1 } # # Check the bond depending on the level given as parameter: $OCF_RESKEY_check_level # # 10: watch for packet counter changes # # # 30: watch for packet counter changes in promiscios mode # # If unsuccessfull in levels 18 and above, # the tests for higher check levels are run. # if_check () { # always check link status first link_status="`get_link_status`" ocf_log debug "link_status: $link_status (up > 0, down = 0)" if [ $link_status -eq 0 ]; then ocf_log notice "link_status: DOWN" return $OCF_NOT_RUNNING fi # if using link_status_only, skip RX count related test if ocf_is_true "$OCF_RESKEY_link_status_only"; then return $OCF_SUCCESS fi # watch for packet counter changes ocf_log debug "watch for packet counter changes" watch_pkt_counter if [ $? -eq 0 ]; then return $OCF_SUCCESS else ocf_log debug "No packets received during packet watch timeout" fi # watch for packet counter changes in promiscios mode # ocf_log debug "watch for packet counter changes in promiscios mode" # be sure switch off promiscios mode in any case # TODO: check first, wether promisc is already on and leave it untouched. # trap "$IP2UTIL link set dev $BOND promisc off; exit" INT TERM EXIT # $IP2UTIL link set dev $BOND promisc on # watch_pkt_counter && return $OCF_SUCCESS # $IP2UTIL link set dev $BOND promisc off # trap - INT TERM EXIT # looks like it's not working (for whatever reason) return $OCF_NOT_RUNNING } ####################################################################### if_usage() { cat < /dev/null` sleep $sleep_time 2> /dev/null runs=$(($runs + 1)) fi if [ $mon_rc -eq $OCF_SUCCESS -a $runs -ne 0 ]; then ocf_log info "Monitoring of $OCF_RESOURCE_INSTANCE recovered from error" fi done ocf_log debug "Monitoring return code: $mon_rc" if [ $mon_rc -eq $OCF_SUCCESS ]; then set_cib_value 1 attr_rc=$? else ocf_log err "Monitoring of $OCF_RESOURCE_INSTANCE failed." set_cib_value 0 attr_rc=$? fi ## The resource should not fail, if the bond is down. It should fail, if the update of the CIB variable has errors. ## To react on the bond failure you must use constraints based on the CIB variable value, not on the resource itself. exit $attr_rc } if_stop() { attrd_updater -D -n $ATTRNAME ha_pseudo_resource $OCF_RESOURCE_INSTANCE stop } if_start() { local rc ha_pseudo_resource $OCF_RESOURCE_INSTANCE start rc=$? if [ $rc -ne $OCF_SUCCESS ]; then ocf_exit_reason "Failure to create ovsmonitor state file" return $rc fi # perform the first monitor during the start operation if_monitor return $? } if_validate() { check_binary ovs-vsctl check_binary ovs-appctl check_binary ovs-ofctl + check_binary bc if_init } case $__OCF_ACTION in meta-data) meta_data ;; usage|help) if_usage exit $OCF_SUCCESS ;; esac if_validate case $__OCF_ACTION in start) if_start exit $? ;; stop) if_stop exit $? ;; monitor|status) if_monitor exit $? ;; validate-all) exit $? ;; *) if_usage exit $OCF_ERR_UNIMPLEMENTED ;; esac