diff --git a/tools/ocft/README.in b/tools/ocft/README.in index e3022c8db..c66b483b0 100644 --- a/tools/ocft/README.in +++ b/tools/ocft/README.in @@ -1,145 +1,147 @@ INTRODUCTION & DESIGN ~~~~~~~~~~~~~~~~~~~~~ - Ocft is a testing tool for resource agents. Instead of the policy of HA, it mainly concerns whether resource agents run correct locally. It can design types of complicated environments to test the reliability of resource agents. Precisely, it is to display whether resource agents can return to correct or expected value. The advantage of the tool provides us with competence to design conditions which can be recorded or reproduced. Hence it is useful to debuggers. * Components ** Test case generator (@sbindir@/ocft) - Turning configuration files of test case to executable scripts. ** Configuration file (@datadir@/@PACKAGE_NAME@/ocft/configs/) - Every configuration file directs only one resource agent and share the same name with resource agent but contains more test cases. ** The testing script (/var/lib/@PACKAGE_NAME@/ocft/cases/) - After the generator reads configuration files and generates many testing scripts and the script is underway, the test begins. * How to customize the environment of testing - Ocft designs the running conditions through two ways, one is changing the environment variables of resource agents (it is the interface left by OCF itself), the other is modifying the OS environment of resource agents, such as altering the permission of some key file or IP address of the machine. * How to test - Firstly, you need to sketch the all complex and uncommon environments against a certain resource agent and keep in mind what consequences may be caused by these uncommon environments. Secondly, write the designed conditions and foreknown consequences into configuration files, and then run the generator to translate the test case to executable scripts. Finally, you need running these scripts to observe the output and learn the running status of each test case, which will compares the predicated result with the actual one. If they differ, you will be able to find the bugs of the resource agent. + - All of the output with test will be recorded into the log files, you can find them + in /var/lib/@PACKAGE_NAME@/ocft/cases/logs. HOW TO WRITE CONFIGURATION FILE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - There are only 6 top level options that are all spelled by capital letters and "-". Every top level option contains sub-options that they are initials. * 'CONFIG' (top level option) - Grammar: CONFIG - The design in this option is global and influences every test case. ** 'Agent' (sub-option) - Grammar: Agent AGENT_NAME - The agent name you want to test. ** 'AgentRoot' (sub-option) - Grammar: AgentRoot /usr/lib/ocf/resource.d/xxx - A few agents will go to "linbit" or "pacemaker" directory, if you define this option, ocft will use it to replace the default directory "heartbeat". ** 'InstallPackage' (sub-option) - Grammar: InstallPackage package [package2 [...]] - It will test whether the system have installed the service of the resource agent. If not, it will download from Internet and have it installed automatically. ** 'HangTimeout' (sub-option) - Grammar: HangTimeout secs - If you alter some key options, some resource agents will get puzzled and stop, which will influence the running of the following test case. Hence timeout setting is needed, if the resource agent stops timeout, the scripts will kill this resource agent. * 'VARIABLE' (top level option) - Garmmar: VARIABLE VAR1=value1 VAR2=value2 ... - Define the global variable here, the variables can be visited everywhere, they can be referenced using $VAR_NAME. Note, the variables in VARIABLE are different from 'Env VAR1=value1', 'Env' can affect the activity of agent, but the variables in VARIABLE just be shared with top level option. * 'SETUP-AGENT' (top level option) - Grammar: SETUP-AGENT bash scripts... ... - Some of Agents may need to be initialized before testing, you can do it here with bash script. * 'CLEANUP-AGENT' (top level option) - Garmmar: CLEANUP-AGENT bash scripts... ... - If SETUP-AGENT set, usually you might be use this option do some cleaning work after test. * 'CASE' & 'CASE-BLOCK' (top level option) - Grammar: CASE "description" & CASE-BLOCK macro_name - Usually, the conditions you designed are more than one and a few 'CASE "..."' will appear in configuration file. It is worth noting that the following sub-options have 2 spellings: One is general, where shell affects the local environment; the other is special, where each options added "@ipaddr". It can remotely execute shell codes. In other words, it is to execute the shell codes from a remote host, which is meaningful when a resource agent needs 2 hosts. This remote shell is not a remote execution only through "ssh", but running a remote shell in the background while the test case is running. The remote shell runs in the background till the end and saves the results during the process. That is to say, you can alternatively carry out local and remote shell code segments. The "CASE-BLOCK" option is a macro definer, the statements in "CASE-BLOCK" will be inserted into "CASE" if you "Include" the "macro_name". ** 'Env' (sub-option) - Grammar: Env VARIABLE=value - It is to set up an environment variable of the resource agent. They usually appear to be OCF_RESKEY_xxx. One point is to be noted is there is no blank by both sides of "=". ** 'Unenv' (sub-option) - Grammer: Unenv VARIABLE [VARIABLE2 [...]] - Remove the environment variable. ** 'Include' (sub-option) - Garmmer: Include macro_name - It will be replaced by statements in 'macro_name', of course, you should define the content of 'macro_name' with 'CASE-BLOCK' first. ** 'Bash' (sub-option) - Grammar: Bash bash_codes - This option is to set up the environment of OS, where you can insert BASH code to customize the system randomly. Note, do not cause unrecoverable consequences to the system. ** 'BashAtExit' (sub-option) - Grammar: BashAtExit bash_codes - This option is to recover the OS environment in order to run another test case correctly. Of cause you can use 'Bash' option to recover it. However, if mistakes occur in the process, the script will quit directly instead of running your recovery codes. If it happens, you ought to use BashAtExit which can restore the system environment before you quit. ** 'RunAgent' (sub-option) - Grammar: RunAgent cmd [ret_value] - This option is to run resource agent. "cmd" is the parameter of the resource agent, such as "start, status, stop ...". The second parameter is optional. It will compare the actual returned value with the expected value when the script has run recourse agent. If differs, bugs will be found. diff --git a/tools/ocft/README.zh_CN.in b/tools/ocft/README.zh_CN.in index cfd79f407..f262ba13f 100644 --- a/tools/ocft/README.zh_CN.in +++ b/tools/ocft/README.zh_CN.in @@ -1,122 +1,124 @@ 1 介绍和设计 ocft是一个测试resource agents的工具。它并不关注HA的策略，而是关注resource agents 是否正常运行在本机。他能对resource agents设计各种复杂环境，来考验resource agnets 是否能正常应对，也就是看resouce agents是否能返回正确的或者说我们所期望的值。 这个工具给我们带来的好处就是我们可以集中的批量的来设计环境，而且这种环境是可被我们 记录、重现的，对Debug人员来说是很有用的。 1.1 组成 1.1.1 解释器 (@sbindir@/ocft) 将test case配置文件转换成可执行的测试脚本。 1.1.2 配置文件 (@datadir@/@PACKAGE_NAME@/ocft/configs/) 每一个configuration file只针对一个resouce agent，配置文件名与resouce agent名相同， 但是它可以容纳很多test case。 1.1.3 测试脚本 (/var/lib/@PACKAGE_NAME@/ocft/cases/) 由generator读取configuration file生成测试脚本，直接运行此脚本就可以开始测试了。 1.2 如何定制环境 ocft 通过两种手段来设计resouce agents的运行环境，一是更改resouce agents的环境变量， 当然这是ocf本身就留给我们的接口。二是更改resource agents所处的系统环境， 比如更改某个关键文件的权限，更改本机ip地址等等... 1.3 如何进行测试 首先你需要针对某个resource agent在脑中勾画各种复杂且异常的环境，并且你能清楚预知 这些异常环境会给resource agent带来什么结果，然后将这些设计好的环境和你预知的结果 都写入配置文件，然后运行generator，将你刚写的test case转换成可执行的脚本。最后运行 这些脚本，观察它们的输出，你可以清楚看到每个test case运行状况，他会比较你的预知结 果和resource agent的实际结果，如果不一样，说明你找到resource agent的bug了。 +所有的测试输出都会被记录到日志文件中,你可以在 /var/lib/@PACKAGE_NAME@/ocft/cases/logs +中找到他们. 2 配置 只有6个top level option，它们是由大写字母和'-'构成的，每个top level option都有若干sub-option， 它们是首字母大写。 2.1 'CONFIG' 选项 语法：CONFIG 此option中的设计是全局的，对每个test case都有所影响。 2.1.1 'Agent' 选项 语法：Agent AGENT_NAME 你要测试的Agent的名字。 2.1.2 'AgentRoot' 选项 语法：AgentRoot /usr/lib/ocf/resource.d/xxx 一些agent将会被移到 "pacemaker" 或 "linbit" 目录，如果你定义了这个选项，ocft将会用它来 替代默认的目录"heartbeat"。 2.1.3 'InstallPackage' 选项 语法:InstallPackage package [package2 [...]] 他会检测系统是否安装了此resource agent的service，如果没有安装，会自动从网络进行安装。 package_name是某个resouce agent在操作系统中必须安装的包。 2.1.4 'HangTimeout' 选项 语法：HangTimeout secs 如果你更改了一些很关键的东西，有些resouce agent会不知所措，停在那里不动，那么就会影响 到后面test case的运行，所以你需要设定超时，如果一旦resouce agent停在那超时了，脚本会 杀死这个resouce agent。 2.2 'VARIABLE' 选项 语法： VARIABLE VAR1=value1 VAR2=value2 ... 在此定义全局变量，这些变量可以用于配置文件中的任何地方，引用时在变量名前加上$。请注意，这些 变量不同于Env定义的变量，Env是定义环境变量，可以改变agent的行为，而这些变量只是用于配置文件中 共享。 2.3 'SETUP-AGENT' 选项 语法： SETUP-AGENT bash scripts... ... 一些Agent在测试前可能需要初始化，你可以用bash脚本在这初始化。 2.4 'CLEANUP-AGENT' 选项 语法： CLEANUP-AGENT bash scripts... ... 如果之前定义了SETUP-AGENT, 你可能还需要此选项在测试完后来作一些清除。 2.5 'CASE' & 'CASE-BLOCK' 选项 语法：CASE "description" & CASE-BLOCK macro_name 通常你设计的环境不止一个，那么配置文件中应该会出现许多 'CASE "..."'，值得注意的是，以下子 选项每个都有两种写法，一是普通写法，产生的shell代码对本地产生作用，二是特殊写法，就是在每 个选项后加上 "@ipaddr"，他可以远程执行shell代码，也就是控制远程机器执行你给的shell代码， 这对需要两台机器以上的resouce agent很有用的，这个远程shell并非简单的用ssh去远程执行，而是在 test case生存期内，后台运行一个远程shell，他始终在后台运行并且保存你的中间结果，也就是说你 可以交替执行本地shell代码段和远程shell代码段。 'CASE-BLOCK'选项是一个宏定义器，它定义的内容将会被插入到"CASE"内容中。 2.5.1 'Env' 选项 语法：Env VARIABLE=value 这是设置resouce agent的环境变量，他们通常是OCF_RESKEY_xxx，注意=号两边不要有空格。 2.5.2 'Unenv' 选项 语法：Unenv VARIABLE 此选项用于删除环境变量。 2.5.3 'Include' 选项 语法：Include macro_name 此选项将会被宏"macro_name"的内容所替代，当然，你得预先用"CASE-BLOCK"来定义宏"macro_name" 的内容。 2.5.4 'Bash' 选项 语法：Bash bash_codes 此选项是用来设置os的环境，你可以嵌入bash代码来对系统作任意设置，不过要注意不要对系统造成 不可恢复的后果就行了。 2.5.5 'BashAtExit' 选项 语法：BashAtExit bash_codes 此选项是用来恢复os的环境，以便另一个test case能正常运行，当然你也可以直接用'Bash'选项来 恢复，但是如果脚本在执行过程中产生某些错误，那么脚本会直接退出，而不会去执行你的恢复代码， 那么你就要用到BashAtExit，他能在你退出前去恢复系统环境。 2.5.6 'RunAgent' 选项 语法：RunAgent cmd [ret_value] 此option会去运行resouce agent，'cmd' 就是resouce agent的参数，如 start，status，stop ... 第二个参数是可选的，它是你对系统环境作出特殊设定以后，你预计resouce agent会返回的值， 如果给出此参数，那么脚本会在运行resouce agent后，会比较此时的返回值和你预期的返回值， 如果不一致，那么说明找到bug了。 diff --git a/tools/ocft/ocft.in b/tools/ocft/ocft.in index 647c580de..22cbf003c 100644 --- a/tools/ocft/ocft.in +++ b/tools/ocft/ocft.in @@ -1,859 +1,863 @@ #!/bin/bash # Copyright (c) 2010-2013 Novell Inc, John Shi # All Rights Reserved. # # This program is free software; you can redistribute it and/or modify # it under the terms of version 2 of the GNU General Public License as # published by the Free Software Foundation. # # This program is distributed in the hope that it would be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # # Further, this software is distributed without any warranty that it is # free of the rightful claim of any third person regarding infringement # or the like. Any license provided herein, whether implied or # otherwise, applies only to this software file. Patent licenses, if # any, provided herein do not apply to combinations of this program with # other software, or any other product whatsoever. # # You should have received a copy of the GNU General Public License # along with this program; if not, write the Free Software Foundation, # Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. die() { local str str="$1" echo "ERROR: $str" >&2 exit 1 } warn() { local str str="$1" echo "WARNING: $str" >&2 } parse_die() { local str str="$1" agent_parse_finish die "${agent}: line ${line_num}: ${str}" } # add quotes to string for Here Documents add_quotes() { local typ str a b typ="$1" str="$2" case "$typ" in 1) a=\'; b=\";; 2) a=\"; b=\';; esac echo "$str" | sed "s/$a/$a$b$a$b$a/g; 1 s/^/$a/; $ s/$/$a/" } # split strings explode() { local str str="$1" echo "$str" | awk -F'"' '{ if (NF > 0 && NF%2 == 0) exit(1); for (i=1; i<=NF; i++) { if (i%2 == 0) print $i; else { len = split($i, str, /[ \t]+/); for (j=1; j<=len; j++) { sb = sub(/#.*/, "", str[j]); if (str[j] != "") print str[j]; if (sb) exit(0); } } } }' } # phase 1: parse the string to 'command' and 'argument collection'. line2trunk() { trunk[0]="${line%%[[:blank:]]*}" trunk[1]="${line#*[[:blank:]]}" } # phase 2: split the argument collection. trunk2branch() { local IFS # Some of statements need one parameter at least. if [ "$line" = "${trunk[0]}" ]; then parse_die "missing parameter." fi IFS=$'\n' branch=($(explode "${trunk[1]}")) if [ $? -ne 0 ]; then parse_die "missing '\"'." fi } preparse_cfg() { local agent line trunk branch macro num host agent="$1" if [ ! -r "$opt_cfgsdir/$agent" ]; then die "${agent}: configuration file not found." fi line_num=0 while read -r line; do let line_num++ num=" $line_num" case "$line" in ""|\#*) continue;; esac line2trunk case "${trunk[0]}" in CASE-BLOCK) trunk2branch macro="$CASES_DIR/${agent}_macro.${branch[0]}" continue ;; Include|Include@*) host=$(echo "${trunk[0]}" | awk -F@ '{print $2}') trunk2branch if [ ! -r "$CASES_DIR/${agent}_macro.${branch[0]}" ]; then parse_die "Macro '${branch[0]}' not found." fi if [ -n "$host" ]; then line="$(sed -e 's/^\([^[:blank:]]*\)@[^[:blank:]]*/\1/' -e "s/^[^[:blank:]]*/&@$host/" "$CASES_DIR/${agent}_macro.${branch[0]}")" else line="$(<"$CASES_DIR/${agent}_macro.${branch[0]}")" fi num= ;; *[!A-Z-]*) : ;; *) macro= ;; esac if [ -n "$macro" ]; then echo "$line$num" >>"$macro" else echo "$line$num" >>"$CASES_DIR/${agent}_preparse" fi done <"$opt_cfgsdir/$agent" } case_parse_finish() { local host if [ -n "$sh" ]; then cat >>$sh <>$sh done echo "quit 0" >>$sh fi atexit_num=0 hosts= sh= } init_cfg_vars() { cfg_agent= cfg_agent_root= cfg_install_package=() cfg_hang_timeout=20 } agent_parse_finish() { local suf for suf in preparse setup cleanup var hosts; do rm -f $CASES_DIR/${agent}_$suf done rm -f $CASES_DIR/${agent}_macro.* init_cfg_vars } need_make() { local src_time obj_time if [ ! -f "$CASES_DIR/0_${agent}.sh" ]; then return 0 fi src_time=$(stat -c '%Y' "$opt_cfgsdir/$agent") obj_time=$(stat -c '%Y' "$CASES_DIR/0_${agent}.sh") test $src_time -ge $obj_time } parse_cfg() { local agents i line stat sh trunk branch atexit_num host hosts if [ $# -eq 0 ]; then agents=($opt_cfgsdir/*) else agents=("$@") fi for agent in "${agents[@]}"; do agent="$(basename "$agent")" if ! need_make; then continue fi agent_obj_clean $agent agent_parse_finish i=0 echo "Making '$agent': " preparse_cfg "$agent" while read -r line; do line_num="${line##* }" line="${line% *}" line2trunk # state switch case "${trunk[0]}" in CONFIG) case_parse_finish stat=1 continue ;; VARIABLE) case_parse_finish stat=2 continue ;; SETUP-AGENT) case_parse_finish stat=3 continue ;; CLEANUP-AGENT) case_parse_finish stat=4 continue ;; CASE) case_parse_finish trunk2branch echo " - case ${i}: ${branch[0]}" sh="$CASES_DIR/${i}_${agent}.sh" cat >$sh <>$CASES_DIR/${agent}_var ;; 3) echo "$line" >>$CASES_DIR/${agent}_setup ;; 4) echo "$line" >>$CASES_DIR/${agent}_cleanup ;; 5) host=$(echo ${trunk[0]} | awk -F@ '{print $2}') if [ -n "$host" ]; then if ! echo "$hosts" | grep -q "$host"; then echo "$host" >>$CASES_DIR/${agent}_hosts hosts=$hosts$'\n'$host cat >>$sh <>$sh if [ -n "$host" ]; then echo "backbash $host <<'CMD'" >>$sh fi case "${trunk[0]}" in Env|Env@*) cat >>$sh <>$sh <>$sh <>$sh <>$sh <>$sh <>$sh <>$sh <>$sh <>$sh fi ;; *) parse_die "unimplemented statement: ${trunk[0]}" ;; esac done <$CASES_DIR/${agent}_preparse if [ -r "$CASES_DIR/${agent}_setup" ]; then cat >$CASES_DIR/setup_${agent}.sh <>$CASES_DIR/setup_${agent}.sh <>$CASES_DIR/setup_${agent}.sh <$CASES_DIR/cleanup_${agent}.sh <>$CASES_DIR/cleanup_${agent}.sh <>$CASES_DIR/cleanup_${agent}.sh </dev/null 2>&1; then die "cases directory not found." fi + if [ ! -d logs ]; then + mkdir logs + fi + export __OCFT__VERBOSE=$opt_verbose if [ $# -eq 0 ]; then agents=($(ls -1 *.sh 2>/dev/null | sed 's/.*_\([^_]*\)\.sh$/\1/' | sort | uniq)) else agents=("$@") fi for shs in "${agents[@]}"; do if [ -z "$opt_incremental" ]; then testsh="setup_${shs}.sh $(ls -1 [0-9]*_${shs}.sh 2>/dev/null | sort -n) cleanup_${shs}.sh" else testsh="setup_${shs}.sh $(ls -1 [0-9]*_${shs}.retest 2>/dev/null | sed 's/retest$/sh/' | sort -n) cleanup_${shs}.sh" fi for sh in $testsh; do if [ -r "$sh" ]; then ./$sh ret=$? case "$sh" in setup*) if [ $ret -ne 0 ]; then warn "SETUP failed, break all tests of '$shs'." break fi ;; cleanup*) if [ $ret -ne 0 ]; then warn "CLEANUP failed." fi ;; [0-9]*) case $ret in 3) die "core function failed, break all tests." ;; 2) warn "core function failed, break all tests of '$shs'."; break ;; 1) touch ${sh%.*}.retest ;; 0) rm -f ${sh%.*}.retest ;; esac ;; esac fi + done 2>&1 | while read -r line; do + echo "$line" + echo "$(date '+%F %T'): $line" | cat -A | + sed -r 's/\^\[\[[0-9]+m|\^I|.$//g' >>logs/$shs.log done - done 2>&1 | while read -r line; do - echo "$line" - echo "$(date '+%F %T'): $line" | cat -A | - sed -r 's/\^\[\[[0-9]+m|\^I|.$//g' >>ocft.log done } agent_clean() { local typ ra typ=$1 shift if [ $# -eq 0 ]; then rm -f $CASES_DIR/*.$typ else for ra in "$@"; do rm -f $CASES_DIR/*_${ra}.$typ done fi } agent_retest_clean() { agent_clean retest "$@" } agent_obj_clean() { agent_clean sh "$@" } usage() { cat <