diff --git a/TODO b/TODO index 03e13e70..cef45dcb 100644 --- a/TODO +++ b/TODO @@ -1,134 +1,133 @@ -------------------------------------------------------- The Corosync Cluster Engine Topic Branches and Backlog -------------------------------------------------------- ---------------------------- Last Updated: October 2011 ---------------------------- -------------------------------------- Current priority list for Needle 2.0 -------------------------------------- 1. implement topic-map 2. replace confdb callers with map callers 3. quorum debugging and rework 4. implement topic-xmlconfig 5. remove external plug-in api 6. remove logsys.h from external headers 7. remove hardcoded values in totempg.c check_q_level 8. check max message size restrictions 9. investigate if https://github.com/asalkeld/libqb/issues/1 is still an issue. -10. remove logging/ipc from man pages. -11. investigate proper sizing for new blackbox (exec/logsys.c:311) -12. allow a cluster name to autogenerate a mcastaddr -13. ring status change via corosync-notifyd +10. investigate proper sizing for new blackbox (exec/logsys.c:311) +11. allow a cluster name to autogenerate a mcastaddr +12. ring status change via corosync-notifyd -------------------------------------- Current priority list for Needle 2.1 -------------------------------------- 1. implement topic-onecrypt 2. implement add/remove nodes from udpu 3. logsys glue layer removal 4. implement topic-zerocopy 5. implement topic-rdmaud 6. harden and finish ykd algorithm We use topic branches in our git repository to develop new disruptive features that define our future roadmap. This file describes the topic branches the developers have interest in investigating further. targets can be: whitetank, needle2.0, needle3.0, or future (3.0+). Finished can be: percentage or date merged to master. Once in a shipped version, please remove from the topic list. ------------------------------------------------------------------------------ topic-map ------------------------------------------------------------------------------ Main Developer: Honza Friesse Started: not started Finished: 20% target: needle2.0 Currently confdb is very difficult to use. We use this component for our diagnostic feature set as well as storing our runtime configuration. A map is a better choice for a data structure here. Current thinking is to use the trie implementation from libqb to provide the core of this functionality ------------------------------------------------------------------------------ topic-xmlconfig ------------------------------------------------------------------------------ Main Developer: Honza Friesse Started: not started Finished: 0% target: needle2.0 Test suites and users alike would like to configure the software via XML configuration. Current thinking is we will implement a separate binary which converts xml to native config format via XSLT. This keeps libxml out of the corosync process address space. During startup, corosync could either fork and exec this process, or it could be part of the system startup mechanism. ------------------------------------------------------------------------------ topic-onecrypt ------------------------------------------------------------------------------ Main Developer: Honza Friesse Started: not started Finished: 0% target: needle2.1 Description: Currently encryption code is located in totemudp.c, totemudpu.c, and iba has no encryption support. This topic merges the encryption code into a new file such as totemcrp.c and provides a mechanism for totemnet.c to register encrypt and decrypt functions with totem[udp|iba|udpu] and use them as requested by the configuration. ------------------------------------------------------------------------------ topic-netmalloc ------------------------------------------------------------------------------ Main Developer: Honza Friesse Started: not started Finished: 0% target: needle2.1 Description: The totemiba.c driver must allocate memory and assign it to a protection domain in order for an infiniband driver to transmit memory. In the current implementation, totemsrp.c also allocates these same frames. This results in an extra memcpy when transmitting with libibverbs technology. Memory copies are to be avoided. The simple solution is to have each network driver provide a memory allocation function. When totemsrp wants a free frame, it requests it from the network driver. ------------------------------------------------------------------------------ topic-rdmaud ------------------------------------------------------------------------------ Main Developer: Honza Friesse Steven Dake Started: not started Finished: 0% target: needle2.1 Description: Currently our RDMA code uses librdmacm to setup connections. We are not certain this extra library is needed, and may be able to use only ibverbs. If this is possible, the totem code may be more reliable, especially around failure conditions. ------------------------------------------------------------------------------ topic-zerocopy ------------------------------------------------------------------------------ Main Developer: Honza Friesse Started: not started Finished: 0% target: needle2.1 Description: Totem has many copies involved in messaging which we would like to investigate removing. Our goal is to deliver wire speed performance for rdma networks, and if this can be achieved by our other topic investigations, we may not further investigate this topic. The basic idea of the topic is to handle message assembly/fragmentation in libcpg, and have totem be responsible for sending these pages that are shared via posix shared memory. ------------------------------------------------------------------------------ other topics not yet defined: * disallow binding to localhost interfae in redundant ring configuation. * doxygenize include and lib directories. * sort out binding to localhost in general * totem multiring * load balancing over different speed links in RRP diff --git a/man/coroipc_overview.8 b/man/coroipc_overview.8 deleted file mode 100644 index 8912f903..00000000 --- a/man/coroipc_overview.8 +++ /dev/null @@ -1,177 +0,0 @@ -.\"/* -.\" * Copyright (c) 2009 Red Hat, Inc. -.\" * -.\" * All rights reserved. -.\" * -.\" * Author: Steven Dake (sdake@redhat.com) -.\" * -.\" * This software licensed under BSD license, the text of which follows: -.\" * -.\" * Redistribution and use in source and binary forms, with or without -.\" * modification, are permitted provided that the following conditions are met: -.\" * -.\" * - Redistributions of source code must retain the above copyright notice, -.\" * this list of conditions and the following disclaimer. -.\" * - Redistributions in binary form must reproduce the above copyright notice, -.\" * this list of conditions and the following disclaimer in the documentation -.\" * and/or other materials provided with the distribution. -.\" * - Neither the name of the MontaVista Software, Inc. nor the names of its -.\" * contributors may be used to endorse or promote products derived from this -.\" * software without specific prior written permission. -.\" * -.\" * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" -.\" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -.\" * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -.\" * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE -.\" * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR -.\" * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF -.\" * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS -.\" * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN -.\" * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) -.\" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF -.\" * THE POSSIBILITY OF SUCH DAMAGE. -.\" */ -.TH COROIPC_OVERVIEW 8 2009-03-21 "corosync Man Page" "Corosync Cluster Engine Programmer's Manual" -.SH NAME -coroipc_overview \- Overview of coroipc libraries -.SH OVERVIEW -The coroipcs and coroipcc libraries provide a generically reusable very high -performance shared memory IPC sytem for client and service applications. -It supports many features including: -.PP -65536 user services and 65536 command ids per service. -.PP -Shared memory implementation for very high performance. -.PP -A synchronous request/response channel and asynchronous response channel -per ipc connection. -.PP -User defined private data per IPC connection. -.PP -Ability to call a function per service on ipc connection and disconnection. -.PP -Authenticated IPC connection with ability for developer to define which -UIDs and GIDs are valid at connection time. -.PP -Fully abstracted poll system so that any poll library may be used. -.PP -User defined selector for determining the proper function to call per -service and id. - -.SH Description of the libraries -There are two shared libraries available for developing IPC client applications. - -The first library is coroipcs.so which is responsible for the server -implementation. This library should be linked with the server and then -initialized with coroipcs_init(3). - -Once the library is initialized, it will provide service to coroipcc.so library -users. - -The second library is coroipcc.so which is responsible for the client -implementation. This library should be linked with the client and requires -no initialization. This library provides communication functions for sending -and receiving synchronous requests, and also reading asynchronous message -requests from the server. - -.SH Initializing the coroipcs library -To use the coroipcs library, the developer creates a coroipcs_init_state -structure and populates it with function names. The functions do various -operations described in coroipcs_init(3) man page. Not all operations must -be specified. If some are missing, the corosync ipcs library will -automatically populate the structure with internal versions which provide -basic functionality. - -.SH Communicating with the coroipcc clients -Every ipc connection is represented by a void * pointer which uniquely -identifies the data set for the IPC connection. Each IPC connection also -contains user defined private data. To obtain this private data pointer, the -function coroipcs_private_data_get(3) function can be called. - -There are two channels for communication. The primary channel is the -synchronous request/response channel. Another channel is available for out of -band asynchronous responses from the server. - -To send a response on the syncronous channel, coroipcs_response_send(3) or -coroipcs_response_iov_send(3) should be used. To send a message on the -asynchronous channel, coroipcs_dispatch_send(3) or coroipc_dispatch_iov_send(3) -should be used. - -.SH The abstracted poll system -There are many different poll systems in use in applications. They are usually -intricately tied up in the implementation of the application and each provide -different APIs and functionality. To manage this, the coroipcs library -provides callbacks in coroipcs_init(3) which should be called when a new -connection should be added for accept system calls or to dispatch messages. - -These callbacks add the relevant fd to the application's poll system. When the -application poll system triggers the callback registered by the user defined -poll adding functions, they then call either coroipc_handler_accept(3) or -coroipc_handler_dispatch(3). - -.SH Initializing the coroipcc library -No initialization is required in the coroipcc library. - -.SH Lifecycle of an IPC connection. -An IPC connection is made to the server with coroipcc_service_connect(3). This -function connects to the server and requests channels be created for -communication. To disconnect, the client either exits or executes the -function coroipcc_service_disconnect(3). - -.SH Synchronous communication -There are two functions for sending a request and receiving a response. The -first function coroipcc_msg_send_reply_receive(3) sends an iovector request -and receives a response. This function copies the response into the response -buffer. the second function coroipcc_msg_end_reply_receive_in_buf(3) does not -copy the response buffer and allows for zero-copy reading of the response -when the lifetime of the response buffer is known. - -.SH Asynchronous communication -The coroipcc_dispatch_recv(3) function receives an out-of-band asyncronous -message. Unlike the synchronous communication channel, the asynchronous -messages are queued and can provide very high out-of-band performance. - -To determine when to call coroipcc_dispatch_recv(3) the corosync_fd_get(3) call -is used to obtain a file descriptor used in the poll(2) or select(2) system -calls. - -Finally the current dispatch flow control state can be obtained with -coroipcc_flow_control_get(3). - -.SH Performance -The ipc system is tuned for very high performance while also being comletely -abstracted from the underlying poll mechanism and any internalisms required -by the server. The ipc system achieves such high performance by using shared -memory as oppossed to slower techniques such as UNIX_PF sockets. - -We intend to do further development to allow syncronous requests to return -messages in an asyncronous way to avoid blocking involved in the syncronous -request/response model used today for higher throughput in some use cases. - -.SH Security -The ipc system uses default operating system security mechanics to ensure -ipc connections are validated. A callback used with coroipcs_init(3) is called -for every new ipc connection with the parameters of UID and GID. The callback -then determines if the UID and GID are authenticated for communication. More -about this topic can be viewed in the coroipcs_init(3) man page. - -.SH "SEE ALSO" -.BR coroipcs_ipc_init (3), -.BR coroipcs_ipc_exit (3), -.BR coroipcs_private_data_get (3), -.BR coroipcs_respone_send (3), -.BR coroipcs_response_iov_send (3), -.BR coroipcs_dispatch_send (3), -.BR coroipcs_dispatch_iov_send (3), -.BR coroipcs_refcount_inc (3), -.BR coroipcs_refcount_dec (3), -.BR coroipcs_handler_accept (3), -.BR coroipcs_handler_dispatch (3), - -.BR cooripcc_service_connect (3), -.BR coroipcc_service_disconnect (3), -.BR coroipcc_msg_send_reply_receive (3), -.BR coroipcc_msg_send_reply_receive_in_buf (3), -.BR coroipcc_dispatch_recv (3), -.BR coroipcc_fd_get(3), -.BR coroipcc_dispatch_flow_control_get (3) diff --git a/man/logsys_overview.8 b/man/logsys_overview.8 deleted file mode 100644 index fb2a26ae..00000000 --- a/man/logsys_overview.8 +++ /dev/null @@ -1,201 +0,0 @@ -.\"/* -.\" * Copyright (c) 2007-2009 Red Hat, Inc. -.\" * -.\" * All rights reserved. -.\" * -.\" * Author: Steven Dake (sdake@redhat.com) -.\" * Author: Fabio M. Di Nitto (fdinitto@redhat.com) -.\" * -.\" * This software licensed under BSD license, the text of which follows: -.\" * -.\" * Redistribution and use in source and binary forms, with or without -.\" * modification, are permitted provided that the following conditions are met: -.\" * -.\" * - Redistributions of source code must retain the above copyright notice, -.\" * this list of conditions and the following disclaimer. -.\" * - Redistributions in binary form must reproduce the above copyright notice, -.\" * this list of conditions and the following disclaimer in the documentation -.\" * and/or other materials provided with the distribution. -.\" * - Neither the name of the MontaVista Software, Inc. nor the names of its -.\" * contributors may be used to endorse or promote products derived from this -.\" * software without specific prior written permission. -.\" * -.\" * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" -.\" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE -.\" * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE -.\" * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE -.\" * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR -.\" * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF -.\" * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS -.\" * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN -.\" * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) -.\" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF -.\" * THE POSSIBILITY OF SUCH DAMAGE. -.\" */ -.TH QB_LOGSYS_OVERVIEW 8 2009-06-16 "corosync Man Page" "Corosync Cluster Engine Programmer's Manual" -.SH NAME -logsys_overview \- Logsys Library Overview -.SH OVERVIEW -The logsys library provides a generically usable logging and tracing system for -use by applications. It supports many features including: -.PP -Configurable subsystem logging or single system logging -.PP -Threaded non-blocking logging of log output data for better non-blocking performance -.PP -Support for 8 tracing levels and tracing the entering and leaving of functions -.PP -Declaration of logging system or subsystem without calling any functions -.PP -Dynamic reconfiguration of the logging system parameters -.PP -Logging to syslog, file, stderr. - -.SH Declaration of the System logger -The logsys library is initially configured by including logsys.h and declaring -a logger. Once the logger is declared, optional subsystem loggers can be -declared in every file. - -The definition QB_LOGSYS_DECLARE_SYSTEM is placed after the include section of one -C file in the application. This declaration creates a constructor function -which will be called automatically before main() is executed. This technique -avoids the need for calling any setup functions in short applications that don't -require it and enables full logging capabilities before any application code is -executed. - -#define QB_LOGSYS_DECLARE_SYSTEM (name, mode, debug, file, file_priority, -syslog_facility, syslog_priority, format, fltsize) - -The name parameter is the name of the application or system. - -The mode parameter is the logging mode of the system. -The following modes can be configured by logically ORing these flags: - -QB_LOGSYS_MODE_OUTPUT_FILE: Output all log data to the file parameter of this declaration - -QB_LOGSYS_MODE_OUTPUT_STDERR: Output all log data to the stderr descriptor - -QB_LOGSYS_MODE_OUTPUT_SYSLOG: Output all log data to syslog using a non-blocking thread - -QB_LOGSYS_MODE_FORK: This flags tells logsys to queue all data untill the application -has forked. The application is then responsible to call logsys_fork_completed to flush -the queue and start logging. - -QB_LOGSYS_MODE_THREADED: Starts a separate thread to handle non-blocking logging operations. -If this flag is not specified, the logging operations are blocking. - -The debug parameter, if enabled, turns off all messages priority filtering, recording -everything everywhere. - -The file parameter specifies the filename that should be used to log messages. -This parameter may be NULL and no log file will be created. - -The file_priority parameter specifies the message priority that should be logged to file. - -The syslog_facility parameter is the syslog facility that should be used when logging -messages. - -The syslog_priority, similar to file_priority, specifies the message priority that should be logged to -syslog. - -The format parameter allows to set custom output format. -Set to NULL to use built-in default. - -The fltsize parameter specifies the flight recorder buffer size in bytes. The requested value -is increased by the size of 2 unsigned ints and rounded to the next PAGE_SIZE. - -An example declaration would be: - -#include - -... (other #includes) - -QB_LOGSYS_DECLARE_SYSTEM ("test", /* name */ - QB_LOGSYS_MODE_OUTPUT_STDERR | QB_LOGSYS_MODE_THREADED, /* mode */ - 0, /* debug */ - NULL, /* logfile path */ - QB_LOGSYS_LEVEL_INFO, /* logfile_priority */ - LOG_DAEMON, /* syslog facility */ - QB_LOGSYS_LEVEL_INFO, /* syslog level */ - NULL, /* use default format */ - 1000000); /* flight recorder size */ - - -.SH Declaration of subsystems -The logsys library supports the logging of information to one main system or -subsystem. This is specified in each individual object C file in the system -and it is entirely optional. - -An example: - -QB_LOGSYS_DECLARE_SUBSYS ("subsystest"); - -It is possible to use the same subsystem name in separate object files. -In this case, the individual logging parameters for those subsystem identifier -will be used. - -A newly created subsystem inherits the system configuration at the time of -creation. - -It is possible to override every configuration option on a subsystem base -throught the configuration API. - -.SH Logging Messages -The definition log_printf is used to log information to the log. It works -in a similiar fashion to printf, except it has a first parameter of level -which may be the following: -QB_LOGSYS_LEVEL_EMERG -QB_LOGSYS_LEVEL_ALERT -QB_LOGSYS_LEVEL_CRIT -QB_LOGSYS_LEVEL_ERR -QB_LOGSYS_LEVEL_WARNING -QB_LOGSYS_LEVEL_NOTICE -QB_LOGSYS_LEVEL_INFO -QB_LOGSYS_LEVEL_DEBUG - -An example of using log_printf would be - -log_printf (QB_LOGSYS_LEVEL_EMERG, "This is an emergency %s value %d\n", string, value); - -Tracing of functions can be done using ENTER(), LEAVE(); - -An example of using ENTER is -void function (char *name) { -ENTER(); -... function contents ... -LEAVE(); -} - -Individual tracing levels are supported through the macros -TRACE1(format, args) -TRACE2(format, args) -.. -TRACE8(format, args) - -An example of using TRACE is - -char *name = "test"; -TRACE7 ("This is a trace 7 log with name %s\n", name); - -Note that ENTER/LEAVE/TRACE* calls are recorded only in the flight recorder. - -.SH "SEE ALSO" -.BR logsys_fork_completed (3), -.BR logsys_atexit (3), -.BR logsys_log_rec_store (3), -.BR logsys_format_set (3), -.BR logsys_format_get (3), -.BR logsys_config_mode_set (3), -.BR logsys_config_file_set (3), -.BR logsys_config_syslog_facility_set (3), -.BR logsys_config_syslog_facility_get (3), -.BR logsys_config_mode_set (3), -.BR logsys_config_mode_get (3), -.BR logsys_config_file_set (3), -.BR logsys_config_logfile_priority_set (3), -.BR logsys_config_debug_set (3), -.BR logsys_facility_id_get (3), -.BR logsys_facility_name_get (3), -.BR logsys_priority_id_get (3), -.BR logsys_priority_name_get (3), -.PP