CI Project
  Overview
  License
  Download
  CVS
  Contributed Code
  Mailing Lists
  Goals
  Project List
  Features
  CLMS Features
  ICS Features
  Current Status
  Kernel Hooks
  Limitations / Enhancements
   
CI Documentation
  Kernel Patch
  Cluster Tools
   
Printer-friendly version

OPEN CI and OPEN SSI CLUSTER PROJECTS

1. ICS Messages and Responses
2. Generating ICS Code Using icsgen
    2.1. Services
    2.2. Operations
    2.3. Datatypes and Encoding
    2.4. Service Invocation Macros
    2.5. Summary Format of Generated Client-side Stubs
    2.6. Summary Format of Generated Server-side Stubs
    2.7. Generated Prototypes
    2.8. Generated Dispatch Table and Service Registration Routine
    2.9. Interface Examples
    2.10. icsgen
        2.10.1. Syntax
        2.10.2. Description
        2.10.3. Options
        2.10.4. Files
        2.10.5. Examples
    2.11. Source and Build Environment
    2.12. Defining Encoding and Decoding
    2.13. XDR-defined Datatypes
    2.14. nsc_rcall() Emulation
    2.15. Verification Tests
3. High-level ICS
    3.1. CI/SSI Services and ICS Communication Channels
    3.2. ICS Server Control Code
    3.3. Typical ICS Client Usage Scenarios
    3.4. Typical ICS Server Usage Scenarios
    3.5. ICS General Purpose Routines
        3.5.1. ics_svc_register()
        3.5.2. ics_init()
        3.5.3. ics_nodeinfo_callback()
        3.5.4. ics_geticsinfo()
        3.5.5. ics_seticsinfo()
        3.5.6. ics_nodeup()
        3.5.7. ics_nodedown()
        3.5.8. ics_getpriority()
        3.5.9. ics_setpriority()
    3.6. ICS Client-side Routines
        3.6.1. icscli_handle_get()
        3.6.2. icscli_wouldthrottle()
        3.6.3. icscli_waitforthrottle()
        3.6.4. icscli_send()
        3.6.5. icscli_wait_callback()
        3.6.6. icscli_wait()
        3.6.7. icscli_get_status()
        3.6.8. icscli_handle_release()
        3.6.9. icscli_handle_release_callback()
    3.7. ICS Client-side Argument Marshalling
        3.7.1. icscli_??code_type()
        3.7.2. icscli_??code*_data_t()
        3.7.3. icscli_??code*_uio_t()
        3.7.4. icscli_??code_mbuf_t()
        3.7.5. icscli_??code_mblk_t()
    3.8. ICS Server-side Routines
        3.8.1. icssvr_handle_get()
        3.8.2. icssvr_recv()
        3.8.3. icssvr_decode_done()
        3.8.4. icssvr_reply()
        3.8.5. icssvr_handle_release()
        3.8.6. icssvr_nodedown_svc_wait()
    3.9. ICS Server-side Argument Marshalling
        3.9.1. icssvr_??code_type()
        3.9.2. icssvr_??code_data_t()
        3.9.3. icssvr_??code_uio_t()
        3.9.4. icssvr_??code_mbuf_t()
        3.9.5. icssvr_??code_mblk_t()
4. Low-level ICS
    4.1. Low-level ICS and Communication Channels
    4.2. Low-level ICS Typical Client Scenarios
    4.3. Low-level ICS Typical Server Scenarios
    4.4. Low-level ICS General Purpose Upcall Routines
        4.4.1. ics_nodedown_notification()
        4.4.2. ics_nodeup_notification()
    4.5. Low-level ICS General Purpose Routines
        4.5.1. ics_llinit()
        4.5.2. ics_llgeticsinfo()
        4.5.3. ics_llseticsinfo()
        4.5.4. ics_llnodeup()
        4.5.5. ics_llnodedown()
    4.6. Low-level ICS Client-side Upcall Routines
        4.6.1. icscli_find_transid_handle()
        4.6.2. icscli_sendup_reply()
    4.7. Low-level ICS Client-side Routines
        4.7.1. icscli_llhandle_init()
        4.7.2. icscli_llwouldthrottle()
        4.7.3. icscli_llwaitfornothrottle()
        4.7.4. icscli_llsend()
        4.7.5. icscli_llhandle_deinit()
    4.8. Low-level ICS Client-side Argument Marshalling
    4.9. Low-level ICS Server-side Upcall Routines
        4.9.1. icssvr_find_recv_handle()
        4.9.2. icssvr_sendup_msg()
        4.9.3. icssvr_sendup_replydone()
    4.10. Low-level ICS Server-side Routines
        4.10.1. icssvr_llhandle_init()
        4.10.2. icssvr_llhandle_init_for_recv()
        4.10.3. icssvr_llhandles_present()
        4.10.4. icssvr_lldecode_done()
        4.10.5. icssvr_llreply()
        4.10.6. icssvr_llhandle_deinit()
    4.11. Low-level ICS Server-side Argument Marshalling Routines



SUBJECT:

AUTHOR:

VERSION:

DATE:

Internode Communication Subsystem (ICS) Design Specification

Now maintained by Compaq Computer Corp.

1.0

Originally 1996; See Errata From 10/15/01 for Updates

 


This document describes ICS, the CI/SSI Internode Communication Subsystem. ICS is used by all CI/SSI components for node-to-node communication.

The first section of this document provides a high-level overview of the message/response paradigm presented by ICS.

The second section of this document describes icsgen, the tool used to generate RPC and message client/server stubs.

The third section of this document describes the ICS interfaces (these interfaces are also known as high-level ICS interfaces, to contrast them with the low-level ICS interfaces described in the next section).

The final section of this document describes the low-level ICS interfaces. This is the transport-specific part of ICS (though the interfaces to low-level ICS are transport-independent). The low-level ICS interfaces are invoked only by high-level ICS. 

1. ICS Messages and Responses

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- ICS_MAX_INLINE_DATA_SIZE is set to 300 bytes.

- There are a limit of 7 out-of-line buffers, where the buffers are

unlimited in size. "Out-of-line" is misleading, since TCP doesn't

offer true OOL data. It was a feature provided by a special

high-speed interconnect such as Servernet. Over TCP, OOL data is

sent out regularly like inline data. All OOL data is simply

appended to the end of the inline data buffer.

- DMA-push/DMA-pull is not supported. This was a feature when using

a high-speed interconnect such as Servernet. It is not available

over regular TCP/IP.
***********************************************************

The Internode Communication Subsystem (ICS) of CI/SSI has the following characteristics:

  • a client/server model is presented;
  • the high-level ICS client code is presented with either a message/response
    model or a reliable message passing model;
  • the high-level ICS server code retrieves messages and sends optional responses back to the client.

This model can be used to efficiently represent both types of messaging paradigms: messages and RPC's (Remote Procedure Calls).

ICS messages and responses consist of:

  • a fixed size, in-line buffer (ICS_MAX_INLINE_DATA_SIZE bytes);

  • zero or more out-of-line buffers, where the buffers are unlimited in size.

The ICS interfaces are structured in a way to allow low-level ICS code to implement efficient transfer of out-of-line buffers using DMA-push, DMA-pull, or other techniques. 

This chapter often refers to input arguments and output arguments . Input arguments refer to the data in messages to the server (both in-line data and out-of-line data). Input arguments apply to both RPCs and messages. Output arguments refer to the data in a response sent from a server (both in-line data and out-of-line data). Output arguments apply to RPCs only.

In order for client code to send a message (and wait for a response), or for the server code to handle a message from the client, the code must deal with handles. Handles are data structures which are opaque to the icsgen-generated stub code. Handles have the following characteristics:

  • contain the state of the transaction (the number of bytes in the message, whether the message has been sent, etc.);
  • contain the fixed size, in-line buffer referenced in prior paragraphs; contain information about any out-of-line buffers that are in use;

Many of routines in the interface make use of callback routines. Callback routines are routines that will be called by the low-level ICS code when some specific event has transpired. The parameters to the callback routine are specified in the relevant interface specifications. Of particular note is the fact that each callback routine has an argument specified as callarg. The callarg argument is an argument to the interface which is supplied solely for the purpose of being used as a callback argument.

2. Generating ICS Code Using icsgen

At the highest level, the interface between ICS and other CI/SSI components (using ICS) is a set of services with each service supporting a set of well-defined operations. These services - comprising both client and server elements - constitute the interface with the ICS subsystem. Services are defined using a definition language which is processed at build-time producing code to bind the local CI/SSI component to ICS; ICS client to ICS server; and thus local CI/SSI component to remote service.

Service definitions are contained in source files processed by a build process called icsgen. icsgen reads service definition files and generates further source files containing:

  • header files of prototype declarations for functions to be supplied to ICS - these prototypes ensure a compatible interface with ICS;

  • header files containing a macro interface calling ICS-supported services;

  • stub routines that map operations between layers of ICS and between nodes - these are principally routines mapping data representations between CI/SSI and internal formats used by ICS;
  • server dispatch tables used internally by ICS.

 

2.1. Services

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- mem_lwm and mem_hwm are currently not being used, since flow

control (aka throttling) is not supported.

***********************************************************

Each distinct service supported by ICS must be declared to ICS. Each service is defined by a file (or set of files) identifying the service to ICS and defining the operations (functions) provided by the service. Service definition files are conventionally given the name  service_name.svc.

A service is defined to ICS by a service prefix. The syntax declaring this is:

	service svc_name svc_id hmin hmax [mem_lwm mem_hwm]

where

 

svc_name

is an identification for the service. It is used in constructing the names of data structures defining the service. Typically the prefix is the same or similar to the .svc filename - for example, the service defined by file fbtok.svc has service name fbtok (for remote file block token operations).

svc_id

is the unique number (or symbol representing this) of the ICS service.
hmin is the minimum number of free ICS server handles held in reserve for this service.

hmax

is the maximum number of free ICS server handles held in reserve for this service.

mem_lwm

is the number of free memory pages below which server-side flow-control (also called throttling) is activated. If not specified, this value is assigned a default at run-time.

mem_hwm

is the number of free memory pages above which server-side flow-control ceases if active. If not specified, this value is assigned a default at run-time.

Service ids are defined in header file <cluster/ics.h> which maps component services to ICS communication channels. Note that many CI/SSI services may be grouped together as sub-services sharing ICS communication channels and flow-controlled as a unit (see the discussion on communication channels and throttling in the high-level ICS section of the document).

The form of the operations definitions are described in the next section.

2.2. Operations

Each operation provided by an ICS-supported service must be fully defined. For each operation, parameters and their attributes must be declared. This both defines the interface with ICS and also dictates how ICS handles data transferred between nodes.

The syntax defining an operation is: 

	
	operation op_name [op_attr] {
		param param_direction[:param_attr] param_utype param_name
		... repeated for each parameter...
	}

where

 

op_name

is the name of the operation, for example: pvpop_fork;

 
op_attr is one of the following, defaulting to RPC:

RPC   

 the operation is a remote procedure call for which at least a returned status value will be awaited.

MSG

 the operation is a simple message with no response expected

ASYC 

the operation is an RPC but the caller requests that send and receive
sub-operations will be separately callable allowing non-blocking behavior.

SKIP 

 

the operation is a dummy placeholder; it enables operation entry numbers to be reserved at a lower-level with no user-level interface corresponding to them. Note that the op_name field must be given.

In addition, there are the following modifiers (which can be concatenated to an operation attribute using the syntax op_attr:op_modifier):
NO_SIG_FORWARD requests that an RPC is not to be interrupted to forward signals on the server.
NO_BLOCK declares that a MSG operation may be called from interrupt level and that the client stub should not sleep awaiting system resources but should return with error (EAGAIN).

 
param_direction is one of the following:
IN for an input paramter;
OUT for an output parameter; or
INOUT for a parameter that is both input and output.

 
param_attr is a composite declaring the data attributes of a parameter, where this is a concatenation of the following separated conventionally by ":" (although any character is accepted):
OOL 

declares the parameter to be out-of-line data.
XDR

declares the parameter is of complex datatype which has been defined as an XDR-able structure and is to be encoded/decoded using an xdr-routine (either generated by rpcgen or hand-written).
VAR

declares the parameter to be of variable length with the length determined by the following implicitly-generated parameter. That is, the caller of this operation supplies an additional argument specifying the length of the variable parameter. This length is generally the number of basic elements in the supplied array: this is the number of bytes for a character array but the number of elements in an array of more complex type. See below for an example of this.
PTR

declares that an input parameter being passed by reference is a a non-scalar type, which is not equivalent to intgeger type, and this pointer may be NULL. Parameters referenced by pointer must always be non-NULL - unless they have the PTR, VAR or OOL attribute.
FREE declares that an input parameter requires an addition call to free resources associated with it on the server-side after the remote server operation has been completed.

 
param_utype is the user type of this parameter, for example: struct vproc or cred_t.

 

param_name is the name of the parameter. Note that the name may be prefixed by "*"s indicating indirection; these are implicitly suffixed to param_utype.

 

In addition to these declared parameters, extra parameters are implicit for operations. If these were to be explicitly declared, these would occur as the first parameters to each operation and would take the following forms. For non-MSG operations returning a value:
param IN node_t node

param OUT int *rval

 

and for MSG operations not returning any value:
param IN node_t node

 

where:
node is the node number of the destination node on which the operation is targeted; and
rval is the return value of the non-message operation.
 

Variable length parameters declared with attribute VAR have an associated length parameter which is implicitly declared as the next parameter. Thus, a declaration of the form:

param IN:VAR char name[MAX_NAME_LEN]
 

will be handled by ICS as if the following were declared:

param IN:VAR char name[MAX_NAME_LEN]
param IN int name_len

This also illustrates the way that the maximum size of a variable length parameter is specified. The main purpose of this is to allocate buffer space into which data is decoded on the server.

icsgen pre-processes service definitions using the C pre-processor. Hence, service definitions can contain embedded C pre-processor directives, macro invocations, and, of course, C-style comments. `

2.3. Datatypes and Encoding

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- cli_encoderesp_ool_param_type() is a NO-OP. This was used to

support DMA-push/DMA-pull over the Servernet interconnect.

- There are no encode/decode routines for uio_t, mblk_t and mbuf_t,

as they are not supported in Linux. Instead, there are new routines

for encoding/decoding msghdr structures.

***********************************************************

Each parameter must have its data type fully defined. If the user data type, param_utype does not quote a C typedef - for example, it is a structure definition like struct procinfo - then a type will be generated by the replacement of whitespace by "_" characters forming, in this example, struct_procinfo.

In addition, indirection indicated through the use of a "*" appended to param_utype will result in _p being appended to the fully qualified parameter type, param_type. So, for example, struct procinfo * becomes struct_procinfo_p. Note that initially only a single level of indirection will be supported. This resulting type dictates the encoding an decoding steps which ICS will use to handle this parameter. For each type, the following routines (or macros) must be callable to ICS:

  • cli_encode_[ptr_][var_][ool_]param_type() to marshal an input parameter into an ICS request buffer on the client-side;
  • cli_decode_[ptr_][var_][ool_]param_type() to demarshal an output parameter from an ICS reply buffer on the client-side;
  • svr_encode_[ptr_][var_][ool_]param_type() to demarshal an input parameter from an ICS request buffer on the server-side;
  • svr_decode_[ptr_][var_][ool_]param_type() to marshal an output parameter into an ICS reply buffer on the server-side.

Here, the presence of ool_ in the routine name indicates that this parameter is being passed out-of-line and hence needs special handling. Likewise, var_ indicates a variable length parameter and ptr_ indicates a possibly null pointer parameter. OOL parameters also require:

  • cli_encoderesp_ool_param_type() to prepare for an OOL output parameter on the client-side prior to sending the request.

Encoding and decoding routines are provided by ICS for the following base datatypes for non-OOL data:

  • int (long)
  • char
  • char_array

and the following for OOL data:

  • data_t
  • uio_t
  • mblk_t

Marshalling for other commonly used kernel datatypes is defined by header file <cluster/ics_proto.h> and source file <cluster/util/nsc_ics.c>. Routines for all other datatypes must be supplied by individual CI/SSI component using ICS.

2.4. Service Invocation Macros

Calling macros are generated for each of the operations in a service. These generated macros invoke the corresponding ICS client stub routine. The form of the calling macro for RPC operations is:

	OP_NAME(node,rval,argument_list)

 
where:

OP_NAME

is the name of the operation (as it appears in the operation definition, without prefix and in upper-case);

node

is the node number identifying a remote node on which the operation is to be performed;

rval

is a variable to be assigned the return value of the remote operation;

argument_list

is the argument list (excluding destination node) for the operation.

For example, the service invocation macros for the remote PVP service will be generated so that invocation of RPVPOP_SETSID() results in a calls to the ICS client stub routine named cli_rpvpop_setsid().

Services including remote asynchronous operations also generate invocation macros for the send and receive operations. That is, 

                OP_NAME_SEND(node,handle,argument_list)
                OP_NAME_RECEIVE(handle,rval,argument_list)

 
where:

handle

is a message handle returned by _SEND and supplied to _RECEIVE.

Note that handle is returned as the second parameter to the _SEND call and the same value must be supplied as the first parameter to the _RECEIVE call. To the caller, handle is of type cli_handle_t ** for the send call and cli_handle_t * for the _RECEIVE call. Note also that the _RECEIVE macro is not provided with the destination node and it returns rval.

Remote operation macros return an integer value indicating the success (or otherwise) of the attempt to perform the remote operation. If a non-zero value is returned, then neither rval nor any output parameter can be assumed valid.

For MSG operations with no value returned to the caller of ICS, rval is absent from the calling macro, which is thus in the form: 

                OP_NAME_MSG(node,argument_list)

Note that operations defined as either MSG or ASYNC can be called in the normally synchronous RPC fashion since this form of client stub code is always generated and the server-side code behaves identically irrespective of the manner of client-side call. In particular, this enables the caller to wait for confirmation that a message operation has been performed remotely.

2.5. Summary Format of Generated Client-side Stubs

The basic form of the client-side data marshaling stub is:

 

	int
	cli_<op_name>(
		node_t                  node,
		int                     *rval,
		<in_param_type1>        <in_param_name1>,
		<inout_param_type1>     <inout_param_name1>,
		<out_param_type1>       <out_param_name1>,
		...
		)
	{
		<Local variables generated here>
		cli_handle_t *handle = icscli_handle_get(node, <service_num>, TRUE);
 
		/*
		 * Marshal input arguments into transport buffer
		 */
		cli_encode_<in_param_type1>(handle,<in_param_name1>,...);
		cli_encode_<inout_param_type1>(handle,<inout_param_name1>,...);
		/* ... as appropriate for all input parameters to be encoded ... */
 
		cli_stat = icscli_send(handle,...);     /* send */
		cli_stat = icscli_wait(...);            /* wait for reply */
 
		/*
		 * Decode the result code for the remote operation
		 */
		cli_decode_int_p(handle,rval);
 
		/*
		 * Marshal output arguments from transport buffer
		 */
		cli_decode_<inout_param_type1>(handle,<in_param_name1>,...);
		cli_decode_<out_param_type1>(handle,<inout_param_name1>,...);
		/* ... as appropriate for all output parameters to be decoded ... */
 
	out:
		icscli_handle_release(handle);
		return (cli_stat);
	}		
 

This pattern is complicated somewhat by additional generated code to handle variable length and OOL datatypes, and by asynchronous versions of the operation.

 

2.6. Summary Format of Generated Server-side Stubs

	int
	svr_<op_name>(svr_handle_t *handle)
	{
		int rval;
		<Operation arguments generated as local (stack) variables here>
 
		/*
		 * Marshal input arguments from transport buffer onto stack
		 */
		svr_decode_<in_param_type1>(handle,<in_param_name1>,...);
		svr_decode_<inout_param_type1>(handle,<inout_param_name1>,...);
		/* ... as appropriate for all input parameters to be encoded ... */
 
		icssvr_decode_done(handle);
 
		<op_name>(this_node, &rval, <argument_list>);
 
		/*
		 * Encode the result of the operation
		 */
		svr_encode_int(handle, rval);
 
		/*
		 * Marshal output arguments from stack into transport buffer
		 */
		svr_encode_<inout_param_type1>(handle,<in_param_name1>,...);
		svr_encode_<out_param_type1>(handle,<inout_param_name1>,...);
		/* ... as appropriate for all output parameters to be decoded ... */
 
		/*
		 * If an error occured while encoding output params, reset the
		 * reply message and encode only the failure code.
		 */
	out:
		if (svr_stat) {
			icssvr_rewind(handle);
			svr_encode_int(handle,svr_stat);
		}
		return(0);	
	}
 

 Stack variables are allocated for parameters in the following manner:

 

  • IN parameters passed by value have buffer space for the demarshaled parameter value reserved.

  • IN parameters pass by reference have a pointer variable allocated containing the address of a stack variable into which to demarshal the parameter value.

  • OUT parameters must be passed by reference. These have a pointer variable allocated containing the address of a stack variable into which the value is output and from which the returned value is encoded.

  • IN:OOL parameters must be passed by reference. A pointer variable with NULL value is allocated.

  • OUT:OOL parameters must be passed by double indirection. A pointer variable is allocated pointing to a NULL-valued pointer to the underlying OOL datatype; the output pointer being returned into this second pointer.

2.7. Generated Prototypes

For each service, header files are generated containing ANSI C function prototype declarations for all stubs and server functions. Inclusion of these header files in CI/SSI components using ICS ensures compatibility of the calling interface with ICS. The generated stubs also include these prototypes for self-consistency.

2.8. Generated Dispatch Table and Service Registration Routine

Additionally, a server dispatch table and a service registration routine is constructed for each service. This table enables the ICS server message handler to call the server function corresponding to the requested operation on the remote node. The form of the table is:

  

	struct {
		int (*<op_name1>)(svr_handle_t *hndl);
		int (*<op_name2>)(svr_handle_t *hndl);
		/* ... repeated for each operation */
		int (*<op_namen>)(svr_handle_t *hndl)
	} icssvr_<svc_name>_stubs = {
		svr_<op_name1>,
		svr_<op_name2>,
		/* ... repeated for each operation */
		svr_<op_namen>
	}

The prototype for the service registration routine is: 

	extern int <svc_name>_svc_init(void);

This routine must be called before ICS will service operation requests for the service. 

2.9. Interface Examples

This section illustrates the interface of ICS with CI/SSI components (the service definer, provider and caller) in the form of a simple example. It shows how a service and its constituent operations are defined and what routines must be provided for ICS to call. 

Suppose we have a service named foo having an RPC operation named bar. Service definition file foo.svc contains:

 

	service	foo NSC_FOO_SVC 1 2
 
	operation rfoo_bar {
		parameter IN:VAR char arg[MAX_ARGLEN]
		parameter OUT result_t *result
	}

Functions must be provided to invoke and to service this operation in the following form:

 

	#include <ics_foo_protos_gen.h>	/* generated prototypes */
 
	int
	foo_bar(node_t node, char *arg, result_t *result)
	{
		if (node != this_node) {
			/*
			 * Call ICS to perform this operation on remote node.
			 */
			int rval;
			int cli_stat;
			cli_stat =  RFOO_BAR(node, &rval, arg, strlen(arg), result);
			if (cli_stat != 0)
				panic("ICS remote operation failed");
			return(rval);
		}
		*result = do_bar(arg);
		result(0);
	}
 
	/*
	 * This routine is called to service a remote RFOO_BAR() request.
	 */
	int
	rfoo_bar(node_t node, int *rval, char_t arg, int arg_len, result_t *result)
	{
		*rval = foo_bar(node, arg, arg_len, result);
		return(0);
	}
 

If instead, this operation is declared asynchronous, viz:

 

	service	foo NSC_FOO_SVC 1 2
	
	operation rfoo_bar ASYNC {
		parameter IN:VAR char arg[MAX_ARGLEN]
		parameter OUT result_t *result
	}

 

then the invoking function might take the following form:

 

	#include <ics_foo_protos_gen.h>		/* generated prototypes */
 
	int
	foo_bar(node_t node, char *arg, result_t *result)
	{
		if (node != this_node) {
			/*
			 * Call ICS to perform this operation on remote node.
			 */
			int rval;
			int cli_stat;
			cli_handle *hndl;
			cli _stat = RFOO_BAR_SEND(node, &hndl, arg, strlen(arg), result);
			if (cli_stat != 0)
				panic("ICS remote operation failed");
 
			/* 
			 * While this is happening, let's do something else...
			 */
			do_be_do();
 
			/*
			 * Now wait for the remote operation to complete.
			 */
			cli_stat = RFOO_BAR_RECEIVE(hndl, &rval, arg, strlen(arg), result);
			if (cli_stat != 0)
				panic("ICS remote operation failed");
 
			return(rval);
		}
 
		*result = do_bar(arg);
		return(0);
	}

Note that the service routine is unchanged since it is unaware whether the client is making the request synchronously or asynchronously.

Finally, removing the output parameter from operation bar, we could redefine this operation as a message:

 

	service	foo NSC_FOO_SVC 1 2
 
	operation bar MSG {
		parameter IN:VAR char arg[MAX_ARGLEN]
	}

then the client and server functions could take following form: 

 
	#include <ics_foo_protos_gen.h>		/* generated prototypes */
 
	int
	foo_bar(node_t node, char *arg)
	{
		if (node != this_node) {
			/*
			 * Call ICS to request this operation on remote node.
			 * But we don't wait around for any confirmation or reply.
			 */
			int cli_stat;
			cli_stat = RFOO_BAR_MSG(node, arg, strlen(arg));
			if (cli_stat != 0)
				panic("ICS remote operation failed");
			return(0);
		}
		
		do_bar(arg);
		
		return(0);
	}
 
	/*
	 * This routine is called to service a remote RFOO_BAR() request.
	 */
	int
	rfoo_bar(node_t node, char_t arg, int arg_len)
	{
		(void) foo_bar(node, arg, arg_len);
		return(0);
	}

2.10. icsgen

The icsgen build phase constructs the following components for defined operations:

1.

2.

3.

4.

5.

Macros to invoke the remote operations.

Prototype declarations for the remote operation routines.

Client interface stubs for remote operations.

Server interface stubs for remote operations.

Dispatch tables used internally by ICS.

Files are generated by icsgen containing the required components. Generated files use a file naming scheme in which the suffix _gen precedes the file extension, e.g. ics_vproc_protos_gen.h.

The build process to generate the required output text files is driven by make commands

using a master script called icsgen. icsgen is defined as follows:

2.10.1. Syntax

	icsgen [[-a] | [-c] [-m] [-p] [-s] [-t]] \
	       [-D def] [-U undef] [-n name] [-o outfile] svc_files

2.10.2. Description

icsgen derives output text selected by option switch from a given service definition file, svc_file. If no svc_file is specified, standard input is read. Output is written to a file bearing the default name of the requested transformation or to a specified outfile. Output text is selected by the following options:

-a generate all the following (this is the default);

-c generate client-side stubs;

-m generates invocation macros;

-p generates prototype declarations for stubs and server routines;

-s generates server-side stubs;

-t generates server dispatch table;

The input is read and passed through the C preprocessor before conducting the required transformation. 

2.10.3. Options

-D preprocessor #define;

-U preprocessor #undefine;

-o specifies outfile filename.

2.10.4. Files

The default output files created by icsgen are as follows:

  • icscli_svc_gen.h - client-side stubs;

  • ics_svc_macros_gen.h - invocation macros;
  • ics_svc_protos_gen.h - prototype declarations;
  • icssvr_svc_gen.c - server-side stubs;
  • icssvr_svc_tables.c - server dispatch table and service registration routine.

 where svc is the service name declared in the service definition file. 

2.10.5. Examples

	icsgen -p pvp.svc as.svc

Generates vproc operations prototype file ics_vproc_protos_gen.h from service master files pvp.svc and as.svc. 

	icsgen -c pvps.svc > icscli_pvps_gen.c

Generates client operation stubs for private virtual system operations defined in file pvps.svc and these are written to file icscli_pvps_gen.c. 

2.11. Source and Build Environment

For a CI/SSI component using ICS to provide transport for a service, in addition to the service operations routines, the following files must be provided:

  • A .svc file defining the service, its operations and their parameters. For services employing the emulated nsc_rcall() interface, an RPCGEN .x file will be present and ICS service declarations may be embedded in this rather than being placed in a separate .svc file.
  • A file containing invocation of remote services via OP_NAME() macros. Typically, individual operations routines include local code and remote invocations.
  • A file containing the remote server routines op_name() which are called by the ICS server stubs routines. These routines may be included in the same file as the local opertaions routines, or they may be placed in a separate source file with name such as svc_name_server.c.
  • A source file to (pre-processor) include the generated ICS client and server stubs and the server tables and registration routine. This file must also include definition of encode/decode macros for special datatypes used by the service.

The nature of the encode/decode macros is discussed further below. 

2.12. Defining Encoding and Decoding

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- cli_encoderesp_ool_macro is a NO-OP. This was used to support

DMA-push/DMA-pull over the Servernet interconnect.

***********************************************************

Generated ICS client and server stubs contain encode and decode calls to marshal parameters of specified datatype. Each invocation may be a call to a routine to perform the required marshaling or it may be a invocation of a macro to do this. However, the macro form is the primary interface and in some cases (when a parameter name needs conversion to its address) a macro will be required to effect a correct entry to a routine. Marshaling for a number of commonly used datatypes is defined as macros in CI/SSI header file <cluster/ics_proto.h>. Marshaling for other datatypes must be defined on a component by component basis.

Note that marshaling required for a particular datatype depends on the type and attributes of the parameter.

  • IN parameters require cli_encode_ and svr_decode_ macros.
  • OUT parameters require svr_encode_ and cli_decode_ macros.
  • INOUT parameters require both of the above.
  • IN:FREE parameters require an additional svr_free_ macro.
  • OUT:OOL parameters require an additional cli_encoderesp_ool_ macro.
  • IN:OOL parameters require an additional svr_free_ool_ macro.

Encode/decode macros for non-OOL and non-VAR types take 2 arguments the handle and the parameter name. OOL and VAR parameter macros additionally take a length parameter.

All OOL encode and decode macros can fail and return a non-zero status.

Whether passed by reference or value, non-OOL parameters will have space reserved for them on the server's stack and svr_decode_ macros use this for demarshaling input values and marshal output values.

OOL parameters require kernel memory to be allocated to receive the transferred data. It is the responsibility of the encode/decode macros to make this allocation and to ensure that the space is freed when appropriate after use. On the server-side, stack space is allocated for a pointer to the OOL datatype. 

2.13. XDR-defined Datatypes

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- This section has been implemented, although not extensively tested.

(In other words, use at your own risk!).

- ICS marshalls all xdr parameters into iovecs pointed to by a msghdr

struct. This msghdr struct is then encoded/decoded using the ICS

routines:

icscli_llencode_ool_struct_msghdr
icscli_lldecode_ool_struct_msghdr

***********************************************************

Special handling is provided for complex datatypes involving trees defined in an xdr-able form (by .x files). Serialization (encoding) and deserialization (decoding) is performed through xdr routines generated by rpcgen. With ICS, this process involves construction of encoded mblk chains which are transferred out-of-line to the target node, decoded and freed. The ICS marshaling code arranges for one mblk chain to be dedicated to each parameter exchanged in this way. The ICS routines icscli_encode_ool_mblk_t(),  icscli_decode_ool_mblk_t(),  icssvr_encode_ool_mblk_t(),  icssvr_decode_ool_mblk_t() are called to perform the transfer of the mblk chains.

To use XDR marshaling, encode/decode macros must call a special handling routine and quote the associated XDR marshaling routine to be used. On the encode side this routine is icscli_encode_xdr() or icssvr_encode_xdr() and on the decode side it is icscli_decode_xdr() or icssvr_decode_xdr(). Also, a generic free routine icssvr_free_xdr() is available for server-side use to free IN parameters after the remote operation has completed. These routines are called with the following parameters:

  • ICS handle associated with the call;
  • address of the data to be coded;
  • XDR routine to marshal the data type.

XDR routines are generated from a conventional .x file by rpcgen or they may be hand-coded. The data address given follows the usual conventions for XDR - in particular, a NULL address will cause memory to be allocated automatically on decode.

Note that using XDR involves a data copy for both encode and decode phases and therefore should be avoided for large data sizes if standard ICS out-of-line support can be exploited for greater efficiency.

 

2.14. nsc_rcall() Emulation

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- This section has been implemented, although not extensively tested.

(In other words, use at your own risk!).

- .x files are generated using rpcgen.

- The icsnsc_rcall macro function is obsoleted. Use nsc_rcall()

directly.

- Credentials are not yet supported.

- Data is marshalled/demarshalled in a msghdr struct.

- RPC client stub for nsc_rcall service is called:

cli_icsnsc_rcall()

- MSG client stub for nsc_rcall service is called:

cli_icsnsc_rcall_msg()

- Server stub for nsc_rcall service is called:

svr_icsnsc_rcall()

***********************************************************

As a compatibility and code migration path from the former nscrpc subsystem, the nsc_call() interface is emulated by ICS. This enables non-ICS style CI/SSI components - with RPC interfaces defined by .x file, compiled by rpcgen and employing the ONC-style RPC paradigm - to use ICS with minimal change to the RPC-calling code.

This emulation is conducted above the ICS data marshaling layer. Firstly, client calls to nsc_rcall() are redirected to an ICS routine, icsnsc_rcall(), by a #define of the form:

 

	#define tnc_rcall(h,proc, xargs, args, xresults, resultsp,flag) \
	    icstnc_rcall(h,proc,xargs,args,xresults,resultsp,flag,  \
		         cli_<svc>_rcall,cli_<svc>_rcall_noreply_msg)

The addtional parameters to icsnsc_rcall() indicate special ICS client stub routines. icsnsc_rcall() then:

  • marshals argument data into mblk's using the specified XDR routine,
  • obtains credentials information for the caller,
  • calls the appropriate ICS client stub according to whether a reply is expected,
  • demarshalls any results data from returning mblk's which are freed.

The special client stubs correspond to two special ICS operations (an RPC and a message) defined for the service as follows:

 

	operation <svc>_rcall {
        	param IN:PTR:FREE cred_t *credp    /* client credentials */
		param IN int             procnum   /* operatioin proc number */
		param IN:OOL mblk_t      *margsp   /* mblks encoded with args */
		param OUT:OOL mblk_t     **mrespp  /* mblks encoded with results */
    	}
	operation <svc>_rcall_noreply MSG {
		param IN:PTR:FREE cred_t  *credp   /* client credentials */
		param IN int              procnum  /* operation proc number */
		param IN:OOL mblk_t       *margsp  /* mblks encoded with args */
	}
 

On the server-side, the corresponding ICS server stubs are called. These direct the requests to a common rcall dispatching routine, icsnsc_rpc_dispatch(), along with the server routine table entry to be dispatched. The mapping from server stub to this dispatcher is performed by further a macro of the form: 

 
	extern int <svc>_0_nproc;
	extern nsc_rpccall_t nsc_rpc_<svc>_0[];
	#define <svc>_rcall(node,rval,redp,procnum,margsp,mrespp)    \
		ASSERT(node == this_node);                           \
		ASSERT(0 <= procnum && procnum < <svc>_0_nproc);     \
		*rval = icsnsc_rpc_dispatch(
			credp,&nsc_rpc_<svc>_0[procnum],margsp,mrespp)
	/*
	 * Map the message-handling server stub to themessage version,
	 * but with a NULL result pointer.
	 */
	#define <svc>_rcall_noreply(node,redp,procnum,margsp)     \
		ASSERT(node == this_node);                        \
		ASSERT(0 <= procnum && procnum < <svc>_0_nproc);  \
		(void) icsnsc_rpc_dispatch(
			credp,&nsc_rpc_<svc>_0[procnum],margsp,NULL)
 

icsnsc_rpc_dispatch() then proceeds to:

  • demarshal argument data from incoming mblk's by calling the corresponding XDR routine and then freeing the mblk's,
  • switch to the client's credentials,
  • call the corresponding server function,
  • revert to the server's credentials,
  • if results are to be returned, XDR these into outgoing mblk's,
  • free the results data if there's a corresponding XDR free function defined in the rcall table.

The above lines of macros must be inserted (prefixed by %'s) into the .x file. Additionally, the ICS service definition is placed in the .x file surrounded by pre-processor conditions which render these visible to the icsgen and not to rpcgen. Finally, other pre-processor directives are added to include ICS-generated stubs at the appropriate place. Thus, by suitable .x and makefile changes alone, a CI/SSI subsystem can be converted to use the ICS nsc_rcall() emulator with no further code change. Moreover, the conversion is achieved as a build option. 

2.15. Verification Tests

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- Not implemented.

***********************************************************

Under the TNC_DEBUG build option, a series of tests of ICS operation is available through the tncsys() system call. At the command-line level these tests can be invoked by the command: 

	node -I nodenum

This verifies a range of data type marshaling/de-marshaling, data transmission, and operation attributes (RPC, message, asynchronous) from the node of the calling process to a nominated second node.

Any test failure is logged to the console of the node detecting the error. Subsequent tests are continued, however. One or more test error being returned to the invoking process.

The source for these tests is to be found in file tnc/tnc_icstest.c and the ICS test operation are defined in file tnc/tnc_icstest.svc.

The test sequence is as follows:

1.

Test primitive types and basic RPC support.

a. Scalar integer values.
IN, OUT, and INOUT integer values are exchanged. This verifies the
basic operation of the ICS subsystem, the generated ICS client and
server stubs and the interface with the ICS subsystem: in particular,
routines icscli_handle_get(),
icscli_encode_int32(), icscli_decode_int32(),
icssvr_decode_int32(), icssvr_encode_int32(),
icscli_send(), and icscli_handle_release().

b. Variable-length integer arrays.
IN, OUT, and INOUTVAR integer arrays are exchanged. This verifies
the generation of ICS stubs for these types and of ICS interface
routines: icscli_encode_int32_array_t(),
icscli_decode_int32_array_t(),
icssvr_decode_int32_array_t(), and
icssvr_encode_int32_array_t().

c. Fixed and variable length character arrays.
IN, OUT, and INOUT fixed-length char arrays, and IN and INOUT
VAR arrays are exchanged. This verifies the generation of ICS stubs for
these types and of ICS interface routines:
icscli_encode_char_array_t(), icscli_decode_char_array_t(), icssvr_decode_char_array_t(),icssvr_encode_char_array_t(). Also, the use of ICS marshaling macros is verified for the handling of fixed-length data stuctures (in this case a simple character array).
2. Out-of-line data.
IN, OUT, and INOUT fixed-length char arrays, and IN and INOUTVAR arrays are exchanged as out-of-line data. This verifies the generation of ICS stubs for these types and of ICS interface routines: icscli_encode_ool_data_t(),
icscli_encoderesp_ool_data_t(),
icscli_decode_ool_data_t(),
icssvr_decode_ool_data_t(), and icssvr_encode_ool_data_t(). In addition, ICS marshaling support routines to allocate and free memory are exercised.
3.

Special data marshaling.
This series of tests verifies special-purpose datatype handling provided at the ICS stub level.

a. PTR and FREE parameter attributes.
PTR parameters are potentially NULL pointers to IN data structures.
The FREE attribute requests that the server stub call a free routine for a
datatype after the remote procedure returns. This test verifies that
non-NULL and NULL pointers are correctly marshalled and that the
associated free routine is called.

b. uio and iovec marshaling.
This test verifies that a combined uio/iovec structure pair is correctly
marshaled to and from a remote node. Note that such data structures are
always INOUT.

c. XDR and credentials type marshaling.
This test exercises several capabilities:

· the use of XDR marshaling routines to transfer data structure in
the form of out-of-line encoded mblks;

· ICS credentials marshaling macros for the exchange of credentials
information (including NULL and system credentials) as ICS
in-line data; and

· the equivalence of credentials marshaled in either of these ways.

This tests generated stub code and the support routines for interfacing
with standard XDR marshaling routines and the production and
consumption of data-encoded mblks.

 

4.

Asynchronous capabilities (ASYNC operation attribute).
In this test, an ASYNC operation is first preformed synchronously as a conventional RPC and then _SEND and _RECEIVE portions are performed separately with a one second delay between.

This is a test of ICS stub generation only; the ICS subsystem does not distinguish between synchronous and split asynchronous operation.
5. Messaging (MSG operation attribute).
This tests the ability of ICS to perform a remote operation without returning data and with or without the client awaiting completion. There are 2 sub-tests. In both tests data is returned to the invoking node by means of a double-hop message: test node to second node and second node back to test node. In the first test, both hops are performed synchronously - the caller blocks until the remote operation has completed. In the second test, each hop is fully asynchronous with the testing node waiting one second to allow the return message to be delivered.
6.

Sundry tests.
Finally, a series of unusual parameter configurations is tested to verify ICS stub generation logic:

a. no input and no output parameters;

b. no output parameter; and

c. no input parameter.

 

7. Operation placeholder (SKIP operation attribute).
A SKIPped operation is included in the .svc file to verify that no stubs are generated for it but that the operation is correctly allocated an unused entry in the generated server table.

3. High-level ICS

This section of the document describes the high-level ICS interface (often referred to as
the ICS interface). The high-level ICS code is used for all inter-node communication.
High-level ICS routines are of several categories:

  • general purpose routines for initializing ICS, declaring nodes up and down, etc. 

  • ICS client routines which are called by icsgen-generated client stubs.

  • server control code, which controls the creation and destruction of server handles and server daemons, and invokes the icsgen-generated server stubs.
  • ICS server routines which are called by icsgen-generated server stubs.

This section of the document contains several subsections. The first describes how ICS
deals with CI/SSI services and communication channels; the next subsection describes
the ICS server control code; the following two subsections describes some typical
scenarios for the invocation of high-level ICS client and server code; and the last
subsections contain interface descriptions for all high-level ICS routines. 

3.1. CI/SSI Services and ICS Communication Channels

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- ICS channels are 2-way communication channels implemented over

TCP.

- ICS_NUM_CHANNELS, ICS_MIN_PRIO_CHAN, ICS_MAX_PRIO_CHAN,

ICS_REPLY_CHAN, ICS_REPLY_PRIO_CHAN have been renamed to

lower-case equivalents:

ics_num_channels

ics_min_prio_chan

ics_max_prio_chan

ics_reply_chan

ics_reply_prio_chan

- Throttling/Flow control not supported. The code is

#ifdef'ed ICS_THROTTLE.

***********************************************************

CI/SSI components utilize services when communicating with ICS. CI/SSI messages and replies are sent out over communication channels. There are ICS_NUM_CHANNELS separate one-way communication channels that are used to communicated between nodes - they are identified by channel number, which goes from zero to (ICS_NUM_CHANNELS-1). Messages and RPCs sent using a CI/SSI service get mapped to a specific communication channel. More than one CI/SSI service may be mapped to a specific communication channel (the mapping of CI/SSI services to communication channels is done in the file <cluster/ics/ics_conf.c>).

Services are grouped together on a channel because communication channels are the entities that are flow-controlled (also known as throttled). Throttling of communication channels is accomplished by stopping all communication over the channel. This is done when the client or server starts running low on memory/resources, and it is desirable for the ICS subsystem (and the CI/SSI components using the ICS susbsystem) to not exhaust all memory/resources. On the server node, the resource limits used to check to see if throttling should occur exist on a per-communication-channel basis (see ics_svr_register() for details), and hence, the grouping of servers on communication channels is relevant.

However, if communication channels could be arbitrarily throttled, deadlock scenarios could arise (e.g., certain RPCs may need to complete to free resources, yet the communication channel needed to complete the RPC stays throttled until the resources are freed). To prevent such deadlocks, the concepts of ICS priority and priority communication channels exist. The communication channels ICS_MIN_PRIO_CHAN..ICS_MAX_PRIO_CHAN are deemed high priority channels, and these channels are never throttled. No CI/SSI services are mapped into the priority channels in <cluster/ics/ics_conf.c>. To make a message or RPC use one of these high priority communication channels, the routine ics_setpriority() should be used to raise the priority level of the current thread. An ICS priority of zero does not use the high priority communication channels (the mapping of CI/SSI services to communication channels specified in <cluster/ics/ics_conf.c> is utilized); an ICS priority of one will use the communication channel ICS_MIN_PRIO_CHAN (regardless of the CI/SSI service); an ICS priority of two will use the communication channel ICS_MIN_PRIO_CHAN+1 (regardless of the CI/SSI service); etc.

Some of the most common deadlock scenarios relate to performing an RPC within an ICS server. An example of this is performing an RPC from node A to node B; the communication channel becomes throttle; the RPC service on node B performs an RPC back to node A; this RPC would free resources on node A, yet the RPC cannot complete because the communication channel is throttled. To prevent these type of deadlock scenarios, the ICS server control code automatically raises the ICS priority of any outgoing message or RPC to be one higher than the incoming RPC which invoked the outgoing message/RPC (if it was an incoming message rather than an RPC, then ICS priorities are not automatically raised).

Replies to non-high-priority RPCs take place over the ICS_REPLY_CHAN communication channel. Replies to all high priority RPCs take place over the ICS_REPLY_PRIO_CHAN communication channel. Neither of these reply channels is ever throttled. No CI/SSI services are mapped to these channels in <cluster/ics/ics_conf.c>. 

3.2. ICS Server Control Code

While a major portion of ICS is devoted to the routines called by the icsgen-generated stubs, another major piece of ICS is the server control code, which controls server daemons and server handles.

For ICS server daemons, the server control code will attempt to keep icsdaemon_avail_lwm to icsdaemon_avail_hwm ICS server daemons in an available state (not processing messages or RPCs). If the number of server daemons is less than icsdaemon_avail_lwm, more daemons are created. If the number of server daemons is greater than icsdaemon_avail_hwm, server daemons are destroyed. Server daemons are used to handle both normal and high priority messages and RPCs. However, there also exists a special server daemon for each priority greater than one. If, when a high priority message or RPC arrives, there are no server daemons that are in an available state, the special daemon for the designated priority level is used. 

For ICS handles, the server control code attempts to keep lwm_svc_handles to hwm_svc_handles in an icssvr_handle__recv() state (these limits are on a per-CI/SSI-service basis - see ics_svc_register() for details). If the number of handles performing icssvr_handle_recv() becomes less than lwm_svc_handles, then more server handles are created. If the number of handles performing icssvr_handle_recv() becomes greater than hwm_svc_handles, then server handles are destroyed. `

3.3. Typical ICS Client Usage Scenarios

***********************************************************
Errata 10/15/01 for Linux CI and SSI Clusters

- ICS messages cannot be sent in interrupt context.

This could be fixed, if we set all ICS channel's sk->allocation

to be GFP_ATOMIC, instead of GFP_KERNEL.

***********************************************************

This section presents a few typical examples of how high-level ICS code interfaces are used on a client node.

The examples in this section describe the typical code sequences generated by icsgen for client stubs.

The first example is for the client to perform an RPC (Remote Procedure Call). Note that this example assumes the client code is in thread context.

1.

Obtain a client handle using icscli_handle_get().

2.

Set up all the necessary input arguments using the icscli_encode_*() routines.

3.

Set up all the necessary information for out-of-line output arguments using the icscli_encoderesp_ool_*() routines.

4.

Call icscli_send() to send the message to the server. icscli_wait_callback() is typically specified as the callback routine.

5.

6.

Wait for the reply to come back, typically using icscli_wait().

Obtain all the output arguments from the response message using the icscli_decode_*() routines.

7.

Release the client handle using icscli_handle_release().

The second example is for the client to send a message to the server. Note that this example can take place in interrupt context, as long as the restrictions described in the example are followed.

1.

Obtain a client handle using icscli_handle_get(). If the IPC is taking place in interrupt context, then make sure that sleep_flag is FALSE.

2.

Set up all the necessary input arguments using the icscli_encode_*() routines. If the IPC is taking place in interrupt context, then make sure that only icscli_encode_*() routines that are appropriate for interrupt context are used.

3.

Call icscli_send() to send the message to the server. The callback routine would typically be icscli_handle_release_callback().

The third example is for the client to perform a ``broadcast RPC'', where the ``send'' portion of the RPC is performed to a series of nodes, then work is done locally, then the ``wait for response'' portion of the RPC is performed for the same series of nodes. This example could be used to implement spanning tree algorithms. This example assumes that the code is running in thread context.

1.

Obtain the appropriate client handles using icscli_handle_get() (one handle for each node).

2.

Set up all the necessary input arguments using the icscli_encode_*() routines (the argument encoding must take place for each handle).

3.

Set up all the necessary information for out-of-line output arguments using the icscli_encoderesp_ool_*() routines (again, this must take place for each handle).

4.

Call icscli_send() for each handle to send the messages to all the servers. The callback routine specified would typically be icscli_wait_callback().

5.

6.

Perform any work that can be performed locally.

Wait for all the replies to arrive back on this node by calling icscli_wait() for each handle.

7.

Obtain all the output arguments using the icscli_decode_*() routines (the argument decoding must take place for each handle).

8.

Release the client handles by calling icscli_handle_release() for each handle.

3.4. Typical ICS Server Usage Scenarios

This section presents a few typical examples of how high-level ICS code interfaces are used on a server node.

The server control code is responsible for obtaining the appropriate number of server handles using icssvr_handle_get(), and then calling icssvr_recv() for each of these handles. If the service is one that is to be performed from thread context, then the server control code is also responsible for creating the appropriate number of ICS server daemons, so that any messages received by the server can be processed in a timely fashion.

The following example describes how the server end of an RPC (Remote Procedure Call) is handled. This example assumes that the callback routine for icssvr_recv() has arranged to wake up a server daemon and that the following server code is executed within the context of the server daemon. In this example, steps 1 though 4 are performed by the icsgen-generated server stubs. The remaining steps are the responsibility of the server control code. 

1. Obtain all arguments from the message by using calls to the
icssvr_decode_*() routines.

2.

Call icssvr_decode_done() to indicate that all argument decoding has been completed.

3.

4.

Perform whatever server work there is to perform.

Put any output arguments for the RPC into the response message by using the icssvr_encode_*() routines.

5.

Call icssvr_reply(). The callback routine specified should merely arrange to call icssvr_recv().

6.

Put the daemon back to sleep, waiting for another message, in whatever fashion the server chooses.

The following example describes how the server end of an IPC (i.e. message send) is handled. This example assumes that the callback routine for icssvr_recv() has arranged to wake up a server daemon and that the following server code is executed within the context of the server daemon. In this example, steps 1 though 4 are performed by the icsgen-generated server stubs. The remaining steps are the responsibility of the server control code.

1. Obtain all arguments from the message by using calls to the
icssvr_decode_*() routines.

2.

Call icssvr_decode_done() to indicate that all argument decoding has been completed.

3.

Immediately reply to the client by using the icssvr_reply() call. The callback routine could merely arrange to call icssvr_recv().

Internode Communication Subsystem (ICS) Design Specification Page 25

4.

Perform whatever server work there is to perform.

5.

Put the daemon back to sleep, waiting for another message, in whatever fashion the server chooses.

The following example describes how the server end of an IPC (i.e. message send) is handled, when the entire server is to run in interrupt context (e.g. to handle STREAMS aqueduct messages). This example assumes that the callback routine for icssvr_recv() has arranged to call the following sequence of code, which possibly r