4 - Domus Clusters:

The following instructions are assumed to be executed by a regular "user".

4.1 - Preliminary Notes:

• types of Domus services: a Domus (logical) cluster is based on a supervisor service (that usually runs in the frontend node of the physical cluster) and a set of regular services (that run on worker nodes);
• binding of Domus services to network interfaces: Domus services (supervisor and regular) are uniquely identified by a network interface (more specifically, by its DNS name or IP address) and a TCP/IP port; because a cluster node may have several network interfaces then, attaching a service to a network interface makes possible to operate a Domus cluster on specialized interconnects (e.g., Myrinet or Infiniband); in any case, supervisor and regular services should all operate on the same network in order to avoid IP routing performance penalties;
• number of Domus regular services per cluster node: this number should be no more than one (for each Domus cluster); it is conceivable to have more than one regular service of the same Domus cluster running in the same cluster node, provided each service is attached to different network interfaces of the node; such scenario, however, is not endorsed by the current Domus implementation, once it violates several architectural invariants;

4.2 - Pre-Configuration:

1) A regular "user" starts by creating a proper local Domus environment in its account:

	
[user@frontend]# cd ~
[user@frontend]# mkdir .domus ./domus/etc
[user@frontend]# cp ~domus/etc/domus#mycluster#* ./domus/etc
[user@frontend]# ls -la ./domus/etc
/home/user/.domus/etc:
total 16
drwxrwxr-x  2 user user 4096 Feb  7 11:00 .
drwxrwxr-x  4 user user 4096 Feb  4 23:16 ..
-rw-r--r--  1 user user 1719 Feb  7 11:00 domus#mycluster#conf
-rw-r--r--  1 user user   36 Feb  4 23:02 domus#mycluster#interfaces

[TODO: make a script to automate the creation of ~/.domus]
The domus#mycluster#* files are tailored to a generic Domus cluster named mycluster. If a user wants to operate several Domus clusters at the same time, specific configuration files will be needed for each one. In this regard, it is a good practice (though not mandatory) to embedded the unique name of each Domus cluster in its configuration files (e.g., as in domus#mycluster#conf and domus#mycluster#interfaces, for a Domus cluster named mycluster). The role of the domus#mycluster#* files is as follows:
• domus#mycluster#interfaces : network interfaces (DNS names or IP addresses) that identify the cluster nodes that support the Domus cluster at its inception; note that primary network interfaces are often identified by the hostname of their host, as illustrated by the supplied domus#mycluster#interfaces file:

  	
  compute-1-0.local
  compute-1-1.local
  ...
  

When deploying Domus in a batch-managed ROCKS cluster under the control of PBS (see 7 - Operation in Batch Clusters) this file will be generated automatically; in such context, the file conveys the subset of cluster nodes that were given to the "user", in order to operate its Domus (logical) cluster. If the target environment is a shared cluster, the "user" that owns the Domus cluster is responsible to populate the file; however, in this context, the file is not mandatory, though it may still be useful, as explained right away.
Thus, the sole purpose of this file is to automate certain tasks, like the creation of a Domus cluster with a certain set of initial services. Moreover, if the set of Domus services is static and specified by this file, one can manage the Domus cluster very conveniently, using the domus_manager_cluster.py tool (see 4.3 - Management Tools). Generically, this file may be viewed as a description of the initial composition (which nodes/services) of a Domus cluster.
[TODO: it should not be difficult, to have the supervisor of a Domus cluster to have this file permanently updated; this would definitely improve its usefulness]
• domus#mycluster#conf : configuration parameters for the Domus services; the contents of the sample file provided are shown bellow:

  	
  
  ################################################################################
  
  DOMUS_DEBUG          1                    # 1 == on; 0 == off; relevant only if
                                            # --debug was assumed during installation
  DOMUS_HOME_DIR_PATH  /home/domus/.domus   # where to find Domus after installation;
                                            # must be NFS-accessible to all users
  
  DOMUS_GANGLIA_HOST     alfa               # ROCKS cluster frontend
  DOMUS_GANGLIA_CHANNEL  235.185.190.208    # see /etc/gmond.conf at the frontend
  DOMUS_GANGLIA_PORT     8649               # see /etc/gmond.conf at the frontend
  DOMUS_GANGLIA_GMETRIC_PATH /opt/ganglia/bin/gmetric   # default: this same path
                                      
  DOMUS_THRESHOLD_CPU    0.8                # (%); resource utilization thresholds;
  DOMUS_THRESHOLD_RAM    0.8                # prevent the creation of DHTs on 
  DOMUS_THRESHOLD_DISK   0.8                # overloaded nodes; also used to weight 
  DOMUS_THRESHOLD_DISKIO 0.8                # the participation of nodes in DHTs 
  DOMUS_THRESHOLD_NET    0.5                # (in combination with benchmark results)
  
  ################################################################################
  
  DOMUS_CLUSTER_NAME                  mycluster #unique name of the Domus cluster;
  DOMUS_CLUSTER_INTERFACES_FILE_PATH  /home/user/.domus/etc/domus#mycluster#interfaces
  
  DOMUS_SUPERVISOR_ROOT_DIR_PATH /home/user/.domus/var  # "supervisor" working dir
                                                        # (shared by all instances)
  
  DOMUS_SUPERVISOR_INTERFACE alfa          # node where the Domus supervisor runs
  DOMUS_SUPERVISOR_PORT      7571          # tcp/ip port of the Domus supervisor
  
  # mainly relevant when operating a Domus cluster in a batch-managed physical cluster
  DOMUS_SUPERVISOR_LIFETIME             0           # (in seconds); 0 is infinite lifetime
  DOMUS_SUPERVISOR_LIFETIME_END_ACTION  terminate   # terminate, shutdown, auto-destroy
  
  # only relevant if --service_threadsafe was assumed during installation
  DOMUS_SUPERVISOR_TCP_FRONTEND_MULTITHREADED 0		# 1 == True ; 0 == False
  DOMUS_SUPERVISOR_TCP_SESSIONS_MULTITHREADED 0		# 1 == True ; 0 == False
  DOMUS_SUPERVISOR_UDP_FRONTEND_MULTITHREADED 0		# 1 == True ; 0 == False
  
  DOMUS_SERVICE_ROOT_DIR_PATH /state/partition1/user/.domus/var  # "service" working dir
                                                                 # (local to each node)
  
  # mainly relevant when operating a Domus cluster in a batch-managed physical cluster
  DOMUS_SERVICE_LIFETIME                0            # (in seconds); 0 is infinite lifetime
  DOMUS_SERVICE_LIFETIME_END_ACTION     terminate    # terminate, shutdown, auto-destroy
  
  ################################################################################
  

  There are two blocks of definitions, delimited by sequences of the character #, like in a domus#nodeinfo#conf file. Moreover, the 1st block of domus#mycluster#conf should match the 1st block of domus#nodeinfo#conf. With regard to the 2nd block, most definitions are self-explanatory. Some, however, deserve a further discussion:
  • DOMUS_SUPERVISOR_ROOT_DIR_PATH : working directory for the supervisor service of the Domus cluster; this directory may reside in the "user" account at the physical cluster frontend because supervisor services do not interact frequently with the file system and their persistent state is small; also, several supervisor services, related to different Domus clusters (operated by the same "user") may share the same working directory;

  • DOMUS_SERVICE_ROOT_DIR_PATH : working directory for regular services of the Domus cluster; it is not advisable for this directory to reside in the "user" account at the physical cluster frontend because such services may interact heavily with the file system for DHT storage purposes; if conducted via NFS, such interactions may be slow and probe to consistency problems; and, most importantly, having the Domus services storing their persistent state in the frontend would defeat the distributed storage goal of the DHTs; thus, the working dir of the regular services should be defined to be local to their hosting nodes; in a ROCKS cluster, /state/partition1 is typically an available local partition, at each node, adequate for such purpose (note, however, that this solution does not ensure durability if the /state/partition1 is configured to be cleaned up during reboot; in such case, a remote (but fast) SAN-based file system could be an alternative); finally, several regular services, related to different Domus clusters (operated by the same user) may also share the same working directory;

2) To complete the Pre-Configuration phase, it is also necessary to update several user environment files:
• the file ~user/.bash_profile must be updated with

  	
  LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~domus/.domus/lib:~domus/.domus/bin:.
  LD_RUN_PATH=$LD_RUN_PATH:~domus/.domus/lib:~domus/.domus/bin:.
  PYTHONPATH=$PYTHONPATH:~domus/.domus/lib:~domus/.domus/bin:.
  PATH=$PATH:$HOME/bin:~domus/.domus/bin:.
  
  export PATH LD_LIBRARY_PATH LD_RUN_PATH PYTHONPATH
  

• the file ~user/.ssh/environment must be updated with (assuming that ~domus is /home/domus):

  	
  PYTHONPATH=/home/domus/.domus/lib
  

4.3 - Management Tools:

Domus provides multiple ways to manage a Domus cluster: i) the domus_manager_cluster.py tool, ii) the domus_console.py tool, iii) the user-level library domus_libusr.py for development of generic client applications (in this regard, methods i) and ii) may be viewed as subcases of iii), where client applications are domus_manager_cluster.py and domus_console.py). In this section, the domus_manager_cluster.py tool is discussed.
Basically, the domus_manager_cluster.py tool provides client applications an alternate execution path to the Domus cluster related methods of the the user-level library domus_libusr.py. By following such path, the methods are not directly called from within the client application; instead, the client application will use OS primitives, like system, in order to execute the domus_manager_cluster.py tool with proper parameters; in turn, the tool will call directly the methods of domus_libusr.py. This allows virtually any client application (weather Python-based or not) to operate Domus clusters.
Usage:

	
[user@frontend]# domus_manager_cluster.py -h
Usage: /home/domus/.domus/bin/domus_manager_cluster.py - h |
{ [-c cluster_conf_file_path] [-N cluster_name] [-I supervisor_interface] 
  [-P supervisor_port] [--debug] -o operation [specific_options] }
operation:        { create | destroy | shutdown | restart | ping | kill | 
                    remove | getpid | log | ls }
specific_options: { -o create ... [-e] } | { -o kill ... [-k kill_signal] }
kill_signal:      { TERM (=> exit) | USR1 (=> shutdown) | USR2 (=>destroy) }

Parameters:
• -h : print usage information;
• -c cluster_conf_file_path : path to the Domus cluster configuration file; for a Domus cluster named cluster_name the file is expected to be ~user/.domus/etc/domus#cluster_name#conf, but an alternate file may be given; the default is ~/.domus/etc/domus#mycluster#conf;
• -N cluster_name : the name of the Domus cluster to be managed; the name must be unique, among the Domus clusters deployed by the same user; the default is the value of DOMUS_CLUSTER_NAME in the Domus cluster configuration file;
• -I supervisor_interface : DNS name or IP address of a network interface of a cluster node selected to run the supervisor service (usually the ROCKS frontend node); the default is the value of DOMUS_SUPERVISOR_INTERFACE in the Domus cluster configuration file;
• -P supervisor_port : TCP/IP port of the supervisor service; the default is the value of DOMUS_SUPERVISOR_PORT in the Domus cluster configuration file;
• --debug : activate debug code during the management process; fully effective only if the --debug was supplied during the installation process;
• -o operation : management operation to perform; the available operations are:
  • create [-e] : create the Domus cluster; at least, this involves starting up the supervisor service; at most, it also involves the launching of a regular service in each one of the cluster nodes that host the network interfaces specified in the interfaces file DOMUS_CLUSTER_INTERFACES_FILE_PATH (as defined in the Domus cluster configuration file); the optional parameter -e forces the Domus cluster to be created "empty" (without regular services), irregardless of the contents of the interfaces file;
  • destroy : destroy the Domus cluster; the supervisor and regular services eliminate all their state and data, and then terminate;
  • shutdown : shutdown (deactivate) the Domus cluster; the supervisor and regular services save to disk all their state and data, and then terminate; the saving takes place in the working directories specified by the DOMUS_SUPERVISOR_ROOT_DIR_PATH and DOMUS_SERVICE_ROOT_DIR_PATH parameters of the Domus cluster configuration file; the Domus cluster becomes inactive, until is brought again to the active state;
  • restart : restart (reactivate) the Domus cluster; the supervisor and regular services start up and recover their previous state from disk; the specific state of each DHT is preserved: DHTs that were offline (online) before the shutdown of the Domus cluster are kept offline (online) after the reactivation of the Domus cluster;
  • ping : check if the Domus cluster is active; this implies probing the supervisor service and all regular services;
  • kill : send a signal to all the Domus cluster services; the signal is specified by the parameter -k kill_signal, where kill_signal may be:
    • TERM : the services will terminate immediately (default if -k kill_signal is not used);
    • USR1 : the services will save any persistent state in disk and then terminate;
    • USR2 : the services will delete all persistent state and then terminate;

  • remove : force the removal of any persistent state of each Domus service;
  • getpid : get the Process ID (PID) of all Domus services; useful to kill "rogue" services;
  • log : launch one X-Window per each Domus service, allowing to see their logs in real-time;
  • ls : shows a recursive listing of the Domus services working directory, as given by the DOMUS_SERVICE_ROOT_DIR_PATH parameter of the Domus cluster configuration file;

Return/Exit Values:
A pair (val1,val2) is sent to the standard output immediately before ending execution; the pair may be captured (e.g., by a popen-like mechanism) by callers in order to know the success / failure of the operation; the generic behavior is to have val1 == 0 for success and val1 != 0 for failure; in either case val2 may be used to convey additional data or error codes, depending on the specific context (error codes are defined in the library ~domus/.domus/lib/domus_libsys.py).