3 - Monitorization Services:

The following instructions are assumed to be executed only (and only!) by the "domus" user.

3.1 - Pre-Configuration:

Edit the configuration files of ~domus/.domus/etc to suit the target physical cluster. If the default configuration was installed (see 2 - Installation), the content of that folder should be as follows:

	
[domus@frontend]# cd ~/.domus/etc
[domus@frontend]# ls -la
total 32
drwxr-xr-x 2 domus domus 4096 2008-02-07 16:57 .
drwxr-xr-x 4 domus domus 4096 2008-02-07 16:57 ..
-rw-r--r-- 1 domus domus 1711 2008-02-04 23:13 domus#mycluster#conf
-rw-r--r-- 1 domus domus   36 2008-02-04 19:43 domus#mycluster#interfaces
-rw-r--r-- 1 domus domus  181 2008-02-02 21:25 domus#nodeinfo#compute-1-0.local#db
-rw-r--r-- 1 domus domus  184 2008-02-02 21:25 domus#nodeinfo#compute-1-1.local#db
-rw-r--r-- 1 domus domus 1075 2008-02-04 23:13 domus#nodeinfo#conf
-rw-r--r-- 1 domus domus   36 2008-02-04 19:43 domus#nodeinfo#nodes

The previous files comprise two complementary groups: one that matches "domus#mycluster#*" and another that matches "domus#nodeinfo#*". The 1st group is only relevant when deploying Domus logical clusters (see 4 - Domus Clusters). The 2nd group is related to the Domus monitorization services of the physical cluster; such services are hereafter named "nodeinfo" services; by default, "nodeinfo" services are assumed to be under the sole control of the "domus" user, who is thus responsible for their configuration; the role of each file in the configuration is as follows:
• domus#nodeinfo#nodes : hostnames of the (physical) cluster nodes to be monitored, one by line; hostnames must be specified using the suffix .local (as defined in the file /etc/hosts, on the ROCKS cluster frontend node), resulting in a domus#nodeinfo#nodes similar to:

  	
  compute-1-0.local
  compute-1-1.local
  ...
  

• domus#nodeinfo#*#db : for each node node.local listed in domus#nodeinfo#nodes, there must be a domus#nodeinfo#node.local#db file; the file describes several capabilities of the node, measured by micro-benchmarks; for instance, the file domus#nodeinfo#compute-1-0.local#db has the following content:

  net_bandwith_iperf 192.168.0.253 610.4 # Mbps
  ram_throughput_berkeley_db 496789.25 # records/s
  disk_throughput_berkeley_db /state/partition1 2144.5 #records/s
  

  where

  • net_bandwith_iperf 192.168.0.253 610.4 # Mbps

    states that 610.4 Mbps is the full-duplex peak-bandwith, measured by iperf, between the 192.168.0.253 interface of the compute-1-0.local node and any other cluster node linked by the same network; generically, more network benchmark results may be added for the same cluster node if additional network interfaces are installed in the node;

  • ram_throughput_berkeley_db 496789.25 # records/s

    states that 496789.25 records/s is the maximum throughput measured by a BerkeleyDB benchmark (see also the related PDF), with RAM-based dictionaries

  • disk_throughput_berkeley_db /state/partition1 2144.5 #records/s

    states that 2144.5 records/s is the maximum throughput measured by the same BerkeleyDB benchmark, with DISK-based dictionaries deployed on the /state/partition1 disk partition; generically, more disk benchmark results may be added for the same cluster node if additional partitions (on the same or different hard-disks) are present;

  The above benchmark results were measured on a commodity cluster of homogeneous nodes, each one with a 3GHz Pentium 4 CPU, 1 GB of RAM and 1Gbps Ethernet onboard NIC. Presently, however, there's no need to repeat the benchmark procedure for a different target cluster: in the current Domus implementation, benchmark data is only used during the creation of DHTs, to attain a weighted distribution of the DHT addressing and storage space among the participating nodes; thus, if the target cluster is homogeneous, one can reuse the supplied domus#nodeinfo#*#db files, provided that the node names, IP addresses and partition identifiers are properly updated.

• domus#nodeinfo#conf : configuration parameters shared by all "nodeinfo" services; the contents of the sample file provided are shown bellow (most parameters are self-explanatory):

  	
  
  ################################################################################
  
  DOMUS_DEBUG          1                    # 1 == on; 0 == off; relevant only if
                                            # --debug was assumed during installation
  DOMUS_HOME_DIR_PATH  /home/domus/.domus   # where to find Domus after installation;
                                            # must be NFS-accessible to all users
  
  DOMUS_GANGLIA_HOST     alfa               # ROCKS cluster frontend
  DOMUS_GANGLIA_CHANNEL  235.185.190.208    # see /etc/gmond.conf at the frontend
  DOMUS_GANGLIA_PORT     8649               # see /etc/gmond.conf at the frontend
  DOMUS_GANGLIA_GMETRIC_PATH /opt/ganglia/bin/gmetric   # default: this same path
                                      
  DOMUS_THRESHOLD_CPU    0.8                # (%); resource utilization thresholds;
  DOMUS_THRESHOLD_RAM    0.8                # prevent the creation of DHTs on 
  DOMUS_THRESHOLD_DISK   0.8                # overloaded nodes; also used to weight 
  DOMUS_THRESHOLD_DISKIO 0.8                # the participation of nodes in DHTs 
  DOMUS_THRESHOLD_NET    0.5                # (in combination with benchmark results)
  
  ################################################################################
  
  DOMUS_NODEINFO_NODES_FILE_PATH /home/domus/.domus/etc/domus#nodeinfo#nodes
  DOMUS_NODEINFO_ROOT_DIR_PATH   /home/domus/.domus/var # "nodeinfo" working dir
                                                        # (shared by all instances)
  DOMUS_NODEINFO_LIFETIME        0    # (in seconds); 0 is infinite lifetime;
                                      # nodeinfos auto-shutdown after lifetime
  DOMUS_NODEINFO_REFRESH_PERIOD  30   # (in seconds); assumes 1 Gbps Ethernet; use
                                      # 30s for 32 bit nodes, 60s for 64 bit nodes
  
  ################################################################################
  

  Typically, only DOMUS_GANGLIA_* attributes need to be edited to conform to the local Ganglia configuration of the ROCKS cluster. The other attributes usually keep their default values.

3.2 - Management Tools:

After the configuration phase, one is ready to launch and manage the "nodeinfo" services. This is performed by means of the domus_manager_nodeinfos.py tool.
Usage:

	
[domus@frontend]# domus_manager_nodeinfos.py -h
Usage: /home/domus/.domus/bin/domus_manager_nodeinfos.py 
-h | { [-c nodeinfo_conf_file_path] [--debug] -o operation [specific_options] }
operation:        { start | kill | getpid | rmpid | log | ls }
specific_options: { -o kill ... [-k kill_signal]}
kill_signal:      { TERM (=> exit) | USR1 (=> shutdown) | USR2 (=> auto-destroy) }

Parameters:
• -h : print usage information;
• -c nodeinfo_conf_file_path : where to find the "nodeinfo" services configuration file; by default, ~domus/.domus/etc/domus#nodeinfo#conf is assumed;
• --debug : activate debug code during the management process; fully effective only if the --debug was supplied during the installation process;
• -o operation : management operation to perform; the available operations are:
  • start : start the distributed set of "nodeinfo" services
  • kill : send a signal to the distributed set of "nodeinfo" services; the signal is specified by the parameter -k kill_signal, where kill_signal may be:
    • TERM : the services terminate immediately (default if -k kill_signal is not used);
    • USR1 : the services save any persistent state in disk and then terminate;
    • USR2 : the services remove all persistent state from disk and then terminate;

  • getpid : get the Process ID (PID) of all "nodeinfo" services; useful to kill "rogue" services;
  • rmpid : force the removal of the PID file of each "nodeinfo" service;
  • ls : shows a recursive listing of the "nodeinfo" services working directory;
  • log : launch one X-Window per each "nodeinfo" service, allowing to see their logs in real-time;

Return/Exit Values:
A pair (val1,val2) is sent to the standard output immediately before ending execution; the pair may be captured (e.g., by a popen-like mechanism) by callers in order to know the success / failure of the operation; the generic behavior is to have val1 == 0 for success and val1 != 0 for failure; in either case val2 may be used to convey additional data or error codes, depending on the specific context (error codes are defined in the library ~domus/.domus/lib/domus_libsys.py).