Interface to LSF with vovlsfd

vovlsfd makes use of the vovlsfd wrapper script that is included as part of the installation. The configuration file for vovlsfd is located in $VOVDIR/local/vovlsf.config.csh. If the file does not exist, create it and populate it with information to configure a shell to use your site's particular LSF setup. Once setup, the following commands should work in with a vovproject enabled terminal.

% vovlsf bsub sleep 10
Job <37655> is submitted to default queue <normal>.
% vovlsf bjobs
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
37655   jeremym PEND  normal     buffalo                 sleep 10   Mar  6 11:56
% vovlsf bqueues
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP
normal           30   Open:Active      -    -    -    -     1     0     1     0
% vovlsf bkill 37655
Job <37655> is being terminated

You can start from the example shown here, by copying it from $VOVDIR/etc/config/vovlsfd/config.tcl.

# Example Configuration file for vovlsfd:
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Remember to add the resource "LSFqueue:$myqueue" to your LSF jobs,
# or vovlsfd will assume the jobs are for direct/already connected taskers.

# How often should the vovlsfd daemon cycle?
# The value is a VOV time spec and the default is two seconds.
set VOVLSFD(refresh)                  2s

# how frequently should we ask bjobs for status?
# The value is a VOV time spec and the default value is one minute.
set VOVLSFD(bjobs,checkfreq)          1m

# How often should we check for sick taskers?
# The value is a VOV time spec and the default is one minute
set VOVLSFD(sick,checkfreq)           1m

# Remove sick taskers that are older than?
# Value is a VOV time spec, and the default value is five minutes.
set VOVLSFD(sick,older)               5m

# Should we dequeue any extra taskers?
# Setting this to "1" will cause a dequeue
# of all not yet running vovtasker submissions for a job bucket.
# This only happens after three consecutive refresh cycles
# have gone by with no work scheduled for that bucket.
set VOVLSFD(dequeueExtraTaskersEnable) 1

# How long should we wait to dequeue any extra taskers?
# The number of refresh cycles
set VOVLSFD(dequeueExtraTaskersDelay)  3

# What is the maximum number taskers we should start?
# Should be set to a high value to enable lots of parallelism.
set VOVLSFD(tasker,max)                99

# What is the maximum number of queued taskers per bucket that we should allow?
set VOVLSFD(tasker,maxQueuedPerBucket) 99

# How many tasker submissions should be done for each
# job bucket, during each refresh cycle?
set VOVLSFD(tasker,maxSubsPerBucket)   1

# What is the minimum number of taskers that will be grouped into an array
# for each resource bucket, during each refresh cycle?
# Setting this to "0" will disable LSF array functionality.
set VOVLSFD(tasker,lsfArrayMin)   0

# What is the maximum number of taskers that will be grouped into an array
# for each resource bucket, during each refresh cycle? The absolute max number
# of taskers supercedes this value.
# Setting this to "0" will disable LSF array functionality.
set VOVLSFD(tasker,lsfArrayMax)   0

# What is the longest a vovtasker should run before self-exiting?
# Ex: if you set it to 8 hours, and queue 4 3-hour jobs:
# the first tasker will run for nine hours (3 x 3-hr > 8-hr) and then exit
# the fourth job will only start when a second tasker has been requeued
# and started by the batch execution system.
# This controls the amount of reuse of a tasker while it processes jobs.
# To avoid the penalties of:
# noticing a tasker is needed
# + submitting to the batch system
# + the batch system to allocate a machine
# You should set this to a high value like a week.
# THe value is a VOV time spec
# This is a default value. It can be overriden on a per job basis by putting
# a resource on the job that looks similar to the following.
# MAXlife:1w
set VOVLSFD(tasker,maxlife)       1w

# How long should a tasker wait idle for a job to arrive?
# The shorter time, the faster the slot is released to the batch system.
# The longer time, the more chances the tasker will be reused.
# The default value is two minutes (usually takes a minute to allocate a
# slot through a batch system).  Value is a VOV time spec
# This is a default value. It can be overriden on a per job basis by putting
# a resource on the job that looks similar to the following.
# MAXidle:2m 
set VOVLSFD(tasker,maxidle)       2m

# Are there any extra resources you wish to pass along to the taskers?
# These resources will be passed directly along to the vovtasker. They
# are not processed in any way by vovlsfd. For example setting
# this to "MAXlife:1w" will not work as you might expect.
set VOVLSFD(tasker,res)           ""

# Is there a default string you wish to pass as "-P" on the bsub
# command line. Per job settings are also supported by putting a 
# resource on the job that looks similar to the following.
# LSFptag:blah
set VOVLSFD(tasker,ptag)          ""

# What is the vovtasker update interval for resource calculation?
# Value is a VOV time spec, and the default value is 15 seconds.
set VOVLSFD(tasker,update)        15s

# How many MB of RAM should we request by default?
# (Default: 256MB)
set VOVLSFD(tasker,ram)           256

# How many CPUs on the same machine should we request by default?
# (Default: 1 core)
set VOVLSFD(tasker,cpus)          1

# What is the default LSF jobname to be used when launching vovtaskers?
set VOVLSFD(tasker,jobname)       "ft_$env(VOV_PROJECT_NAME)"

# Do we want to enable debug messages in the vovtasker log files?
# 0=no; 1=yes; default=0
set VOVLSFD(tasker,debug)         0

# What level of verbosity should the vovtasker use when writing to its the log file?
# Valid values are 0-4; default=1
set VOVLSFD(tasker,verbose)       1

    # How long should the vovtasker try to establish the initial connection to the vovserver?
# Values are in seconds, default is 120 seconds.
set VOVLSFD(tasker,timeout)       120

    # How many hosts should the vovtasker span?
# The value can be overriden by an individual job using the subordinate resource functionality.
# Without that override, this config variable defines a default value for span[hosts=*]
# Setting the value to zero omits the default span[hosts=*] and allows the underlying batch
# scheduler to use its own internally set value.
# Setting to -1 disables the span setting defined for the queue in the batch scheduler configuration.
# Setting to 1 tells the batch system to provide all processors from a single host.
# https://www.ibm.com/support/knowledgecenter/SSETD4_9.1.3/lsf_admin/span_string.html
# Valid values are -1, 0, and 1; default=1
set VOVLSFD(tasker,spanHosts)     1

# Which precmd script should we include in the bsub command line by default?
# The precmd script must be an executable script located in the VOVLSFD(launchers.dirname) directory.
# The script should be self contained so that we can reference it with a single word. (i.e. no arguments, and not full paths)
set VOVLSFD(tasker,precmd)        ""

# How much buffer should we consider when adjusting tasker,max based on available client connections?
# This number will be subtracted from the available maxNormalClient connections
# Twice this number will be subtracted from the available file descriptors
# Set this number based on how many non-vovtasker client connections are anticipated for this session.
set VOVLSFD(client,derate)       50

# What is the name of the launchers directory? (Default: \"./launchers\")
set VOVLSFD(launchers,dirname)   "./launchers"

# How often should we check the launchers directory for a cleanup?
# The value is a VOV time spec and the default is 10 minutes
set VOVLSFD(launchers,checkfreq) 10m

# Remove launchers that are older than?
# Value is a VOV time spec, and the default value is one hour.
set VOVLSFD(launchers,older)     1h

# Generate a CSV report file (pendingrunning.csv) upon each pass
# of the daemon cycle.
set VOVLSFD(savePendingRunningReport) 0

# queue-specific options:
# set VOVLSFD(bsub,options,$qName) 

## To control the name of the executable that is dispatched to LSF.
## The -sr flag in vov_lsf_agent is what 
# set VOVLSFD(exeleaf) "$env(VOVDIR)/scripts/vovboot vov_lsf_agent"
# set VOVLSFD(exeleafargs) "-sr"

This file does not initially exist, so it will have to be manually created. Use the above example as a template. Here is a sequence of commands to set up vovlsfd for a given project.

% vovproject enable <project>
% cd `vovserverdir -p .`
% mkdir vovlsfd
% cp $VOVDIR/etc/config/vovlsfd/config.tcl vovlsfd/config.tcl
% vi vovlsfd/config.tcl ; # Edit config file to suit your installation. 

Use vovdaemonmgr to start vovlsfd manually.

% vovproject enable <project>
% vovdaemonmgr start vovlsfd

In the directory vnc.swd/autostart, create a script called vovlsfd.csh with the following content:

#!/bin/csh -f

vovdaemonmgr start vovlsfd

Don't forget to make the script executable.

% chmod 755 vovlsfd.csh

When first starting vovlsfd, it is helpful to run it in foreground, possibly with the -v and/or -d options to verify operation as expected.

% vovproject enable <project>
% cd `vovserverdir -p vovlsfd`
% vovlsfd  ; # Launch vovlsfd

Examples of job submission with Accelerator:

% nc run -r LSFqueue:normal -- spice abc.spi
% nc run -r LSFqueue:night LSFresource:dc -- dc_shell -f script.tcl

Examples of assigning LSF resources to jobs with FlowTracer:

N "spice"
R "LSFqueue:normal"
J vw spice abc.spi

N "dc_shell"
R "LSFqueue:night LSFresource:dc"
J vw dc_shell -f script.tcl

It is also possible to pass additional LSF bsub options for a particular job to vovlsfd in two ways.

1. Applying the "BSUB_OPTIONS" text property to a job
2. Adding a specially formatted string (beginning with "--") to the end of the FDL 'R' procedure argument

It's up to you to determine which approach to use. It's a matter of preference.

# 1) Append subordinate string to resource string using the following format :
# R {<required resources> -- "<LSF bsub options>"}
N "md5"
R {LSFqueue:medium -- "-R 'select[type==64BIT]' -app 'highpri'"}
J vw md5.pl

# 2) or, apply a property to the job
N "md5"
R "LSFqueue:medium"
set jobId [J vw md5.pl]
vtk_prop_set -text $jobId BSUB_OPTIONS {-R 'select[type==64BIT]' -app 'highpri'}

A police proc should be named police_ concatenated with the array index name of the config variable in question. For example, if you want to write a proc that polices the VOVLSFD(tasker,maxlife) setting used to set the maximum life of a vovtasker submitted to LSF by vovlsfd, then the proc name is police_tasker,maxlife. The proc will be called with a single input, the current value of the variable. The proc should return the desired value.

# Example police.tcl proc that limits maxlife to five minutes or less.
proc police_tasker,maxlife { input } {
    if { [VovParseTimeSpec $input] > 300 } {
        return [vtk_time_pp 300]
    } else {
        return $input
    }
}