Pre-Command and Post-Command Job Conditions

When a job is being submitted, a pre-condition and/or a post-condition can be specified.

  • pre-condition: a script that is executed before the job is executed.
  • post-condition: a script that is executed after the job has completed. The post-condition is typically used to perform cleanup, such as deleting temporary files in /usr/tmp.

Example scripts are available in the following directories: $VOVDIR/etc/pre and $VOVDIR/etc/post.

Pre-condition

A pre-condition is executed before the job is run. It is invoked with a single argument: the ID of the job. A pre-condition is executed with the same credentials as the job (userid, os-groupid) and is in the same directory of the job.
  • If the precondition script fails by exiting with a status different from 0 (zero), the job will not be run and the exit status of the job will be the exit status of the pre-condition script.
  • If the exit status of the pre-condition script is within the range 201-215, the automatic rescheduling condition will occur and the job will be rescheduled on a different host or on a different tasker (see more on auto-rescheduling).

Post-condition

The post-condition script is invoked with two arguments: the ID of the job and the exit status of the job. The post-condition is executed with the same credentials as the job (userid, os-groupid) and in the same directory of the job.
  • When the post-condition script is invoked, the job is still running.
  • The post-condition is executed after the job, even if the job fails, but it is not executed if the pre-condition fails.
  • The exit status of the post condition overrides the exit status of the job. It needs to explicitly return the exit status of the job when that is the requested behavior (see the example scripts).

Submit Jobs with Conditions

Use the options -pre and -post with nc run to specify the pre- and post- conditions.
% nc run -pre $VOVDIR/etc/pre/pre_check.sh sleep 10
% nc run -post $VOVDIR/etc/post/post_cleanup.sh sleep 10

Log Files

The standard output from the pre- and post-commands is saved in log files. The location of the log files is determined by the value of the environment variable NC_LOGDIR. If NC_LOGDIR is not set, the files are stored in the directory ./vnc_logs, relative to the current launch directory.

In the following example, NC_LOGDIR is not set, and the run directory is ~/testrundir:
[goetz@goetz1 ~/testrundir]$ pwd
/home/goetz/testrundir
[goetz@goetz1 ~/testrundir]$ ls
vnc_logs
[goetz@goetz1 ~/testrundir]$ ls -a vnc_logs/
. .. 20210726 .precmd.000083865.log .precmd.000083885.log snapshots

The log files are created with zero size if the pre- and post-commands redirect all the output of the files. At the end of the job, if these files are zero length, they are automatically deleted to reduce disk space overhead.

The log files are named according to the following rules: 
.precmd.$jobID.log
.postcmd.$jobID.log
The pre- and post-command log files can optionally be located in the same directory as the job logfile. For example:
nc run -pre "myprecommand > @JOBLOGDIR@/@JOBID@_pre.out" -l path/to/an/existing/directory/mycommand.out -- mycommand
nc run -post "mypostcommand > @JOBLOGDIR@/@JOBID@_post.out" -l path/to/an/existing/directory/mycommand.out -- mycommand
This would result in the respective pre- and post-command logfiles being written to the directory path/to/an/existing/directory.
Note: When using the nc run command after forgetting jobs that have pre- and/or post-commands, it does not automatically remove the pre- and post-command .log files. If these files are not zero length, they must be removed manually.