Environment Management: Limits

In addition to environment variables, hard and soft limits can affect tool behavior. Hard and soft limits are set in the shell and are imposed by the operating system. Both UNIX and Windows provide a mechanism for processes to communicate information to their subprocesses via environment variables.

VOV uses special environment variables to communicate limit information. The environment variables are named VOV_LIMIT_<name>. name is the name of the limit, such as VOV_LIMIT_cputime.

A complete list of VOV_LIMIT_<name> environment variables include:
  • VOV_LIMIT_cputime
  • VOV_LIMIT_datasize
  • VOV_LIMIT_descriptors
  • VOV_LIMIT_filesize
  • VOV_LIMIT_maxproc
  • VOV_LIMIT_memorylocked
  • VOV_LIMIT_memoryuse
  • VOV_LIMIT_openfiles
  • VOV_LIMIT_stacksize
  • VOV_LIMIT_vmemoryuse

The variables are interpreted in the wrapper program vw2 which uses the C-language getrlimit()/setrlimit() system calls to set the limits for the child process when the job runs.

The value of the variable must be an sequence of digits followed by an optional unit indicator letter (case insensitive), or the string value 'unlimited'. If there is no unit indicator, the units are kilobytes (1024 bytes). The recognized unit indicators are:
  • k or K, kilobytes (same as no unit indicator)
  • M, Megabytes (multiply by 1024)
  • G, Gigabytes (multiply by 1024*1024)
  • T, Terabytes (multiply by 1024*1024*1024)
In Accelerator, the limit can be set at submission time. In the following example, a limit of 20 seconds of CPU is set for a job. 
% nc run -e 'BASE+D(VOV_LIMIT_cputime=20)' .... shortjob.csh
% nc run -e 'BASE+D(VOV_LIMIT_vmemoryuse=5G)' .... bigjob.csh
% nc run -e 'BASE+D(VOV_LIMIT_vmemoryuse=5G)' .... bigjob.csh
In the following example, the command lines provide the roughly equivalent limit of 5GB for a job. 
% nc run -e 'BASE+D(VOV_LIMIT_vmemoryuse=5000M)'    .... bigjob.csh
% nc run -e 'BASE+D(VOV_LIMIT_vmemoryuse=5G)'       .... bigjob.csh
% nc run -e 'BASE+D(VOV_LIMIT_vmemoryuse=5000000k)' .... bigjob.csh

The multipliers used with the memory specifications for limits are as follows: 'k' (KiB, 1024=210, kibibytes) 'M' (MiB, 220, mebibytes) 'G' (GiB, 230, gibibytes) and 'T' (TiB, 240, tebibytes). The multipliers are case-insensitive.

If an integer has no unit specification, kilobytes are the units used. For example, 1024K is the same as 1024. In addition, using unlimited as a value is acceptable.

The following two examples show how to find the current limits.

For csh/tcsh: 
% limit
cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       unlimited
coredumpsize    0 kbytes
vmemoryuse       unlimited
descriptors     1024 
memorylocked    unlimited
maxproc         2048 
openfiles       1024
For sh/bash: 
bash-2.05$ ulimit -a
core file size (blocks)     unlimited
data seg size (kbytes)      unlimited
file size (blocks)          unlimited
max locked memory (kbytes)  unlimited
max memory size (kbytes)    unlimited
open files                  1024
pipe size (512 bytes)       8
stack size (kbytes)         unlimited
cpu time (seconds)          unlimited
max user processes          5119
virtual memory (kbytes)     unlimited

## Use option -H to get the hard-limits
bash-2.05$ ulimit -a -H         
...

In most cases, everything in the shell startup file can be set as unlimited. This setup gives the tools the greatest possibility of a successful run. It is extremely rare for this method to not work.

When an environment snapshot is used when submitting jobs to Accelerator, the limits in the submission environment are captured automatically.

If using a named environment, the following Tcl code in the ENV.start.tcl script can be used to set the variables for limits and umask. This is the same as what is done in the environment snapshot.
if { [info command vtk_umask_get] != {} } {
    setenv VOV_UMASK [vtk_umask_get]
    catch {
        vtk_limits_get limit
        foreach n [array names limit] {
            setenv VOV_LIMIT_$n $limit($n)
        }
    }
}

Tcl-language API

VOV provides two API procedures to get and set the limits, vtk_limits_get and vtk_limits_set. Both procedures take a single array parameter, which contains the limits, keyed by name.

Because different platforms have different limits available, the VTK procedures support only a common subset of limits.

The following names are supported:
VTK Limit Procedure Name
Description
stacksize
Size of process stack segment, bytes
datasize
Size of process data segment, bytes
cputime
Maximum process CPU time, seconds
filesize
Maximum file size, bytes
coredumpsize
Maximum core dump file size, bytes
In the following example, Tcl code is used to eliminate core files by setting the coredump size limit to zero: 
catch {
    vtk_limits_get L ;      # get existing limits into array L
    set L(coredumpsize) 0 ; # set coredump limit
    foreach n [array names L] {
        setenv VOV_LIMIT_$n $limit($n) ; # propagate limits to envVars
    }
}