Health Monitoring and vovnotifyd

The vovnotifyd daemon performs two functions:

• Sends mail notification based on job events, according to the MAILTO property of the jobs; (Altair Accelerator only)
• It performs periodic system health checks, and sends email when it discovers issues. (Monitor and Accelerator)

To manually configure vovnotifyd, you need to create the directory vovnotifyd inside of the server working directory (.swd) and copy the configuration file template, $VOVDIR/etc/config/vovnotifyd/config.tcl into the newly-created vovnotifyd directory. The configuration template will need to be modified to match the settings of your mail server environment.

% cd `vovserverdir -p .`
% mkdir vovnotifyd

File: $VOVDIR/etc/config/vovnotifyd/config.tcl

# Notification configuration file.
# Should be placed in the vovnotifyd directory of the .swd.
# All settings are required unless specified otherwise.
# Unused optional settings should be commented out.

# Create an e-mail address map, stackable, optional
addUserToEmailAddressMap  rtdamgr john@mydomain.com

### Altair Monitor-specific settings
# See notification configuration documentation in Altair Monitor Admin Guide
# ConfigureTag     TAG OPTION VALUE
# ConfigureFeature FEATURE OPTION VALUE

### Examples:
# ConfigureTag MGC -poc { john mary }
# ConfigureFeature EDA/MATLAB -longcheckout 2d -userlongcheckout john 1w -mincap 5 -triggerperc 90
# ConfigureFeature SIMULINK -poc bob -mincap 10 -triggeruse 12

To start the daemon, either use nc cmd vovdaemonmgr start vovnotifyd or start it manually from within the vovnotifyd directory:

% vovproject enable vnc
% cd `vovserverdir -p vovnotifyd`
% vovnotifyd

In the directory vnc.swd/autostart create a script called start_vovnotifyd.tcl with the following content:

% cd `vovserverdir -p .`
% mkdir autostart
% cp $VOVDIR/etc/autostart/start_vovnotifyd.tcl  autostart/start_vovnotifyd.tcl

You can use the config.tcl file to set the email addresses to be used for each user. You can choose one of the following methods:

• Call addUserToEmailAddressMap USERNAME EMAIL
• Override the entire procedure getEmailAddress

# Fragment of vovnotifyd/config.tcl file.

# Method 1.
addUserToEmailAddressMap john  John.Smith@my.company.com

# Method 2. Assume we can get an address from LDAP
#           The LDAP subsystem needs to be configured.
proc getEmailAddress { user } {
    set email [VovLDAP::getEmail $user]
    if { $email != "" } {
       return $email
    } else {
       return $user
    }
}

The health check procedures are loaded using a search path. First, they are loaded from the file given above, then from $VOVDIR/local/vovhealthlib.tcl, and then from vovhealthlib.tcl in the vovnotifyd working directory. This permits you to redefine health check procedures on a site-wide or project-specific basis.

For local procedures to be shown by the browser UI, you need to add a line into your vovhealthlib.tcl file like:

set HEALTHLIB_PRODUCT_MAP(doTestHealthYourProcedure)        "nc"

If your MTA is so configured, you may need to use an alternate method to send mail. One method is to redefine the procedure daemon's sendMail procedure. There is also another way, see below.

# Overriding sendMail in the vovnotifyd/config.tcl file.
proc sendMail { recList subj msg } {
    catch {exec yourMailScript.csh $recList $subj $msg }
}

By default, all checks occur every 10 minutes.

AllJobsFailedOnHost

This check will check for hosts on which all jobs have failed during a given time window (default 3600)

CheckAlerts

This check will send a summary of all the current alerts via notification email

CheckDownTaskers

This check will check for taskers enabled in taskers.tcl but not running

CheckJobsReqstRam

This will alert you for jobs that have used more RAM than requested.

CheckTaskerset

This will check the status of running taskers.

CheckVendorLicenseExpiration

This will alert you when a vendor license is going to expire in a few days.

Daemons

This will check the daemons.

FalseLicenceUsage

This will alert you when jobs use licenses that have not been declared

JobStuck

This will find stuck jobs, which are running but burning no CPU. The default check frequency on this is 2 hours.

JobsWaitingForTooLong

Just as the name suggests, this reports on jobs that have been waiting for too long.

LongJobs

This will check for jobs that run too long.

LongJobsNoCpu

Like the previous check, but will check for jobs that are not using any CPU.

RamSentry

This will check jobs under Ram Sentry protection.

ServerSize

This will check the growth of vovserver.

TooManyFailures

As the name suggests, this will check for a user submitting too many failed jobs.

TooManyOutOfQueueJobs

This will check for too many jobs being run outside of Altair Accelerator.