Create Efficient VOV Scripts

If your flows are small, such as a few thousands jobs, you probably do not need to worry much about efficiency of your scripts. If you expect to operate on flows with hundreds of thousands of jobs, then this section can be useful.

While developing a VOV script, it is important to make sure that they do not needlessly make expensive calls that take a lot of vovserver time.

One useful method is to ask the system to show the service time for all expensive calls, which is activated by setting the environment variable VOV_SHOW_SERVICE_TIME to a positive integer that represents a time in milliseconds.

Note: The integer value is a threshold below which the times are not shown.

Here is an example with a call (i.e. "sanity") that tends to be expensive:

% setenv VOV_SHOW_SERVICE_TIME  1 
% vovproject sanity
vovsh(19194) Nov 12 12:36:35 SERVICE_TIME: Service took  2027ms for 137=SanityCheck 
vovsh(19194) Nov 12 12:36:35 SERVICE_TIME: Total service time for this client:    2.027s

In this example, vovserver took a bit more than 2 seconds to complete the reply to the request "SanityCheck" (internal code 137). This is normal for SanityCheck, and it is a reason why you do not want to run SanityCheck unless really necessary. Most VOV services you really need should be in the low millisecond range.

This method only shows the "slow" services. To see all services requested by a script, use the variable VOV_DEBUG_FLAGS as in this example:

% setenv VOV_DEBUG_FLAGS 16 ; ### This has to be 16 to show the RPC codes.

Experiments

#!/bin/csh -f
# Try this script and compare the load on the server
# Assume it is called "my_test_script"

set id = `vovsh -x 'FDL_INIT; VovUtils:init; set vovutils(feedback) quiet; puts [J vw hostname]'
vovselect status from jobs where id==$id  ;   ######## A common mistake 
vovselect status from $id 

# NC variants
nc info $id | grep Status | awk '{print $2}'
nc list | grep $id | awk '{print $2}'   ;;; ## Another horrible yet common mistake
nc getfield $id status                  ;;; ## BEST way!

####### NOTE:  This experiment run with 500,000 jobs in the flow.
% setenv VOV_SHOW_SERVICE_TIME  1 
% unsetenv VOV_DEBUG_FLAGS
% ./my_test_script
vovsh(9612) Nov 12 15:19:48 SERVICE_TIME: Total service time for this client:    0.000s
vovsh(9617) Nov 12 15:19:48 SERVICE_TIME: Service took     8ms for 307=CreateQuery select:fieldname from:jobs
vovsh(9617) Nov 12 15:19:48 SERVICE_TIME: Service took   707ms for 307=CreateQuery select:status from:jobs where:id==002233767
INVALID
vovsh(9617) Nov 12 15:19:48 SERVICE_TIME: Total service time for this client:    0.715s
INVALID
vovsh(9671) Nov 12 15:19:49 SERVICE_TIME: Total service time for this client:    0.000s
vovsh(9698) Nov 12 15:19:49 SERVICE_TIME: Service took     1ms for 208=GetInfoMap project
vovsh(9698) Nov 12 15:19:49 SERVICE_TIME: Total service time for this client:    0.001s
Idle
vovsh(9724) Nov 12 15:19:52 SERVICE_TIME: Service took  2607ms for 296=ListElementsEnh id:000001041 format:@ID@ @STATUSNC:9@ @PRIORITYPP:6@ @HOST:14@ @COMMAND:40@ range:0--1
vovsh(9724) Nov 12 15:19:54 SERVICE_TIME: Total service time for this client:    2.607s
Idle
INVALID
vovsh(9928) Nov 12 15:19:55 SERVICE_TIME: Total service time for this client:    0.000s