Handling Failures and Rerunning Tasks in Sun Grid Engine Array Jobs

We use Sun Grid Engine here at WTCHG for managing our compute resources. Many of the analyses I’m doing are best run as array jobs, which generally works very well, but sometimes one or more tasks will fail for one reason or another, and I’ve been casting around for best practice when it comes to (a) verifying which tasks succeeded and which failed, and (b) re-running failed tasks.

I found a nice post by Shiran Pasternak on resubmitting failed SGE array tasks, however Shiran doesn’t say how he determined which tasks had failed, and the set of tasks to rerun is specified manually. I have thousands of tasks in each array job, and so I really need an automated way of determining the success/failure of each task and rerunning those that failed.

I came up with the following pattern.

Advertisement

Mounting truecrypt volume from script, without showing password in process list

I found a short article about how to mount a truecrypt volume from a shell script without showing password in process list, see also the comments which provide some alternatives and clarification.

I use…

echo $password | truecrypt -t -k "" --protect-hidden=no /path/to/encrypted.tc /media/truecrypt1

The -t option makes TrueCrypt work in text mode so you can pipe the password to a terminal prompt, and the other options (-k "" --protect-hidden=no) prevent TrueCrypt giving additional prompts which would otherwise confuse the piping of the password variable.