[Jlab-scicomp-briefs] Production release of a new workflow software (SWIF)

[Jie Chen]

Dear Scientific Computing Users:

We are pleased to announce the production release of a new workflow 
software system (SWIF) developed locally by the scientific computing 
group. SWIF, the "scientific workflow indefatigable factotum", 
originated from the recognition of the necessity of providing a 
work-centric data analysis environment in addition to the job-centric 
data analysis platform at Jlab in order to improve the efficiency of 
large data analysis in the 12 GeV era. Its key requirements and 
specifications were the results of meetings and discussions among active 
users, software coordinators from each experimental hall, and 
developers. Its implementation was carried out by developers from the 
scientific computing group, and the whole system underwent a half year 
long beta testing period with feedbacks and bug reports from several 
helpful users. The main features of SWIF are:

1. Job grouping and phased release. User jobs, either Auger jobs or SWIF 
jobs, can be grouped into a named workflow which is released to the 
batch system in multiple user defined phases.

2. Tape savvy job scheduling. The order of user jobs in a workflow can 
be rearranged according to the input file locations on tapes to improve 
the throughput of a workflow.

3. Ease of job modification, resubmission, recall and cancellation.

4. Automatic classification of job errors inside a workflow and 
automatic resubmission of failed jobs if desired.

5. Detailed job metrics during and after a workflow.


In addition, the tape savvy scheduling algorithm has been integrated 
into the Auger system. If a user submits many jobs with input files on 
many tapes, the jobs can be reordered to achieve better throughput.

The SWIF is available as a single command at /site/bin/swif which has 
many options. These options can be viewed either by using /site/bin/swif 
–help or by consulting the SWIF document at 
https://scicomp.jlab.org/docs/swif-cli .

For more information about SWIF, please check out 
https://scicomp.jlab.org/docs/swif . Send any comment, feature request, 
and bug report to scicomp at jlab.org <mailto:scicomp at jlab.org> or through 
the Jlab ccpr system. Thank you very much.


Sincerely,

Your Scientific Computing Group.

-- 
################################################################
#     Jie Chen
#     Scientific Computing Group
#     Jefferson Lab
#     12000 Jefferson Ave
#     Newport News, VA 23606
#     757-269-5046
#     chen at jlab.org
################################################################					
					

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/jlab-scicomp-briefs/attachments/20151015/3be92969/attachment-0002.html>