[Clas12_first_exp] CLAS12 First experiment meeting 12/19 @ 8:30 in F224/25

[Francois-Xavier Girod]

That is not in agreement with what Sandy told me. In any case we should use
swif to manage workflow.

On Thu, Dec 20, 2018, 11:07 William Phelps <wphelps at jlab.org wrote:

> I just want to clear up one point here: submitting jobs using jsub does
> not implement the tape access optimization, that is only available through
> swif.
>
> -Will
>
> On Dec 20, 2018, at 11:01 AM, Francois-Xavier Girod <fxgirod at jlab.org>
> wrote:
>
> Dear Vardan
>
> We did use this script to submit decoding. Sandy mentioned to me that tape
> access optimization from swif is now generally implemented on Auger,
> although swif has additional abilities to handle workflow.
>
> Best regards
> FX
>
> On Thu, Dec 20, 2018, 10:56 Vardan Gyurjyan <gurjyan at jlab.org wrote:
>
>> Hi Raffaella,
>> just to inform that I created a SWIF workflow, designed for CLAS12
>> decoding ( see at https://github.com/JeffersonLab/clas12-swif). Yet, I
>> am not sure if it was in use, and also this was far from being complete,
>> and was just a first attempt using SWIF for CLAS12.
>> Regards,
>> -vg
>> --------------------------------------------------
>> Vardan H. Gyurjyan, Ph.D.
>> Staff Scientist
>> Thomas Jefferson Accelerator Facility
>> Newport News, VA, 23606
>> E-mail: gurjyan at jlab.org <gurjyan at jlab.org>
>> 757-269-5879 (JLAB)
>>
>> On Dec 20, 2018, at 10:40 AM, Raffaella De Vita <
>> Raffaella.Devita at ge.infn.it> wrote:
>>
>> Dear FX,
>> the reason to develop a new SWIF workflow for decoding jobs is indeed to
>> make the whole process more efficient than what we had so far, by finding
>> and eliminating bottlenecks. Once more tests will be completed, we will be
>> able to have a new estimate of the time needed to decode a given number of
>> files. In the meantime, I assume there will be a final run list and number
>> of files to be decoded to estimate the time for the whole process.
>>
>> It would certainly be important to try to do as much as possible before
>> data taking will resume at the end of January. However, even after that,
>> raw data writing will engage only part of the available tape drives (CLAS12
>> is now using only one): if tape drives availability will be the bottleneck,
>> we will have to negotiate to have one drive reserved for the decoding
>> process. For what concerns writing decoded files to tape, the process will
>> be lighter than reading because hipo files are 4 times smaller than raw
>> evio and, also for that, we can negotiate having a dedicated volume to
>> ensure we will write the files sequentially on a tape.
>>
>> During the decoding process, it will be important to plan efficiently
>> other data processing activities:
>> - In February RG-B will be processing routinely data for
>> monitoring/calibration purposes but, making use of the case pinning options
>> we have in place, they will be able to minimise the needs for tape access.
>> - For RG-A, calibration activities will probably continue in
>> January-February: to minimize the interference with the decoding process,
>> it would be advisable to decode now all the files that are needed for
>> calibration, pinning the output on cache so that no further access to tape
>> will be needed.
>>
>> Best regards,
>> Raffaella
>>
>>
>> On 20 Dec 2018, at 02:08, Francois-Xavier Girod <fxgirod at jlab.org> wrote:
>>
>> Here is a very rough order of magnitude estimation of the time required
>> for decoding. We selected 456 runs for the calibration timeline. Some of
>> those are special runs like luminosity scan or empty target, even a couple
>> random trigger runs. Nevertheless, I believe we have more than 400
>> production runs. To process 400 runs in 40 days would demand decoding 10
>> runs a day. This is about or more than twice what we achieved in the past
>> when the accelerator is running. One can also compare a rate of 10 runs a
>> day with the average number of runs we write on tape while taking data,
>> keeping in mind that writing data has a much higher priority than reading
>> back.
>>
>> At this point we should anticipate that we will decode and cook in
>> parallel. I am not certain how that interferes with the current plans to
>> write on tape. We need to make sure that we do not write decoded files and
>> reconstructed files on the same tapes.
>>
>>
>>
>> _______________________________________________
>> Clas12_first_exp mailing list
>> Clas12_first_exp at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/clas12_first_exp
>>
>>
>> _______________________________________________
> Clas12_first_exp mailing list
> Clas12_first_exp at jlab.org
> https://mailman.jlab.org/mailman/listinfo/clas12_first_exp
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_first_exp/attachments/20181220/3085556e/attachment.html>