[Clas12_first_exp] CLAS12 First experiment meeting 12/19 @ 8:30 in F224/25

[William Phelps]

I just want to clear up one point here: submitting jobs using jsub does not implement the tape access optimization, that is only available through swif.

-Will

> On Dec 20, 2018, at 11:01 AM, Francois-Xavier Girod <fxgirod at jlab.org> wrote:
> 
> Dear Vardan
> 
> We did use this script to submit decoding. Sandy mentioned to me that tape access optimization from swif is now generally implemented on Auger, although swif has additional abilities to handle workflow. 
> 
> Best regards 
> FX
> 
> On Thu, Dec 20, 2018, 10:56 Vardan Gyurjyan <gurjyan at jlab.org <mailto:gurjyan at jlab.org> wrote:
> Hi Raffaella,
> just to inform that I created a SWIF workflow, designed for CLAS12 decoding ( see at https://github.com/JeffersonLab/clas12-swif <https://github.com/JeffersonLab/clas12-swif>). Yet, I am not sure if it was in use, and also this was far from being complete, and was just a first attempt using SWIF for CLAS12.
> Regards,
> -vg
> --------------------------------------------------
> Vardan H. Gyurjyan, Ph.D.
> Staff Scientist
> Thomas Jefferson Accelerator Facility
> Newport News, VA, 23606
> E-mail: gurjyan at jlab.org <mailto:gurjyan at jlab.org>
> 757-269-5879 (JLAB)
> 
>> On Dec 20, 2018, at 10:40 AM, Raffaella De Vita <Raffaella.Devita at ge.infn.it <mailto:Raffaella.Devita at ge.infn.it>> wrote:
>> 
>> Dear FX,
>> the reason to develop a new SWIF workflow for decoding jobs is indeed to make the whole process more efficient than what we had so far, by finding and eliminating bottlenecks. Once more tests will be completed, we will be able to have a new estimate of the time needed to decode a given number of files. In the meantime, I assume there will be a final run list and number of files to be decoded to estimate the time for the whole process.
>> 
>> It would certainly be important to try to do as much as possible before data taking will resume at the end of January. However, even after that, raw data writing will engage only part of the available tape drives (CLAS12 is now using only one): if tape drives availability will be the bottleneck, we will have to negotiate to have one drive reserved for the decoding process. For what concerns writing decoded files to tape, the process will be lighter than reading because hipo files are 4 times smaller than raw evio and, also for that, we can negotiate having a dedicated volume to ensure we will write the files sequentially on a tape.
>> 
>> During the decoding process, it will be important to plan efficiently other data processing activities:
>> - In February RG-B will be processing routinely data for monitoring/calibration purposes but, making use of the case pinning options we have in place, they will be able to minimise the needs for tape access.
>> - For RG-A, calibration activities will probably continue in January-February: to minimize the interference with the decoding process, it would be advisable to decode now all the files that are needed for calibration, pinning the output on cache so that no further access to tape will be needed.
>> 
>> Best regards,
>> Raffaella
>> 
>> 
>>> On 20 Dec 2018, at 02:08, Francois-Xavier Girod <fxgirod at jlab.org <mailto:fxgirod at jlab.org>> wrote:
>>> 
>>> Here is a very rough order of magnitude estimation of the time required for decoding. We selected 456 runs for the calibration timeline. Some of those are special runs like luminosity scan or empty target, even a couple random trigger runs. Nevertheless, I believe we have more than 400 production runs. To process 400 runs in 40 days would demand decoding 10 runs a day. This is about or more than twice what we achieved in the past when the accelerator is running. One can also compare a rate of 10 runs a day with the average number of runs we write on tape while taking data, keeping in mind that writing data has a much higher priority than reading back.
>>> 
>>> At this point we should anticipate that we will decode and cook in parallel. I am not certain how that interferes with the current plans to write on tape. We need to make sure that we do not write decoded files and reconstructed files on the same tapes.
>> 
>> 
>> _______________________________________________
>> Clas12_first_exp mailing list
>> Clas12_first_exp at jlab.org <mailto:Clas12_first_exp at jlab.org>
>> https://mailman.jlab.org/mailman/listinfo/clas12_first_exp <https://mailman.jlab.org/mailman/listinfo/clas12_first_exp>
> 
> _______________________________________________
> Clas12_first_exp mailing list
> Clas12_first_exp at jlab.org
> https://mailman.jlab.org/mailman/listinfo/clas12_first_exp

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_first_exp/attachments/20181220/766acdce/attachment-0001.html>