[Clas12_first_exp] CLAS12 First experiment meeting 12/19 @ 8:30 in F224/25

[Francois-Xavier Girod]

Dear Vardan

We did use this script to submit decoding. Sandy mentioned to me that tape
access optimization from swif is now generally implemented on Auger,
although swif has additional abilities to handle workflow.

Best regards
FX

On Thu, Dec 20, 2018, 10:56 Vardan Gyurjyan <gurjyan at jlab.org wrote:

> Hi Raffaella,
> just to inform that I created a SWIF workflow, designed for CLAS12
> decoding ( see at https://github.com/JeffersonLab/clas12-swif). Yet, I am
> not sure if it was in use, and also this was far from being complete, and
> was just a first attempt using SWIF for CLAS12.
> Regards,
> -vg
> --------------------------------------------------
> Vardan H. Gyurjyan, Ph.D.
> Staff Scientist
> Thomas Jefferson Accelerator Facility
> Newport News, VA, 23606
> E-mail: gurjyan at jlab.org <gurjyan at jlab.org>
> 757-269-5879 (JLAB)
>
> On Dec 20, 2018, at 10:40 AM, Raffaella De Vita <
> Raffaella.Devita at ge.infn.it> wrote:
>
> Dear FX,
> the reason to develop a new SWIF workflow for decoding jobs is indeed to
> make the whole process more efficient than what we had so far, by finding
> and eliminating bottlenecks. Once more tests will be completed, we will be
> able to have a new estimate of the time needed to decode a given number of
> files. In the meantime, I assume there will be a final run list and number
> of files to be decoded to estimate the time for the whole process.
>
> It would certainly be important to try to do as much as possible before
> data taking will resume at the end of January. However, even after that,
> raw data writing will engage only part of the available tape drives (CLAS12
> is now using only one): if tape drives availability will be the bottleneck,
> we will have to negotiate to have one drive reserved for the decoding
> process. For what concerns writing decoded files to tape, the process will
> be lighter than reading because hipo files are 4 times smaller than raw
> evio and, also for that, we can negotiate having a dedicated volume to
> ensure we will write the files sequentially on a tape.
>
> During the decoding process, it will be important to plan efficiently
> other data processing activities:
> - In February RG-B will be processing routinely data for
> monitoring/calibration purposes but, making use of the case pinning options
> we have in place, they will be able to minimise the needs for tape access.
> - For RG-A, calibration activities will probably continue in
> January-February: to minimize the interference with the decoding process,
> it would be advisable to decode now all the files that are needed for
> calibration, pinning the output on cache so that no further access to tape
> will be needed.
>
> Best regards,
> Raffaella
>
>
> On 20 Dec 2018, at 02:08, Francois-Xavier Girod <fxgirod at jlab.org> wrote:
>
> Here is a very rough order of magnitude estimation of the time required
> for decoding. We selected 456 runs for the calibration timeline. Some of
> those are special runs like luminosity scan or empty target, even a couple
> random trigger runs. Nevertheless, I believe we have more than 400
> production runs. To process 400 runs in 40 days would demand decoding 10
> runs a day. This is about or more than twice what we achieved in the past
> when the accelerator is running. One can also compare a rate of 10 runs a
> day with the average number of runs we write on tape while taking data,
> keeping in mind that writing data has a much higher priority than reading
> back.
>
> At this point we should anticipate that we will decode and cook in
> parallel. I am not certain how that interferes with the current plans to
> write on tape. We need to make sure that we do not write decoded files and
> reconstructed files on the same tapes.
>
>
>
> _______________________________________________
> Clas12_first_exp mailing list
> Clas12_first_exp at jlab.org
> https://mailman.jlab.org/mailman/listinfo/clas12_first_exp
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_first_exp/attachments/20181220/ff054b1d/attachment.html>