<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I just want to clear up one point here: submitting jobs using jsub does not implement the tape access optimization, that is only available through swif.<div class=""><br class=""></div><div class="">-Will<br class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Dec 20, 2018, at 11:01 AM, Francois-Xavier Girod <<a href="mailto:fxgirod@jlab.org" class="">fxgirod@jlab.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div dir="auto" class="">Dear Vardan<div dir="auto" class=""><br class=""></div><div dir="auto" class="">We did use this script to submit decoding. Sandy mentioned to me that tape access optimization from swif is now generally implemented on Auger, although swif has additional abilities to handle workflow. </div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Best regards </div><div dir="auto" class="">FX</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Thu, Dec 20, 2018, 10:56 Vardan Gyurjyan <<a href="mailto:gurjyan@jlab.org" class="">gurjyan@jlab.org</a> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space" class="">
<div class="">Hi Raffaella,</div>
just to inform that I created a SWIF workflow, designed for CLAS12 decoding ( see at
<a href="https://github.com/JeffersonLab/clas12-swif" target="_blank" rel="noreferrer" class="">https://github.com/JeffersonLab/clas12-swif</a>). Yet, I am not sure if it was in use, and also this was far from being complete, and was just a first attempt using SWIF for CLAS12.
<div class="">Regards,</div>
<div class="">-vg<br class="">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
--------------------------------------------------<br class="">
Vardan H. Gyurjyan, Ph.D.<br class="">
Staff Scientist<br class="">
Thomas Jefferson Accelerator Facility<br class="">
Newport News, VA, 23606<br class="">
<a href="mailto:gurjyan@jlab.org" target="_blank" rel="noreferrer" class="">E-mail: gurjyan@jlab.org</a><br class="">
757-269-5879 (JLAB)</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Dec 20, 2018, at 10:40 AM, Raffaella De Vita <<a href="mailto:Raffaella.Devita@ge.infn.it" target="_blank" rel="noreferrer" class="">Raffaella.Devita@ge.infn.it</a>> wrote:</div>
<br class="m_-7840262720627680420Apple-interchange-newline">
<div class="">
<div class="">Dear FX,<br class="">
the reason to develop a new SWIF workflow for decoding jobs is indeed to make the whole process more efficient than what we had so far, by finding and eliminating bottlenecks. Once more tests will be completed, we will be able to have a new estimate of the
time needed to decode a given number of files. In the meantime, I assume there will be a final run list and number of files to be decoded to estimate the time for the whole process.<br class="">
<br class="">
It would certainly be important to try to do as much as possible before data taking will resume at the end of January. However, even after that, raw data writing will engage only part of the available tape drives (CLAS12 is now using only one): if tape drives
availability will be the bottleneck, we will have to negotiate to have one drive reserved for the decoding process. For what concerns writing decoded files to tape, the process will be lighter than reading because hipo files are 4 times smaller than raw evio
and, also for that, we can negotiate having a dedicated volume to ensure we will write the files sequentially on a tape.<br class="">
<br class="">
During the decoding process, it will be important to plan efficiently other data processing activities:<br class="">
- In February RG-B will be processing routinely data for monitoring/calibration purposes but, making use of the case pinning options we have in place, they will be able to minimise the needs for tape access.<br class="">
- For RG-A, calibration activities will probably continue in January-February: to minimize the interference with the decoding process, it would be advisable to decode now all the files that are needed for calibration, pinning the output on cache so that no
further access to tape will be needed.<br class="">
<br class="">
Best regards,<br class="">
<span class="m_-7840262720627680420Apple-tab-span" style="white-space:pre-wrap"></span>Raffaella<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">On 20 Dec 2018, at 02:08, Francois-Xavier Girod <<a href="mailto:fxgirod@jlab.org" target="_blank" rel="noreferrer" class="">fxgirod@jlab.org</a>> wrote:<br class="">
<br class="">
Here is a very rough order of magnitude estimation of the time required for decoding. We selected 456 runs for the calibration timeline. Some of those are special runs like luminosity scan or empty target, even a couple random trigger runs. Nevertheless, I
believe we have more than 400 production runs. To process 400 runs in 40 days would demand decoding 10 runs a day. This is about or more than twice what we achieved in the past when the accelerator is running. One can also compare a rate of 10 runs a day with
the average number of runs we write on tape while taking data, keeping in mind that writing data has a much higher priority than reading back.<br class="">
<br class="">
At this point we should anticipate that we will decode and cook in parallel. I am not certain how that interferes with the current plans to write on tape. We need to make sure that we do not write decoded files and reconstructed files on the same tapes.<br class="">
</blockquote>
<br class="">
<br class="">
_______________________________________________<br class="">
Clas12_first_exp mailing list<br class="">
<a href="mailto:Clas12_first_exp@jlab.org" target="_blank" rel="noreferrer" class="">Clas12_first_exp@jlab.org</a><br class="">
<a href="https://mailman.jlab.org/mailman/listinfo/clas12_first_exp" target="_blank" rel="noreferrer" class="">https://mailman.jlab.org/mailman/listinfo/clas12_first_exp</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote></div>
_______________________________________________<br class="">Clas12_first_exp mailing list<br class=""><a href="mailto:Clas12_first_exp@jlab.org" class="">Clas12_first_exp@jlab.org</a><br class="">https://mailman.jlab.org/mailman/listinfo/clas12_first_exp<br class=""></div></blockquote></div><br class=""></div></div></body></html>