[Clas12_first_exp] CLAS12 First experiment meeting 12/19 @ 8:30 in F224/25

[Raffaella De Vita]

Dear FX,
the reason to develop a new SWIF workflow for decoding jobs is indeed to make the whole process more efficient than what we had so far, by finding and eliminating bottlenecks. Once more tests will be completed, we will be able to have a new estimate of the time needed to decode a given number of files. In the meantime, I assume there will be a final run list and number of files to be decoded to estimate the time for the whole process.

It would certainly be important to try to do as much as possible before data taking will resume at the end of January. However, even after that, raw data writing will engage only part of the available tape drives (CLAS12 is now using only one): if tape drives availability will be the bottleneck, we will have to negotiate to have one drive reserved for the decoding process. For what concerns writing decoded files to tape, the process will be lighter than reading because hipo files are 4 times smaller than raw evio and, also for that, we can negotiate having a dedicated volume to ensure we will write the files sequentially on a tape.

During the decoding process, it will be important to plan efficiently other data processing activities:
- In February RG-B will be processing routinely data for monitoring/calibration purposes but, making use of the case pinning options we have in place, they will be able to minimise the needs for tape access.
- For RG-A, calibration activities will probably continue in January-February: to minimize the interference with the decoding process, it would be advisable to decode now all the files that are needed for calibration, pinning the output on cache so that no further access to tape will be needed.

Best regards,
	Raffaella


> On 20 Dec 2018, at 02:08, Francois-Xavier Girod <fxgirod at jlab.org> wrote:
> 
> Here is a very rough order of magnitude estimation of the time required for decoding. We selected 456 runs for the calibration timeline. Some of those are special runs like luminosity scan or empty target, even a couple random trigger runs. Nevertheless, I believe we have more than 400 production runs. To process 400 runs in 40 days would demand decoding 10 runs a day. This is about or more than twice what we achieved in the past when the accelerator is running. One can also compare a rate of 10 runs a day with the average number of runs we write on tape while taking data, keeping in mind that writing data has a much higher priority than reading back.
> 
> At this point we should anticipate that we will decode and cook in parallel. I am not certain how that interferes with the current plans to write on tape. We need to make sure that we do not write decoded files and reconstructed files on the same tapes.