[Sbs_software] ROOT files from upcoming SBS data production, file size, and tape storage

[Andrew Puckett]

Dear Helpdesk,

We are about to start our second mass data production from the SBS GMN dataset. This will require roughly 100 kcore-hours to reduce about 2 PB of data down to about 10 TB or less. Each of these ~100k single-thread reconstruction jobs will produce a single ROOT output file ranging in size from about 25 MB at the low end to over 200 MB at the high end. Our intention is to write each of these files to both volatile and to tape under a similar directory structure, and to automate the writing to tape from directly within the farm job itself using the “-output” option of swif2 add-job.

I am writing to give you a heads-up about this plan and ask you to weigh in about the tape aspect because I know these file sizes are on the small side from a tape efficiency standpoint, in terms of overhead of staging in/out, etc. However, we can’t NOT write them to tape since they cost (in total) about 100 kcore-hours to reproduce.

Although the number of these files will be relatively large (about 100k), I normally expect them to be available on /volatile and/or pinned in /cache for further analysis. Moreover, the total amount of data will be relatively small (<10 TB), such that the entire reduced dataset can easily be moved offsite and/or to people’s local machines/workstations/computing setups. As such, I don’t expect a huge amount of “churn” from repeated caching of these files from tape by analysis jobs on the farm itself over the remaining lifecycle of the GMN analysis.

The alternative would be to write the files to /volatile and then attempt to combine them into larger files and/or compressed tarballs that would be more efficient from a tape standpoint. But that is less convenient than writing directly to tape and would require more manual bookkeeping/scripting on our end.

Originally I wanted to chain 10 CODA files of 20 GB each into a single replay job of about 10 core-hours, which would have led to bigger root files that would be more efficient for tape storage, but my experience with that approach was that it reduced our throughput on the farm and increased the failure rate of our jobs due to the logistical overhead of staging 200 GB of input data and requiring it to be available for ~10 hours, as opposed to 20 GB of input data for ~1 hour.

Anyway, if you don’t mind us writing the ~100k ROOT files of 25-200 MB each to tape, then we will write them directly from the farm jobs, which is the most convenient option.

Best,
Andrew


Andrew Puckett
Associate Professor, Physics Department
University of Connecticut
196 Auditorium Road, Unit 3046
Storrs, CT 06269-3046
Office phone: (860) 486-7137
https://puckett.physics.uconn.edu
puckett at jlab.org<mailto:andrew.puckett at uconn.edu>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/sbs_software/attachments/20240124/e5b10619/attachment.html>