[g13] tarring skim files

[Ken Livingston]

Hi Paul & co,
Sorry I didn't make it to tuesday's call. I'm going to work on the
cooking scripts tomorrow and get some test jobs submitted.

I agree with Paul on what's easiest:

We don't want to put lots of tiny files on the SILO, so we should tar
small files together in some way. The easiest way, as Paul says, is to
make a tar file for the all the non-BOS output of each job (ie output
relating to a single input RAW BOS file). This is what was done last
time, despite some whingeing, and it's what I intend to do again.

The alternative is to cache all the output somewhere (eg work disks) and
reorganize it into more sensible tar file before jputting - eg. tarring
files of the same sort together. This is non-trivial and would need some
careful script writing and cron jobs to handle. We don't have time to do
this and debug it, and I don't think it's worth the effort.

Ken

 
On Thu, 2010-07-22 at 15:39 -0400, Paul Mattione wrote:
> From the standpoint of cooking, I think the easiest way of writing  
> our skims, dsts, monitoring histograms, etc to tape is to combine them  
> all into a single tar file for each cooked file.
> 
> However, this means that in order to extract your skim you have to  
> cache much more (all) of the data from the tapes than if you just had  
> a single tar file for a given skim for a given run.  But if you  
> combine things by run, if anything needs to be re-cooked (e.g. a  
> cooking job fails), you have to re-make all of the tar files  
> associated with that run.
> 
> How do you guys think we should do it?
> 
> - Paul
> 
> 
> _______________________________________________
> g13 mailing list
> g13 at jlab.org
> https://mailman.jlab.org/mailman/listinfo/g13