[Halld-online] Fwd: Re: Raw file size for offline reconstruction

[Elliott Wolin]

Hi,

I am writing to begin discussion on the online data file size.

The CC is looking into data transfer from the counting house to the silo 
and is wondering what our preferred online file size might be. Mark took 
a first stab at this based on reconstruction rates on a small number of 
cores.  Other considerations include how often we write files at 300 
MB/s (40 GB is around 2 mins), and whether reconstruction can regularly 
use much more than 4 cores.

Please think about this, it will be discussed in upcoming online and 
offline meetings.

Thanks,



-------- Original Message --------
Subject: 	Re: Raw file size for offline reconstruction
Date: 	Tue, 11 Jun 2013 11:39:08 -0400
From: 	Mark M. Ito <marki at jlab.org>
To: 	Hovanes Egiyan <hovanes at jlab.org>
CC: 	David Lawrence <davidl at jlab.org>, Simon Taylor <staylor at jlab.org>, 
Elliott Wolin <wolin at jlab.org>



Hovanes,

Using numbers from last year's review:

size per event: 15 kB
time to analyze an event, one thread: 130 ms

If we assume the following for starting out with reconstruction

number of threads: 4
time for job to finish: 24 hours

and the job is working on a single complete file, the file size would be
40 GB. To me that is an acceptable number.

In the future, we may be running more than four threads, say 32. Then
you either get a 3 hour job on a 40 GB file or a 24 hour job on a 320 GB
file. The latter sounds like too big a size from a
file-handling-convenience point of view.

   -- Mark

On 06/10/2013 04:07 PM, Hovanes Egiyan wrote:
> Hi,
>
> For the purposes of designing DAQ data transport to the tape silo,
> it would be useful to know if there are any limits (hard or soft) on
> the data files sizes from the point of view of the offline event
> reconstruction.
> The rumor that is going around during the meetings is that the files
> will be about
> 100GB. Is this a reasonable number, how long will such a job take? Can
> the
> files be larger? Since we started (and may be finished) the study of
> the raw events
> in the DAQ we should be able to project this quantity.
>
> Hovanes.
>


-- 
Mark M. Ito, Jefferson Lab, marki at jlab.org, (757)269-5295


-- 

				Sincerely,
					Elliott
  

================================================================================


  Those raised in a morally relative or neutral environment will hold
		    no truths to be self-evident.
				

Elliott Wolin
Staff Physicist, Jefferson Lab
12000 Jefferson Ave
Suite 8 MS 12A1
Newport News, VA 23606
757-269-7365

================================================================================



-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-online/attachments/20130612/f88c7c51/attachment.html