<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
I am writing to begin discussion on the online data file size.<br>
<br>
The CC is looking into data transfer from the counting house to the
silo and is wondering what our preferred online file size might be.
Mark took a first stab at this based on reconstruction rates on a
small number of cores. Other considerations include how often we
write files at 300 MB/s (40 GB is around 2 mins), and whether
reconstruction can regularly use much more than 4 cores.<br>
<br>
Please think about this, it will be discussed in upcoming online and
offline meetings.<br>
<br>
Thanks,<br>
<br>
<div class="moz-forward-container"><br>
<br>
-------- Original Message --------
<table class="moz-email-headers-table" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr>
<th nowrap="nowrap" valign="BASELINE" align="RIGHT">Subject:
</th>
<td>Re: Raw file size for offline reconstruction</td>
</tr>
<tr>
<th nowrap="nowrap" valign="BASELINE" align="RIGHT">Date: </th>
<td>Tue, 11 Jun 2013 11:39:08 -0400</td>
</tr>
<tr>
<th nowrap="nowrap" valign="BASELINE" align="RIGHT">From: </th>
<td>Mark M. Ito <a class="moz-txt-link-rfc2396E" href="mailto:marki@jlab.org"><marki@jlab.org></a></td>
</tr>
<tr>
<th nowrap="nowrap" valign="BASELINE" align="RIGHT">To: </th>
<td>Hovanes Egiyan <a class="moz-txt-link-rfc2396E" href="mailto:hovanes@jlab.org"><hovanes@jlab.org></a></td>
</tr>
<tr>
<th nowrap="nowrap" valign="BASELINE" align="RIGHT">CC: </th>
<td>David Lawrence <a class="moz-txt-link-rfc2396E" href="mailto:davidl@jlab.org"><davidl@jlab.org></a>, Simon Taylor
<a class="moz-txt-link-rfc2396E" href="mailto:staylor@jlab.org"><staylor@jlab.org></a>, Elliott Wolin
<a class="moz-txt-link-rfc2396E" href="mailto:wolin@jlab.org"><wolin@jlab.org></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<pre>Hovanes,
Using numbers from last year's review:
size per event: 15 kB
time to analyze an event, one thread: 130 ms
If we assume the following for starting out with reconstruction
number of threads: 4
time for job to finish: 24 hours
and the job is working on a single complete file, the file size would be
40 GB. To me that is an acceptable number.
In the future, we may be running more than four threads, say 32. Then
you either get a 3 hour job on a 40 GB file or a 24 hour job on a 320 GB
file. The latter sounds like too big a size from a
file-handling-convenience point of view.
-- Mark
On 06/10/2013 04:07 PM, Hovanes Egiyan wrote:
> Hi,
>
> For the purposes of designing DAQ data transport to the tape silo,
> it would be useful to know if there are any limits (hard or soft) on
> the data files sizes from the point of view of the offline event
> reconstruction.
> The rumor that is going around during the meetings is that the files
> will be about
> 100GB. Is this a reasonable number, how long will such a job take? Can
> the
> files be larger? Since we started (and may be finished) the study of
> the raw events
> in the DAQ we should be able to project this quantity.
>
> Hovanes.
>
--
Mark M. Ito, Jefferson Lab, <a class="moz-txt-link-abbreviated" href="mailto:marki@jlab.org">marki@jlab.org</a>, (757)269-5295
</pre>
<br>
<pre class="moz-signature" cols="72">--
Sincerely,
Elliott
================================================================================
Those raised in a morally relative or neutral environment will hold
no truths to be self-evident.
Elliott Wolin
Staff Physicist, Jefferson Lab
12000 Jefferson Ave
Suite 8 MS 12A1
Newport News, VA 23606
757-269-7365
================================================================================
</pre>
<br>
</div>
<br>
</body>
</html>