[Halld-offline] Fwd: Data challenge requirements

[Heyes Graham]

I agree with both Curtis and Richard. In fact GLUEX is in very good shape regarding data challenges. What Chip suggested, and I agree with him, is that the lab should set some expectation, the discussion was then what that expectation should be. While using the OSG is a good resource to use for this sort of thing the expectation, I think from both the lab and the collaboration, is that at least the first pass reconstruction and analysis will happen here. In that case it is important to run the real code on the actual cluster (or as close to it as we can get) where it will finally run. In that way we shake out any issues involving the interaction of GLUEX code not only with the cluster software and hardware but also with other users of the system. The scope of the tests and the schedule are intended to give us (ENP and IT) time to adjust what hardware we buy and the cluster management software to solve any issues found, for example density of computing cores in one box, memory, local disk, network bandwidth etc.

As such then the tests that we proposed are the minimum that we would expect you to do and, being reasonable folks, I expect that they are the sort of thing that you have already considered.

	Regards to all,
		Graham


On Oct 31, 2013, at 10:37 am, Richard Jones <richard.t.jones at uconn.edu> wrote:

> GlueX offline group,
> 
> In stating our goals for the next data challenge, we need to satisfy whatever requirements are provided to us by the Jlab 12 GeV project.  However, let's not forget that simulations on the OSG are an integral part of our computing plan, and that to reach our scientific goals in the most optimum way possible, we need an integrated system for offline processing that includes both on-site and off-site computing.  Let's make sure that our goals for the next data challenge are directed primarily by our own internal criteria as a collaboration, and are not merely dictated to us by the expectations of the lab.  Those should serve as a kind of lower bound on what we expect to achieve.
> 
> -Richard J.
> 
> 
> On Thu, Oct 31, 2013 at 9:40 AM, Mark Ito <marki at jlab.org> wrote:
> Proposal from Graham and Chip, for your information and comment.
> 
> 
> -------- Original Message --------
> Subject:	Data challenge requirements
> Date:	Wed, 30 Oct 2013 13:05:59 -0400
> From:	Heyes Graham <heyes at jlab.org>
> To:	Ito Mark <marki at jlab.org>, Lawrence David <davidl at jlab.org>, Weygand Dennis <weygand at jlab.org>, Veronique Ziegler <ziegler at jlab.org>
> CC:	Rolf Ent <ent at jlab.org>, Rossi Patrizia <rossi at jlab.org>, Watson Chip <watson at jlab.org>
> 
> Dear Offline folks,
> Chip and I had a discussion this morning about what is an acceptable scope and schedule for data challenge tests of your analysis frameworks. The definitive tests should be done at JLab using the systems on which the final analysis will be done, i.e. the ENP analysis cluster (Farm). The suggested scope and schedule is based upon the dates that you have given for the first major bumps in computing requirements, 5000 cores for GLUEX in Q2 of 2015 and 2400 for CLAS12 in Q2 of 2016:
> 10% of final scale 24 months in advance.
> 25% of final scale 18 months in advance.
> 50% of final scale 12 months in advance.
> Each test should last for 2-4 days depending upon how many problems are encountered. The larger scale tests will require significant resources which Chip proposes that he will loan to us from the LQCD clusters so as not to significantly disturb other Farm users. He proposes moving the resources to us on a Monday then moving them back to LQCD on a Friday giving us up to four work days for testing.
> 
> I spoke with Mark this morning and he agreed that this is a reasonable schedule and that GLUEX have already met the 24 month test requirement. However the 25% scale test will be due in the next few months.
> 
> Chip and I both think that this is a reasonable schedule and not difficult to meet. We would like you to consider it as you prepare presentations for the review.
> 
> Regards to all,
> Graham
> 
> 
> 
> 
> 
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline
> 
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20131031/3d9c0265/attachment-0002.html>