<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Richard -<div><br></div><div> this is a useful summary of the grid services. We have a couple of problems that we need to have solutions</div><div>to and it may well be that the grid takes care of one of them. The JLab-centric one is getting production jobs from</div><div>tape, and then smoothly through production (to REST format) on the Jefferson Lab farms. There is a group at</div><div>Jefferson Lab working on this "Workflow System". Unfortunately, this does not match well with the LHC tools,</div><div>but at least for GlueX, is not an crazy task.</div><div><br></div><div> The other one is managing the REST files after they have been produced, and I think that we have always</div><div>felt that srm has to be a part of that. I also expected that there was cataloging systems for that, but was not sure</div><div>how much we were using it. We will hopefully get more experience with this when we do our next large data</div><div>challenge (probably early next year).</div><div><br></div><div> I have ordered a 100TB raid array that I planned to try and contribute to GlueX srm storage. I am still working</div><div>on getting some tome to finalize my cluster upgrades, but my hope is that by January we have several</div><div>hundred CMU cores in the OSG as well as the large disk storage. That would presumably be a good trigger to </div><div>get the LFC services going so that we can get some experience with it.</div><div><br></div><div>Curtis</div><div><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>-------</div><div>Prof. Curtis A. Meyer<span class="Apple-tab-span" style="white-space: pre; "> </span> <span class="Apple-converted-space"> </span>Department of Physics</div><div>Phone: (412) 268-2745<span class="Apple-tab-span" style="white-space: pre; "> </span> <span class="Apple-converted-space"> </span>Carnegie Mellon University</div><div>Fax: (412) 681-0648<span class="Apple-tab-span" style="white-space: pre; "> </span> Pittsburgh, PA 15213</div><div><a href="mailto:curtis.meyer@cmu.edu">curtis.meyer@cmu.edu</a> <a href="http://www.curtismeyer.com/">http://www.curtismeyer.com/</a></div><div><br></div><div><br></div></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br><div><div>On Nov 17, 2013, at 5:04 PM, Richard Jones <<a href="mailto:richard.t.jones@uconn.edu">richard.t.jones@uconn.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Hello Mark,<div><br></div><div>For compute resource discovery and allocation, we used glideinWMS in cooperation with the UCSD and osg GOC job factories. On the client side, the tools are integrated with condor client tools using condor-g.</div>
<div><br></div><div>For storage allocation and redirection services, we used the dcache implementation of SRM.</div><div><br></div><div>For a global file catalog, took some initial steps to set up the LFC file catalog. However I did not use it during the data challenge because no other sites were contributing grid storage to the exercise, and its purpose is to federate grid storage resources. Right now it is on my back burner. As soon as I get a second site with significant grid storage resources to share, I will roll out the LFC service.</div>
<div><br></div><div>To automatically scan job logs and collect statistics after completion for the percent success, total hours used, total hours consumed by failed jobs, and tons of other similar summary information, I used the standard tool condor_userlog in the condor suite.</div>
<div><br></div><div>I don't think a lot of real work exists to be done in this area. It just takes a chat with a few friends on the LHC experiments to learn what the best practices are and how to select the best tools for the major tasks, and stick them together into a workflow with python scripts. It might seem tempting to get into the business of reinventing the wheel, but I don't think we have the manpower resources for that. That seems like the way that Hall B is inclined to go, and that's fine for them. In GlueX, we need to focus on the science and grab the best tools out there to get the job done. That's my two cents.</div>
<div><br></div><div>-Richard Jones</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Nov 17, 2013 at 2:18 PM, Mark Ito <<a href="mailto:marki@jlab.org">marki@jlab.org</a>> wrote:<br>
<blockquote class="gmail_quote">Richard,<br>
<br>
My memory is that for data challenge 1 (last year), you did not use any formal grid project management/tracking/logging tools of the kind we discussed before we did the challenge. Nor any kind of generic grid-toolkit-supplied meta-data catalog. Certainly for what was done at JLab and CMU no such package was used.<br>
<br>
I am planning to say words to this effect to the software review committee, and I just wanted to confirm with you before opening my mouth. I would identify this as an area of future work.<font color="#888888"><br>
<br>
-- Mark<br>
<br>
</font></blockquote></div><br></div>
_______________________________________________<br>Halld-offline mailing list<br><a href="mailto:Halld-offline@jlab.org">Halld-offline@jlab.org</a><br>https://mailman.jlab.org/mailman/listinfo/halld-offline</blockquote></div><br></div></body></html>