[Halld-offline] grid project management tools

[Richard Jones]

Hello Mark,

For compute resource discovery and allocation, we used glideinWMS in
cooperation with the UCSD and osg GOC job factories.  On the client side,
the tools are integrated with condor client tools using condor-g.

For storage allocation and redirection services, we used the dcache
implementation of SRM.

For a global file catalog, took some initial steps to set up the LFC file
catalog.  However I did not use it during the data challenge because no
other sites were contributing grid storage to the exercise, and its purpose
is to federate grid storage resources.  Right now it is on my back burner.
 As soon as I get a second site with significant grid storage resources to
share, I will roll out the LFC service.

To automatically scan job logs and collect statistics after completion for
the percent success, total hours used, total hours consumed by failed jobs,
and tons of other similar summary information, I used the standard tool
condor_userlog in the condor suite.

I don't think a lot of real work exists to be done in this area.  It just
takes a chat with a few friends on the LHC experiments to learn what the
best practices are and how to select the best tools for the major tasks,
and stick them together into a workflow with python scripts.  It might seem
tempting to get into the business of reinventing the wheel, but I don't
think we have the manpower resources for that.  That seems like the way
that Hall B is inclined to go, and that's fine for them.  In GlueX, we need
to focus on the science and grab the best tools out there to get the job
done.  That's my two cents.

-Richard Jones


On Sun, Nov 17, 2013 at 2:18 PM, Mark Ito <marki at jlab.org> wrote:

> Richard,
>
> My memory is that for data challenge 1 (last year), you did not use any
> formal grid project management/tracking/logging tools of the kind we
> discussed before we did the challenge. Nor any kind of generic
> grid-toolkit-supplied meta-data catalog. Certainly for what was done at
> JLab and CMU no such package was used.
>
> I am planning to say words to this effect to the software review
> committee, and I just wanted to confirm with you before opening my mouth. I
> would identify this as an area of future work.
>
>   -- Mark
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20131117/0cfeaeca/attachment-0002.html>