[Halld-offline] Fwd: Data Challenge Meeting Minutes, March 21, 2014

Sat Mar 22 09:11:25 EDT 2014

2nd try at sending this. Not sure what happened the first time...

-------- Original Message --------
Subject: 	Data Challenge Meeting Minutes, March 21, 2014
Date: 	Sat, 22 Mar 2014 01:30:17 -0400
From: 	Mark Ito <marki at jlab.org>
To: 	GlueX Offline Software Email List <halld-offline at jlab.org>

Folks,

Find the minutes below and at

https://halldweb1.jlab.org/wiki/index.php/GlueX_Data_Challenge_Meeting,_March_21,_2014#Minutes

   -- Mark
_________________________________________________________

GlueX Data Challenge Meeting, March 21, 2014
Minutes

    Present:
      * CMU: Paul Mattione
      * IU: Kei Moriya
      * JLab: Mark Ito, Simon Taylor
      * MIT: Justin Stevens
      * NU: Sean Dobbs
      * UConn: Alex Barnes, Richard Jones

Announcements

    Yesterday we froze the code and tagged version dc-2.7. This tag brings
    in the change that Paul made to solve the reproducibility problem (see
    below). Richard also discussed how the change we had requested, to
    store the random number seed from bggen in its output for later use by
    hdgeant, was ill-advised and thus was not included in this tag. He
    wrote [36]email to the group yesterday on this. He noted that Eugene
    Chudakov has replaced the original Pythia random number generator with
    a more modern one. This means our simple practice of seeding the
    generator with the file number will not get us into trouble with
    tightly repeating loops.

    We went over the [37]conditions page to make sure we are all in
    agreement on details of the data challenge. We agreed that no changes
    that affect functionality were needed, but Mark should clean up some of
    the presentation on the page.

Review of Minutes from Last Time

    We went through the [38]minutes of the March 14 meeting.
      * Paul commented on [39]his fix to the reproducibility problem. At
        root was a dependence on the order of objects stored in an STL map
        which could change from one run on the data to the next. This
        occurred in matching tracks in the CDC with those in the FDC.
      * Mark reported that the return of nodes from the LQCD farm to the
        physics batch farm is complete. SciComp is ready for us to start
        submitting jobs.
      * We noted that all action items listed at the end of the minutes had
        in fact been acted on and resolved.

Activities at Production Sites

    We reviewed activities at the various sites.

CMU

    Paul has started a check-out and build of the latest tag. He reminded
    us that the various conditions have different processing rates, in
    terms of events per CPU-seconds. [40]His spreadsheet can help folks
    plan their processing. The cluster will contribute 384 cores.

FSU

    FSU had trouble connecting, understandable due to the evasive action we
    took to establish communications for this meeting. [Added in press:
    Aristeidis sent in a [41]report via email.]

JLab

      * Last night, a few jobs were run through with the dc-2.7 tag. They
        seem to have gone through.
      * Plan is to run 24-hour jobs, with the number of events per job
        depending on the EM background intensity.
      * JLab will have roughly 1200 cores for DC2.

MIT

    Justin led us through [42]his wiki page summarizing status at MIT. He
    is using OpenStack in two contexts: the MIT Reuse Cluster and
    FutureGrid (links on his wiki page). He expects to have 300 cores to
    contribute.

OSG

    Richard reported that the next step for OSG running is to set-up a
    CernVM system to provide access to our software stack at all available
    OSG sites. The system consists of a distributed set of read-only
    replica servers.

    MIT has access to an LHC tier-2 site in principle, we have not built up
    any credit on that site and so those resources are probably not
    available for this exercise.

    The number of cores that we get from the OSG is uncertain due to the
    nature of opportunistic running. Last time we peaked at 10,000 cores
    and nothing that we know now says that this time should be any
    different.

IU

    Kei reported that there are significant computing resources in
    Bloomington. These will likely be deployed for future large-scale
    computing efforts.

Monitoring Discussion

    We threw around some ideas on how to monitor quality of the results. We
    will be creating ROOT histograms for each job. At a minimum we could do
    spot checks on these histograms. Kei suggested processing the output of
    hd_dump for each output file to get an event count and prove
    analyzability. Mark proposed using hddm-xml and counting the number of
    event tags found using simple tools grep and wc. Richard volunteered to
    implement a program to do event counting at a lower level; it will be
    more efficient. We all thought that this would be a nice tool to have.
    [Added in press: this last change turned out to be a bit more complex
    than anticipated. We will proceed without it; it may be added to the
    mix later.]

File Distribution Discussion

    We did not come to a general plan for how to ship output data around
    for use in analysis. Paul pointed out that last time the OSG
    contribution dwarfed everything else and there the mechanism is the
    SRM. More thought will have to go into this.

    Retrieved from
"https://halldweb1.jlab.org/wiki/index.php/GlueX_Data_Challenge_Meeting,_March_21,_2014"

References

   36.
https://mailman.jlab.org/pipermail/halld-offline/2014-March/001612.html
   37.
https://halldweb1.jlab.org/data_challenge/02/conditions/data_challenge_2.html
   38.
https://halldweb1.jlab.org/wiki/index.php/GlueX_Data_Challenge_Meeting,_March_14,_2014#Minutes
   39.
https://mailman.jlab.org/pipermail/halld-offline/2014-March/001609.html
   40.
https://mailman.jlab.org/pipermail/halld-offline/2014-March/001577.html
   41.
https://mailman.jlab.org/pipermail/halld-offline/2014-March/001618.html
   42.
https://halldweb1.jlab.org/wiki/index.php/MIT/FutureGrid_Data_Challenge_2_Production

-- 
Mark M. Ito, Jefferson Lab, marki at jlab.org, (757)269-5295

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140322/bee857b6/attachment-0001.html 