<div dir="ltr">Justin,<div><br></div><div>Good catch. I am diagnosing now, expect to issue a fix by end of tomorrow at latest.</div><div><br></div><div>-Richard J.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Mon, Mar 3, 2014 at 8:28 AM, David Lawrence <<a href="mailto:davidl@jlab.org">davidl@jlab.org</a>> wrote:<br><blockquote class="gmail_quote">
<br>
Phew! That’s good. You had me worried for a minute! Now that I re-read your e-mail, I see what you were saying the first time around. Thanks for the clarification.<br>
<font color="#888888"><br>
<br>
-David<br>
</font><div class="HOEnZb"><div class="h5"><br>
On Mar 3, 2014, at 8:24 AM, Justin Stevens <<a href="mailto:jrsteven@mit.edu">jrsteven@mit.edu</a>> wrote:<br>
<br>
> Hi David,<br>
><br>
> Sorry, what I was trying to say is the failure rate I'm seeing is 0%. All of the 700 jobs processed all the events and they all had the correct size REST file which should contain all the events. So by turning the compression off I don't see this problem with small REST files anymore. Sorry for the poor wording early in the morning... it was before caffeine.<br>
><br>
> -Justin<br>
><br>
> On Mar 3, 2014, at 8:16 AM, David Lawrence wrote:<br>
><br>
>><br>
>> Hi Justin,<br>
>><br>
>> Can you clarify a bit. It sounds like you’re saying all 700 jobs failed to process all events and all 700 had small REST files. In other words a 100% failure rate. Is this correct?<br>
>><br>
>> -David<br>
>><br>
>> On Mar 3, 2014, at 6:52 AM, Justin Stevens <<a href="mailto:jrsteven@mit.edu">jrsteven@mit.edu</a>> wrote:<br>
>><br>
>>> Hi Data Challengers,<br>
>>><br>
>>> Over the weekend I was running test jobs with branches/sim-recon-dc-2 and branches/hdds-dc-2 (checked out as of the time of Friday's meeting), and in hd_root (ie. creating the rest file) I switched to using the option -PHDDM:USE_COMPRESSION=0. With 700 jobs finished so far, I haven't seen any jobs where the monitoring histograms showed the full 25K events were processed, but the REST file was small and didn't contain all 25K events I'm generating. The fraction of "small" REST files was something like 2-3% when I was running with compression turned on earlier.<br>
>>><br>
>>> These jobs were all using the 10^7 EM background with a gate of +/-800 ns, and I had 1 job which crashed in hdgeant, similar to what Paul has already reported:<br>
>>><br>
>>> !!!!! ZFATAL called from MZPUSH<br>
>>><br>
>>> !!!!! ZFATAL reached from MZPUSH for Case= 3<br>
>>><br>
>>> FYI,<br>
>>> Justin<br>
>>><br>
>>> On Mar 2, 2014, at 10:02 PM, Mark Ito wrote:<br>
>>><br>
>>>> Colleagues,<br>
>>>><br>
>>>> Please fine the minutes below and at<br>
>>>><br>
>>>> <a href="https://halldweb1.jlab.org/wiki/index.php/GlueX_Data_Challenge_Meeting,_February_28,_2014#Minutes">https://halldweb1.jlab.org/wiki/index.php/GlueX_Data_Challenge_Meeting,_February_28,_2014#Minutes</a><br>
>>>><br>
>>>> -- Mark<br>
>>>> _________________________________________________________<br>
>>>><br>
>>>> GlueX Data Challenge Meeting, February 28, 2014<br>
>>>> Minutes<br>
>>>><br>
>>>> Present:<br>
>>>> * CMU: Paul Mattione, Curtis Meyer<br>
>>>> * FSU: Volker Crede, Priyashree Roy, Aristeidis Tsaris,<br>
>>>> * IU: Kei Moriya<br>
>>>> * JLab: Mark Dalton, Mark Ito (chair), Chris Larrieu, Simon Taylor<br>
>>>> * NU: Sean Dobbs<br>
>>>> * UConn: Richard Jones<br>
>>>><br>
>>>> Announcements<br>
>>>><br>
>>>> * Mark announced an [21]update of the branch. Changes include:<br>
>>>> 1. I fix from Simon for single-ended TOF counters.<br>
>>>> 2. Improvements from Paul for cutting off processing for<br>
>>>> multi-lap curling tracks.<br>
>>>> 3. A change from David Lawrence to [22]allow compression to be<br>
>>>> turned off in producing REST format data.<br>
>>>> o David noticed that all three of the programs hdgeant,<br>
>>>> mcsmear, and DANA produced HDDM-like output, but only<br>
>>>> DANA has compression turned on (REST data in this case).<br>
>>>> This feature will allow us to test if this has anything<br>
>>>> to do with short REST files. On a side note, David<br>
>>>> reported that the short-REST-file was not reproducible.<br>
>>>> Mark produced some example hdgeant_smeared.hddm files<br>
>>>> that produced short output for him to test.<br>
>>>><br>
>>>> Running Jobs at JLab<br>
>>>><br>
>>>> Mark has submitted some test jobs against the new branch. [Added in<br>
>>>> press: 1,000 50 k-event jobs have been submitted.]<br>
>>>><br>
>>>> Status of Preparations<br>
>>>><br>
>>>> Random number seeds procedure<br>
>>>><br>
>>>> Paul spoke to David about this. It seems that mcsmear is currently<br>
>>>> generating its own random number seed. We still have details to fill in<br>
>>>> on this story.<br>
>>>><br>
>>>> Running Jobs at FSU<br>
>>>><br>
>>>> FSU has started running data challenge test jobs on their cluster.<br>
>>>> Aristeidis has started with 50 jobs, but an early look shows problems<br>
>>>> with some of them in hd_root. Also there was the GlueX-software-induced<br>
>>>> crash of the FSU cluster[?].<br>
>>>><br>
>>>> Running jobs at CMU<br>
>>>><br>
>>>> Paul is seeing ZFATAL errors from hdgeant. He will send a bug report to<br>
>>>> Richard who will look into a fix beyond merely increasing ZEBRA memory.<br>
>>>><br>
>>>> Richard asked about an issue where JANA takes a long time to identify a<br>
>>>> CODA file as not an HDDM file. Richard would like to fix the HDDM<br>
>>>> parser such that this is not the case. Mark D. will send Richard an<br>
>>>> example.<br>
>>>><br>
>>>> Running Jobs at NU<br>
>>>><br>
>>>> Sean regaled us with tales of site specific problems.<br>
>>>><br>
>>>> Lots of jobs crashed at REST generation. Site configuration changes<br>
>>>> helped. But there were still a lot of jobs hanging, usually with new<br>
>>>> nodes. Reducing the number of submit slots fixed most of the problems.<br>
>>>> Many of the remaining symptoms were jobs hung on the first event when<br>
>>>> accessing the magnetic field. Jobs are single-threaded. [23]Some<br>
>>>> statistics on the results were presented as well.<br>
>>>><br>
>>>> Richard remarked that on the OSG, jobs will start much faster if<br>
>>>> declared as single-threaded.<br>
>>>><br>
>>>> Richard proposed the following standards:<br>
>>>><br>
>>>> BGRATE 1.1 (equivalent to 10^7) BGGATE -800 800 (in ns, time gate for<br>
>>>> EM background addition)<br>
>>>><br>
>>>> We agreed on these as standard settings.<br>
>>>><br>
>>>> Mark proposed the following split of running:<br>
>>>><br>
>>>> 15% with no EM background 70% with EM background corresponding to 10^7<br>
>>>> 15% with EM background corresponding to 5\×10^7<br>
>>>><br>
>>>> There was general agreement; adjustment may happen in the future.<br>
>>>><br>
>>>> Running Jobs at MIT<br>
>>>><br>
>>>> Justin has been running with the dc-2.2 tag. The OpenStack cluster at<br>
>>>> MIT has about 180 cores and he has been running jobs for a couple of<br>
>>>> days with good success. BGGATE was set at -200 to 200.<br>
>>>><br>
>>>> Electromagnetic Background<br>
>>>><br>
>>>> Kei gave us an [24]update on his studies of EM background with hdds-2.0<br>
>>>> and sim-recon-dc-2.1. Slides covered:<br>
>>>> * Memory Usage<br>
>>>> * CPU time<br>
>>>> * mcsmear File Sizes<br>
>>>> * REST File Sizes<br>
>>>> * Another Bad File<br>
>>>> * Sum of parentid=0<br>
>>>> * Correlation of CDC hits<br>
>>>> * Correlation of FDC hits<br>
>>>> * pπ^+π^- Events<br>
>>>><br>
>>>> Proposed Schedule<br>
>>>><br>
>>>> The schedule has slipped. The new schedule is as follows:<br>
>>>> 1. Launch of Data Challenge Thursday March 6, 2014 (est.).<br>
>>>> 2. Test jobs going successfully by Tuesday March 4.<br>
>>>> 3. Distribution ready by Monday March 3.<br>
>>>><br>
>>>> Justin pointed out that the short REST file problem might be something<br>
>>>> that we could live with for this data challenge.<br>
>>>><br>
>>>> Richard asked that Mark assign run numbers and run conditions for the<br>
>>>> various sites.<br>
>>>><br>
>>>> Action Items<br>
>>>><br>
>>>> 1. Understand random number seed system.<br>
>>>> 2. Solve ZFATAL crashes.<br>
>>>> 3. Make a table of conditions vs. sites where the entries are assigned<br>
>>>> file numbers.<br>
>>>> 4. Report the broken Polycom in L207.<br>
>>>><br>
>>>> Retrieved from<br>
>>>> "<a href="https://halldweb1.jlab.org/wiki/index.php/GlueX_Data_Challenge_Meeting,_February_28,_2014">https://halldweb1.jlab.org/wiki/index.php/GlueX_Data_Challenge_Meeting,_February_28,_2014</a>"<br>
>>>><br>
>>>> References<br>
>>>><br>
>>>> 21.<br>
>>>> <a href="https://mailman.jlab.org/pipermail/halld-offline/2014-February/001511.html">https://mailman.jlab.org/pipermail/halld-offline/2014-February/001511.html</a><br>
>>>> 22.<br>
>>>> <a href="https://mailman.jlab.org/pipermail/halld-offline/2014-February/001512.html">https://mailman.jlab.org/pipermail/halld-offline/2014-February/001512.html</a><br>
>>>> 23.<br>
>>>> <a href="https://halldweb1.jlab.org/wiki/images/f/f8/DC2-Meeting-sdobbs-20140228.pdf">https://halldweb1.jlab.org/wiki/images/f/f8/DC2-Meeting-sdobbs-20140228.pdf</a><br>
>>>> 24. <a href="https://halldweb1.jlab.org/wiki/images/e/ea/2014-02-28-DC2.pdf">https://halldweb1.jlab.org/wiki/images/e/ea/2014-02-28-DC2.pdf</a><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> Halld-offline mailing list<br>
>>>> <a href="mailto:Halld-offline@jlab.org">Halld-offline@jlab.org</a><br>
>>>> <a href="https://mailman.jlab.org/mailman/listinfo/halld-offline">https://mailman.jlab.org/mailman/listinfo/halld-offline</a><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Halld-offline mailing list<br>
>>> <a href="mailto:Halld-offline@jlab.org">Halld-offline@jlab.org</a><br>
>>> <a href="https://mailman.jlab.org/mailman/listinfo/halld-offline">https://mailman.jlab.org/mailman/listinfo/halld-offline</a><br>
>><br>
><br>
<br>
<br>
_______________________________________________<br>
Halld-offline mailing list<br>
<a href="mailto:Halld-offline@jlab.org">Halld-offline@jlab.org</a><br>
<a href="https://mailman.jlab.org/mailman/listinfo/halld-offline">https://mailman.jlab.org/mailman/listinfo/halld-offline</a></div></div></blockquote></div><br></div>