Was this with the root look fixed? I merged them into the master. We will need another tag. Idk if the unfixed code is reliable when it doesn't crash.<div><br></div><div>Naomi.</div><div>On a bus in England heh bye </div><div><br><br>On Friday, June 28, 2024, Igal Jaegle <<a href="mailto:ijaegle@jlab.org">ijaegle@jlab.org</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The second attempt worked, the job took slightly more than 3h with 100% success rate but the skims are almost all larger than the one produced locally. So, there is still an issue.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The files are on volatile and will be moved tomorrow on cache</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
/volatile/halld/offsite_prod/<wbr>RunPeriod-2022-05/recon/ver99-<wbr>perl/RUN101586/</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
tks ig.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
s -lrth /volatile/halld/offsite_prod/<wbr>RunPeriod-2022-05/recon/ver99-<wbr>perl/RUN101586/FILE000/<wbr>RUN101586/FILE000/</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
total 5.2G</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 784 Jun 25 05:42 tree_sc_eff.root</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 315K Jun 27 22:13 hd_rawdata_101586_000.CCAL-<wbr>LED.evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 16M Jun 27 22:13 hd_rawdata_101586_000.BCAL-<wbr>LED.evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 7.5K Jun 27 22:13 syncskim.root</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 174K Jun 27 22:13 job_info_101586_000.tgz</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 227M Jun 27 22:14 hd_rawdata_101586_000.cpp_2c.<wbr>evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 778M Jun 27 22:14 hd_rawdata_101586_000.ctof.<wbr>evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 42M Jun 27 22:15 converted_random_101586_000.<wbr>hddm</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 49M Jun 27 22:15 tree_TPOL.root</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 8.5K Jun 27 22:15 tree_TS_scaler.root</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 1.2G Jun 27 22:15 dana_rest.hddm</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 93M Jun 27 22:15 tree_PSFlux.root</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 129K Jun 27 22:15 tree_fcal_hadronic_eff.root</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 7.2K Jun 27 22:15 tree_bcal_hadronic_eff.root</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 37M Jun 27 22:15 hd_root.root</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 308K Jun 27 22:16 hd_rawdata_101586_000.sync.<wbr>evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 315K Jun 27 22:16 hd_rawdata_101586_000.DIRC-<wbr>LED.evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 127M Jun 27 22:18 hd_rawdata_101586_000.npp_<wbr>2pi0.evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 106M Jun 27 22:18 hd_rawdata_101586_000.npp_2g.<wbr>evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 19M Jun 27 22:18 hd_rawdata_101586_000.FCAL-<wbr>LED.evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 180M Jun 27 22:21 hd_rawdata_101586_000.random.<wbr>evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 2.7G Jun 27 22:21 hd_rawdata_101586_000.ps.evio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-rw-r--r-- 1 gxproj4 halld 1.1M Jun 27 22:27 tree_tof_eff.root</div>
<div></div>
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Igal Jaegle <<a href="mailto:ijaegle@jlab.org" target="_blank">ijaegle@jlab.org</a>><br>
<b>Sent:</b> Wednesday, June 26, 2024 10:38 AM<br>
<b>To:</b> Alexander Austregesilo <<a href="mailto:aaustreg@jlab.org" target="_blank">aaustreg@jlab.org</a>>; <a href="mailto:halld-cpp@jlab.org" target="_blank">halld-cpp@jlab.org</a> <<a href="mailto:halld-cpp@jlab.org" target="_blank">halld-cpp@jlab.org</a>><br>
<b>Cc:</b> Naomi Jarvis <<a href="mailto:nsj@cmu.edu" target="_blank">nsj@cmu.edu</a>>; Sean Dobbs <<a href="mailto:sdobbs@jlab.org" target="_blank">sdobbs@jlab.org</a>><br>
<b>Subject:</b> Re: CPP REST production</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
PERLMUTTER is down due to a maintenance day. But tomorrow I can grab the other logs.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
tks ig.</div>
<div></div>
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Alexander Austregesilo <<a href="mailto:aaustreg@jlab.org" target="_blank">aaustreg@jlab.org</a>><br>
<b>Sent:</b> Wednesday, June 26, 2024 10:32 AM<br>
<b>To:</b> Igal Jaegle <<a href="mailto:ijaegle@jlab.org" target="_blank">ijaegle@jlab.org</a>>; <a href="mailto:halld-cpp@jlab.org" target="_blank">halld-cpp@jlab.org</a> <<a href="mailto:halld-cpp@jlab.org" target="_blank">halld-cpp@jlab.org</a>><br>
<b>Cc:</b> Naomi Jarvis <<a href="mailto:nsj@cmu.edu" target="_blank">nsj@cmu.edu</a>>; Sean Dobbs <<a href="mailto:sdobbs@jlab.org" target="_blank">sdobbs@jlab.org</a>><br>
<b>Subject:</b> Re: CPP REST production</font>
<div> </div>
</div>
<div>
<p>Hi Igal,</p>
<p>This failure rate is pretty bad. Can you point me to the log files of the failed jobs? The messages you sent concerning root dictionaries and ps_counts_thresholds are not fatal, they will not cause a failures. You should still confirm with Sasha if the ps_counts_thresholds
are important.<br>
</p>
<p><br>
</p>
<p>The file /work/halld/home/gxproj4/<wbr>public/ForAlexAndSean/std.out seems to be cut off. We may want to add this option to the config file to remove the number of processed events:</p>
<p><br>
</p>
<p>JANA:BATCH_MODE 1</p>
<p><br>
</p>
<p>You can find my output files here: /work/halld2/home/aaustreg/<wbr>Analysis/cpp/REST/</p>
<p>I don't have the logs, but all files were closed at exactly the same time.</p>
<p><br>
</p>
<p>Cheers,</p>
<p>Alex<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div>On 6/26/24 08:59, Igal Jaegle wrote:<br>
</div>
<blockquote type="cite">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Alex,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Could you provide the path to your output files and most importantly the logs?</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The results of the test on NERSC for the same run are as follows</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
108 evio files were cooked properly out of 126, thus 15% failure rate which is way too much to proceed with the cooking.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
tks ig.</div>
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Alexander Austregesilo
<a href="mailto:aaustreg@jlab.org" target="_blank"><aaustreg@jlab.org></a><br>
<b>Sent:</b> Monday, June 24, 2024 6:06 PM<br>
<b>To:</b> <a href="mailto:halld-cpp@jlab.org" target="_blank">
halld-cpp@jlab.org</a> <a href="mailto:halld-cpp@jlab.org" target="_blank">
<halld-cpp@jlab.org></a><br>
<b>Cc:</b> Naomi Jarvis <a href="mailto:nsj@cmu.edu" target="_blank">
<nsj@cmu.edu></a>; Igal Jaegle <a href="mailto:ijaegle@jlab.org" target="_blank">
<ijaegle@jlab.org></a>; Sean Dobbs <a href="mailto:sdobbs@jlab.org" target="_blank">
<sdobbs@jlab.org></a><br>
<b>Subject:</b> CPP REST production</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>Dear Colleagues,<br>
<br>
I processed one single file of a typical CPP production run on Pb <br>
target. Here is a list of all files and their sizes which will be <br>
produced suring the REST prodution:<br>
<br>
1.7G hd_rawdata_101586_000.ps.evio<br>
1.2G dana_rest.hddm<br>
495M hd_rawdata_101586_000.ctof.<wbr>evio<br>
160M hd_rawdata_101586_000.cpp_2c.<wbr>evio<br>
115M hd_rawdata_101586_000.random.<wbr>evio<br>
93M tree_PSFlux.root<br>
90M hd_rawdata_101586_000.npp_<wbr>2pi0.evio<br>
72M hd_rawdata_101586_000.npp_2g.<wbr>evio<br>
49M tree_TPOL.root<br>
42M converted_random.hddm<br>
41M hd_root.root<br>
16M hd_rawdata_101586_000.FCAL-<wbr>LED.evio<br>
13M hd_rawdata_101586_000.BCAL-<wbr>LED.evio<br>
1.1M tree_tof_eff.root<br>
164K tree_fcal_hadronic_eff.root<br>
105K hd_rawdata_101586_000.DIRC-<wbr>LED.evio<br>
105K hd_rawdata_101586_000.CCAL-<wbr>LED.evio<br>
94K hd_rawdata_101586_000.sync.<wbr>evio<br>
24K tree_TS_scaler.root<br>
24K tree_bcal_hadronic_eff.root<br>
24K syncskim.root<br>
<br>
The trigger skims for ps (with thick converter) and ctof are quite <br>
large. Do we actually need them or were they maybe already produced <br>
during the calibration stages?<br>
<br>
As far as I understand, Igal has started a test of the REST production <br>
at NERSC. We are getting closer to launch!<br>
<br>
Cheers,<br>
<br>
Alex<br>
<br>
<br>
<br>
<br>
<br>
-- <br>
Alexander Austregesilo<br>
<br>
Staff Scientist - Experimental Nuclear Physics<br>
Thomas Jefferson National Accelerator Facility<br>
Newport News, VA<br>
<a href="mailto:aaustreg@jlab.org" target="_blank">aaustreg@jlab.org</a><br>
(757) 269-6982<br>
<br>
</div>
</span></font></div>
</blockquote>
<pre cols="72">--
Alexander Austregesilo
Staff Scientist - Experimental Nuclear Physics
Thomas Jefferson National Accelerator Facility
Newport News, VA
<a href="mailto:aaustreg@jlab.org" target="_blank">aaustreg@jlab.org</a>
(757) 269-6982
</pre>
</div>
</div>
</div>
</blockquote></div>