<div dir="ltr">I tagged the current master since there are many bug fixes already since 5a.1.0 from last week<div><br></div><div>Is there one person who has the expertise to systematically assess the DB accesses of all services, or do we have to rely on every system expert to certify that their service will not do DB access every event? I agree with Vardan that this, and the systematic printing of error messages at every event, need to be fixed in priority. This seems like it should be easy to fix and can dramatically improve performances.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 2, 2018 at 8:13 PM, Vardan Gyurjyan <span dir="ltr"><<a href="mailto:gurjyan@jlab.org" target="_blank">gurjyan@jlab.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Hi Silvester,<div>I also notice that FTCAL and FTHODO services are making many dB accesses during the initialization. I expect to see a single access to the database followed by a single dB disconnect.</div><div>-Vardan</div><div><br><br><div id="m_7933519319901114395AppleMailSignature">Sent from my iPhone</div><div><br>On Mar 2, 2018, at 7:50 PM, Sylvester J. Joosten <<a href="mailto:tuf42480@temple.edu" target="_blank">tuf42480@temple.edu</a>> wrote:<br><br></div><blockquote type="cite"><div>Hi Rafaella, hi Vardan,<div><br></div><div>In case this is useful: I recognize this error in the context of accessing tables from CCDB from COATJAVA. If this happens when</div><div><br></div><div>if(this.entries.hasItem(index)<wbr>==false) (org.jlab.utils.groups.<wbr>IndexedTable)</div><div><br></div><div>fails. entries.hasItem() contains the following checks:</div><div> </div><div><div>—> calls IndexedList.hasItem(int... index) (org.jlab.utils.groups.<wbr>IndexedList)</div><div>—> has 2 internal checks:</div><div> 1. if(index.length!=this.<wbr>indexSize)</div><div> 2. IndexGenerator.hashCode(<wbr>index);</div><div><br></div><div>In my experience, this implies that there is some kind of issue or inconsistency when accessing CCDB, at least for this particular run.</div><div><br></div><div>Just my 2 cents.</div><div>Best,</div><div>Sylvester</div><div><div class="h5"><div><br></div><div><br><blockquote type="cite"><div>On Mar 2, 2018, at 7:37 PM, Vardan Gyurjyan <<a href="mailto:gurjyan@jlab.org" target="_blank">gurjyan@jlab.org</a>> wrote:</div><br class="m_7933519319901114395Apple-interchange-newline"><div><div dir="auto">Hi Raffaella,<div>I do not know what is exactly the cause but whenever I use one of these services in the data processing chain I get out memory exception. This exception is not recoverable. As I mention this happens every single time on clonfar0 node. I never saw this error on the farm machines though. May be FX can comment. I am getting these error on data over the ET as well as decoded files from FX’s decoded files directory. <br>Vardan<br><div>Sent from my iPhone</div><div><br>On Mar 2, 2018, at 4:53 PM, Raffaella De Vita <<a href="mailto:Raffaella.Devita@ge.infn.it" target="_blank">Raffaella.Devita@ge.infn.it</a>> wrote:<br><br></div><blockquote type="cite"><div>
Hi Vardan,<br>
I'm the author of those services. The only change that was done
recently was a modification of an hardcoded constant and, after that
was done, FX cooked several files for FT studies. Nothing else was
changed in more than one month. Anyway, I will try to reproduce the
problem and debug it.What data are you processing?<br>
Regards,<br>
Raffaella<br>
<br>
<div class="m_7933519319901114395moz-cite-prefix">Vardan Gyurjyan wrote:<br>
</div>
<blockquote type="cite">
Hi FX,
<div><br>
</div>
<div>Since I am not sure who is the FTCAL and FTHODO
engines author I am cc-ing this email to clas12.</div>
<div>There is a critical bug introduced in these service
engine’s code for the 5a.1.0 release. It is 100% reproducible on
clonfarm0 node (online reconstruction node), where JVM crashes
with the out of memory exception. Reconstruction chain without
these services function properly.</div>
<div>These are service engines that also print for every
event warning messages such as “ [IndexedTable] ---> error..
entry does not exist” (I consider them warning since if this is
a real error the processing should be stoped). Any ways, this is
a serious bug that can result in large number of job failures on
the farm.</div>
<div> </div>
<div>-vardan<br>
<div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">------------------------------<wbr>--------------------<br>
Vardan H. Gyurjyan, Ph.D.<br>
Staff Scientist<br>
Thomas Jefferson Accelerator Facility<br>
Newport News, VA, 23606<br>
E-mail: <a href="mailto:gurjyan@jlab.org" target="_blank">gurjyan@jlab.org</a><br>
<a href="tel:(757)%20269-5879" value="+17572695879" target="_blank">757-269-5879</a> (JLAB)</div>
</div>
<br>
</div>
<br>
<fieldset class="m_7933519319901114395mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Clas12_software mailing list
<a class="m_7933519319901114395moz-txt-link-abbreviated" href="mailto:Clas12_software@jlab.org" target="_blank">Clas12_software@jlab.org</a>
<a class="m_7933519319901114395moz-txt-link-freetext" href="https://mailman.jlab.org/mailman/listinfo/clas12_software" target="_blank">https://mailman.jlab.org/<wbr>mailman/listinfo/clas12_<wbr>software</a></pre>
</blockquote>
<br>
</div></blockquote></div></div>______________________________<wbr>_________________<br>Clas12_software mailing list<br><a href="mailto:Clas12_software@jlab.org" target="_blank">Clas12_software@jlab.org</a><br><a href="https://mailman.jlab.org/mailman/listinfo/clas12_software" target="_blank">https://mailman.jlab.org/<wbr>mailman/listinfo/clas12_<wbr>software</a></div></blockquote></div><br></div></div></div></div></blockquote></div></div><br>______________________________<wbr>_________________<br>
Clas12_software mailing list<br>
<a href="mailto:Clas12_software@jlab.org">Clas12_software@jlab.org</a><br>
<a href="https://mailman.jlab.org/mailman/listinfo/clas12_software" rel="noreferrer" target="_blank">https://mailman.jlab.org/<wbr>mailman/listinfo/clas12_<wbr>software</a><br></blockquote></div><br></div>