<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">Hi All,</div><div class=""><br class=""></div><div class=""> Yes, I have seen this crash quite a bit over the past few days. It usually has DFCALCluster in the stack trace,</div><div class="">but often will have several frames with “???” above it. It also seems mysteriously temporal. I have seen periods</div><div class="">where all jobs crash within one minute of starting and others where they all survive the entire run. I’m still</div><div class="">looking into it. Maybe this clue Alex points to will yield something. Stay tuned….</div><div class=""><br class=""></div><div class="">-David</div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 8, 2017, at 3:08 PM, Sean Dobbs <<a href="mailto:s-dobbs@northwestern.edu" class="">s-dobbs@northwestern.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">For what it's worth, looking at the bracktrace under gdb shows that the stack appears corrupt. Maybe running it under valgrind would help find the memory error?<div class=""><br class=""></div><div class="">Snippet:</div><div class=""><div class="">#38 0x0000000000a2dee9 in jana::JEventLoop::Get<DFCALCluster> (this=0x7fffd0873998, t=<error reading variable: Cannot access memory at address 0x54>, </div><div class=""> tag=0x7fffb5b8fc20 "P\t\032\001", allow_deftag=<optimized out>)</div><div class=""> at /u/group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5/jana/jana_0.7.5p2^root6.06.08/Linux_CentOS7-x86_64-gcc4.8.5/include/JANA/JEventLoop.h:304</div><div class="">Backtrace stopped: previous frame inner to this frame (corrupt stack?)</div></div><div class=""><br class=""></div><div class="">---Sean</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Wed, Feb 8, 2017 at 1:56 PM Alexander Austregesilo <<a href="mailto:aaustreg@jlab.org" class="">aaustreg@jlab.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="gmail_msg"><p class="gmail_msg">Hi David,</p><p class="gmail_msg">I saw that you fixed a bug in occupancy_online plugin. Unfortunately, there seems to be another one in highlevel_online. In about 10% of all jobs, I see crashes without any reasonable hint to what is happening. However, I could track it down to this plugin.
Here is an example:</p><p class="gmail_msg">hd_root -PEVENTS_TO_SKIP=155600 -PPLUGINS=highlevel_online /cache/halld/RunPeriod-2017-01/rawdata/Run030280/hd_rawdata_030280_041.evio</p><p class="gmail_msg">This should crash after 75events and the stack trace looks awful.</p><p class="gmail_msg">Would you mind having a look?</p><p class="gmail_msg">Cheers,</p><p class="gmail_msg">Alex<br class="gmail_msg">
</p><p class="gmail_msg"><br class="gmail_msg">
</p>
<br class="gmail_msg">
<div class="gmail_msg m_2276503083908144802moz-cite-prefix">On 02/06/2017 03:25 PM, gluex wrote:<br class="gmail_msg">
</div>
<blockquote type="cite" class="gmail_msg"><p class="gmail_msg">Test status for this pull request: SUCCESS</p><p class="gmail_msg">Summary: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__halldweb.jlab.org_pull-5Frequest-5Ftest_sim-2Drecon-255Edavidl-5FL1trigger-5Fmonitor_tests-2Dsummary.txt&d=CwMGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=T8VpbeyfSWpCctf2DXXwHXHUoJu6h76LJ1kyhxsnxVw&s=vvfBME-Ay7Y1_fXZEGyZ9VAZRgvpPpiy3X3yr4E8PwM&e=" class="gmail_msg" target="_blank">
/work/halld/pull_request_test/sim-recon^davidl_L1trigger_monitor/tests/summary.txt</a><br class="gmail_msg">
Logs: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__halldweb.jlab.org_pull-5Frequest-5Ftest_sim-2Drecon-255Edavidl-5FL1trigger-5Fmonitor_tests-2Dlogs&d=CwMGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=T8VpbeyfSWpCctf2DXXwHXHUoJu6h76LJ1kyhxsnxVw&s=ZTnhZcea46_6IiibB8Hb989B9rRfgZxafX0UqxzJBSU&e=" class="gmail_msg" target="_blank">
/work/halld/pull_request_test/sim-recon^davidl_L1trigger_monitor/tests/log</a></p><p class="gmail_msg">Build log: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__halldweb.jlab.org_pull-5Frequest-5Ftest_sim-2Drecon-255Edavidl-5FL1trigger-5Fmonitor_make-5Fdavidl-5FL1trigger-5Fmonitor.log&d=CwMGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=T8VpbeyfSWpCctf2DXXwHXHUoJu6h76LJ1kyhxsnxVw&s=yv9lEc5U-oeLGXxzsHhzEnKK6_aC9lnAyUuk1ubLyqE&e=" class="gmail_msg" target="_blank">
/work/halld/pull_request_test/sim-recon^davidl_L1trigger_monitor/make_davidl_L1trigger_monitor.log</a><br class="gmail_msg">
Build report: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__halldweb.jlab.org_pull-5Frequest-5Ftest_sim-2Drecon-255Edavidl-5FL1trigger-5Fmonitor_report-5Fdavidl-5FL1trigger-5Fmonitor.txt&d=CwMGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=T8VpbeyfSWpCctf2DXXwHXHUoJu6h76LJ1kyhxsnxVw&s=bMmJ-y-gjgxbMWIqtdnTp5sUXu7vWfj9YITnNxtpqe4&e=" class="gmail_msg" target="_blank">
/work/halld/pull_request_test/sim-recon^davidl_L1trigger_monitor/report_davidl_L1trigger_monitor.txt</a><br class="gmail_msg">
Location of build: /work/halld/pull_request_test/sim-recon^davidl_L1trigger_monitor</p><p style="font-size:small;color:#666" class="gmail_msg">—<br class="gmail_msg">
You are receiving this because you are subscribed to this thread.<br class="gmail_msg">
Reply to this email directly, <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_JeffersonLab_sim-2Drecon_pull_693-23issuecomment-2D277802649&d=CwMGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=T8VpbeyfSWpCctf2DXXwHXHUoJu6h76LJ1kyhxsnxVw&s=iPPY5ZXi4p72SSOFBHTVHpKIMZ9JnyP5BgaW3jhCAb0&e=" class="gmail_msg" target="_blank">
view it on GitHub</a>, or <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIZiyGShO0ZM4XGcSbvmtnDDIBkiQpO3ks5rZ4HJgaJpZM4L4piG&d=CwMGaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=T8VpbeyfSWpCctf2DXXwHXHUoJu6h76LJ1kyhxsnxVw&s=wYQ_LWCpzKZEziZHzlJpbdNM0mthtURBt_Np9E8oe84&e=" class="gmail_msg" target="_blank">
mute the thread</a>.<img alt="" src="https://github.com/notifications/beacon/AIZiyCvRQ5A1P6_J8W9ei5yDMX_RaX8bks5rZ4HJgaJpZM4L4piG.gif" height="1" width="1" class="gmail_msg"></p>
<div class="gmail_msg">
<div class="gmail_msg">
</div>
</div>
</blockquote>
<br class="gmail_msg">
</div>
</blockquote></div>
</div></blockquote></div><br class=""></body></html>