[Halld-offline] hd_ana hanging after variable number of bggen events

Richard Jones richard.t.jones at uconn.edu
Wed Mar 16 10:18:21 EDT 2011


David,

Yes, I have reproduced it as well, and like you point out, it can be reproduced with just a single event.  I have found the instructions that cause the problem, and am checking into how compare is failing to go false before it runs off the end of the list.  One possibility is that there is a NaN in one of the registers, but I have to remember how to dump the floating point stack in the IA32 floating point unit.  This compare is being performed by the FUCOMPP instruction, which sets the Unordered flag in the FP status register if you try to do a compare with an "uncomparable value" like NaN.  I don't think the SortInteractions() function checks for that, and the Perp() method probably does a square root which might make a NaN under some circumstances.  I suspect that it is something related, but only have a few minutes here and there to look at it.  Have to go teach, more later today...

-Richard J.


On 3/16/2011 8:21 AM, David Lawrence wrote:
> Hi Richard (and offliners),
>
>      Just a quick update that I have been able to reproduce the problem and it does appear to be the same problem earlier reported. I have added a note to the issue in Mantis (https://halldnew.jlab.org/mantisbt/view.php?id=38) indicating this. I now have an event which seems to reliably cause the problem though the exact symptoms seem to vary a little. This is likely due to the sort-gone-crazy problem corrupting memory in a way that causes different behavior depending on details of how things are laid out when the corruption occurs.
>
>      No real clues yet on the underlying cause. If it is a bug in the optimizer as Matt suspects, then we'll have to treat it symptomatically. I'm going to try to get a little more proof that that is the case before resorting to that however.
>
>      We'll have to discuss the backtrace issue some more at the offline meeting. I may not be understanding all of the details. I will say for this particular problem, I was able to launch gdb and attach it to an existing process while it was hung and view the stack trace for all threads.
>
> Regards,
> -Dave
>
> On 3/14/11 8:45 AM, Richard Jones wrote:
> Dave,
>
> Ok, I thought it might be related.  I think the following represents progress on this front:
>
>   1.  It is not a "hang" but a segfault in child thread #2
>   2.  Segfault in a child thread causes the code to hang.  This seems to be because we are going through the root signal handling mechanism, which is badly broken in the JANA context.
>
> I suggest that the top priority is to fix item #2, by writing your own signal recovery and backtrace mechanism for the JANA framework.  This seems like a first-order requirement for our analysis framework, to have a signal recovery and backtrace mechanism with an appropriate behavior.  Once that is done, tracing other problems, such as item #1, will be more feasible.
>
> -Richard J.
>
>
>
>
> On 3/14/2011 8:37 AM, David Lawrence wrote:
>
>
> Hi Richard,
>
>      The DTrackCandidate_factory_CDC::FindThetaZRegression() has come up before when Kei and Jake reported problems with DANA programs hanging as far back as December. This led to the "fix" currently used (though not incorporated into the build system) where optimization is turned off when compiling DTrackCandidate_factory_CDC.cc. As of yet, we have not been able to identify the bug exactly as the behavior is not deterministic.
>
>      I will take another look at this today to see if I can make some more headway on the problem.
>
> Regards,
> -Dave
>
>
>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4092 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20110316/4d51ded5/attachment.p7s>


More information about the Halld-offline mailing list