[Halld-offline] segfaults in hd_root

[Richard Jones]

Dear David and all,

I confirm that the latest patch to Jana that David created this morning
solved the segfault problems that I was seeing. Thank you for the rapid
response, am switching production to the new Jana now.

-Richard J.


On Tue, Mar 25, 2014 at 9:52 AM, Mark Ito <marki at jlab.org> wrote:

> Richard and David,
>
> Sounds like we should switch to this new release for everyone, though I
> admit that that is not obvious. For me, I think I would like to eliminate
> the problem as a possibility. Note that in any case, everything we have
> done so far is OK output-file-wise, so we are not under the gun to do the
> switch "now, now, now!" Probably should be done soon though, if we are
> going to switch everyone.
>
>   -- Mark
>
>
> On 03/25/2014 09:45 AM, David Lawrence wrote:
>
>> Hi Richard and Mark,
>>
>>    I believe I now have a fix for this. I was able to reproduce the
>> problem and after making a "minor" modification, it seems to have gone
>> away. The only change was to add the RTLD_NODELETE flag to the call to
>> dlopen when attaching a plugin. This basically prevents the shared object
>> from actually being detached, therefore, preventing the multiple destructor
>> calls. I've made a new JANA release, 0.7.1p2 that has this one change in
>> it. You can download the tar ball from here:
>>
>> http://www.jlab.org/JANA/releases/jana_0.7.1p2.tgz
>>
>> or check out the source from subversion here:
>>
>> svn co https://phys12svn.jlab.org/repos/tags/jana_0.7.1p2
>>
>>
>> The minimal set of unit tests I have seemed to work OK with this so
>> hopefully it will not give you any problems.
>>
>> Regards,
>> -David
>>
>> On Mar 25, 2014, at 6:55 AM, David Lawrence <davidl at jlab.org> wrote:
>>
>>  Hi Richard,
>>>
>>>   This is a problem that I have seen before and thought I fixed. It is
>>> because the linker automatically connects the global jout and jerr objects
>>> of the plugins and the main executable when they are attached. When a
>>> plugin is detached (via dlclose) the destructor is automatically called.
>>> The other plugin though still has reference to jout and jerr which causes a
>>> seg. fault when it tries to use them (even if that is just calling the
>>> destructors).
>>>
>>>   This is an annoying problem that did not always exist and seems to
>>> have come about with a change in the system somehow rather than a change in
>>> the code. It shouldn't affect the output files, but does cause the program
>>> to report that it crashed which causes confusion and needs to be fixed.
>>> I'll take another look to see what can be done.
>>>
>>> Regards,
>>> -Dave
>>>
>>>
>>> On Mar 25, 2014, at 6:47 AM, Richard Jones <richard.t.jones at uconn.edu>
>>> wrote:
>>>
>>>  Hello Dave, Mark,
>>>>
>>>> I am getting jobs failing with segfault at the end of the hd_root
>>>> command, as documented on the wiki page.  As long as I only declare one
>>>> plugin, all ends well, but as soon as I declare two as in
>>>> "-PPLUGINS=monitoring_hists,danarest" then the job crashes at the end
>>>> with a segfault.  The error seems to be in the output to jout at the end of
>>>> the ~JApplication destructor, second time around the loop.  The first time
>>>> around the loop it is fine, but the second time (after dlclose() has been
>>>> called on danarest.so) the std::ostream::operator<<() method goes belly-up.
>>>>
>>>> As a workaround, I have split the monitoring_hists and danarest plugins
>>>> into separate runs, and both seem to complete ok.  It is a bit of a waste
>>>> of time, though.  I guess there is some dependency somewhere in an existing
>>>> object associated with jout that is being invalidated by the dlclose on
>>>> danarest.so ?
>>>>
>>>> -Richard J.
>>>>
>>>
> --
> Mark M. Ito, Jefferson Lab, marki at jlab.org, (757)269-5295
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140325/740ee46e/attachment.html