[Halld-tagger] jana hangs

[Richard Jones]

Dave,

I have collected a bunch of log files for you to browse through. Look at

http://zeus.phys.uconn.edu/halld/gridwork/badlogs/dana-hangs

These logs come in stderr.N and stdout.N pairs.  They contain all of the output from the start of the job, including some inane messages from the job setup scripts.  The jobs attempt to run the analysis chain twice, so you can see what happens when the same thing is attempted two times in a row on the same machine, to look for reproducibility.

Generically what you see is the following: hd_ana is running along merrily, and then into the error log pops a message like:

JANA ERROR>> Thread 0 hasn't responded in 30 seconds. (run:event=9000:5926) Cancelling ...
JANA ERROR>>Caught HUP signal for thread 0x2aaab3b55940 thread exiting...
JANA ERROR>> Launching new thread ...

Then after the launching new thread, everything goes dark.  The process just sits dormant, neither writing to disk nor consuming cpu.  You can see the command line used to start hd_ana in the stdout file.  As far as I know, there is just one processing thread in operation.  In most of these logs, you see the signal sent by the batch job monitor after the wall clock has passed the quota for the job.  In some of the later jobs, you might also see evidence of the signal 15 that my watchdog script uses to try and clear out hung jobs.

You can also see evidence of an occasional problem reported by the bzlib compression library, that an invalid compression code has been requested.  I should probably look into that after all of this is over.  That only happens on certain runs, but from the logs it appears to be reproducible.

If you are interested, you can also see logs from the other general class of failed jobs, where hdgeant segfaulted. That is reproducible and happily it appears upstream of mcsmear where irreproducible behavior begins, so it should be quick to find and fix.  A sampling of logs of this type are found in the web folder:

http://zeus.phys.uconn.edu/halld/gridwork/badlogs/hdgeant-segfaults

-Richard J.





On 12/12/2012 5:03 PM, David Lawrence wrote:
>
> Hi Richard,
>
>    At this point, there shouldn't be anything in JANA that looks for things on the web. I know both gxtwist and the hdparsim plugin used curl to grab files from the web, but I don't believe you are using either of those for the data challenge. You probably are also not using the CCDB since that is not yet the default. I'm at a loss as to what might cause a network access if that is what's going on.
>
>   I would really like to get to the bottom of the JANA hangs. A long-term solution that requires a watcher program seems unacceptable to me. At least before we make a strong attempt to track down the cause. Can you give me a little more information:
>
> - What JANA program is hanging?
> - How many processing threads are you using?
> - Is the output just a continuous cycling of the "thread X has not responded ..." messages?
>
> Regards,
> -David
>
>
> On 12/12/12 3:20 PM, Richard Jones wrote:
>> Mark and David,
>>
>> Ok, sounds good.  I was late for the meeting because I triggered a security flag at Fermilab.  It is being worked out. Apparently too many offsite connections per hour to be physics. Hmmm, what could he be doing??  It is being worked out.
>>
>> With regards to this, something looks suggestive about the way the hd_ana is hanging in streaks.  It seems to run fine for a while, then across a site a wave of hangs seems to set in. Another thing looks odd, that I have seen the message "DParticle: loading constants from the data base" or something like that, and then - lights out!  Is it possible that there are secret web accesses going on from inside the application, something like fetches of data or documents from remote web servers on-the-fly inside the application.  I know it is a long shot, but this security flag got me to thinking about what I could be doing that I don't realize.
>>
>> -Richard J.
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-tagger/attachments/20121212/afe51481/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3232 bytes
Desc: S/MIME Cryptographic Signature
Url : https://mailman.jlab.org/pipermail/halld-tagger/attachments/20121212/afe51481/attachment.bin