[Halld-offline] Fwd: non-reproducibility study

Paul Mattione pmatt at jlab.org
Fri Mar 14 13:34:25 EDT 2014 

Yes, it's the second one.  Sometimes z0 is 112.78, other times it's 112.36 (with different pz as well).  The diff:

318236c318236
<       <trackFit Ndof="12" chisq="15.8713" e11="0" e12="0" e13="0" e14="0" e15="0" e22="0" e23="0" e24="0" e25="0" e33="0" e34="0" e35="0" e44="0" e45="0" e55="0" lunit="cm" punit="GeV/c" px="-0.0649556" py="0.0811795" pz="0.294878" t0="-999" t0det="0" t0err="-999" tunit="ns" x0="-7.33991" y0="6.01047" z0="112.782" />
---
>       <trackFit Ndof="12" chisq="15.8713" e11="0" e12="0" e13="0" e14="0" e15="0" e22="0" e23="0" e24="0" e25="0" e33="0" e34="0" e35="0" e44="0" e45="0" e55="0" lunit="cm" punit="GeV/c" px="-0.0649556" py="0.0811795" pz="0.296839" t0="-999" t0det="0" t0err="-999" tunit="ns" x0="-7.33991" y0="6.01047" z0="112.36" />

This track is being added to the factory in DTrackCandidate_factory::MatchMethod2().  I'm running a quick test writing DTrackCandidates to REST from the "CDC" factory tag, and then will do so with the "FDCCathodes" tag, to see if the problem is in one of these factories.  If anyone would like to double check this, they could use the attached file and adjust the factory tag as desired.

 - Paul


On Mar 14, 2014, at 1:22 PM, Sean Dobbs wrote:

> Hi Paul,
> 
> I'd like to look at that event in some detail, and am looking at the hd_dump output right now.  Can you confirm what the problem DTrackCandidate is on the list below (i.e. is it the second one?)
> 
> 
> DTrackCandidate:
>  PID:    Name:  q: x(cm): y(cm): z(cm): E(GeV): t(ns): p(GeV/c): theta(deg): phi(deg):         id:     chisq: Ndof:
> -------------------------------------------------------------------------------------------------------------------
>    0  Unknown  +1   -7.1   -6.3  100.1  0.6591  0.000     0.659      15.849  -148.710  0x1b1f6a70  35.961708    10 
>    0  Unknown  +1   -7.3    6.0  112.8  0.3127  0.000     0.313      19.422   128.665  0x11d642a0  15.871287    12 
>    0  Unknown  -1    4.9    8.1   95.3  0.8981  0.000     0.898      22.813    62.311  0x112adea0  10.248627     9 
>    0  Unknown  +1   -6.3   -7.1   75.5  0.7139  0.000     0.714       6.655  -147.993  0x1d7c6050  11.376116     3 
>    0  Unknown  -1    4.6   10.1  167.9  1.0669  0.000     1.067       6.823    79.946  0x1e7a4ca0   3.746347     8 
>    0  Unknown  -1   -7.4  -12.4  167.9  0.5313  0.000     0.531       8.868   -90.505  0x1d29d230  38.236427    12 
>    0  Unknown  -1   -3.7   -1.7  167.9  0.2730  0.000     0.273       2.788   -96.878  0x132281a0   7.396164     6 
>    0  Unknown  +1  -25.4   -1.5  167.9  6.2217  0.000     6.222      13.620   169.034  0x11346220  18.706890     7 
>    0  Unknown  +1   -9.7  -11.0  233.7  2.2539  0.000     2.254       6.024  -147.778  0x1d0cc7d0   8.066071    10 
>    0  Unknown  -1  -19.5   -2.7  292.1  0.0753  0.000     0.075       5.091    70.522  0x1fea48a0  40.620304     6 
>    0  Unknown  -1    4.5    8.8  234.7  0.8029  0.000     0.803       3.772  -151.003  0x1ed4cf60  31.882702     4 
>    0  Unknown  +1  -20.2   -2.8  236.8  1.0370  0.000     1.037       7.027   157.478  0x1835f360  85.737801     4 
> 
> 
> 
> On Fri, Mar 14, 2014 at 10:44 AM, Paul Mattione <pmatt at jlab.org> wrote:
> The difference is for:
> 
> Event #1339, Track Candidate #2, DTrackCandidate pz & z0.
> 
> We need to find which track candidate factory this is coming from still.  
> 
>  - Paul
> 
> On Mar 14, 2014, at 8:44 AM, Beni Zihlmann wrote:
> 
>> Hi Paul,
>> yea! I see a very similar thing. I run single threaded but with two different groups
>> once using hd_root and once using hd_dump both with my plugin. The group
>> using hd_root is consistent in itself and the group with hd_dump agrees with
>> the group of hd_root except for one run! And yes its event 1339!
>> 
>> Event  #Tracks #CDCHits #FDCHits #+tracsk #-tracks #Neutrals
>> ---------------------------------------------------------------------------------------
>> 1336    8     51  169  6  2   11
>> 1337    9    68  491  7  2   7
>> 1338    16  81  817  10  6   6
>> 1339    21  43  1398  11  10   17     >>>>> 1339    21  43  1398  11  10   18
>> 1340    14  42  1067  8  6   16
>> 
>> 
>> different # of Neutrals!
>> 
>> cheers,
>> Beni
>> 
>>> I ran 32 single-threaded jobs with the new software, and I see nearly identical results, but not quite.  Half of the REST files have one identical file size (and identical contents), and the other half have a different identical file size (and identical contents).  This is true whether I run with saving the time-based tracking results or the wire-based tracking results to REST.  The attached "diff" files show the difference for each case (via hddm-xml).  
>>> 
>>> This difference is isolated to one event: #1339 in the file I linked everyone to a week ago.  In fact, while three wire-based and time-based tracks are listed, they all have the same candidate id: 2.  
>>> 
>>> We've got it cornered Simon!  Now let's finish it off! 
>>> 
>>>  - Paul
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mar 13, 2014, at 6:56 PM, Sean Dobbs wrote:
>>> 
>>>> 
>>>> 
>>>> Hi all,
>>>> 
>>>> I checked out and built a clean version with the new tags and am now seeing consistent results when running with one thread and 4 threads.
>>>> 
>>>> 
>>>> ---Sean
>>>> 
>>>> 
>>>> On Thu, Mar 13, 2014 at 2:37 PM, Mark Ito <marki at jlab.org> wrote:
>>>> Still seeing differences of the same ilk as previously reported.
>>>> 
>>>> On 03/13/2014 02:31 PM, Mark Ito wrote:
>>>> > I've re-tagged to reflect this change: tags/sim-recon-2.5 .
>>>> >
>>>> > On 03/13/2014 02:12 PM, Simon Taylor wrote:
>>>> >> I have checked in some changes to the tracking code that appear to
>>>> >> address the valgrind errors mentioned below.
>>>> >>
>>>> >> Simon
>>>> >>
>>>> >> On 03/12/2014 03:59 PM, Matthew Shepherd wrote:
>>>> >>> Having just spent many frustrating hours hunting down my own separate non-deterministic bug I was motivated to run hd_dump -DTrackWireBased through valgrind.
>>>> >>>
>>>> >>> The error below seems suspicious and could result in non-determinstic behaviour, although valgrind is known to generate "errors" where there are none.  I didn't have time to look at the code since I have to run to another meeting, but thought I would pass it on.
>>>> >>>
>>>> >>> Matt
>>>> >>>
>>>> >>>
>>>> >>> ==7443== Conditional jump or move depends on uninitialised value(s)
>>>> >>> ==7443==    at 0x88F769: DTrackFitterKalmanSIMD::KalmanForwardCDC(double, DMatrix5x1&, DMatrix5x5&, double&, unsigned int&) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x8926C4: DTrackFitterKalmanSIMD::ForwardCDCFit(DMatrix5x1 const&, DMatrix5x5 const&) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x89762A: DTrackFitterKalmanSIMD::KalmanLoop() (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x898315: DTrackFitterKalmanSIMD::FitTrack() (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x84B8CB: DTrackFitter::FindHitsAndFitTrack(DKinematicData const&, DReferenceTrajectory const*, jana::JEventLoop*, double, int, double, DetectorSystem_t) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x8C25D7: DTrackWireBased_factory::DoFit(unsigned int, DTrackCandidate const*, DReferenceTrajectory*, jana::JEventLoop*, double) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x8C4803: DTrackWireBased_factory::evnt(jana::JEventLoop*, int) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x6C8F38: jana::JFactory<DTrackWireBased>::Get(std::vector<DTrackWireBased const*, std::allocator<DTrackWireBased const*> >&) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x6C97EC: jana::JFactory<DTrackWireBased>* jana::JEventLoop::GetFromFactory<DTrackWireBased>(std::vector<DTrackWireBased const*, std::allocator<DTrackWireBased const*> >&, char const*, jana::JEventLoop::data_source_t&, bool) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x6C9A84: jana::JFactory<DTrackWireBased>* jana::JEventLoop::Get<DTrackWireBased>(std::vector<DTrackWireBased const*, std::allocator<DTrackWireBased const*> >&, char const*, bool) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x6CA153: jana::JFactory<DTrackWireBased>::GetNrows() (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>> ==7443==    by 0x571A39: MyProcessor::evnt(jana::JEventLoop*, int) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
>>>> >>>
>>>> >>>
>>>> >> _______________________________________________
>>>> >> Halld-offline mailing list
>>>> >> Halld-offline at jlab.org
>>>> >> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>>> 
>>>> --
>>>> Mark M. Ito, Jefferson Lab, marki at jlab.org, (757)269-5295
>>>> 
>>>> _______________________________________________
>>>> Halld-offline mailing list
>>>> Halld-offline at jlab.org
>>>> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Sean Dobbs
>>>> Department of Physics & Astronomy 
>>>> Northwestern University
>>>> phone: 847-467-2826
>>>> _______________________________________________
>>>> Halld-offline mailing list
>>>> Halld-offline at jlab.org
>>>> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Halld-offline mailing list
>>> Halld-offline at jlab.org
>>> https://mailman.jlab.org/mailman/listinfo/halld-offline
>> 
>> _______________________________________________
>> Halld-offline mailing list
>> Halld-offline at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/halld-offline
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140314/2b069cd5/attachment-0002.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DEventWriterREST.cc
Type: application/octet-stream
Size: 17266 bytes
Desc: not available
Url : https://mailman.jlab.org/pipermail/halld-offline/attachments/20140314/2b069cd5/attachment-0001.obj 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140314/2b069cd5/attachment-0003.html

More information about the Halld-offline mailing list