[Halld-offline] Fwd: non-reproducibility study

[Paul Mattione]

I ran 32 single-threaded jobs with the new software, and I see nearly identical results, but not quite.  Half of the REST files have one identical file size (and identical contents), and the other half have a different identical file size (and identical contents).  This is true whether I run with saving the time-based tracking results or the wire-based tracking results to REST.  The attached "diff" files show the difference for each case (via hddm-xml).  

This difference is isolated to one event: #1339 in the file I linked everyone to a week ago.  In fact, while three wire-based and time-based tracks are listed, they all have the same candidate id: 2.  

We've got it cornered Simon!  Now let's finish it off! 

 - Paul





On Mar 13, 2014, at 6:56 PM, Sean Dobbs wrote:

> 
> 
> Hi all,
> 
> I checked out and built a clean version with the new tags and am now seeing consistent results when running with one thread and 4 threads.
> 
> 
> ---Sean
> 
> 
> On Thu, Mar 13, 2014 at 2:37 PM, Mark Ito <marki at jlab.org> wrote:
> Still seeing differences of the same ilk as previously reported.
> 
> On 03/13/2014 02:31 PM, Mark Ito wrote:
> > I've re-tagged to reflect this change: tags/sim-recon-2.5 .
> >
> > On 03/13/2014 02:12 PM, Simon Taylor wrote:
> >> I have checked in some changes to the tracking code that appear to
> >> address the valgrind errors mentioned below.
> >>
> >> Simon
> >>
> >> On 03/12/2014 03:59 PM, Matthew Shepherd wrote:
> >>> Having just spent many frustrating hours hunting down my own separate non-deterministic bug I was motivated to run hd_dump -DTrackWireBased through valgrind.
> >>>
> >>> The error below seems suspicious and could result in non-determinstic behaviour, although valgrind is known to generate "errors" where there are none.  I didn't have time to look at the code since I have to run to another meeting, but thought I would pass it on.
> >>>
> >>> Matt
> >>>
> >>>
> >>> ==7443== Conditional jump or move depends on uninitialised value(s)
> >>> ==7443==    at 0x88F769: DTrackFitterKalmanSIMD::KalmanForwardCDC(double, DMatrix5x1&, DMatrix5x5&, double&, unsigned int&) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x8926C4: DTrackFitterKalmanSIMD::ForwardCDCFit(DMatrix5x1 const&, DMatrix5x5 const&) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x89762A: DTrackFitterKalmanSIMD::KalmanLoop() (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x898315: DTrackFitterKalmanSIMD::FitTrack() (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x84B8CB: DTrackFitter::FindHitsAndFitTrack(DKinematicData const&, DReferenceTrajectory const*, jana::JEventLoop*, double, int, double, DetectorSystem_t) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x8C25D7: DTrackWireBased_factory::DoFit(unsigned int, DTrackCandidate const*, DReferenceTrajectory*, jana::JEventLoop*, double) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x8C4803: DTrackWireBased_factory::evnt(jana::JEventLoop*, int) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x6C8F38: jana::JFactory<DTrackWireBased>::Get(std::vector<DTrackWireBased const*, std::allocator<DTrackWireBased const*> >&) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x6C97EC: jana::JFactory<DTrackWireBased>* jana::JEventLoop::GetFromFactory<DTrackWireBased>(std::vector<DTrackWireBased const*, std::allocator<DTrackWireBased const*> >&, char const*, jana::JEventLoop::data_source_t&, bool) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x6C9A84: jana::JFactory<DTrackWireBased>* jana::JEventLoop::Get<DTrackWireBased>(std::vector<DTrackWireBased const*, std::allocator<DTrackWireBased const*> >&, char const*, bool) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x6CA153: jana::JFactory<DTrackWireBased>::GetNrows() (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>> ==7443==    by 0x571A39: MyProcessor::evnt(jana::JEventLoop*, int) (in /home/fs1/mashephe/gluex/my_src/bin/Linux_CentOS6-x86_64-gcc4.4.6/hd_dump)
> >>>
> >>>
> >> _______________________________________________
> >> Halld-offline mailing list
> >> Halld-offline at jlab.org
> >> https://mailman.jlab.org/mailman/listinfo/halld-offline
> 
> --
> Mark M. Ito, Jefferson Lab, marki at jlab.org, (757)269-5295
> 
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline
> 
> 
> 
> 
> -- 
> Sean Dobbs
> Department of Physics & Astronomy 
> Northwestern University
> phone: 847-467-2826
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140314/04f6b02a/attachment.html 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: time-based.txt
Url: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140314/04f6b02a/attachment.txt 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140314/04f6b02a/attachment-0001.html 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: wire-based.txt
Url: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140314/04f6b02a/attachment-0001.txt 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halld-offline/attachments/20140314/04f6b02a/attachment-0002.html