[Halld-offline] non-reproducibility study

[Simon Taylor]

On 03/11/2014 05:32 PM, Paul Mattione wrote:
> 3) It looks like everything through wire-based tracking is OK, except DTrackWireBased::t0() is non-deterministic (the rest of the tracking parameters are fine).  This was discovered by saving the DTrackWireBased results to REST instead of DTrackTimeBased in our local test builds, then running hddm-xml and diff on the output (I agree this is the best way to go).

I have checked in a change to DTrackFitterKalmanSIMD that I believe may 
address this issue.  I changed how and where this t0 is computed slightly,
although I do not understand why the previous method caused 
non-deterministic behavior.  Please let me know if the new version helps.

> 4) It looks like the track-BCAL/FCAL/TOF/SC matching is now deterministic (it wasn't before).  However, it looks like there are occasional differences when multithreading.   Perhaps this is due to problems with 3)??
>
> 5) It looks like time-based tracking still has issues, which are worse when running multithreaded.  Perhaps these are related to 3)??
>

I have been looking into the multi-threading issues with valgrind. There 
are some indications that our use of TVector2, TVector3, and TMatrix in 
various parts of the tracking code are not thread safe because of an 
apparent use of global memory by TObject (from which all of the above 
classes inherit) through something called TObjectTable.   I have 
developed my own DVector2 and DVector3 classes to avoid this TObject 
dependency but
it will take more effort to move away from TMatrix if this is what we 
want to do.

Simon