[Halld-offline] [EXTERNAL] logbook on gluex reconstruction reproducibility in jana1

[Richard Jones]

Hello Raiqa and all,

My major effort on understanding and resolving reproducibility issues with
gluex reconstruction under jana 1 was carried out back in the first half of
2021, in the context
of trying to prototype the ability to run raw data reconstruction for Gluex
on the open science grid. Here is a link to my online logbook for this
entire effort,

https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1KNfBbapnycAvlfBYEb9u0795cdtV5YIPDIuucjHcNx4_edit-3Fusp-3Dsharing&d=DwIBaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=Te_hCR4EUlJ6iCDYLJ8Viv2aDOR7D9ZZMoBAvf2H0M4&m=w_imHiEV-InK8x5D8uiWQZYb1r_zvSZoEEF6OYmjfrGMeyLUODqSsAUT6cSfFBvJ&s=XcU1dAJTZagJfpg8J3QLt8S8oXDcug1kM95ywlJ0AFc&e= 

You can look for "reproducibility" or "inconsistency" or "divergence"
keywords to see where this part of the project is discussed. I never
completely finished this, but I got far enough with it to be satisfied that
I could test the osg production framework. The part that is relevant starts
on page 71 with "run-to-run inconsistency", and continues with "take two"
and "divergence, take 2", and then "divergence, take 3". There are a lot of
details there, maybe too dense to be useful to you.

My tools for this work were: (1) valgrind, (2) hddmdiff, and (3) rootdiff
to compare complete sets of output histogram with one single command. To
diagnose divergences in a multithread context, I wrote the dilog divergence
detector class that was a critical tool for tracing differences back to
where they originate in the event processing. If there is anything I can do
to help, eg. set up a tutorial illustrating how to use dilog in a practical
case, please let me know, I want to help get this issue resolved for jana2.

-Richard Jones
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20240422/2d21897d/attachment.html>