[Halld-offline] Software Meeting Minutes, May 28, 2019
Mark Ito
marki at jlab.org
Fri May 31 13:30:36 EDT 2019
Friends,
Please find the minutes below and here
<https://halldweb.jlab.org/wiki/index.php/GlueX_Software_Meeting,_May_28,_2019#Minutes>.
-- Mark
____________________________________________________________
Minutes, GlueX Software Meeting, May 28, 2019
Present:
* *CMU: * Naomi Jarvis
* *FIU: * Mahmoud Kamel, Joerg Reinhold
* *JLab: * Shankar Adhikari, Alexander Austregesilo, Thomas Britton,
Mark Dalton, Sean Dobbs, Mark Ito (chair), David Lawrence, Justin
Stevens, Simon Taylor, Beni Zihlmann
There is a recording of this meeting <https://bluejeans.com/s/QUAf6/> on
the BlueJeans site. Use your JLab credentials to access it.
Announcements
1. New version set consistent with recon-2018_08, version 00
<https://mailman.jlab.org/pipermail/halld-offline/2019-May/003635.html>,
as announced May 8.
2. MCwrapper v2.2.0 released
<https://mailman.jlab.org/pipermail/halld-offline/2019-May/003642.html>.
MCwrapper can now sense viable version sets. The new release uses
XROOTD to stream random triggers. Those files are now kept on the
work disk. Version 2.2.1 is due out soon. The robot will now submit
to the JLab farm when it senses that the OSG is full.
3. PBS to Slurm on June 3rd
<https://mailman.jlab.org/pipermail/jlab-scicomp-briefs/2019q2/000203.html>.
This morning SciComp had a meeting on this. Beni reminded us that
either SWIF or Auger can be used with the new system. Changes under
the new system should be transparent to users, except for the fact
when choosing a custom directory for standard output and error
files, that directory must be made in advance under slurm.
* Added in press: during the meeting, SciComp sent out
[httpd://halldweb.jlab.org/talks/2019/Slurm-rev2.pdf slides from
this morning's presentation].
4. A script for backups of disk files
<https://mailman.jlab.org/pipermail/halld-offline/2019-May/003652.html>.
Mark described the new script. He has been using it to clean-up
recent congestion on the work disk.
Discussion of splitting analysis libraries from halld_recon
Justin referred us to a recent discussion on the offline email list
<https://mailman.jlab.org/pipermail/halld-offline/2019-May/003628.html>
and introduced the topic:
* *Goal*: maintain ability to analyze REST data produced with fixed
halld_recon version using an updated version of "analysis" libraries
currently in halld_recon.
* *Proposal*: extract ANALYSIS, KINFITTER, and PID libraries (and
required plugins and programs) to a separate repo halld_ana.
Some notes on the discussion:
* Fundamental problem: analysis launch data is created from REST data
which was produced with a known version of halld_recon. The analysis
launch itself may be run with a different version of halld_recon to
take advantage of changes made since the reconstruction launch. The
problem is best illustrated in the case of Monte Carlo. There, to
maintain fidelity with real data, one should do the reconstruction
of the simulated data with the reconstruction-launch version of
halld_recon, and do analysis with the analysis-launch version of
halld_recon. We have no mechanism to insure this practice and most
users will not be aware that this is even an issue. The current
version set system has no mechanism to provide two different
versions of a package (halld_recon in this case), and even if it did
the user must know which hd_root is the right one for reconstruction
and which is the right one for analysis. On top of this, it is
possible to do reconstruction and analysis both starting from raw
(or simulated raw) data. In that the case "wrong" version of
reconstruction or analysis will be done by construction.
* The solution we have been discussing before is the one Justin
proposed: break out some of the libraries in a separate package that
can be version controlled separately. Then there is a natural
distinction between reconstruction versions and and analysis
versions based on the tagged version of analysis libraries used.
David quickly recognized that there is fundamental difficulty in
this scheme. Currently the analysis libraries depend on code in the
other parts of halld_recon, and other other parts of halld_recon
depend on the analysis libraries. Since there is no dependency
hierarchy in the code, they cannot be separated into independent
packages, i.e., neither one can be built first (chicken and egg
problem).
* Beni pointed out that if it were the case that analysis could be
done only on REST data, then the necessary dependency hierarchy
would obtain and the analysis packages could be separated. That case
could be enforced by policy. That would break the current feature
where reconstruction and analysis can be done in the same invocation
of hd_root. People were reluctant to give up this feature.
* David thought that the chicken and egg problem could be solved, but
it would require careful separation of routines into reconstruction
libraries, analysis libraries, and common libraries. Such a clean
separation with the roster of routines in the current libraries may
not be possible.
* Justin suggested having two version of halld_recon in the version
sets (version.xml files), one for reconstruction and one for analysis.
* David commented that a system where the analysis libraries were
plug-ins (selected at run-time) that supersede statically linked
versions of the analysis libraries is possible.
* Thomas pointed out that it is possible to modify MCwrapper to switch
from one version set, used for reconstruction, to another, used for
analysis, in a single job. This does not solve the problem of which
reconstruction version goes with which analysis launch, and it does
no good at all for folks not using MCwrapper.
* Another possibility for MCwrapper jobs is to copy the
analysis-compatible version of hd_root along with the job on the OSG
or elsewhere.
* In order to save on disk space with a possibly exploding number of
builds needed (depending on the solution we settle on) Sean
suggested making stripped down version of halld_recon that exclude
the plug-ins and other appropriately chosen files.
* It was noted that having to deal with two complete version sets is
confusing; an independent package solution is easier to understand.
* A method for maintaining the correct correlation of reconstruction
versions and analysis versions should be a part of any solution.
* Alex pointed out the the name of the analysis launch from which ROOT
trees were generated is encoded in each tree file.
* We did not arrive at a solution. Mark encouraged us to think about one.
Review of the previous Software and HDGeant4 meetings
We went over the minutes from the April 30 Software meeting
<https://halldweb.jlab.org/wiki/index.php/GlueX_Software_Meeting,_April_30,_2019#Minutes>
and those from the May 21 HDGeant4 meeting
<https://halldweb.jlab.org/wiki/index.php/HDGeant4_Meeting,_May_21,_2019#Minutes>
without significant comment.
Review of recent issues and pull requests
Two issues were reviewed:
halld_recon: crash in ST matching #71
<https://github.com/JeffersonLab/halld_recon/issues/71>. This problem is
still causing a huge fraction of monitoring launch jobs to crash. Work
continues on this.
halld_recon: Vertex z dependent reconstruction efficiency #166
<https://github.com/JeffersonLab/halld_recon/issues/166>. Alex recently
reported that the vertex position for ρ events shows structure as a
function of z. The effect is seen in HDGeant, HDGeant4, and real data.
This has not been understood yet.
Review of recent discussion on the GlueX Software Help List
We went over message on the help list
<https://groups.google.com/forum/#!forum/gluex-software>.
Matt and Mark's convo on gluex_install
<https://groups.google.com/forum/#!topic/gluex-software/WuKlWUsEydo>
went private at some point, but in the end the problem was resolved.
Retrieved from
"https://halldweb.jlab.org/wiki/index.php?title=GlueX_Software_Meeting,_May_28,_2019&oldid=92819"
* This page was last modified on 31 May 2019, at 13:26.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20190531/50f7d220/attachment.html>
More information about the Halld-offline
mailing list