<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">Alex and Mark,</div>
<div class=""><br class="">
</div>
<div class="">If we can implement these requirements on extra particles prior to trying to reconstruct particular topologies, then I expect we will also dramatically increase the speed of our analysis launches.</div>
<div class=""><br class="">
</div>
<div class="">I've looked at both eta pi+ pi- pi0 p and eta pi+ pi- pi+ pi- p, which have 4 and 2 photons respectively. Both seemed to show similar behavior for photons as Mark reported for tracks. There is marginal gain allowing one extra photon and not
much after that. The eta pi+ pi- pi0 p had some single events with hundreds of combos -- if we can get rid of that kinematic fitting things will be more efficient. I will try to make some slides to illustrate this so there is a record of it.</div>
<div class=""><br class="">
</div>
<div class="">For monitoring reconstruction efficiency one probably wants the analysis step of particular topologies to be as efficient as possible. So these extra requirements may not be desirable in some cases.</div>
<div class=""><br class="">
</div>
<div class="">However, if we're going to start looking for interesting things, we need to lower the threshold to be able to interact with the data. Optimizing the efficiency for something interesting is secondary to finding something interesting to begin with.</div>
<div class=""><br class="">
</div>
<div class="">Matt</div>
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 23, 2019, at 2:34 PM, Mark-Macrae Dalton <<a href="mailto:dalton@jlab.org" class="">dalton@jlab.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi all,
<div class=""><br class="">
</div>
<div class="">I agree with both Alex and Matt. </div>
<div class=""><br class="">
</div>
<div class="">Matt suggests that we have, by default, a very small, flat data set which can be used for many exploratory studies but probably not a final analysis. I think this would be very useful.</div>
<div class="">
<div class=""><br class="">
</div>
</div>
<div class="">Alex suggests that we decrease the size of the trees themselves and I totally agree. We should change the default max number of extra tracks to 1. We should implement the ability to cut on the number of
<b class="">extra</b> photons and choose a more reasonable default. At the moment we allow up to 15 photons per event. As Matt says, there’s no useful data for chi2 above 50—we should cut there in the trees. If somebody wants more than these defaults, they
can easily request it. As we’ve learned with retirement savings, sensible defaults really matter. </div>
<div class=""><br class="">
</div>
<div class="">More than a year ago (with somewhat different software) I found that 2 and 3 extra tracks didn’t change the omega yield while 1 extra track could be up to 20% of the yield at high luminosity. I was not able study the number of extra photons with
that software.</div>
<div class=""><a href="https://logbooks.jlab.org/entry/3545789" class="">https://logbooks.jlab.org/entry/3545789</a></div>
<div class=""><br class="">
</div>
<div class="">Mark</div>
<div class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 18, 2019, at 11:45 AM, Alexander Austregesilo <<a href="mailto:aaustreg@jlab.org" class="">aaustreg@jlab.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Dear Matt,<br class="">
<br class="">
Thank you for your ideas. I just want to add a few alternatives:<br class="">
<br class="">
* Any exploratory analysis can request a smaller number of extra charged tracks, even through the reaction submit form online. The analysis library could also be modified to enable individual cuts on the number of neutrals and a cut on the chi2 in the creation
of the trees, which then could be added to the web form. If you can quantify the loss by applying these cuts, we could even make these cuts the default for the analysis library. Both disk footprint and processing time would benefit from tighter default cuts.<br class="">
<br class="">
* The user can run a DSelector with any number of cuts over the trees once, and write out a much smaller subselection in the same format. This can be easily done on the JLab farm, without the need of copying anything offsite. The produced tree can then be copied
or flattened by the user.<br class="">
<br class="">
* In order to get 'up to speed with the JLab queue and job submission system' as well as the DSelector in general, we organize annual tutorials which provide maintained scripts and examples.<br class="">
<br class="">
Best regards,<br class="">
<br class="">
Alex<br class="">
<br class="">
<br class="">
On 7/17/19 5:03 PM, Shepherd, Matthew wrote:<br class="">
<blockquote type="cite" class="">Hi all,<br class="">
<br class="">
As a follow-on to some earlier discussions at the collaboration meeting and workshop, I'd like to put forth the idea of including a secondary skim or data processing step in our analysis launches, and perhaps, in some cases, discarding the standard ROOT analysis
trees. The goal in this is to take up less disk space and facilitate easy exploratory interaction with the GlueX data.<br class="">
<br class="">
First some comments from recent experience:<br class="">
<br class="">
In order to complement some work that others in our group were doing, I set out to try to analyze the reaction: gamma p -> pi+ pi- pi0 eta p. Specifically, I was interested in gamma p -> ( eta pi- pi0 ) + ( pi+ p ), where the pi+ p was a Delta++. The motivation
is not so relevant for this discussion, but one wants to look at Dalitz plots of the eta pi- pi0 system as a function of eta pi- pi0 mass, etc. There are a lot of things to potentially plot (1D and 2D) and it useful to be able to do this interactively on
the command line wile making cuts interactively. I opted to use the FSRoot package that Ryan wrote (and I demonstrated at the workshop in May).<br class="">
<br class="">
Since I wasn't up to speed on the JLab queue and job submission system, I wanted to test the model of doing "analysis at home" using products of the analysis launch produced at JLab. Fortunately ver27 analysis launch of the 2017 data had what I was looking
for: tree_pi0pippimeta__B3_M17. The ROOT files were 3.5 TB. I had to find a place to park these files and then it took two days to transfer at an average speed of 27 MB/s.<br class="">
<br class="">
While I was waiting for the transfer to complete, I took one file (one run) and began to analyze it for mechanisms to reduce the size. A single run was a 17 GB root file, which I could "flatten" (by throwing away some information I didn't need) into a 8 GB
file. I could then interact with this file with the FSRoot package on the ROOT command line to explore useful skim cuts.<br class="">
<br class="">
As expected, a significant contributor to file size is photon combinatorics. There is a pi0 and an eta in the final state, so there are many combinations of two pairs of photons. Using FSRoot I could easily plot the eta invariant mass for various cuts and
realized that there was little evidence of signal when the number of photons over 100 MeV in an event exceeded 5 (NumNeutralHypos > 5 in analysis tree speak). And looking at the case when there were exactly 4 and exactly 5 suggested diminishing returns for
including 5. Likewise I looked at chi2DOF for various combinations of unused tracks and concluded there was only evidence of signal (clear peak at very low chi2DOF) if the number of extra tracks was 1 or 0.<br class="">
<br class="">
I then used the flatten program (hd_utilities/FlattenForFSRoot), with the following requirements:<br class="">
<br class="">
numNeutralHypos = 4<br class="">
numUnusedTracks <= 1<br class="">
chi2DOF < 50<br class="">
<br class="">
To run flatten on the 3.5 TB of files, I had to submit jobs to our local cluster -- it takes about 30 minutes per file to skim and process, so this took an afternoon. When I was done, I now I had a set of ROOT files that was 105 GB. I went from 3.5 TB to
105 GB by making cuts that largely throw out junk events or poorly fitted hypotheses (only about a factor of two was due to discarding information in going from the analysis tree format to the flat tree format).<br class="">
<br class="">
So, 105 GB is better than 3.5 TB, but still not quite small enough to copy to my laptop and "play" with interactively using ROOT. I made a second step with some more specialized cuts (e.g., proton pi+ invariant mass to isolate the delta, beam energy, etc.)
and I was left with a file that is now less than 10 GB and has really everything I desired to work with that was in the initial 3.5 TB set of files. And using this 10 GB file, I was able make plots on the fly for this topology showing all the substructure
in the eta, pi-, pi0 system that I was interested in. (The second skim step is very easy since FSRoot is distributed with command line tools that let one make rather complicated skim cuts without every having to write any code.)<br class="">
<br class="">
This whole job would have been much easier (and I could have made my desired plots in an hour or so) if I could copy 105 GB instead of 3.5 TB and have it already in a format that facilitates easy interaction.<br class="">
<br class="">
So, here's the proposal:<br class="">
<br class="">
Can we incorporate a standard "flatten" step into the analysis launch framework to beat down the file size a bit and make it easier for portable interactive analysis offsite? I'm envisioning that for each reaction we specify the values for the three cuts noted
above (these are arguments to flatten). And (maybe) we have an option to purge the analysis trees in order to reduce disk usage at JLab. The purge is particularly useful for exploratory analyses.<br class="">
<br class="">
In some cases one may need to go back to the analysis trees for more detailed looks at data or to run specialized algorithms. But in many cases one is trying to survey the landscape and look for low hanging fruit. My biggest concern is that starting with
a DSelector running over analysis trees presents a significant impediment to playing with the data and looking for interesting things (and this ultimately diminishes the discovery potential for GlueX). While it is important to be able to exercise the capability
of the DSelector framework and all the info provided the standard trees, we also need to enable fast and easy browsing of data, especially offsite.<br class="">
<br class="">
I'd be interested in hearing thoughts and ideas -- is there a desire for something like this?<br class="">
<br class="">
Matt<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Halld-offline mailing list<br class="">
<a href="mailto:Halld-offline@jlab.org" class="">Halld-offline@jlab.org</a><br class="">
<a href="https://mailman.jlab.org/mailman/listinfo/halld-offline" class="">https://mailman.jlab.org/mailman/listinfo/halld-offline</a><br class="">
</blockquote>
_______________________________________________<br class="">
Halld-offline mailing list<br class="">
<a href="mailto:Halld-offline@jlab.org" class="">Halld-offline@jlab.org</a><br class="">
<a href="https://mailman.jlab.org/mailman/listinfo/halld-offline" class="">https://mailman.jlab.org/mailman/listinfo/halld-offline</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>