[Hallcsw] Hall C 12 GeV software update (mailing list & request for additional info)
Brad Sawatzky
brads at jlab.org
Tue Aug 23 14:23:42 EDT 2011
Forwarding this thread to the software list. I'd like to get the
discussion off the general SHMS_users list.
----- Forwarded message from Brad Sawatzky <brads at jlab.org> -----
From: Brad Sawatzky <brads at jlab.org>
Subject: Hall C 12 GeV software update (mailing list & request for
additional info)
To: shms_users at jlab.org
Cc: hallcsw at jlab.org
Dear all,
-------- Mailing List for Software Development --------
During the software discussions at the SHMS Detector Working group and
Hall C Summer Workshop I suggested starting a mailing list to provide
a public development forum.
Mark Jones pointed out that there is already a Hall C Software mailing
list in place. I took a look at the archives and it has seen 3 messages
since July 2009. Rather than generating a new "12 GeV" list, I
recommend just using that one. Those interested in development of
online/offline analysis software should sign up using this page:
https://mailman.jlab.org/mailman/listinfo/hallcsw
I also note that a call for software volunteers in the June 2011 SHMS
Users newsletter points to this wiki page:
https://hallcweb.jlab.org/wiki/index.php/Hall_c_soft_main
This would also serve as a good reference/documentation center. Gabi
and John Arrington were tagged as primary contacts for the Software
working group.
-------- Hall C Analyzer Resources --------
These are a few items that were discussed over the last few days. I
would like to encourage "folks in the know" to make a post to the
hallcsw at jlab.org mailing list if they can answer/comment on any of these
topics. Updates to the above wiki would be great too!
- Documentation for FORTRAN analyzer?
https://hallcweb.jlab.org/analysis_documentation/
- Is there anything more current? Does the build procedure
indicated here work with the present RHEL5/64bit mix at JLab?
- Anyone try it with RHEL6 (which is the standard for new builds
now)?
- General Documentation for Hall A analyzer can be found here:
http://hallaweb.jlab.org/podd/
https://hallaweb.jlab.org/wiki/index.php/Analysis_resources_for_d2n#Hall_A_Analyzer_Documentation
- Note that this will _not_ decode Hall C data streams by default.
- Hall C analyzer based on PODD (Hall A analyzer framework)?
- A written note that gave a general status update would be very
much appreciated. I'm not really clear on where things are at.
- Gabi: Do you have a CVS or other source tree for your modified
Hall A analyzer?
-------- Notes/Suggested Outline for Future Progress --------
Many of these items were discussed at the recent meetings and I thought
it might be of value to put them down for general reference. Note that
I don't think we really reached a clear consensus at last week's
meetings, so please consider this my "2 cents" and take it for what it's
worth :-)
Define a few canonical reference CODA runs/dataset that we can use as
a baseline for development. Ideally these should be runs from a
quasi-recent HMS+SOS experiment at a stage where the DAQ was running
smoothly.
- e+P elastic data would be best(?) so we know what we should expect
- reasonable statistics preferred
- A bundle of 'reference' histograms generated by the FORTRAN analyzer
would also be great.
- Some associated documentation on the meta-data needed for decoding
will also be needed. This includes crate, slot, module information
along with ADC and TDC channel map data. A pointer to the FORTRAN
engine's input/config files, along with a little documentation on
the config file format would be sufficient.
- At some point we'll want to push any new analyzer a little harder
with some more difficult-to-analyze runs (ie. high luminosity, large
tracking multiplicity, whatever). Let's start simple though.
FORTRAN Engine:
- Rebuild analyzer on one of the RHEL5/64bit ifarm nodes, replay the
above reference runs and sanity check the output.
- It would be great to add a software switch (or just hack up a
"test" version of the code) to output an event-by-event list of
tracking information (in particular). This would allow for a
quantitative comparison of tracking accuracy and efficiency
between the 'stock' code and any new software.
- Update FORTRAN analyzer to support baseline FADC readout.
- non-blocking, non-pipelined mode; FADCs behave as drop-in
replacement of FB 1881 ADCs
- code should also make provision for TDC data arriving in the
same data block
- note that, just like conventional F1s and CAEN VME TDCs, this
will require a 'reference time' correction to be extracted
from a conventional TDC and applied to the raw data
- can talk to Brad S. and/or DAQ group for format specs, example
code, and CODA files with FADC data
C++ Analyzer:
- Adapt Hall A raw-decode stage to parse Hall C CODA files. (DAQ
folks should know what I mean here.)
- I had the impression that this may already be complete? If so, a
link to the code would be great!
- Incorporate the ADC and TDC channel map config files.
- Simplest approach would be to write a quick translation script to
convert the Hall C legacy config files to Hall A's DB setup.
- Adding "proper" parsers to the Hall A code that understand the
Hall C format wouldn't be too difficult either.
- NOTE: The 'database' backend in the Hall A analyzer has be
long-recognized as a big weakness. Multiple hacks have been
implemented to address this -- none are very satisfying. A
proper solution would greatly benefit both Halls. This would be
a clean, well-defined project for a good student to take on.
- Start comparing low-level output of the C++ analyzer to the
legacy FORTRAN results
- start with raw ADC/TDC comparisons (crate, card, channel), then
"work up" to calibrated output (Energy, Coincidence timing, etc)
- compare track reconstruction event-by-event
- It would be disconcerting if there are significant differences
between what the "Hall A" tracking algorithms generate and what
the FORTRAN analyzer ends up with. I would like to think there
is minimal special case "magic" in either tracking codes.
- compare aggregate tracking output
- efficiencies, momentum distributions, resolutions, etc
-- Brad
--
Brad Sawatzky, PhD <brads at jlab.org> -<>- Jefferson Lab / Hall C / C111
Ph: 757-269-5947 -<>- Fax: 757-269-5235 -<>- Pager: brads-page at jlab.org
The most exciting phrase to hear in science, the one that heralds new
discoveries, is not "Eureka!" but "That's funny..." -- Isaac Asimov
----- End forwarded message -----
----- Forwarded message from Brad Sawatzky <brads at jlab.org> -----
From: Brad Sawatzky <brads at jlab.org>
Subject: Re: [Shms_users] Hall C 12 GeV software update (mailing list &
request for additional info)
To: shms_users at jlab.org
This is Ole Hansen's response. It got bounced from the mailing list for
some reason (we'll get that fixed ASAP).
---------------------
On Mon, 22 Aug 2011, Ole Hansen wrote:
---------------------
Hi folks,
a few quick comments below.
On 8/22/2011 17:30, Brad Sawatzky wrote:
-------- Notes/Suggested Outline for Future Progress --------
C++ Analyzer:
- Adapt Hall A raw-decode stage to parse Hall C CODA files. (DAQ
folks should know what I mean here.)
- I had the impression that this may already be complete? If so, a
link to the code would be great!
I am curious about the differences between Hall A and Hall C CODA files. I would think that the Hall A CODA decoder should at least be able to run through a Hall C data stream, even if it doesn't actually decode module data. The CODA format (event headers, offset pointers, etc.) is well documented, and we are not doing anything non-standard in Hall A.
Whatever differences we discover would be a good starting point for generalizing the decoder logic. There is admittedly quite a bit of hardcoded stuff in there.
Of course, we have most probably not implemented all the Hall C hardware, which will cause certain decoder error messages. That brings me to the weakest point of the decoder: processing of different hardware modules (mostly VME) is coded in the bad old Fortran way - one long string of if-elseifs. This should be completely rewritten in an object-oriented way. Once that is in place, code for new electronics can be loaded dynamically from user-written shared libraries at run time, as we already do with detectors etc.
- NOTE: The 'database' backend in the Hall A analyzer has be
long-recognized as a big weakness. Multiple hacks have been
implemented to address this -- none are very satisfying. A
proper solution would greatly benefit both Halls. This would be
a clean, well-defined project for a good student to take on.
True. One staff member and one postdoc worked on a "real" database solution at different points in time, and neither attempt was successful (i.e. I never saw a line of code checked in). But before someone merrily hacks away, please let's design it first.
- compare track reconstruction event-by-event
- It would be disconcerting if there are significant differences
between what the "Hall A" tracking algorithms generate and what
the FORTRAN analyzer ends up with. I would like to think there
is minimal special case "magic" in either tracking codes.
Keep in mind that the tracking algorithm for the Hall A HRS has been specifically written for our set of VDCs. I don't think you have VDC reconstruction in your Fortran analyzer, or do you? Even if you do, I do think there may very well be "special case magic". I doubt such a comparison would work out of the box. But yes, we did the same thing, we compared Fortran and C++ tracking output in exhaustive detail - after we wrote the C++ tracking code from scratch.
Gabi and I talked at PANIC about having a meeting at JLab sometime in August/September. We should make it a plan. Hopefully, we can use the new old mailing list for further discussions. (I guess I'm waiting for moderator approval.)
Cheers,
Ole
----- End forwarded message -----
----- Forwarded message from Rolf Ent <ent at jlab.org> -----
From: Rolf Ent <ent at jlab.org>
Subject: Re: [Shms_users] Hall C 12 GeV software update (mailing list &
request for additional info)
To: Brad Sawatzky <brads at jlab.org>
cc: shms_users at jlab.org
> - compare track reconstruction event-by-event
> - It would be disconcerting if there are significant differences
> between what the "Hall A" tracking algorithms generate and what
> the FORTRAN analyzer ends up with. I would like to think there
> is minimal special case "magic" in either tracking codes.
>
> Keep in mind that the tracking algorithm for the Hall A HRS has been
> specifically written for our set of VDCs. I don't think you have
> VDC reconstruction in your Fortran analyzer, or do you? Even if you
> do, I do think there may very well be "special case magic". I doubt
> such a comparison would work out of the box. But yes, we did the
> same thing,
> we compared Fortran and C++ tracking output in exhaustive detail - after
> we wrote the C++ tracking code from scratch.
Ole is correct, it is a non-starter to assume the tracking codes
will give similar results given that the Hall A setup uses VDCs. Even
more, the Hall C tracking analysis has been painstakingly optimized with
various "pruning" additions to have a well-known tracking efficiency for
very high rates within the spectrometer, with many bells and whistles how
to check it. Given this, my point of view would be that the starting
point to optimize for the precision L/T program in Hall C should be the
robust Hall C tracking algorithm. Maybe one could argue we should update
the documentation if for instance Eric Christy's tracking note is not
readily accessible, and adding sections from Vladas Tvaskis' thesis.
Don Geesaman did a good job doing the initial documentation.
best regards, Rolf
----- End forwarded message -----
----- Forwarded message from Brad Sawatzky <brads at jlab.org> -----
From: Brad Sawatzky <brads at jlab.org>
Subject: Re: [Shms_users] Hall C 12 GeV software update (mailing list &
request for additional info)
To: shms_users at jlab.org
On Tue, 23 Aug 2011, Rolf Ent wrote:
> > - compare track reconstruction event-by-event
> > - It would be disconcerting if there are significant
> > differences between what the "Hall A" tracking algorithms
> > generate and what the FORTRAN analyzer ends up with. I
> > would like to think there is minimal special case "magic" in
> > either tracking codes.
> >
> >Keep in mind that the tracking algorithm for the Hall A HRS has been
> >specifically written for our set of VDCs. I don't think you have VDC
> >reconstruction in your Fortran analyzer, or do you? Even if you do, I
> >do think there may very well be "special case magic". I doubt such a
> >comparison would work out of the box. But yes, we did the same thing,
> >we compared Fortran and C++ tracking output in exhaustive detail -
> >after we wrote the C++ tracking code from scratch.
>
> Ole is correct, it is a non-starter to assume the tracking codes will
> give similar results given that the Hall A setup uses VDCs. Even more,
> the Hall C tracking analysis has been painstakingly optimized with
> various "pruning" additions to have a well-known tracking efficiency
> for very high rates within the spectrometer, with many bells and
> whistles how to check it. Given this, my point of view would be that
> the starting point to optimize for the precision L/T program in Hall C
> should be the robust Hall C tracking algorithm. Maybe one could argue
> we should update the documentation if for instance Eric Christy's
> tracking note is not readily accessible, and adding sections from
> Vladas Tvaskis' thesis. Don Geesaman did a good job doing the initial
> documentation.
Fair enough. I had been thinking of using the new algorithm Ole first
implemented for BigBite as a starting point, but whatever works. Bottom
line is that we need to make sure any new code matches the output of the
old code (or, at least, that any differences are understood).
It would be very helpful to get pointers/links to any tracking notes and
other docs posted to one or both of these pages, if only to get them in
one place.
https://hallcweb.jlab.org/wiki/index.php/Hall_c_soft_main
https://hallcweb.jlab.org/analysis_documentation/
-- Brad
--
Brad Sawatzky, PhD <brads at jlab.org> -<>- Jefferson Lab / Hall C / C111
Ph: 757-269-5947 -<>- Fax: 757-269-5235 -<>- Pager: brads-page at jlab.org
The most exciting phrase to hear in science, the one that heralds new
discoveries, is not "Eureka!" but "That's funny..." -- Isaac Asimov
----- End forwarded message -----
----- Forwarded message from Eric Christy <meric.christy at gmail.com> -----
From: Eric Christy <meric.christy at gmail.com>
Subject: Re: [Shms_users] Hall C 12 GeV software update (mailing list &
request for additional info)
To: Brad Sawatzky <brads at jlab.org>
Cc: shms_users at jlab.org
On Tue, Aug 23, 2011 at 10:26 AM, Brad Sawatzky <brads at jlab.org> wrote:
> On Tue, 23 Aug 2011, Rolf Ent wrote:
>
> > > - compare track reconstruction event-by-event
> > > - It would be disconcerting if there are significant
> > > differences between what the "Hall A" tracking algorithms
> > > generate and what the FORTRAN analyzer ends up with. I
> > > would like to think there is minimal special case "magic" in
> > > either tracking codes.
> > >
> > >Keep in mind that the tracking algorithm for the Hall A HRS has been
> > >specifically written for our set of VDCs. I don't think you have VDC
> > >reconstruction in your Fortran analyzer, or do you? Even if you do, I
> > >do think there may very well be "special case magic". I doubt such a
> > >comparison would work out of the box. But yes, we did the same thing,
> > >we compared Fortran and C++ tracking output in exhaustive detail -
> > >after we wrote the C++ tracking code from scratch.
> >
> > Ole is correct, it is a non-starter to assume the tracking codes will
> > give similar results given that the Hall A setup uses VDCs. Even more,
> > the Hall C tracking analysis has been painstakingly optimized with
> > various "pruning" additions to have a well-known tracking efficiency
> > for very high rates within the spectrometer, with many bells and
> > whistles how to check it. Given this, my point of view would be that
> > the starting point to optimize for the precision L/T program in Hall C
> > should be the robust Hall C tracking algorithm. Maybe one could argue
> > we should update the documentation if for instance Eric Christy's
> > tracking note is not readily accessible, and adding sections from
> > Vladas Tvaskis' thesis. Don Geesaman did a good job doing the initial
> > documentation.
>
> Fair enough. I had been thinking of using the new algorithm Ole first
> implemented for BigBite as a starting point, but whatever works. Bottom
> line is that we need to make sure any new code matches the output of the
> old code (or, at least, that any differences are understood).
>
> It would be very helpful to get pointers/links to any tracking notes and
> other docs posted to one or both of these pages, if only to get them in
> one place.
> https://hallcweb.jlab.org/wiki/index.php/Hall_c_soft_main
> https://hallcweb.jlab.org/analysis_documentation/
>
>
I think it is well covered now from this discussion that the HRS VDC
algorithms should not be expected to work for the Hall C chambers. I am
sorry that I was not able to attend these discussions during the Hall C
meeting last week. I don't know what the BigBite tracking methodology is,
but I do know that the HMS code has been optimized over the last decade to
work well for high rate precision inclusive cross section measurements,
utilizing information from the basic HMS detector package for selecting the
'best' track at high multiplicities (the pruning method to which Rolf
referred). I had a hand in many of the developments since the base code was
established and I can give a basic chronology. I would not want to have to
valid a new set of algorithms from scratch for L/T experiments, when what we
have should work.
Most of this has been presented in various Hall C workshops, although a
single tech note does not exist for all improvements. The basics of the
'pruning' method and high rate optimization and validation is documented in
the dissertation of Vladas Tvaskis, as mentioned. Peter Bosted worked on
further optimizing the pruning and better integrating parameters tunable
from the parameter files. There is a tech note on optimizing and correctly
determining the HMS tracking efficiency, including an explicit correction
for the software deadtime incurred by the exclusion of hits which occur
during the time that the trigger gate is open. This also includes a pile-up
model based on the intrinsic efficiencies for both 1-track and 2-track
events (which can be determined by cuts on the calorimeter energy), which
describes the determined efficiency quite well. The conclusion from this is
that the efficiencies look to be dominated by a minimal spatial separation
of pile up track hits (mainly in Y). This note was written by Tanja Horn,
Edwin Segbegia, and myself. I believe that Tanja put in on Docdb. I expect
that this spatial pileup should be further minimized by the design of the
SHMS chambers, which have have the U/V planes rotated and a larger angle
relative to the X to provide more information in Y.
-Eric
----- End forwarded message -----
--
Brad Sawatzky, PhD <brads at jlab.org> -<>- Jefferson Lab / Hall C / C111
Ph: 757-269-5947 -<>- Fax: 757-269-5235 -<>- Pager: brads-page at jlab.org
The most exciting phrase to hear in science, the one that heralds new
discoveries, is not "Eureka!" but "That's funny..." -- Isaac Asimov
More information about the Hallcsw
mailing list