[Halld-offline] hddm with fifos (named pipes)

[Richard Jones]

Hello Beni, Paul and all,

At the last offline meeting, I understood that you had seen problems
getting hddm streams to work over named pipes (fifos). I was surprised to
hear this, as I had seen it work back when I first tested the hddm tools,
but I was able to confirm the problem you reported and solve it. The main
problem was related to how the JANA framework handles input files, and I am
not sure that everyone is going to like the way I resolved it, except that
it works. For the details of the problem and how I worked around it, see
the long explanation below. For the shortcut to a solution, just check out
the latest updates to the trunk and rebuild from the top (src) dir.

*A quick test*
See http://zeus.phys.uconn.edu/halld/tagger/fp-microscope/sim-recon-rj-pm/work,
particularly the run scripts therein.

   1. run_bggen.sh
   2. run_hdgeant.sh
   3. run_mcsmear.sh
   4. run_dana_hddm.sh
   5. run_dana_rest.sh

To run my test, create a work directory and copy these 5 scripts from the
above web folder into it, plus the input control.ini file that goes with
them. Then open 5 new xterms on your workstation, go to your work
directory, and run the above scripts in the order listed, one in each
xterm. As you start each one, they will hang as soon as they try to write
to their output fifo until the next script starts up and can read from the
fifo. You can watch the flow of action as it cascades through the chain,
ending up with the final script writing a regular output file in rest
format. None of the intermediate streams are ever written to disk.

*What the main problem was:*
A jana application treats a provided input file as a generic source of
events, and tries to figure out what kind of data it contains, and match it
up with a class derived from JEventSource that can decode the input and
turn it into jana objects that the consumers are asking for. Right now
there are at least 3 different JEventSource classes that I know of:

   1. DEventSourceHDDM - reads hits-level hddm files, like the ones
   generated by mcsmear.
   2. DEventSourceREST - reads REST hddm files, like the ones generated by
   the danarest plugin.
   3. JEventSource_EVIO - reads evio files, like the ones generated by the
   rawevent plugin, and by the experiment.

Before jana can instantiate the correct JEventSource, it has to figure out
what kind of input file it was given. That job is delegated to the various
JEventSourceGenerator classes, which compete with one another for the right
to handle the file. The is a JEventSourceGenerator object for each of the 3
types listed above, and each one gets to try to claim the file by having
its CheckOpenable() method return a probability [0,1] that the file is of
its type. The winner gets to generate the JEventSource that actually opens
the file.

Clearly the best way to figure out what kind a data a file contains is to
read the first few bytes from the file. That is what the existing versions
of the CheckOpenable() method of DEventSourceHDDMGenerator and
DEventSourceRESTGenerator did. This way the correct one always was able to
claim the file. But then, when the JEventSourceGenerator when to make the
actual JEventSource to open the file and read it, if the input file was a
pipe then the header had already been stripped off the stream by the
CheckOpenable() method, and so the headerless hddm stream was not readable.

*My workaround*
If the CheckOpenable() method cannot read bytes from the actual stream
header then it only has the filename to go on, which is not a lot of
information. However the shoot-out scheme in jana tolerates some guessing,
so I implemented code in the revised CheckOpenable() methods that looks for
a handful of clues in the filename and tries to make a best-guess
assignment of what the file is.

   1. Does it contain the string"_rest" or "rest_" and end with ".hddm"? It
   is probably hddm "r" (REST) format.
   2. Does it contain the string "_smeared" or "smeared_" and end with
   ".hddm"? It is probably hddm "s" type hits data.
   3. Does it contain the string "hdgeant" and end with ".hddm"? It is
   probably hddm "s" type hits data.

These heuristics will probably work in most cases. Perhaps the offline
group can come up with a more clear set of conventions to enable jana to
always get the right answer.

-Richard Jones
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20140823/a437af47/attachment-0002.html>