[Halld-offline] hddm with fifos (named pipes)

[David Lawrence]

Hi Richard,

  Thanks for the detailed explanation. I’m still a little confused since it looks like the CheckOpenable code
that actually does a test open the source(=file) was added only 2 years ago. Prior to that, it just checked if the
suffix was “.hddm”. The more sophisticated CheckOpenable was needed with the introduction REST.
My confusion is just that I thought Beni tried using pipes more than 2 years ago and had problems. I guess
all that is important is that you have it working now.

  One thing we could consider is checking if the source is a regular file (and not a named pipe) using the stat()
function. If it is, then the old method of test opening the file could still be used. The name-based algorithm would
only then need to be used for named pipes. This might close off some of the cyber-space where a possible ambiguity 
could arise. Just a thought.

Regards,
-David


On Aug 23, 2014, at 7:55 PM, Richard Jones <richard.t.jones at uconn.edu> wrote:

> Hello Beni, Paul and all,
> 
> At the last offline meeting, I understood that you had seen problems getting hddm streams to work over named pipes (fifos). I was surprised to hear this, as I had seen it work back when I first tested the hddm tools, but I was able to confirm the problem you reported and solve it. The main problem was related to how the JANA framework handles input files, and I am not sure that everyone is going to like the way I resolved it, except that it works. For the details of the problem and how I worked around it, see the long explanation below. For the shortcut to a solution, just check out the latest updates to the trunk and rebuild from the top (src) dir.
> 
> A quick test
> See http://zeus.phys.uconn.edu/halld/tagger/fp-microscope/sim-recon-rj-pm/work, particularly the run scripts therein.
> run_bggen.sh
> run_hdgeant.sh
> run_mcsmear.sh
> run_dana_hddm.sh
> run_dana_rest.sh
> To run my test, create a work directory and copy these 5 scripts from the above web folder into it, plus the input control.ini file that goes with them. Then open 5 new xterms on your workstation, go to your work directory, and run the above scripts in the order listed, one in each xterm. As you start each one, they will hang as soon as they try to write to their output fifo until the next script starts up and can read from the fifo. You can watch the flow of action as it cascades through the chain, ending up with the final script writing a regular output file in rest format. None of the intermediate streams are ever written to disk.
> 
> What the main problem was:
> A jana application treats a provided input file as a generic source of events, and tries to figure out what kind of data it contains, and match it up with a class derived from JEventSource that can decode the input and turn it into jana objects that the consumers are asking for. Right now there are at least 3 different JEventSource classes that I know of:
> DEventSourceHDDM - reads hits-level hddm files, like the ones generated by mcsmear.
> DEventSourceREST - reads REST hddm files, like the ones generated by the danarest plugin.
> JEventSource_EVIO - reads evio files, like the ones generated by the rawevent plugin, and by the experiment.
> Before jana can instantiate the correct JEventSource, it has to figure out what kind of input file it was given. That job is delegated to the various JEventSourceGenerator classes, which compete with one another for the right to handle the file. The is a JEventSourceGenerator object for each of the 3 types listed above, and each one gets to try to claim the file by having its CheckOpenable() method return a probability [0,1] that the file is of its type. The winner gets to generate the JEventSource that actually opens the file.
> 
> Clearly the best way to figure out what kind a data a file contains is to read the first few bytes from the file. That is what the existing versions of the CheckOpenable() method of DEventSourceHDDMGenerator and DEventSourceRESTGenerator did. This way the correct one always was able to claim the file. But then, when the JEventSourceGenerator when to make the actual JEventSource to open the file and read it, if the input file was a pipe then the header had already been stripped off the stream by the CheckOpenable() method, and so the headerless hddm stream was not readable.
> 
> My workaround
> If the CheckOpenable() method cannot read bytes from the actual stream header then it only has the filename to go on, which is not a lot of information. However the shoot-out scheme in jana tolerates some guessing, so I implemented code in the revised CheckOpenable() methods that looks for a handful of clues in the filename and tries to make a best-guess assignment of what the file is.
> Does it contain the string"_rest" or "rest_" and end with ".hddm"? It is probably hddm "r" (REST) format.
> Does it contain the string "_smeared" or "smeared_" and end with ".hddm"? It is probably hddm "s" type hits data.
> Does it contain the string "hdgeant" and end with ".hddm"? It is probably hddm "s" type hits data.
> These heuristics will probably work in most cases. Perhaps the offline group can come up with a more clear set of conventions to enable jana to always get the right answer.
> 
> -Richard Jones
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20140823/c50bdf24/attachment-0002.html>