[Halla12_software] Fun4Spin slides, topNode

[Thomas K Hemmick]

Hi Ole

Thanks for the careful reading.  Comments are interspersed below.

On Wed, Apr 1, 2015 at 12:31 AM, Ole Hansen <ole at jlab.org> wrote:

> Hi Tom,
>
> I finally had some time to browse the slides of the presentation you
> sent last week. It's definitely interesting.
>
> About the "topNode", I can see two reasons why one might want to pass a
> pointer like that to user code. First, on slide 6, there is a comment
> "Fun4All can keep multiple nodes trees" (but "not mainstream yet"). So
> this pointer would then select the appropriate tree. Second, and perhaps
> more importantly, if this is multithreaded code, with each analysis
> thread working on a separate event, there will presumably be one node
> tree per thread, so user code in different threads would need to be
> given different pointers.
>
> It is not multi-threaded, but this is a very interesting idea and would
indeed be most convenient with a "passed" topNode.  The main way in which
we do use more than one node tree (actually it is one with multiple
"co-tops" below the very top) is for so-called "Embedding Simulation".
Here we embed a simulated event (one node tree branch) into a real event
(second node tree branch) making a "merged" event (third node tree branch).

At the system level (Fun4All) is is quite elegant and nice.  The user
implementation below this is kind of a mess and is to this very day leaking
bugs due to the fact that the "digi" stage in the simulation uses fake
reverse simulations while the real event uses actual calibrations.  We *DO*
measure correctly the loss in efficiency for the embedded simulated track
in the presence of hits from a real event.  That said, almost no one has
noticed that the tracks from the real event were not reconstructed with
full efficiency (below 50%!!!) due to the inability of the simulation and
real data to agree on the model of drift in the drift chamber.



> Just a few more questions:
>
> 1) On page 10 an example is given how to retrieve an object from the
> node tree:
>
> PHCentralTrack* cnt =
> findNode::getClass<PHCentralTrack>(topNode,"PHCentralTrack");
>
> I'm puzzled why the getClass method is templated and /also/ given the
> class name as a string. Could the two ever be different? What would that
> mean?
>

The char* name is a name while the <> is the class type.  The best example
of the distinction is for detectors with multiple stations.  In PHENIX
there are three layers of pad chamber (PC1, PC2, and PC3).  The pad chamber
hit container class is the same for all three layers, but these are held as
three instances (with different names) in the node tree.  In that case the
code would say:

PadHit *pc1hits = findNode::getClass<PadHit>(topNode, "PC1Hits");
PadHit *pc2hits = findNode::getClass<PadHit>(topNode, "PC2Hits");
PadHit *pc3hits = findNode::getClass<PadHit>(topNode, "PC3Hits");

In many places, where there is only need for a single instance of a
container, people have named the node using the same string as the class
type, but this is not a requirement.


> 2) The example analysis macro on page 11 does not seem to set up any
> output manager. Where would the output from that example analysis go?
> How would a user select what goes into the output? There is an example
> of how to write histograms to a separate file, and a brief discussion
> that it is possible to add user-defined objects to the output node tree.
> But I assume that DST -> DST processing is not encouraged because, as
> you mentioned, DSTs are big, and bandwidth is limited. What output do
> users typically write? Histograms?
>

Most users write Histograms and NTuples.  NTuples are currently much less
frequent except when organized by Physics Working Groups for two reasons
(they you already alluded to).  First, if a user wants to store even a few
variables per event, from *every* recorded event, they will soon run into
disk space issues (users have 10's of TB each and working group common
areas have 100's of TB each).  Also, data access on "common network disks"
will RAPIDLY hit bandwidth limits when being farmed out to many analysis
nodes.  Once you get upwards of 1000 nodes trying to read the same disk at
the same time over the same network pipe, it does not end well for anyone.
It was absolute HELL trying to teach this to the users in the early days of
PHENIX.  There was even a postdoc who refused to believe us (computing team
and analysis coordinator) and this postdoc stole the accounts of the grad
students he was working with to hide his tracks while analysis for the
whole experiment ground to a halt due to the actions of one guy.

This is the basis of the analysis taxi that I mentioned before.  You should
note that the taxi is partly a computing facility thing (aggregate file
requests and sequence these to optimize tape robot speed) and partly an
experiment thing (choosing what data goes to what node's local disk, and
sequentially sending jobs to the data instead of data to the jobs).
Because of the PHENIX way of doing things (Pinkenburg's Analysis Taxi
upgrade to Hemmick's Analysis Train), PHENIX pushes something like 40X as
much data through the *SAME* computing hardware as STAR manages to do.
Here are last years numbers:

12000 disks distributed among 12000 compute nodes
Each of these 12000 disks saw the passage of 635 TB of data in a year.
Total throughput = 7.6 million TeraBytes per year delivered to user
analysis code.

To the end user, they simply ask for a complete pass of the data by riding
the taxi and then have their histograms ready in three days.  The thousands
of histogram files produced from the thousands of nodes being used in
parallel are automatically accumulated so that the result are found in a
single histogram file with lots of data inside.


> 3) Does the "node tree for real" on slide 6 reflect the contents of a
> typical DST file? Is this tree built "from scratch" from the raw data?
> Is this step (raw data -> DST) what you call "reconstruction"? What
> analysis would users typically do in their Analysis Modules? Some type
> of physics analysis, or more "reconstruction"? Basically, a more
> complete overview of the data flow would be interesting.
>

This will take more time for detail, but basically there is a
"reconstruction pass" that involves a single pass over the full raw
dataset.  After this, users primarily access the DST summaries of the data
riding the taxi.


>
> 4) Where is the PID done?
>

PID is at the user level.


>
> 5) On slide 4, I'm puzzled to see "Simulated PRDF" on the output side. I
> can only guess: are these simulation data generated from the "PISA
> Hits", which in turn come from some kind of simulation engine? So the
> Fun4All server may act as both a reconstruction and a digitization
> framework?
>

Fun4All can read all input file types/formats and it can write all output
types/formats.  Basically, there are only two types of data:  PRDF (Phenix
Raw Data Format) and PHObjects with root streamers.  All classes that run
in and out are set to inherit from a common Object type and Fun4All will
stream whatever it finds into memory and stream whatever it is told (by
node name) to output file.



>
> 6) Slide 7 mentions "reading simulated data together with real data".
> How does one distinguish between the two in the analysis code? Can one
> reconstruct simulated and real data simultaneously? Are reconstructed
> simulated data included in DSTs optionally/routinely?
>

This is mentioned above as "embedding".  PHENIX suffers a ~20% loss in
tracks reconstruction efficiency for the highest multiplicity events.  We
measure this by embedding simulated tracks into real events and asking
whether they do or do not make it to the output stream.


>
> You'll probably cover some or all of this at one of the upcoming
> meetings, but I'm sure it may help preparations to have a list of
> questions beforehand.
>
> Ole
>
> On 03/24/2015 03:45 PM, Thomas K Hemmick wrote:
> > Hi guys
> >
> > Last time there was a request for more info on Fun4All for our next
> > chat.  Chris Pinkenburg (author of Fun4All) sent me these (old) slides
> > that give a little more detail.  I'll also give a presentation during
> > our next meeting, since these slides are a user-oriented presentation
> > while our task is more architect-oriented.
> >
> > The one design feature implemented here that I would change is having
> > the so-called "topNode" is passed around from place-to-place as an
> > argument.  In my opinion, singletons inherently don't need to pass
> > arguments since you can always wake up clueless, grab a pointer to the
> > instance of the singleton, and then learn ANYTHING AND EVERYTHING (not
> > just the selected info passed around via arguments).  Indeed, the
> > current implementation of Fun4All *does* have a function that returns
> > the topNode and I encourage my students as they write layers of code to
> > *NOT* keep passing the pointer to this node around to each level of
> > their own code, but instead get the topNode when & if they need it.
> >
> > Overall that is a pretty short list of complaints (for me).
> >
> > Enjoy!
> > Tom
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mailman.jlab.org/pipermail/halla12_software/attachments/20150401/48a4eb60/attachment-0001.html