[Halld-offline] subtle bug in ROOT

[Dominik Werthmueller]

Hi Richard

No problem. In general, your point about different types of histograms and their limits and precisions is actually quite important and can be easily overseen by people (including me).

Cheers,
Dominik

> Am 25.02.2017 um 14:35 schrieb Richard Jones <richard.t.jones at uconn.edu>:
> 
> Dominik,
> 
> You are right, I was wrong to correct you. There is no truncation needed to explain this other than that implied by the 24-bit mantissa of IEEE 32-bit floats. Thank you for clarifying this and setting me straight.
> 
> -Richard Jones
> 
> On Sat, Feb 25, 2017 at 9:15 AM, Dominik Werthmueller <werthm at jlab.org> wrote:
> Dear Richard
> 
> This issue is not really a bug in ROOT but related to general floating point arithmetics on computers. Basically the difference between two representable numbers increases as their values increase and it will exceed 1 at the limit you mentioned. Hence adding 1 does not change the number anymore.
> 
> https://root-forum.cern.ch/t/maximum-histogram-bin-content/5891/2
> https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
> 
> Cheers,
> Dominik
> 
> > Am 25.02.2017 um 13:14 schrieb Richard Jones <richard.t.jones at uconn.edu>:
> >
> > Dear colleagues,
> >
> > Maybe this is common knowledge to people more versed in ROOT than I am, but this was a surprise to me, that took me quite some effort to discover why my TTree analysis was producing nonsense.
> >
> > >>  TH1F histograms silently truncate their bin contents at 16,777,216 (2^28) <<
> >
> > unless you fill with non-unity weights, in which case they truncate at OTHER SMALL VALUES. To be specific, if you create a TH1F histogram h1 and then do h1.Fill(0.) repeatedly then this bin will silently stop incrementing when the bin content reaches 1.677216e+7. If you fill it with a non-unity weight then it will gradually lose precision as the number of Fill calls on that bin exceeds some threshold like a few M, and then silently stop incrementing altogether when the bin content reaches some limit. For w=100 I found this limit to be 2.147e+9. This was unexpected because the letter F in TH1F implies "float" which has a max value about 3.4e+38. What use is a histogram object that silently discards entries as soon as the count reaches some small value that we expect to commonly hit in high-statistics analysis? They must be doing some kind of range-truncating-compression in the storage of TH1F bin contents. Personally, I would rather get the right answer, even if it means using more memory, but that's just me.
> >
> > A workaround would be never to use TH1F, always TH1D. I have not been able to discover a similar silent truncation in TH1D. That, plus the fact that TH1F::Fill() is a couple orders of magnitude slower than TH1F::Fill(). Apparently it takes a lot of cpu time to generate bugs of this kind?
> >
> > Meanwhile, beware. This is especially insidious because the command tree.Draw("px") in your interactive ROOT session implicitly creates and fills a TH1F, not a TH1D, even if px is declared double in your tree. In my present analysis, my tree has 200M rows, but in principle that will bite you even if you have only 20M rows in your tree.
> >
> > -Richard Jones
> > _______________________________________________
> > Halld-offline mailing list
> > Halld-offline at jlab.org
> > https://mailman.jlab.org/mailman/listinfo/halld-offline
> 
>