[Halld-offline] subtle bug in ROOT

[David Lawrence]

++ to Dominik and Paul’s answer


> On Feb 25, 2017, at 9:28 AM, Paul Mattione <pmatt at jlab.org> wrote:
> 
> No, he's right.  With floats you only have 23 bits for the significand (1 for sign, 8 for exponent), meaning you only have 6 or 7 significant digits (2^23).  Then you can't increment by one any more. 
> 
>  - Paul 
> 
> On 02/25/2017 09:20 AM, Richard Jones wrote:
>> Dominik,
>> 
>> Such limits exist of course, but this is NOT what I am reporting. The limit of IEEE 32-bit floats is 3.4e+38. TH1F is not using ordinary float-type values. It must be converting the floats to something less than 32 bits (maybe 16 bits?) and trying to save space that way.
>> 
>> -Richard J
>> 
>> On Sat, Feb 25, 2017 at 9:15 AM, Dominik Werthmueller <werthm at jlab.org <mailto:werthm at jlab.org>> wrote:
>> Dear Richard
>> 
>> This issue is not really a bug in ROOT but related to general floating point arithmetics on computers. Basically the difference between two representable numbers increases as their values increase and it will exceed 1 at the limit you mentioned. Hence adding 1 does not change the number anymore.
>> 
>> https://root-forum.cern.ch/t/maximum-histogram-bin-content/5891/2 <https://root-forum.cern.ch/t/maximum-histogram-bin-content/5891/2>
>> https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html <https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html>
>> 
>> Cheers,
>> Dominik
>> 
>> > Am 25.02.2017 um 13:14 schrieb Richard Jones <richard.t.jones at uconn.edu <mailto:richard.t.jones at uconn.edu>>:
>> >
>> > Dear colleagues,
>> >
>> > Maybe this is common knowledge to people more versed in ROOT than I am, but this was a surprise to me, that took me quite some effort to discover why my TTree analysis was producing nonsense.
>> >
>> > >>  TH1F histograms silently truncate their bin contents at 16,777,216 (2^28) <<
>> >
>> > unless you fill with non-unity weights, in which case they truncate at OTHER SMALL VALUES. To be specific, if you create a TH1F histogram h1 and then do h1.Fill(0.) repeatedly then this bin will silently stop incrementing when the bin content reaches 1.677216e+7. If you fill it with a non-unity weight then it will gradually lose precision as the number of Fill calls on that bin exceeds some threshold like a few M, and then silently stop incrementing altogether when the bin content reaches some limit. For w=100 I found this limit to be 2.147e+9. This was unexpected because the letter F in TH1F implies "float" which has a max value about 3.4e+38. What use is a histogram object that silently discards entries as soon as the count reaches some small value that we expect to commonly hit in high-statistics analysis? They must be doing some kind of range-truncating-compression in the storage of TH1F bin contents. Personally, I would rather get the right answer, even if it means using more memory, but that's just me.
>> >
>> > A workaround would be never to use TH1F, always TH1D. I have not been able to discover a similar silent truncation in TH1D. That, plus the fact that TH1F::Fill() is a couple orders of magnitude slower than TH1F::Fill(). Apparently it takes a lot of cpu time to generate bugs of this kind?
>> >
>> > Meanwhile, beware. This is especially insidious because the command tree.Draw("px") in your interactive ROOT session implicitly creates and fills a TH1F, not a TH1D, even if px is declared double in your tree. In my present analysis, my tree has 200M rows, but in principle that will bite you even if you have only 20M rows in your tree.
>> >
>> > -Richard Jones
>> > _______________________________________________
>> > Halld-offline mailing list
>> > Halld-offline at jlab.org <mailto:Halld-offline at jlab.org>
>> > https://mailman.jlab.org/mailman/listinfo/halld-offline <https://mailman.jlab.org/mailman/listinfo/halld-offline>
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Halld-offline mailing list
>> Halld-offline at jlab.org <mailto:Halld-offline at jlab.org>
>> https://mailman.jlab.org/mailman/listinfo/halld-offline <https://mailman.jlab.org/mailman/listinfo/halld-offline>
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20170225/565e899b/attachment-0002.html>