[Halld-offline] hd_ana hanging after variable number of bggen events

Wed Mar 16 11:48:57 EDT 2011

For reference, I'm pasting below the message I sent to Dave some time ago when I was digging into this... I don't think it contains more info than we have already discussed.

-Matt

Begin forwarded message:

> From: Matthew Shepherd <mashephe at indiana.edu>
> Date: January 13, 2011 11:34:13 AM EST
> To: David Lawrence <davidl at jlab.org>, Simon Taylor <staylor at jlab.org>
> Cc: Kei Moriya <kmoriya at andrew.cmu.edu>, Jake Bennett <jvbennet at indiana.edu>
> Subject: Infinite loop segfault problem
> 
> 
> Hi David and Simon,
> 
> I've started looking more at the infinite loop problem.  I'm totally confused, but I thought I would share my confusion with you -- maybe we can have a group debugging effort.  Be sure to scroll all the way to the bottom of this message.
> 
> I think I've narrowed the problem down to the sort routine in DTrackCandidate_factory_CDC.cc.  It gets called here:
> 
> sort( intersections.begin(), intersections.end(), SortIntersections )
> 
> I've printed out the addresses of begin and end and the elements.  Then I modified SortIntersections to print out the addresses of the items being compared.  Here is some output:
> 
> size of intersections: 42
> 0xd039258: 0
> 0xd03927c: 28.8495
> 0xd0392a0: 28.5845
> 0xd0392c4: 28.7133
> 0xd0392e8: 27.4125
> 0xd03930c: 27.1888
> 0xd039330: 27.3444
> 0xd039354: 27.5862
> 0xd039378: 25.2423
> 0xd03939c: 25.4766
> 0xd0393c0: 25.7988
> 0xd0393e4: 24.029
> 0xd039408: 23.8549
> 0xd03942c: 24.1266
> 0xd039450: 24.4596
> 0xd039474: 22.3323
> 0xd039498: 20.845
> 0xd0394bc: 21.0201
> 0xd0394e0: 19.07
> 0xd039504: 17.5303
> 0xd039528: 17.7948
> 0xd03954c: 48.4739
> 0xd039570: 48.4739
> 0xd039594: 48.303
> 0xd0395b8: 48.303
> 0xd0395dc: 46.9341
> 0xd039600: 46.802
> 0xd039624: 45.1266
> 0xd039648: 45.1266
> 0xd03966c: 45.039
> 0xd039690: 43.7441
> 0xd0396b4: 43.6517
> 0xd0396d8: 43.6517
> 0xd0396fc: 41.9293
> 0xd039720: 41.2372
> 0xd039744: 40.5346
> 0xd039768: 40.5711
> 0xd03978c: 39.4966
> 0xd0397b0: 38.8455
> 0xd0397d4: 38.9204
> 0xd0397f8: 38.084
> 0xd03981c: 37.5229
> now calling sort:  0xd039258 to 0xd039840     <<<----- here is the beginning and end of the array
> a: 0xd039258 Ra: 0 b: 0xd03954c
> a: 0xd03954c Ra: 48.4739 b: 0xd03981c
> a: 0xd039258 Ra: 0 b: 0xd03981c
> a: 0xd039258 Ra: 0 b: 0xb73fd898
> a: 0xd03927c Ra: 28.8495 b: 0xb73fd898
> a: 0xd0392a0 Ra: 28.5845 b: 0xb73fd898
> a: 0xd0392c4 Ra: 28.7133 b: 0xb73fd898
> a: 0xd0392e8 Ra: 27.4125 b: 0xb73fd898
> a: 0xd03930c Ra: 27.1888 b: 0xb73fd898
> a: 0xd039330 Ra: 27.3444 b: 0xb73fd898
> a: 0xd039354 Ra: 27.5862 b: 0xb73fd898
> a: 0xd039378 Ra: 25.2423 b: 0xb73fd898
> a: 0xd03939c Ra: 25.4766 b: 0xb73fd898
> a: 0xd0393c0 Ra: 25.7988 b: 0xb73fd898
> a: 0xd0393e4 Ra: 24.029 b: 0xb73fd898
> a: 0xd039408 Ra: 23.8549 b: 0xb73fd898
> a: 0xd03942c Ra: 24.1266 b: 0xb73fd898
> a: 0xd039450 Ra: 24.4596 b: 0xb73fd898
> a: 0xd039474 Ra: 22.3323 b: 0xb73fd898
> a: 0xd039498 Ra: 20.845 b: 0xb73fd898
> a: 0xd0394bc Ra: 21.0201 b: 0xb73fd898
> a: 0xd0394e0 Ra: 19.07 b: 0xb73fd898
> a: 0xd039504 Ra: 17.5303 b: 0xb73fd898
> a: 0xd039528 Ra: 17.7948 b: 0xb73fd898
> a: 0xd03954c Ra: 48.4739 b: 0xb73fd898
> a: 0xb73fd898 Ra: 37.5229 b: 0xd03981c
> a: 0xd039570 Ra: 48.4739 b: 0xb73fd898
> a: 0xb73fd898 Ra: 37.5229 b: 0xd0397f8
> a: 0xb73fd898 Ra: 37.5229 b: 0xd0397d4
> a: 0xb73fd898 Ra: 37.5229 b: 0xd0397b0
> a: 0xb73fd898 Ra: 37.5229 b: 0xd03978c
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039768
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039744
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039720
> a: 0xb73fd898 Ra: 37.5229 b: 0xd0396fc
> a: 0xb73fd898 Ra: 37.5229 b: 0xd0396d8
> a: 0xb73fd898 Ra: 37.5229 b: 0xd0396b4
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039690
> a: 0xb73fd898 Ra: 37.5229 b: 0xd03966c
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039648
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039624
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039600
> a: 0xb73fd898 Ra: 37.5229 b: 0xd0395dc
> a: 0xb73fd898 Ra: 37.5229 b: 0xd0395b8
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039594
> a: 0xb73fd898 Ra: 37.5229 b: 0xd039570
> a: 0xb73fd898 Ra: 37.5229 b: 0xd03954c
> a: 0xd039570 Ra: 48.4739 b: 0xd0396d8
> a: 0xd039570 Ra: 48.4739 b: 0xd03981c
> a: 0xd039570 Ra: 48.4739 b: 0xb73fd828
> a: 0xd039594 Ra: 48.303 b: 0xb73fd828
> a: 0xd0395b8 Ra: 48.303 b: 0xb73fd828
> a: 0xd0395dc Ra: 46.9341 b: 0xb73fd828
> a: 0xd039600 Ra: 46.802 b: 0xb73fd828
> a: 0xd039624 Ra: 45.1266 b: 0xb73fd828
> a: 0xd039648 Ra: 45.1266 b: 0xb73fd828
> a: 0xd03966c Ra: 45.039 b: 0xb73fd828
> a: 0xd039690 Ra: 43.7441 b: 0xb73fd828
> a: 0xd0396b4 Ra: 43.6517 b: 0xb73fd828
> a: 0xd0396d8 Ra: 43.6517 b: 0xb73fd828
> a: 0xd0396fc Ra: 41.9293 b: 0xb73fd828
> a: 0xd039720 Ra: 41.2372 b: 0xb73fd828
> a: 0xd039744 Ra: 40.5346 b: 0xb73fd828
> a: 0xd039768 Ra: 40.5711 b: 0xb73fd828
> a: 0xd03978c Ra: 39.4966 b: 0xb73fd828
> a: 0xd0397b0 Ra: 38.8455 b: 0xb73fd828
> a: 0xd0397d4 Ra: 38.9204 b: 0xb73fd828
> a: 0xd0397f8 Ra: 38.084 b: 0xb73fd828
> a: 0xd03981c Ra: 48.4739 b: 0xb73fd828
> a: 0xd039840 Ra: 0 b: 0xb73fd828                <<<<---- comparing against the element at end -- should never happen
> a: 0xd039864 Ra: 0 b: 0xb73fd828                 <<<<---- now blown past the end and keeps going
> a: 0xd039888 Ra: 0 b: 0xb73fd828
> a: 0xd0398ac Ra: 0 b: 0xb73fd828
> a: 0xd0398d0 Ra: 0 b: 0xb73fd828
> a: 0xd0398f4 Ra: 0 b: 0xb73fd828
> a: 0xd039918 Ra: 0 b: 0xb73fd828
> a: 0xd03993c Ra: 0 b: 0xb73fd828
> a: 0xd039960 Ra: 0 b: 0xb73fd828
> a: 0xd039984 Ra: 0 b: 0xb73fd828
> 
> 
> eventually segfault....    If I print Rb in the SortIntersections method then the crash does not happen.  Here is the print.  Of course the addresses are not the same, but you can get the idea.  I'm sure sort makes a temporary variable to copy into as it is sorting.  You seen below and above there are two well-separated groups of addresses.
> 
> 
> size of intersections: 42
> 0xc3b3810: 0
> 0xc3b3834: 28.8495
> 0xc3b3858: 28.5845
> 0xc3b387c: 28.7133
> 0xc3b38a0: 27.4125
> 0xc3b38c4: 27.1888
> 0xc3b38e8: 27.3444
> 0xc3b390c: 27.5862
> 0xc3b3930: 25.2423
> 0xc3b3954: 25.4766
> 0xc3b3978: 25.7988
> 0xc3b399c: 24.029
> 0xc3b39c0: 23.8549
> 0xc3b39e4: 24.1266
> 0xc3b3a08: 24.4596
> 0xc3b3a2c: 22.3323
> 0xc3b3a50: 20.845
> 0xc3b3a74: 21.0201
> 0xc3b3a98: 19.07
> 0xc3b3abc: 17.5303
> 0xc3b3ae0: 17.7948
> 0xc3b3b04: 48.4739
> 0xc3b3b28: 48.4739
> 0xc3b3b4c: 48.303
> 0xc3b3b70: 48.303
> 0xc3b3b94: 46.9341
> 0xc3b3bb8: 46.802
> 0xc3b3bdc: 45.1266
> 0xc3b3c00: 45.1266
> 0xc3b3c24: 45.039
> 0xc3b3c48: 43.7441
> 0xc3b3c6c: 43.6517
> 0xc3b3c90: 43.6517
> 0xc3b3cb4: 41.9293
> 0xc3b3cd8: 41.2372
> 0xc3b3cfc: 40.5346
> 0xc3b3d20: 40.5711
> 0xc3b3d44: 39.4966
> 0xc3b3d68: 38.8455
> 0xc3b3d8c: 38.9204
> 0xc3b3db0: 38.084
> 0xc3b3dd4: 37.5229
> now calling sort:  0xc3b3810 to 0xc3b3df8
> a: 0xc3b3810 Ra: 0 b: 0xc3b3b04 Rb: 48.4739
> a: 0xc3b3b04 Ra: 48.4739 b: 0xc3b3dd4 Rb: 37.5229
> a: 0xc3b3810 Ra: 0 b: 0xc3b3dd4 Rb: 37.5229
> a: 0xc3b3810 Ra: 0 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3834 Ra: 28.8495 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3858 Ra: 28.5845 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b387c Ra: 28.7133 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b38a0 Ra: 27.4125 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b38c4 Ra: 27.1888 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b38e8 Ra: 27.3444 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b390c Ra: 27.5862 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3930 Ra: 25.2423 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3954 Ra: 25.4766 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3978 Ra: 25.7988 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b399c Ra: 24.029 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b39c0 Ra: 23.8549 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b39e4 Ra: 24.1266 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3a08 Ra: 24.4596 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3a2c Ra: 22.3323 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3a50 Ra: 20.845 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3a74 Ra: 21.0201 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3a98 Ra: 19.07 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3abc Ra: 17.5303 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3ae0 Ra: 17.7948 b: 0xb759f898 Rb: 37.5229
> a: 0xc3b3b04 Ra: 48.4739 b: 0xb759f898 Rb: 37.5229
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3dd4 Rb: 37.5229
> a: 0xc3b3b28 Ra: 48.4739 b: 0xb759f898 Rb: 37.5229
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3db0 Rb: 38.084
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3d8c Rb: 38.9204
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3d68 Rb: 38.8455
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3d44 Rb: 39.4966
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3d20 Rb: 40.5711
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3cfc Rb: 40.5346
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3cd8 Rb: 41.2372
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3cb4 Rb: 41.9293
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3c90 Rb: 43.6517
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3c6c Rb: 43.6517
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3c48 Rb: 43.7441
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3c24 Rb: 45.039
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3c00 Rb: 45.1266
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3bdc Rb: 45.1266
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3bb8 Rb: 46.802
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3b94 Rb: 46.9341
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3b70 Rb: 48.303
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3b4c Rb: 48.303
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3b28 Rb: 48.4739
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3b04 Rb: 37.5229
> a: 0xc3b3b28 Ra: 48.4739 b: 0xc3b3c90 Rb: 43.6517
> a: 0xc3b3b28 Ra: 48.4739 b: 0xc3b3dd4 Rb: 48.4739
> a: 0xc3b3c90 Ra: 43.6517 b: 0xc3b3dd4 Rb: 48.4739
> a: 0xc3b3b28 Ra: 48.4739 b: 0xb759f828 Rb: 48.4739    
> a: 0xb759f828 Ra: 48.4739 b: 0xc3b3dd4 Rb: 48.4739   
> a: 0xc3b3b4c Ra: 48.303 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3b70 Ra: 48.303 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3b94 Ra: 46.9341 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3bb8 Ra: 46.802 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3bdc Ra: 45.1266 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3c00 Ra: 45.1266 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3c24 Ra: 45.039 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3c48 Ra: 43.7441 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3c6c Ra: 43.6517 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3c90 Ra: 43.6517 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3cb4 Ra: 41.9293 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3cd8 Ra: 41.2372 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3cfc Ra: 40.5346 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3d20 Ra: 40.5711 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3d44 Ra: 39.4966 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3d68 Ra: 38.8455 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3d8c Ra: 38.9204 b: 0xb759f828 Rb: 48.4739
> a: 0xc3b3db0 Ra: 38.084 b: 0xb759f828 Rb: 48.4739  <<--- this was the last good point in the other run of the program
> a: 0xc3b3dd4 Ra: 48.4739 b: 0xb759f828 Rb: 48.4739  <<--- here it looks like value has been copied into the last element 
> a: 0xb759f828 Ra: 48.4739 b: 0xc3b3db0 Rb: 38.084
> a: 0xc3b3b28 Ra: 48.4739 b: 0xc3b3c6c Rb: 43.6517
> a: 0xc3b3b28 Ra: 48.4739 b: 0xc3b3db0 Rb: 38.084
> a: 0xc3b3c6c Ra: 43.6517 b: 0xc3b3db0 Rb: 38.084
> a: 0xc3b3b28 Ra: 48.4739 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3db0 Rb: 38.084
> a: 0xc3b3b4c Ra: 48.303 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3d8c Rb: 38.9204
> a: 0xc3b3b70 Ra: 48.303 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3d68 Rb: 38.8455
> a: 0xc3b3b94 Ra: 46.9341 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3d44 Rb: 39.4966
> a: 0xc3b3bb8 Ra: 46.802 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3d20 Rb: 40.5711
> a: 0xc3b3bdc Ra: 45.1266 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3cfc Rb: 40.5346
> a: 0xc3b3c00 Ra: 45.1266 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3cd8 Rb: 41.2372
> a: 0xc3b3c24 Ra: 45.039 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3cb4 Rb: 41.9293
> a: 0xc3b3c48 Ra: 43.7441 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3c90 Rb: 43.6517
> a: 0xc3b3c6c Ra: 43.6517 b: 0xb759f828 Rb: 43.6517
> a: 0xb759f828 Ra: 43.6517 b: 0xc3b3c6c Rb: 43.6517
> a: 0xc3b3810 Ra: 0 b: 0xc3b399c Rb: 24.029
> a: 0xc3b399c Ra: 24.029 b: 0xc3b3b04 Rb: 37.5229
> a: 0xc3b3810 Ra: 0 b: 0xb759f898 Rb: 24.029
> a: 0xc3b3834 Ra: 28.8495 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3b04 Rb: 37.5229
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3ae0 Rb: 17.7948
> a: 0xc3b3858 Ra: 28.5845 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3abc Rb: 17.5303
> a: 0xc3b387c Ra: 28.7133 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3a98 Rb: 19.07
> a: 0xc3b38a0 Ra: 27.4125 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3a74 Rb: 21.0201
> a: 0xc3b38c4 Ra: 27.1888 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3a50 Rb: 20.845
> a: 0xc3b38e8 Ra: 27.3444 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3a2c Rb: 22.3323
> a: 0xc3b390c Ra: 27.5862 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3a08 Rb: 24.4596
> a: 0xb759f898 Ra: 24.029 b: 0xc3b39e4 Rb: 24.1266
> a: 0xb759f898 Ra: 24.029 b: 0xc3b39c0 Rb: 23.8549
> a: 0xc3b3930 Ra: 25.2423 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b399c Rb: 24.029
> a: 0xc3b3954 Ra: 25.4766 b: 0xb759f898 Rb: 24.029
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3978 Rb: 25.7988
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3954 Rb: 25.4766
> a: 0xb759f898 Ra: 24.029 b: 0xc3b3930 Rb: 24.029
> a: 0xb759f814 Ra: 17.7948 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 17.7948 b: 0xc3b3810 Rb: 0
> a: 0xb759f814 Ra: 17.5303 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 17.5303 b: 0xc3b3834 Rb: 17.7948
> a: 0xb759f838 Ra: 17.5303 b: 0xc3b3810 Rb: 0
> a: 0xb759f814 Ra: 19.07 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 19.07 b: 0xc3b3858 Rb: 17.7948
> a: 0xb759f814 Ra: 21.0201 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 21.0201 b: 0xc3b387c Rb: 19.07
> a: 0xb759f814 Ra: 20.845 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 20.845 b: 0xc3b38a0 Rb: 21.0201
> a: 0xb759f838 Ra: 20.845 b: 0xc3b387c Rb: 19.07
> a: 0xb759f814 Ra: 22.3323 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 22.3323 b: 0xc3b38c4 Rb: 21.0201
> a: 0xb759f814 Ra: 23.8549 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 23.8549 b: 0xc3b38e8 Rb: 22.3323
> a: 0xb759f814 Ra: 24.029 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 24.029 b: 0xc3b390c Rb: 23.8549
> a: 0xb759f814 Ra: 25.4766 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 25.4766 b: 0xc3b3930 Rb: 24.029
> a: 0xb759f814 Ra: 25.7988 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 25.7988 b: 0xc3b3954 Rb: 25.4766
> a: 0xb759f814 Ra: 25.2423 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 25.2423 b: 0xc3b3978 Rb: 25.7988
> a: 0xb759f838 Ra: 25.2423 b: 0xc3b3954 Rb: 25.4766
> a: 0xb759f838 Ra: 25.2423 b: 0xc3b3930 Rb: 24.029
> a: 0xb759f814 Ra: 27.5862 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 27.5862 b: 0xc3b399c Rb: 25.7988
> a: 0xb759f814 Ra: 24.1266 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 24.1266 b: 0xc3b39c0 Rb: 27.5862
> a: 0xb759f838 Ra: 24.1266 b: 0xc3b399c Rb: 25.7988
> a: 0xb759f838 Ra: 24.1266 b: 0xc3b3978 Rb: 25.4766
> a: 0xb759f838 Ra: 24.1266 b: 0xc3b3954 Rb: 25.2423
> a: 0xb759f838 Ra: 24.1266 b: 0xc3b3930 Rb: 24.029
> a: 0xb759f814 Ra: 24.4596 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 24.4596 b: 0xc3b39e4 Rb: 27.5862
> a: 0xb759f838 Ra: 24.4596 b: 0xc3b39c0 Rb: 25.7988
> a: 0xb759f838 Ra: 24.4596 b: 0xc3b399c Rb: 25.4766
> a: 0xb759f838 Ra: 24.4596 b: 0xc3b3978 Rb: 25.2423
> a: 0xb759f838 Ra: 24.4596 b: 0xc3b3954 Rb: 24.1266
> a: 0xb759f814 Ra: 27.3444 b: 0xc3b3810 Rb: 0
> a: 0xb759f838 Ra: 27.3444 b: 0xc3b3a08 Rb: 27.5862
> a: 0xb759f838 Ra: 27.3444 b: 0xc3b39e4 Rb: 25.7988
> a: 0xb759f898 Ra: 27.1888 b: 0xc3b3a2c Rb: 27.5862
> a: 0xb759f898 Ra: 27.1888 b: 0xc3b3a08 Rb: 27.3444
> a: 0xb759f898 Ra: 27.1888 b: 0xc3b39e4 Rb: 25.7988
> a: 0xb759f898 Ra: 27.4125 b: 0xc3b3a50 Rb: 27.5862
> a: 0xb759f898 Ra: 27.4125 b: 0xc3b3a2c Rb: 27.3444
> a: 0xb759f898 Ra: 28.7133 b: 0xc3b3a74 Rb: 27.5862
> a: 0xb759f898 Ra: 28.5845 b: 0xc3b3a98 Rb: 28.7133
> a: 0xb759f898 Ra: 28.5845 b: 0xc3b3a74 Rb: 27.5862
> a: 0xb759f898 Ra: 28.8495 b: 0xc3b3abc Rb: 28.7133
> a: 0xb759f898 Ra: 37.5229 b: 0xc3b3ae0 Rb: 28.8495
> a: 0xb759f898 Ra: 38.084 b: 0xc3b3b04 Rb: 37.5229
> a: 0xb759f898 Ra: 38.9204 b: 0xc3b3b28 Rb: 38.084
> a: 0xb759f898 Ra: 38.8455 b: 0xc3b3b4c Rb: 38.9204
> a: 0xb759f898 Ra: 38.8455 b: 0xc3b3b28 Rb: 38.084
> a: 0xb759f898 Ra: 39.4966 b: 0xc3b3b70 Rb: 38.9204
> a: 0xb759f898 Ra: 40.5711 b: 0xc3b3b94 Rb: 39.4966
> a: 0xb759f898 Ra: 40.5346 b: 0xc3b3bb8 Rb: 40.5711
> a: 0xb759f898 Ra: 40.5346 b: 0xc3b3b94 Rb: 39.4966
> a: 0xb759f898 Ra: 41.2372 b: 0xc3b3bdc Rb: 40.5711
> a: 0xb759f898 Ra: 41.9293 b: 0xc3b3c00 Rb: 41.2372
> a: 0xb759f898 Ra: 43.6517 b: 0xc3b3c24 Rb: 41.9293
> a: 0xb759f898 Ra: 43.6517 b: 0xc3b3c48 Rb: 43.6517
> a: 0xb759f898 Ra: 43.7441 b: 0xc3b3c6c Rb: 43.6517
> a: 0xb759f898 Ra: 45.039 b: 0xc3b3c90 Rb: 43.7441
> a: 0xb759f898 Ra: 45.1266 b: 0xc3b3cb4 Rb: 45.039
> a: 0xb759f898 Ra: 45.1266 b: 0xc3b3cd8 Rb: 45.1266
> a: 0xb759f898 Ra: 46.802 b: 0xc3b3cfc Rb: 45.1266
> a: 0xb759f898 Ra: 46.9341 b: 0xc3b3d20 Rb: 46.802
> a: 0xb759f898 Ra: 48.303 b: 0xc3b3d44 Rb: 46.9341
> a: 0xb759f898 Ra: 48.303 b: 0xc3b3d68 Rb: 48.303
> a: 0xb759f898 Ra: 48.4739 b: 0xc3b3d8c Rb: 48.303
> a: 0xb759f898 Ra: 48.4739 b: 0xc3b3db0 Rb: 48.4739
> 
> This could explain some of the behavior.  If sort keeps cruising through memory without ever stopping at the end, that could be like an runway "infinite loop" behavior.  It it stumbles on some bad memory that it can't dereference, then a segfault will happen.  I still have no idea what is causing this, but maybe a fresh pair of eyes would help.
> 
> -Matt
> 
> 