[Halld-offline] [EXTERNAL] Re: heads-up: pull requests require review for halld_recon and halld_sim

Thu Apr 30 16:07:24 EDT 2020

These tests are still run on every pull request, but they only check
to see if the programs/plugins run without crashing.  They could be
built out to have more comparisons - I agree we want numerical
comparisons, not plots.  Maybe a project for an enterprising grad
student?

Also, a couple years ago Thomas worked on putting together a nice
dashboard for a similar sort of analysis for the automated tests we
ran.  The hope was to add some other periodic tests as well, but this
also got buried under more pressing issues...

---Sean

On Thu, Apr 30, 2020 at 1:15 PM Mark Ito <marki at jlab.org> wrote:
>
> We used to have some very simple tests, set up by Nathan Sparks, that at least checked that the code would run. He worked with Sean to run tests on every pull request. But that went by the wayside at some point. It's on the list to revive that, but admittedly pretty darn far down.
>
> On 4/30/20 10:00 AM, Naomi Jarvis wrote:
>
> We could use more automated testing.  It would be nice to have a build+run+compare test instead of just a build test.
>
> in my previous job I wrote the back-end software for this https://urldefense.proofpoint.com/v2/url?u=https-3A__www.phe-2Dprotectionservices.org.uk_imba_&d=DwIFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=Te_hCR4EUlJ6iCDYLJ8Viv2aDOR7D9ZZMoBAvf2H0M4&m=GK0lBCMKmzgyX_qXbere5id2N74O97-6LAd7SecvGBE&s=iZwt-4Ya4kVLMy_cRgAiebcqt6T4rhllznhiQa9rBp8&e=    Eventually I wrote an automated test facility for myself to use to compare results between old and new versions using many strategically picked scenarios, to check that the changes did what they were supposed to and nothing else.  Writing the test facility was a discouraged diversion initially but well worth it a few years later.
>
> GlueX could use something like this, at least to compare tagged releases.  I am not suggesting checking histograms by eye on a website, I would prefer an automatic numerical comparison which flags differences and % differences, for many plugins, many histograms.
>
> Naomi.
>
>
>
> On Wed, Apr 29, 2020 at 9:56 PM David Lawrence <davidl at jlab.org> wrote:
>>
>> I just meant the analogy doesn’t seem complete since lots of people do actually download and build
>> the Hall-D software, unlike a data analysis.
>>
>> Regards,
>> -David
>>
>> -------------------------------------------------------------
>> David Lawrence Ph.D.
>> Staff Scientist, Thomas Jefferson National Accelerator Facility
>> Newport News, VA
>> davidl at jlab.org
>> (757) 269-5567 W
>> (757) 746-6697 C
>>
>>
>> On Apr 29, 2020, at 1:47 PM, Richard Jones <richard.t.jones at uconn.edu> wrote:
>>
>> @David, this is already automated, I thought.
>> -rtj
>>
>> On Wed, Apr 29, 2020 at 1:40 PM David Lawrence <davidl at jlab.org> wrote:
>>>
>>>
>>> *Message sent from a system outside of UConn.*
>>>
>>>
>>>
>>> The reviewer though will likely never need to download the data and run the analysis whereas someone elese will need to compile and run the code.
>>> Just a thought.
>>> Regards,
>>> -David
>>>
>>> -------------------------------------------------------------
>>> David Lawrence Ph.D.
>>> Staff Scientist, Thomas Jefferson National Accelerator Facility
>>> Newport News, VA
>>> davidl at jlab.org
>>> (757) 269-5567 W
>>> (757) 746-6697 C
>>>
>>>
>>> On Apr 29, 2020, at 10:32 AM, Richard Jones <richard.t.jones at uconn.edu> wrote:
>>>
>>> Or how about just emailing the author and asking, have you done this? Or can you do this one more test, and I will approve it. Something like the way peer review of articles works. Usually the reviewer does not download the data and carry out independent checks of the results.
>>>
>>> -rtj
>>>
>>> On Wed, Apr 29, 2020 at 9:37 AM Naomi Jarvis <nsj at cmu.edu> wrote:
>>>>
>>>>
>>>> *Message sent from a system outside of UConn.*
>>>>
>>>>
>>>> I thought the review process would be something like download the new branch, make sure it compiles, make sure it runs, make sure it does whatever it was supposed to do, then go to the website and click the merge button (and presumably one more now).  That's what I do before approving PRs (& why I don't approve many).
>>>>
>>>> Naomi.
>>>>
>>>> On Wed, Apr 29, 2020 at 9:00 AM Sean Dobbs <sdobbs at fsu.edu> wrote:
>>>>>
>>>>> We could just start with a short comment like “looks good” for now, and then revisit the question next software meeting when more people are familiar with what functionality is available.
>>>>>
>>>>> Cheers
>>>>> Sean
>>>>>
>>>>> On Wed, Apr 29, 2020 at 8:46 AM Mark Ito <marki at jlab.org> wrote:
>>>>>>
>>>>>> Mark,
>>>>>>
>>>>>> That is a good point, well taken. Anyone who was at the meeting
>>>>>> yesterday care to take this one?
>>>>>>
>>>>>>    -- Mark
>>>>>>
>>>>>> On 4/29/20 8:08 AM, Mark-Macrae Dalton wrote:
>>>>>> > It might be useful to spell out more clearly what is expected from a review of a pull request, other than clicking the button.
>>>>>> >
>>>>>> > Mark
>>>>>> >
>>>>>> >> On Apr 28, 2020, at 8:06 PM, Mark Ito <marki at jlab.org> wrote:
>>>>>> >>
>>>>>> >> Cool. Good to know.
>>>>>> >>
>>>>>> >> On 4/28/20 7:59 PM, Justin Stevens wrote:
>>>>>> >>> FYI, just successfully executed the review -> merge PR sequence.  There’s basically just one more button to push to “Approve” the review, and then you can merge as usual.
>>>>>> >>> -Justin
>>>>>> >>>
>>>>>> >>>> On Apr 28, 2020, at 5:04 PM, Mark Ito <marki at jlab.org> wrote:
>>>>>> >>>>
>>>>>> >>>> Folks,
>>>>>> >>>>
>>>>>> >>>> At the software meeting just now, we decided to require "reviews" of pull requests before they can be merged onto the master branch. For now this is only for halld_sim and halld_recon. The webpages for the individual pull requests now reflect this. The idea is to prevent inadvertent changes to the master branches.
>>>>>> >>>>
>>>>>> >>>>    -- Mark
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> _______________________________________________
>>>>>> >>>> Halld-offline mailing list
>>>>>> >>>> Halld-offline at jlab.org
>>>>>> >>>> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>>>>> >> _______________________________________________
>>>>>> >> Halld-offline mailing list
>>>>>> >> Halld-offline at jlab.org
>>>>>> >> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>>>>> _______________________________________________
>>>>>> Halld-offline mailing list
>>>>>> Halld-offline at jlab.org
>>>>>> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>>>>
>>>>> _______________________________________________
>>>>> Halld-offline mailing list
>>>>> Halld-offline at jlab.org
>>>>> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>>>
>>>> _______________________________________________
>>>> Halld-offline mailing list
>>>> Halld-offline at jlab.org
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__nam10.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fmailman.jlab.org-252Fmailman-252Flistinfo-252Fhalld-2Doffline-26amp-3Bdata-3D02-257C01-257Crichard.t.jones-2540uconn.edu-257C27fd9227d73c4da2c00a08d7ec41b3d3-257C17f1a87e2a254eaab9df9d439034b080-257C0-257C0-257C637237639251886232-26amp-3Bsdata-3DdoNOCLzRPLj59h9l4LLjZ04WFccf2s7KPmLa0vXRHEs-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=Te_hCR4EUlJ6iCDYLJ8Viv2aDOR7D9ZZMoBAvf2H0M4&m=GK0lBCMKmzgyX_qXbere5id2N74O97-6LAd7SecvGBE&s=sdNpqS0Ana7WyQ6sbOQEzdDZTQpkQ7Gq16BHHt3D3_8&e= 
>>>
>>> _______________________________________________
>>> Halld-offline mailing list
>>> Halld-offline at jlab.org
>>> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>>
>>>
>>
>
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline
>
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline