<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">All,<div class=""><br class=""></div><div class="">I checked with histograms of the previous days and it seems that there is nothing strange. </div><div class="">So please disregard my previous e-mail</div><div class=""><br class=""></div><div class="">patrizia<br class=""><div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">---------------------------------------------------<br class=""># Patrizia Rossi<br class=""># Jefferson Lab <br class=""># 12000 Jefferson Avenue<br class=""># Suite 5<br class=""># Newport News VA 23606 (USA)<br class=""># e-mail: <a href="mailto:rossi@jlab.org" class="">rossi@jlab.org</a><br class=""># phone: +1 (757) 269 7740<br class=""># fax: + 1 (757) 269 5800</div><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""># cell: +1 (757) 528 7606<br class=""><br class=""><br class=""><br class=""></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On 23/feb/2018, at 15:22, Patrizia Rossi <<a href="mailto:rossi@jlab.org" class="">rossi@jlab.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Dear all,<div class=""><br class=""><div class="">we just started a new run after ~2hours of beam off. I checked the RICH occupancy and TDC time and it seems to me that the TDC time is a little bit weird.</div><div class="">The three red bands are not separated now. May you please have a look at them and let me know if it is OK? I made an entry in both the HBLog and RICH log (<span style="color: rgb(49, 49, 49); font-family: 'Trebuchet MS', 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: 14px; font-weight: bold; background-color: rgb(255, 255, 255);" class="">Lognumber </span><a href="https://logbooks.jlab.org/entry/3537256" class="lognumber" data-lognumber="3537256" style="text-decoration: none; color: rgb(0, 0, 238); font-family: 'Trebuchet MS', 'Helvetica Neue', Arial, Helvetica, sans-serif; font-size: 14px; font-weight: bold;">3537256</a>)</div><div class=""><br class=""></div><div class="">thanks</div><div class="">patrizia</div><div class=""><br class=""></div><div apple-content-edited="true" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">---------------------------------------------------<br class=""># Patrizia Rossi<br class=""># Jefferson Lab <br class=""># 12000 Jefferson Avenue<br class=""># Suite 5<br class=""># Newport News VA 23606 (USA)<br class=""># e-mail: <a href="mailto:rossi@jlab.org" class="">rossi@jlab.org</a><br class=""># phone: +1 (757) 269 7740<br class=""># fax: + 1 (757) 269 5800</div><div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""># cell: +1 (757) 528 7606<br class=""><br class=""><br class=""><br class=""></div>
</div>
<br class=""><div class=""><blockquote type="cite" class=""><div class="">On 23/feb/2018, at 10:38, Tyler Lemon <<a href="mailto:tlemon@jlab.org" class="">tlemon@jlab.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><div data-marker="__QUOTED_TEXT__" class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><div class="">Hi Matteo,<br class=""></div><br class=""><div class="">I've added you to DSG-RICH mailing list.<br class=""></div><br class=""><div class="">Below is the email thread with Ben discussing the PMT scalers.<br class=""></div><br class=""><div class="">You can also get to messages sent to the mailing list in the past through its archive at <a href="https://mailman.jlab.org/pipermail/dsg-rich/" class="">https://mailman.jlab.org/pipermail/dsg-rich/</a><br class=""></div><br class=""><div class="">Best regards,<br class=""></div><div class="">Tyler<br class=""></div><br class=""><hr id="zwchr" class=""><div class=""><b class="">From: </b>"Benjamin Raydo" <<a href="mailto:braydo@jlab.org" class="">braydo@jlab.org</a>><br class=""><b class="">To: </b>"Amrit Yegneswaran" <<a href="mailto:yeg@jlab.org" class="">yeg@jlab.org</a>><br class=""><b class="">Cc: </b>"dsg-rich" <<a href="mailto:dsg-rich@jlab.org" class="">dsg-rich@jlab.org</a>><br class=""><b class="">Sent: </b>Thursday, February 22, 2018 7:08:09 PM<br class=""><b class="">Subject: </b>Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating<br class=""></div><br class=""><div class="">Hi Amrit,<br class=""><br class="">Not really a contradiction, here's the two possible scenarios I can guess:<br class=""><br class="">cause 1) Radiation induced error corrupts either the FPGA or ASIC. This won't effect the scalers, they will continue counting as normal. Once the DAQ is started again some point in the future the problem becomes apparent when trying to talk to the ASIC during initialization at which point the driver stops sending scalers.<br class=""><br class="">cause 2) Let's say ASIC identification works 99.9% of the time (due to a firmware bug) - I don't know what the real number is, but something like this . Since this is only done during DAQ initialization that's when the chances are evaluated...So DAQ init may screw it up 1 time, but probably not 2 times in a row.<br class=""><br class="">If you all see any pattern or suspicious behavior feel free to let me know - I'd know there are some surprises to find and fix still.<br class=""><br class="">Ben<br class=""><br class="">----- Original Message -----<br class="">From: "Amrit Yegneswaran" <<a href="mailto:yeg@jlab.org" class="">yeg@jlab.org</a>><br class="">To: "dsg-rich" <<a href="mailto:dsg-rich@jlab.org" class="">dsg-rich@jlab.org</a>>, "Benjamin Raydo" <<a href="mailto:braydo@jlab.org" class="">braydo@jlab.org</a>><br class="">Sent: Thursday, February 22, 2018 6:21:07 PM<br class="">Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating<br class=""><br class="">tyler/ben <br class=""><br class="">thanks for the info. <br class=""><br class=""><br class="">however, isn't there a contradiction between point#1 and point #5? <br class="">1. The most likely cause of PMT scalers frozen at zero is a general issue i n initializing the ASIC when DAQ is restarted. <br class="">so: <br class="">5. If scalers are stuck at zero, an LV power cycle and DAQ restart fixes the problem because it re-initializes the ASIC and gives it a chance to start communicating properly again. <br class=""><br class="">is the ben/tyler version of the epimenides paradox? <br class="">amrit <br class=""><br class=""><br class="">From: "Tyler Lemon" <<a href="mailto:tlemon@jlab.org" class="">tlemon@jlab.org</a>> <br class="">To: "Benjamin Raydo" <<a href="mailto:braydo@jlab.org" class="">braydo@jlab.org</a>>, "dsg-rich" <<a href="mailto:dsg-rich@jlab.org" class="">dsg-rich@jlab.org</a>> <br class="">Sent: Thursday, February 22, 2018 3:48:54 PM <br class="">Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating <br class=""><br class="">Hi Ben, <br class=""><br class="">Thanks for your reply. From what you said, I understand that: <br class=""><br class="">1.) The most likely cause of PMT scalers frozen at zero is a general issue in initializing the ASIC when DAQ is restarted. <br class=""><br class="">2.) The error could happen anytime DAQ restarts because RICH readout hardware is initialized at that time. <br class=""><br class="">3.) If during this initialization, something happens and the DAQ readout cannot determine whether a tile has two or three ASICs, the scalers for that tile will be invalid and read as zeros. <br class=""><br class="">4.) The tile's temperature and voltages readout do not depend on ASIC configuration. This is why we still read temperature and voltage from the tile when its scalers are all zero. <br class=""><br class="">5.) If scalers are stuck at zero, an LV power cycle and DAQ restart fixes the problem because it re-initializes the ASIC and gives it a chance to start communicating properly again. <br class=""><br class=""><br class="">Since the likely cause of the issue is a problem when DAQ is restarted, I looked back at the Hall B Logbook to see if a DAQ restart happened around 4:00AM today when the three PMTs all froze at zero. <br class=""><br class="">In today's owl shift summary (<a href="https://logbooks.jlab.org/entry/3536095" class="">https://logbooks.jlab.org/entry/3536095</a>), there are entries noting issues with DAQ and DAQ was restarted. From this, it does seem that there was some initialization problem for tile 66's ASICs when DAQ was restarted. <br class=""><br class="">Right now, we will continue to monitor throughout the day to see if the same tiles keep dropping out, maybe indicating another problem. <br class=""><br class="">Best regards, <br class="">Tyler <br class=""><br class=""><br class=""><br class=""><br class="">From: "Benjamin Raydo" <<a href="mailto:braydo@jlab.org" class="">braydo@jlab.org</a>> <br class="">To: "Tyler Lemon" <<a href="mailto:tlemon@jlab.org" class="">tlemon@jlab.org</a>> <br class="">Cc: "dsg-rich" <<a href="mailto:dsg-rich@jlab.org" class="">dsg-rich@jlab.org</a>>, "Patrizia Rossi" <<a href="mailto:rossi@jlab.org" class="">rossi@jlab.org</a>> <br class="">Sent: Thursday, February 22, 2018 1:53:32 PM <br class="">Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating <br class=""><br class="">Tyler, <br class=""><br class="">It's not clear what causes that issue yet (perhaps this is a new issue and we have a problem talking to the ASIC), but I believe it can happen anytime the DAQ system is restarted since it re-initializes the RICH readout hardware where it will check the ASIC configuration - if during this DAQ configuration a tile ASIC isn't found as a 2 or 3 ASIC module then scalers will be invalid (but temperature/voltages on the other hand don't depend on the ASIC configuration, so those continue to work). It's quite possible the FPGA or ASIC has a radiation induced errors or we just have an general issue initializing the ASIC sometimes (it's not clear) - in either case the procedure to fix this is an LV power cycle and then DAQ restart to reinitialize the RICH. In the future we'll add additional status information to see if we are dealing with radiation induced problems. <br class=""><br class="">Let me know if you'd like me to do something - I normally don't monitor and keep track of the status for RICH, but can certainly help get an issue resolved. <br class=""><br class="">Thanks, <br class="">Ben <br class=""><br class="">----- Original Message ----- <br class="">From: "Tyler Lemon" <<a href="mailto:tlemon@jlab.org" class="">tlemon@jlab.org</a>> <br class="">To: "dsg-rich" <<a href="mailto:dsg-rich@jlab.org" class="">dsg-rich@jlab.org</a>>, "Benjamin Raydo" <<a href="mailto:braydo@jlab.org" class="">braydo@jlab.org</a>> <br class="">Cc: "rossi patrizia" <<a href="mailto:rossi@jlab.org" class="">rossi@jlab.org</a>> <br class="">Sent: Thursday, February 22, 2018 1:22:28 PM <br class="">Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating <br class=""><br class="">Hello, <br class="">After our email, DSG looked at the data available on EPICS for tile 66 with all PMTs stuck at zero. We saw that it is getting the correct low voltage and the PMTs are getting the correct high voltage. We also saw that the FPGA is reading out temperatures. From this, it leads DSG to believe that it is a problem with the hardware responsible for reading the PMT data. <br class=""><br class=""><br class="">Hello Ben, <br class="">From your email below, it seems that there is an issue with communicating to the tile in question. What would cause a board to stop working correctly at seemingly random time? <br class=""><br class="">You mentioned that a power-cycle of LV and the DAQ should fix the issue. Has this been the procedure to fix a "frozen" tile that is not responding in the past? <br class=""><br class="">Best regards, <br class="">Tyler <br class=""><br class=""><br class=""><br class="">From: "Benjamin Raydo" <<a href="mailto:braydo@jlab.org" class="">braydo@jlab.org</a>> <br class="">To: "Valery Kubarovsky" <<a href="mailto:vpk@jlab.org" class="">vpk@jlab.org</a>>, <a href="mailto:tlemon@jlab.org" class="">tlemon@jlab.org</a> <br class="">Sent: Thursday, February 22, 2018 11:28:40 AM <br class="">Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating <br class=""><br class="">Hi Tyler and Valery, <br class=""><br class="">I just checked the tile status and see that the link is up to the unit, but it failed to identify the ASIC configuration (so the system doesn't know whether it is a 2 or 3 ASIC tile, so it puts 0) - you can see that in the status printout below for SSP slot 5 fiber 1 (#ASIC is 0, but ChUp is 1 saying a tile exists). This is observed from time-to-time and likely can be fixed by restarting the DAQ, but a power cycle of the low voltage for that tile and CODA restart may be necessary to fix this. This behavior is on the list of issues to clean up - please keep me informed if it becomes a serious problem for you all and I may be able to make it more of a priority. <br class=""><br class="">Thanks, <br class="">Ben <br class=""><br class=""><br class="">clasrun@clondaq6:clasrun> tcpClient rich4 'sspRich_PrintFiberStatus(5)' <br class="">SSP RICH Fiber Status (slot=5): <br class="">------------------------------------------------------------------------ <br class="">Fiber Rst ChUp HrdErr FrmErr ErrCnt EvtEn Valid #ASIC FWRev <br class="">------------------------------------------------------------------------ <br class="">0 0 1 0 0 0 1 1 3 00000001 <br class="">1 0 1 0 0 0 1 1 0 00000001 <br class="">2 0 1 0 0 0 1 1 3 00000001 <br class="">3 0 1 0 0 0 1 1 3 00000001 <br class="">4 0 1 0 0 0 1 1 2 00000001 <br class="">5 0 1 0 0 0 1 1 3 00000001 <br class="">6 0 1 0 0 0 1 1 3 00000001 <br class="">7 0 1 0 0 0 1 1 3 00000001 <br class="">8 0 1 0 0 0 1 1 3 00000001 <br class="">9 0 1 0 0 0 1 1 3 00000001 <br class="">10 0 1 0 0 0 1 1 3 00000001 <br class="">11 0 1 0 0 0 1 1 3 00000001 <br class="">12 0 1 0 0 0 1 1 3 00000001 <br class="">13 0 1 0 0 0 1 1 3 00000001 <br class="">14 0 1 0 0 0 1 1 3 00000001 <br class="">15 0 0 0 0 0 0 0 0 00000000 <br class="">16 0 1 0 0 0 1 1 3 00000001 <br class="">17 0 1 0 0 0 1 1 3 00000001 <br class="">18 0 1 0 0 0 1 1 3 00000001 <br class="">19 0 1 0 0 0 1 1 2 00000001 <br class="">20 0 1 0 0 0 1 1 2 00000001 <br class="">21 0 1 0 0 0 1 1 3 00000001 <br class="">22 0 1 0 0 0 1 1 3 00000001 <br class="">23 0 1 0 0 0 1 1 3 00000001 <br class="">24 0 1 0 0 0 1 1 3 00000001 <br class="">25 0 1 0 0 0 1 1 3 00000001 <br class="">26 0 1 0 0 0 1 1 3 00000001 <br class="">27 0 1 0 0 0 1 1 3 00000001 <br class="">28 0 1 0 0 0 1 1 2 00000001 <br class="">29 0 1 0 0 0 1 1 3 00000001 <br class="">30 0 1 0 0 0 1 1 3 00000001 <br class="">31 0 1 0 0 0 1 1 3 00000001 <br class=""><br class=""><br class="">----- Original Message ----- <br class="">From: "Valery Kubarovsky" <<a href="mailto:vpk@jlab.org" class="">vpk@jlab.org</a>> <br class="">To: "Benjamin Raydo" <<a href="mailto:braydo@jlab.org" class="">braydo@jlab.org</a>> <br class="">Sent: Thursday, February 22, 2018 8:59:15 AM <br class="">Subject: Fwd: [Dsg-rich] scalers for PMTs on one RICH tile not updating <br class=""><br class="">Ben, <br class="">Please reply to DSG group about the reasons why we have sometimes frozen scalers. <br class="">Thanks, <br class="">Valery <br class=""><br class=""><br class="">Begin forwarded message: <br class=""><br class="">> From: Tyler Lemon <<a href="mailto:tlemon@jlab.org" class="">tlemon@jlab.org</a>> <br class="">> Date: February 22, 2018 at 08:22:50 EST <br class="">> To: dsg-rich <<a href="mailto:dsg-rich@jlab.org" class="">dsg-rich@jlab.org</a>> <br class="">> Subject: [Dsg-rich] scalers for PMTs on one RICH tile not updating <br class="">> <br class="">> Hello, <br class="">> <br class="">> DSG noticed this morning that there is there is a group of three PMTs on the scaler map EPICS screen that are not updating. They have all been at zero for a few hours this morning. Attached is a screenshot with the three PMT scalers as zero. <br class="">> <br class="">> The PMTs in question are PMT 182, 183, and 184. They all look to be on the same electronic board (tile 66) that is read out in SSP slot 5, fiber 1. The temperatures for this board are still being read. <br class="">> <br class="">> Scalers for PMTs not updating for long periods until the IOC is rebooted or something else is done seems to be a common occurrence. <br class="">> <br class="">> What would be causing the scalers to stop updating while the temperatures are still read? <br class="">> <br class="">> Best regards, <br class="">> Tyler <br class="">> _______________________________________________ <br class="">> Dsg-rich mailing list <br class="">> <a href="mailto:Dsg-rich@jlab.org" class="">Dsg-rich@jlab.org</a> <br class="">> <a href="https://mailman.jlab.org/mailman/listinfo/dsg-rich" class="">https://mailman.jlab.org/mailman/listinfo/dsg-rich</a> <br class=""><br class=""><br class="">_______________________________________________ <br class="">Dsg-rich mailing list <br class=""><a href="mailto:Dsg-rich@jlab.org" class="">Dsg-rich@jlab.org</a> <br class=""><a href="https://mailman.jlab.org/mailman/listinfo/dsg-rich" class="">https://mailman.jlab.org/mailman/listinfo/dsg-rich</a> <br class="">G <br class="">M <br class="">T <br class=""> <br class=""> <br class=""> <br class=""> <br class=""> <br class=""> <br class="">Text-to-speech function is limited to 200 characters <br class=""> <br class=""> Options : History : Feedback : Donate Close<br class="">_______________________________________________<br class="">Dsg-rich mailing list<br class=""><a href="mailto:Dsg-rich@jlab.org" class="">Dsg-rich@jlab.org</a><br class=""><a href="https://mailman.jlab.org/mailman/listinfo/dsg-rich" class="">https://mailman.jlab.org/mailman/listinfo/dsg-rich</a></div></div><br class=""></div></div></div>_______________________________________________<br class="">Dsg-rich mailing list<br class=""><a href="mailto:Dsg-rich@jlab.org" class="">Dsg-rich@jlab.org</a><br class=""><a href="https://mailman.jlab.org/mailman/listinfo/dsg-rich" class="">https://mailman.jlab.org/mailman/listinfo/dsg-rich</a><br class=""></div></blockquote></div><br class=""></div></div>_______________________________________________<br class="">Dsg-rich mailing list<br class=""><a href="mailto:Dsg-rich@jlab.org" class="">Dsg-rich@jlab.org</a><br class="">https://mailman.jlab.org/mailman/listinfo/dsg-rich<br class=""></div></blockquote></div><br class=""></div></body></html>