[Dsg-rich] Fwd: scalers for PMTs on one RICH tile not updating

Fri Feb 23 10:38:46 EST 2018

Hi Matteo, 

I've added you to DSG-RICH mailing list. 

Below is the email thread with Ben discussing the PMT scalers. 

You can also get to messages sent to the mailing list in the past through its archive at https://mailman.jlab.org/pipermail/dsg-rich/ 

Best regards, 
Tyler 

From: "Benjamin Raydo" <braydo at jlab.org> 
To: "Amrit Yegneswaran" <yeg at jlab.org> 
Cc: "dsg-rich" <dsg-rich at jlab.org> 
Sent: Thursday, February 22, 2018 7:08:09 PM 
Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating 

Hi Amrit, 

Not really a contradiction, here's the two possible scenarios I can guess: 

cause 1) Radiation induced error corrupts either the FPGA or ASIC. This won't effect the scalers, they will continue counting as normal. Once the DAQ is started again some point in the future the problem becomes apparent when trying to talk to the ASIC during initialization at which point the driver stops sending scalers. 

cause 2) Let's say ASIC identification works 99.9% of the time (due to a firmware bug) - I don't know what the real number is, but something like this . Since this is only done during DAQ initialization that's when the chances are evaluated...So DAQ init may screw it up 1 time, but probably not 2 times in a row. 

If you all see any pattern or suspicious behavior feel free to let me know - I'd know there are some surprises to find and fix still. 

Ben 

----- Original Message ----- 
From: "Amrit Yegneswaran" <yeg at jlab.org> 
To: "dsg-rich" <dsg-rich at jlab.org>, "Benjamin Raydo" <braydo at jlab.org> 
Sent: Thursday, February 22, 2018 6:21:07 PM 
Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating 

tyler/ben 

thanks for the info. 

however, isn't there a contradiction between point#1 and point #5? 
1. The most likely cause of PMT scalers frozen at zero is a general issue i n initializing the ASIC when DAQ is restarted. 
so: 
5. If scalers are stuck at zero, an LV power cycle and DAQ restart fixes the problem because it re-initializes the ASIC and gives it a chance to start communicating properly again. 

is the ben/tyler version of the epimenides paradox? 
amrit 

From: "Tyler Lemon" <tlemon at jlab.org> 
To: "Benjamin Raydo" <braydo at jlab.org>, "dsg-rich" <dsg-rich at jlab.org> 
Sent: Thursday, February 22, 2018 3:48:54 PM 
Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating 

Hi Ben, 

Thanks for your reply. From what you said, I understand that: 

1.) The most likely cause of PMT scalers frozen at zero is a general issue in initializing the ASIC when DAQ is restarted. 

2.) The error could happen anytime DAQ restarts because RICH readout hardware is initialized at that time. 

3.) If during this initialization, something happens and the DAQ readout cannot determine whether a tile has two or three ASICs, the scalers for that tile will be invalid and read as zeros. 

4.) The tile's temperature and voltages readout do not depend on ASIC configuration. This is why we still read temperature and voltage from the tile when its scalers are all zero. 

5.) If scalers are stuck at zero, an LV power cycle and DAQ restart fixes the problem because it re-initializes the ASIC and gives it a chance to start communicating properly again. 

Since the likely cause of the issue is a problem when DAQ is restarted, I looked back at the Hall B Logbook to see if a DAQ restart happened around 4:00AM today when the three PMTs all froze at zero. 

In today's owl shift summary (https://logbooks.jlab.org/entry/3536095), there are entries noting issues with DAQ and DAQ was restarted. From this, it does seem that there was some initialization problem for tile 66's ASICs when DAQ was restarted. 

Right now, we will continue to monitor throughout the day to see if the same tiles keep dropping out, maybe indicating another problem. 

Best regards, 
Tyler 

From: "Benjamin Raydo" <braydo at jlab.org> 
To: "Tyler Lemon" <tlemon at jlab.org> 
Cc: "dsg-rich" <dsg-rich at jlab.org>, "Patrizia Rossi" <rossi at jlab.org> 
Sent: Thursday, February 22, 2018 1:53:32 PM 
Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating 

Tyler, 

It's not clear what causes that issue yet (perhaps this is a new issue and we have a problem talking to the ASIC), but I believe it can happen anytime the DAQ system is restarted since it re-initializes the RICH readout hardware where it will check the ASIC configuration - if during this DAQ configuration a tile ASIC isn't found as a 2 or 3 ASIC module then scalers will be invalid (but temperature/voltages on the other hand don't depend on the ASIC configuration, so those continue to work). It's quite possible the FPGA or ASIC has a radiation induced errors or we just have an general issue initializing the ASIC sometimes (it's not clear) - in either case the procedure to fix this is an LV power cycle and then DAQ restart to reinitialize the RICH. In the future we'll add additional status information to see if we are dealing with radiation induced problems. 

Let me know if you'd like me to do something - I normally don't monitor and keep track of the status for RICH, but can certainly help get an issue resolved. 

Thanks, 
Ben 

----- Original Message ----- 
From: "Tyler Lemon" <tlemon at jlab.org> 
To: "dsg-rich" <dsg-rich at jlab.org>, "Benjamin Raydo" <braydo at jlab.org> 
Cc: "rossi patrizia" <rossi at jlab.org> 
Sent: Thursday, February 22, 2018 1:22:28 PM 
Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating 

Hello, 
After our email, DSG looked at the data available on EPICS for tile 66 with all PMTs stuck at zero. We saw that it is getting the correct low voltage and the PMTs are getting the correct high voltage. We also saw that the FPGA is reading out temperatures. From this, it leads DSG to believe that it is a problem with the hardware responsible for reading the PMT data. 

Hello Ben, 
>From your email below, it seems that there is an issue with communicating to the tile in question. What would cause a board to stop working correctly at seemingly random time? 

You mentioned that a power-cycle of LV and the DAQ should fix the issue. Has this been the procedure to fix a "frozen" tile that is not responding in the past? 

Best regards, 
Tyler 

From: "Benjamin Raydo" <braydo at jlab.org> 
To: "Valery Kubarovsky" <vpk at jlab.org>, tlemon at jlab.org 
Sent: Thursday, February 22, 2018 11:28:40 AM 
Subject: Re: [Dsg-rich] scalers for PMTs on one RICH tile not updating 

Hi Tyler and Valery, 

I just checked the tile status and see that the link is up to the unit, but it failed to identify the ASIC configuration (so the system doesn't know whether it is a 2 or 3 ASIC tile, so it puts 0) - you can see that in the status printout below for SSP slot 5 fiber 1 (#ASIC is 0, but ChUp is 1 saying a tile exists). This is observed from time-to-time and likely can be fixed by restarting the DAQ, but a power cycle of the low voltage for that tile and CODA restart may be necessary to fix this. This behavior is on the list of issues to clean up - please keep me informed if it becomes a serious problem for you all and I may be able to make it more of a priority. 

Thanks, 
Ben 

clasrun at clondaq6:clasrun> tcpClient rich4 'sspRich_PrintFiberStatus(5)' 
SSP RICH Fiber Status (slot=5): 
------------------------------------------------------------------------ 
Fiber Rst ChUp HrdErr FrmErr ErrCnt EvtEn Valid #ASIC FWRev 
------------------------------------------------------------------------ 
0 0 1 0 0 0 1 1 3 00000001 
1 0 1 0 0 0 1 1 0 00000001 
2 0 1 0 0 0 1 1 3 00000001 
3 0 1 0 0 0 1 1 3 00000001 
4 0 1 0 0 0 1 1 2 00000001 
5 0 1 0 0 0 1 1 3 00000001 
6 0 1 0 0 0 1 1 3 00000001 
7 0 1 0 0 0 1 1 3 00000001 
8 0 1 0 0 0 1 1 3 00000001 
9 0 1 0 0 0 1 1 3 00000001 
10 0 1 0 0 0 1 1 3 00000001 
11 0 1 0 0 0 1 1 3 00000001 
12 0 1 0 0 0 1 1 3 00000001 
13 0 1 0 0 0 1 1 3 00000001 
14 0 1 0 0 0 1 1 3 00000001 
15 0 0 0 0 0 0 0 0 00000000 
16 0 1 0 0 0 1 1 3 00000001 
17 0 1 0 0 0 1 1 3 00000001 
18 0 1 0 0 0 1 1 3 00000001 
19 0 1 0 0 0 1 1 2 00000001 
20 0 1 0 0 0 1 1 2 00000001 
21 0 1 0 0 0 1 1 3 00000001 
22 0 1 0 0 0 1 1 3 00000001 
23 0 1 0 0 0 1 1 3 00000001 
24 0 1 0 0 0 1 1 3 00000001 
25 0 1 0 0 0 1 1 3 00000001 
26 0 1 0 0 0 1 1 3 00000001 
27 0 1 0 0 0 1 1 3 00000001 
28 0 1 0 0 0 1 1 2 00000001 
29 0 1 0 0 0 1 1 3 00000001 
30 0 1 0 0 0 1 1 3 00000001 
31 0 1 0 0 0 1 1 3 00000001 

----- Original Message ----- 
From: "Valery Kubarovsky" <vpk at jlab.org> 
To: "Benjamin Raydo" <braydo at jlab.org> 
Sent: Thursday, February 22, 2018 8:59:15 AM 
Subject: Fwd: [Dsg-rich] scalers for PMTs on one RICH tile not updating 

Ben, 
Please reply to DSG group about the reasons why we have sometimes frozen scalers. 
Thanks, 
Valery 

Begin forwarded message: 

> From: Tyler Lemon <tlemon at jlab.org> 
> Date: February 22, 2018 at 08:22:50 EST 
> To: dsg-rich <dsg-rich at jlab.org> 
> Subject: [Dsg-rich] scalers for PMTs on one RICH tile not updating 
> 
> Hello, 
> 
> DSG noticed this morning that there is there is a group of three PMTs on the scaler map EPICS screen that are not updating. They have all been at zero for a few hours this morning. Attached is a screenshot with the three PMT scalers as zero. 
> 
> The PMTs in question are PMT 182, 183, and 184. They all look to be on the same electronic board (tile 66) that is read out in SSP slot 5, fiber 1. The temperatures for this board are still being read. 
> 
> Scalers for PMTs not updating for long periods until the IOC is rebooted or something else is done seems to be a common occurrence. 
> 
> What would be causing the scalers to stop updating while the temperatures are still read? 
> 
> Best regards, 
> Tyler 
> _______________________________________________ 
> Dsg-rich mailing list 
> Dsg-rich at jlab.org 
> https://mailman.jlab.org/mailman/listinfo/dsg-rich 

_______________________________________________ 
Dsg-rich mailing list 
Dsg-rich at jlab.org 
https://mailman.jlab.org/mailman/listinfo/dsg-rich 
G 
M 
T 

Text-to-speech function is limited to 200 characters 

Options : History : Feedback : Donate Close 
_______________________________________________ 
Dsg-rich mailing list 
Dsg-rich at jlab.org 
https://mailman.jlab.org/mailman/listinfo/dsg-rich 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20180223/35b526b3/attachment-0002.html>