[Dsg-rich] Summary of yesterday's EP temperature/humidity sensor readout concern

[Tyler Lemon]

Hello,
Below is a summary of yesterday's RICH EP temperature and humidity sensor readout concern. It's a little bit lengthy, but I wanted to be sure to explain the concern and what was done to investigate/alleviate the concern.

Yesterday, Valery emailed me with the concern that the RICH Electronic Panel (EP) temperature and humidity sensors appeared to be frozen.

To check this, I opened the RICH EP Hardware Interlock Control Systems Studio (CSS) screen in clascss on the PC clonsl2 and did see that the indicators for the sensors did not appear to be updating. I then used the terminal command "caget" (caget is usable from any system with EPICS installed) to get multiple data points for all EP temperature and humidity sensors. From these few samples, I could see that the values were changing with the changes on a ~0.001 level. Because the indicators on the CSS screen display values to a precision of 0.01, they did not display these changes. I also opened all EP temperature and humidity PVs in a live view plot (live view plot displays EPICS data for the PVs as it gets it without any archiving deadband) and saw that the PV values were updating there as well. After sharing these findings there was still concern over the values not appearing to update.

I then remembered the problem we had in November 2019 where the same issue happened and some sensors looked like they were frozen. In that situation, the PVs' monitoring deadband field was the issue. This monitoring deadband is not related to any archiving deadband in the Mya Archiver. The notation for this field is .MDEL appended to the PV name (e.g., B_DET_RICH_INTLK_TEMP1.MDEL). The monitoring deadband field is used to "throttle" the data in the EPICS IOC. For example, if the MDEL field of a PV is set to 1, the IOC will not acknowledge or do anything with the PV's value unless the IOC's observed change is over 1. This will also prevent the values from updating on the CSS screen and live plot unless that change is greater than 1.

After remembering this, I changed all EP hardware interlock PVs' monitoring deadband from 0.01 to 0. This causes any change, no matter how small, to be acknowledged by the interlock system's IOC. After the change, I could see more oscillation in the live view plot of the PVs (attached screenshot). Despite this change, there are still a few interlock sensors that appear to not be changing on the CSS screen because the changes are still less than the precision of the indicators on the CSS screen. However, this does not mean that the values are frozen since the PVs are indeed updating as shown by the result of caget and the resulting plots in the live plots.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-rich/attachments/20200402/9f310523/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Post MDEL field change to zero 2020-04-01.jpg
Type: image/jpeg
Size: 289309 bytes
Desc: Post MDEL field change to zero 2020-04-01.jpg
URL: <https://mailman.jlab.org/pipermail/dsg-rich/attachments/20200402/9f310523/attachment-0001.jpg>