[Dsg-rich] Summary of yesterday's EP temperature/humidity sensor readout concern

[Amrit Yegneswaran]

excellent summary tyler.
so if the main purpose is to determine that the sensors aren't frozen, i have a suggestion:
put a counter in for each sensor,  that increases by one every time the sensor value changes.
when we look at the counter the next time after lets say n hours we would know how many value changes each sensor registered.
In a matter of a few days, ceteris paribus, we could know which sensors are the slackers?
talk to you about this soon


________________________________
From: Dsg-rich <dsg-rich-bounces at jlab.org> on behalf of Tyler Lemon <tlemon at jlab.org>
Sent: Thursday, April 2, 2020 10:24 AM
To: dsg-rich <dsg-rich at jlab.org>
Subject: [Dsg-rich] Summary of yesterday's EP temperature/humidity sensor readout concern

Hello,
Below is a summary of yesterday's RICH EP temperature and humidity sensor readout concern. It's a little bit lengthy, but I wanted to be sure to explain the concern and what was done to investigate/alleviate the concern.

Yesterday, Valery emailed me with the concern that the RICH Electronic Panel (EP) temperature and humidity sensors appeared to be frozen.

To check this, I opened the RICH EP Hardware Interlock Control Systems Studio (CSS) screen in clascss on the PC clonsl2 and did see that the indicators for the sensors did not appear to be updating. I then used the terminal command "caget" (caget is usable from any system with EPICS installed) to get multiple data points for all EP temperature and humidity sensors. From these few samples, I could see that the values were changing with the changes on a ~0.001 level. Because the indicators on the CSS screen display values to a precision of 0.01, they did not display these changes. I also opened all EP temperature and humidity PVs in a live view plot (live view plot displays EPICS data for the PVs as it gets it without any archiving deadband) and saw that the PV values were updating there as well. After sharing these findings there was still concern over the values not appearing to update.

I then remembered the problem we had in November 2019 where the same issue happened and some sensors looked like they were frozen. In that situation, the PVs' monitoring deadband field was the issue. This monitoring deadband is not related to any archiving deadband in the Mya Archiver. The notation for this field is .MDEL appended to the PV name (e.g., B_DET_RICH_INTLK_TEMP1.MDEL). The monitoring deadband field is used to "throttle" the data in the EPICS IOC. For example, if the MDEL field of a PV is set to 1, the IOC will not acknowledge or do anything with the PV's value unless the IOC's observed change is over 1. This will also prevent the values from updating on the CSS screen and live plot unless that change is greater than 1.

After remembering this, I changed all EP hardware interlock PVs' monitoring deadband from 0.01 to 0. This causes any change, no matter how small, to be acknowledged by the interlock system's IOC. After the change, I could see more oscillation in the live view plot of the PVs (attached screenshot). Despite this change, there are still a few interlock sensors that appear to not be changing on the CSS screen because the changes are still less than the precision of the indicators on the CSS screen. However, this does not mean that the values are frozen since the PVs are indeed updating as shown by the result of caget and the resulting plots in the live plots.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20200402/f2464203/attachment-0002.html>