[Dsg-rich] [Rich] FPGA temp alarm

Tue Oct 16 14:14:41 EDT 2018

The software protocol is similar. Disable HV and LV, alarm shift (and others). 
You have to clear interlock and switch LV and HV manually. 
Valery 

> From: "Tyler Lemon" <tlemon at jlab.org>
> To: "Chris Cuevas" <cuevas at jlab.org>
> Cc: "Valery Kubarovsky" <vpk at jlab.org>, "Marco Mirazita"
> <Marco.Mirazita at lnf.infn.it>, rich at jlab.org, "dsg-rich" <dsg-rich at jlab.org>
> Sent: Tuesday, October 16, 2018 1:52:00 PM
> Subject: Re: [Rich] [Dsg-rich] FPGA temp alarm

> Chris,

> For the hardware interlocks, if any of the cRIO-based system's sensors (RTDs,
> humidity, airflow, air pressure, N2 flow; sensors not dependent on FPGA DAQ
> readout) go out of limit, the cRIO disables HV and LV by opening a relay
> contact to the Interlock port on the CAEN mainframe.

> This disables HV and LV and will only allow HV and LV to be re-enabled if the
> interlock is clear and a user has reset the interlock.

> Regards,
> Tyler

> From: "Chris Cuevas" <cuevas at jlab.org>
> To: "Valery Kubarovsky" <vpk at jlab.org>, "Mirazita Marco"
> <Marco.Mirazita at lnf.infn.it>
> Cc: rich at jlab.org, "dsg-rich" <dsg-rich at jlab.org>
> Sent: Tuesday, October 16, 2018 1:38:14 PM
> Subject: Re: [Rich] [Dsg-rich] FPGA temp alarm

> Hi All,

> The FPGAs have a firmware set point for the die temperature. Ben can verify this
> set point, but it sure sounds like the 'interlock' systems rely on software
> code. Are there watchdog timers set up in case the code stops running?

> For a true hardware interlock, one that does not depend on software, there
> should be temperature probes coupled directly to a comparator circuit which
> latches off the Low_Voltage outputs to the FPGA boards. From these email
> threads, it sounds like all the temperature devices are monitored with
> software, and if a software set point threshold is violated, then the
> Low_Voltage is turned OFF. If the monitored temperature devices go below the
> software set point threshold, is the Low_Voltage power reapplied automatically?

> Thanks,

> -Chris

> ~~~~~~~~~~~~~~~~~
> On 10/16/2018 9:40 AM, Valery Kubarovsky wrote:

>> The detector (actually LV in this particular case, because the HV was off at
>> that moment)
>> was shut down by software interlock that monitors the FPGS temperature.

>>> From: "Marco Mirazita" <Marco.Mirazita at lnf.infn.it>
>>> To: "Tyler Lemon" <tlemon at jlab.org>
>>> Cc: "Valery Kubarovsky" <vpk at jlab.org> , rich at jlab.org , "dsg-rich"
>>> <dsg-rich at jlab.org>
>>> Sent: Tuesday, October 16, 2018 9:37:41 AM
>>> Subject: Re: [Rich] [Dsg-rich] FPGA temp alarm

>>> But we have a soft interlock shutting down the system if fpga
>>> temperature goes above 75 deg.

>>> Il 2018-10-16 14:24 Tyler Lemon ha scritto:
>>> > Hello Marco,

>>> > The hardware interlock system does not monitor FPGA temperatures. It
>>> > monitors the RTDs installed in the EP.

>>> > The hardware interlock system did not trip off the electronics because
>>> > the strange temperature readings were only seen on the FPGA
>>> > temperature readouts.

>>> > Best regards,
>>> > Tyler

>>> > -------------------------

>>> > FROM: "Mirazita Marco" <Marco.Mirazita at lnf.infn.it>
>>> > TO: "Valery Kubarovsky" <vpk at jlab.org>
>>> > CC: rich at jlab.org , "dsg-rich" <dsg-rich at jlab.org>
>>> > SENT: Tuesday, October 16, 2018 8:16:06 AM
>>> > SUBJECT: Re: [Rich] [Dsg-rich] FPGA temp alarm

>>> > Hi Valery,
>>> > thank you, in fact my main worry was that the interlock didn't shut
>>> > down
>>> > the electronics.
>>> > It would be anyway good to understand why we had this event.
>>> > I saw from the strip charts that also the LV currents and voltages
>>> > have
>>> > frequent random spikes.
>>> > Marco

>>> > Il 2018-10-16 13:55 Valery Kubarovsky ha scritto:
>>> >> Marco,
>>> >> If you take a more careful look you will find out that the LV was
>>> > shut
>>> >> down.
>>> >> The HV was off. We investigate the reason. It is the first time we
>>> > had
>>> >> such an event.
>>> >> Probably it was connected with the software update that was done
>>> >> approximately at the same time.
>>> >> Regards,
>>> >> Valery

>>> >> -------------------------

>>> >>> FROM: "Marco Mirazita" <Marco.Mirazita at lnf.infn.it>
>>> >>> TO: dsg-rich at jlab.org , rich at jlab.org
>>> >>> SENT: Tuesday, October 16, 2018 4:03:24 AM
>>> >>> SUBJECT: [Dsg-rich] FPGA temp alarm

>>> >>> Hi all,
>>> >>> yesterday at about 4:20 pm jlab time I received several alarm
>>> >>> messages
>>> >>> saying that the FPGA temperatures reached values around 115 deg.
>>> >>> Since it looked like a readout error, I checked the strip chart,
>>> >>> where
>>> >>> there is actually a spike in the readout at that time, but to
>>> > values
>>> >>> of
>>> >>> the order of -10^5. See the attached plot.
>>> >>> So, it is clear that it was a readout error, but it is strange that
>>> >>> the
>>> >>> alarm message and epics report different values.
>>> >>> Also, according to the logbook, the electronics was not shut down
>>> > by
>>> >>> the
>>> >>> interlocks, as should have happened if the temperature really went
>>> >>> above
>>> >>> 100 deg.
>>> >>> So, perhaps the values reported in the alarm messages are not the
>>> >>> correct ones?
>>> >>> Marco

>>> >>> _______________________________________________
>>> >>> Dsg-rich mailing list
>>> >>> Dsg-rich at jlab.org
>>> >>> https://mailman.jlab.org/mailman/listinfo/dsg-rich

>>> >> _______________________________________________
>>> >> Dsg-rich mailing list
>>> >> Dsg-rich at jlab.org
>>> >> https://mailman.jlab.org/mailman/listinfo/dsg-rich
>>> > _______________________________________________
>>> > Rich mailing list
>>> > Rich at jlab.org
>>> > https://mailman.jlab.org/mailman/listinfo/rich

>> _______________________________________________
>> Rich mailing list Rich at jlab.org https://mailman.jlab.org/mailman/listinfo/rich

> _______________________________________________
> Rich mailing list
> Rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/rich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20181016/984884f0/attachment-0002.html>