[Dsg-rich] [Rich] FPGA temp alarm

[Valery Kubarovsky]

Hi Tyler, 

When t goes over the threshold one more measurement is scheduled in 10 seconds (if I remember correctly). If the second measurement shows problem the program trips CAEN. 

Valery 

> From: "Tyler Lemon" <tlemon at jlab.org>
> To: "Valery Kubarovsky" <vpk at jlab.org>
> Cc: "Marco Mirazita" <Marco.Mirazita at lnf.infn.it>, rich at jlab.org, "dsg-rich"
> <dsg-rich at jlab.org>
> Sent: Tuesday, October 16, 2018 10:34:42 AM
> Subject: Re: [Rich] [Dsg-rich] FPGA temp alarm

> I went through the archiver to see if I could find the data that backs up the
> sequence of events. Attached are the plots I found.

> I was able to see that HV had been off for about 6 hours (RICH HV with FPGA
> interlock.JPG). However, from the tiles in Marco's plot in his initial email,
> none of them went over the 75 deg C soft interlock limit.

> Looking at all FPGA temperatures, it seems that tile 138's temperature went up
> around the time of the interlock (all fpga temps 2018-10-16.JPG).

> Zooming in on when the tile temperature jumped, the data show that it went up to
> ~115 deg C (RICH tile 138 with LV.JPG). However, the LV appears to have been
> shut off on the third time the temperature jumped to 115 deg C.

> Is there some sort of time over threshold or delay that would keep the soft
> interlock from tripping on the first two temperature jumps?

> Best regards,
> Tyler

> From: "Valery Kubarovsky" <vpk at jlab.org>
> To: "Mirazita Marco" <Marco.Mirazita at lnf.infn.it>
> Cc: "Tyler Lemon" <tlemon at jlab.org>, rich at jlab.org, "dsg-rich"
> <dsg-rich at jlab.org>
> Sent: Tuesday, October 16, 2018 9:40:12 AM
> Subject: Re: [Rich] [Dsg-rich] FPGA temp alarm

> The detector (actually LV in this particular case, because the HV was off at
> that moment)
> was shut down by software interlock that monitors the FPGS temperature.

>> From: "Marco Mirazita" <Marco.Mirazita at lnf.infn.it>
>> To: "Tyler Lemon" <tlemon at jlab.org>
>> Cc: "Valery Kubarovsky" <vpk at jlab.org>, rich at jlab.org, "dsg-rich"
>> <dsg-rich at jlab.org>
>> Sent: Tuesday, October 16, 2018 9:37:41 AM
>> Subject: Re: [Rich] [Dsg-rich] FPGA temp alarm

>> But we have a soft interlock shutting down the system if fpga
>> temperature goes above 75 deg.

>> Il 2018-10-16 14:24 Tyler Lemon ha scritto:
>> > Hello Marco,

>> > The hardware interlock system does not monitor FPGA temperatures. It
>> > monitors the RTDs installed in the EP.

>> > The hardware interlock system did not trip off the electronics because
>> > the strange temperature readings were only seen on the FPGA
>> > temperature readouts.

>> > Best regards,
>> > Tyler

>> > -------------------------

>> > FROM: "Mirazita Marco" <Marco.Mirazita at lnf.infn.it>
>> > TO: "Valery Kubarovsky" <vpk at jlab.org>
>> > CC: rich at jlab.org, "dsg-rich" <dsg-rich at jlab.org>
>> > SENT: Tuesday, October 16, 2018 8:16:06 AM
>> > SUBJECT: Re: [Rich] [Dsg-rich] FPGA temp alarm

>> > Hi Valery,
>> > thank you, in fact my main worry was that the interlock didn't shut
>> > down
>> > the electronics.
>> > It would be anyway good to understand why we had this event.
>> > I saw from the strip charts that also the LV currents and voltages
>> > have
>> > frequent random spikes.
>> > Marco

>> > Il 2018-10-16 13:55 Valery Kubarovsky ha scritto:
>> >> Marco,
>> >> If you take a more careful look you will find out that the LV was
>> > shut
>> >> down.
>> >> The HV was off. We investigate the reason. It is the first time we
>> > had
>> >> such an event.
>> >> Probably it was connected with the software update that was done
>> >> approximately at the same time.
>> >> Regards,
>> >> Valery

>> >> -------------------------

>> >>> FROM: "Marco Mirazita" <Marco.Mirazita at lnf.infn.it>
>> >>> TO: dsg-rich at jlab.org, rich at jlab.org
>> >>> SENT: Tuesday, October 16, 2018 4:03:24 AM
>> >>> SUBJECT: [Dsg-rich] FPGA temp alarm

>> >>> Hi all,
>> >>> yesterday at about 4:20 pm jlab time I received several alarm
>> >>> messages
>> >>> saying that the FPGA temperatures reached values around 115 deg.
>> >>> Since it looked like a readout error, I checked the strip chart,
>> >>> where
>> >>> there is actually a spike in the readout at that time, but to
>> > values
>> >>> of
>> >>> the order of -10^5. See the attached plot.
>> >>> So, it is clear that it was a readout error, but it is strange that
>> >>> the
>> >>> alarm message and epics report different values.
>> >>> Also, according to the logbook, the electronics was not shut down
>> > by
>> >>> the
>> >>> interlocks, as should have happened if the temperature really went
>> >>> above
>> >>> 100 deg.
>> >>> So, perhaps the values reported in the alarm messages are not the
>> >>> correct ones?
>> >>> Marco

>> >>> _______________________________________________
>> >>> Dsg-rich mailing list
>> >>> Dsg-rich at jlab.org
>> >>> https://mailman.jlab.org/mailman/listinfo/dsg-rich

>> >> _______________________________________________
>> >> Dsg-rich mailing list
>> >> Dsg-rich at jlab.org
>> >> https://mailman.jlab.org/mailman/listinfo/dsg-rich
>> > _______________________________________________
>> > Rich mailing list
>> > Rich at jlab.org
>> > https://mailman.jlab.org/mailman/listinfo/rich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20181016/567cc6ab/attachment-0002.html>