[Dsg-rich] [Rich] FPGA temp alarm

Tue Oct 16 10:34:42 EDT 2018

I went through the archiver to see if I could find the data that backs up the sequence of events. Attached are the plots I found. 

I was able to see that HV had been off for about 6 hours (RICH HV with FPGA interlock.JPG). However, from the tiles in Marco's plot in his initial email, none of them went over the 75 deg C soft interlock limit. 

Looking at all FPGA temperatures, it seems that tile 138's temperature went up around the time of the interlock (all fpga temps 2018-10-16.JPG). 

Zooming in on when the tile temperature jumped, the data show that it went up to ~115 deg C (RICH tile 138 with LV.JPG). However, the LV appears to have been shut off on the third time the temperature jumped to 115 deg C. 

Is there some sort of time over threshold or delay that would keep the soft interlock from tripping on the first two temperature jumps? 

Best regards, 
Tyler 

From: "Valery Kubarovsky" <vpk at jlab.org> 
To: "Mirazita Marco" <Marco.Mirazita at lnf.infn.it> 
Cc: "Tyler Lemon" <tlemon at jlab.org>, rich at jlab.org, "dsg-rich" <dsg-rich at jlab.org> 
Sent: Tuesday, October 16, 2018 9:40:12 AM 
Subject: Re: [Rich] [Dsg-rich] FPGA temp alarm 

The detector (actually LV in this particular case, because the HV was off at that moment) 
was shut down by software interlock that monitors the FPGS temperature. 

From: "Marco Mirazita" <Marco.Mirazita at lnf.infn.it> 
To: "Tyler Lemon" <tlemon at jlab.org> 
Cc: "Valery Kubarovsky" <vpk at jlab.org>, rich at jlab.org, "dsg-rich" <dsg-rich at jlab.org> 
Sent: Tuesday, October 16, 2018 9:37:41 AM 
Subject: Re: [Rich] [Dsg-rich] FPGA temp alarm 

BQ_BEGIN
But we have a soft interlock shutting down the system if fpga 
temperature goes above 75 deg. 

Il 2018-10-16 14:24 Tyler Lemon ha scritto: 
> Hello Marco, 
> 
> The hardware interlock system does not monitor FPGA temperatures. It 
> monitors the RTDs installed in the EP. 
> 
> The hardware interlock system did not trip off the electronics because 
> the strange temperature readings were only seen on the FPGA 
> temperature readouts. 
> 
> Best regards, 
> Tyler 
> 
> ------------------------- 
> 
> FROM: "Mirazita Marco" <Marco.Mirazita at lnf.infn.it> 
> TO: "Valery Kubarovsky" <vpk at jlab.org> 
> CC: rich at jlab.org, "dsg-rich" <dsg-rich at jlab.org> 
> SENT: Tuesday, October 16, 2018 8:16:06 AM 
> SUBJECT: Re: [Rich] [Dsg-rich] FPGA temp alarm 
> 
> Hi Valery, 
> thank you, in fact my main worry was that the interlock didn't shut 
> down 
> the electronics. 
> It would be anyway good to understand why we had this event. 
> I saw from the strip charts that also the LV currents and voltages 
> have 
> frequent random spikes. 
> Marco 
> 
> Il 2018-10-16 13:55 Valery Kubarovsky ha scritto: 
>> Marco, 
>> If you take a more careful look you will find out that the LV was 
> shut 
>> down. 
>> The HV was off. We investigate the reason. It is the first time we 
> had 
>> such an event. 
>> Probably it was connected with the software update that was done 
>> approximately at the same time. 
>> Regards, 
>> Valery 
>> 
>> ------------------------- 
>> 
>>> FROM: "Marco Mirazita" <Marco.Mirazita at lnf.infn.it> 
>>> TO: dsg-rich at jlab.org, rich at jlab.org 
>>> SENT: Tuesday, October 16, 2018 4:03:24 AM 
>>> SUBJECT: [Dsg-rich] FPGA temp alarm 
>> 
>>> Hi all, 
>>> yesterday at about 4:20 pm jlab time I received several alarm 
>>> messages 
>>> saying that the FPGA temperatures reached values around 115 deg. 
>>> Since it looked like a readout error, I checked the strip chart, 
>>> where 
>>> there is actually a spike in the readout at that time, but to 
> values 
>>> of 
>>> the order of -10^5. See the attached plot. 
>>> So, it is clear that it was a readout error, but it is strange that 
>>> the 
>>> alarm message and epics report different values. 
>>> Also, according to the logbook, the electronics was not shut down 
> by 
>>> the 
>>> interlocks, as should have happened if the temperature really went 
>>> above 
>>> 100 deg. 
>>> So, perhaps the values reported in the alarm messages are not the 
>>> correct ones? 
>>> Marco 
>>> 
>>> _______________________________________________ 
>>> Dsg-rich mailing list 
>>> Dsg-rich at jlab.org 
>>> https://mailman.jlab.org/mailman/listinfo/dsg-rich 
>> 
>> _______________________________________________ 
>> Dsg-rich mailing list 
>> Dsg-rich at jlab.org 
>> https://mailman.jlab.org/mailman/listinfo/dsg-rich 
> _______________________________________________ 
> Rich mailing list 
> Rich at jlab.org 
> https://mailman.jlab.org/mailman/listinfo/rich 
BQ_END

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20181016/6b1fa077/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RICH tile 138 with LV.JPG
Type: image/jpeg
Size: 261280 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20181016/6b1fa077/attachment-0006.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: all fpga temps 2018-10-16.JPG
Type: image/jpeg
Size: 976485 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20181016/6b1fa077/attachment-0007.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RICH HV with FPGA interlock.JPG
Type: image/jpeg
Size: 271643 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20181016/6b1fa077/attachment-0008.jpe>