<html><body><div style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000"><div>I went through the archiver to see if I could find the data that backs up the sequence of events. Attached are the plots I found.<br></div><div><br data-mce-bogus="1"></div><div>I was able to see that HV had been off for about 6 hours (RICH HV with FPGA interlock.JPG). However, from the tiles in Marco's plot in his initial email, none of them went over the 75 deg C soft interlock limit.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Looking at all FPGA temperatures, it seems that tile 138's temperature went up around the time of the interlock (all fpga temps 2018-10-16.JPG).<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Zooming in on when the tile temperature jumped, the data show that it went up to ~115 deg C (RICH tile 138 with LV.JPG). However, the LV appears to have been shut off on the third time the temperature jumped to 115 deg C.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Is there some sort of time over threshold or delay that would keep the soft interlock from tripping on the first two temperature jumps?<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Best regards,<br data-mce-bogus="1"></div><div>Tyler<br data-mce-bogus="1"></div><div><br></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Valery Kubarovsky" <vpk@jlab.org><br><b>To: </b>"Mirazita Marco" <Marco.Mirazita@lnf.infn.it><br><b>Cc: </b>"Tyler Lemon" <tlemon@jlab.org>, rich@jlab.org, "dsg-rich" <dsg-rich@jlab.org><br><b>Sent: </b>Tuesday, October 16, 2018 9:40:12 AM<br><b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp alarm<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><div style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000"><div>The detector (actually LV in this particular case, because the HV was off at that moment)</div><div>was shut down by software interlock that monitors the FPGS temperature.</div><br><hr id="zwchr"><div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Marco Mirazita" <Marco.Mirazita@lnf.infn.it><br><b>To: </b>"Tyler Lemon" <tlemon@jlab.org><br><b>Cc: </b>"Valery Kubarovsky" <vpk@jlab.org>, rich@jlab.org, "dsg-rich" <dsg-rich@jlab.org><br><b>Sent: </b>Tuesday, October 16, 2018 9:37:41 AM<br><b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp alarm<br></blockquote></div><div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">But we have a soft interlock shutting down the system if fpga <br>temperature goes above 75 deg.<br><br><br>Il 2018-10-16 14:24 Tyler Lemon ha scritto:<br>> Hello Marco,<br>> <br>> The hardware interlock system does not monitor FPGA temperatures. It<br>> monitors the RTDs installed in the EP.<br>> <br>> The hardware interlock system did not trip off the electronics because<br>> the strange temperature readings were only seen on the FPGA<br>> temperature readouts.<br>> <br>> Best regards,<br>> Tyler<br>> <br>> -------------------------<br>> <br>> FROM: "Mirazita Marco" <Marco.Mirazita@lnf.infn.it><br>> TO: "Valery Kubarovsky" <vpk@jlab.org><br>> CC: rich@jlab.org, "dsg-rich" <dsg-rich@jlab.org><br>> SENT: Tuesday, October 16, 2018 8:16:06 AM<br>> SUBJECT: Re: [Rich] [Dsg-rich] FPGA temp alarm<br>> <br>> Hi Valery,<br>> thank you, in fact my main worry was that the interlock didn't shut<br>> down<br>> the electronics.<br>> It would be anyway good to understand why we had this event.<br>> I saw from the strip charts that also the LV currents and voltages<br>> have<br>> frequent random spikes.<br>> Marco<br>> <br>> Il 2018-10-16 13:55 Valery Kubarovsky ha scritto:<br>>> Marco,<br>>> If you take a more careful look you will find out that the LV was<br>> shut<br>>> down.<br>>> The HV was off. We investigate the reason. It is the first time we<br>> had<br>>> such an event.<br>>> Probably it was connected with the software update that was done<br>>> approximately at the same time.<br>>> Regards,<br>>> Valery<br>>> <br>>> -------------------------<br>>> <br>>>> FROM: "Marco Mirazita" <Marco.Mirazita@lnf.infn.it><br>>>> TO: dsg-rich@jlab.org, rich@jlab.org<br>>>> SENT: Tuesday, October 16, 2018 4:03:24 AM<br>>>> SUBJECT: [Dsg-rich] FPGA temp alarm<br>>> <br>>>> Hi all,<br>>>> yesterday at about 4:20 pm jlab time I received several alarm<br>>>> messages<br>>>> saying that the FPGA temperatures reached values around 115 deg.<br>>>> Since it looked like a readout error, I checked the strip chart,<br>>>> where<br>>>> there is actually a spike in the readout at that time, but to<br>> values<br>>>> of<br>>>> the order of -10^5. See the attached plot.<br>>>> So, it is clear that it was a readout error, but it is strange that<br>>>> the<br>>>> alarm message and epics report different values.<br>>>> Also, according to the logbook, the electronics was not shut down<br>> by<br>>>> the<br>>>> interlocks, as should have happened if the temperature really went<br>>>> above<br>>>> 100 deg.<br>>>> So, perhaps the values reported in the alarm messages are not the<br>>>> correct ones?<br>>>> Marco<br>>>> <br>>>> _______________________________________________<br>>>> Dsg-rich mailing list<br>>>> Dsg-rich@jlab.org<br>>>> https://mailman.jlab.org/mailman/listinfo/dsg-rich<br>>> <br>>> _______________________________________________<br>>> Dsg-rich mailing list<br>>> Dsg-rich@jlab.org<br>>> https://mailman.jlab.org/mailman/listinfo/dsg-rich<br>> _______________________________________________<br>> Rich mailing list<br>> Rich@jlab.org<br>> https://mailman.jlab.org/mailman/listinfo/rich</blockquote></div></div><br></div></div></body></html>