<html><body><div style="font-family: tahoma,new york,times,serif; font-size: 14pt; color: #009900"><div>tyler,<br></div><div>thank you for your crystal clear explanations.<br data-mce-bogus="1"></div><div>Now , i understand, what i thought i understood.<br data-mce-bogus="1"></div><div>feel exhilarated, like when i plunged (long time ago) in a icy-cold, unpolluted, clear-water spring in Barrow, Alaska<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>the two questions you posed at the end of the e-mail are quintessential.<br data-mce-bogus="1"></div><div>perhaps, some Illuminati from the RICH group will shed some light on this matter, thus enlightening us.<br data-mce-bogus="1"></div><div>amrit<br data-mce-bogus="1"></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Tyler Lemon" <tlemon@jlab.org><br><b>To: </b>"Chris Cuevas" <cuevas@jlab.org>, "Valery Kubarovsky" <vpk@jlab.org>, "Mirazita Marco" <Marco.Mirazita@lnf.infn.it>, rich@jlab.org, "dsg-rich" <dsg-rich@jlab.org><br><b>Sent: </b>Wednesday, October 17, 2018 10:47:00 AM<br><b>Subject: </b>Re: [Dsg-rich] [Rich] FPGA temp alarm<br></div><div><br></div><div data-marker="__QUOTED_TEXT__"><div style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000" data-mce-style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000;"><div><div style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000" data-mce-style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000;"><div><p style="margin: 0px;" data-mce-style="margin: 0px;">Hello,</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">RICH's software interlocks are EPICS-based and were developed by Baltzell and Kubarovsky. The soft interlocks directly monitor the FPGA temperatures using the chip’s data readout. Hardware interlock system cannot monitor the FPGA internal temperature, even with some sort of probe.</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">The five-second rate for the FPGA temperatures readout, I presume was set by Raydo during development. I also presume that the 10-second delay between temperature readings to trip soft interlocks was determined during the development of the soft interlock system by Baltzell and Kubarovsky.</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">The cRIO-based hardware interlock system developed by DSG monitors temperatures, humidity measurements, flows, and pressures using independent sensors.</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">The CAEN power supply mainframe has its own internal interlocks that monitor HV for the MAPMTs and LV for the FPGAs. The CAEN monitors voltage ans current outputs and can internally handle faults.</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">We have seen recently that FPGA temperatures falsely jump to high values for five seconds before returning to a normal value. Why do the temperatures suddenly jump 50 deg C for one measurement period? I do not know and nobody else seems to know why the temperatures readings are spiking.<br></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">We have also seen throughout the entirety of RICH operations that fibers suddenly disconnect. Sometimes the DAQ needs to be restarted to get them to reconnect, but sometimes they reconnect on their own after a few seconds. Why do the fibers disconnect? Again, I do not know and nobody else seems to know why..</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">Are there any concrete answers as to why the temperature readouts are suddenly jumping above the software interlock limit or as to why fibers keep disconnecting?</p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;">Best regards,</p><p style="margin: 0px;" data-mce-style="margin: 0px;">Tyler</p></div><br><hr id="zwchr"><div><b>From: </b>"Chris Cuevas" <cuevas@jlab.org><br><b>To: </b>"Tyler Lemon" <tlemon@jlab.org><br><b>Cc: </b>"Valery Kubarovsky" <vpk@jlab.org>, "Mirazita Marco" <Marco.Mirazita@lnf.infn.it>, rich@jlab.org, "dsg-rich" <dsg-rich@jlab.org><br><b>Sent: </b>Tuesday, October 16, 2018 3:56:57 PM<br><b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp alarm<br></div><br><div><p>Hi,</p><p>Software can stop for a variety of reasons, but good to know that there is a default watchdog timer on the cRIO. How much time elapses before software stoppage is detected and the relay opens? <br></p><p>I am not concerned about a power off sequence because I know the CAEN mainframe and modules will latch OFF until new commands are issued.</p><p>Thanks,</p><p>-Chris</p><p>~~~~~~~~~~~~~~~~~<br></p><br><div class="moz-cite-prefix">On 10/16/2018 3:11 PM, Tyler Lemon wrote:<br></div><blockquote cite="mid:2016969297.3697450.1539717108610.JavaMail.zimbra@jlab.org"><div style="font-family: times new roman,new york,times,serif;
font-size: 12pt; color: #000000" data-mce-style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000;"><div>If the software stops running, the relay goes into its default open state, disabling the CAEN.</div><div><br></div><div>The most likely cause of the cRIO's software stopping is a power outage. In the event of a power outage, there will be a sequence of events (compressor losing power, airflow/tank pressure dropping, interlock tripping) that will disable the CAEN. All components of the hardware interlock system are on UPS, so the hardware interlock components will stay alive long enough after the power outage to disable the CAEN. When the power is restored, the cRIO is set up to automatically run the interlock program on boot. On power restore, the CAEN will remain disabled until all normal operating conditions (airflow, buffer tank pressure, temperatures) are restored and the CAEN's interlocks are cleared.<br> <br></div><div>--Tyler<br></div><hr id="zwchr"><div><b>From: </b>"Chris Cuevas" <a class="moz-txt-link-rfc2396E" href="mailto:cuevas@jlab.org" target="_blank" data-mce-href="mailto:cuevas@jlab.org"><cuevas@jlab.org></a><br> <b>To: </b>"Tyler Lemon" <a class="moz-txt-link-rfc2396E" href="mailto:tlemon@jlab.org" target="_blank" data-mce-href="mailto:tlemon@jlab.org"><tlemon@jlab.org></a><br> <b>Cc: </b>"Valery Kubarovsky" <a class="moz-txt-link-rfc2396E" href="mailto:vpk@jlab.org" target="_blank" data-mce-href="mailto:vpk@jlab.org"><vpk@jlab.org></a>, "Mirazita Marco" <a class="moz-txt-link-rfc2396E" href="mailto:Marco.Mirazita@lnf.infn.it" target="_blank" data-mce-href="mailto:Marco.Mirazita@lnf.infn.it"><Marco.Mirazita@lnf.infn.it></a>, <a class="moz-txt-link-abbreviated" href="mailto:rich@jlab.org" target="_blank" data-mce-href="mailto:rich@jlab.org">rich@jlab.org</a>, "dsg-rich" <a class="moz-txt-link-rfc2396E" href="mailto:dsg-rich@jlab.org" target="_blank" data-mce-href="mailto:dsg-rich@jlab.org"><dsg-rich@jlab.org></a><br> <b>Sent: </b>Tuesday, October 16, 2018 2:53:02 PM<br> <b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp alarm<br></div><div><br></div><div><p>Hi Tyler,</p><p>So the cRIO software must always be running to detect out of limit signals from the front end sensors? What happens if the software stops running? Is there a default timeout[watchdog] setting that will open the final relay contact?</p><p>Thanks for the feedback,</p><p>-Chris</p><p>~~~~~~~~~~~<br></p><br><div class="moz-cite-prefix">On 10/16/2018 1:52 PM, Tyler Lemon wrote:<br></div><blockquote cite="mid:1675170298.2052603.1539712320930.JavaMail.zimbra@jlab.org"><div style="font-family: times new roman,new
york,times,serif; font-size: 12pt; color: #000000" data-mce-style="font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000;"><div>Chris,<br></div><div><br></div><div>For the hardware interlocks, if any of the cRIO-based system's sensors (RTDs, humidity, airflow, air pressure, N2 flow; sensors not dependent on FPGA DAQ readout) go out of limit, the cRIO disables HV and LV by opening a relay contact to the Interlock port on the CAEN mainframe.<br></div><div><br></div><div>This disables HV and LV and will only allow HV and LV to be re-enabled if the interlock is clear and a user has reset the interlock.<br></div><div><br></div><div>Regards,<br></div><div>Tyler<br></div><div><br></div><div><br></div><div><br></div><div><br></div><hr id="zwchr"><div><b>From: </b>"Chris Cuevas" <a class="moz-txt-link-rfc2396E" href="mailto:cuevas@jlab.org" target="_blank" data-mce-href="mailto:cuevas@jlab.org"><cuevas@jlab.org></a><br> <b>To: </b>"Valery Kubarovsky" <a class="moz-txt-link-rfc2396E" href="mailto:vpk@jlab.org" target="_blank" data-mce-href="mailto:vpk@jlab.org"><vpk@jlab.org></a>, "Mirazita Marco" <a class="moz-txt-link-rfc2396E" href="mailto:Marco.Mirazita@lnf.infn.it" target="_blank" data-mce-href="mailto:Marco.Mirazita@lnf.infn.it"><Marco.Mirazita@lnf.infn.it></a><br> <b>Cc: </b><a class="moz-txt-link-abbreviated" href="mailto:rich@jlab.org" target="_blank" data-mce-href="mailto:rich@jlab.org">rich@jlab.org</a>, "dsg-rich" <a class="moz-txt-link-rfc2396E" href="mailto:dsg-rich@jlab.org" target="_blank" data-mce-href="mailto:dsg-rich@jlab.org"><dsg-rich@jlab.org></a><br> <b>Sent: </b>Tuesday, October 16, 2018 1:38:14 PM<br> <b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp alarm<br></div><div><br></div><div><p>Hi All,</p><p>The FPGAs have a firmware set point for the die temperature. Ben can verify this set point, but it sure sounds like the 'interlock' systems rely on software code. Are there watchdog timers set up in case the code stops running? <br></p><p>For a true hardware interlock, one that does not depend on software, there should be temperature probes coupled directly to a comparator circuit which latches off the Low_Voltage outputs to the FPGA boards. From these email threads, it sounds like all the temperature devices are monitored with software, and if a software set point threshold is violated, then the Low_Voltage is turned OFF. If the monitored temperature devices go below the software set point threshold, is the Low_Voltage power reapplied automatically?</p><p>Thanks,</p><p>-Chris</p><p>~~~~~~~~~~~~~~~~~</p><br><div class="moz-cite-prefix">On 10/16/2018 9:40 AM, Valery Kubarovsky wrote:<br></div><blockquote cite="mid:26066286.4049022.1539697212523.JavaMail.zimbra@jlab.org"><div style="font-family: arial,helvetica,sans-serif;
font-size: 12pt; color: #000000" data-mce-style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000;"><div>The detector (actually LV in this particular case, because the HV was off at that moment)</div><div>was shut down by software interlock that monitors the FPGS temperature.</div><div><br></div><hr id="zwchr"><div><blockquote style="border-left:2px solid
#1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid#1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Marco Mirazita" <a class="moz-txt-link-rfc2396E" href="mailto:Marco.Mirazita@lnf.infn.it" target="_blank" data-mce-href="mailto:Marco.Mirazita@lnf.infn.it"><Marco.Mirazita@lnf.infn.it></a><br> <b>To: </b>"Tyler Lemon" <a class="moz-txt-link-rfc2396E" href="mailto:tlemon@jlab.org" target="_blank" data-mce-href="mailto:tlemon@jlab.org"><tlemon@jlab.org></a><br> <b>Cc: </b>"Valery Kubarovsky" <a class="moz-txt-link-rfc2396E" href="mailto:vpk@jlab.org" target="_blank" data-mce-href="mailto:vpk@jlab.org"><vpk@jlab.org></a>, <a class="moz-txt-link-abbreviated" href="mailto:rich@jlab.org" target="_blank" data-mce-href="mailto:rich@jlab.org">rich@jlab.org</a>, "dsg-rich" <a class="moz-txt-link-rfc2396E" href="mailto:dsg-rich@jlab.org" target="_blank" data-mce-href="mailto:dsg-rich@jlab.org"><dsg-rich@jlab.org></a><br> <b>Sent: </b>Tuesday, October 16, 2018 9:37:41 AM<br> <b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp alarm<br></blockquote></div><div><blockquote style="border-left:2px solid
#1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid#1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;">But we have a soft interlock shutting down the system if fpga <br> temperature goes above 75 deg.<br> <br> <br> Il 2018-10-16 14:24 Tyler Lemon ha scritto:<br> > Hello Marco,<br> > <br> > The hardware interlock system does not monitor FPGA temperatures. It<br> > monitors the RTDs installed in the EP.<br> > <br> > The hardware interlock system did not trip off the electronics because<br> > the strange temperature readings were only seen on the FPGA<br> > temperature readouts.<br> > <br> > Best regards,<br> > Tyler<br> > <br> > -------------------------<br> > <br> > FROM: "Mirazita Marco" <a class="moz-txt-link-rfc2396E" href="mailto:Marco.Mirazita@lnf.infn.it" target="_blank" data-mce-href="mailto:Marco.Mirazita@lnf.infn.it"><Marco.Mirazita@lnf.infn.it></a><br> > TO: "Valery Kubarovsky" <a class="moz-txt-link-rfc2396E" href="mailto:vpk@jlab.org" target="_blank" data-mce-href="mailto:vpk@jlab.org"><vpk@jlab.org></a><br> > CC: <a class="moz-txt-link-abbreviated" href="mailto:rich@jlab.org" target="_blank" data-mce-href="mailto:rich@jlab.org">rich@jlab.org</a>, "dsg-rich" <a class="moz-txt-link-rfc2396E" href="mailto:dsg-rich@jlab.org" target="_blank" data-mce-href="mailto:dsg-rich@jlab.org"><dsg-rich@jlab.org></a><br> > SENT: Tuesday, October 16, 2018 8:16:06 AM<br> > SUBJECT: Re: [Rich] [Dsg-rich] FPGA temp alarm<br> > <br> > Hi Valery,<br> > thank you, in fact my main worry was that the interlock didn't shut<br> > down<br> > the electronics.<br> > It would be anyway good to understand why we had this event.<br> > I saw from the strip charts that also the LV currents and voltages<br> > have<br> > frequent random spikes.<br> > Marco<br> > <br> > Il 2018-10-16 13:55 Valery Kubarovsky ha scritto:<br> >> Marco,<br> >> If you take a more careful look you will find out that the LV was<br> > shut<br> >> down.<br> >> The HV was off. We investigate the reason. It is the first time we<br> > had<br> >> such an event.<br> >> Probably it was connected with the software update that was done<br> >> approximately at the same time.<br> >> Regards,<br> >> Valery<br> >> <br> >> -------------------------<br> >> <br> >>> FROM: "Marco Mirazita" <a class="moz-txt-link-rfc2396E" href="mailto:Marco.Mirazita@lnf.infn.it" target="_blank" data-mce-href="mailto:Marco.Mirazita@lnf.infn.it"><Marco.Mirazita@lnf.infn.it></a><br> >>> TO: <a class="moz-txt-link-abbreviated" href="mailto:dsg-rich@jlab.org" target="_blank" data-mce-href="mailto:dsg-rich@jlab.org">dsg-rich@jlab.org</a>, <a class="moz-txt-link-abbreviated" href="mailto:rich@jlab.org" target="_blank" data-mce-href="mailto:rich@jlab.org">rich@jlab.org</a><br> >>> SENT: Tuesday, October 16, 2018 4:03:24 AM<br> >>> SUBJECT: [Dsg-rich] FPGA temp alarm<br> >> <br> >>> Hi all,<br> >>> yesterday at about 4:20 pm jlab time I received several alarm<br> >>> messages<br> >>> saying that the FPGA temperatures reached values around 115 deg.<br> >>> Since it looked like a readout error, I checked the strip chart,<br> >>> where<br> >>> there is actually a spike in the readout at that time, but to<br> > values<br> >>> of<br> >>> the order of -10^5. See the attached plot.<br> >>> So, it is clear that it was a readout error, but it is strange that<br> >>> the<br> >>> alarm message and epics report different values.<br> >>> Also, according to the logbook, the electronics was not shut down<br> > by<br> >>> the<br> >>> interlocks, as should have happened if the temperature really went<br> >>> above<br> >>> 100 deg.<br> >>> So, perhaps the values reported in the alarm messages are not the<br> >>> correct ones?<br> >>> Marco<br> >>> <br> >>> _______________________________________________<br> >>> Dsg-rich mailing list<br> >>> <a class="moz-txt-link-abbreviated" href="mailto:Dsg-rich@jlab.org" target="_blank" data-mce-href="mailto:Dsg-rich@jlab.org">Dsg-rich@jlab.org</a><br> >>> <a class="moz-txt-link-freetext" href="https://mailman.jlab.org/mailman/listinfo/dsg-rich" target="_blank" data-mce-href="https://mailman.jlab.org/mailman/listinfo/dsg-rich">https://mailman.jlab.org/mailman/listinfo/dsg-rich</a><br> >> <br> >> _______________________________________________<br> >> Dsg-rich mailing list<br> >> <a class="moz-txt-link-abbreviated" href="mailto:Dsg-rich@jlab.org" target="_blank" data-mce-href="mailto:Dsg-rich@jlab.org">Dsg-rich@jlab.org</a><br> >> <a class="moz-txt-link-freetext" href="https://mailman.jlab.org/mailman/listinfo/dsg-rich" target="_blank" data-mce-href="https://mailman.jlab.org/mailman/listinfo/dsg-rich">https://mailman.jlab.org/mailman/listinfo/dsg-rich</a><br> > _______________________________________________<br> > Rich mailing list<br> > <a class="moz-txt-link-abbreviated" href="mailto:Rich@jlab.org" target="_blank" data-mce-href="mailto:Rich@jlab.org">Rich@jlab.org</a><br> > <a class="moz-txt-link-freetext" href="https://mailman.jlab.org/mailman/listinfo/rich" target="_blank" data-mce-href="https://mailman.jlab.org/mailman/listinfo/rich">https://mailman.jlab.org/mailman/listinfo/rich</a><br></blockquote></div></div><br><fieldset class="mimeAttachmentHeader"></fieldset><br><pre>_______________________________________________
Rich mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Rich@jlab.org" target="_blank" data-mce-href="mailto:Rich@jlab.org">Rich@jlab.org</a>
<a class="moz-txt-link-freetext" href="https://mailman.jlab.org/mailman/listinfo/rich" target="_blank" data-mce-href="https://mailman.jlab.org/mailman/listinfo/rich">https://mailman.jlab.org/mailman/listinfo/rich</a>
</pre></blockquote><br> <br> _______________________________________________<br> Rich mailing list<br> <a class="moz-txt-link-abbreviated" href="mailto:Rich@jlab.org" target="_blank" data-mce-href="mailto:Rich@jlab.org">Rich@jlab.org</a><br> <a class="moz-txt-link-freetext" href="https://mailman.jlab.org/mailman/listinfo/rich" target="_blank" data-mce-href="https://mailman.jlab.org/mailman/listinfo/rich">https://mailman.jlab.org/mailman/listinfo/rich</a><br></div></div></blockquote><br> <br></div></div></blockquote></div></div><br></div></div><br>_______________________________________________<br>Dsg-rich mailing list<br>Dsg-rich@jlab.org<br>https://mailman.jlab.org/mailman/listinfo/dsg-rich<br></div></div></body></html>