<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>Software can stop for a variety of reasons, but good to know that
there is a default watchdog timer on the cRIO. How much time
elapses before software stoppage is detected and the relay opens?
<br>
</p>
<p>I am not concerned about a power off sequence because I know the
CAEN mainframe and modules will latch OFF until new commands are
issued.</p>
<p>Thanks,</p>
<p>-Chris</p>
<p>~~~~~~~~~~~~~~~~~<br>
</p>
<br>
<div class="moz-cite-prefix">On 10/16/2018 3:11 PM, Tyler Lemon
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:2016969297.3697450.1539717108610.JavaMail.zimbra@jlab.org">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div style="font-family: times new roman,new york,times,serif;
font-size: 12pt; color: #000000">
<div>If the software stops running, the relay goes into its
default open state, disabling the CAEN.</div>
<div><br data-mce-bogus="1">
</div>
<div>The most likely cause of the cRIO's software stopping is a
power outage. In the event of a power outage, there will be a
sequence of events (compressor losing power, airflow/tank
pressure dropping, interlock tripping) that will disable the
CAEN. All components of the hardware interlock system are on
UPS, so the hardware interlock components will stay alive long
enough after the power outage to disable the CAEN. When the
power is restored, the cRIO is set up to automatically run the
interlock program on boot. On power restore, the CAEN will
remain disabled until all normal operating conditions
(airflow, buffer tank pressure, temperatures) are restored and
the CAEN's interlocks are cleared.<br>
<br>
</div>
<div>--Tyler<br data-mce-bogus="1">
</div>
<hr id="zwchr" data-marker="__DIVIDER__">
<div data-marker="__HEADERS__"><b>From: </b>"Chris Cuevas"
<a class="moz-txt-link-rfc2396E" href="mailto:cuevas@jlab.org"><cuevas@jlab.org></a><br>
<b>To: </b>"Tyler Lemon" <a class="moz-txt-link-rfc2396E" href="mailto:tlemon@jlab.org"><tlemon@jlab.org></a><br>
<b>Cc: </b>"Valery Kubarovsky" <a class="moz-txt-link-rfc2396E" href="mailto:vpk@jlab.org"><vpk@jlab.org></a>,
"Mirazita Marco" <a class="moz-txt-link-rfc2396E" href="mailto:Marco.Mirazita@lnf.infn.it"><Marco.Mirazita@lnf.infn.it></a>,
<a class="moz-txt-link-abbreviated" href="mailto:rich@jlab.org">rich@jlab.org</a>, "dsg-rich" <a class="moz-txt-link-rfc2396E" href="mailto:dsg-rich@jlab.org"><dsg-rich@jlab.org></a><br>
<b>Sent: </b>Tuesday, October 16, 2018 2:53:02 PM<br>
<b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp alarm<br>
</div>
<div><br>
</div>
<div data-marker="__QUOTED_TEXT__">
<p>Hi Tyler,</p>
<p>So the cRIO software must always be running to detect out
of limit signals from the front end sensors? What happens if
the software stops running? Is there a default
timeout[watchdog] setting that will open the final relay
contact?</p>
<p>Thanks for the feedback,</p>
<p>-Chris</p>
<p>~~~~~~~~~~~<br>
</p>
<br>
<div class="moz-cite-prefix">On 10/16/2018 1:52 PM, Tyler
Lemon wrote:<br>
</div>
<blockquote
cite="mid:1675170298.2052603.1539712320930.JavaMail.zimbra@jlab.org">
<div style="font-family: times new roman,new
york,times,serif; font-size: 12pt; color: #000000">
<div>Chris,<br>
</div>
<div><br>
</div>
<div>For the hardware interlocks, if any of the cRIO-based
system's sensors (RTDs, humidity, airflow, air pressure,
N2 flow; sensors not dependent on FPGA DAQ readout) go
out of limit, the cRIO disables HV and LV by opening a
relay contact to the Interlock port on the CAEN
mainframe.<br>
</div>
<div><br>
</div>
<div>This disables HV and LV and will only allow HV and LV
to be re-enabled if the interlock is clear and a user
has reset the interlock.<br>
</div>
<div><br>
</div>
<div>Regards,<br>
</div>
<div>Tyler<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<hr id="zwchr">
<div><b>From: </b>"Chris Cuevas" <a
class="moz-txt-link-rfc2396E"
href="mailto:cuevas@jlab.org" target="_blank"
moz-do-not-send="true"><cuevas@jlab.org></a><br>
<b>To: </b>"Valery Kubarovsky" <a
class="moz-txt-link-rfc2396E"
href="mailto:vpk@jlab.org" target="_blank"
moz-do-not-send="true"><vpk@jlab.org></a>,
"Mirazita Marco" <a class="moz-txt-link-rfc2396E"
href="mailto:Marco.Mirazita@lnf.infn.it"
target="_blank" moz-do-not-send="true"><Marco.Mirazita@lnf.infn.it></a><br>
<b>Cc: </b><a class="moz-txt-link-abbreviated"
href="mailto:rich@jlab.org" target="_blank"
moz-do-not-send="true">rich@jlab.org</a>, "dsg-rich" <a
class="moz-txt-link-rfc2396E"
href="mailto:dsg-rich@jlab.org" target="_blank"
moz-do-not-send="true"><dsg-rich@jlab.org></a><br>
<b>Sent: </b>Tuesday, October 16, 2018 1:38:14 PM<br>
<b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp alarm<br>
</div>
<div><br>
</div>
<div>
<p>Hi All,</p>
<p>The FPGAs have a firmware set point for the die
temperature. Ben can verify this set point, but it
sure sounds like the 'interlock' systems rely on
software code. Are there watchdog timers set up in
case the code stops running? <br>
</p>
<p>For a true hardware interlock, one that does not
depend on software, there should be temperature probes
coupled directly to a comparator circuit which latches
off the Low_Voltage outputs to the FPGA boards. From
these email threads, it sounds like all the
temperature devices are monitored with software, and
if a software set point threshold is violated, then
the Low_Voltage is turned OFF. If the monitored
temperature devices go below the software set point
threshold, is the Low_Voltage power reapplied
automatically?</p>
<p>Thanks,</p>
<p>-Chris</p>
<p>~~~~~~~~~~~~~~~~~</p>
<br>
<div class="moz-cite-prefix">On 10/16/2018 9:40 AM,
Valery Kubarovsky wrote:<br>
</div>
<blockquote
cite="mid:26066286.4049022.1539697212523.JavaMail.zimbra@jlab.org">
<div style="font-family: arial,helvetica,sans-serif;
font-size: 12pt; color: #000000">
<div>The detector (actually LV in this particular
case, because the HV was off at that moment)</div>
<div>was shut down by software interlock that
monitors the FPGS temperature.</div>
<div><br>
</div>
<hr id="zwchr">
<div>
<blockquote style="border-left:2px solid
#1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From:
</b>"Marco Mirazita" <a
class="moz-txt-link-rfc2396E"
href="mailto:Marco.Mirazita@lnf.infn.it"
target="_blank" moz-do-not-send="true"><Marco.Mirazita@lnf.infn.it></a><br>
<b>To: </b>"Tyler Lemon" <a
class="moz-txt-link-rfc2396E"
href="mailto:tlemon@jlab.org" target="_blank"
moz-do-not-send="true"><tlemon@jlab.org></a><br>
<b>Cc: </b>"Valery Kubarovsky" <a
class="moz-txt-link-rfc2396E"
href="mailto:vpk@jlab.org" target="_blank"
moz-do-not-send="true"><vpk@jlab.org></a>,
<a class="moz-txt-link-abbreviated"
href="mailto:rich@jlab.org" target="_blank"
moz-do-not-send="true">rich@jlab.org</a>,
"dsg-rich" <a class="moz-txt-link-rfc2396E"
href="mailto:dsg-rich@jlab.org"
target="_blank" moz-do-not-send="true"><dsg-rich@jlab.org></a><br>
<b>Sent: </b>Tuesday, October 16, 2018 9:37:41
AM<br>
<b>Subject: </b>Re: [Rich] [Dsg-rich] FPGA temp
alarm<br>
</blockquote>
</div>
<div>
<blockquote style="border-left:2px solid
#1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">But
we have a soft interlock shutting down the
system if fpga <br>
temperature goes above 75 deg.<br>
<br>
<br>
Il 2018-10-16 14:24 Tyler Lemon ha scritto:<br>
> Hello Marco,<br>
> <br>
> The hardware interlock system does not
monitor FPGA temperatures. It<br>
> monitors the RTDs installed in the EP.<br>
> <br>
> The hardware interlock system did not trip
off the electronics because<br>
> the strange temperature readings were only
seen on the FPGA<br>
> temperature readouts.<br>
> <br>
> Best regards,<br>
> Tyler<br>
> <br>
> -------------------------<br>
> <br>
> FROM: "Mirazita Marco" <a
class="moz-txt-link-rfc2396E"
href="mailto:Marco.Mirazita@lnf.infn.it"
target="_blank" moz-do-not-send="true"><Marco.Mirazita@lnf.infn.it></a><br>
> TO: "Valery Kubarovsky" <a
class="moz-txt-link-rfc2396E"
href="mailto:vpk@jlab.org" target="_blank"
moz-do-not-send="true"><vpk@jlab.org></a><br>
> CC: <a class="moz-txt-link-abbreviated"
href="mailto:rich@jlab.org" target="_blank"
moz-do-not-send="true">rich@jlab.org</a>,
"dsg-rich" <a class="moz-txt-link-rfc2396E"
href="mailto:dsg-rich@jlab.org"
target="_blank" moz-do-not-send="true"><dsg-rich@jlab.org></a><br>
> SENT: Tuesday, October 16, 2018 8:16:06 AM<br>
> SUBJECT: Re: [Rich] [Dsg-rich] FPGA temp
alarm<br>
> <br>
> Hi Valery,<br>
> thank you, in fact my main worry was that
the interlock didn't shut<br>
> down<br>
> the electronics.<br>
> It would be anyway good to understand why
we had this event.<br>
> I saw from the strip charts that also the
LV currents and voltages<br>
> have<br>
> frequent random spikes.<br>
> Marco<br>
> <br>
> Il 2018-10-16 13:55 Valery Kubarovsky ha
scritto:<br>
>> Marco,<br>
>> If you take a more careful look you
will find out that the LV was<br>
> shut<br>
>> down.<br>
>> The HV was off. We investigate the
reason. It is the first time we<br>
> had<br>
>> such an event.<br>
>> Probably it was connected with the
software update that was done<br>
>> approximately at the same time.<br>
>> Regards,<br>
>> Valery<br>
>> <br>
>> -------------------------<br>
>> <br>
>>> FROM: "Marco Mirazita" <a
class="moz-txt-link-rfc2396E"
href="mailto:Marco.Mirazita@lnf.infn.it"
target="_blank" moz-do-not-send="true"><Marco.Mirazita@lnf.infn.it></a><br>
>>> TO: <a
class="moz-txt-link-abbreviated"
href="mailto:dsg-rich@jlab.org"
target="_blank" moz-do-not-send="true">dsg-rich@jlab.org</a>,
<a class="moz-txt-link-abbreviated"
href="mailto:rich@jlab.org" target="_blank"
moz-do-not-send="true">rich@jlab.org</a><br>
>>> SENT: Tuesday, October 16, 2018
4:03:24 AM<br>
>>> SUBJECT: [Dsg-rich] FPGA temp alarm<br>
>> <br>
>>> Hi all,<br>
>>> yesterday at about 4:20 pm jlab
time I received several alarm<br>
>>> messages<br>
>>> saying that the FPGA temperatures
reached values around 115 deg.<br>
>>> Since it looked like a readout
error, I checked the strip chart,<br>
>>> where<br>
>>> there is actually a spike in the
readout at that time, but to<br>
> values<br>
>>> of<br>
>>> the order of -10^5. See the
attached plot.<br>
>>> So, it is clear that it was a
readout error, but it is strange that<br>
>>> the<br>
>>> alarm message and epics report
different values.<br>
>>> Also, according to the logbook, the
electronics was not shut down<br>
> by<br>
>>> the<br>
>>> interlocks, as should have happened
if the temperature really went<br>
>>> above<br>
>>> 100 deg.<br>
>>> So, perhaps the values reported in
the alarm messages are not the<br>
>>> correct ones?<br>
>>> Marco<br>
>>> <br>
>>>
_______________________________________________<br>
>>> Dsg-rich mailing list<br>
>>> <a
class="moz-txt-link-abbreviated"
href="mailto:Dsg-rich@jlab.org"
target="_blank" moz-do-not-send="true">Dsg-rich@jlab.org</a><br>
>>> <a class="moz-txt-link-freetext"
href="https://mailman.jlab.org/mailman/listinfo/dsg-rich"
target="_blank" moz-do-not-send="true">https://mailman.jlab.org/mailman/listinfo/dsg-rich</a><br>
>> <br>
>>
_______________________________________________<br>
>> Dsg-rich mailing list<br>
>> <a class="moz-txt-link-abbreviated"
href="mailto:Dsg-rich@jlab.org"
target="_blank" moz-do-not-send="true">Dsg-rich@jlab.org</a><br>
>> <a class="moz-txt-link-freetext"
href="https://mailman.jlab.org/mailman/listinfo/dsg-rich"
target="_blank" moz-do-not-send="true">https://mailman.jlab.org/mailman/listinfo/dsg-rich</a><br>
>
_______________________________________________<br>
> Rich mailing list<br>
> <a class="moz-txt-link-abbreviated"
href="mailto:Rich@jlab.org" target="_blank"
moz-do-not-send="true">Rich@jlab.org</a><br>
> <a class="moz-txt-link-freetext"
href="https://mailman.jlab.org/mailman/listinfo/rich"
target="_blank" moz-do-not-send="true">https://mailman.jlab.org/mailman/listinfo/rich</a><br>
</blockquote>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
Rich mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Rich@jlab.org" target="_blank" moz-do-not-send="true">Rich@jlab.org</a>
<a class="moz-txt-link-freetext" href="https://mailman.jlab.org/mailman/listinfo/rich" target="_blank" moz-do-not-send="true">https://mailman.jlab.org/mailman/listinfo/rich</a>
</pre>
</blockquote>
<br>
<br>
_______________________________________________<br>
Rich mailing list<br>
<a class="moz-txt-link-abbreviated"
href="mailto:Rich@jlab.org" target="_blank"
moz-do-not-send="true">Rich@jlab.org</a><br>
<a class="moz-txt-link-freetext"
href="https://mailman.jlab.org/mailman/listinfo/rich"
target="_blank" moz-do-not-send="true">https://mailman.jlab.org/mailman/listinfo/rich</a><br>
</div>
</div>
</blockquote>
<br>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>