[Dsg-rich] [Rich] FPGA temp alarm
Chris Cuevas
cuevas at jlab.org
Tue Oct 16 15:56:57 EDT 2018
Hi,
Software can stop for a variety of reasons, but good to know that there
is a default watchdog timer on the cRIO. How much time elapses before
software stoppage is detected and the relay opens?
I am not concerned about a power off sequence because I know the CAEN
mainframe and modules will latch OFF until new commands are issued.
Thanks,
-Chris
~~~~~~~~~~~~~~~~~
On 10/16/2018 3:11 PM, Tyler Lemon wrote:
> If the software stops running, the relay goes into its default open
> state, disabling the CAEN.
>
> The most likely cause of the cRIO's software stopping is a power
> outage. In the event of a power outage, there will be a sequence of
> events (compressor losing power, airflow/tank pressure dropping,
> interlock tripping) that will disable the CAEN. All components of the
> hardware interlock system are on UPS, so the hardware interlock
> components will stay alive long enough after the power outage to
> disable the CAEN. When the power is restored, the cRIO is set up to
> automatically run the interlock program on boot. On power restore, the
> CAEN will remain disabled until all normal operating conditions
> (airflow, buffer tank pressure, temperatures) are restored and the
> CAEN's interlocks are cleared.
>
> --Tyler
> ------------------------------------------------------------------------
> *From: *"Chris Cuevas" <cuevas at jlab.org>
> *To: *"Tyler Lemon" <tlemon at jlab.org>
> *Cc: *"Valery Kubarovsky" <vpk at jlab.org>, "Mirazita Marco"
> <Marco.Mirazita at lnf.infn.it>, rich at jlab.org, "dsg-rich"
> <dsg-rich at jlab.org>
> *Sent: *Tuesday, October 16, 2018 2:53:02 PM
> *Subject: *Re: [Rich] [Dsg-rich] FPGA temp alarm
>
> Hi Tyler,
>
> So the cRIO software must always be running to detect out of limit
> signals from the front end sensors? What happens if the software stops
> running? Is there a default timeout[watchdog] setting that will open
> the final relay contact?
>
> Thanks for the feedback,
>
> -Chris
>
> ~~~~~~~~~~~
>
>
> On 10/16/2018 1:52 PM, Tyler Lemon wrote:
>
> Chris,
>
> For the hardware interlocks, if any of the cRIO-based system's
> sensors (RTDs, humidity, airflow, air pressure, N2 flow; sensors
> not dependent on FPGA DAQ readout) go out of limit, the cRIO
> disables HV and LV by opening a relay contact to the Interlock
> port on the CAEN mainframe.
>
> This disables HV and LV and will only allow HV and LV to be
> re-enabled if the interlock is clear and a user has reset the
> interlock.
>
> Regards,
> Tyler
>
>
>
>
> ------------------------------------------------------------------------
> *From: *"Chris Cuevas" <cuevas at jlab.org>
> *To: *"Valery Kubarovsky" <vpk at jlab.org>, "Mirazita Marco"
> <Marco.Mirazita at lnf.infn.it>
> *Cc: *rich at jlab.org, "dsg-rich" <dsg-rich at jlab.org>
> *Sent: *Tuesday, October 16, 2018 1:38:14 PM
> *Subject: *Re: [Rich] [Dsg-rich] FPGA temp alarm
>
> Hi All,
>
> The FPGAs have a firmware set point for the die temperature. Ben
> can verify this set point, but it sure sounds like the 'interlock'
> systems rely on software code. Are there watchdog timers set up in
> case the code stops running?
>
> For a true hardware interlock, one that does not depend on
> software, there should be temperature probes coupled directly to a
> comparator circuit which latches off the Low_Voltage outputs to
> the FPGA boards. From these email threads, it sounds like all the
> temperature devices are monitored with software, and if a software
> set point threshold is violated, then the Low_Voltage is turned
> OFF. If the monitored temperature devices go below the software
> set point threshold, is the Low_Voltage power reapplied automatically?
>
> Thanks,
>
> -Chris
>
> ~~~~~~~~~~~~~~~~~
>
>
> On 10/16/2018 9:40 AM, Valery Kubarovsky wrote:
>
> The detector (actually LV in this particular case, because the
> HV was off at that moment)
> was shut down by software interlock that monitors the FPGS
> temperature.
>
> ------------------------------------------------------------------------
>
> *From: *"Marco Mirazita" <Marco.Mirazita at lnf.infn.it>
> *To: *"Tyler Lemon" <tlemon at jlab.org>
> *Cc: *"Valery Kubarovsky" <vpk at jlab.org>, rich at jlab.org,
> "dsg-rich" <dsg-rich at jlab.org>
> *Sent: *Tuesday, October 16, 2018 9:37:41 AM
> *Subject: *Re: [Rich] [Dsg-rich] FPGA temp alarm
>
> But we have a soft interlock shutting down the system if fpga
> temperature goes above 75 deg.
>
>
> Il 2018-10-16 14:24 Tyler Lemon ha scritto:
> > Hello Marco,
> >
> > The hardware interlock system does not monitor FPGA
> temperatures. It
> > monitors the RTDs installed in the EP.
> >
> > The hardware interlock system did not trip off the
> electronics because
> > the strange temperature readings were only seen on the FPGA
> > temperature readouts.
> >
> > Best regards,
> > Tyler
> >
> > -------------------------
> >
> > FROM: "Mirazita Marco" <Marco.Mirazita at lnf.infn.it>
> > TO: "Valery Kubarovsky" <vpk at jlab.org>
> > CC: rich at jlab.org, "dsg-rich" <dsg-rich at jlab.org>
> > SENT: Tuesday, October 16, 2018 8:16:06 AM
> > SUBJECT: Re: [Rich] [Dsg-rich] FPGA temp alarm
> >
> > Hi Valery,
> > thank you, in fact my main worry was that the interlock
> didn't shut
> > down
> > the electronics.
> > It would be anyway good to understand why we had this event.
> > I saw from the strip charts that also the LV currents
> and voltages
> > have
> > frequent random spikes.
> > Marco
> >
> > Il 2018-10-16 13:55 Valery Kubarovsky ha scritto:
> >> Marco,
> >> If you take a more careful look you will find out that
> the LV was
> > shut
> >> down.
> >> The HV was off. We investigate the reason. It is the
> first time we
> > had
> >> such an event.
> >> Probably it was connected with the software update that
> was done
> >> approximately at the same time.
> >> Regards,
> >> Valery
> >>
> >> -------------------------
> >>
> >>> FROM: "Marco Mirazita" <Marco.Mirazita at lnf.infn.it>
> >>> TO: dsg-rich at jlab.org, rich at jlab.org
> >>> SENT: Tuesday, October 16, 2018 4:03:24 AM
> >>> SUBJECT: [Dsg-rich] FPGA temp alarm
> >>
> >>> Hi all,
> >>> yesterday at about 4:20 pm jlab time I received
> several alarm
> >>> messages
> >>> saying that the FPGA temperatures reached values
> around 115 deg.
> >>> Since it looked like a readout error, I checked the
> strip chart,
> >>> where
> >>> there is actually a spike in the readout at that time,
> but to
> > values
> >>> of
> >>> the order of -10^5. See the attached plot.
> >>> So, it is clear that it was a readout error, but it is
> strange that
> >>> the
> >>> alarm message and epics report different values.
> >>> Also, according to the logbook, the electronics was
> not shut down
> > by
> >>> the
> >>> interlocks, as should have happened if the temperature
> really went
> >>> above
> >>> 100 deg.
> >>> So, perhaps the values reported in the alarm messages
> are not the
> >>> correct ones?
> >>> Marco
> >>>
> >>> _______________________________________________
> >>> Dsg-rich mailing list
> >>> Dsg-rich at jlab.org
> >>> https://mailman.jlab.org/mailman/listinfo/dsg-rich
> >>
> >> _______________________________________________
> >> Dsg-rich mailing list
> >> Dsg-rich at jlab.org
> >> https://mailman.jlab.org/mailman/listinfo/dsg-rich
> > _______________________________________________
> > Rich mailing list
> > Rich at jlab.org
> > https://mailman.jlab.org/mailman/listinfo/rich
>
>
>
> _______________________________________________
> Rich mailing list
> Rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/rich
>
>
>
> _______________________________________________
> Rich mailing list
> Rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/rich
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20181016/11ed8fb8/attachment-0002.html>
More information about the Dsg-rich
mailing list