[Halld-controls] [New Logentry] PXI Downtime

[beng at jlab.org]

Logentry Text:
--
Hovanes reported that the PXI was down, initial symptoms were that it wasn't even responding to pings.

I went to the hall with Tyler to investigate and initially we found it had nothing on the console indicating an error or why it wasn't running, but upon power cycling it we go the same console message as this comment: https://logbooks.jlab.org/comment/10881

We changed the controller to use a static IP for NIC1, disabled NIC2, and also upgraded the base software from LabVIEW 2015 to 2016.

Even after doing so we managed to get the same message and/or the controller would reboot itself a few times before going into safe mode (which was the same behavior we saw with 2015).

Thinking this might have something to do with the real-time executable we ran the VI manually from the project, this worked (once). Upon rebooting to test a real-time executable that had debugging disabled (which had the same behavior) running the VI manually resulted in the controller rebooting itself as well.

We redeployed the real-time executable that has debugging enabled and connected to the debug session, initially the VI was stuck initializing, but stopping and starting the VI allowed it to run successfully.

End result: we just rebooted it enough times for it to boot into a working state (without really changing anything from when we started).

As a side note: we rebooted the IOC for the PXI many times (as should be done if the PXI is restarted) and several times it got stuck (all indicators showing disconnected) after a soft reboot which required a hard reboot.



Possible things we could try:
- Add some indication if the VI is running or not (e.g. blink an LED)
- Look into running a serial line to the controller to see if it produces more useful debugging information than on the monitor
- The controller has some diagnostics for the hard drive and memory, but since it said they could take hours to complete we didn't run them

---

This is a plain text email for clients that cannot display HTML.  The full logentry can be found online at https://logbooks.jlab.org/entry/3474459
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-controls/attachments/20170602/064a9128/attachment.html>