<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This should definitely help reduce the number of hung consoles and I'm all for it. But there's something else that happens between the ioc and the console server that locks up the connection and is resolvable only by an ioc reboot.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
For example yesterday 6 of the 14 iocs in injsbcon4 did not have working consoles. Anthony rebooted the console server and that number dropped to 4. Now it is back up to 5. I would love to find out what causes this, but it's very difficult to investigate once
the console is locked up.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The tool I run to reinitialize CA security attempts to use the console of every operational ioc and logs the ones with problems while ignoring iocs it can't ping. Earlier today that number was 37 locked up out of 613 total iocs. Now it's 46. If we really want
to cut down on this it might be useful to run it every maintenance day to generate a report and reboot the console servers and/or iocs that aren't working (excepting cryo, etc.)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--Brian<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Ace <ace-bounces@jlab.org> on behalf of Anthony Cuffe <cuffe@jlab.org><br>
<b>Sent:</b> Friday, July 1, 2022 3:39 PM<br>
<b>To:</b> Gary Croke <gcroke@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org>; ace@jlab.org <ace@jlab.org><br>
<b>Subject:</b> Re: [Ace] [Controls_dept] Advise on rebooting all console servers</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">We do monitor them but there is really not way of knowing which ones have hung ports other than the connection attempts. Some of the problems are also the IOCs themselves. Since it seems to take many months and connections for the problems
to arise, I think a reboot once (or twice) a month will mostly resolve the issues. I can also provide a mechanism for developers to reboot the console servers, although this is a bit involved and I have not figured out the best way to allow this globally
for each of you.<br>
<br>
<br>
=========================<br>
Anthony Cuffe<br>
voice : 757 269-6213<br>
e-mail : cuffe@jlab.org<br>
=========================<br>
<br>
On 7/1/22 3:32 PM, Gary Croke wrote:<br>
> We usually discover several console problems in the lead up to, or in the early stages of a run. We either realize we can't connect to a console, or look for information in the logs that isn't there because the console stopped logging at some point. I think
anything that can be done to reduce instances of these problems is definitely worthwhile. If it's not easy to monitor the console servers, the proactively rebooting periodically sounds like a good idea to me.<br>
> <br>
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> *From:* Controls_dept <controls_dept-bounces@jlab.org> on behalf of Anthony Cuffe <cuffe@jlab.org><br>
> *Sent:* Friday, July 1, 2022 2:29 PM<br>
> *To:* controls_dept@jlab.org <controls_dept@jlab.org>; ace@jlab.org <ace@jlab.org><br>
> *Subject:* [Controls_dept] Advise on rebooting all console servers<br>
> I would like to setup a task to reboot all console servers on a regular schedule. This would be to address things like port hang ups. I propose doing this once a month on a Weds at around 5am. This way, the chances of anyone using one would be minimal
and if a console server does not reboot properly, one of us can address it when we arrive in the morning.<br>
> <br>
> Any suggestions or comments?<br>
> <br>
> <br>
> -- <br>
> =========================<br>
> Anthony Cuffe<br>
> voice : 757 269-6213<br>
> e-mail : cuffe@jlab.org<br>
> =========================<br>
> _______________________________________________<br>
> Controls_dept mailing list<br>
> Controls_dept@jlab.org<br>
> <a href="https://mailman.jlab.org/mailman/listinfo/controls_dept">https://mailman.jlab.org/mailman/listinfo/controls_dept</a> <<a href="https://mailman.jlab.org/mailman/listinfo/controls_dept">https://mailman.jlab.org/mailman/listinfo/controls_dept</a>><br>
<br>
_______________________________________________<br>
Ace mailing list<br>
Ace@jlab.org<br>
<a href="https://mailman.jlab.org/mailman/listinfo/ace">https://mailman.jlab.org/mailman/listinfo/ace</a></div>
</span></font></div>
</body>
</html>