<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Anthony and I cleaned up some issues in opsbat3 around the time of the last email flurry. If these problems have since resolved it may have been the source.<span id="ms-outlook-android-cursor"></span></div>
<div><br>
</div>
<div id="ms-outlook-mobile-signature">Get <a href="https://aka.ms/AAb9ysg">Outlook for Android</a></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Brad Cumbia <cumbia@jlab.org><br>
<b>Sent:</b> Tuesday, September 14, 2021 5:00:11 PM<br>
<b>To:</b> Gary Croke <gcroke@jlab.org>; Theo Larrieu <theo@jlab.org>; Scott Higgins <higgins@jlab.org>; Adam Carpenter <adamc@jlab.org>; Michele Joyce <erb@jlab.org>; Omar Garza <garza@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: Network timeouts being reported from various applications -- feels ominous.</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:"Segoe UI"}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_xxxxmsonormal, li.x_xxxxmsonormal, div.x_xxxxmsonormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_xxxxxxmsonormal, li.x_xxxxxxmsonormal, div.x_xxxxxxmsonormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle22
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">Has anyone noticed or stumbled upon anymore of these timeouts or have they resolved themselves? I have been periodically check the Elog but have not seen any significant entries.</p>
<p class="x_MsoNormal"> </p>
<div>
<div>
<div>
<p class="x_MsoNormal">-- </p>
</div>
<div>
<p class="x_MsoNormal">Brad Cumbia</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Accelerator Network Engineer</p>
<p class="x_MsoNormal">Accelerator SysAdmin Group</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Thomas Jefferson National Accelerator Facility</p>
<p class="x_MsoNormal">12000 Jefferson Avenue</p>
<p class="x_MsoNormal">Newport News, Virginia 23606</p>
<p class="x_MsoNormal">Phone (757)269-5839</p>
</div>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">Gary Croke <gcroke@jlab.org><br>
<b>Date: </b>Tuesday, September 7, 2021 at 3:56 PM<br>
<b>To: </b>Theo Larrieu <theo@jlab.org>, Brad Cumbia <cumbia@jlab.org>, Scott Higgins <higgins@jlab.org>, Adam Carpenter <adamc@jlab.org>, Michele Joyce <erb@jlab.org>, Omar Garza <garza@jlab.org>, ace@jlab.org <ace@jlab.org>, controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject: </b>Re: Network timeouts being reported from various applications -- feels ominous.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Not sure if this is related, but iocnl13diag went unresponsive around 13:47 today and had to be rebooted, here are some messages from the console log:</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"><Sep/07 13:46:19>vxwrks@iocnl13diag -> dbCa:exceptionCallback stat "Virtual circuit unresponsive" channel "unknown" context "opsbat3.acc.jlab.org:34694"
</span></p>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"><Sep/07 13:46:19> nativeType DBR_invalid requestType DBR_invalid nativeCount 0 requestCount 0 noReadAccess noWriteAccess</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"><Sep/07 13:46:31>DB CA Link Exception: "Virtual circuit unresponsive", context "iocin1b.acc.jlab.org:5064"</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"><Sep/07 13:46:33>DB CA Link Exception: "Virtual circuit unresponsive", context "opsbat3.acc.jlab.org:5064"</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"><Sep/07 13:47:46>CAS: TCP send to "129.57.236.210:43608" failed because "errno = 0x43"</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"><Sep/07 13:47:46>rsrv: socket shutdown error was errno = 0x43</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"><Sep/07 13:51:02>NTPTimeSync: NTP requests failing - errno = 0x710003</span></p>
</div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Theo Larrieu <theo@jlab.org><br>
<b>Sent:</b> Tuesday, September 7, 2021 1:47 PM<br>
<b>To:</b> Brad Cumbia <cumbia@jlab.org>; Scott Higgins <higgins@jlab.org>; Gary Croke <gcroke@jlab.org>; Adam Carpenter <adamc@jlab.org>; Michele Joyce <erb@jlab.org>; Omar Garza <garza@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> RE: Network timeouts being reported from various applications -- feels ominous.</span>
</p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="color:#1F497D">The timeouts being reported via cavitybypass are wider than just iocnl4</span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"> </span></p>
<p class="x_xmsonormal"><span style="color:#1F497D">this morning to iocnl2:</span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"> </span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"><a href="https://logbooks.jlab.org/entry/3901900">https://logbooks.jlab.org/entry/3901900</a></span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"> </span></p>
<p class="x_xmsonormal"><span style="color:#1F497D">Over the weekend the timeouts happened for both iocnl4 and iocsl4</span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"> </span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"><a href="https://logbooks.jlab.org/entry/3900887">https://logbooks.jlab.org/entry/3900887</a></span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"><a href="https://logbooks.jlab.org/entry/3900743">https://logbooks.jlab.org/entry/3900743</a>
</span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"> </span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_xmsonormal"><b>From:</b> Brad Cumbia <cumbia@jlab.org> <br>
<b>Sent:</b> Tuesday, September 7, 2021 12:12 PM<br>
<b>To:</b> Scott Higgins <higgins@jlab.org>; Gary Croke <gcroke@jlab.org>; Adam Carpenter <adamc@jlab.org>; Michele Joyce <erb@jlab.org>; Omar Garza <garza@jlab.org>; Theo Larrieu <theo@jlab.org>; ace@jlab.org; controls_dept@jlab.org<br>
<b>Subject:</b> Re: Network timeouts being reported from various applications -- feels ominous.</p>
</div>
</div>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Just to start a list of the reported problems, here is what I could gather from the emails.</p>
<p class="x_xmsonormal"><span style="font-size:10.5pt; font-family:"Segoe UI",sans-serif">iochallmag - myrestore caput issue</span></p>
<p class="x_xmsonormal"><span style="font-size:10.5pt; font-family:"Segoe UI",sans-serif">iocnl4 - cavity bypass failure</span></p>
<p class="x_xmsonormal"><span style="font-size:10.5pt; font-family:"Segoe UI",sans-serif">Hall B line - unkown fopt problem</span></p>
<p class="x_xmsonormal"><span style="font-size:10.5pt; font-family:"Segoe UI",sans-serif">NL05 rf - myrestore issue - new zone new rf type hardware, embedded ioc's with soft ioc.</span></p>
<p class="x_xmsonormal"><span style="font-size:10.5pt; font-family:"Segoe UI",sans-serif"> </span></p>
<p class="x_xmsonormal"><span style="font-size:10.5pt; font-family:"Segoe UI",sans-serif">Don't see any network related errors/issues/utilization with any of the associated network ports or network as a whole. I will continue to keep an eye on things.</span></p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Please reply if there is anything I may have missed.</p>
<div>
<div>
<div>
<p class="x_xmsonormal">-- </p>
</div>
<div>
<p class="x_xmsonormal">Brad Cumbia</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Accelerator Network Engineer</p>
<p class="x_xmsonormal">Accelerator SysAdmin Group</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Thomas Jefferson National Accelerator Facility</p>
<p class="x_xmsonormal">12000 Jefferson Avenue</p>
<p class="x_xmsonormal">Newport News, Virginia 23606</p>
</div>
</div>
</div>
<p class="x_xmsonormal">Phone (757)269-5839</p>
<p class="x_xmsonormal"> </p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_xmsonormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">Controls_dept <<a href="mailto:controls_dept-bounces@jlab.org">controls_dept-bounces@jlab.org</a>> on behalf of Scott Higgins <<a href="mailto:higgins@jlab.org">higgins@jlab.org</a>><br>
<b>Date: </b>Tuesday, September 7, 2021 at 11:24 AM<br>
<b>To: </b>Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>>, Adam Carpenter <<a href="mailto:adamc@jlab.org">adamc@jlab.org</a>>, Michele Joyce <<a href="mailto:erb@jlab.org">erb@jlab.org</a>>, Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>,
Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>, <a href="mailto:ace@jlab.org">
ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>, <a href="mailto:controls_dept@jlab.org">
controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject: </b>Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Had an issue on Friday Sept. 3 where I rebooted iocsoftnl05rf for a change and myRestore restored from Aug 19th instead</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">of Sept 3 right before the reboot. This prevented OPS from restoring beam.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Worked through the issue with Theo and found that the archiver stopped archiving some EPICS PV's that were part</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">of the active Restore Set for iocsoftnl05rf. These were random signals from the restore set with no noticeable pattern. The archiver</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">has an algorithm were if it thinks a PV has gone away after some period of time it stops archiving it. However these PV's</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">never went away. This was the first time I have seen this happen.</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="x_xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Controls_dept <<a href="mailto:controls_dept-bounces@jlab.org">controls_dept-bounces@jlab.org</a>> on behalf of Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>><br>
<b>Sent:</b> Tuesday, September 7, 2021 11:13 AM<br>
<b>To:</b> Adam Carpenter <<a href="mailto:adamc@jlab.org">adamc@jlab.org</a>>; Michele Joyce <<a href="mailto:erb@jlab.org">erb@jlab.org</a>>; Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>; Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>;
<a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>;
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span>
</p>
<div>
<p class="x_xmsonormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Here's one from Friday that probably falls in this category. Failed on some caputs, although channels were available:</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"><a href="https://logbooks.jlab.org/entry/3900716">https://logbooks.jlab.org/entry/3900716</a></span></p>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Another attempt shortly after succeeded:</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"><a href="https://logbooks.jlab.org/entry/3900725">https://logbooks.jlab.org/entry/3900725</a></span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_divRplyFwdMsg">
<p class="x_xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Adam Carpenter <<a href="mailto:adamc@jlab.org">adamc@jlab.org</a>><br>
<b>Sent:</b> Tuesday, September 7, 2021 11:06 AM<br>
<b>To:</b> Michele Joyce <<a href="mailto:erb@jlab.org">erb@jlab.org</a>>; Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>>; Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>; Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>;
<a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>;
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> Re: Network timeouts being reported from various applications -- feels ominous.</span>
</p>
<div>
<p class="x_xmsonormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Omar,</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Do you remember which IOCs these were? PC104s or something else? Was it part of a mass reboot or just a single IOC?</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Thanks,</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Adam</span></p>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div id="x_x_x_x_Signature">
<div>
<div id="x_x_x_x_divtagdefaultwrapper">
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"><br>
Adam Carpenter<br>
Accelerator Operations Software Department<br>
Thomas Jefferson National Accelerator Facility</span></p>
</div>
</div>
</div>
</div>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_x_divRplyFwdMsg">
<p class="x_xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Ace <<a href="mailto:ace-bounces@jlab.org">ace-bounces@jlab.org</a>> on behalf of Michele Joyce <<a href="mailto:erb@jlab.org">erb@jlab.org</a>><br>
<b>Sent:</b> Tuesday, September 7, 2021 10:25 AM<br>
<b>To:</b> Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>>; Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>; Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>;
<a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>;
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> Re: [Ace] Network timeouts being reported from various applications -- feels ominous.</span>
</p>
<div>
<p class="x_xmsonormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xxxxmsonormal">Is anything machine dependent?</p>
<p class="x_xxxxmsonormal"> </p>
<p class="x_xxxxmsonormal"> </p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_xxxxmsonormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">Controls_dept <<a href="mailto:controls_dept-bounces@jlab.org">controls_dept-bounces@jlab.org</a>> on behalf of Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>><br>
<b>Date: </b>Sunday, September 5, 2021 at 11:09 AM<br>
<b>To: </b>Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>, Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>,
<a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>,
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject: </b>Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span></p>
</div>
<div>
<p class="x_xxxxmsonormal"><span style="font-size:12.0pt; color:black">What's the suspicion? Has there been a big uptick in network traffic recently, or have we gradually been approaching some limit? Or just some failing hardware somewhere?</span></p>
</div>
<div>
<div>
<p class="x_xxxxmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_x_x_divRplyFwdMsg">
<p class="x_xxxxmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Controls_dept <<a href="mailto:controls_dept-bounces@jlab.org">controls_dept-bounces@jlab.org</a>> on behalf of Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>><br>
<b>Sent:</b> Saturday, September 4, 2021 7:58 PM<br>
<b>To:</b> Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>; <a href="mailto:ace@jlab.org">
ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>; <a href="mailto:controls_dept@jlab.org">
controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span>
</p>
<div>
<p class="x_xxxxmsonormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xxxxmsonormal"><span style="font-family:"Arial",sans-serif; color:black">This nay also be related
</span></p>
</div>
<div>
<p class="x_xxxxmsonormal"><span style="font-family:"Arial",sans-serif; color:black">In the past weeks, we have also performed hard reboots on iocs with multiple tries failing to complete network connections .</span></p>
</div>
<div>
<p class="x_xxxxmsonormal"><span style="font-family:"Arial",sans-serif; color:black">2-3 hard reboots to succeed.</span></p>
</div>
<div>
<p class="x_xxxxmsonormal"><span style="font-family:"Arial",sans-serif; color:black">Omar</span></p>
</div>
<div>
<p class="x_xxxxmsonormal"> </p>
</div>
<div id="x_x_x_x_x_x_ms-outlook-mobile-signature">
<p class="x_xxxxmsonormal">Sent from my Verizon, Samsung Galaxy smartphone<br>
Get <a href="https://aka.ms/AAb9ysg">Outlook for Android</a></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_x_x_x_divRplyFwdMsg">
<p class="x_xxxxmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Ace <<a href="mailto:ace-bounces@jlab.org">ace-bounces@jlab.org</a>> on behalf of Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>><br>
<b>Sent:</b> Friday, September 3, 2021 5:27:32 PM<br>
<b>To:</b> <a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>;
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> [Ace] Network timeouts being reported from various applications -- feels ominous.</span>
</p>
<div>
<p class="x_xxxxmsonormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xxxxxxmsonormal"> </p>
<p class="x_xxxxxxmsonormal">Observing that in the past 24 hours there have been several seemingly unrelated applications having trouble making connections to EPICS signals.</p>
<p class="x_xxxxxxmsonormal"> </p>
<p class="x_xxxxxxmsonormal">Cavity History</p>
<p class="x_xxxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900743">https://logbooks.jlab.org/entry/3900743</a></p>
<p class="x_xxxxxxmsonormal"> </p>
<p class="x_xxxxxxmsonormal">FOPT</p>
<p class="x_xxxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900317">https://logbooks.jlab.org/entry/3900317</a></p>
<p class="x_xxxxxxmsonormal"> </p>
<p class="x_xxxxxxmsonormal">myRestore</p>
<p class="x_xxxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900705">https://logbooks.jlab.org/entry/3900705</a></p>
<p class="x_xxxxxxmsonormal"> </p>
<p class="x_xxxxxxmsonormal">It might be wise to start looking for signs of distress in name server, softIOC hosts, network switches, etc.</p>
<p class="x_xxxxxxmsonormal"> </p>
<p class="x_xxxxxxmsonormal">-Theo</p>
<p class="x_xxxxxxmsonormal"> </p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>