<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xxxxmsonormal, li.xxxxmsonormal, div.xxxxmsonormal
{mso-style-name:x_xxxmsonormal;
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xxxxxxmsonormal, li.xxxxxxmsonormal, div.xxxxxxmsonormal
{mso-style-name:x_xxxxxmsonormal;
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">129.57.236.210:43608 is opsmyafb07</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Brad Cumbia<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Accelerator Network Engineer<o:p></o:p></p>
<p class="MsoNormal">Accelerator SysAdmin Group<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thomas Jefferson National Accelerator Facility<o:p></o:p></p>
<p class="MsoNormal">12000 Jefferson Avenue<o:p></o:p></p>
<p class="MsoNormal">Newport News, Virginia 23606<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal">Phone (757)269-5839<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Gary Croke <gcroke@jlab.org><br>
<b>Date: </b>Tuesday, September 7, 2021 at 3:56 PM<br>
<b>To: </b>Theo Larrieu <theo@jlab.org>, Brad Cumbia <cumbia@jlab.org>, Scott Higgins <higgins@jlab.org>, Adam Carpenter <adamc@jlab.org>, Michele Joyce <erb@jlab.org>, Omar Garza <garza@jlab.org>, ace@jlab.org <ace@jlab.org>, controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject: </b>Re: Network timeouts being reported from various applications -- feels ominous.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Not sure if this is related, but iocnl13diag went unresponsive around 13:47 today and had to be rebooted, here are some messages from the console log:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><Sep/07 13:46:19>vxwrks@iocnl13diag -> dbCa:exceptionCallback stat "Virtual circuit unresponsive" channel "unknown" context "opsbat3.acc.jlab.org:34694"
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><Sep/07 13:46:19> nativeType DBR_invalid requestType DBR_invalid nativeCount 0 requestCount 0 noReadAccess noWriteAccess<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><Sep/07 13:46:31>DB CA Link Exception: "Virtual circuit unresponsive", context "iocin1b.acc.jlab.org:5064"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><Sep/07 13:46:33>DB CA Link Exception: "Virtual circuit unresponsive", context "opsbat3.acc.jlab.org:5064"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><Sep/07 13:47:46>CAS: TCP send to "129.57.236.210:43608" failed because "errno = 0x43"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><Sep/07 13:47:46>rsrv: socket shutdown error was errno = 0x43<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><Sep/07 13:51:02>NTPTimeSync: NTP requests failing - errno = 0x710003<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Theo Larrieu <theo@jlab.org><br>
<b>Sent:</b> Tuesday, September 7, 2021 1:47 PM<br>
<b>To:</b> Brad Cumbia <cumbia@jlab.org>; Scott Higgins <higgins@jlab.org>; Gary Croke <gcroke@jlab.org>; Adam Carpenter <adamc@jlab.org>; Michele Joyce <erb@jlab.org>; Omar Garza <garza@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> RE: Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="color:#1F497D">The timeouts being reported via cavitybypass are wider than just iocnl4</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D">this morning to iocnl2:</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"><a href="https://logbooks.jlab.org/entry/3901900">https://logbooks.jlab.org/entry/3901900</a></span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D">Over the weekend the timeouts happened for both iocnl4 and iocsl4</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"><a href="https://logbooks.jlab.org/entry/3900887">https://logbooks.jlab.org/entry/3900887</a></span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"><a href="https://logbooks.jlab.org/entry/3900743">https://logbooks.jlab.org/entry/3900743</a>
</span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xmsonormal"><b>From:</b> Brad Cumbia <cumbia@jlab.org> <br>
<b>Sent:</b> Tuesday, September 7, 2021 12:12 PM<br>
<b>To:</b> Scott Higgins <higgins@jlab.org>; Gary Croke <gcroke@jlab.org>; Adam Carpenter <adamc@jlab.org>; Michele Joyce <erb@jlab.org>; Omar Garza <garza@jlab.org>; Theo Larrieu <theo@jlab.org>; ace@jlab.org; controls_dept@jlab.org<br>
<b>Subject:</b> Re: Network timeouts being reported from various applications -- feels ominous.<o:p></o:p></p>
</div>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">Just to start a list of the reported problems, here is what I could gather from the emails.<o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">iochallmag - myrestore caput issue</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">iocnl4 - cavity bypass failure</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">Hall B line - unkown fopt problem</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">NL05 rf - myrestore issue - new zone new rf type hardware, embedded ioc's with soft ioc.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">Don't see any network related errors/issues/utilization with any of the associated network ports or network as a whole. I will continue to keep an eye on things.</span><o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">Please reply if there is anything I may have missed.<o:p></o:p></p>
<div>
<div>
<div>
<p class="xmsonormal">-- <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal">Brad Cumbia<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">Accelerator Network Engineer<o:p></o:p></p>
<p class="xmsonormal">Accelerator SysAdmin Group<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">Thomas Jefferson National Accelerator Facility<o:p></o:p></p>
<p class="xmsonormal">12000 Jefferson Avenue<o:p></o:p></p>
<p class="xmsonormal">Newport News, Virginia 23606<o:p></o:p></p>
</div>
</div>
</div>
<p class="xmsonormal">Phone (757)269-5839<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xmsonormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Controls_dept <<a href="mailto:controls_dept-bounces@jlab.org">controls_dept-bounces@jlab.org</a>> on behalf of Scott Higgins <<a href="mailto:higgins@jlab.org">higgins@jlab.org</a>><br>
<b>Date: </b>Tuesday, September 7, 2021 at 11:24 AM<br>
<b>To: </b>Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>>, Adam Carpenter <<a href="mailto:adamc@jlab.org">adamc@jlab.org</a>>, Michele Joyce <<a href="mailto:erb@jlab.org">erb@jlab.org</a>>, Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>,
Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>, <a href="mailto:ace@jlab.org">
ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>, <a href="mailto:controls_dept@jlab.org">
controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject: </b>Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Had an issue on Friday Sept. 3 where I rebooted iocsoftnl05rf for a change and myRestore restored from Aug 19th instead</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">of Sept 3 right before the reboot. This prevented OPS from restoring beam.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Worked through the issue with Theo and found that the archiver stopped archiving some EPICS PV's that were part</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">of the active Restore Set for iocsoftnl05rf. These were random signals from the restore set with no noticeable pattern. The archiver</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">has an algorithm were if it thinks a PV has gone away after some period of time it stops archiving it. However these PV's</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">never went away. This was the first time I have seen this happen.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Controls_dept <<a href="mailto:controls_dept-bounces@jlab.org">controls_dept-bounces@jlab.org</a>> on behalf of Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>><br>
<b>Sent:</b> Tuesday, September 7, 2021 11:13 AM<br>
<b>To:</b> Adam Carpenter <<a href="mailto:adamc@jlab.org">adamc@jlab.org</a>>; Michele Joyce <<a href="mailto:erb@jlab.org">erb@jlab.org</a>>; Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>; Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>;
<a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>;
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Here's one from Friday that probably falls in this category. Failed on some caputs, although channels were available:</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"><a href="https://logbooks.jlab.org/entry/3900716">https://logbooks.jlab.org/entry/3900716</a></span><o:p></o:p></p>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Another attempt shortly after succeeded:</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"><a href="https://logbooks.jlab.org/entry/3900725">https://logbooks.jlab.org/entry/3900725</a></span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Adam Carpenter <<a href="mailto:adamc@jlab.org">adamc@jlab.org</a>><br>
<b>Sent:</b> Tuesday, September 7, 2021 11:06 AM<br>
<b>To:</b> Michele Joyce <<a href="mailto:erb@jlab.org">erb@jlab.org</a>>; Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>>; Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>; Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>;
<a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>;
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> Re: Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Omar,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Do you remember which IOCs these were? PC104s or something else? Was it part of a mass reboot or just a single IOC?</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Thanks,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black">Adam</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div id="x_x_x_Signature">
<div>
<div id="x_x_x_divtagdefaultwrapper">
<div>
<p class="xmsonormal"><span style="font-size:12.0pt;color:black"><br>
Adam Carpenter<br>
Accelerator Operations Software Department<br>
Thomas Jefferson National Accelerator Facility</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Ace <<a href="mailto:ace-bounces@jlab.org">ace-bounces@jlab.org</a>> on behalf of Michele Joyce <<a href="mailto:erb@jlab.org">erb@jlab.org</a>><br>
<b>Sent:</b> Tuesday, September 7, 2021 10:25 AM<br>
<b>To:</b> Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>>; Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>; Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>;
<a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>;
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> Re: [Ace] Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxxxmsonormal">Is anything machine dependent?<o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxmsonormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xxxxmsonormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Controls_dept <<a href="mailto:controls_dept-bounces@jlab.org">controls_dept-bounces@jlab.org</a>> on behalf of Gary Croke <<a href="mailto:gcroke@jlab.org">gcroke@jlab.org</a>><br>
<b>Date: </b>Sunday, September 5, 2021 at 11:09 AM<br>
<b>To: </b>Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>>, Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>,
<a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>,
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject: </b>Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-size:12.0pt;color:black">What's the suspicion? Has there been a big uptick in network traffic recently, or have we gradually been approaching some limit? Or just some failing hardware somewhere?</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="xxxxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_x_divRplyFwdMsg">
<p class="xxxxmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Controls_dept <<a href="mailto:controls_dept-bounces@jlab.org">controls_dept-bounces@jlab.org</a>> on behalf of Omar Garza <<a href="mailto:garza@jlab.org">garza@jlab.org</a>><br>
<b>Sent:</b> Saturday, September 4, 2021 7:58 PM<br>
<b>To:</b> Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>>; <a href="mailto:ace@jlab.org">
ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>; <a href="mailto:controls_dept@jlab.org">
controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="xxxxmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxxxmsonormal"><span style="font-family:"Arial",sans-serif;color:black">This nay also be related
</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-family:"Arial",sans-serif;color:black">In the past weeks, we have also performed hard reboots on iocs with multiple tries failing to complete network connections .</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-family:"Arial",sans-serif;color:black">2-3 hard reboots to succeed.</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"><span style="font-family:"Arial",sans-serif;color:black">Omar</span><o:p></o:p></p>
</div>
<div>
<p class="xxxxmsonormal"> <o:p></o:p></p>
</div>
<div id="x_x_x_x_x_ms-outlook-mobile-signature">
<p class="xxxxmsonormal">Sent from my Verizon, Samsung Galaxy smartphone<br>
Get <a href="https://aka.ms/AAb9ysg">Outlook for Android</a><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_x_x_divRplyFwdMsg">
<p class="xxxxmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Ace <<a href="mailto:ace-bounces@jlab.org">ace-bounces@jlab.org</a>> on behalf of Theo Larrieu <<a href="mailto:theo@jlab.org">theo@jlab.org</a>><br>
<b>Sent:</b> Friday, September 3, 2021 5:27:32 PM<br>
<b>To:</b> <a href="mailto:ace@jlab.org">ace@jlab.org</a> <<a href="mailto:ace@jlab.org">ace@jlab.org</a>>;
<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a> <<a href="mailto:controls_dept@jlab.org">controls_dept@jlab.org</a>><br>
<b>Subject:</b> [Ace] Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="xxxxmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxxmsonormal">Observing that in the past 24 hours there have been several seemingly unrelated applications having trouble making connections to EPICS signals.<o:p></o:p></p>
<p class="xxxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxxmsonormal">Cavity History<o:p></o:p></p>
<p class="xxxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900743">https://logbooks.jlab.org/entry/3900743</a><o:p></o:p></p>
<p class="xxxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxxmsonormal">FOPT<o:p></o:p></p>
<p class="xxxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900317">https://logbooks.jlab.org/entry/3900317</a><o:p></o:p></p>
<p class="xxxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxxmsonormal">myRestore<o:p></o:p></p>
<p class="xxxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900705">https://logbooks.jlab.org/entry/3900705</a><o:p></o:p></p>
<p class="xxxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxxmsonormal">It might be wise to start looking for signs of distress in name server, softIOC hosts, network switches, etc.<o:p></o:p></p>
<p class="xxxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxxmsonormal">-Theo<o:p></o:p></p>
<p class="xxxxxxmsonormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>