<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.xxxmsonormal, li.xxxmsonormal, div.xxxmsonormal
{mso-style-name:x_x_x_msonormal;
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xxxxxmsonormal, li.xxxxxmsonormal, div.xxxxxmsonormal
{mso-style-name:x_x_x_xxmsonormal;
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Just to start a list of the reported problems, here is what I could gather from the emails.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">iochallmag - myrestore caput issue<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">iocnl4 - cavity bypass failure<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">Hall B line - unkown fopt problem<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">NL05 rf - myrestore issue - new zone new rf type hardware, embedded ioc's with soft ioc.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif">Don't see any network related errors/issues/utilization with any of the associated network ports or network as a whole. I will continue to keep an eye on things.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please reply if there is anything I may have missed.<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Brad Cumbia<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Accelerator Network Engineer<o:p></o:p></p>
<p class="MsoNormal">Accelerator SysAdmin Group<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thomas Jefferson National Accelerator Facility<o:p></o:p></p>
<p class="MsoNormal">12000 Jefferson Avenue<o:p></o:p></p>
<p class="MsoNormal">Newport News, Virginia 23606<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal">Phone (757)269-5839<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Controls_dept <controls_dept-bounces@jlab.org> on behalf of Scott Higgins <higgins@jlab.org><br>
<b>Date: </b>Tuesday, September 7, 2021 at 11:24 AM<br>
<b>To: </b>Gary Croke <gcroke@jlab.org>, Adam Carpenter <adamc@jlab.org>, Michele Joyce <erb@jlab.org>, Omar Garza <garza@jlab.org>, Theo Larrieu <theo@jlab.org>, ace@jlab.org <ace@jlab.org>, controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject: </b>Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Had an issue on Friday Sept. 3 where I rebooted iocsoftnl05rf for a change and myRestore restored from Aug 19th instead<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">of Sept 3 right before the reboot. This prevented OPS from restoring beam.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Worked through the issue with Theo and found that the archiver stopped archiving some EPICS PV's that were part<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">of the active Restore Set for iocsoftnl05rf. These were random signals from the restore set with no noticeable pattern. The archiver<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">has an algorithm were if it thinks a PV has gone away after some period of time it stops archiving it. However these PV's<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">never went away. This was the first time I have seen this happen.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Controls_dept <controls_dept-bounces@jlab.org> on behalf of Gary Croke <gcroke@jlab.org><br>
<b>Sent:</b> Tuesday, September 7, 2021 11:13 AM<br>
<b>To:</b> Adam Carpenter <adamc@jlab.org>; Michele Joyce <erb@jlab.org>; Omar Garza <garza@jlab.org>; Theo Larrieu <theo@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Here's one from Friday that probably falls in this category. Failed on some caputs, although channels were available:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><a href="https://logbooks.jlab.org/entry/3900716">https://logbooks.jlab.org/entry/3900716</a><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Another attempt shortly after succeeded:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><a href="https://logbooks.jlab.org/entry/3900725">https://logbooks.jlab.org/entry/3900725</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Adam Carpenter <adamc@jlab.org><br>
<b>Sent:</b> Tuesday, September 7, 2021 11:06 AM<br>
<b>To:</b> Michele Joyce <erb@jlab.org>; Gary Croke <gcroke@jlab.org>; Omar Garza <garza@jlab.org>; Theo Larrieu <theo@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Omar,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Do you remember which IOCs these were? PC104s or something else? Was it part of a mass reboot or just a single IOC?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Adam<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div id="x_x_Signature">
<div>
<div id="x_x_divtagdefaultwrapper">
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><br>
Adam Carpenter<br>
Accelerator Operations Software Department<br>
Thomas Jefferson National Accelerator Facility<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Ace <ace-bounces@jlab.org> on behalf of Michele Joyce <erb@jlab.org><br>
<b>Sent:</b> Tuesday, September 7, 2021 10:25 AM<br>
<b>To:</b> Gary Croke <gcroke@jlab.org>; Omar Garza <garza@jlab.org>; Theo Larrieu <theo@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: [Ace] Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxxmsonormal">Is anything machine dependent?<o:p></o:p></p>
<p class="xxxmsonormal"> <o:p></o:p></p>
<p class="xxxmsonormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xxxmsonormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Controls_dept <controls_dept-bounces@jlab.org> on behalf of Gary Croke <gcroke@jlab.org><br>
<b>Date: </b>Sunday, September 5, 2021 at 11:09 AM<br>
<b>To: </b>Omar Garza <garza@jlab.org>, Theo Larrieu <theo@jlab.org>, ace@jlab.org <ace@jlab.org>, controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject: </b>Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="font-size:12.0pt;color:black">What's the suspicion? Has there been a big uptick in network traffic recently, or have we gradually been approaching some limit? Or just some failing hardware somewhere?</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="xxxmsonormal"><span style="font-size:12.0pt;color:black"> </span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_divRplyFwdMsg">
<p class="xxxmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Controls_dept <controls_dept-bounces@jlab.org> on behalf of Omar Garza <garza@jlab.org><br>
<b>Sent:</b> Saturday, September 4, 2021 7:58 PM<br>
<b>To:</b> Theo Larrieu <theo@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="xxxmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxxmsonormal"><span style="font-family:"Arial",sans-serif;color:black">This nay also be related
</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="font-family:"Arial",sans-serif;color:black">In the past weeks, we have also performed hard reboots on iocs with multiple tries failing to complete network connections .</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="font-family:"Arial",sans-serif;color:black">2-3 hard reboots to succeed.</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"><span style="font-family:"Arial",sans-serif;color:black">Omar</span><o:p></o:p></p>
</div>
<div>
<p class="xxxmsonormal"> <o:p></o:p></p>
</div>
<div id="x_x_x_x_ms-outlook-mobile-signature">
<p class="xxxmsonormal">Sent from my Verizon, Samsung Galaxy smartphone<br>
Get <a href="https://aka.ms/AAb9ysg">Outlook for Android</a><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_x_x_x_divRplyFwdMsg">
<p class="xxxmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Ace <ace-bounces@jlab.org> on behalf of Theo Larrieu <theo@jlab.org><br>
<b>Sent:</b> Friday, September 3, 2021 5:27:32 PM<br>
<b>To:</b> ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> [Ace] Network timeouts being reported from various applications -- feels ominous.</span>
<o:p></o:p></p>
<div>
<p class="xxxmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">Observing that in the past 24 hours there have been several seemingly unrelated applications having trouble making connections to EPICS signals.<o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">Cavity History<o:p></o:p></p>
<p class="xxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900743">https://logbooks.jlab.org/entry/3900743</a><o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">FOPT<o:p></o:p></p>
<p class="xxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900317">https://logbooks.jlab.org/entry/3900317</a><o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">myRestore<o:p></o:p></p>
<p class="xxxxxmsonormal"><a href="https://logbooks.jlab.org/entry/3900705">https://logbooks.jlab.org/entry/3900705</a><o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">It might be wise to start looking for signs of distress in name server, softIOC hosts, network switches, etc.<o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
<p class="xxxxxmsonormal">-Theo<o:p></o:p></p>
<p class="xxxxxmsonormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>