<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Had an issue on Friday Sept. 3 where I rebooted iocsoftnl05rf for a change and myRestore restored from Aug 19th instead</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
of Sept 3 right before the reboot. This prevented OPS from restoring beam.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Worked through the issue with Theo and found that the archiver stopped archiving some EPICS PV's that were part</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
of the active Restore Set for iocsoftnl05rf. These were random signals from the restore set with no noticeable pattern. The archiver</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
has an algorithm were if it thinks a PV has gone away after some period of time it stops archiving it. However these PV's</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
never went away. This was the first time I have seen this happen.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Controls_dept <controls_dept-bounces@jlab.org> on behalf of Gary Croke <gcroke@jlab.org><br>
<b>Sent:</b> Tuesday, September 7, 2021 11:13 AM<br>
<b>To:</b> Adam Carpenter <adamc@jlab.org>; Michele Joyce <erb@jlab.org>; Omar Garza <garza@jlab.org>; Theo Larrieu <theo@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Here's one from Friday that probably falls in this category. Failed on some caputs, although channels were available:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<a href="https://logbooks.jlab.org/entry/3900716" id="LPlnk129696">https://logbooks.jlab.org/entry/3900716</a><br>
</div>
<div>
<div id="x_appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Another attempt shortly after succeeded:<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<a href="https://logbooks.jlab.org/entry/3900725" id="LPlnk202857">https://logbooks.jlab.org/entry/3900725</a></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Adam Carpenter <adamc@jlab.org><br>
<b>Sent:</b> Tuesday, September 7, 2021 11:06 AM<br>
<b>To:</b> Michele Joyce <erb@jlab.org>; Gary Croke <gcroke@jlab.org>; Omar Garza <garza@jlab.org>; Theo Larrieu <theo@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: Network timeouts being reported from various applications -- feels ominous.</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Omar,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Do you remember which IOCs these were? PC104s or something else? Was it part of a mass reboot or just a single IOC?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Adam</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_Signature">
<div>
<meta content="text/html; charset=UTF-8">
<div id="x_x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px; margin-top:0; margin-bottom:0">
</p>
<div><br>
Adam Carpenter<br>
Accelerator Operations Software Department<br>
Thomas Jefferson National Accelerator Facility</div>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin-top:0px; margin-bottom:0px">
</p>
</div>
</div>
</div>
</div>
<div id="x_x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Ace <ace-bounces@jlab.org> on behalf of Michele Joyce <erb@jlab.org><br>
<b>Sent:</b> Tuesday, September 7, 2021 10:25 AM<br>
<b>To:</b> Gary Croke <gcroke@jlab.org>; Omar Garza <garza@jlab.org>; Theo Larrieu <theo@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: [Ace] Network timeouts being reported from various applications -- feels ominous.</font>
<div> </div>
</div>
<div style="word-wrap:break-word" lang="EN-US">
<div class="x_x_x_WordSection1">
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
Is anything machine dependent?</p>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif; margin-bottom:12.0pt">
<b><span style="font-size:12.0pt; color:black">From: </span></b><span style="font-size:12.0pt; color:black">Controls_dept <controls_dept-bounces@jlab.org> on behalf of Gary Croke <gcroke@jlab.org><br>
<b>Date: </b>Sunday, September 5, 2021 at 11:09 AM<br>
<b>To: </b>Omar Garza <garza@jlab.org>, Theo Larrieu <theo@jlab.org>, ace@jlab.org <ace@jlab.org>, controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject: </b>Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span></p>
</div>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<span style="font-size:12.0pt; color:black">What's the suspicion? Has there been a big uptick in network traffic recently, or have we gradually been approaching some limit? Or just some failing hardware somewhere?</span></p>
</div>
<div>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_x_x_MsoNormal" style="margin:0in; font-size:11pt; font-family:"Calibri",sans-serif; text-align:center" align="center">
<hr width="98%" size="2" align="center">
</div>
<div id="x_x_x_divRplyFwdMsg">
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<b><span style="color:black">From:</span></b><span style="color:black"> Controls_dept <controls_dept-bounces@jlab.org> on behalf of Omar Garza <garza@jlab.org><br>
<b>Sent:</b> Saturday, September 4, 2021 7:58 PM<br>
<b>To:</b> Theo Larrieu <theo@jlab.org>; ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.</span>
</p>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
</div>
</div>
<div>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<span style="font-family:"Arial",sans-serif; color:black">This nay also be related
</span></p>
</div>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<span style="font-family:"Arial",sans-serif; color:black">In the past weeks, we have also performed hard reboots on iocs with multiple tries failing to complete network connections .</span></p>
</div>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<span style="font-family:"Arial",sans-serif; color:black">2-3 hard reboots to succeed.</span></p>
</div>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<span style="font-family:"Arial",sans-serif; color:black">Omar</span></p>
</div>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
</div>
<div id="x_x_x_x_ms-outlook-mobile-signature">
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
Sent from my Verizon, Samsung Galaxy smartphone<br>
Get <a href="https://aka.ms/AAb9ysg">Outlook for Android</a></p>
</div>
<div class="x_x_x_MsoNormal" style="margin:0in; font-size:11pt; font-family:"Calibri",sans-serif; text-align:center" align="center">
<hr width="98%" size="2" align="center">
</div>
<div id="x_x_x_x_divRplyFwdMsg">
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<b><span style="color:black">From:</span></b><span style="color:black"> Ace <ace-bounces@jlab.org> on behalf of Theo Larrieu <theo@jlab.org><br>
<b>Sent:</b> Friday, September 3, 2021 5:27:32 PM<br>
<b>To:</b> ace@jlab.org <ace@jlab.org>; controls_dept@jlab.org <controls_dept@jlab.org><br>
<b>Subject:</b> [Ace] Network timeouts being reported from various applications -- feels ominous.</span>
</p>
<div>
<p class="x_x_x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
</div>
</div>
<div>
<div>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
Observing that in the past 24 hours there have been several seemingly unrelated applications having trouble making connections to EPICS signals.</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
Cavity History</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<a href="https://logbooks.jlab.org/entry/3900743">https://logbooks.jlab.org/entry/3900743</a></p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
FOPT</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<a href="https://logbooks.jlab.org/entry/3900317">https://logbooks.jlab.org/entry/3900317</a></p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
myRestore</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
<a href="https://logbooks.jlab.org/entry/3900705">https://logbooks.jlab.org/entry/3900705</a></p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
It might be wise to start looking for signs of distress in name server, softIOC hosts, network switches, etc.</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
-Theo</p>
<p class="x_x_x_xxmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px; margin:0in; font-size:11pt; font-family:"Calibri",sans-serif">
</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>