[Ace] Network timeouts being reported from various applications -- feels ominous.

[Adam Carpenter]

Omar,

Do you remember which IOCs these were?  PC104s or something else?  Was it part of a mass reboot or just a single IOC?

Thanks,
Adam


Adam Carpenter
Accelerator Operations Software Department
Thomas Jefferson National Accelerator Facility

________________________________
From: Ace <ace-bounces at jlab.org> on behalf of Michele Joyce <erb at jlab.org>
Sent: Tuesday, September 7, 2021 10:25 AM
To: Gary Croke <gcroke at jlab.org>; Omar Garza <garza at jlab.org>; Theo Larrieu <theo at jlab.org>; ace at jlab.org <ace at jlab.org>; controls_dept at jlab.org <controls_dept at jlab.org>
Subject: Re: [Ace] Network timeouts being reported from various applications -- feels ominous.


Is anything machine dependent?





From: Controls_dept <controls_dept-bounces at jlab.org> on behalf of Gary Croke <gcroke at jlab.org>
Date: Sunday, September 5, 2021 at 11:09 AM
To: Omar Garza <garza at jlab.org>, Theo Larrieu <theo at jlab.org>, ace at jlab.org <ace at jlab.org>, controls_dept at jlab.org <controls_dept at jlab.org>
Subject: Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.

What's the suspicion?  Has there been a big uptick in network traffic recently, or have we gradually been approaching some limit?  Or just some failing hardware somewhere?



________________________________

From: Controls_dept <controls_dept-bounces at jlab.org> on behalf of Omar Garza <garza at jlab.org>
Sent: Saturday, September 4, 2021 7:58 PM
To: Theo Larrieu <theo at jlab.org>; ace at jlab.org <ace at jlab.org>; controls_dept at jlab.org <controls_dept at jlab.org>
Subject: Re: [Controls_dept] Network timeouts being reported from various applications -- feels ominous.



This nay also be related

In the past weeks, we have also performed hard reboots on iocs with  multiple tries failing to complete network connections  .

2-3 hard reboots to  succeed.

Omar



Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________

From: Ace <ace-bounces at jlab.org> on behalf of Theo Larrieu <theo at jlab.org>
Sent: Friday, September 3, 2021 5:27:32 PM
To: ace at jlab.org <ace at jlab.org>; controls_dept at jlab.org <controls_dept at jlab.org>
Subject: [Ace] Network timeouts being reported from various applications -- feels ominous.





Observing that in the past 24 hours there have been several seemingly unrelated applications having trouble making connections to EPICS signals.



Cavity History

https://logbooks.jlab.org/entry/3900743



FOPT

https://logbooks.jlab.org/entry/3900317



myRestore

https://logbooks.jlab.org/entry/3900705



It might be wise to start looking for signs of distress in name server, softIOC hosts, network switches, etc.



-Theo


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/ace/attachments/20210907/7b5aa831/attachment.html>