[acg] Machine instability and oscillations

[Gary Croke]

Hi All,

I hope everyone has had a nice weekend.  It seems Ops is still struggling to find the source of machine instability with a roughly 80 second period.  Some seem particularly paranoid about control system processes (a few examples copied below).  I'm dubious, but to perform our due diligence in support of Ops, please think carefully about systems and applications you are involved with, and whether there is anything, functioning correctly or misbehaving, that could contribute.

Thank you,

Gary





Hi Gary,

Thanks. Maybe you are doing this but I am not asking you to look for oscillations in the archived data. Rather, search all applications and processes (all controls) that has ~80 s scan rate, PID loop, update rate, etc, similar to the example I sent you.

It is enough to have the name of  these, if any. Then we will take it from there.

Yours,
Riad.


-------- Forwarded Message --------
Subject:         80-second oscillation
Date:    Sat, 17 May 2025 09:53:06 -0400
From:    Stepan Stepanyan <stepanya at jlab.org>
To:      Jay Benesch <benesch at jlab.org>

Hi Jey,

I’m far from having any understanding of the cause of this 80-second
oscillation, but I remember some reported issues with slow control
response during startup. So, I wonder if communication delays—possibly
in the feedback loops of some controls in IOCs—could be contributing to
this unusual frequency.

Regards,
Stepan


Hi folks,

While listening to the 8 AM meeting this morning and thinking about
the 80s orbit oscillation, one question kept sticking with me: why
don't orbit locks take out this oscillation? The PID locks should
be fast enough to -- it is a slow oscillation. We could add PID
locks in e.g. 1S/1E/2E that should take out most of them before
the coupling in 2R.

Is it feasible that orbit locks somewhere (in the injector?) are part
of the problem? A mis-setting of some orbit lock PID setup might
create something like this slow oscillation.

In the larger picture, is there a list somewhere of what is being
checked so we can systematically check off ideas and items? Maybe
a thread in the BTLOG would be useful. Right now it seems that the
information is in people's heads so some ideas are being revisited:

https://logbooks.jlab.org/entry/4376279

Maybe the orbit lock idea is a revisit too and they were already checked.

                                                 -Todd





________________________________
From: acg <acg-bounces at jlab.org> on behalf of Gary Croke via acg <acg at jlab.org>
Sent: Thursday, May 15, 2025 9:50 AM
To: Gary Croke via acg <acg at jlab.org>
Subject: [acg] Machine instability and oscillations

Hi All,

Ops have been trying to track down the source of two different periodic instabilities in the machine.  They believe they've tracked a 13 second one to 0L03 cavity 8.  However there seems to also be an 80 second oscillation that they are still hunting.

If you can think of anything that could be the source of this, either in software or the systems you are familiar with, please bring it forward.

Thank you,

Gary

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/acg/attachments/20250518/61797ebb/attachment.htm>