[dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Mon Jul 14 11:31:27 EDT 2025

Hi Tyler,
It seems like the cRIO spontaneously fixed itself; see https://logbooks.jlab.org/entry/4416371
I am not sure that this is the real fix of the problem.
Are you going to take a look during maintenance day this Thursday?
Thanks,
Valery

________________________________
From: Tyler Lemon <tlemon at jlab.org>
Sent: Monday, July 14, 2025 11:01
To: Christopher Dilks <dilks at jlab.org>; Valery Kubarovsky <vpk at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Hi Chris,

Brian saw that the EPICS part of the interlock program was hung up for some of the gas system PVs, so that might be the cause of those PVs appearing frozen.

We can remotely restart just the EPICS portion of the program, but I am hesitant to do that over the short access period since other PVs are still working.

-Tyler
________________________________
From: Christopher Dilks <dilks at jlab.org>
Sent: Monday, July 14, 2025 10:50 AM
To: Tyler Lemon <tlemon at jlab.org>; Valery Kubarovsky <vpk at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Hi Tyler,

Thanks for taking a look! I can ask Nathan if there's anything we could do in EPICS.

By the way, Thursday Jul. 17th will be a maintenance day.

Cheers,
Chris

Christopher Dilks

Jefferson Lab Staff Scientist

Office: B122

Phone: 757-269-7750

Email: dilks at jlab.org

________________________________
From: Tyler Lemon <tlemon at jlab.org>
Sent: Monday, July 14, 2025 09:48
To: Christopher Dilks <dilks at jlab.org>; Valery Kubarovsky <vpk at jlab.org>; Tyler Lemon <tlemon at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Hi Chris,

The raw voltage readback for those channels are updating.
Attached are two screenshots of the raw values taken about one minute apart.

AI0 is N2 flow
AI2 is Airflow 1
AI3 is Airflow 2
CH0 is air pressure

You can see that the values for the channels are different in the two screenshots.

So the cRIO is correctly monitoring the airflows, N2 flows, and air pressure. It might be something with an EPICS deadband making it look like those PVs aren't updating.

-Tyler
________________________________
From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of Tyler Lemon via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Sent: Monday, July 14, 2025 9:06 AM
To: Christopher Dilks <dilks at jlab.org>; Valery Kubarovsky <vpk at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Hi Chris,

I‘m not sure if I will be able to make it around noon to go down to the hall.
I’m currently at the mechanic for my car and probably will be for most of the morning.

I can take a look remotely, though.

-Tyler

________________________________
From: Christopher Dilks <dilks at jlab.org>
Sent: Monday, July 14, 2025 8:32:29 AM
To: Valery Kubarovsky <vpk at jlab.org>; Tyler Lemon <tlemon at jlab.org>
Cc: Brian Eng <beng at jlab.org>; Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [EXTERNAL] Re: [dsg-hallb_rich] RICH S4 CRIO heartbeat stopped

Dear Tyler,

There will be a 20-30 minute access opportunity around noon today (for ECAL work). I'm not sure if that's enough time to take a look at the cRIO, but it is an access opportunity.

Also, I noticed that some of the S4 PVs, namely airflow, air pressure, and N2 flow, look like they are frozen; the heartbeat is beating though.

Cheers,
Chris

Christopher Dilks

Jefferson Lab Staff Scientist

Office: B122

Phone: 757-269-7750

Email: dilks at jlab.org

________________________________
From: Christopher Dilks <dilks at jlab.org>
Sent: Saturday, July 12, 2025 06:43
To: Valery Kubarovsky <vpk at jlab.org>; Tyler Lemon <tlemon at jlab.org>
Cc: Brian Eng <beng at jlab.org>; Marco.Mirazita at lnf.infn.it <marco.mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [EXTERNAL] Re: [dsg-hallb_rich] RICH S4 CRIO heartbeat stopped

Dear all,

It seems like the cRIO spontaneously fixed itself; see https://logbooks.jlab.org/entry/4416371

Thanks,
Chris

Christopher Dilks

Jefferson Lab Staff Scientist

Office: B122

Phone: 757-269-7750

Email: dilks at jlab.org

________________________________
From: Marco Mirazita <Marco.Mirazita at lnf.infn.it>
Sent: Thursday, July 10, 2025 10:06
To: Valery Kubarovsky <vpk at jlab.org>
Cc: Brian Eng <beng at jlab.org>; Christopher Dilks <dilks at jlab.org>; Tyler Lemon <tlemon at jlab.org>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: [EXTERNAL] Re: [dsg-hallb_rich] RICH S4 CRIO heartbeat stopped

Hello,
I agree with Valery, no reason to stop the data taking, but in case
please be ready.
Thank you,
Marco

Il 2025-07-10 16:04 Valery Kubarovsky via dsg-hallb_rich ha scritto:
> I understand this, and I have no intention of interrupting data
> taking. Nevertheless, if there is significant downtime, I would prefer
> to take the opportunity to fix the cRio. Please be prepared.
>
> -------------------------
>
>  From: Brian Eng <beng at jlab.org>
> Sent: Thursday, July 10, 2025 09:37
> To: Valery Kubarovsky <vpk at jlab.org>; Christopher Dilks
> <dilks at jlab.org>; Tyler Lemon <tlemon at jlab.org>
> Cc: Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Subject: Re: RICH S4 CRIO heartbeat stopped
>
>  It can probably wait until the next planned maintenance window on
> 7/17; at the very least I don't think this is something the beam
> should be stopped for as troubleshooting can take longer than expected
> if it isn't something obvious/simple.
>
>  Since we know there's still flow to S1, S4 should also be okay.
>
> -------------------------
>
>  From: Valery Kubarovsky <vpk at jlab.org>
> Sent: Thursday, July 10, 2025 9:25 AM
> To: Brian Eng <beng at jlab.org>; Christopher Dilks <dilks at jlab.org>;
> Tyler Lemon <tlemon at jlab.org>
> Cc: Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Subject: Re: RICH S4 CRIO heartbeat stopped
>
>  Hi Brian,
>  Thanks for the update. I will let you know when the access will be
> available.
>  Regards,
>  Valery
>
> -------------------------
>
>  From: Brian Eng <beng at jlab.org>
> Sent: Thursday, July 10, 2025 09:22
> To: Christopher Dilks <dilks at jlab.org>; Tyler Lemon <tlemon at jlab.org>;
> Valery Kubarovsky <vpk at jlab.org>
> Cc: Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Subject: Re: RICH S4 CRIO heartbeat stopped
>
>  It looks like the sbRIO that was replaced 3 weeks ago is offline, or
> at least isn't responding to pings.
>
>  There's nothing that can be done remotely so it would require an
> access to further troubleshoot.
>
> -------------------------
>
>  From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of
> Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Sent: Thursday, July 10, 2025 9:05 AM
> To: Christopher Dilks <dilks at jlab.org>; Tyler Lemon <tlemon at jlab.org>
> Cc: Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Subject: Re: [dsg-hallb_rich] RICH S4 CRIO heartbeat stopped
>
>  Hi Tyler,
>  Please take a look to the RICH cRIO sector 4.
>  Thank you,
>  Valery
>
> -------------------------
>
>  From: Christopher Dilks <dilks at jlab.org>
> Sent: Thursday, July 10, 2025 02:15
> To: Tyler Lemon <tlemon at jlab.org>
> Cc: Valery Kubarovsky <vpk at jlab.org>
> Subject: RICH S4 CRIO heartbeat stopped
>
> Hi Tyler,
>
> The heartbeat stopped around 1am:
> https://logbooks.jlab.org/entry/4415043
>
> All Sector 4 N2 volume PVs are frozen. The frozen values look normal.
>
> Can you take a look when you get this message? Meanwhile the shift
> crew is calling engineering on-call (following the alarm guidance).
>
> Thanks,
> Chris
>
>> Christopher Dilks
>>
>> Jefferson Lab Staff Scientist
>>
>> Office: B122
>>
>> Phone: 757-269-7750
>>
>> Email: dilks at jlab.org
> _______________________________________________
> dsg-hallb_rich mailing list
> dsg-hallb_rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20250714/8cdfae31/attachment-0001.htm>