[dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Tue Jul 15 12:38:13 EDT 2025

The plan is still that on Thursday during the scheduled maintenance I'll try restarting software/hardware to see if that resolves the issue.

If not we have the spare that Tyler already prepared.

This is why we're not going to do anything during the short access today as there still are some signals that are updating and we don't want to lose everything completely.

________________________________
From: Valery Kubarovsky <vpk at jlab.org>
Sent: Tuesday, July 15, 2025 12:35 PM
To: Tyler Lemon <tlemon at jlab.org>; Christopher Dilks <dilks at jlab.org>; Brian Eng <beng at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Ok, then. Let us try to fix it.
________________________________
From: Tyler Lemon <tlemon at jlab.org>
Sent: Tuesday, July 15, 2025 12:33
To: Valery Kubarovsky <vpk at jlab.org>; Christopher Dilks <dilks at jlab.org>; Brian Eng <beng at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Hi Valery,

June 17 was the date the S4 N2 volume cRIO was swapped out, so it may be an EPICS communication problem with that cRIO.

On Thursday, the EPICS client reboot and/or entire system reboot Brian mentioned may fix the problem.

-Tyler

________________________________
From: Valery Kubarovsky <vpk at jlab.org>
Sent: Tuesday, July 15, 2025 12:27:15 PM
To: Tyler Lemon <tlemon at jlab.org>; Christopher Dilks <dilks at jlab.org>; Brian Eng <beng at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Hello Tyler and Brian,
Take a look to the logentry
https://logbooks.jlab.org/book/RICH
It looks like something happened on June 17 at 7:30AM.
After this point the air pressure and air flow1 behave differently.
I am not sure that it is connected with EPICS.
We will have the possibility to take a look at this issue this Thursday.
Thank you,
Valery

________________________________
From: Tyler Lemon <tlemon at jlab.org>
Sent: Monday, July 14, 2025 09:48
To: Christopher Dilks <dilks at jlab.org>; Valery Kubarovsky <vpk at jlab.org>; Tyler Lemon <tlemon at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Hi Chris,

The raw voltage readback for those channels are updating.
Attached are two screenshots of the raw values taken about one minute apart.

AI0 is N2 flow
AI2 is Airflow 1
AI3 is Airflow 2
CH0 is air pressure

You can see that the values for the channels are different in the two screenshots.

So the cRIO is correctly monitoring the airflows, N2 flows, and air pressure. It might be something with an EPICS deadband making it look like those PVs aren't updating.

-Tyler
________________________________
From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of Tyler Lemon via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Sent: Monday, July 14, 2025 9:06 AM
To: Christopher Dilks <dilks at jlab.org>; Valery Kubarovsky <vpk at jlab.org>
Cc: Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: RICH S4 CRIO heartbeat stopped

Hi Chris,

I‘m not sure if I will be able to make it around noon to go down to the hall.
I’m currently at the mechanic for my car and probably will be for most of the morning.

I can take a look remotely, though.

-Tyler

________________________________
From: Christopher Dilks <dilks at jlab.org>
Sent: Monday, July 14, 2025 8:32:29 AM
To: Valery Kubarovsky <vpk at jlab.org>; Tyler Lemon <tlemon at jlab.org>
Cc: Brian Eng <beng at jlab.org>; Marco.Mirazita at lnf.infn.it <Marco.Mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [EXTERNAL] Re: [dsg-hallb_rich] RICH S4 CRIO heartbeat stopped

Dear Tyler,

There will be a 20-30 minute access opportunity around noon today (for ECAL work). I'm not sure if that's enough time to take a look at the cRIO, but it is an access opportunity.

Also, I noticed that some of the S4 PVs, namely airflow, air pressure, and N2 flow, look like they are frozen; the heartbeat is beating though.

Cheers,
Chris

Christopher Dilks

Jefferson Lab Staff Scientist

Office: B122

Phone: 757-269-7750

Email: dilks at jlab.org

________________________________
From: Christopher Dilks <dilks at jlab.org>
Sent: Saturday, July 12, 2025 06:43
To: Valery Kubarovsky <vpk at jlab.org>; Tyler Lemon <tlemon at jlab.org>
Cc: Brian Eng <beng at jlab.org>; Marco.Mirazita at lnf.infn.it <marco.mirazita at lnf.infn.it>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: Re: [EXTERNAL] Re: [dsg-hallb_rich] RICH S4 CRIO heartbeat stopped

Dear all,

It seems like the cRIO spontaneously fixed itself; see https://logbooks.jlab.org/entry/4416371

Thanks,
Chris

Christopher Dilks

Jefferson Lab Staff Scientist

Office: B122

Phone: 757-269-7750

Email: dilks at jlab.org

________________________________
From: Marco Mirazita <Marco.Mirazita at lnf.infn.it>
Sent: Thursday, July 10, 2025 10:06
To: Valery Kubarovsky <vpk at jlab.org>
Cc: Brian Eng <beng at jlab.org>; Christopher Dilks <dilks at jlab.org>; Tyler Lemon <tlemon at jlab.org>; Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
Subject: [EXTERNAL] Re: [dsg-hallb_rich] RICH S4 CRIO heartbeat stopped

Hello,
I agree with Valery, no reason to stop the data taking, but in case
please be ready.
Thank you,
Marco

Il 2025-07-10 16:04 Valery Kubarovsky via dsg-hallb_rich ha scritto:
> I understand this, and I have no intention of interrupting data
> taking. Nevertheless, if there is significant downtime, I would prefer
> to take the opportunity to fix the cRio. Please be prepared.
>
> -------------------------
>
>  From: Brian Eng <beng at jlab.org>
> Sent: Thursday, July 10, 2025 09:37
> To: Valery Kubarovsky <vpk at jlab.org>; Christopher Dilks
> <dilks at jlab.org>; Tyler Lemon <tlemon at jlab.org>
> Cc: Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Subject: Re: RICH S4 CRIO heartbeat stopped
>
>  It can probably wait until the next planned maintenance window on
> 7/17; at the very least I don't think this is something the beam
> should be stopped for as troubleshooting can take longer than expected
> if it isn't something obvious/simple.
>
>  Since we know there's still flow to S1, S4 should also be okay.
>
> -------------------------
>
>  From: Valery Kubarovsky <vpk at jlab.org>
> Sent: Thursday, July 10, 2025 9:25 AM
> To: Brian Eng <beng at jlab.org>; Christopher Dilks <dilks at jlab.org>;
> Tyler Lemon <tlemon at jlab.org>
> Cc: Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Subject: Re: RICH S4 CRIO heartbeat stopped
>
>  Hi Brian,
>  Thanks for the update. I will let you know when the access will be
> available.
>  Regards,
>  Valery
>
> -------------------------
>
>  From: Brian Eng <beng at jlab.org>
> Sent: Thursday, July 10, 2025 09:22
> To: Christopher Dilks <dilks at jlab.org>; Tyler Lemon <tlemon at jlab.org>;
> Valery Kubarovsky <vpk at jlab.org>
> Cc: Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Subject: Re: RICH S4 CRIO heartbeat stopped
>
>  It looks like the sbRIO that was replaced 3 weeks ago is offline, or
> at least isn't responding to pings.
>
>  There's nothing that can be done remotely so it would require an
> access to further troubleshoot.
>
> -------------------------
>
>  From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of
> Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Sent: Thursday, July 10, 2025 9:05 AM
> To: Christopher Dilks <dilks at jlab.org>; Tyler Lemon <tlemon at jlab.org>
> Cc: Valery Kubarovsky via dsg-hallb_rich <dsg-hallb_rich at jlab.org>
> Subject: Re: [dsg-hallb_rich] RICH S4 CRIO heartbeat stopped
>
>  Hi Tyler,
>  Please take a look to the RICH cRIO sector 4.
>  Thank you,
>  Valery
>
> -------------------------
>
>  From: Christopher Dilks <dilks at jlab.org>
> Sent: Thursday, July 10, 2025 02:15
> To: Tyler Lemon <tlemon at jlab.org>
> Cc: Valery Kubarovsky <vpk at jlab.org>
> Subject: RICH S4 CRIO heartbeat stopped
>
> Hi Tyler,
>
> The heartbeat stopped around 1am:
> https://logbooks.jlab.org/entry/4415043
>
> All Sector 4 N2 volume PVs are frozen. The frozen values look normal.
>
> Can you take a look when you get this message? Meanwhile the shift
> crew is calling engineering on-call (following the alarm guidance).
>
> Thanks,
> Chris
>
>> Christopher Dilks
>>
>> Jefferson Lab Staff Scientist
>>
>> Office: B122
>>
>> Phone: 757-269-7750
>>
>> Email: dilks at jlab.org
> _______________________________________________
> dsg-hallb_rich mailing list
> dsg-hallb_rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich

T
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20250715/8207598f/attachment-0001.htm>