[dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to RICH N2 cRIO in hall

Tue Apr 6 04:09:44 EDT 2021

For the webopi version

  1.  go to RICH Overview from the main menu
  2.  then click on the Hard Interlocks button in the upper right and select Interlock Controls
  3.  click on the Details button near the middle of the screen in the N2 Volume and Gas section
  4.  finally click on the N2 Volume Sensor Locations

Here's a direct link to that screen once you've already logged in to the webopi page if you don't want to go through those steps.

https://hallbopi.jlab.org/webopi3.3/w?opi=https://hallbopi.jlab.org/opis/apps/cRioIntlkApp/RICH_S1_N2.opi

When I go to that page I get non-zero values for the humidity sensors, like in the attached image.

________________________________
From: Marco Mirazita <Marco.Mirazita at lnf.infn.it>
Sent: Tuesday, April 6, 2021 3:41 AM
To: Valery Kubarovsky <vpk at jlab.org>
Cc: Tyler Lemon <tlemon at jlab.org>; Brian Eng <beng at jlab.org>; dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV <mcookiv at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to RICH N2 cRIO in hall

Hello,
thank you everyone for this work.
But I have one question now, how can I check the status of the N2
system? From the web OPI interface it looks that all the humidity values
are at 0.
Marco

Il 2021-04-03 18:35 Valery Kubarovsky ha scritto:
> Thank you, folks!
>  I inspected the interlock limits, and they are as expected.
>  Valery
>
> -------------------------
>
> From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of
> Tyler Lemon <tlemon at jlab.org>
> Sent: Saturday, April 3, 2021 08:02
> To: Brian Eng <beng at jlab.org>; Marco Mirazita
> <Marco.Mirazita at lnf.infn.it>
> Cc: Morgan Cook IV <mcookiv at jlab.org>; Bob Miller <rmiller at jlab.org>;
> Denny Insley <dinsley at jlab.org>; dsg-hallb_rich at jlab.org
> <dsg-hallb_rich at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
>
>  Hi Brian,
>
>  Thanks for swapping out the cRIOs.
>
>  It looks good to me, too; all the values are as expected.
>
>  The only thing I think that would need double checking by Valery or
> Marco is the interlock limits since the limits might not have been
> restored properly given that this is a different cRIO and there were
> issues with the previous cRIO.
>
>  -Tyler
>
> -------------------------
>
> From: Brian Eng <beng at jlab.org>
> Sent: Saturday, April 3, 2021 12:51 AM
> To: Tyler Lemon <tlemon at jlab.org>; Marco Mirazita
> <Marco.Mirazita at lnf.infn.it>
> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
>
>  The cRIO has been swapped out with a spare.
>
> It was a totally Benny Hill show swapping it out with everything that
> could take longer or go wrong doing so, but at least it's complete.
>
>  All indicators are green in EPICS so I'm assuming everything is okay,
> but someone else can give it a more thorough check later.
>
> -------------------------
>
> From: Tyler Lemon <tlemon at jlab.org>
> Sent: Friday, April 2, 2021 1:22 PM
> To: Brian Eng <beng at jlab.org>; Marco Mirazita
> <Marco.Mirazita at lnf.infn.it>
> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
>
>  Hi Marco,
>
>  Like Brian said, we're in the process of configuring a spare cRIO to
> swap it for the the failed one in Hall B.
>
>  We've asked Mark Taylor to verify the hostname of the cRIO at Peter's
> desk so we can get its network settings configured ahead of time.
> Brian will then go in to the lab to swap the cRIOs.
>
>  Despite the N2 cRIO being down, we can still see from the EP cRIO
> that the humidity is still low in RICH, low enough to where it is
> below the sensors' capability to measure.
>
>  In the meantime, the N2 flow can be verified in person by a Hall B
> Tech if they go up to the top level of the Forward Carriage.
>
>  -Tyler
>
> -------------------------
>
> From: Brian Eng <beng at jlab.org>
> Sent: Friday, April 2, 2021 10:07 AM
> To: Marco Mirazita <Marco.Mirazita at lnf.infn.it>; Tyler Lemon
> <tlemon at jlab.org>
> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
>
>  Hi Marco,
>
>  I filed a support ticket with NI to see if there was any way to get
> it working without swapping controllers, but they're quite slow with
> replies.
>
>  I'll work with Sergey on getting IPs swapped in the mean time.
>
> -------------------------
>
> From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of
> Marco Mirazita <Marco.Mirazita at lnf.infn.it>
> Sent: Friday, April 2, 2021 7:59 AM
> To: Tyler Lemon <tlemon at jlab.org>
> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
>
> Hi all,
> any update on this? Do you know how much time will be needed to
> replace
> or fix the broken cRIO? If it takes long time, would it be possible to
>
> switch the nitrogen system readout on the one that is now connected to
>
> the EP and is basically not in use?
> Thank you,
> Marco
>
> Il 2021-03-30 18:12 Marco Mirazita ha scritto:
>> Hi Tyler,
>> isn't the alarm system based on the cRIO readings? If so, it cannot
>> work even if it is still enabled.
>> Thank you,
>> Marco
>>
>>
>>
>> Il 2021-03-30 16:39 Tyler Lemon ha scritto:
>>> Hi Marco,
>>>
>>>  - do we still have remote access to the information on the
> nitrogen
>>> flow?
>>>
>>>  No, the flow can only be remotely read out through that cRIO.
>>>  There is a local display on the nitrogen panel on top of the
> forward
>>> carriage someone can read in person.
>>>
>>>  - do we still have remote access to the information on the
> humidity
>>> level inside the RICH?
>>>  The humidity in the electronic panel can still be read by the EP
>>> cRIO. Those humidity readings are still very low, making me
> confident
>>> that there is still N2 flow to RICH.
>>>
>>>  We cannot read the humidity in the nitrogen volume because the
> cRIO
>>> appears to have failed.
>>>
>>>  - do we still have the alarm system working?
>>>  To my knowledge, all the alarms are still enabled.
>>>
>>>  - do we still have the backup system working?
>>>  Yes, the backup N2 system is still connected and will provide N2
> if
>>> the main supply pressure stops. This system has no electrically
>>> controlled or remote controlled elements so the backup flow
>>> automatically starts if N2 supply pressure is lost through the use
> of
>>> a normally-open pneumatic valve.
>>>
>>> -------------------------
>>>
>>> From: Marco Mirazita <Marco.Mirazita at lnf.infn.it>
>>> Sent: Tuesday, March 30, 2021 10:26 AM
>>> To: Tyler Lemon <tlemon at jlab.org>
>>> Cc: Denny Insley <dinsley at jlab.org>; Morgan Cook IV
>>> <mcookiv at jlab.org>; Bob Miller <rmiller at jlab.org>;
>>> dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Valery
> Kubarovsky
>>> <vpk at jlab.org>
>>> Subject: [EXTERNAL] Re: [dsg-hallb_rich] Fw: unable to connect to
> RICH
>>> N2 cRIO in hall
>>>
>>> Hi Tyler,
>>> could you please clarify the situation? In particular:
>>> - do we still have remote access to the information on the nitrogen
>>> flow?
>>> - do we still have remote access to the information on the humidity
>>> level inside the RICH?
>>> - do we still have the alarm system working?
>>> - do we still have the backup system working?
>>> Thank you,
>>> Marco
>>>
>>> Il 2021-03-29 14:52 Tyler Lemon ha scritto:
>>>> Hi Denny and Morgan,
>>>>
>>>>  Please see the email chain below.
>>>>
>>>>  Brian went in to Hall B last week to check the the RICH N2 volume
>>>> cRIO that failed and it looks like the cRIO will have to be
>>> replaced.
>>>>
>>>>  Until the cRIO issue is resolved, can Hall B Engineering
>>> periodically
>>>> check the local N2 flow readout for RICH on the top of the forward
>>>> carriage to make sure the detector is still getting appropriate N2
>>>> flow? We're still all working from home and have to make special
>>>> arrangements to go in to the lab.
>>>>
>>>>  We can remotely see that the humidity in the RICH electronic
> panel
>>> is
>>>> still very low (below the point we're able to read with the
> humidity
>>>> sensors), but it'd still be good to check the flow meters in case
>>>> something happens to that cRIO.
>>>>
>>>>  Thanks.
>>>>
>>>>  -Tyler
>>>>
>>>> -------------------------
>>>>
>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>> Tyler
>>>> Lemon <tlemon at jlab.org>
>>>> Sent: Monday, March 29, 2021 8:32 AM
>>>> To: Amrit Yegneswaran <yeg at jlab.org>; 'dsg-internal'
>>>> <dsg-internal at jlab.org>
>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>> hall
>>>>
>>>> Remotely, no there is not a way.
>>>>
>>>> If you go up to the top of the forward carriage, there are local
>>>> displays on the flow meters.
>>>>
>>>> -------------------------
>>>>
>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>> Amrit
>>>> Yegneswaran <yeg at jlab.org>
>>>> Sent: Monday, March 29, 2021 8:29:04 AM
>>>> To: 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>> hall
>>>>
>>>>  thanks brian
>>>>  we'll talk about this issue in today's meeting
>>>>  is there anyway to see wheteher there is nitrogen flow?
>>>>
>>>> -------------------------
>>>>
>>>> From: Brian Eng <beng at jlab.org>
>>>> Sent: Saturday, March 27, 2021 12:15 AM
>>>> To: Amrit Yegneswaran <yeg at jlab.org>; Tyler Lemon
> <tlemon at jlab.org>;
>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>>
>>>>  It wouldn't force boot into safe mode (still the same blinking
>>> status
>>>> LED) nor be detected by MAX when connected via USB or directly
> with
>>> a
>>>> network cable.
>>>>
>>>>  Looks like might have to swap the cRIO out, but I'm going to file
> a
>>>> ticket with NI first to make sure there isn't something else I can
>>> do
>>>> locally before doing that since that seems like kind of a pain to
>>> do.
>>>>
>>>> -------------------------
>>>>
>>>> From: Brian Eng <beng at jlab.org>
>>>> Sent: Friday, March 26, 2021 12:15 AM
>>>> To: Amrit Yegneswaran <yeg at jlab.org>; Tyler Lemon
> <tlemon at jlab.org>;
>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>>
>>>>  First look isn't good, the status LED on the cRIO is constantly
>>>> blinking which according to the manual means:
>>>>
>>>>  The cRIO-9035 has not booted into NI Linux Real-Time. The
> cRIO-9035
>>>> either booted into an unsupported operating system, was
> interrupted
>>>> during the boot process, or detected an unrecoverable software
>>> error.
>>>>
>>>>  I did notice that the network went down/up several times along
> with
>>>> it trying to access the SD card, so it seems like it's in some
> sort
>>> of
>>>> boot loop.
>>>>
>>>>  I didn't bring the Windows laptop down with me today and it's
> later
>>>> than I planned on being here so I'll try formatting it locally
>>>> tomorrow to see if I can't just get it on the network so that
>>>> Tyler/Pete can reinstall the needed software.
>>>>
>>>> -------------------------
>>>>
>>>> From: Amrit Yegneswaran <yeg at jlab.org>
>>>> Sent: Friday, March 19, 2021 4:45 PM
>>>> To: Brian Eng <beng at jlab.org>; Tyler Lemon <tlemon at jlab.org>;
>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>>
>>>>  brian,
>>>>  from tyler's e-mail it appears that since there is still N2 so
>>> there
>>>> should be no urgent problem.
>>>>  which means you can decide when you wish to go.
>>>>  like always late in the evening.
>>>>
>>>>  pity the hallb techs aren't able to help.
>>>>
>>>>  on the good news aide patrizia, is willing to consider the
> robotic
>>>> version, but we need to come up with a plan on what other project
>>>> (which are on the books we can use the system)
>>>>  so all, put on your thinking caps
>>>>
>>>> -------------------------
>>>>
>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>> Brian
>>>> Eng <beng at jlab.org>
>>>> Sent: Friday, March 19, 2021 2:48 PM
>>>> To: Tyler Lemon <tlemon at jlab.org>; 'dsg-internal'
>>>> <dsg-internal at jlab.org>
>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>> hall
>>>>
>>>>  I can go in and check it tonight if need be.
>>>>
>>>> -------------------------
>>>>
>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>> Tyler
>>>> Lemon <tlemon at jlab.org>
>>>> Sent: Friday, March 19, 2021 1:45 PM
>>>> To: 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: [Dsg-internal] unable to connect to RICH N2 cRIO in hall
>>>>
>>>>  Hello,
>>>>
>>>>  I'm going to preface this by saying that from the EP cRIO, we can
>>> see
>>>> that the humidity in RICH is still low and at a good level so RICH
>>> is
>>>> still getting N2 without issues.
>>>>
>>>>  That being said, I cannot connect to the N2 volume cRIO (hostname
>>>> richcrio) over the network.
>>>>
>>>>  Peter noticed this a few days ago and I asked Hall B engineering
> to
>>>> check on the cRIO. Their response was not helpful since they just
>>>> looked at the cRIO chassis from the outside. I have since asked
> them
>>>> to recheck the chassis, looking inside to see if there are any
>>> lights
>>>> flashing inside, but this probably won't be done today.
>>>>
>>>>  It seems more likely than not that someone will have to go in to
>>> Hall
>>>> B to check the cRIO.
>>>>
>>>>  Since we can still monitor the internal humidity from the EP
> cRIO,
>>> I
>>>> don't think this is something that has to be absolutely taken care
>>> of
>>>> right this movement.
>>>>
>>>>  Should I start the process of getting approval to go in myself?
> Or
>>>> would Brian be able to check the cRIO?
>>>>
>>>>  -Tyler
>>>> _______________________________________________
>>>> dsg-hallb_rich mailing list
>>>> dsg-hallb_rich at jlab.org
>>>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>>> _______________________________________________
>>> dsg-hallb_rich mailing list
>>> dsg-hallb_rich at jlab.org
>>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>> _______________________________________________
>> dsg-hallb_rich mailing list
>> dsg-hallb_rich at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
> _______________________________________________
> dsg-hallb_rich mailing list
> dsg-hallb_rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
> _______________________________________________
> dsg-hallb_rich mailing list
> dsg-hallb_rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20210406/b2a89851/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RICH N2 webopi.png
Type: image/png
Size: 312510 bytes
Desc: RICH N2 webopi.png
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20210406/b2a89851/attachment-0001.png>