[dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to RICH N2 cRIO in hall

Tue Apr 6 04:13:52 EDT 2021

Thank you!
Marco

Il 2021-04-06 10:09 Brian Eng ha scritto:
> For the webopi version
> 
>  	* go to RICH Overview from the main menu
> 	* then click on the Hard Interlocks button in the upper right and
> select Interlock Controls
> 	* click on the Details button near the middle of the screen in the N2
> Volume and Gas section
> 	* finally click on the N2 Volume Sensor Locations
> 
> Here's a direct link to that screen once you've already logged in to
> the webopi page if you don't want to go through those steps.
> 
> https://hallbopi.jlab.org/webopi3.3/w?opi=https://hallbopi.jlab.org/opis/apps/cRioIntlkApp/RICH_S1_N2.opi
> 
> When I go to that page I get non-zero values for the humidity sensors,
> like in the attached image.
> 
> -------------------------
> 
> From: Marco Mirazita <Marco.Mirazita at lnf.infn.it>
> Sent: Tuesday, April 6, 2021 3:41 AM
> To: Valery Kubarovsky <vpk at jlab.org>
> Cc: Tyler Lemon <tlemon at jlab.org>; Brian Eng <beng at jlab.org>;
> dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
> 
> Hello,
> thank you everyone for this work.
> But I have one question now, how can I check the status of the N2
> system? From the web OPI interface it looks that all the humidity
> values
> are at 0.
> Marco
> 
> Il 2021-04-03 18:35 Valery Kubarovsky ha scritto:
>> Thank you, folks!
>>  I inspected the interlock limits, and they are as expected.
>>  Valery
>> 
>> -------------------------
>> 
>> From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of
>> Tyler Lemon <tlemon at jlab.org>
>> Sent: Saturday, April 3, 2021 08:02
>> To: Brian Eng <beng at jlab.org>; Marco Mirazita
>> <Marco.Mirazita at lnf.infn.it>
>> Cc: Morgan Cook IV <mcookiv at jlab.org>; Bob Miller
> <rmiller at jlab.org>;
>> Denny Insley <dinsley at jlab.org>; dsg-hallb_rich at jlab.org
>> <dsg-hallb_rich at jlab.org>
>> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect
> to
>> RICH N2 cRIO in hall
>> 
>>  Hi Brian,
>> 
>>  Thanks for swapping out the cRIOs.
>> 
>>  It looks good to me, too; all the values are as expected.
>> 
>>  The only thing I think that would need double checking by Valery or
>> Marco is the interlock limits since the limits might not have been
>> restored properly given that this is a different cRIO and there were
>> issues with the previous cRIO.
>> 
>>  -Tyler
>> 
>> -------------------------
>> 
>> From: Brian Eng <beng at jlab.org>
>> Sent: Saturday, April 3, 2021 12:51 AM
>> To: Tyler Lemon <tlemon at jlab.org>; Marco Mirazita
>> <Marco.Mirazita at lnf.infn.it>
>> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
>> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
>> <mcookiv at jlab.org>
>> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect
> to
>> RICH N2 cRIO in hall
>> 
>>  The cRIO has been swapped out with a spare.
>> 
>> It was a totally Benny Hill show swapping it out with everything
> that
>> could take longer or go wrong doing so, but at least it's complete.
>> 
>>  All indicators are green in EPICS so I'm assuming everything is
> okay,
>> but someone else can give it a more thorough check later.
>> 
>> -------------------------
>> 
>> From: Tyler Lemon <tlemon at jlab.org>
>> Sent: Friday, April 2, 2021 1:22 PM
>> To: Brian Eng <beng at jlab.org>; Marco Mirazita
>> <Marco.Mirazita at lnf.infn.it>
>> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
>> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
>> <mcookiv at jlab.org>
>> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect
> to
>> RICH N2 cRIO in hall
>> 
>>  Hi Marco,
>> 
>>  Like Brian said, we're in the process of configuring a spare cRIO
> to
>> swap it for the the failed one in Hall B.
>> 
>>  We've asked Mark Taylor to verify the hostname of the cRIO at
> Peter's
>> desk so we can get its network settings configured ahead of time.
>> Brian will then go in to the lab to swap the cRIOs.
>> 
>>  Despite the N2 cRIO being down, we can still see from the EP cRIO
>> that the humidity is still low in RICH, low enough to where it is
>> below the sensors' capability to measure.
>> 
>>  In the meantime, the N2 flow can be verified in person by a Hall B
>> Tech if they go up to the top level of the Forward Carriage.
>> 
>>  -Tyler
>> 
>> -------------------------
>> 
>> From: Brian Eng <beng at jlab.org>
>> Sent: Friday, April 2, 2021 10:07 AM
>> To: Marco Mirazita <Marco.Mirazita at lnf.infn.it>; Tyler Lemon
>> <tlemon at jlab.org>
>> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
>> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
>> <mcookiv at jlab.org>
>> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect
> to
>> RICH N2 cRIO in hall
>> 
>>  Hi Marco,
>> 
>>  I filed a support ticket with NI to see if there was any way to get
>> it working without swapping controllers, but they're quite slow with
>> replies.
>> 
>>  I'll work with Sergey on getting IPs swapped in the mean time.
>> 
>> -------------------------
>> 
>> From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of
>> Marco Mirazita <Marco.Mirazita at lnf.infn.it>
>> Sent: Friday, April 2, 2021 7:59 AM
>> To: Tyler Lemon <tlemon at jlab.org>
>> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
>> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
>> <mcookiv at jlab.org>
>> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect
> to
>> RICH N2 cRIO in hall
>> 
>> Hi all,
>> any update on this? Do you know how much time will be needed to
>> replace
>> or fix the broken cRIO? If it takes long time, would it be possible
> to
>> 
>> switch the nitrogen system readout on the one that is now connected
> to
>> 
>> the EP and is basically not in use?
>> Thank you,
>> Marco
>> 
>> Il 2021-03-30 18:12 Marco Mirazita ha scritto:
>>> Hi Tyler,
>>> isn't the alarm system based on the cRIO readings? If so, it cannot
>>> work even if it is still enabled.
>>> Thank you,
>>> Marco
>>> 
>>> 
>>> 
>>> Il 2021-03-30 16:39 Tyler Lemon ha scritto:
>>>> Hi Marco,
>>>> 
>>>>  - do we still have remote access to the information on the
>> nitrogen
>>>> flow?
>>>> 
>>>>  No, the flow can only be remotely read out through that cRIO.
>>>>  There is a local display on the nitrogen panel on top of the
>> forward
>>>> carriage someone can read in person.
>>>> 
>>>>  - do we still have remote access to the information on the
>> humidity
>>>> level inside the RICH?
>>>>  The humidity in the electronic panel can still be read by the EP
>>>> cRIO. Those humidity readings are still very low, making me
>> confident
>>>> that there is still N2 flow to RICH.
>>>> 
>>>>  We cannot read the humidity in the nitrogen volume because the
>> cRIO
>>>> appears to have failed.
>>>> 
>>>>  - do we still have the alarm system working?
>>>>  To my knowledge, all the alarms are still enabled.
>>>> 
>>>>  - do we still have the backup system working?
>>>>  Yes, the backup N2 system is still connected and will provide N2
>> if
>>>> the main supply pressure stops. This system has no electrically
>>>> controlled or remote controlled elements so the backup flow
>>>> automatically starts if N2 supply pressure is lost through the use
>> of
>>>> a normally-open pneumatic valve.
>>>> 
>>>> -------------------------
>>>> 
>>>> From: Marco Mirazita <Marco.Mirazita at lnf.infn.it>
>>>> Sent: Tuesday, March 30, 2021 10:26 AM
>>>> To: Tyler Lemon <tlemon at jlab.org>
>>>> Cc: Denny Insley <dinsley at jlab.org>; Morgan Cook IV
>>>> <mcookiv at jlab.org>; Bob Miller <rmiller at jlab.org>;
>>>> dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Valery
>> Kubarovsky
>>>> <vpk at jlab.org>
>>>> Subject: [EXTERNAL] Re: [dsg-hallb_rich] Fw: unable to connect to
>> RICH
>>>> N2 cRIO in hall
>>>> 
>>>> Hi Tyler,
>>>> could you please clarify the situation? In particular:
>>>> - do we still have remote access to the information on the
> nitrogen
>>>> flow?
>>>> - do we still have remote access to the information on the
> humidity
>>>> level inside the RICH?
>>>> - do we still have the alarm system working?
>>>> - do we still have the backup system working?
>>>> Thank you,
>>>> Marco
>>>> 
>>>> Il 2021-03-29 14:52 Tyler Lemon ha scritto:
>>>>> Hi Denny and Morgan,
>>>>> 
>>>>>  Please see the email chain below.
>>>>> 
>>>>>  Brian went in to Hall B last week to check the the RICH N2
> volume
>>>>> cRIO that failed and it looks like the cRIO will have to be
>>>> replaced.
>>>>> 
>>>>>  Until the cRIO issue is resolved, can Hall B Engineering
>>>> periodically
>>>>> check the local N2 flow readout for RICH on the top of the
> forward
>>>>> carriage to make sure the detector is still getting appropriate
> N2
>>>>> flow? We're still all working from home and have to make special
>>>>> arrangements to go in to the lab.
>>>>> 
>>>>>  We can remotely see that the humidity in the RICH electronic
>> panel
>>>> is
>>>>> still very low (below the point we're able to read with the
>> humidity
>>>>> sensors), but it'd still be good to check the flow meters in case
>>>>> something happens to that cRIO.
>>>>> 
>>>>>  Thanks.
>>>>> 
>>>>>  -Tyler
>>>>> 
>>>>> -------------------------
>>>>> 
>>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>>> Tyler
>>>>> Lemon <tlemon at jlab.org>
>>>>> Sent: Monday, March 29, 2021 8:32 AM
>>>>> To: Amrit Yegneswaran <yeg at jlab.org>; 'dsg-internal'
>>>>> <dsg-internal at jlab.org>
>>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>>> hall
>>>>> 
>>>>> Remotely, no there is not a way.
>>>>> 
>>>>> If you go up to the top of the forward carriage, there are local
>>>>> displays on the flow meters.
>>>>> 
>>>>> -------------------------
>>>>> 
>>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>>> Amrit
>>>>> Yegneswaran <yeg at jlab.org>
>>>>> Sent: Monday, March 29, 2021 8:29:04 AM
>>>>> To: 'dsg-internal' <dsg-internal at jlab.org>
>>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>>> hall
>>>>> 
>>>>>  thanks brian
>>>>>  we'll talk about this issue in today's meeting
>>>>>  is there anyway to see wheteher there is nitrogen flow?
>>>>> 
>>>>> -------------------------
>>>>> 
>>>>> From: Brian Eng <beng at jlab.org>
>>>>> Sent: Saturday, March 27, 2021 12:15 AM
>>>>> To: Amrit Yegneswaran <yeg at jlab.org>; Tyler Lemon
>> <tlemon at jlab.org>;
>>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>>> 
>>>>>  It wouldn't force boot into safe mode (still the same blinking
>>>> status
>>>>> LED) nor be detected by MAX when connected via USB or directly
>> with
>>>> a
>>>>> network cable.
>>>>> 
>>>>>  Looks like might have to swap the cRIO out, but I'm going to
> file
>> a
>>>>> ticket with NI first to make sure there isn't something else I
> can
>>>> do
>>>>> locally before doing that since that seems like kind of a pain to
>>>> do.
>>>>> 
>>>>> -------------------------
>>>>> 
>>>>> From: Brian Eng <beng at jlab.org>
>>>>> Sent: Friday, March 26, 2021 12:15 AM
>>>>> To: Amrit Yegneswaran <yeg at jlab.org>; Tyler Lemon
>> <tlemon at jlab.org>;
>>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>>> 
>>>>>  First look isn't good, the status LED on the cRIO is constantly
>>>>> blinking which according to the manual means:
>>>>> 
>>>>>  The cRIO-9035 has not booted into NI Linux Real-Time. The
>> cRIO-9035
>>>>> either booted into an unsupported operating system, was
>> interrupted
>>>>> during the boot process, or detected an unrecoverable software
>>>> error.
>>>>> 
>>>>>  I did notice that the network went down/up several times along
>> with
>>>>> it trying to access the SD card, so it seems like it's in some
>> sort
>>>> of
>>>>> boot loop.
>>>>> 
>>>>>  I didn't bring the Windows laptop down with me today and it's
>> later
>>>>> than I planned on being here so I'll try formatting it locally
>>>>> tomorrow to see if I can't just get it on the network so that
>>>>> Tyler/Pete can reinstall the needed software.
>>>>> 
>>>>> -------------------------
>>>>> 
>>>>> From: Amrit Yegneswaran <yeg at jlab.org>
>>>>> Sent: Friday, March 19, 2021 4:45 PM
>>>>> To: Brian Eng <beng at jlab.org>; Tyler Lemon <tlemon at jlab.org>;
>>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>>> 
>>>>>  brian,
>>>>>  from tyler's e-mail it appears that since there is still N2 so
>>>> there
>>>>> should be no urgent problem.
>>>>>  which means you can decide when you wish to go.
>>>>>  like always late in the evening.
>>>>> 
>>>>>  pity the hallb techs aren't able to help.
>>>>> 
>>>>>  on the good news aide patrizia, is willing to consider the
>> robotic
>>>>> version, but we need to come up with a plan on what other project
>>>>> (which are on the books we can use the system)
>>>>>  so all, put on your thinking caps
>>>>> 
>>>>> -------------------------
>>>>> 
>>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>>> Brian
>>>>> Eng <beng at jlab.org>
>>>>> Sent: Friday, March 19, 2021 2:48 PM
>>>>> To: Tyler Lemon <tlemon at jlab.org>; 'dsg-internal'
>>>>> <dsg-internal at jlab.org>
>>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>>> hall
>>>>> 
>>>>>  I can go in and check it tonight if need be.
>>>>> 
>>>>> -------------------------
>>>>> 
>>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>>> Tyler
>>>>> Lemon <tlemon at jlab.org>
>>>>> Sent: Friday, March 19, 2021 1:45 PM
>>>>> To: 'dsg-internal' <dsg-internal at jlab.org>
>>>>> Subject: [Dsg-internal] unable to connect to RICH N2 cRIO in hall
>>>>> 
>>>>>  Hello,
>>>>> 
>>>>>  I'm going to preface this by saying that from the EP cRIO, we
> can
>>>> see
>>>>> that the humidity in RICH is still low and at a good level so
> RICH
>>>> is
>>>>> still getting N2 without issues.
>>>>> 
>>>>>  That being said, I cannot connect to the N2 volume cRIO
> (hostname
>>>>> richcrio) over the network.
>>>>> 
>>>>>  Peter noticed this a few days ago and I asked Hall B engineering
>> to
>>>>> check on the cRIO. Their response was not helpful since they just
>>>>> looked at the cRIO chassis from the outside. I have since asked
>> them
>>>>> to recheck the chassis, looking inside to see if there are any
>>>> lights
>>>>> flashing inside, but this probably won't be done today.
>>>>> 
>>>>>  It seems more likely than not that someone will have to go in to
>>>> Hall
>>>>> B to check the cRIO.
>>>>> 
>>>>>  Since we can still monitor the internal humidity from the EP
>> cRIO,
>>>> I
>>>>> don't think this is something that has to be absolutely taken
> care
>>>> of
>>>>> right this movement.
>>>>> 
>>>>>  Should I start the process of getting approval to go in myself?
>> Or
>>>>> would Brian be able to check the cRIO?
>>>>> 
>>>>>  -Tyler
>>>>> _______________________________________________
>>>>> dsg-hallb_rich mailing list
>>>>> dsg-hallb_rich at jlab.org
>>>>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>>>> _______________________________________________
>>>> dsg-hallb_rich mailing list
>>>> dsg-hallb_rich at jlab.org
>>>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>>> _______________________________________________
>>> dsg-hallb_rich mailing list
>>> dsg-hallb_rich at jlab.org
>>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>> _______________________________________________
>> dsg-hallb_rich mailing list
>> dsg-hallb_rich at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>> _______________________________________________
>> dsg-hallb_rich mailing list
>> dsg-hallb_rich at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
> _______________________________________________
> dsg-hallb_rich mailing list
> dsg-hallb_rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich