[dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to RICH N2 cRIO in hall

Tue Apr 6 03:41:08 EDT 2021

Hello,
thank you everyone for this work.
But I have one question now, how can I check the status of the N2 
system? From the web OPI interface it looks that all the humidity values 
are at 0.
Marco

Il 2021-04-03 18:35 Valery Kubarovsky ha scritto:
> Thank you, folks!
>  I inspected the interlock limits, and they are as expected.
>  Valery
> 
> -------------------------
> 
> From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of
> Tyler Lemon <tlemon at jlab.org>
> Sent: Saturday, April 3, 2021 08:02
> To: Brian Eng <beng at jlab.org>; Marco Mirazita
> <Marco.Mirazita at lnf.infn.it>
> Cc: Morgan Cook IV <mcookiv at jlab.org>; Bob Miller <rmiller at jlab.org>;
> Denny Insley <dinsley at jlab.org>; dsg-hallb_rich at jlab.org
> <dsg-hallb_rich at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
> 
>  Hi Brian,
> 
>  Thanks for swapping out the cRIOs.
> 
>  It looks good to me, too; all the values are as expected.
> 
>  The only thing I think that would need double checking by Valery or
> Marco is the interlock limits since the limits might not have been
> restored properly given that this is a different cRIO and there were
> issues with the previous cRIO.
> 
>  -Tyler
> 
> -------------------------
> 
> From: Brian Eng <beng at jlab.org>
> Sent: Saturday, April 3, 2021 12:51 AM
> To: Tyler Lemon <tlemon at jlab.org>; Marco Mirazita
> <Marco.Mirazita at lnf.infn.it>
> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
> 
>  The cRIO has been swapped out with a spare.
> 
> It was a totally Benny Hill show swapping it out with everything that
> could take longer or go wrong doing so, but at least it's complete.
> 
>  All indicators are green in EPICS so I'm assuming everything is okay,
> but someone else can give it a more thorough check later.
> 
> -------------------------
> 
> From: Tyler Lemon <tlemon at jlab.org>
> Sent: Friday, April 2, 2021 1:22 PM
> To: Brian Eng <beng at jlab.org>; Marco Mirazita
> <Marco.Mirazita at lnf.infn.it>
> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
> 
>  Hi Marco,
> 
>  Like Brian said, we're in the process of configuring a spare cRIO to
> swap it for the the failed one in Hall B.
> 
>  We've asked Mark Taylor to verify the hostname of the cRIO at Peter's
> desk so we can get its network settings configured ahead of time.
> Brian will then go in to the lab to swap the cRIOs.
> 
>  Despite the N2 cRIO being down, we can still see from the EP cRIO
> that the humidity is still low in RICH, low enough to where it is
> below the sensors' capability to measure.
> 
>  In the meantime, the N2 flow can be verified in person by a Hall B
> Tech if they go up to the top level of the Forward Carriage.
> 
>  -Tyler
> 
> -------------------------
> 
> From: Brian Eng <beng at jlab.org>
> Sent: Friday, April 2, 2021 10:07 AM
> To: Marco Mirazita <Marco.Mirazita at lnf.infn.it>; Tyler Lemon
> <tlemon at jlab.org>
> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
> 
>  Hi Marco,
> 
>  I filed a support ticket with NI to see if there was any way to get
> it working without swapping controllers, but they're quite slow with
> replies.
> 
>  I'll work with Sergey on getting IPs swapped in the mean time.
> 
> -------------------------
> 
> From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of
> Marco Mirazita <Marco.Mirazita at lnf.infn.it>
> Sent: Friday, April 2, 2021 7:59 AM
> To: Tyler Lemon <tlemon at jlab.org>
> Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller
> <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV
> <mcookiv at jlab.org>
> Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to
> RICH N2 cRIO in hall
> 
> Hi all,
> any update on this? Do you know how much time will be needed to
> replace
> or fix the broken cRIO? If it takes long time, would it be possible to
> 
> switch the nitrogen system readout on the one that is now connected to
> 
> the EP and is basically not in use?
> Thank you,
> Marco
> 
> Il 2021-03-30 18:12 Marco Mirazita ha scritto:
>> Hi Tyler,
>> isn't the alarm system based on the cRIO readings? If so, it cannot
>> work even if it is still enabled.
>> Thank you,
>> Marco
>> 
>> 
>> 
>> Il 2021-03-30 16:39 Tyler Lemon ha scritto:
>>> Hi Marco,
>>> 
>>>  - do we still have remote access to the information on the
> nitrogen
>>> flow?
>>> 
>>>  No, the flow can only be remotely read out through that cRIO.
>>>  There is a local display on the nitrogen panel on top of the
> forward
>>> carriage someone can read in person.
>>> 
>>>  - do we still have remote access to the information on the
> humidity
>>> level inside the RICH?
>>>  The humidity in the electronic panel can still be read by the EP
>>> cRIO. Those humidity readings are still very low, making me
> confident
>>> that there is still N2 flow to RICH.
>>> 
>>>  We cannot read the humidity in the nitrogen volume because the
> cRIO
>>> appears to have failed.
>>> 
>>>  - do we still have the alarm system working?
>>>  To my knowledge, all the alarms are still enabled.
>>> 
>>>  - do we still have the backup system working?
>>>  Yes, the backup N2 system is still connected and will provide N2
> if
>>> the main supply pressure stops. This system has no electrically
>>> controlled or remote controlled elements so the backup flow
>>> automatically starts if N2 supply pressure is lost through the use
> of
>>> a normally-open pneumatic valve.
>>> 
>>> -------------------------
>>> 
>>> From: Marco Mirazita <Marco.Mirazita at lnf.infn.it>
>>> Sent: Tuesday, March 30, 2021 10:26 AM
>>> To: Tyler Lemon <tlemon at jlab.org>
>>> Cc: Denny Insley <dinsley at jlab.org>; Morgan Cook IV
>>> <mcookiv at jlab.org>; Bob Miller <rmiller at jlab.org>;
>>> dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Valery
> Kubarovsky
>>> <vpk at jlab.org>
>>> Subject: [EXTERNAL] Re: [dsg-hallb_rich] Fw: unable to connect to
> RICH
>>> N2 cRIO in hall
>>> 
>>> Hi Tyler,
>>> could you please clarify the situation? In particular:
>>> - do we still have remote access to the information on the nitrogen
>>> flow?
>>> - do we still have remote access to the information on the humidity
>>> level inside the RICH?
>>> - do we still have the alarm system working?
>>> - do we still have the backup system working?
>>> Thank you,
>>> Marco
>>> 
>>> Il 2021-03-29 14:52 Tyler Lemon ha scritto:
>>>> Hi Denny and Morgan,
>>>> 
>>>>  Please see the email chain below.
>>>> 
>>>>  Brian went in to Hall B last week to check the the RICH N2 volume
>>>> cRIO that failed and it looks like the cRIO will have to be
>>> replaced.
>>>> 
>>>>  Until the cRIO issue is resolved, can Hall B Engineering
>>> periodically
>>>> check the local N2 flow readout for RICH on the top of the forward
>>>> carriage to make sure the detector is still getting appropriate N2
>>>> flow? We're still all working from home and have to make special
>>>> arrangements to go in to the lab.
>>>> 
>>>>  We can remotely see that the humidity in the RICH electronic
> panel
>>> is
>>>> still very low (below the point we're able to read with the
> humidity
>>>> sensors), but it'd still be good to check the flow meters in case
>>>> something happens to that cRIO.
>>>> 
>>>>  Thanks.
>>>> 
>>>>  -Tyler
>>>> 
>>>> -------------------------
>>>> 
>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>> Tyler
>>>> Lemon <tlemon at jlab.org>
>>>> Sent: Monday, March 29, 2021 8:32 AM
>>>> To: Amrit Yegneswaran <yeg at jlab.org>; 'dsg-internal'
>>>> <dsg-internal at jlab.org>
>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>> hall
>>>> 
>>>> Remotely, no there is not a way.
>>>> 
>>>> If you go up to the top of the forward carriage, there are local
>>>> displays on the flow meters.
>>>> 
>>>> -------------------------
>>>> 
>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>> Amrit
>>>> Yegneswaran <yeg at jlab.org>
>>>> Sent: Monday, March 29, 2021 8:29:04 AM
>>>> To: 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>> hall
>>>> 
>>>>  thanks brian
>>>>  we'll talk about this issue in today's meeting
>>>>  is there anyway to see wheteher there is nitrogen flow?
>>>> 
>>>> -------------------------
>>>> 
>>>> From: Brian Eng <beng at jlab.org>
>>>> Sent: Saturday, March 27, 2021 12:15 AM
>>>> To: Amrit Yegneswaran <yeg at jlab.org>; Tyler Lemon
> <tlemon at jlab.org>;
>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>> 
>>>>  It wouldn't force boot into safe mode (still the same blinking
>>> status
>>>> LED) nor be detected by MAX when connected via USB or directly
> with
>>> a
>>>> network cable.
>>>> 
>>>>  Looks like might have to swap the cRIO out, but I'm going to file
> a
>>>> ticket with NI first to make sure there isn't something else I can
>>> do
>>>> locally before doing that since that seems like kind of a pain to
>>> do.
>>>> 
>>>> -------------------------
>>>> 
>>>> From: Brian Eng <beng at jlab.org>
>>>> Sent: Friday, March 26, 2021 12:15 AM
>>>> To: Amrit Yegneswaran <yeg at jlab.org>; Tyler Lemon
> <tlemon at jlab.org>;
>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>> 
>>>>  First look isn't good, the status LED on the cRIO is constantly
>>>> blinking which according to the manual means:
>>>> 
>>>>  The cRIO-9035 has not booted into NI Linux Real-Time. The
> cRIO-9035
>>>> either booted into an unsupported operating system, was
> interrupted
>>>> during the boot process, or detected an unrecoverable software
>>> error.
>>>> 
>>>>  I did notice that the network went down/up several times along
> with
>>>> it trying to access the SD card, so it seems like it's in some
> sort
>>> of
>>>> boot loop.
>>>> 
>>>>  I didn't bring the Windows laptop down with me today and it's
> later
>>>> than I planned on being here so I'll try formatting it locally
>>>> tomorrow to see if I can't just get it on the network so that
>>>> Tyler/Pete can reinstall the needed software.
>>>> 
>>>> -------------------------
>>>> 
>>>> From: Amrit Yegneswaran <yeg at jlab.org>
>>>> Sent: Friday, March 19, 2021 4:45 PM
>>>> To: Brian Eng <beng at jlab.org>; Tyler Lemon <tlemon at jlab.org>;
>>>> 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>> 
>>>>  brian,
>>>>  from tyler's e-mail it appears that since there is still N2 so
>>> there
>>>> should be no urgent problem.
>>>>  which means you can decide when you wish to go.
>>>>  like always late in the evening.
>>>> 
>>>>  pity the hallb techs aren't able to help.
>>>> 
>>>>  on the good news aide patrizia, is willing to consider the
> robotic
>>>> version, but we need to come up with a plan on what other project
>>>> (which are on the books we can use the system)
>>>>  so all, put on your thinking caps
>>>> 
>>>> -------------------------
>>>> 
>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>> Brian
>>>> Eng <beng at jlab.org>
>>>> Sent: Friday, March 19, 2021 2:48 PM
>>>> To: Tyler Lemon <tlemon at jlab.org>; 'dsg-internal'
>>>> <dsg-internal at jlab.org>
>>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>>> hall
>>>> 
>>>>  I can go in and check it tonight if need be.
>>>> 
>>>> -------------------------
>>>> 
>>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>>> Tyler
>>>> Lemon <tlemon at jlab.org>
>>>> Sent: Friday, March 19, 2021 1:45 PM
>>>> To: 'dsg-internal' <dsg-internal at jlab.org>
>>>> Subject: [Dsg-internal] unable to connect to RICH N2 cRIO in hall
>>>> 
>>>>  Hello,
>>>> 
>>>>  I'm going to preface this by saying that from the EP cRIO, we can
>>> see
>>>> that the humidity in RICH is still low and at a good level so RICH
>>> is
>>>> still getting N2 without issues.
>>>> 
>>>>  That being said, I cannot connect to the N2 volume cRIO (hostname
>>>> richcrio) over the network.
>>>> 
>>>>  Peter noticed this a few days ago and I asked Hall B engineering
> to
>>>> check on the cRIO. Their response was not helpful since they just
>>>> looked at the cRIO chassis from the outside. I have since asked
> them
>>>> to recheck the chassis, looking inside to see if there are any
>>> lights
>>>> flashing inside, but this probably won't be done today.
>>>> 
>>>>  It seems more likely than not that someone will have to go in to
>>> Hall
>>>> B to check the cRIO.
>>>> 
>>>>  Since we can still monitor the internal humidity from the EP
> cRIO,
>>> I
>>>> don't think this is something that has to be absolutely taken care
>>> of
>>>> right this movement.
>>>> 
>>>>  Should I start the process of getting approval to go in myself?
> Or
>>>> would Brian be able to check the cRIO?
>>>> 
>>>>  -Tyler
>>>> _______________________________________________
>>>> dsg-hallb_rich mailing list
>>>> dsg-hallb_rich at jlab.org
>>>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>>> _______________________________________________
>>> dsg-hallb_rich mailing list
>>> dsg-hallb_rich at jlab.org
>>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>> _______________________________________________
>> dsg-hallb_rich mailing list
>> dsg-hallb_rich at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
> _______________________________________________
> dsg-hallb_rich mailing list
> dsg-hallb_rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
> _______________________________________________
> dsg-hallb_rich mailing list
> dsg-hallb_rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich