[dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to RICH N2 cRIO in hall

Fri Apr 2 13:22:02 EDT 2021

Hi Marco,

Like Brian said, we're in the process of configuring a spare cRIO to swap it for the the failed one in Hall B.

We've asked Mark Taylor to verify the hostname of the cRIO at Peter's desk so we can get its network settings configured ahead of time. Brian will then go in to the lab to swap the cRIOs.

Despite the N2 cRIO being down, we can still see from the EP cRIO that the humidity is still low in RICH, low enough to where it is below the sensors' capability to measure.

In the meantime, the N2 flow can be verified in person by a Hall B Tech if they go up to the top level of the Forward Carriage.

-Tyler

________________________________
From: Brian Eng <beng at jlab.org>
Sent: Friday, April 2, 2021 10:07 AM
To: Marco Mirazita <Marco.Mirazita at lnf.infn.it>; Tyler Lemon <tlemon at jlab.org>
Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV <mcookiv at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to RICH N2 cRIO in hall

Hi Marco,

I filed a support ticket with NI to see if there was any way to get it working without swapping controllers, but they're quite slow with replies.

I'll work with Sergey on getting IPs swapped in the mean time.

________________________________
From: dsg-hallb_rich <dsg-hallb_rich-bounces at jlab.org> on behalf of Marco Mirazita <Marco.Mirazita at lnf.infn.it>
Sent: Friday, April 2, 2021 7:59 AM
To: Tyler Lemon <tlemon at jlab.org>
Cc: dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Bob Miller <rmiller at jlab.org>; Denny Insley <dinsley at jlab.org>; Morgan Cook IV <mcookiv at jlab.org>
Subject: Re: [dsg-hallb_rich] [EXTERNAL] Re: Fw: unable to connect to RICH N2 cRIO in hall

Hi all,
any update on this? Do you know how much time will be needed to replace
or fix the broken cRIO? If it takes long time, would it be possible to
switch the nitrogen system readout on the one that is now connected to
the EP and is basically not in use?
Thank you,
Marco

Il 2021-03-30 18:12 Marco Mirazita ha scritto:
> Hi Tyler,
> isn't the alarm system based on the cRIO readings? If so, it cannot
> work even if it is still enabled.
> Thank you,
> Marco
>
>
>
> Il 2021-03-30 16:39 Tyler Lemon ha scritto:
>> Hi Marco,
>>
>>  - do we still have remote access to the information on the nitrogen
>> flow?
>>
>>  No, the flow can only be remotely read out through that cRIO.
>>  There is a local display on the nitrogen panel on top of the forward
>> carriage someone can read in person.
>>
>>  - do we still have remote access to the information on the humidity
>> level inside the RICH?
>>  The humidity in the electronic panel can still be read by the EP
>> cRIO. Those humidity readings are still very low, making me confident
>> that there is still N2 flow to RICH.
>>
>>  We cannot read the humidity in the nitrogen volume because the cRIO
>> appears to have failed.
>>
>>  - do we still have the alarm system working?
>>  To my knowledge, all the alarms are still enabled.
>>
>>  - do we still have the backup system working?
>>  Yes, the backup N2 system is still connected and will provide N2 if
>> the main supply pressure stops. This system has no electrically
>> controlled or remote controlled elements so the backup flow
>> automatically starts if N2 supply pressure is lost through the use of
>> a normally-open pneumatic valve.
>>
>> -------------------------
>>
>> From: Marco Mirazita <Marco.Mirazita at lnf.infn.it>
>> Sent: Tuesday, March 30, 2021 10:26 AM
>> To: Tyler Lemon <tlemon at jlab.org>
>> Cc: Denny Insley <dinsley at jlab.org>; Morgan Cook IV
>> <mcookiv at jlab.org>; Bob Miller <rmiller at jlab.org>;
>> dsg-hallb_rich at jlab.org <dsg-hallb_rich at jlab.org>; Valery Kubarovsky
>> <vpk at jlab.org>
>> Subject: [EXTERNAL] Re: [dsg-hallb_rich] Fw: unable to connect to RICH
>> N2 cRIO in hall
>>
>> Hi Tyler,
>> could you please clarify the situation? In particular:
>> - do we still have remote access to the information on the nitrogen
>> flow?
>> - do we still have remote access to the information on the humidity
>> level inside the RICH?
>> - do we still have the alarm system working?
>> - do we still have the backup system working?
>> Thank you,
>> Marco
>>
>> Il 2021-03-29 14:52 Tyler Lemon ha scritto:
>>> Hi Denny and Morgan,
>>>
>>>  Please see the email chain below.
>>>
>>>  Brian went in to Hall B last week to check the the RICH N2 volume
>>> cRIO that failed and it looks like the cRIO will have to be
>> replaced.
>>>
>>>  Until the cRIO issue is resolved, can Hall B Engineering
>> periodically
>>> check the local N2 flow readout for RICH on the top of the forward
>>> carriage to make sure the detector is still getting appropriate N2
>>> flow? We're still all working from home and have to make special
>>> arrangements to go in to the lab.
>>>
>>>  We can remotely see that the humidity in the RICH electronic panel
>> is
>>> still very low (below the point we're able to read with the humidity
>>> sensors), but it'd still be good to check the flow meters in case
>>> something happens to that cRIO.
>>>
>>>  Thanks.
>>>
>>>  -Tyler
>>>
>>> -------------------------
>>>
>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>> Tyler
>>> Lemon <tlemon at jlab.org>
>>> Sent: Monday, March 29, 2021 8:32 AM
>>> To: Amrit Yegneswaran <yeg at jlab.org>; 'dsg-internal'
>>> <dsg-internal at jlab.org>
>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>> hall
>>>
>>> Remotely, no there is not a way.
>>>
>>> If you go up to the top of the forward carriage, there are local
>>> displays on the flow meters.
>>>
>>> -------------------------
>>>
>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>> Amrit
>>> Yegneswaran <yeg at jlab.org>
>>> Sent: Monday, March 29, 2021 8:29:04 AM
>>> To: 'dsg-internal' <dsg-internal at jlab.org>
>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>> hall
>>>
>>>  thanks brian
>>>  we'll talk about this issue in today's meeting
>>>  is there anyway to see wheteher there is nitrogen flow?
>>>
>>> -------------------------
>>>
>>> From: Brian Eng <beng at jlab.org>
>>> Sent: Saturday, March 27, 2021 12:15 AM
>>> To: Amrit Yegneswaran <yeg at jlab.org>; Tyler Lemon <tlemon at jlab.org>;
>>> 'dsg-internal' <dsg-internal at jlab.org>
>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>
>>>  It wouldn't force boot into safe mode (still the same blinking
>> status
>>> LED) nor be detected by MAX when connected via USB or directly with
>> a
>>> network cable.
>>>
>>>  Looks like might have to swap the cRIO out, but I'm going to file a
>>> ticket with NI first to make sure there isn't something else I can
>> do
>>> locally before doing that since that seems like kind of a pain to
>> do.
>>>
>>> -------------------------
>>>
>>> From: Brian Eng <beng at jlab.org>
>>> Sent: Friday, March 26, 2021 12:15 AM
>>> To: Amrit Yegneswaran <yeg at jlab.org>; Tyler Lemon <tlemon at jlab.org>;
>>> 'dsg-internal' <dsg-internal at jlab.org>
>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>
>>>  First look isn't good, the status LED on the cRIO is constantly
>>> blinking which according to the manual means:
>>>
>>>  The cRIO-9035 has not booted into NI Linux Real-Time. The cRIO-9035
>>> either booted into an unsupported operating system, was interrupted
>>> during the boot process, or detected an unrecoverable software
>> error.
>>>
>>>  I did notice that the network went down/up several times along with
>>> it trying to access the SD card, so it seems like it's in some sort
>> of
>>> boot loop.
>>>
>>>  I didn't bring the Windows laptop down with me today and it's later
>>> than I planned on being here so I'll try formatting it locally
>>> tomorrow to see if I can't just get it on the network so that
>>> Tyler/Pete can reinstall the needed software.
>>>
>>> -------------------------
>>>
>>> From: Amrit Yegneswaran <yeg at jlab.org>
>>> Sent: Friday, March 19, 2021 4:45 PM
>>> To: Brian Eng <beng at jlab.org>; Tyler Lemon <tlemon at jlab.org>;
>>> 'dsg-internal' <dsg-internal at jlab.org>
>>> Subject: Re: unable to connect to RICH N2 cRIO in hall
>>>
>>>  brian,
>>>  from tyler's e-mail it appears that since there is still N2 so
>> there
>>> should be no urgent problem.
>>>  which means you can decide when you wish to go.
>>>  like always late in the evening.
>>>
>>>  pity the hallb techs aren't able to help.
>>>
>>>  on the good news aide patrizia, is willing to consider the robotic
>>> version, but we need to come up with a plan on what other project
>>> (which are on the books we can use the system)
>>>  so all, put on your thinking caps
>>>
>>> -------------------------
>>>
>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>> Brian
>>> Eng <beng at jlab.org>
>>> Sent: Friday, March 19, 2021 2:48 PM
>>> To: Tyler Lemon <tlemon at jlab.org>; 'dsg-internal'
>>> <dsg-internal at jlab.org>
>>> Subject: Re: [Dsg-internal] unable to connect to RICH N2 cRIO in
>> hall
>>>
>>>  I can go in and check it tonight if need be.
>>>
>>> -------------------------
>>>
>>> From: Dsg-internal <dsg-internal-bounces at jlab.org> on behalf of
>> Tyler
>>> Lemon <tlemon at jlab.org>
>>> Sent: Friday, March 19, 2021 1:45 PM
>>> To: 'dsg-internal' <dsg-internal at jlab.org>
>>> Subject: [Dsg-internal] unable to connect to RICH N2 cRIO in hall
>>>
>>>  Hello,
>>>
>>>  I'm going to preface this by saying that from the EP cRIO, we can
>> see
>>> that the humidity in RICH is still low and at a good level so RICH
>> is
>>> still getting N2 without issues.
>>>
>>>  That being said, I cannot connect to the N2 volume cRIO (hostname
>>> richcrio) over the network.
>>>
>>>  Peter noticed this a few days ago and I asked Hall B engineering to
>>> check on the cRIO. Their response was not helpful since they just
>>> looked at the cRIO chassis from the outside. I have since asked them
>>> to recheck the chassis, looking inside to see if there are any
>> lights
>>> flashing inside, but this probably won't be done today.
>>>
>>>  It seems more likely than not that someone will have to go in to
>> Hall
>>> B to check the cRIO.
>>>
>>>  Since we can still monitor the internal humidity from the EP cRIO,
>> I
>>> don't think this is something that has to be absolutely taken care
>> of
>>> right this movement.
>>>
>>>  Should I start the process of getting approval to go in myself? Or
>>> would Brian be able to check the cRIO?
>>>
>>>  -Tyler
>>> _______________________________________________
>>> dsg-hallb_rich mailing list
>>> dsg-hallb_rich at jlab.org
>>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
>> _______________________________________________
>> dsg-hallb_rich mailing list
>> dsg-hallb_rich at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
> _______________________________________________
> dsg-hallb_rich mailing list
> dsg-hallb_rich at jlab.org
> https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
_______________________________________________
dsg-hallb_rich mailing list
dsg-hallb_rich at jlab.org
https://mailman.jlab.org/mailman/listinfo/dsg-hallb_rich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/dsg-hallb_rich/attachments/20210402/4311fb14/attachment-0001.html>