[Sbs_gems] [Revised Logentry] Reason of last week's gas flow problem

[chandan at jlab.org]

Logentry Text:
--
During the last week (3rd July), the GEM gas supply was stopped (See Fig 1).
It was found that the bottle switchover didn't happen and it was thought the nitrogen bottles that were used for opening the pneumatic values were empty.

Actually, there was a power outage on the Hall A gas shed on July 2nd at 10 am. This can be seen from Fig 2. The SBS side EPICS readout stopped immediately as there is no UPS power backup.
For the BB side, the IOC ran for 20 min after the power outage as there is a UPS backup.
Figure 2 shows the the two N2 bottle pressures (Ch_26 and Ch_27), the N2 pressure in the line (Ch_20) which controls the pneumatic values. One can see the Ch_20 dropped to zero immediately as the power went off, although the individual bottle pressures remained almost the same.
This zero pressure in Ch_20 would close all the pneumatic valves that connect individual bottles to the supply lines.

In this week I simulated the situation that happened last week by turning off power to the BB side IOC and gas control. What I found was that after rebooting the gas control and IOC, the gas switchover control got into the situation that is shown in Fig 3. All the bottles were 'ready' to use. But, none of them (either Ar, CO2, or N2) was in the 'run' states.
So, after that, the mixing system used the existing gas that was in the supply line. That gas lasted until 10 pm on the 2nd and on the 3rd the GEM people realized that there was no gas flow into their system.
On the 3rd, Ellen forced the N2 bottle to be in the 'run' state and made the bottle switchover possible and gas flow started. Since then the system has been working as it is supposed to be.


I also noticed that the IOC stopped reading EPICS PVs after the power outage (as can be seen in Fig 1). The EPICS PVs were not read back until this week when I reset the crate. It is still not clear to me why that is the case. However, for the SBS side, the EPICS readout started as soon as power returned (See the Magenta trace in Fig 2). The SBS EPICS readout is done by an RPI, not via an IOC.

The only problem I noticed after the power glitch was that the BB side binary gas analyzer (BGA) display stopped working (see Fig. 4). I checked the gas mixing ratio from this BGA, and that looks okay.
So, it might be just the display went back. For the upcoming runs, the SBS experiment will use GEM detectors only on the SBS side. So, we will discuss this with GEM experts and take action on this BGA.


---

This is a plain text email for clients that cannot display HTML.  The full logentry can be found online at https://logbooks.jlab.org/entry/4312122
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/sbs_gems/attachments/20240709/db83a98c/attachment.html>