[Alert_collaboration] [New Logentry] RC Summary July 26 - July 27 Owl

[schmidta at jlab.org]

Logentry Text:
--
This is the RC summary for Day Shift July 26 through Owl shift July 27.

Yesterday started with a loss of communication to an HV crate powering part of EC Sector 2 (and part of the LTCC). Remote reboots were only partially able to restore communications. (https://logbooks.jlab.org/entry/4427418). Furthermore, the problem precipitated a complete loss of triggers in sector 2 (https://logbooks.jlab.org/entry/4427525), which was only corrected by Sergey by about 12:00. At 12:30, the decision was made for Nathan to do a rapid access to do a power cycle of the crate, and to replace any bad boards. This work was successful at restoring communications (https://logbooks.jlab.org/entry/4427575).

Simultaneous to this, we discovered a problem with the sound coming from the alarm handler. The sound message being played when the DAQ is ready conflicted with the alarm handler and caused new alarms to make no sound. We found that we had missed an AHDC trip for several hours because there were many other alarms flashing red, and this one made no new noise. Nathan made a stop gap solution to have the alarm annunciator play on clonpc16 rather than clonpc17 (https://logbooks.jlab.org/entry/4427597). My takeaways are 1) we need to be vigilant about making sure the alarms are making noise like they are supposed to (and if they are not, call in the cavalry!), 2) Linux audio is fragile and that if possible we should not add any new sound-playing scripts that might accidentally interfere with our alarm systems. 

Once the above problems were remedied, data taking was mostly smooth over swing and owl.

The Accelerator Overview page (down yesterday) is up and working again. 

- 7/25 day shift summary: https://logbooks.jlab.org/entry/4427415
- 7/25 swing shift summary: https://logbooks.jlab.org/entry/4427658
- 7/26 owl shift summary: https://logbooks.jlab.org/entry/4427885

At the time of this writing, there is one open issue to note: two trigger bits (19 and 22) associated with the muon trigger went down in rate. The alarm range has been adjusted to avoid constant alarms. 

ABU for last 24h

- 18.6h of ABUs ~ 78% efficiency,

- Total ABUs since April 5th: 1208.7+18.6=1227.3 --> 1227.3 / (75.5 * 24) = 67.7%

- Day 113 of 151 scheduled: 113/151= 74.8%

- BANU: 3.1h due to access to restore HV communications, work to debug trigger problems, DAQ crashes

Plan for today:
* Production on He with 325nA

RC meeting:

We will not have an RC meeting today or tomorrow. If any issues arise, please pick up the phone and call me, any hour: 757-575-7540.

---

This is a plain text email for clients that cannot display HTML.  The full logentry can be found online at https://logbooks.jlab.org/entry/4428226
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/alert_collaboration/attachments/20250727/a53d3ffd/attachment.htm>