[Sbs_gems] [EXTERNAL] RE: GEM MPD DAQ - APV hang up issue for January run

Wed Dec 22 09:21:29 EST 2021

Hi Ben,
Thanks for the feedback. Yes the plan is to make some warning message available in the RunControl GUI but not to stop the run. Then shift crew will be instructed to check this error message during each run and alert the GEM expert immediately.
It will also be very helpful to have a log entry pass/fail to be emailed directly to GEM expert as you suggested

I understand that it is a hardware issue that get fixed only after access to the hall but it is certainly not the only issue that causes the MPD/DAQ to hang

Best regards
Kondo

From: Benjamin Raydo <braydo at jlab.org>
Sent: Tuesday, December 21, 2021 9:16 PM
To: Gnanvo, Kondo (kg6cq) <kg6cq at virginia.edu>; Bryan Moffit <moffit at jlab.org>; Alexandre Camsonne <camsonne at jlab.org>
Cc: sbs_gems at jlab.org
Subject: Re: GEM MPD DAQ - APV hang up issue for January run

Hi Kondo,

It doesn't seem like a big deal to change the readout list to work that way (automatically disable the APVs that failed to initialize and report a warning). Somehow someone will have to be on top of this to ensure long runs are not taken before noticing this issue (which would probably boil down to someone looking at RunControl GUI warnings and reporting this to GEM experts).

I would definitely have to defer to Bryan on implementing this and he's likely a lot more aware of possibilities on how this could be reported (maybe a GMN GEM status log entry for pass/fail that has a GEM expert emailed to help keep them on top of the issue? Not sure how reasonable/feasible this would be).

I definitely understand nothing here is ideal short of fixing the APV init issues. If you think it would be helpful to summarize the type issues and ways they are recovered maybe we could try to figure out a software or firmware fix (we have some of these issues showing up in EEL125 setup too so I was planning to look more at this in January anyway - I'm definitely aware of some cases where the MPD firmware is in a funny state that causes all APVs to fail to initialize that I'm hoping to address for example).

Ben
________________________________
From: Gnanvo, Kondo (kg6cq) <kg6cq at virginia.edu<mailto:kg6cq at virginia.edu>>
Sent: Tuesday, December 21, 2021 11:55 AM
To: Bryan Moffit <moffit at jlab.org<mailto:moffit at jlab.org>>; Alexandre Camsonne <camsonne at jlab.org<mailto:camsonne at jlab.org>>; Benjamin Raydo <braydo at jlab.org<mailto:braydo at jlab.org>>
Cc: sbs_gems at jlab.org<mailto:sbs_gems at jlab.org> <Sbs_gems at jlab.org<mailto:Sbs_gems at jlab.org>>
Subject: [EXTERNAL] GEM MPD DAQ - APV hang up issue for January run

Dear Bryan, Ben.

We discussed at the RC meeting yesterday whether MPD readout list could be modified to allow the DAQ to keep on running even if a single APV fail to configure properly at the beginning of a run.

Right now, we have had a lot of occurrence where one APV failure cause an MPD / fiber link error that a shift crew can not really debug or fix. A GEM or DAQ expert will ultimately be called and it would results to this APV been disabled. This usually results in a lot of wasted beam time during the shift without even fixing the APV problem.

So it will be more efficient if the APV25 configuration fail is ignored and allow the DAQ to continue with some warning that will allow the shift crew to alert the GEM expert who can take a look at the issue when there is an opportunity. This way, there is no beam time lost.

Maybe the DAQ configuration will be allow to crash only if a number of APV25 (that we decide on, maybe 3 or 5 ...), or all APVs on a backplane or MPD  all fail to configure which would indicate a more serious problem.

I was wondering if it is possible to implement such scheme in the MPD readout list

Best regards

Kondo

From: Sbs_gems <sbs_gems-bounces at jlab.org<mailto:sbs_gems-bounces at jlab.org>> On Behalf Of Holly Szumila-Vance
Sent: Monday, December 20, 2021 4:12 PM
To: sbs_gems at jlab.org<mailto:sbs_gems at jlab.org>
Subject: [Sbs_gems] No SBS GEM meeting on Wednesday this week

Dear all,

There will be no SBS GEM meeting this Wednesday. Please look for updates by e-mail.

Enjoy the holidays, and see you in the New Year!

-Holly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/sbs_gems/attachments/20211222/66a91ba1/attachment-0001.html>