[Sbs_software] Hydra online monitoring - plots and labeling

[Ole Hansen]

Hi everyone,

our friends in the EPSCI group have more or less finished porting Hall 
D's AI-based online monitoring system, Hydra, to Hall A/SBS. The way it 
works is roughly like this:

- We run the 10k/50k replay as we usually do
- A slightly modified version of Panguin writes the output plots not 
only to PDFs but also to individual image files
- Hydra detects new image files on disk and uses ML inference to 
classify each one
- If there are anomalies, i.e. Hydra thinks a plot is "bad", an alarm sounds

Unlike our current alarm handler, AI-based image classification can 
detect subtle problems, like hot channels, unusually low yield, 
distorted spectra etc. Of course, it can also get hopelessly confused. 
We almost certainly will have to fine-tune things when the next run 
period starts.

Now, for Hydra to work, its ML model needs to be trained with existing 
data. For this purpose, we've run Panguin over the ROOT files from the 
GMn experiment, extracted images of the plots (a subset—the macro-based 
Panguin plots aren't there yet), and loaded a few of them into the Hydra 
database for labeling.

Labeling means that a human expert has to manually classify each plot. 
(This only needs to be done once, of course, after which the model makes 
predictions on its own.) The labeling can be done through a web 
interface, which also gives a nice overview of the run-by-run data 
quality for each individual plot.

It would be great if our online monitoring experts (Paul, ...) could 
take a peek at the labeler web page (CUE login required) and give us 
some feedback:

https://hallaweb.jlab.org/hydra/labeler.html

It should be mostly self-explanatory, but instructions are here:

https://docs.google.com/document/d/1h4AC5UL_DB4Ugjw1GlybNdaq23uAYrc9m0R37sTxG1s/edit?usp=sharing

The page will probably tell you that you do not have permission to label 
anything, but you can look around. We need to decide who should be given 
write access.

If you don't see any plots, choose a different plot type from the 
drop-down list (top center) or switch on Editor Mode (top left).

As you will see, the database contains only 5 different types of plots 
at the moment. These are a few simple ones we uploaded for testing. We 
need to decide on the full set of plots we'd like to have monitored by 
Hydra.

The images that Panguin extracted for each run are in these directories 
on the counting house computers:

/chafs2/work1/hydra/<run_number>/

Again, it would be great if our online experts could look at these plots 
and make a list of the ones worth monitoring. Also, a list of plots 
still missing that should definitely be monitored. As I said, the 
macro-based Panguin plots aren't there yet.

I'll be happy to give a 10 minute demonstration of the labeler and 
answer questions at the software meeting later today.

Ole