[Vetroc_daq_sbs] VETROC testing status and open questions

[Benjamin Raydo]

Hi Evan,

Perhaps it would be best if I could meet with you at your setup sometime next week (or later if needed)? In the meantime, if you could let me know a machine I can logon to and where the readout list and library for the VETROC exist I could take a peak to see if I understand any reasons for the issues.

I seem to recall hearing about point 2 you're asking Bob about - perhaps this is related to the need for the TS (or master TI) to generate some trigger to reach a complete block on the end of a run? You may want to discuss with Bryan Moffit who would be very familiar with how this is handled.

For the questions you're asking me:
1) No clue - the hardware can't anticipate the end of a run, so I don't see how it could bias the data if it were the last block of the run! I suspect this may be related to the end of run failing.
2) The trigger rate sounds too high for the running conditions (and/or buffer level * block size is huge):
     -For low occupancy, the dominating rate limit is based on the readout window size - so you should try to make this only as big as needed, otherwise the firmware spends more time per event search for hits (I believe the firmware you have needs roughly the readout window size amount of time to build the event, so a 1us readout window you might be able to run at 1MHz - for low occupancy).
     -The hit rate limit should be about 4MHz per channel on average to prevent from loosing hits. I could improve this with a firmware change if needed, but I suspect you're not at this limit. Readout hit rate should be on the order of 20MHz assuming you saturate the VME bus reading out a single board.

I would be interested in finding out:
  1) Expected average trigger rate needed to run
  2) Expected readout window size
  3) Expected hit rate per channel (just raw, not readout, in Hz)

If you are trying to push limit to the maximum, we should introduce a BUSY signal that comes from the VETROC that will tell the TI to stop taking triggers when it gets backed up. Otherwise, we will need to set large hold-off times after each trigger. I would also like to introduce another firmware feature that limits the number of hits built per event this way there is bound on how long an event may take to build (right now event building time/size is unbounded which could result in loss of data at large windows/high trigger rates).

Anyhow, if we can meet and exchange a few words/idea on how things are working and what it needs to achieve then I should be able to update firmware/drivers pretty quickly as needed to reach that goal.

Thanks,
Ben

----- Original Message -----
From: "R. Evan McClellan" <randallm at jlab.org>
To: "Vetroc_daq_sbs" <vetroc_daq_sbs at jlab.org>
Cc: "Benjamin Raydo" <braydo at jlab.org>
Sent: Thursday, February 9, 2017 4:26:00 PM
Subject: VETROC testing status and open questions

Hi all,

Carlos and I ran another small set of tests. This time, we used an 'always on' LED to generate random hits on all PMTs. (The trigger rate was at ~10kHz for all tests)

The conclusions are roughly the same:

1- Somewhere around 100 kHz, we start missing ~50% of the events.
2- At ~350 kHz and above, the raw data starts corrupting (missing Event Headers and Trigger Time words)
3- Even for much lower rate (~10Hz), we noticed that the events at the end of the last block (after the final trigger), which should be empty, often have hits in signal channels.


I would summarize our current status as five questions (two for Bob, two for Ben, and one in general):

Bob:
1- ROC 9 is sometimes very slow (lots of "waiting for ROC9" in CODA). It is usually fixed by rebooting. Is this a common issue? Is there a known fix?
2- Occasionally, the last Block of a run will go missing, and then apparently get read out with the next run. Have you seen this before?

Ben:
1- How are hits showing up in the 'empty' events at the end of the last block of a run?
2- What causes the Event Header and Trigger Time words to go missing at high rate? Can we alter the firmware to make the readout format immune to high signal rates?

General:
1- What is our actually rate requirement for the GRINCH DAQ? (I have previously heard 1MHz per channel. It's pretty clear that the VETROC could not handle that...)

Cheers,
Evan



R. Evan McClellan, PhD
Hall A Postdoctoral Fellow
Jefferson Lab