[Moller_daq] [EXTERNAL] MOLLER raw data file size

[mgericke]

Hi Paul,

In that case, you need the first 3 and a subset of 4 (the 
minimum/maximum values, plus maybe trigger information), plus the 3 
final window block running sum values.
Could there be an advantage to keep the running sums, rather than just 
the final total value ? If there is data loss, it could tell your in 
which block it occurred. But maybe
we don't care and just throw away the entire sequence at that point.

Thanks,

Michael


On 2025-09-05 11:12 a.m., Paul King wrote:
> Dear Michael,
>
> We would store the raw data from the ADC boards in binary form.  However, I had thought to do a data filtering step either in the CODA readout list on the CPU, since we  will not need the top longword of the 64-bits on the sum of all samples, for example, and would not need to keep each instance of the running sum.
>
> Paul.
>
> ________________________________________
> From: Michael Gericke <mgericke at physics.umanitoba.ca>
> Sent: Friday, September 5, 2025 12:01 PM
> To: Paul King
> Cc: moller_analysis at jlab.org; moller_daq at jlab.org
> Subject: [EXTERNAL] MOLLER raw data file size
>
> Hi Paul,
>
> Is the plan for the MOLLER data to write the raw data (as it comes from
> the ADC boards) to file in binary form,
> or only the "condensed" root files?
>
> Right now, including the total helicity window data (which I added to
> the firmware last night) as well as the block
> data, the raw data files size for a 1 minute run is 445440000 bytes.
> This is for a reversal rate of 2 kHz and 4 blocks
> per window. Each block of data within a given helicity window is a
> separate packet, which is written to file.
>
> Each block includes 16x7 (16 channels times 7 data items) 64 bit data
> registers, plus a 4x64 bit packet header, including
> block number, time stamp, and total sample count (over all 16 channels).
>
> Within each block we have the following data items:
>
> 1) 64 bit sum of all samples per block for each of 16 channels
> 2) 64 bit sum-of-squares per block for each of 16 channels
> 3) 64 bit number of samples per block for each of 16 channels
> 4) 64 bit:  20 bit maximum + 20 bit minimum  sample values + 2 bit
> trigger state (both input trigger Lemo signals) + 22 bits empty/free per
> block for each of 16 channels
> 5) 64 bit running sum of all samples over the sequence of blocks in a
> helicity window for each of 16 channels
> 6) 64 bit running sum-of-squares over the sequence of blocks in a
> helicity window for each of 16 channels
> 7) 64 bit running sample number over the sequence of blocks in a
> helicity window for each of 16 channels
>
> The running sums over all blocks per helicity window are cleared with
> each new trigger, but are kept for all
> block packets rather than only sending the final values with the last
> block packet. I think doing the latter would
> require a more complicated setup for the firmware with some sort of
> interleaved packet transmission (since they
> would have different sizes).
>
> I tested the transmission in longer runs last night and with regard to
> that there doesn't seem to be a problem (e.g. dead
> time). From a data storage point of view this is a little wasteful, but
> we could throw away the intermediate running
> values before writing it to file.
>
> Right now, I am just dumping every packet as I receive it to a binary file.
>
> Of course, if you change the number of blocks per window, the file size
> changes. Right now, the maximum number
> of blocks that can be specified per helicity window is 16 (based on the
> 8 bit register size). So if that were to be chosen
> for some reason, it would increase the file size by a factor of 4
> relative to size I stated at the beginning.
>
> Thanks,
>
> Michael
>
> _______________________________________________
> Moller_daq mailing list
> Moller_daq at jlab.org
> https://mailman.jlab.org/mailman/listinfo/moller_daq