<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
It sounds like from David's message, that the evio data file headers are written as bigendian, and then from what Ole says the evio reader interprets that as the whole data file needs to be byteswapped.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
That would maybe explain why I needed to byteswap the helicity scaler data words.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Does that make sense? <span id="ms-outlook-android-cursor"></span></div>
<div><br>
</div>
<div id="ms-outlook-mobile-signature">Sent from my mobile device.<br>
Get <a href="https://aka.ms/ghei36">Outlook for Android</a></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Ole Hansen <ole@jlab.org><br>
<b>Sent:</b> Sunday, October 3, 2021 3:06:20 PM<br>
<b>To:</b> Alexandre Camsonne <camsonne@jlab.org>; Paul King <pking@jlab.org><br>
<b>Cc:</b> Andrew Puckett <puckett@jlab.org>; Robert Michaels <rom@jlab.org>; sbs_daq@jlab.org <sbs_daq@jlab.org><br>
<b>Subject:</b> Re: Big endian raw data?</font>
<div> </div>
</div>
<div>When EVIO opens a file, it looks for a "magic" word in a certain place early in the file. If it isn't found, it byte-swaps that word and checks if it matches the magic value then. If it does, it sets a flag that this file needs byte-swapping, and all header
and data words, except "unknown" data (see the swap_data routine in evioswap.c), are then swapped. There is no (easy) way to enable/disable this flag outside of the library; it's hardcoded, pre-determined behavior.<br>
<br>
I am also not sure if there are any flags in any of the headers to indicate the endianness of the data. It would just require a single bit. The headers are not user-accessible anyway, BTW, so it looks like you're on your own to find out whether you need swapping,
on a per-module or per-ROC basis, I guess. Like I said, it's not very clear. What does "VME is big endian" really mean? An Intel CPU in a VME crate produces what endianness ... 🤔<br>
<br>
Ole<br>
<br>
<div class="x_moz-cite-prefix">On 3.10.21 at 14:21, Alexandre Camsonne wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">
<div>Everything is intel besided VTP. Though Dave mentionned VME was big endian.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Alexandre<br>
<br>
<div class="x_gmail_quote" dir="auto">
<div dir="ltr" class="x_gmail_attr">On Sun, Oct 3, 2021, 13:56 Paul King <<a href="mailto:pking@jlab.org">pking@jlab.org</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
I can comment that when I wrote the helicity scaler library, I found that I needed to byte swap the data words (module data and diagnostic counters) on the crate in order to be decoded correctly.
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
I'm not sure if halladaq8 is an Intel or arm cpu.<br>
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
Does Podd or evio2xml do a dynamic check of endianness and then byteswap, or is that explicitly enabled?
<span id="x_m_-188557246818857469ms-outlook-android-cursor"></span></div>
<div><br>
</div>
<div id="x_m_-188557246818857469ms-outlook-mobile-signature">Sent from my mobile device.<br>
Get <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_ghei36&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=FxpSg1qAggzMg8YCjcvm4w&m=W_Py47zEQFMF4YLSxKlQ6YvCZkjxlCgMIaViW0IvmiA&s=HtjtqwZf9TY9s3jftpebziiNiD97Y3BAaURjA_dYHjc&e=" target="_blank" rel="noreferrer">
Outlook for Android</a></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-188557246818857469divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Sbs_daq <<a href="mailto:sbs_daq-bounces@jlab.org" target="_blank" rel="noreferrer">sbs_daq-bounces@jlab.org</a>> on
behalf of Andrew Puckett <<a href="mailto:puckett@jlab.org" target="_blank" rel="noreferrer">puckett@jlab.org</a>><br>
<b>Sent:</b> Sunday, October 3, 2021 1:31:05 PM<br>
<b>To:</b> Robert Michaels <<a href="mailto:rom@jlab.org" target="_blank" rel="noreferrer">rom@jlab.org</a>>; Ole Hansen <<a href="mailto:ole@jlab.org" target="_blank" rel="noreferrer">ole@jlab.org</a>>;
<a href="mailto:sbs_daq@jlab.org" target="_blank" rel="noreferrer">sbs_daq@jlab.org</a> <<a href="mailto:sbs_daq@jlab.org" target="_blank" rel="noreferrer">sbs_daq@jlab.org</a>><br>
<b>Subject:</b> Re: [Sbs_daq] Big endian raw data?</font>
<div> </div>
</div>
<div lang="EN-US" style="word-wrap:break-word">
<div>
<p>Interesting. So perhaps I’m being naïve here, but other than the byte-swapping inefficiency Ole pointed out in processing the raw data on the compute farm nodes, is there an actual problem here? Do we need to check/care about this in the software in writing
our raw data decoders?</p>
<p> </p>
<p>The cause of Bradley’s crash while processing GRINCH data doesn’t necessarily seem related to this…
</p>
<p> </p>
<p>Andrew </p>
<p> </p>
<div style="border:none; border-top:solid #b5c4df
1.0pt; padding:3.0pt 0in 0in 0in">
<p style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">Robert Michaels <<a href="mailto:rom@jlab.org" target="_blank" rel="noreferrer">rom@jlab.org</a>><br>
<b>Date: </b>Sunday, October 3, 2021 at 1:21 PM<br>
<b>To: </b>Ole Hansen <<a href="mailto:ole@jlab.org" target="_blank" rel="noreferrer">ole@jlab.org</a>>, Andrew Puckett <<a href="mailto:puckett@jlab.org" target="_blank" rel="noreferrer">puckett@jlab.org</a>>,
<a href="mailto:sbs_daq@jlab.org" target="_blank" rel="noreferrer">sbs_daq@jlab.org</a> <<a href="mailto:sbs_daq@jlab.org" target="_blank" rel="noreferrer">sbs_daq@jlab.org</a>><br>
<b>Subject: </b>Re: [Sbs_daq] Big endian raw data?</span></p>
</div>
<div>
<p><span style="font-size:12.0pt; color:black">I believe there are byte-swapping routines available in the DAQ libraries which allow to put the bytes in the right state and be consistent. But the DAQ expert needs to make this happen. Below is a snippet of
an email from Dave Abbott about a year ago when I was having some trouble, which I think is relevant.. Dave is a good person to ask. Can ask Bryan Moffit or Alexandre, too.</span></p>
</div>
<div>
<p><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p><span style="font-size:12.0pt; color:black">---------------------- snippet of email from Dave Abbott ------------------------</span></p>
</div>
<div>
<p><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p><span style="font-size:12.0pt; color:black; background:white">The CODA data files are written from a Java Event Builder. JAVA is inherently Big Endian. The EVIO</span><span style="font-size:12.0pt; color:black">
</span></p>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black">files will be by default in big endian.</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black">However, ALL Banks of User data - created in your readout list - will NOT be swapped. They will stay</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black">whatever Endian it was when it was written.</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black">Typically the ROC will run in Linux on Intel which is Little Endian. Therefore the Data banks you create will stay</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black">little endian. However the Bank headers will be swapped to be compatible with the rest of the CODA file.</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black">An even more confusing possibility is that you might do a DMA from the VME bus into a CODA data Bank.</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black">The VME bus is Big endian. Therefore the data from the VME bus will stay Big endian in this bank.</span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p style="background:white"><span style="font-size:12.0pt; color:black">Our general rule for CODA 3 is that for purposes of DAQ we will not touch (or modify) the User's data in any way.</span></p>
</div>
<p><span style="font-size:12.0pt; color:black; background:white">We will only modify the EVIO headers to match the endianess of whatever System writes the file.</span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<div>
<p><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div id="x_m_-188557246818857469x_Signature">
<div>
<div id="x_m_-188557246818857469x_divtagdefaultwrapper">
<p><span style="font-size:12.0pt; color:black"> </span></p>
</div>
</div>
</div>
</div>
<div align="center" style="text-align:center">
<hr width="100%" size="0" align="center">
</div>
<div id="x_m_-188557246818857469x_divRplyFwdMsg">
<p><b><span style="color:black">From:</span></b><span style="color:black"> Sbs_daq <<a href="mailto:sbs_daq-bounces@jlab.org" target="_blank" rel="noreferrer">sbs_daq-bounces@jlab.org</a>> on behalf of Ole Hansen <<a href="mailto:ole@jlab.org" target="_blank" rel="noreferrer">ole@jlab.org</a>><br>
<b>Sent:</b> Sunday, October 3, 2021 1:06 PM<br>
<b>To:</b> Andrew Puckett <<a href="mailto:puckett@jlab.org" target="_blank" rel="noreferrer">puckett@jlab.org</a>>;
<a href="mailto:sbs_daq@jlab.org" target="_blank" rel="noreferrer">sbs_daq@jlab.org</a> <<a href="mailto:sbs_daq@jlab.org" target="_blank" rel="noreferrer">sbs_daq@jlab.org</a>><br>
<b>Subject:</b> Re: [Sbs_daq] Big endian raw data?</span> </p>
<div>
<p> </p>
</div>
</div>
<div>
<p style="margin-bottom:12.0pt">Maybe our various front-ends differ in endianness, so we write mixed-endian data?!? That would be disastrous since it is not supported by EVIO. A file can only be one or the other—a very binary view. (I guess EVIO was written
before we became diversity-aware ;) ).<br>
<br>
Ole</p>
<div>
<p>On 3.10.21 at 13:03, Andrew Puckett wrote:</p>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<div>
<p>Hi Ole, </p>
<p> </p>
<p>This is interesting. The GRINCH data are being read out by the new VETROC modules, I don’t know if they differ from the other modules in terms of “endian-ness”. Maybe a DAQ expert can weigh in here?</p>
<p> </p>
<p>Andrew </p>
<p> </p>
<div style="border:none; border-top:solid
#b5c4df 1.0pt; padding:3.0pt 0in 0in 0in">
<p style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt; color:black">From:
</span></b><span style="font-size:12.0pt; color:black">Sbs_daq <a href="mailto:sbs_daq-bounces@jlab.org" target="_blank" rel="noreferrer">
<sbs_daq-bounces@jlab.org></a> on behalf of Ole Hansen <a href="mailto:ole@jlab.org" target="_blank" rel="noreferrer">
<ole@jlab.org></a><br>
<b>Date: </b>Sunday, October 3, 2021 at 1:00 PM<br>
<b>To: </b><a href="mailto:sbs_daq@jlab.org" target="_blank" rel="noreferrer">sbs_daq@jlab.org</a>
<a href="mailto:sbs_daq@jlab.org" target="_blank" rel="noreferrer"><sbs_daq@jlab.org></a><br>
<b>Subject: </b>[Sbs_daq] Big endian raw data?</span></p>
</div>
<p style="margin-bottom:12.0pt">Hi guys,<br>
<br>
Bradley reported a crash of the replay (actually in EVIO) with /adaq1/data1/sbs/grinch_72.evio.0 (see
<a href="https://logbooks.jlab.org/entry/3916105" target="_blank" rel="noreferrer">
https://logbooks.jlab.org/entry/3916105</a>).<br>
<br>
When digging into the cause of this crash, I discovered that these raw data are written in big-endian format. How can this be? I thought the front-ends are Intel processors. Are we taking data with ARM chips that are configured for big-endian mode? Is this
a mistake, or is there some plan to it?<br>
<br>
These big-endian data have to be byte-swapped when processing them on x86, which is what all our compute nodes run. That's a LOT of work. It leads to significant and seemingly completely unnecessary overhead. I.e. we're burning CPU cycles for nothing good,
it seems.<br>
<br>
Please explain.<br>
<br>
Ole</p>
</div>
</blockquote>
<p> </p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Sbs_daq mailing list<br>
<a href="mailto:Sbs_daq@jlab.org" target="_blank" rel="noreferrer">Sbs_daq@jlab.org</a><br>
<a href="https://mailman.jlab.org/mailman/listinfo/sbs_daq" rel="noreferrer noreferrer" target="_blank">https://mailman.jlab.org/mailman/listinfo/sbs_daq</a><br>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</body>
</html>