<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Annalisa,<div><br></div><div>I was made aware of this issue, I looked through the files that are causing problems and discovered that the files are missing the dictionary, that’s why the denoising is failing.</div><div>However, I could not find the cause of this, my best guess is that there was a failure to read the disk during decoding that did not fill the dictionaries into the file.</div><div><br></div><div>I recovered the file by simply running an update on the decoded file, which re-writes the dictionary. Then everything works.</div><div>Maybe this step can be added to the workflow, I think it is in the workflow, but if I’m not mistaken (Nathan can comment on this) it is after the decoder is run,</div><div>If the update dictionary code is run before the decoder, everything should work in principle. </div><div><br></div><div>Nathan, can you comment on when do you run the hypo-utils -update?</div><div><br></div><div><br><div>
<meta charset="UTF-8"><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>— Gagik Gavalian<br>— Staff Scientist<br>— Jefferson Lab<br>— email: gavalian@jlab.org</div></div>
</div>
<div><br><blockquote type="cite"><div>On Oct 11, 2023, at 8:08 AM, Annalisa D'Angelo <annalisa.dangelo@roma2.infn.it> wrote:</div><br class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><p>Dear Gagik,</p><p>thank you for all the work you are carrying on for the CLAS
collaboration and for RG-K in particular.</p><p>I understand that there are about 10 files, which are being
processed for pass2, whose reconstruction keep failing because of
some incompatibility of the de-noising code and the input file
format.</p><p>This problem has been holding us up for several weeks already.
Can you please help us out at your earliest convenience? it is
becoming urgent...</p><p>Thank you in advance <br>
</p><p>Annalisa</p><p>for RG-K<br>
</p>
<div class="moz-forward-container"><br>
<br>
-------- Messaggio Inoltrato --------
<table class="moz-email-headers-table" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Oggetto:
</th>
<td>Fwd: [EXTERNAL] weird files</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Data: </th>
<td>Wed, 11 Oct 2023 12:44:29 +0200</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Mittente:
</th>
<td>Raffaella De Vita <a class="moz-txt-link-rfc2396E" href="mailto:Raffaella.Devita@ge.infn.it"><Raffaella.Devita@ge.infn.it></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">A: </th>
<td>Annalisa D'Angelo <a class="moz-txt-link-rfc2396E" href="mailto:annalisa@roma2.infn.it"><annalisa@roma2.infn.it></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<br>
<div><br>
<blockquote type="cite">
<div>Begin forwarded message:</div>
<br class="Apple-interchange-newline">
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>From: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Gagik Gavalian <<a href="mailto:gavalian@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">gavalian@jlab.org</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Subject: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;"><b>Re: [EXTERNAL] weird files</b><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Date: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">5 October 2023 at 19:14:47 CEST<br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>To: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Raffaella De Vita <<a href="mailto:Raffaella.Devita@ge.infn.it" class="moz-txt-link-freetext" moz-do-not-send="true">Raffaella.Devita@ge.infn.it</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Cc: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Nathan Baltzell <<a href="mailto:baltzell@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">baltzell@jlab.org</a>>,
Lucilla Lanza <<a href="mailto:lucilla.lanza@roma2.infn.it" class="moz-txt-link-freetext" moz-do-not-send="true">lucilla.lanza@roma2.infn.it</a>><br>
</span></div>
<br>
<div>
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
I am looking into it, in principle, this should never
happen. I’ll figure it out, (hopefully)
<div><br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>— Gagik Gavalian<br>
— Staff Scientist<br>
— Jefferson Lab<br>
— <a href="mailto:gavalian@jlab.org" moz-do-not-send="true">email: gavalian@jlab.org</a></div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Oct 5, 2023, at 1:10 PM, Raffaella
De Vita <<a href="mailto:Raffaella.Devita@ge.infn.it" class="moz-txt-link-freetext" moz-do-not-send="true">Raffaella.Devita@ge.infn.it</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
So the decoded file has no schema?? How can that
happen?<br>
<div><br>
<blockquote type="cite">
<div>On 5 Oct 2023, at 17:59, Gagik
Gavalian <<a href="mailto:gavalian@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">gavalian@jlab.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hi Guys,
<div><br>
</div>
<div>When I run decoder on
this file, it doe not crash, but
rather gracefully exits with a
message:</div>
<div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0);">
<span style="font-variant-ligatures: no-common-ligatures">-- created
denoiser with 8 threads</span></div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0); min-height: 21px;">
<br>
<span style="font-variant-ligatures: no-common-ligatures"></span></div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0);">
<span style="font-variant-ligatures: no-common-ligatures">hipo::dictionary
(ERROR) schema {DC::tdc} does
not exist... exiting</span></div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0); min-height: 21px;">
<br>
<span style="font-variant-ligatures: no-common-ligatures"></span></div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0); min-height: 21px;">
hipo::dictionary (ERROR) schema
{DC::tdc} does not exist...
exiting<span style="font-variant-ligatures: no-common-ligatures"></span></div>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div><br>
</div>
<div><br>
</div>
<div>However, if I run
the -update option first:</div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0);">
<span style="font-variant-ligatures: no-common-ligatures">$PROJECT/distribution/coatjava/coatjava/bin/hipo-utils
-update -d
$PROJECT/distribution/coatjava/etc/bankdefs/hipo4
-o
clas_005886.evio.dict.hipo
clas_005886.evio.00325-00329.hipo</span></div>
</div>
<div><br>
</div>
<div>Then the new file
with the dictionary runs
through de-noiser without any
problems.</div>
<div><br>
</div>
<div><br>
</div>
<div>— Gagik Gavalian<br>
— Staff Scientist<br>
— Jefferson Lab<br>
— <a href="mailto:gavalian@jlab.org" moz-do-not-send="true">email:
gavalian@jlab.org</a></div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Oct 5, 2023, at
11:41 AM, Raffaella De Vita
<<a href="mailto:Raffaella.Devita@ge.infn.it" class="moz-txt-link-freetext" moz-do-not-send="true">Raffaella.Devita@ge.infn.it</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>Hi Gagik,</div>
<div><br>
</div>
<div>Here is the
message about the RG-K
problematic files.</div>
<div><br>
</div>
<div>Raffaella</div>
<div><br>
<blockquote type="cite">
<div>Begin
forwarded message:</div>
<br class="Apple-interchange-newline">
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>From:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Nathan
Baltzell <<a href="mailto:baltzell@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">baltzell@jlab.org</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Subject:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;"><b>weird
files</b><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Date:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">September
22, 2023 at
12:33:27 PM EDT<br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>To:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Gagik
Gavalian <<a href="mailto:gavalian@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">gavalian@jlab.org</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Cc:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Lucilla
Lanza <<a href="mailto:lanza@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">lanza@jlab.org</a>><br>
</span></div>
<br>
<div>
<div>Hi
Gagik,<br>
<br>
Do you see
anything
interesting about
this file?<br>
<br>
/cache/clas12/rg-k/production/decoded/6b.2.0/005886/clas_005886.evio.00325-00329.hipo<br>
<br>
Size doesn't look
too odd,
hipo-utils -test
is happy, but
seems it has no
schema somehow,
and I confirmed
its checksum
matches the file
on tape. Haven't
seen this set of
conditions before.<br>
<br>
-Nathan</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div></blockquote></div><br></div></body></html>