<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Annalisa,<div><br></div><div>That is correct, once you update the dictionary, everything should work fine.</div><div>I talked to Nathan, and we will come up with better solution for the long run. </div><div>We can have the workflow detect this problems and run the update if needed.</div><div>But this has to be implemented.</div><div><br></div><div>For now, one has to do it manually.</div><div><br><div>
<meta charset="UTF-8"><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>— Gagik Gavalian<br>— Staff Scientist<br>— Jefferson Lab<br>— email: gavalian@jlab.org</div></div>
</div>
<div><br><blockquote type="cite"><div>On Oct 11, 2023, at 10:29 AM, Annalisa D'Angelo <annalisa.dangelo@roma2.infn.it> wrote:</div><br class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div><p>Dear Gagik, <br>
</p><p>thank you for your prompt reply. </p><p>Do I understand correctly that we need to run the following
command:</p><p><span style="font-variant-ligatures: no-common-ligatures">$PROJECT/distribution/coatjava/coatjava/bin/hipo-utils
-update -d $PROJECT/distribution/coatjava/etc/bankdefs/hipo4 -o
clas_005886.evio.dict.hipo
clas_005886.evio.00325-00329.hipo</span></p><p>on the decoded files that are problematic to restore the
dictionaries to allow the workflow to be completed?</p><p>All the best</p><p>Annalisa<br>
</p>
<div class="moz-cite-prefix">Il 11/10/23 14:33, Gagik Gavalian ha
scritto:<br>
</div>
<blockquote type="cite" cite="mid:800CA002-D720-46C5-9D38-54B0D6128790@jlab.org">
Hi Annalisa,
<div><br>
</div>
<div>I was made aware of this issue, I looked through the files
that are causing problems and discovered that the files are
missing the dictionary, that’s why the denoising is failing.</div>
<div>However, I could not find the cause of this, my best guess is
that there was a failure to read the disk during decoding that
did not fill the dictionaries into the file.</div>
<div><br>
</div>
<div>I recovered the file by simply running an update on the
decoded file, which re-writes the dictionary. Then everything
works.</div>
<div>Maybe this step can be added to the workflow, I think it is
in the workflow, but if I’m not mistaken (Nathan can comment on
this) it is after the decoder is run,</div>
<div>If the update dictionary code is run before the decoder,
everything should work in principle. </div>
<div><br>
</div>
<div>Nathan, can you comment on when do you run the hypo-utils
-update?</div>
<div><br>
</div>
<div><br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>— Gagik Gavalian<br>
— Staff Scientist<br>
— Jefferson Lab<br>
— email: <a class="moz-txt-link-abbreviated" href="mailto:gavalian@jlab.org">gavalian@jlab.org</a></div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Oct 11, 2023, at 8:08 AM, Annalisa D'Angelo
<a class="moz-txt-link-rfc2396E" href="mailto:annalisa.dangelo@roma2.infn.it"><annalisa.dangelo@roma2.infn.it></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><p>Dear Gagik,</p><p>thank you for all the work you are carrying on for
the CLAS collaboration and for RG-K in particular.</p><p>I understand that there are about 10 files, which are
being processed for pass2, whose reconstruction keep
failing because of some incompatibility of the
de-noising code and the input file format.</p><p>This problem has been holding us up for several weeks
already. Can you please help us out at your earliest
convenience? it is becoming urgent...</p><p>Thank you in advance <br>
</p><p>Annalisa</p><p>for RG-K<br>
</p>
<div class="moz-forward-container"><br>
<br>
-------- Messaggio Inoltrato --------
<table class="moz-email-headers-table" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Oggetto: </th>
<td>Fwd: [EXTERNAL] weird files</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Data: </th>
<td>Wed, 11 Oct 2023 12:44:29 +0200</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Mittente: </th>
<td>Raffaella De Vita <a class="moz-txt-link-rfc2396E" href="mailto:Raffaella.Devita@ge.infn.it" moz-do-not-send="true"><Raffaella.Devita@ge.infn.it></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">A: </th>
<td>Annalisa D'Angelo <a class="moz-txt-link-rfc2396E" href="mailto:annalisa@roma2.infn.it" moz-do-not-send="true"><annalisa@roma2.infn.it></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<br>
<div><br>
<blockquote type="cite">
<div>Begin forwarded message:</div>
<br class="Apple-interchange-newline">
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>From:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Gagik
Gavalian <<a href="mailto:gavalian@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">gavalian@jlab.org</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Subject:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;"><b>Re:
[EXTERNAL] weird files</b><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Date:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">5
October 2023 at 19:14:47 CEST<br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>To:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Raffaella
De Vita <<a href="mailto:Raffaella.Devita@ge.infn.it" class="moz-txt-link-freetext" moz-do-not-send="true">Raffaella.Devita@ge.infn.it</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;"><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Cc:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Nathan
Baltzell <<a href="mailto:baltzell@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">baltzell@jlab.org</a>>,
Lucilla Lanza <<a href="mailto:lucilla.lanza@roma2.infn.it" class="moz-txt-link-freetext" moz-do-not-send="true">lucilla.lanza@roma2.infn.it</a>><br>
</span></div>
<br>
<div>
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
I am looking into it, in principle, this
should never happen. I’ll figure it out,
(hopefully)
<div><br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>— Gagik Gavalian<br>
— Staff Scientist<br>
— Jefferson Lab<br>
— <a href="mailto:gavalian@jlab.org" moz-do-not-send="true">email:
gavalian@jlab.org</a></div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Oct 5, 2023, at 1:10 PM,
Raffaella De Vita <<a href="mailto:Raffaella.Devita@ge.infn.it" class="moz-txt-link-freetext" moz-do-not-send="true">Raffaella.Devita@ge.infn.it</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
So the decoded file has no schema??
How can that happen?<br>
<div><br>
<blockquote type="cite">
<div>On 5 Oct 2023, at 17:59,
Gagik Gavalian <<a href="mailto:gavalian@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">gavalian@jlab.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hi Guys,
<div><br>
</div>
<div>When I run decoder on
this file, it doe not
crash, but rather
gracefully exits with a
message:</div>
<div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0);">
<span style="font-variant-ligatures: no-common-ligatures">-- created denoiser
with 8 threads</span></div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0); min-height: 21px;">
<br>
<span style="font-variant-ligatures: no-common-ligatures"></span></div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0);">
<span style="font-variant-ligatures: no-common-ligatures">hipo::dictionary
(ERROR) schema
{DC::tdc} does not
exist... exiting</span></div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0); min-height: 21px;">
<br>
<span style="font-variant-ligatures: no-common-ligatures"></span></div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0); min-height: 21px;">
hipo::dictionary
(ERROR) schema
{DC::tdc} does not
exist... exiting<span style="font-variant-ligatures: no-common-ligatures"></span></div>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div><br>
</div>
<div><br>
</div>
<div>However, if I
run the -update
option first:</div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 18px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(248, 155, 18); background-color: rgb(0, 0, 0);">
<span style="font-variant-ligatures: no-common-ligatures">$PROJECT/distribution/coatjava/coatjava/bin/hipo-utils
-update -d
$PROJECT/distribution/coatjava/etc/bankdefs/hipo4
-o
clas_005886.evio.dict.hipo
clas_005886.evio.00325-00329.hipo</span></div>
</div>
<div><br>
</div>
<div>Then the new
file with the
dictionary runs
through de-noiser
without any
problems.</div>
<div><br>
</div>
<div><br>
</div>
<div>— Gagik
Gavalian<br>
— Staff Scientist<br>
— Jefferson Lab<br>
— <a href="mailto:gavalian@jlab.org" moz-do-not-send="true">email:
gavalian@jlab.org</a></div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Oct 5, 2023,
at 11:41 AM,
Raffaella De Vita
<<a href="mailto:Raffaella.Devita@ge.infn.it" class="moz-txt-link-freetext" moz-do-not-send="true">Raffaella.Devita@ge.infn.it</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div>Hi Gagik,</div>
<div><br>
</div>
<div>Here is
the message
about the RG-K
problematic
files.</div>
<div><br>
</div>
<div>Raffaella</div>
<div><br>
<blockquote type="cite">
<div>Begin
forwarded
message:</div>
<br class="Apple-interchange-newline">
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>From:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Nathan
Baltzell <<a href="mailto:baltzell@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">baltzell@jlab.org</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Subject:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;"><b>weird
files</b><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Date:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">September
22, 2023 at
12:33:27 PM
EDT<br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>To:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Gagik
Gavalian <<a href="mailto:gavalian@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">gavalian@jlab.org</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;"><b>Cc:
</b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;">Lucilla
Lanza <<a href="mailto:lanza@jlab.org" class="moz-txt-link-freetext" moz-do-not-send="true">lanza@jlab.org</a>><br>
</span></div>
<br>
<div>
<div>Hi Gagik,<br>
<br>
Do you see
anything
interesting
about this
file?<br>
<br>
/cache/clas12/rg-k/production/decoded/6b.2.0/005886/clas_005886.evio.00325-00329.hipo<br>
<br>
Size doesn't
look too odd,
hipo-utils
-test is
happy, but
seems it has
no schema
somehow, and I
confirmed its
checksum
matches the
file on tape.
Haven't seen
this set of
conditions
before.<br>
<br>
-Nathan</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
================================================
Prof. Annalisa D'Angelo
Dip. Fisica, Universita' di Roma "Tor Vergata"
Istituto Nazionale di Fisica Nucleare
Sezione di Roma Tor Vergata, Rome Italy
<a class="moz-txt-link-abbreviated" href="mailto:email:annalisa.dangelo@roma2.infn.it">email:annalisa.dangelo@roma2.infn.it</a>
Jefferson Laboratory, Newport News, VA USA
Email: <a class="moz-txt-link-abbreviated" href="mailto:annalisa@jlab.org">annalisa@jlab.org</a>
Tel: + 39 06 72594562
</pre>
</div>
</div></blockquote></div><br></div></body></html>