[clas12_rgk] Please help! - wd: Fwd: [EXTERNAL] weird files

Wed Oct 11 10:29:10 EDT 2023

Dear Gagik,

thank you for your prompt reply.

Do I understand correctly that we need to  run the following command:

$PROJECT/distribution/coatjava/coatjava/bin/hipo-utils -update -d 
$PROJECT/distribution/coatjava/etc/bankdefs/hipo4 -o 
clas_005886.evio.dict.hipo clas_005886.evio.00325-00329.hipo

on the decoded files that are problematic to restore the dictionaries to 
allow the workflow to be completed?

All the best

Annalisa

Il 11/10/23 14:33, Gagik Gavalian ha scritto:
> Hi Annalisa,
>
> I was made aware of this issue, I looked through the files that are 
> causing problems and discovered that the files are missing the 
> dictionary, that’s why the denoising is failing.
> However, I could not find the cause of this, my best guess is that 
> there was a failure to read the disk during decoding that did not fill 
> the dictionaries into the file.
>
> I recovered the file by simply running an update on the decoded file, 
> which re-writes the dictionary. Then everything works.
> Maybe this step can be added to the workflow, I think it is in the 
> workflow, but if I’m not mistaken (Nathan can comment on this) it is 
> after the decoder is run,
> If the update dictionary code is run before the decoder, everything 
> should work in principle.
>
> Nathan, can you comment on when do you run the hypo-utils -update?
>
>
> — Gagik Gavalian
> — Staff Scientist
> — Jefferson Lab
> — email: gavalian at jlab.org
>
>> On Oct 11, 2023, at 8:08 AM, Annalisa D'Angelo 
>> <annalisa.dangelo at roma2.infn.it> wrote:
>>
>> Dear Gagik,
>>
>> thank you for all the work you are carrying on for the CLAS 
>> collaboration and for RG-K in particular.
>>
>> I understand that there are about 10 files, which are being processed 
>> for pass2, whose reconstruction keep failing because of some 
>> incompatibility of the de-noising code and the input file format.
>>
>> This problem has been holding us up for several weeks already. Can 
>> you please help us out at your earliest convenience? it is becoming 
>> urgent...
>>
>> Thank you in advance
>>
>> Annalisa
>>
>> for RG-K
>>
>>
>>
>> -------- Messaggio Inoltrato --------
>> Oggetto: 	Fwd: [EXTERNAL] weird files
>> Data: 	Wed, 11 Oct 2023 12:44:29 +0200
>> Mittente: 	Raffaella De Vita <Raffaella.Devita at ge.infn.it>
>> A: 	Annalisa D'Angelo <annalisa at roma2.infn.it>
>>
>>
>>
>>
>>
>>> Begin forwarded message:
>>>
>>> *From: *Gagik Gavalian <gavalian at jlab.org>
>>> *Subject: **Re: [EXTERNAL] weird files*
>>> *Date: *5 October 2023 at 19:14:47 CEST
>>> *To: *Raffaella De Vita <Raffaella.Devita at ge.infn.it>
>>> *Cc: *Nathan Baltzell <baltzell at jlab.org>, Lucilla Lanza 
>>> <lucilla.lanza at roma2.infn.it>
>>>
>>> I am looking into it, in principle, this should never happen. I’ll 
>>> figure it out, (hopefully)
>>>
>>> — Gagik Gavalian
>>> — Staff Scientist
>>> — Jefferson Lab
>>> — email: gavalian at jlab.org <mailto:gavalian at jlab.org>
>>>
>>>> On Oct 5, 2023, at 1:10 PM, Raffaella De Vita 
>>>> <Raffaella.Devita at ge.infn.it> wrote:
>>>>
>>>> So the decoded file has no schema?? How can that happen?
>>>>
>>>>> On 5 Oct 2023, at 17:59, Gagik Gavalian <gavalian at jlab.org> wrote:
>>>>>
>>>>> Hi Guys,
>>>>>
>>>>> When I run decoder on this file, it doe not crash, but rather 
>>>>> gracefully exits with a message:
>>>>> -- created denoiser with 8 threads
>>>>>
>>>>> hipo::dictionary (ERROR) schema {DC::tdc} does not exist... exiting
>>>>>
>>>>> hipo::dictionary (ERROR) schema {DC::tdc} does not exist... exiting
>>>>>
>>>>>
>>>>> However, if I run the -update option first:
>>>>>
>>>>> $PROJECT/distribution/coatjava/coatjava/bin/hipo-utils -update -d 
>>>>> $PROJECT/distribution/coatjava/etc/bankdefs/hipo4 -o 
>>>>> clas_005886.evio.dict.hipo clas_005886.evio.00325-00329.hipo
>>>>>
>>>>> Then the new file with the dictionary runs through de-noiser 
>>>>> without any problems.
>>>>>
>>>>>
>>>>> — Gagik Gavalian
>>>>> — Staff Scientist
>>>>> — Jefferson Lab
>>>>> — email: gavalian at jlab.org <mailto:gavalian at jlab.org>
>>>>>
>>>>>> On Oct 5, 2023, at 11:41 AM, Raffaella De Vita 
>>>>>> <Raffaella.Devita at ge.infn.it> wrote:
>>>>>>
>>>>>> Hi Gagik,
>>>>>>
>>>>>> Here is the message about the RG-K problematic files.
>>>>>>
>>>>>> Raffaella
>>>>>>
>>>>>>> Begin forwarded message:
>>>>>>>
>>>>>>> *From: *Nathan Baltzell <baltzell at jlab.org>
>>>>>>> *Subject: **weird files*
>>>>>>> *Date: *September 22, 2023 at 12:33:27 PM EDT
>>>>>>> *To: *Gagik Gavalian <gavalian at jlab.org>
>>>>>>> *Cc: *Lucilla Lanza <lanza at jlab.org>
>>>>>>>
>>>>>>> Hi Gagik,
>>>>>>>
>>>>>>> Do you see anything interesting about this file?
>>>>>>>
>>>>>>> /cache/clas12/rg-k/production/decoded/6b.2.0/005886/clas_005886.evio.00325-00329.hipo
>>>>>>>
>>>>>>> Size doesn't look too odd, hipo-utils -test is happy, but seems 
>>>>>>> it has no schema somehow, and I confirmed its checksum matches 
>>>>>>> the file on tape.  Haven't seen this set of conditions before.
>>>>>>>
>>>>>>> -Nathan
>>>>>>
>>>>>
>>>>
>>>
>>
>
-- 
================================================
Prof. Annalisa D'Angelo
Dip. Fisica, Universita' di Roma "Tor Vergata"
Istituto Nazionale di Fisica Nucleare
Sezione di Roma Tor Vergata, Rome Italy
email:annalisa.dangelo at roma2.infn.it  
Jefferson Laboratory, Newport News, VA USA
Email:annalisa at jlab.org
Tel: + 39 06 72594562
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_rgk/attachments/20231011/8c03f704/attachment-0001.html>