[clas12_rgk] Please help! - wd: Fwd: [EXTERNAL] weird files

Wed Oct 11 10:31:22 EDT 2023

Hi Annalisa,

That is correct, once you update the dictionary, everything should work fine.
I talked to Nathan, and we will come up with better solution for the long run. 
We can have the workflow detect this problems and run the update if needed.
But this has to be implemented.

For now, one has to do it manually.

— Gagik Gavalian
— Staff Scientist
— Jefferson Lab
— email: gavalian at jlab.org

> On Oct 11, 2023, at 10:29 AM, Annalisa D'Angelo <annalisa.dangelo at roma2.infn.it> wrote:
> 
> Dear Gagik, 
> 
> thank you for your prompt reply. 
> 
> Do I understand correctly that we need to  run the following command:
> 
> $PROJECT/distribution/coatjava/coatjava/bin/hipo-utils -update -d $PROJECT/distribution/coatjava/etc/bankdefs/hipo4 -o clas_005886.evio.dict.hipo clas_005886.evio.00325-00329.hipo
> 
> on the decoded files that are problematic to restore the dictionaries to allow the workflow to be completed?
> 
> All the best
> 
> Annalisa
> 
> Il 11/10/23 14:33, Gagik Gavalian ha scritto:
>> Hi Annalisa,
>> 
>> I was made aware of this issue, I looked through the files that are causing problems and discovered that the files are missing the dictionary, that’s why the denoising is failing.
>> However, I could not find the cause of this, my best guess is that there was a failure to read the disk during decoding that did not fill the dictionaries into the file.
>> 
>> I recovered the file by simply running an update on the decoded file, which re-writes the dictionary. Then everything works.
>> Maybe this step can be added to the workflow, I think it is in the workflow, but if I’m not mistaken (Nathan can comment on this) it is after the decoder is run,
>> If the update dictionary code is run before the decoder, everything should work in principle. 
>> 
>> Nathan, can you comment on when do you run the hypo-utils -update?
>> 
>> 
>> — Gagik Gavalian
>> — Staff Scientist
>> — Jefferson Lab
>> — email: gavalian at jlab.org <mailto:gavalian at jlab.org>
>> 
>>> On Oct 11, 2023, at 8:08 AM, Annalisa D'Angelo <annalisa.dangelo at roma2.infn.it> <mailto:annalisa.dangelo at roma2.infn.it> wrote:
>>> 
>>> Dear Gagik,
>>> 
>>> thank you for all the work you are carrying on for the CLAS collaboration and for RG-K in particular.
>>> 
>>> I understand that there are about 10 files, which are being processed for pass2, whose reconstruction keep failing because of some incompatibility of the de-noising code and the input file format.
>>> 
>>> This problem has been holding us up for several weeks already. Can you please help us out at your earliest convenience? it is becoming urgent...
>>> 
>>> Thank you in advance 
>>> 
>>> Annalisa
>>> 
>>> for RG-K
>>> 
>>> 
>>> 
>>> -------- Messaggio Inoltrato --------
>>> Oggetto:	Fwd: [EXTERNAL] weird files
>>> Data:	Wed, 11 Oct 2023 12:44:29 +0200
>>> Mittente:	Raffaella De Vita <Raffaella.Devita at ge.infn.it> <mailto:Raffaella.Devita at ge.infn.it>
>>> A:	Annalisa D'Angelo <annalisa at roma2.infn.it> <mailto:annalisa at roma2.infn.it>
>>> 
>>> 
>>> 
>>>> Begin forwarded message:
>>>> 
>>>> From: Gagik Gavalian <gavalian at jlab.org <mailto:gavalian at jlab.org>>
>>>> Subject: Re: [EXTERNAL] weird files
>>>> Date: 5 October 2023 at 19:14:47 CEST
>>>> To: Raffaella De Vita <Raffaella.Devita at ge.infn.it <mailto:Raffaella.Devita at ge.infn.it>>
>>>> Cc: Nathan Baltzell <baltzell at jlab.org <mailto:baltzell at jlab.org>>, Lucilla Lanza <lucilla.lanza at roma2.infn.it <mailto:lucilla.lanza at roma2.infn.it>>
>>>> 
>>>> I am looking into it, in principle, this should never happen. I’ll figure it out, (hopefully)
>>>> 
>>>> — Gagik Gavalian
>>>> — Staff Scientist
>>>> — Jefferson Lab
>>>> — email: gavalian at jlab.org <mailto:gavalian at jlab.org>
>>>> 
>>>>> On Oct 5, 2023, at 1:10 PM, Raffaella De Vita <Raffaella.Devita at ge.infn.it <mailto:Raffaella.Devita at ge.infn.it>> wrote:
>>>>> 
>>>>> So the decoded file has no schema?? How can that happen?
>>>>> 
>>>>>> On 5 Oct 2023, at 17:59, Gagik Gavalian <gavalian at jlab.org <mailto:gavalian at jlab.org>> wrote:
>>>>>> 
>>>>>> Hi Guys,
>>>>>> 
>>>>>> When I run decoder on this file, it doe not crash, but rather gracefully exits with a message:
>>>>>> -- created denoiser with 8 threads
>>>>>> 
>>>>>> hipo::dictionary (ERROR) schema {DC::tdc} does not exist... exiting
>>>>>> 
>>>>>> hipo::dictionary (ERROR) schema {DC::tdc} does not exist... exiting
>>>>>> 
>>>>>> 
>>>>>> However, if I run the -update option first:
>>>>>> 
>>>>>> $PROJECT/distribution/coatjava/coatjava/bin/hipo-utils -update -d $PROJECT/distribution/coatjava/etc/bankdefs/hipo4 -o clas_005886.evio.dict.hipo clas_005886.evio.00325-00329.hipo
>>>>>> 
>>>>>> Then the new file with the dictionary runs through de-noiser without any problems.
>>>>>> 
>>>>>> 
>>>>>> — Gagik Gavalian
>>>>>> — Staff Scientist
>>>>>> — Jefferson Lab
>>>>>> — email: gavalian at jlab.org <mailto:gavalian at jlab.org>
>>>>>> 
>>>>>>> On Oct 5, 2023, at 11:41 AM, Raffaella De Vita <Raffaella.Devita at ge.infn.it <mailto:Raffaella.Devita at ge.infn.it>> wrote:
>>>>>>> 
>>>>>>> Hi Gagik,
>>>>>>> 
>>>>>>> Here is the message about the RG-K problematic files.
>>>>>>> 
>>>>>>> Raffaella
>>>>>>> 
>>>>>>>> Begin forwarded message:
>>>>>>>> 
>>>>>>>> From: Nathan Baltzell <baltzell at jlab.org <mailto:baltzell at jlab.org>>
>>>>>>>> Subject: weird files
>>>>>>>> Date: September 22, 2023 at 12:33:27 PM EDT
>>>>>>>> To: Gagik Gavalian <gavalian at jlab.org <mailto:gavalian at jlab.org>>
>>>>>>>> Cc: Lucilla Lanza <lanza at jlab.org <mailto:lanza at jlab.org>>
>>>>>>>> 
>>>>>>>> Hi Gagik,
>>>>>>>> 
>>>>>>>> Do you see anything interesting about this file?
>>>>>>>> 
>>>>>>>> /cache/clas12/rg-k/production/decoded/6b.2.0/005886/clas_005886.evio.00325-00329.hipo
>>>>>>>> 
>>>>>>>> Size doesn't look too odd, hipo-utils -test is happy, but seems it has no schema somehow, and I confirmed its checksum matches the file on tape.  Haven't seen this set of conditions before.
>>>>>>>> 
>>>>>>>> -Nathan
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> -- 
> ================================================
> Prof. Annalisa D'Angelo
> Dip. Fisica, Universita' di Roma "Tor Vergata"
> Istituto Nazionale di Fisica Nucleare
> Sezione di Roma Tor Vergata, Rome Italy
> email:annalisa.dangelo at roma2.infn.it <mailto:email:annalisa.dangelo at roma2.infn.it> 
> Jefferson Laboratory, Newport News, VA USA
> Email: annalisa at jlab.org <mailto:annalisa at jlab.org>
> Tel: + 39 06 72594562

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_rgk/attachments/20231011/d1bd85e4/attachment-0001.html>