[Halld-online] [New Logentry] Missing tar files

[davidl at jlab.org]

Logentry Text:
--
DAQ/Trigger configuration files are packaged up into a single tar file for each run be copied to tape alongside the raw data for archiving purposes. It was noticed that several of these tar files fail to show up on the tape. A sampling survey of runs with >1M events showed roughly 30% of these are lost. Checking the logs indicates that the tar files are being created in the volatile directory and at least an attempt is made to create a hard link to them in the staging directory where they will be picked up by the jmigrate script and copied to tape. No error is recorded in making this link, but there are also no records indicating that the link is ever made in staging for those files that are missing (though the history is not complete).

The current belief is that the jmigrate script is deleting the empty directory from staging where the link to the tar file is about to be made by the stage_to_tape script so that the link fails. Sometimes the link is made first which accounts for those files that do get copied to tape. The two scripts involved are run via separate cron jobs every 10 minutes and so the timing of these operations will be roughly the same creating the race condition.

Adjustments are being made to the jmigrate script to address this issue for the future.

Tar files that still existed on both gluonraid1 and gluonraid2 but were not on tape as of this morning had appropriate directories and hard links created by so that they would be copied to tape. The list of these is:

gluonaid1
-----------------
Run010729/RunLog010729.tar
Run011312/RunLog011312.tar
Run011313/RunLog011313.tar
Run011314/RunLog011314.tar
Run011315/RunLog011315.tar
Run011320/RunLog011320.tar
Run011321/RunLog011321.tar
Run011322/RunLog011322.tar
Run011324/RunLog011324.tar
Run011325/RunLog011325.tar
Run011328/RunLog011328.tar
Run011329/RunLog011329.tar
Run011330/RunLog011330.tar
Run011331/RunLog011331.tar
Run011333/RunLog011333.tar
Run011335/RunLog011335.tar
Run011336/RunLog011336.tar
Run011337/RunLog011337.tar
Run011338/RunLog011338.tar
Run011339/RunLog011339.tar
Run011340/RunLog011340.tar
Run011341/RunLog011341.tar
Run011343/RunLog011343.tar
Run011345/RunLog011345.tar
Run011349/RunLog011349.tar
Run011350/RunLog011350.tar
Run011351/RunLog011351.tar
Run011352/RunLog011352.tar
Run011353/RunLog011353.tar
Run011355/RunLog011355.tar
Run011356/RunLog011356.tar
Run011357/RunLog011357.tar
Run011359/RunLog011359.tar
Run011631/RunLog011631.tar
Run011632/RunLog011632.tar
Run011633/RunLog011633.tar
Run011634/RunLog011634.tar
Run011635/RunLog011635.tar
Run011636/RunLog011636.tar
Run011637/RunLog011637.tar
Run011638/RunLog011638.tar
Run011639/RunLog011639.tar
Run011640/RunLog011640.tar
Run011641/RunLog011641.tar


gluonraid2
-----------------
Run000214/RunLog000214.tar
Run000215/RunLog000215.tar
Run000216/RunLog000216.tar
Run011253/RunLog011253.tar
Run011254/RunLog011254.tar
Run011255/RunLog011255.tar
Run011256/RunLog011256.tar
Run011258/RunLog011258.tar
Run011259/RunLog011259.tar
Run011368/RunLog011368.tar
Run011369/RunLog011369.tar
Run011370/RunLog011370.tar
Run011373/RunLog011373.tar
Run011374/RunLog011374.tar
Run011375/RunLog011375.tar
Run011377/RunLog011377.tar
Run011378/RunLog011378.tar
Run011379/RunLog011379.tar
Run011380/RunLog011380.tar
Run011381/RunLog011381.tar
Run011426/RunLog011426.tar
Run011427/RunLog011427.tar
Run011428/RunLog011428.tar
Run011556/RunLog011556.tar
Run011557/RunLog011557.tar
Run011558/RunLog011558.tar
Run011605/RunLog011605.tar
Run011606/RunLog011606.tar
Run011608/RunLog011608.tar
Run011610/RunLog011610.tar
Run011611/RunLog011611.tar
Run011612/RunLog011612.tar
Run011613/RunLog011613.tar
Run011614/RunLog011614.tar




---

This is a plain text email for clients that cannot display HTML.  The full logentry can be found online at https://logbooks.jlab.org/entry/3403062
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-online/attachments/20160426/a6108cb0/attachment.html>