From ungaro at jlab.org Mon Nov 3 10:13:46 2025 From: ungaro at jlab.org (Maurizio Ungaro) Date: Mon, 3 Nov 2025 15:13:46 +0000 Subject: [Clas12_software] OSG Jobs Output Message-ID: <56AE997D-4E2C-4CE8-8B0C-A9D5A85A6DD6@jlab.org> Dear all, We are experiencing a pelican failure and there no output coming back from the OSG Jobs. It?s under investigation and it may be solved soon, but lots of data has been lost already. For now, newly submitted jobs will revert to the original file transfer method. If you need output from the current running jobs, let me know the ID and I can resubmit. Or please resubmit them yourself. Regards, Mauri From ungaro at jlab.org Mon Nov 3 10:32:53 2025 From: ungaro at jlab.org (Maurizio Ungaro) Date: Mon, 3 Nov 2025 15:32:53 +0000 Subject: [Clas12_software] OSG Jobs Output In-Reply-To: <56AE997D-4E2C-4CE8-8B0C-A9D5A85A6DD6@jlab.org> References: <56AE997D-4E2C-4CE8-8B0C-A9D5A85A6DD6@jlab.org> Message-ID: The issue has been fixed. The current running jobs are correctly writing jobs to /volatile. Please resubmit jobs in case you need to make up for the loss of data. Regards, Mauri > On Nov 3, 2025, at 10:13?AM, Maurizio Ungaro via Clas12_software wrote: > > Dear all, > > We are experiencing a pelican failure and there no output coming back from the OSG Jobs. > It?s under investigation and it may be solved soon, but lots of data has been lost already. > > For now, newly submitted jobs will revert to the original file transfer method. > > If you need output from the current running jobs, let me know the ID and I can resubmit. Or please resubmit them yourself. > > Regards, > Mauri > > > > _______________________________________________ > Clas12_software mailing list > Clas12_software at jlab.org > https://mailman.jlab.org/mailman/listinfo/clas12_software From devita at jlab.org Mon Nov 3 19:14:03 2025 From: devita at jlab.org (Raffaella De Vita) Date: Tue, 4 Nov 2025 00:14:03 +0000 Subject: [Clas12_software] CLAS12 software&computing office hours Message-ID: ?Dear All, This is just a reminder that CLAS12 software&computing office hours are tomorrow, November 4, at 9:30 am EST. Collaborators can join the zoom meeting https://jlab-org.zoomgov.com/j/1605511359?pwd=TTVRUW16UklWcXpuS1N0Y0I1Unk1UT09 Best regards, Raffaella -------------- next part -------------- An HTML attachment was scrubbed... URL: From ungaro at jlab.org Tue Nov 4 14:22:19 2025 From: ungaro at jlab.org (Maurizio Ungaro) Date: Tue, 4 Nov 2025 19:22:19 +0000 Subject: [Clas12_software] OSG submissions Message-ID: <549331ED-4F76-4E31-ADF1-DB34D5C8D936@jlab.org> Hi all, There are still some failures when using the pelican transfer protocol. Looks like related to the security tokens. As a result, the jobs failed to submit and disappeared from the portal (after initially being listed as ?Not Submitted?) Sorry for the inconvenience. The token generation frequency is increased and that should help. Please resubmit the failed jobs and let me know if/when this happens. Regards, Mauri From baltzell at jlab.org Wed Nov 5 18:07:39 2025 From: baltzell at jlab.org (Nathan Baltzell) Date: Wed, 5 Nov 2025 23:07:39 +0000 Subject: [Clas12_software] clas12 software meeting tomorrow Message-ID: Dear Everyone, We'll have a software meeting tomorrow at 11:00 in L210A. The AI version (1st Thursday of the month) will resume in December. Here's the links: https://clasweb.jlab.org/wiki/index.php/CLAS12_Software_Center https://jlab-org.zoomgov.com/j/1614901536?pwd=fUSamaIgF6cLkpsdmlWp66T2mvjQm8.1 Regards, Nathan From devita at jlab.org Mon Nov 10 19:22:08 2025 From: devita at jlab.org (Raffaella De Vita) Date: Tue, 11 Nov 2025 00:22:08 +0000 Subject: [Clas12_software] CLAS12 software&computing office hours Message-ID: <84EB71FC-2E2B-4B4B-A5CB-DC1E4F7FA3D8@jlab.org> ?Dear All, This is just a reminder that CLAS12 software&computing office hours are tomorrow, November 11, at 9:30 am EST. Collaborators can join the zoom meeting https://jlab-org.zoomgov.com/j/1605511359?pwd=TTVRUW16UklWcXpuS1N0Y0I1Unk1UT09 Best regards, Raffaella -------------- next part -------------- An HTML attachment was scrubbed... URL: From baltzell at jlab.org Wed Nov 12 22:17:03 2025 From: baltzell at jlab.org (Nathan Baltzell) Date: Thu, 13 Nov 2025 03:17:03 +0000 Subject: [Clas12_software] clas12 software meeting tomorrow at 11:00 in L210 Message-ID: Dear Everyone, We'll meet tomorrow at 11:00 in L210. Here's the wiki agenda and zoom links: https://clasweb.jlab.org/wiki/index.php/CLAS12_Software_Center https://jlab-org.zoomgov.com/j/1614901536?pwd=fUSamaIgF6cLkpsdmlWp66T2mvjQm8.1 Regards, Nathan From ungaro at jlab.org Wed Nov 19 09:40:49 2025 From: ungaro at jlab.org (Maurizio Ungaro) Date: Wed, 19 Nov 2025 14:40:49 +0000 Subject: [Clas12_software] OSG Jobs with background merging are failing Message-ID: Looks like fetching the background files from the xrootd server is not working at the moment. This causes all jobs that requests background merging to fail and go to HOLD. I submitted a ticket about this. Regards, Mauri From ungaro at jlab.org Wed Nov 19 17:15:59 2025 From: ungaro at jlab.org (Maurizio Ungaro) Date: Wed, 19 Nov 2025 22:15:59 +0000 Subject: [Clas12_software] OSG Jobs with background merging are failing In-Reply-To: References: Message-ID: <249B3836-5EDA-4BD5-80DA-CD06D25F3C02@jlab.org> Dear all: It turns out that after Tuesday's updates the xrootd interface is no longer supported at JLab That?s bad news many current OSG jobs. We?ll work out a solution hopefully as soon as by tomorrow so jobs can be resumed. In the meantime, the OSG jobs requesting background merging have to be removed from the OSG. Once the situation is resolved, I will place them first in line to be re-submitted. Mauri > On Nov 19, 2025, at 9:40?AM, Maurizio Ungaro wrote: > > Looks like fetching the background files from the xrootd server is not working at the moment. > This causes all jobs that requests background merging to fail and go to HOLD. > I submitted a ticket about this. > > Regards, > Mauri From ungaro at jlab.org Fri Nov 21 08:04:27 2025 From: ungaro at jlab.org (Maurizio Ungaro) Date: Fri, 21 Nov 2025 13:04:27 +0000 Subject: [Clas12_software] OSG new submissions are suspended Message-ID: <79333A09-02D3-4721-B6E7-573FCFD9EBEB@jlab.org> Dear all, There are a few issues piling up: 1. Pelican file transfers: The new file transfer system (pelican) is acting up. This affects files to and back from OSG, including LUND submissions. 2. xrootd deprecation at JLab: We learned that JLab?s IT decommissioned our xrootd support last maintenance day. This impacts all jobs that rely on background merging. These jobs are being are skipped for the moment, to be resumed when we move to the alternative (pelican) The path forward is: a. Temporarely suspend new submissions while these issues are worked out b. Continue monitoring of the successful / failed jobs in the current queue c. Clean up and resubmit failed jobs once the problems are resolved. If you had submissions that did not go through properly, please keep a list but hold off on sending it to me for now, as it may duplicate what I?m tracking. May the force be with us, Mauri Jobs currently skipped because of bg merging: 9972 9973 9982 9983 9984 9987 9988 9989 10006 10007 10008 10009 10010 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ungaro at jlab.org Mon Nov 24 14:59:01 2025 From: ungaro at jlab.org (Maurizio Ungaro) Date: Mon, 24 Nov 2025 19:59:01 +0000 Subject: [Clas12_software] OSG Portal now available - update Message-ID: <50E276C4-5FA3-4F95-A397-ED197B2CC2F0@jlab.org> Dear all, We are now successfully using pelican to transfer to the OSG node: - LUND files from the user requested location - background files when requested by the job So the portal is back and available for new submissions, and I resubmitted the jobs listed below. You should see them back on the portal soon. Please let me know if I missed any. HOWEVER: Unfortunately a few issues are affecting the jobs, so that they need to be babysit until their completion. So you may see slow downs and jobs back and forth from HOLD status. The issues, under investigation, are: - pelican still shaky on both LUND and background files - possible CVMFS issues Regards, Mauri valerii 9972 anya 9973 avakian 9977 elouan 9982 elouan 9983 elouan 9984 marianat 9987 marianat 9988 marianat 9989 jsalvg 9994 storyf 10006 storyf 10007 storyf 10008 storyf 10009 storyf 10010 elouan 10013 elouan 10014 elouan 10015 elouan 10016 mronay 10017 jsalvg 10018