From wmoore at jlab.org Thu Nov 13 14:15:31 2025 From: wmoore at jlab.org (Wesley Moore) Date: Thu, 13 Nov 2025 19:15:31 +0000 Subject: [Jlab-scicomp-briefs] =?windows-1252?q?Interactive_Node_Usage_Re?= =?windows-1252?q?minder_=97_ifarm?= Message-ID: Dear Users, The interactive node ifarm2402 has required several recent reboots due to resource overload. Please note that interactive nodes are intended for interactive work and code development only. Do not run long-running or CPU/memory-intensive jobs on these nodes. For all compute-heavy workflows, please use the batch system instead. Stricter system-level limits will be applied during the next maintenance window (Tues, Nov 18th) to help prevent future disruptions. Thank you for your cooperation. Best regards, Scientific Computing Operations Team -------------- next part -------------- An HTML attachment was scrubbed... URL: From lsh at jlab.org Mon Nov 17 11:00:59 2025 From: lsh at jlab.org (Laura Hild) Date: Mon, 17 Nov 2025 16:00:59 +0000 Subject: [Jlab-scicomp-briefs] Farm AL9.6 to 9.7 tomorrow Message-ID: Happy Monday, everyone- We regularly patch the Farm on the CST Division's monthly maintenance day, the third Tuesday. We do not ordinarily announce the patches because they are within a single minor release of the operating system. This time, tomorrow, the 18th, the updates will cross a minor release, taking us from AlmaLinux 9.6 to AlmaLinux 9.7. A summary of changes can be found at https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/9.7_release_notes/overview#overview-major-changes and the upstream operating system vendor's general compatibility guide can be found at https://access.redhat.com/articles/rhel9-abi-compatibility I do not expect the update to require you to take any action, but it is more likely with the minor-version update than with the regular patches, so I am forewarning you. Please file a ServiceNow Incident at https://jlab.servicenowservices.com/scicomp with any questions or concerns. Thank you! -Laura From brads at jlab.org Mon Nov 17 18:02:15 2025 From: brads at jlab.org (Brad Sawatzky) Date: Mon, 17 Nov 2025 18:02:15 -0500 Subject: [Jlab-scicomp-briefs] SciComp Farm Maintenance day notices for Tues 11/18 Message-ID: Dear SciComp Farm Users, In addition to the minor Alma release bump announced earlier today [1], we want to highlight a couple of additional maintenance day items for Tuesday, Nov 18: - The interactive ifarm nodes will rebooted. - Python3.12 and Python3.13 packages will be installed on all Farm systems. Note that these packages do /not/ override the RHEL default python3 version, but may be invoked by calling the version you need: /usr/bin/python3.12 /usr/bin/python3.13 -- Brad [1] https://mailman.jlab.org/pipermail/jlab-scicomp-briefs/2025q4/000986.html From brads at jlab.org Thu Nov 20 17:31:58 2025 From: brads at jlab.org (Brad Sawatzky) Date: Thu, 20 Nov 2025 17:31:58 -0500 Subject: [Jlab-scicomp-briefs] Tape system downtime -- awaiting repairs Message-ID: Dear SciComp Users, We wanted to let you know that the Tape Library is currently experiencing mechanical issues and requires service. The support vendor has been on-site and we are awaiting parts. Given the proximity to the weekend, there is a good chance that repairs will not be completed until early next week. In the interim tape operations may be slowed or halted. This will, of course, have an impact on jobs that have tape dependencies. Jobs with files in /cache already will continue, and files submitted to tape will continue to spool for the time being. If possible, please defer 'manual' tape puts until next week to stretch our buffer-time. We will post to this list when the system is fully back online. Stay tuned, -- Brad From brads at jlab.org Tue Nov 25 08:42:48 2025 From: brads at jlab.org (Brad Sawatzky) Date: Tue, 25 Nov 2025 08:42:48 -0500 Subject: [Jlab-scicomp-briefs] Tape system downtime -- repairs continue Message-ID: Dear SciComp Users, Partial repairs to the Tape Library system have been completed and it is back in intermittent operation. Two technicians have been on-site and working the problem as of Monday. Some of the most significant repairs have been completed but they have identified additional parts that need replacement. Those parts are inbound. This will, of course, continue to have an impact on jobs that have tape dependencies. Jobs with files in /cache already will continue, and files submitted to tape will spool until the library can get to them. As before, where possible, please defer large 'manual' tape puts until the system fully back online to stretch our buffer time. We will post to this list when that is the case. Stay tuned, -- Brad -- Brad Sawatzky (he/him), PhD -<>- -<>- Ph: 757-269-5947 Nuclear Physics IT Lead -<>- Jefferson Lab/SciComp/F272 The most exciting phrase to hear in science, the one that heralds new discoveries, is not "Eureka!" but "That's funny..." -- Isaac Asimov From brads at jlab.org Tue Nov 25 08:54:52 2025 From: brads at jlab.org (Brad Sawatzky) Date: Tue, 25 Nov 2025 08:54:52 -0500 Subject: [Jlab-scicomp-briefs] Please report blocked/degraded web access to helpdesk Message-ID: Dear SciComp Users, You may have noticed issues access JLab resources from offsite, particularly web-based systems. Most of those issues have been driven by a marked increase in bot traffic of various types resulting in degraded access and some denial of service (DoS) situations. The CNI team wants to get the word out and will be posting the following in the Jlab Weekly, but we thought this would be a good venue as well. Please note the last paragraph below and let the team know of issues so the mitigation systems can be tuned. --- Jefferson Lab has been moving externally accessible websites behind the commercial product Cloudflare for the past several months. Cloudflare counters the increasing amount of bot and AI scraping automation attempting to connect to these websites. For example, from Nov. 17-24, there were 2.91 million accesses to our external websites with 603,000 identified as bot or AI website scrapping automation. The number of valid accesses over that period was roughly 1 million with another 1 million served by Cloudflare using cached data. This has reduced the load on the lab?s external web servers, and CST continues to tweak these configurations to reduce the bot activity as much as possible. As these adjustments take place, some legitimate accesses may be blocked or generate a ?prove you are a human? challenge. If you are legitimately accessing lab websites from offsite or the ?JLab Guest? network and get blocked or challenged, please email the Help Desk at helpdesk at jlab.org with the full URL you were accessing, any ?RayID? displayed on the block page, and your IP address. This information will help CST identify the issue and add any needed exceptions. --- -- Brad (on behalf of the CNI team) -- Brad Sawatzky (he/him), PhD -<>- -<>- Ph: 757-269-5947 Nuclear Physics IT Lead -<>- Jefferson Lab/SciComp/F272 The most exciting phrase to hear in science, the one that heralds new discoveries, is not "Eureka!" but "That's funny..." -- Isaac Asimov From brads at jlab.org Wed Nov 26 08:49:41 2025 From: brads at jlab.org (Brad Sawatzky) Date: Wed, 26 Nov 2025 08:49:41 -0500 Subject: [Jlab-scicomp-briefs] Tape system back in operation Message-ID: Dear SciComp Users, We're happy to report that the tape system is back in full production. Please give the system a little time to 'settle in' after the disruption, but do let us know if you see any longer term issues. Enjoy your regularly scheduled turkey. -- Brad