[Hps] [Jlab-scicomp-briefs] Upcoming Change: Enforcement of /scratch Disk Requests on the JLab Compute Farm

[Wesley Moore via Jlab-scicomp-briefs]

Dear Users,
Feb 17th: If your job uses /scratch, you must request disk space explicitly using gres. Jobs that do not request disk (or request too little) may fail after this change.
Example (Slurm):

#SBATCH --gres=disk:10G

This requests 10 GB of /scratch space for your job.
________________________________
Since at least 2019, the JLab Compute Farm batch system has supported requesting a generic resource (gres), disk, to represent the amount of space a job expects to use on a compute node’s /scratch filesystem. For example, if four jobs each requested 100 GB, the scheduler would avoid placing all four on a node with only 315 GB of /scratch space available.
Until now, however, these disk requests were advisory only. Unlike CPU and memory limits, jobs were not prevented from using more /scratch space than they requested.
What’s changing
On the February 17th maintenance day, disk requests will be strictly enforced. Jobs will be limited to the amount of /scratch space they request. This change is intended to improve overall Farm reliability by preventing /scratch filesystems from filling up and causing ENOSPC (“no space left on device”) errors for other jobs.
As a result, jobs that do not request disk space—or request too little—may begin to fail if they previously relied on unused /scratch space being available.
What you may need to do

  *
Swif users:
Swif automatically calculates a disk request based on the size of input files. If your jobs use additional temporary space beyond the inputs and begin failing, you may need to request extra space using the -disk-scratch option.
  *
Slurm users:
By default, Slurm jobs receive no /scratch allocation. If your job runs entirely in memory and shared filesystems, this may be fine. However, many applications implicitly use temporary scratch space. If you encounter ENOSPC errors, you will need to request disk explicitly, for example:

#SBATCH --gres=disk:1G

If your job requires /scratch space, declaring it will now be mandatory.

Please review your workflows ahead of time to ensure appropriate disk requests are being made.
If you have questions or run into issues after the change, feel free to reach out.
On behalf of Scientific Computing Operations,
Wesley


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/hps/attachments/20260203/eaec9be2/attachment.htm>
-------------- next part --------------
--

This is an announcement-only list for Jefferson Lab Scientific Computing Updates .

Subscription and List Archive: https://mailman.jlab.org/mailman/listinfo/jlab-scicomp-briefs

For help: https://jlab.servicenowservices.com/scicomp