<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Dear Users,</div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Feb 17th: If your job uses <code>/scratch</code>, you <b>must</b> request disk space explicitly using
<code>gres</code>. Jobs that do not request disk (or request too little) may fail after this change.</div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<b>Example (Slurm):</b></div>
<pre class="elementToProof"><div style="white-space: pre-wrap; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof"><code>#SBATCH --gres=disk:10G</code></div></pre>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
This requests 10 GB of <code>/scratch</code> space for your job.</div>
<hr>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Since at least 2019, the JLab Compute Farm batch system has supported requesting a generic resource (gres),
<b>disk</b>, to represent the amount of space a job expects to use on a compute node’s
<code>/scratch</code> filesystem. For example, if four jobs each requested 100 GB, the scheduler would avoid placing all four on a node with only 315 GB of
<code>/scratch</code> space available.</div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Until now, however, these disk requests were <b>advisory only</b>. Unlike CPU and memory limits, jobs were not prevented from using more
<code>/scratch</code> space than they requested.</div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<b>What’s changing</b></div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
On the <b>February 17th maintenance day</b>, disk requests will be <b>strictly enforced</b>. Jobs will be limited to the amount of
<code>/scratch</code> space they request. This change is intended to improve overall Farm reliability by preventing
<code>/scratch</code> filesystems from filling up and causing ENOSPC (“no space left on device”) errors for other jobs.</div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
As a result, jobs that do not request disk space—or request too little—may begin to fail if they previously relied on unused
<code>/scratch</code> space being available.</div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<b>What you may need to do</b></div>
<ul data-spread="true">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 1em 0px;" role="presentation" class="elementToProof"><b>Swif users:</b><br>
Swif automatically calculates a disk request based on the size of input files. If your jobs use additional temporary space beyond the inputs and begin failing, you may need to request extra space using the
<code>-disk-scratch</code> option.</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 1em 0px;" role="presentation" class="elementToProof"><b>Slurm users:</b><br>
By default, Slurm jobs receive <b>no <code>/scratch</code> allocation</b>. If your job runs entirely in memory and shared filesystems, this may be fine. However, many applications implicitly use temporary scratch space. If you encounter ENOSPC errors, you will
need to request disk explicitly, for example:</div>
<pre role="presentation" class="elementToProof"><div style="white-space: pre-wrap; margin: 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif;" class="elementToProof"><code>#SBATCH --gres=disk:1G</code></div></pre>
<div style="margin: 1em 0px;" role="presentation" class="elementToProof">If your job requires
<code>/scratch</code> space, declaring it will now be mandatory.</div>
</li></ul>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Please review your workflows ahead of time to ensure appropriate disk requests are being made.</div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
If you have questions or run into issues after the change, feel free to reach out.</div>
<div style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;" class="elementToProof">
On behalf of Scientific Computing Operations,</div>
<div style="text-align: left; text-indent: 0px; background-color: rgb(255, 255, 255); margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: black;" class="elementToProof">
Wesley</div>
<div style="margin: 1em 0px; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
</body>
</html>