<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Vardan,<br class="">
<br class="">
The memory footprint of CLAS12 reconstruction jobs is always evolving, e.g., due to software/algorithm changes and new detectors. And the more general systems at JLab react to physical memory, not virtual. But both of those have been pretty static for a few
years, with no signs of changing any time soon.<br class="">
<br class="">
The motivation of this email thread was rather to warn our collaborators that they may experience new issues if their resource requests on their particular jobs are deemed inappropriate, and try to provide some guidance on how to measure and address that.<br class="">
<br class="">
-Nathan<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Feb 8, 2023, at 7:59 AM, Vardan Gyurjyan <<a href="mailto:gurjyan@jlab.org" class="">gurjyan@jlab.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hi Nathan,<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
I wanted to know if there were changes in the memory footprint of the reconstruction application and if SLURM is reacting to resident memory and not the virtual memory. I have to say that your previous email answered my questions.<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
-Vardan<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
--------------------------------------------------<br class="">
Vardan H. Gyurjyan, Ph.D.<br class="">
Staff Scientist<br class="">
Thomas Jefferson Accelerator Facility<br class="">
Newport News, VA, 23606<br class="">
E-mail:<span class="Apple-converted-space"> </span><a href="mailto:gurjyan@jlab.org" style="color: purple; text-decoration: underline;" class="">gurjyan@jlab.org</a><br class="">
757-269-5879 (JLAB)<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Nathan Baltzell <<a href="mailto:baltzell@jlab.org" class="">baltzell@jlab.org</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Tuesday, February 7, 2023 at 9:37 PM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>Vardan Gyurjyan <<a href="mailto:gurjyan@jlab.org" class="">gurjyan@jlab.org</a>><br class="">
<b class="">Cc:<span class="Apple-converted-space"> </span></b>clas12 software <<a href="mailto:clas12_software@jlab.org" class="">clas12_software@jlab.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [Clas12_software] jlab batch job memory requests<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
And for exclusive jobs, the memory request is moot anyway. What are you really after here?<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Feb 7, 2023, at 11:14 AM, Nathan Baltzell <<a href="mailto:baltzell@jlab.org" style="color: purple; text-decoration: underline;" class="">baltzell@jlab.org</a>> wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Hi Vardan,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
What is the "reconstruction application"? The number I quoted was for standard simulation jobs. The larger jobs used for real data of course depend on the size of the job, but never more than 1 GB per thread.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
-Nathan<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<br class="">
<br class="">
<o:p class=""></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
On Feb 7, 2023, at 11:11 AM, Vardan Gyurjyan <<a href="mailto:gurjyan@jlab.org" style="color: purple; text-decoration: underline;" class="">gurjyan@jlab.org</a>> wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
What is the reconstruction application's resident (not the virtual) memory usage for the exclusive usage of a node?<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
--------------------------------------------------<br class="">
Vardan H. Gyurjyan, Ph.D.<br class="">
Staff Scientist<br class="">
Thomas Jefferson Accelerator Facility<br class="">
Newport News, VA, 23606<br class="">
E-mail:<span class="apple-converted-space"> </span><a href="mailto:gurjyan@jlab.org" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">gurjyan@jlab.org</span></a><br class="">
757-269-5879 (JLAB)<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 12pt;" class="">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Clas12_software <<a href="mailto:clas12_software-bounces@jlab.org" style="color: purple; text-decoration: underline;" class="">clas12_software-bounces@jlab.org</a>>
on behalf of Nathan Baltzell via Clas12_software <<a href="mailto:clas12_software@jlab.org" style="color: purple; text-decoration: underline;" class="">clas12_software@jlab.org</a>><br class="">
<b class="">Reply-To:<span class="apple-converted-space"> </span></b>Nathan Baltzell <<a href="mailto:baltzell@jlab.org" style="color: purple; text-decoration: underline;" class="">baltzell@jlab.org</a>><br class="">
<b class="">Date:<span class="apple-converted-space"> </span></b>Tuesday, February 7, 2023 at 11:05 AM<br class="">
<b class="">To:<span class="apple-converted-space"> </span></b>clas12 software <<a href="mailto:clas12_software@jlab.org" style="color: purple; text-decoration: underline;" class="">clas12_software@jlab.org</a>><br class="">
<b class="">Subject:<span class="apple-converted-space"> </span></b>[Clas12_software] jlab batch job memory requests</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
FYI Everyone,<span class="apple-converted-space"> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
We've brought up memory efficiency of batch job requests at CLAS12 collaboration and software meetings in previous years. Lots of jobs requesting a lot more memory than they actually use can make the farm unnecessarily idle and significantly reduce throughput
for everyone.<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
*** And now Scicomp has a larger initiative to improve farm efficiency, which includes contacting people running memory-inefficient jobs and potentially throttling their jobs if no action is taken. ***<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
You can check metrics of your batch jobs at:<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<a href="https://scicomp.jlab.org/" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">https://scicomp.jlab.org</span></a> <o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
There's a search feature at 'Slurm Jobs' (left sidebar) -> 'Jobs Query' (top), and 'Recent Jobs' (top), and also 'Memory Efficiency' (top). <o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Before launching a large number of new types of jobs, you can measure how much memory your jobs use. For example, by submitting a couple jobs and using that website, or by running your job interactively and checking in htop or ps or other system utilities.
And then set your SLURM/SWIF job memory request accordingly.<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
Note, standard CLAS12 simulation jobs (gemc plus recon-util) require less than 1.7 GB of memory.<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">
-Nathan</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>