[Clas12_software] jlab batch job memory requests

[Vardan Gyurjyan]

What is the reconstruction application's resident (not the virtual) memory usage for the exclusive usage of a node?

--------------------------------------------------
Vardan H. Gyurjyan, Ph.D.
Staff Scientist
Thomas Jefferson Accelerator Facility
Newport News, VA, 23606
E-mail: gurjyan at jlab.org
757-269-5879 (JLAB)

From: Clas12_software <clas12_software-bounces at jlab.org> on behalf of Nathan Baltzell via Clas12_software <clas12_software at jlab.org>
Reply-To: Nathan Baltzell <baltzell at jlab.org>
Date: Tuesday, February 7, 2023 at 11:05 AM
To: clas12 software <clas12_software at jlab.org>
Subject: [Clas12_software] jlab batch job memory requests

FYI Everyone,

We've brought up memory efficiency of batch job requests at CLAS12 collaboration and software meetings in previous years.  Lots of jobs requesting a lot more memory than they actually use can make the farm unnecessarily idle and significantly reduce throughput for everyone.

*** And now Scicomp has a larger initiative to improve farm efficiency, which includes contacting people running memory-inefficient jobs and potentially throttling their jobs if no action is taken.  ***

You can check metrics of your batch jobs at:

https://scicomp.jlab.org

There's a search feature at 'Slurm Jobs'  (left sidebar) -> 'Jobs Query' (top), and  'Recent Jobs' (top), and also 'Memory Efficiency' (top).

Before launching a large number of new types of jobs, you can measure how much memory your jobs use.  For example, by submitting a couple jobs and using that website, or by running your job interactively and checking in htop or ps or other system utilities.  And then set your SLURM/SWIF job memory request accordingly.

Note, standard CLAS12 simulation jobs (gemc plus recon-util) require less than 1.7 GB of memory.

-Nathan


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_software/attachments/20230207/d00241dc/attachment.html>