[Halld-offline] Fwd: Re: Scientific Computing and Physics topics

Mon Jan 6 12:58:00 EST 2020

People,

We have a new scheme for assigning priorities on the JLab farm. Some 
details are in Bryan Hess's messages below. This should correct some 
anomalies we saw in farm priority at the end of last year. Let me know 
if things do not go as you expect.

   -- Mark

-------- Forwarded Message --------
Subject: 	Re: Scientific Computing and Physics topics
Date: 	Mon, 6 Jan 2020 09:58:17 -0500
From: 	Bryan Hess <bhess at jlab.org>
To: 	Ying Chen <ychen at jlab.org>, Mark Ito <marki at jlab.org>, Nathan 
Baltzell <baltzell at jlab.org>
CC: 	Graham Heyes <heyes at jlab.org>, David Lawrence <davidl at jlab.org>, 
Ole Hansen <ole at jlab.org>, Brad Sawatzky <brads at jlab.org>, Thomas 
Britton <tbritton at jlab.org>

The changes to the hall B and hall D slurm accounts that I emailed about 
on Friday were completed this morning. Ying and I will monitor it.

Just to recap what this means for submitting jobs going forward:

  * In Auger, there are two new projects names  "gluex-pro" and 
    "hallb-pro" that should be used for the production accounts starting
    now to get the highest share.
  * non-production users are unaffected by these changes

Let me know if anything looks odd.

--Bryan

------------------------------------------------------------------------
*From:* Bryan Hess <bhess at jlab.org>
*Sent:* Friday, January 3, 2020 3:20 PM
*To:* Ying Chen <ychen at jlab.org>; Mark Ito <marki at jlab.org>; Nathan 
Baltzell <baltzell at jlab.org>
*Cc:* Graham Heyes <heyes at jlab.org>; David Lawrence <davidl at jlab.org>; 
Ole Hansen <ole at jlab.org>; Brad Sawatzky <brads at jlab.org>; Thomas 
Britton <tbritton at jlab.org>
*Subject:* Re: Scientific Computing and Physics topics
This afternoon I normalized the shares in slurm so that they sum to 100 
at each level. I have also rearranged the fairshare tree, which you can 
see with the sshare command or on the fairshare web pages.

On Monday morning I will modify the production slurm accounts for both 
hall B and hall D to give them access to the hallb-pro and halld-pro 
slurm accounts, respectively. I will also turn off the 
"fairshare=parent" trick, and then we will be in the final state for the 
tree as shown. I'll email once the change is made. This will not break 
any jobs that are queued, but I want to watch things after the change.

Because the production accounts have the "fairshare=parent" set as of 
this writing, they will run with no problem over the weekend.

Here's the updated picture, which matches the sshare command.

------------------------------------------------------------------------
*From:* Ying Chen <ychen at jlab.org>
*Sent:* Friday, January 3, 2020 2:01 PM
*To:* Mark Ito <marki at jlab.org>; Bryan Hess <bhess at jlab.org>; Nathan 
Baltzell <baltzell at jlab.org>
*Cc:* Graham Heyes <heyes at jlab.org>; David Lawrence <davidl at jlab.org>; 
Ole Hansen <ole at jlab.org>; Brad Sawatzky <brads at jlab.org>; Thomas 
Britton <tbritton at jlab.org>
*Subject:* Re: Scientific Computing and Physics topics
Mark,

I did the mapping from user name to queue before (regular user's job go 
to general queue and production user's job go to production queue.

But few months ago, we remove general queue and all jobs are go to 
production queue.

This change is different, since we made the fairshare setting to a tree 
structure, the project to slurm account mapping will only map to tree 
leaf accounts. So there is no simple and straight way to map the project 
to slurm account just using the user name.

Ying
------------------------------------------------------------------------
*From:* Mark Ito <marki at jlab.org>
*Sent:* Friday, January 3, 2020 12:02 PM
*To:* Ying Chen <ychen at jlab.org>; Bryan Hess <bhess at jlab.org>; Nathan 
Baltzell <baltzell at jlab.org>
*Cc:* Graham Heyes <heyes at jlab.org>; David Lawrence <davidl at jlab.org>; 
Ole Hansen <ole at jlab.org>; Brad Sawatzky <brads at jlab.org>; Thomas 
Britton <tbritton at jlab.org>
*Subject:* Re: Scientific Computing and Physics topics

Folks,

I'm confused.

I thought that the way this would work is that the priority for 
production accounts gets assigned based solely on username of the 
account. There would be no need to submit to a special project. Is this 
a backward-compatibility thing or is my understanding fundamentally 
off-base?

   -- Mark

On 1/3/20 11:32 AM, Ying Chen wrote:
> Yes, if any user (regular or production) calls jsub with old project 
> names, everything will work and job will submit to slurm with old 
> slurm accounts (like hallb, halld). The point is we don't want to make 
> the old script fail.
>
> If these production user accounts want their jobs have high priority, 
> they must change the project name to new pro name (like gluex-pro or 
> hallb-pro).
>
> Ying
> ------------------------------------------------------------------------
> *From:* Bryan Hess <bhess at jlab.org> <mailto:bhess at jlab.org>
> *Sent:* Friday, January 3, 2020 11:16 AM
> *To:* Nathan Baltzell <baltzell at jlab.org> <mailto:baltzell at jlab.org>
> *Cc:* Mark Ito <marki at jlab.org> <mailto:marki at jlab.org>; Graham Heyes 
> <heyes at jlab.org> <mailto:heyes at jlab.org>; David Lawrence 
> <davidl at jlab.org> <mailto:davidl at jlab.org>; Ole Hansen <ole at jlab.org> 
> <mailto:ole at jlab.org>; Brad Sawatzky <brads at jlab.org> 
> <mailto:brads at jlab.org>; Thomas Britton <tbritton at jlab.org> 
> <mailto:tbritton at jlab.org>; Ying Chen <ychen at jlab.org> 
> <mailto:ychen at jlab.org>
> *Subject:* Re: Scientific Computing and Physics topics
> Yes, that's been the behavior in the past. Ying can correct me if 
> something has changed.
> ------------------------------------------------------------------------
> *From:* Nathan Baltzell <baltzell at jlab.org> <mailto:baltzell at jlab.org>
> *Sent:* Friday, January 3, 2020 11:10 AM
> *To:* Bryan Hess <bhess at jlab.org> <mailto:bhess at jlab.org>
> *Cc:* Mark Ito <marki at jlab.org> <mailto:marki at jlab.org>; Graham Heyes 
> <heyes at jlab.org> <mailto:heyes at jlab.org>; David Lawrence 
> <davidl at jlab.org> <mailto:davidl at jlab.org>; Ole Hansen <ole at jlab.org> 
> <mailto:ole at jlab.org>; Brad Sawatzky <brads at jlab.org> 
> <mailto:brads at jlab.org>; Thomas Britton <tbritton at jlab.org> 
> <mailto:tbritton at jlab.org>; Ying Chen <ychen at jlab.org> 
> <mailto:ychen at jlab.org>
> *Subject:* Re: Scientific Computing and Physics topics
> And if those “production” users submit with the old project (e.g. 
> “clas12”), will it still work but with the non-production fairshare?
>
> I think it’s also fine with me (or maybe better) if it instead errs on 
> job submission with the old project, just want to know what to expect ...
>
>
>
> > On Jan 3, 2020, at 10:45, Bryan Hess <bhess at jlab.org> 
> <mailto:bhess at jlab.org> wrote:
> >
> > Yes, the web page code is ready. There is one change that will 
> affect only the production users who submit via Auger once we make the 
> change to the tree:
> >
> > Hall B production users will need to change their Auger project to 
> "hallb-pro"
> > Hall D production users will need to change their Auger project to 
> "gluex-pro"
> >
> > No other users will need to do anything different than they do now.
> >
> >
> > --Bryan
> > From: Mark Ito <marki at jlab.org> <mailto:marki at jlab.org>
> > Sent: Friday, January 3, 2020 10:03 AM
> > To: Bryan Hess <bhess at jlab.org> <mailto:bhess at jlab.org>; Graham 
> Heyes <heyes at jlab.org> <mailto:heyes at jlab.org>; David Lawrence 
> <davidl at jlab.org> <mailto:davidl at jlab.org>; Ole Hansen <ole at jlab.org> 
> <mailto:ole at jlab.org>; Nathan Baltzell <baltzell at jlab.org> 
> <mailto:baltzell at jlab.org>; Brad Sawatzky <brads at jlab.org> 
> <mailto:brads at jlab.org>; Thomas Britton <tbritton at jlab.org> 
> <mailto:tbritton at jlab.org>; Ying Chen <ychen at jlab.org> 
> <mailto:ychen at jlab.org>
> > Subject: Re: Scientific Computing and Physics topics
> >
> > Bryan, et al.
> >
> > On 1/2/20 1:44 PM, Bryan Hess wrote:
> >>
> >> Unfortunately we can't recycle the names "hallb" and "halld" as 
> anything other than leaf nodes because of the way the database is 
> structured. This does introduce some inconsistency in the naming. We 
> can do anything we want with a new account name, but the old ones need 
> to continue to be leaf nodes once they are leaf nodes, at least until 
> Auger is gone.
> > OK. If that's the way it is, then that is the way it is. This falls 
> under the category "something I was missing".
> >>
> >> We could create a convention like "halld-root" or "halld-top" for 
> each if that helps to regularize. Any new name works.
> > I think either of those would be better than just "d". So good.
> >
> >>
> >> I'll copy Ying in.
> >> Once we agree, we will make the change.
> > Have the web page changes been dealt with then?
> >
> >   -- Mark
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20200106/fa07e7ff/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-nsozgseo.png
Type: image/png
Size: 139089 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20200106/fa07e7ff/attachment-0001.png>