[Frost] Fwd: [Clas_offline] Fwd: ENP consumption of disk space under /work

Fri Jun 9 16:49:05 EDT 2017

Eugene,
I think you should look a bit more into the directory trees. 
clasg9 owns quite a lot from calibration and cooking phases, but also because users ran their code as 'clasg9' (e.g. the g9b/ directory by Ross: 148GB).

If there is no complaint, I will clean up at least all old (pass0) stuff plus log & histogram files in g9frost/ plus previous_constants/ plus sc_calib/

Cheers,
Franz

----- Original Message -----
From: "Eugene Pasyuk" <pasyuk at jlab.org>
To: "frost" <frost at jlab.org>
Sent: Friday, June 9, 2017 4:14:50 PM
Subject: [Frost] Fwd: [Clas_offline] Fwd: ENP consumption of disk space	under /work

Folks, 

A general work disk has grown to unexpected size. Computer center wants us to shrink it. See messages below. 
clasg9 is not the biggest offender. Out of almost 100 TB used by Hall B we have only 1.2 TB. However, we should keep the only thing which are used on a regular basis. Work disk is not meant to be used for the permanent storage and it is not backed up. 
Here is the list of the directories there with ownership: 

-rw-r--r-- 1 6348 451 Nov 25 2016 76runs.list 
drwxrwsr-x 3 clasg9 4096 Jun 1 2010 TAG 
drwxr-sr-x 7 natalie 4096 Aug 6 2013 bin 
-rw-rw-r-- 1 clasg9 1361489920 Oct 17 2016 clasg9_g9a.tar 
drwxrwsr-x 3 clasg9 4096 Feb 15 00:49 clasweb_monitor 
drwxrwsr-x 4 clasg9 4096 Mar 10 2010 cooklinks 
drwxr-sr-x 2 crede 4096 Aug 16 2014 crede 
drwxrwsr-x 2 clasg9 4096 Oct 27 2011 dcalign 
drwxrwsr-x 3 dschott 4096 Jul 31 2014 dschott 
drwxr-sr-x 5 dugger 4096 May 26 10:54 dugger 
drwxr-s--- 2 fklein 4096 Apr 19 11:30 fklein 
drwxrwsr-x 3 clasg9 4096 Oct 19 2012 g9a 
drwxrwsr-x 3 clasg9 4096 Oct 21 2012 g9b 
-rwxr-xr-x 1 6890 474 Nov 8 2010 g9b_sub_script.pl 
drwxrwsr-x 8 clasg9 4096 Dec 4 2012 g9frost 
drwxr-sr-x 3 6659 4096 Jan 31 2011 mcandrew 
drwxrwsr-x 3 natalie 36864 May 4 10:43 natalie 
drwxrwsr-x 4 clasg9 4096 Aug 14 2014 pass1 
drwxr-sr-x 4 6890 4096 May 19 2010 previous_constants 
-rwxrwxrwx 1 clasg9 15539 Nov 11 2010 runs_g9b_pass0 
-rwxrwxrwx 1 6890 15563 Nov 8 2010 runs_g9b_pass0~ 
drwxrwsrwx 2 clasg9 20480 Feb 15 01:14 sc_calib 
drwxr-sr-x 2 sfegan 73728 Feb 15 00:49 sfegan 
drwxrwsr-x 3 supark 4096 Aug 8 2012 skpark 
drwxr-sr-x 7 7216 4096 Jan 21 2016 yuqing 
drwxrwsr-x 8 clasg9 4096 Sep 19 2015 zana 

Here is the space taken by these directories: 

129K 76runs.list 
6.9M TAG 
191M bin 
1.3G clasg9_g9a.tar 
363M clasweb_monitor 
137M cooklinks 
4.0K crede 
261K dcalign 
1.6G dschott 
4.2G dugger 
1.9G fklein 
2.8G g9a 
148G g9b 
129K g9b_sub_script.pl 
578G g9frost 
8.0K mcandrew 
131G natalie 
18G pass1 
121G previous_constants 
129K runs_g9b_pass0 
129K runs_g9b_pass0~ 
53G sc_calib 
34G sfegan 
112M skpark 
33G yuqing 
4.7G zana 

Most of the file were not used for ages. 
Please review an let me know what is being used NOW and should be kept. Unclaimed files will be deleted. 
If there is something that you don't use but plan to use in a distant future and it is not easily recreatable you may think of putting this on tape. 
Please do it ASAP. 

Thanks, 

-Eugene 

From: "Harut Avagyan" <avakian at jlab.org> 
To: "clas offline" <clas_offline at jlab.org> 
Sent: Thursday, June 1, 2017 11:01:24 AM 
Subject: [Clas_offline] Fwd: ENP consumption of disk space under /work 

Dear All, 

As you can see from the e-mail below, keeping all our work disk space requires some additional funding. 

Option 3 will inevitably impact on farm operations, removing of ~20% space from Lustre. 

We can also choose something between options 1) and 3). 

Please revise the content and move at least 75% of what is in /work/clas to either /cache or /volatile? 

The current Hall-B usage includes: 

550G hallb/bonus 
1.5T hallb/clase1 
3.6T hallb/clase1-6 
3.3T hallb/clase1dvcs 
2.8T hallb/clase1dvcs2 
987G hallb/clase1f 
1.8T hallb/clase2 
1.6G hallb/clase5 
413G hallb/clase6 
2.2T hallb/claseg1 
3.9T hallb/claseg1dvcs 
1.2T hallb/claseg3 
4.1T hallb/claseg4 
2.7T hallb/claseg5 
1.7T hallb/claseg6 
367G hallb/clas-farm-output 
734G hallb/clasg10 
601G hallb/clasg11 
8.1T hallb/clasg12 
2.4T hallb/clasg13 
2.4T hallb/clasg14 
28G hallb/clasg3 
5.8G hallb/clasg7 
269G hallb/clasg8 
1.2T hallb/clasg9 
1.3T hallb/clashps 
1.8T hallb/clas-production 
5.6T hallb/clas-production2 
1.4T hallb/clas-production3 
12T hallb/hps 
13T hallb/prad 

Regards, Harut 

P.S. Few times we had crashes and they may also happen in future, so keeping important files in /work is not recommended. 
You can see the list of lost files in /site/scicomp/lostfiles.txt and /site/scicomp/lostfiles-jan-2017.txt 

-------- Forwarded Message -------- Subject: 	ENP consumption of disk space under /work 
Date: 	Wed, 31 May 2017 10:35:51 -0400 
From: 	Chip Watson <watson at jlab.org> 

To: 	Sandy Philpott <philpott at jlab.org> , Graham Heyes <heyes at jlab.org> , Ole Hansen <ole at jlab.org> , Harut Avakian <avakian at jlab.org> , Brad Sawatzky <brads at jlab.org> , Mark M. Ito <marki at jlab.org> 

All,

As I have started on the procurement of the new /work file server, I 
have discovered that Physics' use of /work has grown unrestrained over 
the last year or two.

"Unrestrained" because there is no way under Lustre to restrain it 
except via a very unfriendly Lustre quota system.  As we leave some 
quota headroom to accommodate large swings in usage for each hall for 
cache and volatile, then /work continues to grow.

Total /work has now reached 260 TB, several times larger than I was 
anticipating.  This constitutes more than 25% of Physics' share of 
Lustre, compared to LQCD which uses less than 5% of its disk space on 
the un-managed /work.

It would cost Physics an extra $25K (total $35K - $40K) to treat the 260 
TB as a requirement.

There are 3 paths forward:

(1) Physics cuts its use of /work by a factor of 4-5.
(2) Physics increases funding to $40K
(3) We pull a server out of Lustre, decreasing Physics' share of the 
system, and use that as half of the new active-active pair, beefing it 
up with SSDs and perhaps additional memory; this would actually shrink 
Physics near term costs, but puts higher pressure on the file system for 
the farm

The decision is clearly Physics', but I do need a VERY FAST response to 
this question, as I need to move quickly now for LQCD's needs.

Hall D + GlueX,  96 TB
CLAS + CLAS12, 98 TB
Hall C,                35 TB
Hall A <unknown, still scanning>

Email, call (x7101), or drop by today 1:30-3:00 p.m. for discussion.

thanks,
Chip 

_______________________________________________ 
Clas_offline mailing list 
Clas_offline at jlab.org 
https://mailman.jlab.org/mailman/listinfo/clas_offline 

_______________________________________________
Frost mailing list
Frost at jlab.org
https://mailman.jlab.org/mailman/listinfo/frost