[Halld-offline] ENP consumption of disk space under /work

[Mark Ito]

In my previous estimate, of the cache portion, 278 TB, only 105 TB of 
that is pinned. The unpinned part is presumeably old files that should 
be gone, but have not been deleted since there happens to be no demand 
for the space. If we use 105 TB as our cache usage then re-doing your 
estimate gives 555 TB, which means in 9 months we will have 270 TB of 
unused space. Which would mean that we have room to increase our usage 
without buying anything!


On 06/07/2017 05:54 PM, Chip Watson wrote:
>
> Mark,
>
> I still need you to answer the question of how to further reduce usage 
> and how to configure.  Your usage as you report it is about 370 TB.  
> Assuming that Hall B needs the same within 9 months, and that A+C need 
> half as much, then that leads to a total of 925TB which is more than 
> Physics owns, by 100 TB (NOT CURRENT USAGE, JUST PROJECTION BASED ON 
> GLUEX USAGE).
>
> There is also the question of how to split the storage budget. In 
> budget, you can have new half a JBOD: 21 disks configured as 3 RAID z2 
> stripes of 5+2 disks, 8TB, thus 120 raw data, 108 in a file system, 
> and 86 TB at 80% -- for all of GlueX, CLAS-12, A and C.  If GlueX is 
> 40% of the total, that makes 35TB, and you are still high by 70%.
>
> The other low cost option is to re-purpose a 2016 Lustre node so that 
> /work is twice this size (one full JBOD), and GlueX can use 70TB as 
> /work.  But then you must reduce /cache + /volatile by a comparable 
> amount since we have to pull a node out of production.  And this still 
> isn't free since we'll need a total of 4 RAID cards instead of 2 to 
> provide correct performance, and we'll need to add SSD's to the mix.
>
> So, in the absence of money (which clearly seems to be the case), do 
> you choose (a) reduce your use of work by 1.7x, or (b) reduce your use 
> of cache + volatile by 25%.  There is no middle case.
>
> thanks,
>
> Chip
>
>
> On 6/7/17 5:30 PM, Mark Ito wrote:
>>
>> Summarizing Hall D work disk usage (/work/halld only):
>>
>> o using du, today 2017-06-06, 59 TB
>>
>> o from our disk-management database, a couple of days ago, 
>> 2017-06-04, 86 TB
>>
>> I also know that one of our students got rid of about 20 TB of 
>> unneeded files yesterday. That accounts for part of the drop.
>>
>> We produce a report from that database 
>> <https://halldweb.jlab.org/disk_management/work_report.html> that is 
>> updated every few days.
>>
>> From the SciComp pages, Hall D is using 287 TB on cache and 21 TB on 
>> volatile.
>>
>> My view is that this level of work disk usage is more or less as 
>> expected, consistent with our previous estimates, and not 
>> particularly abusive. That having been said, I am sure there is a lot 
>> that can be cleaned up. But as Ole pointed out, disk usage grows 
>> naturally and we were not aware that this was a problem. I seem to 
>> recall that we agreed to respond to emails that would be sent when we 
>> reached 90% of too much, no? Was the email sent out?
>>
>> One mystery: when I ask Lustre what we are using I get:
>>
>> ifarm1402:marki:marki>   lfs quota -gh halld /lustre
>> Disk quotas for group halld (gid 267):
>>       Filesystem    used   quota   limit   grace   files   quota   limit   grace
>>          /lustre    290T    470T    500T       - 15106047       0       0       -
>> which is less than cache + volatile, not to mention work. I thought 
>> that to a good approximation this 290 TB should be the sum of all 
>> three. What am I missing?
>>
>> On 05/31/2017 10:35 AM, Chip Watson wrote:
>>> All,
>>>
>>> As I have started on the procurement of the new /work file server, I 
>>> have discovered that Physics' use of /work has grown unrestrained 
>>> over the last year or two.
>>>
>>> "Unrestrained" because there is no way under Lustre to restrain it 
>>> except via a very unfriendly Lustre quota system. As we leave some 
>>> quota headroom to accommodate large swings in usage for each hall 
>>> for cache and volatile, then /work continues to grow.
>>>
>>> Total /work has now reached 260 TB, several times larger than I was 
>>> anticipating.  This constitutes more than 25% of Physics' share of 
>>> Lustre, compared to LQCD which uses less than 5% of its disk space 
>>> on the un-managed /work.
>>>
>>> It would cost Physics an extra $25K (total $35K - $40K) to treat the 
>>> 260 TB as a requirement.
>>>
>>> There are 3 paths forward:
>>>
>>> (1) Physics cuts its use of /work by a factor of 4-5.
>>> (2) Physics increases funding to $40K
>>> (3) We pull a server out of Lustre, decreasing Physics' share of the 
>>> system, and use that as half of the new active-active pair, beefing 
>>> it up with SSDs and perhaps additional memory; this would actually 
>>> shrink Physics near term costs, but puts higher pressure on the file 
>>> system for the farm
>>>
>>> The decision is clearly Physics', but I do need a VERY FAST response 
>>> to this question, as I need to move quickly now for LQCD's needs.
>>>
>>> Hall D + GlueX,  96 TB
>>> CLAS + CLAS12, 98 TB
>>> Hall C,                35 TB
>>> Hall A <unknown, still scanning>
>>>
>>> Email, call (x7101), or drop by today 1:30-3:00 p.m. for discussion.
>>>
>>> thanks,
>>> Chip
>>>
>>
>> -- 
>> Mark Ito,marki at jlab.org, (757)269-5295
>

-- 
Mark Ito, marki at jlab.org, (757)269-5295

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20170608/eee7fc4e/attachment-0002.html>