[Halld-offline] distributing REST files from initial launch

[Naomi Jarvis]

We'd use it.

Naomi.


>
> Richard,
>
> The useful life of these files, in principle, could be quite long.  I
> think anticipating a life of a year or so would be good.
>
> The IU data capacitor will keep files live as long as they have been
> accessed in the preceding 60 days.  Presently that disk is fairly full:
> 102 TB free out of 3.5 PB capacity.  A few weeks ago when I looked there
> were about 600 TB free.  This disk has been around for years so I'm happy
> just using it and letting the IT folks deal with managing/making available
> free space.
>
> To preserve files, one thing I can do is copy to the IU "Scholarly Data
> Archive" - this can be done directly from the data capacitor.  The SDA is
> a 15 PB tape system with with a 1.8 PB disk cache that is also configured
> as a globus endpoint.  This would allow preservation of the REST at IU at
> least for its usable life and would avoid subsequent fetching from JLab if
> the data expire on the data capacitor.
>
> Matt
>
>> On Jun 13, 2016, at 9:21 AM, Richard Jones <richard.t.jones at uconn.edu>
>> wrote:
>>
>> Hello Matt,
>>
>> I fully support the plan you propose. UConn would be one of your first
>> subscribers. Either your option 1 or 2 would be acceptable for us,
>> whichever you prefer. We have a similar resource here at UConn to the IU
>> data capacitor, soon to be upgraded to 2PB. Not all of that would be
>> available to us, of course, and the effective lifetime of files on this
>> service is still unknown. The globus endpoint name is
>> uchc#starfish_test. Typically for such a thing the lifetime starts out
>> very long when the full storage first becomes available, and then
>> shrinks as demand grows over time. A couple of questions to ponder
>> regarding life-cycle of these REST files.
>> 	• How long might they be expected to live, before another launch or
>> larger dataset makes them less relevant? My guess might be something on
>> the order 4-6 months, at least for now.
>> 	• What would the lifetime of a dataset be on the IU data capacitor?
>> As long as it is equal to or greater than the answer to question 1, the
>> IU solution sounds like the right one.
>> -Richard Jones
>>
>> On Mon, Jun 13, 2016 at 9:03 AM, Shepherd, Matthew
>> <mashephe at indiana.edu> wrote:
>>
>> Hi all,
>>
>> I'm interesting in copying the REST files from the initial launch back
>> to Indiana for analysis.  In order to minimize load on disk/network
>> coming out of the lab, I wanted to try to coordinate this with others
>> who might be interested in having a copy of the data.
>>
>> My initial plan is to copy this to the IU "data capacitor" which is a
>> university managed 3.5 PB high throughput scratch drive that is mounted
>> on a couple of our large centralized clusters.  It is configured as a
>> globus endpoint so this copy should be easy.  A quick test last week
>> showed 150 MB/s transfer from JLab.
>>
>> Once it is here, then I could share with others.  Access to the data
>> capacitor requires IU credentials.  There are two ways to get the files
>> from here to another institution:
>>
>> (1) I can request an affiliate account for a representative at an
>> institution.  This requires submitting some forms, etc., but results in
>> an account that has full access to IU systems.  Using this credential
>> you can authenticate yourself and access files via the data capacitor
>> globus endpoint.
>>
>> (2) You can give me access to your globus endpoint and I can initiate
>> the transfer using the web interface.  (This is probably easiest.)
>>
>> Alternatively, if there is another institution who is interested in
>> hosting and distributing the REST data, then I'd be happy to copy from
>> there.
>>
>> Thoughts?  How many are interested in getting a local copy of REST?
>>
>> Matt
>>
>> ---------------------------------------------------------------------
>> Matthew Shepherd, Associate Professor
>> Department of Physics, Indiana University, Swain West 265
>> 727 East Third Street, Bloomington, IN 47405
>>
>> Office Phone:  +1 812 856 5808
>>
>>
>> _______________________________________________
>> Halld-offline mailing list
>> Halld-offline at jlab.org
>> https://mailman.jlab.org/mailman/listinfo/halld-offline
>>
>
>
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline