[Halld-offline] distributing REST files from initial launch

[Richard Jones]

Hello Matt,

I fully support the plan you propose. UConn would be one of your first
subscribers. Either your option 1 or 2 would be acceptable for us,
whichever you prefer. We have a similar resource here at UConn to the IU
data capacitor, soon to be upgraded to 2PB. Not all of that would be
available to us, of course, and the effective lifetime of files on this
service is still unknown. The globus endpoint name is uchc#starfish_test.
Typically for such a thing the lifetime starts out very long when the full
storage first becomes available, and then shrinks as demand grows over
time. A couple of questions to ponder regarding life-cycle of these REST
files.

   1. How long might they be expected to live, before another launch or
   larger dataset makes them less relevant? My guess might be something on the
   order 4-6 months, at least for now.
   2. What would the lifetime of a dataset be on the IU data capacitor? As
   long as it is equal to or greater than the answer to question 1, the IU
   solution sounds like the right one.

-Richard Jones

On Mon, Jun 13, 2016 at 9:03 AM, Shepherd, Matthew <mashephe at indiana.edu>
wrote:

>
> Hi all,
>
> I'm interesting in copying the REST files from the initial launch back to
> Indiana for analysis.  In order to minimize load on disk/network coming out
> of the lab, I wanted to try to coordinate this with others who might be
> interested in having a copy of the data.
>
> My initial plan is to copy this to the IU "data capacitor" which is a
> university managed 3.5 PB high throughput scratch drive that is mounted on
> a couple of our large centralized clusters.  It is configured as a globus
> endpoint so this copy should be easy.  A quick test last week showed 150
> MB/s transfer from JLab.
>
> Once it is here, then I could share with others.  Access to the data
> capacitor requires IU credentials.  There are two ways to get the files
> from here to another institution:
>
> (1) I can request an affiliate account for a representative at an
> institution.  This requires submitting some forms, etc., but results in an
> account that has full access to IU systems.  Using this credential you can
> authenticate yourself and access files via the data capacitor globus
> endpoint.
>
> (2) You can give me access to your globus endpoint and I can initiate the
> transfer using the web interface.  (This is probably easiest.)
>
> Alternatively, if there is another institution who is interested in
> hosting and distributing the REST data, then I'd be happy to copy from
> there.
>
> Thoughts?  How many are interested in getting a local copy of REST?
>
> Matt
>
> ---------------------------------------------------------------------
> Matthew Shepherd, Associate Professor
> Department of Physics, Indiana University, Swain West 265
> 727 East Third Street, Bloomington, IN 47405
>
> Office Phone:  +1 812 856 5808
>
>
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20160613/bb338d17/attachment-0002.html>