[Halld-offline] advice on high speed file copies offsite

[Michael Staib]

Hi Matt,

The globus node is indeed very restricted. However, if you set up IU as an endpoint, the transfers can be easily handled through the web interface at www.globus.org. I have been using this to send data to CMU. It does considerably better than scp or rsync. I get effective speeds of 200-300 Mb/s.

Curtis did the configuration of the endpoint at CMU, so I am not exactly sure of the details.

Best,
Mike





On Jan 27, 2015, at 2:01 PM, Shepherd, Matthew <mashephe at indiana.edu> wrote:

> 
> Hi all,
> 
> Does anyone have suggestions for doing high speed
> file copies offsite?
> 
> I consulted this page:
> 
> https://scicomp.jlab.org/docs/node/11
> 
> Which suggests to use the node:  globus.jlab.org
> 
> This node has real limited functionality.  I can't get rsync to
> work and there is no tar to use with ssh.
> 
> I tried just:
> 
> scp -r
> 
> This works, but I'd like to know if I can go faster.
> I'm getting speeds that are orders of magnitude below
> the 10 Gbps pipe that is at JLab and also at my
> end.  I'm sure there is other competing traffic, but
> I think part of the problem might be the scp is poorly
> optimized for coping many small files.
> 
> Any experience with this in the offline group?
> 
> Matt
> 
> ---------------------------------------------------------------------
> Matthew Shepherd, Associate Professor
> Department of Physics, Indiana University, Swain West 265
> 727 East Third Street, Bloomington, IN 47405
> 
> Office Phone:  +1 812 856 5808
> 
> 
> _______________________________________________
> Halld-offline mailing list
> Halld-offline at jlab.org
> https://mailman.jlab.org/mailman/listinfo/halld-offline

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20150127/38230391/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2015-01-27 at 2.07.50 PM.png
Type: image/png
Size: 72470 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20150127/38230391/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2015-01-27 at 2.07.42 PM.png
Type: image/png
Size: 91704 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20150127/38230391/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2205 bytes
Desc: not available
URL: <https://mailman.jlab.org/pipermail/halld-offline/attachments/20150127/38230391/attachment.p7s>