[Halld-offline] Offline Software Meeting Minutes, June 14, 2017

[Chip Watson]

Team,

Do we have any evidence that file copies at job launch are a problem?

Is it true that file copies (cp) do not use large block transfers (1 MB 
or larger)?

If so, can we recommend an alternative way to stage files which will 
definitely use large block transfers?

Chip


On 6/16/17 6:16 PM, Mark Ito wrote:
> Recently calibration jobs run under the gxproj3 account were flagged 
> by SciComp as causing excessive IO-ops on Lustre. Sean believes that 
> this was due to many jobs starting at the same time and each one 
> copying its input data file to the local farm node disk before 
> processing. This practice was recommended by SciComp some months ago. 
> After further consultation with SciComp he has switched to analyzing 
> the data directly from the Lustre-based cache disk (no local copy). 
> SciComp has not weighed in about whether this helps the situation or 
> not. No news is good news?