[Clas12_software] Fwd: class12-2 jobs performing lots of small i/o

[John Price]

If the data are being analyzed "as-is", then there will definitely be
many small I/O calls.  The batch system copies the files from tape to
the /cache filesystem, and then makes a symlink to the disk on the local
compute node.  The way to avoid this is to copy the data files to the
local disk before starting the analysis.  

I have a script that I have been using for this purpose with g12
(granted, that's CLAS6, not CLAS12, but the principle is the same).

John

On Tue, 2018-07-03 at 08:03 +0900, Francois-Xavier Girod wrote:
> The I/O to those jobs is defined as per CC guidelines, there is no
> “small” I/O the hippo files are about 5 GB merging 10 evio files
> together. I think the CC need to have better diagnostics. 
> 
> On Tue, Jul 3, 2018 at 7:56 AM Harout Avakian <avakian at jlab.org>
> wrote:
> 
>         FYI
>         
>         I understood that was fixed.  FX could you please check what
>         is the problem.
>         
>         
>         Harut
>         
>         -------- Forwarded Message -------- 
>                               Subject: 
>         class12-2 jobs performing lots
>         of small i/o
>                                  Date: 
>         Mon, 2 Jul 2018 08:58:04 -0400
>         (EDT)
>                                  From: 
>         Kurt Strosahl
>         <strosahl at jlab.org>
>                                    To: 
>         Harut Avagyan
>         <avakian at jlab.org>
>                                    CC: 
>         sciops <sciops at jlab.org>
>         
>         
>         Harut,
>         
>             There are a large number of clas12 jobs running through the farm under user clas12-2, these jobs are performing lots of small i/o.
>         
>         An example of one of these jobs is:
>         
>         Job Index:	55141495
>         User Name:	clas12-2
>         Job Name:	R4013_13
>         Project: 	clas12
>         Queue:		prod64
>         Hostname:	farm12021
>         CPU Req: 	1 centos7 core requested
>         MemoryReq:	9 GB
>         Status:		ACTIVE
>         
>         You can see the small i/o by looking: https://scicomp.jlab.org/scicomp/index.html#/lustre/users
>         
>         w/r,
>         Kurt J. Strosahl
>         System Administrator: Lustre, HPC
>         Scientific Computing Group, Thomas Jefferson National Accelerator Facility
>         _______________________________________________
>         Clas12_software mailing list
>         Clas12_software at jlab.org
>         https://mailman.jlab.org/mailman/listinfo/clas12_software
> _______________________________________________
> Clas12_software mailing list
> Clas12_software at jlab.org
> https://mailman.jlab.org/mailman/listinfo/clas12_software

-- 
John W. Price
Professor and Chair, CSUDH Department of Physics
Coordinator, Science, Mathematics, and Technology Program
Director, Office of Undergraduate Research, Scholarship, and Creative
Activity
310-243-3403