[Clas12_software] Fwd: class12-2 jobs performing lots of small i/o

[Francois-Xavier Girod]

1) sounds like poor design to me

2) the sym links are removed and the files are copied from cache already

On Tue, Jul 3, 2018, 08:23 John Price <jprice at csudh.edu> wrote:

> If the data are being analyzed "as-is", then there will definitely be
> many small I/O calls.  The batch system copies the files from tape to
> the /cache filesystem, and then makes a symlink to the disk on the local
> compute node.  The way to avoid this is to copy the data files to the
> local disk before starting the analysis.
>
> I have a script that I have been using for this purpose with g12
> (granted, that's CLAS6, not CLAS12, but the principle is the same).
>
> John
>
> On Tue, 2018-07-03 at 08:03 +0900, Francois-Xavier Girod wrote:
> > The I/O to those jobs is defined as per CC guidelines, there is no
> > “small” I/O the hippo files are about 5 GB merging 10 evio files
> > together. I think the CC need to have better diagnostics.
> >
> > On Tue, Jul 3, 2018 at 7:56 AM Harout Avakian <avakian at jlab.org>
> > wrote:
> >
> >         FYI
> >
> >         I understood that was fixed.  FX could you please check what
> >         is the problem.
> >
> >
> >         Harut
> >
> >         -------- Forwarded Message --------
> >                               Subject:
> >         class12-2 jobs performing lots
> >         of small i/o
> >                                  Date:
> >         Mon, 2 Jul 2018 08:58:04 -0400
> >         (EDT)
> >                                  From:
> >         Kurt Strosahl
> >         <strosahl at jlab.org>
> >                                    To:
> >         Harut Avagyan
> >         <avakian at jlab.org>
> >                                    CC:
> >         sciops <sciops at jlab.org>
> >
> >
> >         Harut,
> >
> >             There are a large number of clas12 jobs running through the
> farm under user clas12-2, these jobs are performing lots of small i/o.
> >
> >         An example of one of these jobs is:
> >
> >         Job Index:    55141495
> >         User Name:    clas12-2
> >         Job Name:     R4013_13
> >         Project:      clas12
> >         Queue:                prod64
> >         Hostname:     farm12021
> >         CPU Req:      1 centos7 core requested
> >         MemoryReq:    9 GB
> >         Status:               ACTIVE
> >
> >         You can see the small i/o by looking:
> https://scicomp.jlab.org/scicomp/index.html#/lustre/users
> >
> >         w/r,
> >         Kurt J. Strosahl
> >         System Administrator: Lustre, HPC
> >         Scientific Computing Group, Thomas Jefferson National
> Accelerator Facility
> >         _______________________________________________
> >         Clas12_software mailing list
> >         Clas12_software at jlab.org
> >         https://mailman.jlab.org/mailman/listinfo/clas12_software
> > _______________________________________________
> > Clas12_software mailing list
> > Clas12_software at jlab.org
> > https://mailman.jlab.org/mailman/listinfo/clas12_software
>
> --
> John W. Price
> Professor and Chair, CSUDH Department of Physics
> Coordinator, Science, Mathematics, and Technology Program
> Director, Office of Undergraduate Research, Scholarship, and Creative
> Activity
> 310-243-3403
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.jlab.org/pipermail/clas12_software/attachments/20180703/3a05efe8/attachment-0002.html>