<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Guys,</div><div class=""><br class=""></div><div class=""> In the past we have had problems with the servers getting hit really hard when lots of farm jobs</div><div class="">try accessing them at once. Pointing RCDB_CONNECTION to a JLab server is OK for now if there</div><div class="">are only a few jobs using it, but I think we are going to need a solution where the RCDB </div><div class="">(or relevant parts of it) are packaged in the container itself.</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">-David</div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 17, 2017, at 3:58 PM, Richard Jones <<a href="mailto:richard.t.jones@uconn.edu" class="">richard.t.jones@uconn.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Sean,<div class=""><ul class=""><li class="">Thanks a lot, this sounds perfect. I presume the http_proxy changes were made in the container, since the template script wasn't updated?<br class=""></li></ul><div style="font-size:12.8px" class="">Yes, in the container.</div><ul class=""><li class="">One more comment: it may be useful to have a default setting for the connection to the RCDB, e.g. RCDB_CONNECTION=mysql:<a href="mailto://rcdb@hallddb.jlab.org/rcdb" target="_blank" style="font-size:12.8px" class="">//rcdb@<wbr class="">hallddb.jlab.org/rcdb</a><br class=""></li></ul>Yes, I agree. I have added this. It might take another 30 minutes before the update appears in /cvmfs.</div><div class=""><br class=""></div><div class="">-Richard J.</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Jul 17, 2017 at 3:23 PM, Sean Dobbs <span dir="ltr" class=""><<a href="mailto:s-dobbs@northwestern.edu" target="_blank" class="">s-dobbs@northwestern.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Richard,<div class=""><br class=""></div><div class="">Thanks a lot, this sounds perfect. I presume the http_proxy changes were made in the container, since the template script wasn't updated?</div><div class="">One more comment: it may be useful to have a default setting for the connection to the RCDB, e.g. RCDB_CONNECTION=mysql:<a href="mailto://rcdb@hallddb.jlab.org/rcdb" target="_blank" class="">//rcdb@<wbr class="">hallddb.jlab.org/rcdb</a></div><span class="HOEnZb"><font color="#888888" class=""><div class=""><br class=""></div></font></span><div class=""><span class="HOEnZb"><font color="#888888" class="">---Sean<br class=""><br class=""><br class=""></font></span><div class="gmail_quote"><span class=""><div dir="ltr" class="">On Mon, Jul 17, 2017 at 1:57 PM Richard Jones <<a href="mailto:richard.t.jones@uconn.edu" target="_blank" class="">richard.t.jones@uconn.edu</a>> wrote:<br class=""></div></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<div class="">
<div dir="ltr" class="">Sean,
<div class=""><span style="font-size:12.8px" class=""><br class="">
</span></div>
<ul class="">
<li class="">[ability to download the latest ccdb.sqlite file from <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__halldweb.jlab.org_dist&d=DwMFaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=yDkkICgQySpvpZthngnDLMXHfbHiYtlR9GwTTeg0H2U&s=EXwR_dhU_P5byRiLiITjm2p7KAbl57HJimO5BnoaUpw&e=" target="_blank" class="">halldweb.jlab.org/dist</a> <wbr class="">and
use it, instead of copying from a standard location on cvmfs, where it is likely to be stale] This is helpful, but is there no good way to automatically update the SQLite file on OASIS (perhaps weekly)? I worry that when we do a large scale production, having
every job copying from halldweb would place too much of a load on the server, so that we would need to do some server-side caching/load balancing.<br class="">
</li></ul>
<div class="">For things that are expected to be updated on a regular schedule, osg users should use the standard OSG_SQUID_LOCATION variable so that multiple jobs running on a site share a single copy. The cvmfs cache is per-host, whereas the squid cache is per-site,
so this is much more efficient in terms of occupation of the site cache space for something like the sqlite file. I had forgotten to translate the OSG_SQUID_LOCATION variable into environment variables http_proxy, https_proxy, and this was why you were seeing
repeated hits on the jlab server for each job. This should be fixed now. The site squid cache will be invalidated on each site once per day, each time the sqlite file is updated, so there should be a max of one download from
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__halldweb.jlab.org&d=DwMFaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=yDkkICgQySpvpZthngnDLMXHfbHiYtlR9GwTTeg0H2U&s=hhaIasIMSKCtlNwozKQWuSJ86DpR-UcS05HgpeE3PFY&e=" target="_blank" class="">
halldweb.jlab.org</a> per site per day. Is this going to be ok?</div>
<div class="">
<div class="">
<ul style="font-size:12.8px" class="">
<li class="">[ability to use non-sequential run lists for simulating a series of runs, instead of assuming that you want to fill in a sequential range with every run number; if you supply a plain text file called runsequence.txt with your job then this file will be
used to look up the run sequence instead of assuming a contiguous sequence.] Could you please add the attached file into some standard location? I think it would be useful for users to have standard run lists stored somewhere in the CVMFS namespace.</li></ul>
<span style="font-size:12.8px" class="">Ok, this is done. The file you shared with me has been uploaded to /cvmfs/<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__oasis.opensciencegrid.org_gluex_templates&d=DwMFaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=yDkkICgQySpvpZthngnDLMXHfbHiYtlR9GwTTeg0H2U&s=IDqQxjmiWD_xUBymCJtf1vC2TdntFY8GYoGdovAtCms&e=" target="_blank" class="">oasis.opensciencegrid.<wbr class="">org/gluex/templates</a></span></div>
<div class="">
<ul class="">
<li class="">[<span style="font-size:12.8px" class="">You also asked if we could have more than one build of sim-recon available simultaneously on the oasis filesystem. That is already a feature of the system I set up, so no updates were required to provide this.] </span><span style="font-size:12.8px" class="">That
great to hear. When do you think you can provide a copy of sim-recon branch recon-2017_01-ver01-</span><span style="font-size:12.8px" class="">bat<wbr class="">ch01-mcsmear?</span></li></ul>
</div>
</div>
<span style="font-size:12.8px" class="">This is available now. Just do "source $GLUEX_TOP/.hdpm/env/</span><span style="font-size:12.8px" class="">recon-<wbr class="">2017_01-ver01-</span><span style="font-size:12.8px" class="">batch01-mcsmear.<wbr class="">sh" at the beginning of
the job script, and the binaries and libraries in your path variables will come from the branch build.</span></div></div><div class=""><div dir="ltr" class="">
<div class=""><span style="font-size:12.8px" class=""><br class="">
</span></div>
<div class=""><span style="font-size:12.8px" class="">-Richard Jones<br class="">
</span>
<div class="">
<div style="font-size:12.8px" class=""><br class="">
</div>
</div>
</div>
</div></div></span><div class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote"><span class="">On Sun, Jul 16, 2017 at 7:13 PM, Sean Dobbs <span dir="ltr" class="">
<<a href="mailto:s-dobbs@northwestern.edu" target="_blank" class="">s-dobbs@northwestern.edu</a>></span> wrote:<br class="">
</span><div class=""><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">Richard,
<div class=""><br class="">
</div>
<div class="">Thanks for adding these features. I have some comments below:<br class="">
<br class="">
<div class="gmail_quote"><span class="">
<div dir="ltr" class="">On Sat, Jul 15, 2017 at 6:00 AM Richard Jones <<a href="mailto:richard.t.jones@uconn.edu" target="_blank" class="">richard.t.jones@uconn.edu</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class=""><br class="">
<div class="">
<ol class="">
<li class="">ability to download the latest ccdb.sqlite file from <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__halldweb.jlab.org_dist&d=DwMFaQ&c=yHlS04HhBraes5BQ9ueu5zKhE7rtNXt_d012z2PA6ws&r=y4ZD58I4nPR6tZqjerSGt-WlWVAhqa3FHDMXqQ_5aUc&m=yDkkICgQySpvpZthngnDLMXHfbHiYtlR9GwTTeg0H2U&s=EXwR_dhU_P5byRiLiITjm2p7KAbl57HJimO5BnoaUpw&e=" target="_blank" class="">
halldweb.jlab.org/dist</a> and use it, instead of copying from a standard location on cvmfs, where it is likely to be stale.</li></ol>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">This is helpful, but is there no good way to automatically update the SQLite file on OASIS (perhaps weekly)? I worry that when we do a large scale production, having every job copying from halldweb would place too much of a load on the server, so that
we would need to do some server-side caching/load balancing.</div>
<div class=""><br class="">
</div>
<div class="">For what it's worth, in some ~50 test job runs during the week, I noticed that sometimes wget'ing the SQLite file took ~30 minutes (though apparently it's faster during the weekend!)</div>
<span class="">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">
<ol class="">
<li class="">ability to use non-sequential run lists for simulating a series of runs, instead of assuming that you want to fill in a sequential range with every run number; if you supply a plain text file called runsequence.txt with your job then this file will be used
to look up the run sequence instead of assuming a contiguous sequence.</li></ol>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">Could you please add the attached file into some standard location? I think it would be useful for users to have standard run lists stored somewhere in the CVMFS namespace.</div>
<span class="">
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">You also asked if we could have more than one build of sim-recon available simultaneously on the oasis filesystem. That is already a feature of the system I set up, so no updates were required to provide this.</div>
</div>
<div dir="ltr" class="">
<div class=""><br class="">
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</span>
<div class="">That great to hear. When do you think you can provide a copy of sim-recon branch recon-2017_01-ver01-<wbr class="">batch01-mcsmear?</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">----Sean</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class=""></div>
<div class="">-Richard Jones</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div></div></div>
<br class="">
</div>
</div></blockquote></div></div></div>
</blockquote></div><br class=""></div>
_______________________________________________<br class="">Halld-offline mailing list<br class=""><a href="mailto:Halld-offline@jlab.org" class="">Halld-offline@jlab.org</a><br class="">https://mailman.jlab.org/mailman/listinfo/halld-offline</div></blockquote></div><br class=""></body></html>