<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Folks,<br>
</p>
<p>No objections were lodged. The switch has been made (a few
minutes early). The change should be transparent to
default-build_scripts users, so let me know if you see any
problems.</p>
<p> -- Mark<br>
</p>
<div class="moz-cite-prefix">On 4/29/19 11:57 AM, Mark Ito wrote:<br>
</div>
<blockquote type="cite"
cite="mid:f8af0a1b-7104-a1ca-e249-6b1c4f2eb425@jlab.org">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Folks,</p>
<p>You will recall that we switched the default choice for the
database engine for CCDB at JLab from SQLite to MySQL back on
March 12 (<a class="moz-txt-link-freetext"
href="https://mailman.jlab.org/pipermail/halld-offline/2019-March/003561.html"
moz-do-not-send="true">https://mailman.jlab.org/pipermail/halld-offline/2019-March/003561.html</a>).
Since then we have been studying usage patterns and CCDB
performance on MySQL. And, not unexpectedly, we have experienced
"storms" where the MySQL server is overloaded with CCDB requests
from farm jobs and performance is severely degraded. Jobs don't
crash but they take a long time to read in constants before
starting to analyze events, up to two hours in the worst cases.
These have been occurring once every day or two and last for a
few hours.<br>
<br>
We have learned a lot. The storms are definitely coming from
farm jobs when many of them start at the same time. The queries
have been analyzed (no big inefficiencies found). These things
were invisible to us when we were running with the SQLite
default. As a result IT is putting up a new beefier server for
us to use for the farm.<br>
<br>
Until the new server comes online (a week or two), <b>I am
proposing that we switch back to using SQLite as the default
for CCDB</b>, but with two differences from how we used SQLite
at JLab in the past.<br>
<br>
(1) SQLite will be used <b>for the farm only</b>. Interactive
users at JLab will continue to use the MySQL server.<br>
(2) Rather than accessing the ccdb.sqlite file on the group
disk, farm jobs will access one of 100 copies of ccdb.sqlite,
located on the work disk, chosen randomly for each job.<br>
<br>
The idea is to separate farm demand for CCDB constants from
other functions of our main database server while we wait for
the new server that will do just that. Other functions include
the writing of calibration constants to the CCDB, RCDB access,
and MCwrapper job control functions.<br>
<br>
Note again that this change only affects processes using the
defaults from the build_scripts-based environment set-up. Those
that set their own JANA_CALIB_URL environment variable
explicitly will get what they ask for, as always.<br>
<br>
<b>The proposed switch-over time is COB today.</b> I will make
the change unless I hear objections.<br>
</p>
<p> -- Mark<br>
</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Halld-offline mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Halld-offline@jlab.org">Halld-offline@jlab.org</a>
<a class="moz-txt-link-freetext" href="https://mailman.jlab.org/mailman/listinfo/halld-offline">https://mailman.jlab.org/mailman/listinfo/halld-offline</a></pre>
</blockquote>
</body>
</html>