<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The /cache and /volatile Lustre metadata server for the farm rebooted unexpectedly Saturday night. The filesystem recovery process, normally quick, began last night and continues with an estimated completion time later this morning. <span style="margin: 0px; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">We
are monitoring the progress and will have the system in service as soon as possible.</span><br>
<span style="margin: 0px; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Until the filesystem recovery completes file I/O to /cache and /volatile will hang. <span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Thank you for your patience.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Lustre usage tips for /cache and /volatile:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<ul>
<li><span>Avoid small or unbuffered writes to Lustre when possible. </span></li><li><span>Write log files to /farm_out when possible, which is tuned for high IOPS/small files. <br>
</span></li><li><span>Avoid file metadata operations in tight loops, such as hundreds of open() and close() operations per second.</span></li><li><span>Lustre perform best when reading or writing files using large buffers. </span></li><li>Per-User Farm Job IOPS are shown here <a href="https://scicomp.jlab.org/scicomp/slurmJob/jobLimit" id="LPlnk838643">https://scicomp.jlab.org/scicomp/slurmJob/jobLimit</a></li><li><span>FarmJobs with an I/O mix that is challenging for Lustre are shown <a href="https://scicomp.jlab.org/scicomp/slurmJob/jobLimit" id="LPlnk211024">https://scicomp.jlab.org/scicomp/slurmJob/jobLimit</a> </span></li></ul>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
</body>
</html>