<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The previously reported Lustre issue has been resolved.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> Jlab-scicomp-briefs <jlab-scicomp-briefs-bounces@jlab.org> on behalf of Bryan Hess <bhess@jlab.org><br>
<b>Sent:</b> Sunday, February 6, 2022 6:31 AM<br>
<b>To:</b> jlab-scicomp-briefs@jlab.org <jlab-scicomp-briefs@jlab.org><br>
<b>Subject:</b> [Jlab-scicomp-briefs] Farm Lustre outage for /cache and /volatile - Sun Feb 6, 2022</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The /cache and /volatile Lustre metadata server for the farm rebooted unexpectedly Saturday night. The filesystem recovery process, normally quick, began last night and continues with an estimated completion time later this morning. <span style="margin: 0px; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">We
are monitoring the progress and will have the system in service as soon as possible.</span><br>
<span style="margin: 0px; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Until the filesystem recovery completes file I/O to /cache and /volatile will hang. <span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">Thank you for your patience.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Lustre usage tips for /cache and /volatile:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<ul>
<li><span>Avoid small or unbuffered writes to Lustre when possible. </span></li><li><span>Write log files to /farm_out when possible, which is tuned for high IOPS/small files. <br>
</span></li><li><span>Avoid file metadata operations in tight loops, such as hundreds of open() and close() operations per second.</span></li><li><span>Lustre perform best when reading or writing files using large buffers. </span></li><li>Per-User Farm Job IOPS are shown here <a href="https://scicomp.jlab.org/scicomp/slurmJob/jobLimit" id="LPlnk838643">https://scicomp.jlab.org/scicomp/slurmJob/jobLimit</a></li><li><span>FarmJobs with an I/O mix that is challenging for Lustre are shown <a href="https://scicomp.jlab.org/scicomp/slurmJob/jobLimit" id="LPlnk211024">https://scicomp.jlab.org/scicomp/slurmJob/jobLimit</a> </span></li></ul>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
</div>
</body>
</html>