<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=windows-1252">
</head>
<body dir="ltr">
<p>News from Bryan Hess, FYI. He addresses interactive response on
the ifarms and the situation with tape file recovery.<br>
</p>
<div class="moz-forward-container"><br>
<br>
-------- Forwarded Message --------
<table class="moz-email-headers-table" cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Subject:
</th>
<td>An update on ongoing issues</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Date: </th>
<td>Fri, 12 Feb 2021 16:07:12 -0500</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">From: </th>
<td>Bryan Hess <a class="moz-txt-link-rfc2396E" href="mailto:bhess@jlab.org"><bhess@jlab.org></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">To: </th>
<td>Brad Sawatzky <a class="moz-txt-link-rfc2396E" href="mailto:brads@jlab.org"><brads@jlab.org></a>, Ole Hansen
<a class="moz-txt-link-rfc2396E" href="mailto:ole@jlab.org"><ole@jlab.org></a>, Nathan Baltzell
<a class="moz-txt-link-rfc2396E" href="mailto:baltzell@jlab.org"><baltzell@jlab.org></a>, Mark Ito <a class="moz-txt-link-rfc2396E" href="mailto:marki@jlab.org"><marki@jlab.org></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">CC: </th>
<td>Graham Heyes <a class="moz-txt-link-rfc2396E" href="mailto:heyes@jlab.org"><heyes@jlab.org></a>, Andy Kowalski
<a class="moz-txt-link-rfc2396E" href="mailto:kowalski@jlab.org"><kowalski@jlab.org></a>, David Lawrence
<a class="moz-txt-link-rfc2396E" href="mailto:davidl@jlab.org"><davidl@jlab.org></a>, Jie Chen <a class="moz-txt-link-rfc2396E" href="mailto:chen@jlab.org"><chen@jlab.org></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt;">I owe you updates on a several topics after
our meeting last week.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt;">Interactive Response:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt;">We've been looking into reports of
intermittent slow interactive response on the ifarm/farm
networks . We are making three change that address aspects of
this:</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<ol>
<li>The scientific computing firewall will be replaced Tuesday
as part of scheduled monthly maintenance.
<span style="color: rgb(0, 0, 0); background-color: rgb(255,
255, 255); display: inline !important;">
The switchover is quick and will increase bandwidth to
from 1 to 10Gigabit/sec for firewalled connections.<span> I
will get a full maintenance reminder out soon.</span></span></li>
<li>NIS and LDAP lookups have been intermittently slow. We are
implementing lookup caching on ifarm systems as a mitigation
while we look for the root cause. </li>
<li>There is a procurement in progress for a new pair of work
file servers dedicated to ENP (no more shared failover with
LQCD). Once we take delivery, we will configure some more
modern filesystem features including a better SSD caching
layer.</li>
</ol>
<div><br>
</div>
<div><br>
</div>
<div>Tape Library News:</div>
<div><br>
</div>
<div><br>
</div>
<div>We met with IBM about the corroded tape drive problem again
today. There were no unexpected findings from the chemical
analysis of the head corrosion, which rules out some causes.
Electron microscopy shows severe corrosion on only one drive
so far. Two more drives are being disassembled for teh same
analysis.</div>
<div><br>
</div>
<div>The working theory is that that the tape errors stem from
one bad drive and its corrosion was exacerbated by
higher-than-normal humidity (although it was in spec). As a
result, we are installing additional humidity monitors in the
library and test coupons to check for corrosive agents. We are
working with facilities management to decrease peak humidity
in the data center, aiming for 30% - 50% as the operating
range.</div>
<div><br>
</div>
<div>It is possible that we are past this problem, but out of
caution we will continue to write duplicate tapes of all new
data for the interim. We will continue to do a verify pass on
all data during the period when we believe the bad tape drive
was in service. These two tasks are placing additional load on
the tape library, which will increase service time for a
while. </div>
<div><br>
</div>
<div>Our next steps will depend on what we find from IBM's next
drive analysis.</div>
<div><br>
</div>
<div>Recovery of the initial 18 volumes continues. IBM needs ~96
hours per tape recovery because of the block-by-block process
between a pair of drives. </div>
<div><br>
</div>
<div>That's the news for Friday. Have a good weekend. </div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</body>
</html>