<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=windows-1252">
</head>
<body dir="ltr">
<p>from Bryan Hess...<br>
</p>
<div class="moz-forward-container"><br>
<br>
-------- Forwarded Message --------
<table class="moz-email-headers-table" cellspacing="0"
cellpadding="0" border="0">
<tbody>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Subject:
</th>
<td>IBM Tape Library Issue</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">Date: </th>
<td>Tue, 15 Dec 2020 12:20:20 -0500</td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">From: </th>
<td>Bryan Hess <a class="moz-txt-link-rfc2396E" href="mailto:bhess@jlab.org"><bhess@jlab.org></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">To: </th>
<td>Mark Ito <a class="moz-txt-link-rfc2396E" href="mailto:marki@jlab.org"><marki@jlab.org></a>, Nathan Baltzell
<a class="moz-txt-link-rfc2396E" href="mailto:baltzell@jlab.org"><baltzell@jlab.org></a>, Brad Sawatzky
<a class="moz-txt-link-rfc2396E" href="mailto:brads@jlab.org"><brads@jlab.org></a>, Ole Hansen <a class="moz-txt-link-rfc2396E" href="mailto:ole@jlab.org"><ole@jlab.org></a>,
Robert Edwards <a class="moz-txt-link-rfc2396E" href="mailto:edwards@jlab.org"><edwards@jlab.org></a></td>
</tr>
<tr>
<th valign="BASELINE" nowrap="nowrap" align="RIGHT">CC: </th>
<td>Graham Heyes <a class="moz-txt-link-rfc2396E" href="mailto:heyes@jlab.org"><heyes@jlab.org></a>, Andy Kowalski
<a class="moz-txt-link-rfc2396E" href="mailto:kowalski@jlab.org"><kowalski@jlab.org></a>, Jie Chen <a class="moz-txt-link-rfc2396E" href="mailto:chen@jlab.org"><chen@jlab.org></a>,
David Lawrence <a class="moz-txt-link-rfc2396E" href="mailto:davidl@jlab.org"><davidl@jlab.org></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
I wanted update you about a tape library issue that we are
working on. During the last run we switched to writing LTO8
media, which gave us an increase in both capacity and
throughput. Now, reading back those tapes, we are seeing a
higher than usual I/O error rate. We have an open case with IBM,
and yesterday I escalated it for a second time, so that it is
now a "Critical Situation" in IBM's parlance. This gets us
support from the tape drive engineers at the Factory.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
The problem has been typically observed in CLAS12 and GlueX
workflows where a batch of farm jobs stall because the tape is
unavailable. Tapes are automatically disabled by Jasmine when
they have unrecoverable I/O errors, so a tape with an unreadable
segment will cause files after that position on the tape to
queue until the tape is put through a manual recovery/verify
procedure. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
Current evidence suggests that we may have received a batch of
problematic tapes. The good news is that since we use IBM media,
IBM is on the hook to fix the problem under our support
agreement, whether it is a tape or a drive problem. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
Until we have full confidence that the issue is resolved, we are
taking some measures to curb the impact:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<ul>
<li>We are initiating a verify of files that may have been
written to problematic media</li>
<li>We are falling back to writing M8 media while IBM does
analysis on the read-back errors<br>
</li>
<li>We are investigating the possibility of writing temporary
duplicates of all files as a hedge. This will require some
coding to implement.</li>
<li>IBM is replacing a significant number of the LTO8 tape
drives, based on error accounting, to rule out a bad drive
as the cause.</li>
</ul>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
I expect that the trajectory of this plan will change as we
receive more information from IBM, but I wanted to give you a
snapshot of where we are rather than merely updating service now
tickets with "we have an open case with IBM"</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
Please let me know if you have any questions. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255,
255, 255);">
--Bryan</div>
</div>
</body>
</html>