<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In case someone has valuable data on gluonraid3 which has not been copied elsewhere - let Sergey Furletov know before Monday.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The device has to been reconfigured - see a mail from Paul Letta. Sergey will give Paul the green light.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Eugene<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Sergey Furletov <furletov@jlab.org><br>
<b>Sent:</b> Friday, April 17, 2020 10:25 AM<br>
<b>To:</b> David Lawrence <davidl@jlab.org><br>
<b>Cc:</b> Paul Letta <letta@jlab.org>; Eugene Chudakov <gen@jlab.org><br>
<b>Subject:</b> Re: Failed drive in gluonraid3</font>
<div> </div>
</div>
<div><span class="" style="font-size:14px">Hi All,</span>
<div class=""><span class="" style="font-size:14px"><br class="">
</span></div>
<div class="">
<div><span class="" style="font-size:14px">since we are going to wipe /data1 anyway, we are not in hurry to replace the disk in it, </span></div>
<div><span class="" style="font-size:14px">and because other disks are endangered I would suggest first asking colleagues to check if there is any other data on these disks. </span></div>
<div><span class="" style="font-size:14px"><br class="">
</span></div>
<div><span class="" style="font-size:14px">Sergey</span></div>
<div><br class="">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 17, 2020, at 10:11, David Lawrence <<a href="mailto:davidl@jlab.org" class="">davidl@jlab.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap:break-word; line-break:after-white-space">Hi All,
<div class=""><br class="">
</div>
<div class=""> All of the raw data has been copied to tape so there should be no danger of permanent loss.</div>
<div class="">I’ll leave the decision on re-wiring to Sergey and Eugene.</div>
<div class=""><br class="">
<div class="">
<div dir="auto" class="" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; word-wrap:break-word; line-break:after-white-space">
Regards,<br class="">
-David<br class="">
<br class="">
-------------------------------------------------------------<br class="">
David Lawrence Ph.D.<br class="">
Staff Scientist, Thomas Jefferson National Accelerator Facility<br class="">
Newport News, VA<br class="">
<a href="mailto:davidl@jlab.org" class="">davidl@jlab.org</a><br class="">
(757) 269-5567 W<br class="">
(757) 746-6697 C<br class="">
<br class="">
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 17, 2020, at 9:50 AM, Paul Letta <<a href="mailto:letta@jlab.org" class="">letta@jlab.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)">Hi All.</span></div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)">The disks on gluonraid3 were discontinued. I have a compatible replacement. However the raid card is not accepting it due so block size differences. I do believe
I can get this drive into use… but I will have to destroy and recreate the /data1 partition on gluonraid3 to do it. Also – gluonraid3 has a legacy issue that this might be a good time to correct. Recall I had to change the SAS cabling in gluonraid4/5 so
that each of the 2 raid cards would only see their own drives. (i.e. half the drives are on raid card 1, and the other half of the drives are on raid card 2). That was not done on gluonraid3. This is needed because having 2 Raid cards in the same machine
that both see all the drives could cause issues, and LSI does not recommend it. That’s why after I learned this, I rewired gluonraid4/5. I think I can preserve the files in /data2,3,4 through this.. but maybe not… So can we ensure everything on gluonraid3
is written to tape so that I could do this?</span></div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)">/data1 needs to be wiped regardless to get the replacement drive into it.</span></div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)">Oh.. gluonraid5 failed a drive yesterday as well. It’s still under warranty, so a replacement drive will be on the way for it.</span></div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)">Paul</span></div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt; font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="">
<div class="" style="border-style:solid none none; border-top-width:1pt; border-top-color:rgb(225,225,225); padding:3pt 0in 0in">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<b class=""><span class="" style="font-size:11pt; font-family:Calibri,sans-serif">From:</span></b><span class="" style="font-size:11pt; font-family:Calibri,sans-serif"><span class="Apple-converted-space"> </span>Eugene Chudakov <<a href="mailto:gen@jlab.org" class="">gen@jlab.org</a>><span class="Apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, April 3, 2020 3:50 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Paul Letta <<a href="mailto:letta@jlab.org" class="">letta@jlab.org</a>>; Sergey Furletov <<a href="mailto:furletov@jlab.org" class="">furletov@jlab.org</a>>; David Lawrence <<a href="mailto:davidl@jlab.org" class="">davidl@jlab.org</a>><br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span>Andy Kowalski <<a href="mailto:kowalski@jlab.org" class="">kowalski@jlab.org</a>>; Kelvin Edwards <<a href="mailto:kelvin@jlab.org" class="">kelvin@jlab.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: Failed drive in gluonraid3</span></div>
</div>
</div>
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
</div>
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-family:Calibri,sans-serif">Hi Paul,</span></div>
</div>
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-family:Calibri,sans-serif"> </span></div>
</div>
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-family:Calibri,sans-serif">Please use DOPS.PHALLD</span></div>
</div>
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-family:Calibri,sans-serif"> </span></div>
</div>
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-family:Calibri,sans-serif">Thanks,</span></div>
</div>
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-family:Calibri,sans-serif">Eugene</span></div>
</div>
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-family:Calibri,sans-serif"> </span></div>
</div>
<div class="MsoNormal" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif; text-align:center" align="center">
<hr class="" width="98%" size="2" align="center">
</div>
<div id="divRplyFwdMsg" class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<b class=""><span class="" style="font-size:11pt; font-family:Calibri,sans-serif">From:</span></b><span class="" style="font-size:11pt; font-family:Calibri,sans-serif"><span class="Apple-converted-space"> </span>Paul Letta <<a href="mailto:letta@jlab.org" class="" style="color:purple; text-decoration:underline">letta@jlab.org</a>><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, April 3, 2020 3:47 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Sergey Furletov <<a href="mailto:furletov@jlab.org" class="" style="color:purple; text-decoration:underline">furletov@jlab.org</a>>; David Lawrence <<a href="mailto:davidl@jlab.org" class="" style="color:purple; text-decoration:underline">davidl@jlab.org</a>>;
Eugene Chudakov <<a href="mailto:gen@jlab.org" class="" style="color:purple; text-decoration:underline">gen@jlab.org</a>><br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span>Andy Kowalski <<a href="mailto:kowalski@jlab.org" class="" style="color:purple; text-decoration:underline">kowalski@jlab.org</a>>; Kelvin Edwards <<a href="mailto:kelvin@jlab.org" class="" style="color:purple; text-decoration:underline">kelvin@jlab.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Failed drive in gluonraid3</span></div>
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="" style="margin:0in 0in 0.0001pt; font-size:12pt; font-family:"Times New Roman",serif">
<span class="" style="font-size:11pt">Gluonraid3 does have a degraded area due to a failed drive. I do not think gluonraid3 is under warranty anymore. <span class="Apple-converted-space"> </span><br class="">
I'll need to get the serial number off the chassis and call the vendor to be sure. I'll do that the next time I'm onsite, which may be 1-2 weeks.<br class="">
<br class="">
Or -- I can just order a replacement through webstock.. it’s a 8TB drive.. $377. Give me the charge code and I'll order it.<br class="">
<br class="">
It’s the /data1 area on gluonraid3. All the files have be copied to tape, so there is no risk of data loss.<br class="">
Just do not use data1 until its fixed.<br class="">
<br class="">
I also found and fixed the error that prevented me from getting emails about this from the raid card. I've checked gluonraid1-5 and make sure email alerts are working.<br class="">
<br class="">
Paul<br class="">
<br class="">
<br class="">
-----Original Message-----<br class="">
From: Sergey Furletov <<a href="mailto:furletov@jlab.org" class="" style="color:purple; text-decoration:underline">furletov@jlab.org</a>><span class="Apple-converted-space"> </span><br class="">
Sent: Friday, April 3, 2020 12:50 PM<br class="">
To: Paul Letta <<a href="mailto:letta@jlab.org" class="" style="color:purple; text-decoration:underline">letta@jlab.org</a>>; David Lawrence <<a href="mailto:davidl@jlab.org" class="" style="color:purple; text-decoration:underline">davidl@jlab.org</a>><br class="">
Cc: Eugene Chudakov <<a href="mailto:gen@jlab.org" class="" style="color:purple; text-decoration:underline">gen@jlab.org</a>><br class="">
Subject: Fwd: Re: Audible Alarm<br class="">
<br class="">
<br class="">
Hi Paul and David,<br class="">
<br class="">
gluonraid3 /data1 in degraded state:<br class="">
<br class="">
----------<br class="">
Virtual Drive: 2 (Target Id: 2)<br class="">
Name :data1<br class="">
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3<br class="">
Size : 65.491 TB<br class="">
Sector Size : 512<br class="">
Is VD emulated : Yes<br class="">
Parity Size : 7.276 TB<br class="">
State : Degraded<br class="">
Strip Size : 256 KB<br class="">
Number Of Drives : 10<br class="">
----------<br class="">
PD: 3 Information<br class="">
Enclosure Device ID: 44<br class="">
Slot Number: 4<br class="">
Drive's position: DiskGroup: 3, Span: 0, Arm: 3 Enclosure position: 2 Device Id: 12<br class="">
WWN: 5000CCA23BB79157<br class="">
Sequence Number: 5<br class="">
Media Error Count: 1<br class="">
Other Error Count: 11<br class="">
Predictive Failure Count: 0<br class="">
Last Predictive Failure Event Seq Number: 0 PD Type: SAS<br class="">
-------------------<br class="">
<br class="">
David, how can we verify that all data is copied to tapes?<br class="">
<br class="">
Sergey<br class="">
<br class="">
-------- Forwarded Message --------<br class="">
Subject: Re: Audible Alarm<br class="">
Date: Fri, 3 Apr 2020 12:12:23 -0400<br class="">
From: Timothy Whitlatch <<a href="mailto:whitey@jlab.org" class="" style="color:purple; text-decoration:underline">whitey@jlab.org</a>><br class="">
To: Hovanes Egiyan <<a href="mailto:hovanes@jlab.org" class="" style="color:purple; text-decoration:underline">hovanes@jlab.org</a>>, Sergey Furletov <<a href="mailto:furletov@jlab.org" class="" style="color:purple; text-decoration:underline">furletov@jlab.org</a>><br class="">
CC: Walt Akers <<a href="mailto:akers@jlab.org" class="" style="color:purple; text-decoration:underline">akers@jlab.org</a>>, Eugene Chudakov <<a href="mailto:gen@jlab.org" class="" style="color:purple; text-decoration:underline">gen@jlab.org</a>><br class="">
<br class="">
<br class="">
<br class="">
Thanks Walt. I am forwarding this to people that may know.<br class="">
<br class="">
Looks like Gluon Raid3<br class="">
<br class="">
Hovanes or Sergey please see link below. Anything that should be done?<br class="">
<br class="">
Thanks,<br class="">
<br class="">
Tim Whitlatch<br class="">
<br class="">
Hall D Engineer<br class="">
<br class="">
Jefferson Lab<br class="">
<br class="">
757-269-5087<br class="">
<br class="">
<br class="">
------------------------------------------------------------------------<br class="">
*From:* Walt Akers <<a href="mailto:akers@jlab.org" class="" style="color:purple; text-decoration:underline">akers@jlab.org</a>><br class="">
*Sent:* Friday, April 3, 2020 11:46 AM<br class="">
*To:* Timothy Whitlatch <<a href="mailto:whitey@jlab.org" class="" style="color:purple; text-decoration:underline">whitey@jlab.org</a>><br class="">
*Subject:* Audible Alarm<br class="">
Hey Tim:<br class="">
<br class="">
We found what appears to be a disk array alarming in the Hall D Counting House computer enclosure.<br class="">
Take a look at today's daily report and let me know if there is anything that we need to do.<br class="">
<br class="">
<a href="https://physdiv.jlab.org/2020MedConOps/ConditionReports/20200403-ConditionReport.pdf" class="" style="color:purple; text-decoration:underline">https://physdiv.jlab.org/2020MedConOps/ConditionReports/20200403-ConditionReport.pdf</a><br class="">
<br class="">
Walt</span></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>