<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
<!--
html{color:#555555;}body{line-height:1.5;font-family:'Trebuchet MS','Helvetica Neue',Arial,Helvetica,sans-serif;font-size:87.5%;}h1{font-size:1.6em;}h2.field-label{display:inline-block;font-size:1em;padding-right:5px;min-width:10em;margin:0.3em;}.problem_report{line-height:1.5;max-width:60em;}fieldset.problem_report.resolved
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVQ4jWP8//8/AyWAiZACd3f3/xYWFrht+f//P1a84t3e/0obff4rbfT5D1GGXR0LuoEr3+/7X3W4n2gvwA0gVSOKAcqbfPGGpImJCU45JgYGBoa7fpsZ22wLSbadgYGBgRE9GrF55Vf2BYbHjx8zYjWB0ljAcAGGExkZ/0MtwuoCggmJEBh4AzBS4pMnT/7fuXOH4dKlSwwnT56EiwcGBv43MDBgMDExYdDX12eQkZGBhAlyiC5YsOA/AwMDUXjLli3/iYoFQgAA+pSxZrXofD0AAAAASUVORK5CYII=);background-repeat:no-repeat;padding-left:18px;}fieldset.problem_report.needs_attention
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABmJLR0QA/wD/AP+gvaeTAAAA9ElEQVR42sWTvUoDQRSFv9wMKWxSBVmzdhZJIwTWv9pyLWxTpbE1kBeJPoLxBazzBgGFKNqlHXAhsITUw1y7sMpmjER0YJrDPWcO556BPzhjYPJjlhfT82LUpxcK6Lo5U0JMgcu56tXy7BQajeBDpkAcAEdz1W6uyrLdYieK2DMmKCArcqczpH/ddc0msy+OkyQJC4h3N0ynx+Q5ALtUNs5q5U+8e+R+VPFi4kjk3NZqd++qUK+TZVnYwSfAOyvejeLXt/2qVG/dYoG1dqseBNco27bs/wXKWhIDB8AhcFLAH4Bn4Al4AUqT7RVC++6mv/JVPwDi3VGzomYvyAAAAABJRU5ErkJggg==);background-repeat:no-repeat;padding-left:18px;}.problem_report div.field-items{display:inline-block;}div.date-vitals p{font-size:87.5%;}a{text-decoration:none;}.Readme a:link,.Readme a:visited,.Readme
a:active{color:red;}
-->
</style>
</head>
<body id="mimemail-body" class="elog-logentry-notify">
<div id="center">
<div id="main">
<style>
<!--/*--><![CDATA[/* ><!--*/
div.field-vitals{
margin: 0.5em 0;
}
div.field-vitals .field-type-taxonomy-term-reference {
margin: 0.1em 0;
}
article.comment {
padding-left: 10px;
}
article.comment.odd {
background-color: #EEEEEE;
}
article.comment.even {
background-color: #DDDDDD;
}
div.node-content.logentry table{
width: auto;
border-collapse: collapse;
border-spacing: 0;
border-width: 1px;
}
div.node-content.logentry th{
border: inherit;
}
div.node-content.logentry blockquote{
background-color: #FFFFFF;
}
div.node-content.logentry caption{
font-size: 1em;
font-weight: normal;
}
table.field-vitals{
margin-top: 1em;
margin-bottom: 1em;
font-size: 87.5%;
}
table.field-vitals th{
vertical-align: middle;
text-align: left;
width: 15%;
padding: 0.1em;
}
table.field-vitals td{
vertical-align: middle;
text-align: left;
width: auto;
padding: 0.1em;
}
table.field-vitals td li {
margin-left: 0;
list-style-type: none;
list-style-image: none;
}
table.downtime {
width: 30em;
margin-bottom: 1em;
border: 1px black dotted;
}
table.downtime th {
text-align: center;
}
table.downtime td {
text-align: center;
}
tr.caption th {
border-bottom: none;
}
table.downtime tfoot{
background-color:#EEEEEE;
}
div.field-name-body{
margin: 1em 0;
font-size: 110%;
}
div.date-vitals p{
margin: .1em 0;
}
article div.ctools-collapsible-container{
margin-left: -5px;
clear: both;
}
#comment-form{
margin-left: 5px;
border: graytext outset medium;
-moz-border-radius: 15px;
border-radius: 15px;
padding: 1em;
}
div.comments-form-box {
margin-top: 2em;
margin-bottom: 5em;
}
h3.comment-title {
/* display: none; */
}
p.author-datetime{
font-weight: bold;
}
/*--><!]]>*/
</style><article id="node-523173" class="node node-logentry article ia-n clearfix" role="article"><header class="node-header"><h1 class="node-title">
<a href="https://logbooks.jlab.org/entry/3450335" rel="bookmark">Follow-up Re: Follow-up Re: Corrupted data</a>
</h1>
</header><div class="date-vitals">
<p class="author-datetime">
Lognumber <a href="https://logbooks.jlab.org/entry/3450335" class="lognumber" data-lognumber="3450335">3450335</a>. Submitted by <a href="https://logbooks.jlab.org/user/davidl">davidl</a> on <time datetime="2017-01-12T14:51:52-0500" pubdate="pubdate">Thu, 01/12/2017 - 14:51</time>. </p>
<table class="field-vitals"><tr><th>Logbooks: </th><td><a href="https://logbooks.jlab.org/book/hdlog">HDLOG</a></td></tr><tr><th>References: </th><td><a href="https://logbooks.jlab.org/entry/3449814">3449814 - Follow-up Re: Corrupted data</a></td></tr></table></div>
<div class="logentry node-content">
<p>Upon further investigation by Sergey, he found that the issue was due to the RAID0 configuration used for partition 3 and partition 4. As a reminder we had:</p>
<p>/gluonraid3/data1 zfs<br />
/gluonraid3/data2 zfs<br />
/gluonraid3/data3 software RAID0 + xfs<br />
/gluonraid3/data4 software RAID0 + xfs</p>
<p>The configuration of the RAID partitions used disk names like "/dev/sdaa", "/dev/sdbc", ... It turns out that these names are not always assigned to the same physical disk when the system is rebooted. There exists a directory /dev/disk/by-vdev that contains symbolic links with names like "slot1", "slot2", ... that point to the correct device file assigned to the disk in that physical slot at boot time. When we configured the RAID partitions we used the "/dev/sdXX" names (archived from a previous boot) instead of the "/dev/disk/by-vdev/slotX" names. This caused both zfs and xfs to try using the same physical disk, leading to the corruption.</p>
<p>This was discovered after rebooting gluonraid3 today which again shuffled the device names. As a consequence, the data on both partition 3 and partition 4 is now compromised (and probably on the first 2 zfs partitions). No data was lost since everything important was copied to tape back in December. However, it means that the Fall 2016 data is no longer available in the counting house.</p>
</div>
<div class="attachment-box">
</div>
</article> </div>
</div>
</body>
</html>