<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
<br id="lineBreakAtBeginningOfSignature">
<div dir="ltr">Thomas Britton</div>
<div dir="ltr"><br>
Begin forwarded message:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr"><b>From:</b> Marty Wise <wise@jlab.org><br>
<b>Date:</b> January 24, 2024 at 8:36:56 AM EST<br>
<b>To:</b> Thomas Britton <tbritton@jlab.org><br>
<b>Cc:</b> Kelvin Edwards <kelvin@jlab.org>, Paul Letta <letta@jlab.org>, Myung Bang <bangdm@jlab.org><br>
<b>Subject:</b> <b>disk IO issues on epscidb-a</b><br>
<br>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style>@font-face { font-family: "Cambria Math"; }
@font-face { font-family: Calibri; }
p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; }
span.EmailStyle17 { font-family: Calibri, sans-serif; color: windowtext; }
.MsoChpDefault { font-family: Calibri, sans-serif; }
@page WordSection1 { size: 8.5in 11in; margin: 1in; }
div.WordSection1 { page: WordSection1; }</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Thomas,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I may have found the source of the disk IO issues I’ve seen on epscidb-a. Looking at the ganglia graphs for it over the last couple of months, it shows a large amount of IOWait (usually kind of orange/salmon colored on the graphs). This
is generally something we don’t see much if any at all, so this is definitely unusual, and could produce a variety of symptoms on the system.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Over the past several months, we have begun deploying a cyber security tool (required by the feds) called CrowdStrike/Falcon. For most linux systems, I believe it is deployed only in “monitoring” mode – i.e. it’s not configured to block
anything or take any action, just to alert if something is wrong. I noticed there were a lot of these processes running on the system. I turned it off and waited overnight to see how the disk IO/IOWait status changed.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><img alt="image001.png" src="cid:image001.png@01DA4EA0.79F4A120"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As you can see, Wait state stats dropped dramatically about the time I disabled CrowdStrike. Now.. maybe something else happened around that time, so this isn’t conclusive yet, but looks very suspicious. Are you aware of any significant
change late yesterday afternoon that might account for the difference?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So, I am continuing to monitor the system. Please let me know if you experience any problems or notice anything unusual.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">BTW – we are meeting with the CrowdStrike reps soon and will discuss this apparent problem with them. I should say, that it’s entirely possible that this issue is related to some misconfiguration or system peculiarity and not a problem
with CrowdStrike itself (I have not noticed similar issues elsewhere)… so hopefully we can work out the problem and re-enable CrowdStrike on the system. But, I will leave it disabled for now.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="mso-ligatures:none">Marty Wise<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-ligatures:none">JLab CST/CNI<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</body>
</html>