<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div class="elementToProof">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thomas Britton</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Staff Scientist in Scientific Computing</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Jefferson Lab</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
x7624<br>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Marty Wise <wise@jlab.org><br>
<b>Sent:</b> Wednesday, January 24, 2024 8:36 AM<br>
<b>To:</b> Thomas Britton <tbritton@jlab.org><br>
<b>Cc:</b> Kelvin Edwards <kelvin@jlab.org>; Paul Letta <letta@jlab.org>; Myung Bang <bangdm@jlab.org><br>
<b>Subject:</b> disk IO issues on epscidb-a</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">Thomas,</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I may have found the source of the disk IO issues I’ve seen on epscidb-a. Looking at the ganglia graphs for it over the last couple of months, it shows a large amount of IOWait (usually kind of orange/salmon colored on the graphs). This
is generally something we don’t see much if any at all, so this is definitely unusual, and could produce a variety of symptoms on the system.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Over the past several months, we have begun deploying a cyber security tool (required by the feds) called CrowdStrike/Falcon. For most linux systems, I believe it is deployed only in “monitoring” mode – i.e. it’s not configured to block
anything or take any action, just to alert if something is wrong. I noticed there were a lot of these processes running on the system. I turned it off and waited overnight to see how the disk IO/IOWait status changed.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><img width="486" height="334" id="x_Picture_x0020_1" style="width:5.0572in; height:3.4791in" data-outlook-trace="F:1|T:1" src="cid:image001.png@01DA4EA0.79F4A120"></p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">As you can see, Wait state stats dropped dramatically about the time I disabled CrowdStrike. Now.. maybe something else happened around that time, so this isn’t conclusive yet, but looks very suspicious. Are you aware of any significant
change late yesterday afternoon that might account for the difference?</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">So, I am continuing to monitor the system. Please let me know if you experience any problems or notice anything unusual.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">BTW – we are meeting with the CrowdStrike reps soon and will discuss this apparent problem with them. I should say, that it’s entirely possible that this issue is related to some misconfiguration or system peculiarity and not a problem
with CrowdStrike itself (I have not noticed similar issues elsewhere)… so hopefully we can work out the problem and re-enable CrowdStrike on the system. But, I will leave it disabled for now.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="">Marty Wise</span></p>
<p class="x_MsoNormal"><span style="">JLab CST/CNI</span></p>
<p class="x_MsoNormal"> </p>
</div>
</div>
</body>
</html>