<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hi Richard -<div><br></div><div> thanks for the update on the OSG. That is very interesting on what happens when you went above 10k cores.</div><div><br></div><div> Curtis<br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><div>---------</div><div>Curtis A. Meyer<span class="Apple-tab-span" style="white-space: pre; "> </span>MCS Associate Dean for Faculty and Graduate Affairs</div><div>Wean: (412) 268-2745<span class="Apple-tab-span" style="white-space: pre; "> </span>Professor of Physics</div><div>Doherty: (412) 268-3090<span class="Apple-tab-span" style="white-space: pre; "> </span>Carnegie Mellon University</div><div>Fax: (412) 681-0648<span class="Apple-tab-span" style="white-space: pre; "> </span>Pittsburgh, PA 15213</div><div><a href="mailto:curtis.meyer@cmu.edu">curtis.meyer@cmu.edu</a><span class="Apple-tab-span" style="white-space: pre; "> </span>http://www.curtismeyer.com/</div></div><div><br></div></span><br class="Apple-interchange-newline">
</div>
<br><div><div>On Apr 7, 2014, at 8:27 AM, Richard Jones <<a href="mailto:richard.t.jones@uconn.edu">richard.t.jones@uconn.edu</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Hello dc2 crew,<div><br></div><div>As you can see from the graph of running processes,</div><div><br></div><div><a href="http://gryphn.phys.uconn.edu/vofrontend/monitor/frontendStatus.html">http://gryphn.phys.uconn.edu/vofrontend/monitor/frontendStatus.html</a><br>
</div><div><br></div><div>we hit an instability in the OSG production at around 11,000 cores. To get the time on the graph right, you need to select the "-4 hr" timezone from the menu at the bottom of the page.</div>
<div><br></div><div>We had been running on Saturday around 10k cores for a few hours and everything seemed ok, so I decided to bump up our maximum cores request to 12k. I did this around 6:00pm. Immediately we started getting more cores, which looked great. But as soon as the running process count hit around 10.5k I started seeing big fluctuations in the swap rate on the submit host. By the time we hit 11k running processes, my submit host had reached >50% of cpu time spent swapping, which quickly led to a runaway situation where running processes were queuing up for attention, leading to more swapping, etc. Within 10 minutes the submit host had reached a cpu load of 4200 processes, and 99.7% cpu time spent swapping. If the submit host goes away, the jobs automatically exit after some timeout currently around 10 minutes, so within 15 minutes our production had cleared out.</div>
<div><br></div><div>I restarted production with a cap at 9k cores just to be conservative, and after that ran smoothly for 24 hours, I have increased it to 10k cores. I plan to run with this throttle in place until a new shipment of ram for my submit host arrives later on this week.</div>
<div><br></div><div>This is why we do these exercises, to find out where the critical points are and resolve them, right?</div><div><br></div><div>-Richard Jones</div></div>
_______________________________________________<br>Halld-offline mailing list<br><a href="mailto:Halld-offline@jlab.org">Halld-offline@jlab.org</a><br>https://mailman.jlab.org/mailman/listinfo/halld-offline</blockquote></div><br></div></body></html>