<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
<!--
html{color:#555555;}body{line-height:1.5;font-family:'Trebuchet MS','Helvetica Neue',Arial,Helvetica,sans-serif;font-size:87.5%;}h1{font-size:1.6em;}h2.field-label{display:inline-block;font-size:1em;padding-right:5px;min-width:10em;margin:0.3em;}.problem_report{line-height:1.5;max-width:60em;}fieldset.problem_report.resolved
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVQ4jWP8//8/AyWAiZACd3f3/xYWFrht+f//P1a84t3e/0obff4rbfT5D1GGXR0LuoEr3+/7X3W4n2gvwA0gVSOKAcqbfPGGpImJCU45JgYGBoa7fpsZ22wLSbadgYGBgRE9GrF55Vf2BYbHjx8zYjWB0ljAcAGGExkZ/0MtwuoCggmJEBh4AzBS4pMnT/7fuXOH4dKlSwwnT56EiwcGBv43MDBgMDExYdDX12eQkZGBhAlyiC5YsOA/AwMDUXjLli3/iYoFQgAA+pSxZrXofD0AAAAASUVORK5CYII=);background-repeat:no-repeat;padding-left:18px;}fieldset.problem_report.needs_attention
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABmJLR0QA/wD/AP+gvaeTAAAA9ElEQVR42sWTvUoDQRSFv9wMKWxSBVmzdhZJIwTWv9pyLWxTpbE1kBeJPoLxBazzBgGFKNqlHXAhsITUw1y7sMpmjER0YJrDPWcO556BPzhjYPJjlhfT82LUpxcK6Lo5U0JMgcu56tXy7BQajeBDpkAcAEdz1W6uyrLdYieK2DMmKCArcqczpH/ddc0msy+OkyQJC4h3N0ynx+Q5ALtUNs5q5U+8e+R+VPFi4kjk3NZqd++qUK+TZVnYwSfAOyvejeLXt/2qVG/dYoG1dqseBNco27bs/wXKWhIDB8AhcFLAH4Bn4Al4AUqT7RVC++6mv/JVPwDi3VGzomYvyAAAAABJRU5ErkJggg==);background-repeat:no-repeat;padding-left:18px;}.problem_report div.field-items{display:inline-block;}div.date-vitals p{font-size:87.5%;}a{text-decoration:none;}.Readme a:link,.Readme a:visited,.Readme
a:active{color:red;}
-->
</style>
</head>
<body id="mimemail-body" class="elog-logentry-notify">
<div id="center">
<div id="main">
<style>
<!--/*--><![CDATA[/* ><!--*/
div.field-vitals{
margin: 0.5em 0;
}
div.field-vitals .field-type-taxonomy-term-reference {
margin: 0.1em 0;
}
article.comment {
padding-left: 10px;
}
article.comment.odd {
background-color: #EEEEEE;
}
article.comment.even {
background-color: #DDDDDD;
}
div.node-content.logentry table{
width: auto;
border-collapse: collapse;
border-spacing: 0;
border-width: 1px;
}
div.node-content.logentry th{
border: inherit;
}
div.node-content.logentry blockquote{
background-color: #FFFFFF;
}
div.node-content.logentry caption{
font-size: 1em;
font-weight: normal;
}
table.field-vitals{
margin-top: 1em;
margin-bottom: 1em;
font-size: 87.5%;
}
table.field-vitals th{
vertical-align: middle;
text-align: left;
width: 15%;
padding: 0.1em;
}
table.field-vitals td{
vertical-align: middle;
text-align: left;
width: auto;
padding: 0.1em;
}
table.field-vitals td li {
margin-left: 0;
list-style-type: none;
list-style-image: none;
}
table.downtime {
width: 30em;
margin-bottom: 1em;
border: 1px black dotted;
}
table.downtime th {
text-align: center;
}
table.downtime td {
text-align: center;
}
tr.caption th {
border-bottom: none;
}
table.downtime tfoot{
background-color:#EEEEEE;
}
div.field-name-body{
margin: 1em 0;
font-size: 110%;
}
div.date-vitals p{
margin: .1em 0;
}
article div.ctools-collapsible-container{
margin-left: -5px;
clear: both;
}
#comment-form{
margin-left: 5px;
border: graytext outset medium;
-moz-border-radius: 15px;
border-radius: 15px;
padding: 1em;
}
div.comments-form-box {
margin-top: 2em;
margin-bottom: 5em;
}
h3.comment-title {
/* display: none; */
}
p.author-datetime{
font-weight: bold;
}
/*--><!]]>*/
</style><article id="node-420478" class="node node-logentry article ia-n clearfix" role="article"><header class="node-header"><h1 class="node-title">
<a href="https://logbooks.jlab.org/entry/3347682" rel="bookmark">"maximum number of clients reached" problem solved (hopefully)</a>
</h1>
</header><div class="date-vitals">
<p class="author-datetime">
Lognumber <a href="https://logbooks.jlab.org/entry/3347682" class="lognumber" data-lognumber="3347682">3347682</a>. Submitted by <a href="https://logbooks.jlab.org/user/davidl">davidl</a> on <time datetime="2015-08-06T09:34:48-0400" pubdate="pubdate">Thu, 08/06/2015 - 09:34</time>. </p>
<table class="field-vitals"><tr><th>Logbooks: </th><td><a href="https://logbooks.jlab.org/book/hdlog">HDLOG</a></td></tr><tr><th>Entry Makers: </th><td>davidl, furletov</td></tr></table></div>
<div class="logentry node-content">
<p>For quite some time there has been a problem when running the DAQ system from the Hall-D counting house where we would eventually get errors reporting "maximum number of clients reached". When this happened, new windows would not open and the DAQ system could not be run until the user logged out and then back in. We believe we have finally found the cause and a solution for this. The problem stems from the Xorg server process maintaining open file descriptors to the file:</p>
<p>/etc/ati/atiapfuser.blb</p>
<p>There is a hardcoded limit of 255 in the Xorg server for the number of open file descriptors it is able to have. If this is exceeded, then the "maximum number of clients reached" error is reported. One can see a large number of open file descriptors being used for the above file by the Xorg server whenever the DAQ system is run. One needs root privileges to run this, but the command to list open file descriptors for a process is: </p>
<p>> sudo /usr/sbin/lsof -p `/sbin/pidof Xorg` | sort -n -k4</p>
<p>An example of some of the lines output of this is:</p>
<p>Xorg 32072 root 191r REG 8,3 0 17696773 /etc/ati/atiapfuser.blb<br />
Xorg 32072 root 192r REG 8,3 0 17696773 /etc/ati/atiapfuser.blb<br />
Xorg 32072 root 193r REG 8,3 0 17696773 /etc/ati/atiapfuser.blb<br />
Xorg 32072 root 194r REG 8,3 0 17696773 /etc/ati/atiapfuser.blb<br />
Xorg 32072 root 195r REG 8,3 0 17696773 /etc/ati/atiapfuser.blb</p>
<p>Here, the "195r" in the last line indicates that the Xorg process has 196 file descriptors open total (first one is fd=0). In this particular case, 120 of these are for the file /etc/ati/atiapfuser.blb . Whenever the DAQ system is started from rcm.sh, several windows are opened and this leads to the Xorg process using another file descriptor for each one to open the /etc/ati/atiapfuser.blb file. As it turns out, when one stops the DAQ system using the "STOP" button on the rcm.sh gui, the DAQ processes are killed, but the Xorg process maintains its open file descriptors to the /etc/ati/atiapfuser.blb file. This eventually leads to the failure mode after a couple of cycles of starting and stopping the DAQ.</p>
<p>We tried reproducing the problem with a script that simply opened many xterm windows. This did suceed in exhausting the file descriptors and did produce the "maximum number of clients reached" error. However, these did not cause the /etc/ati/atiapfuser.blb file to be opened. At the end of the test, the number of open file descriptors in the Xorg process returned to what it had been before the test. This indicates there is something different about the windows opened for the DAQ by rcm.sh compared to simply launching xterms. At this time, it is unclear what that is.</p>
<p>The problem file /etc/ati/atiapfuser.blb is related to the ATI Radeon graphics card. It is unclear where it comes from since rpm claims it is not owned by any package (nor is any other file in the /etc/ati directory). The file itself is owned by root and empty. (That's right, there is *nothing* in this file!) It appears not to be required since deleting it does not seem to have any ill effects. Logging in/out and even rebooting the computer does not seem to cause the file to be recreated. We found some reference to this issue and the file here:</p>
<p><a href="https://bugs.mageia.org/show_bug.cgi?id=10163">https://bugs.mageia.org/show_bug.cgi?id=10163</a></p>
<p>One workaround used to address the "maximum number of clients reached" problem previosuly was to run the DAQ from a VNC session. Running in this mode never exhibited this problem. Similarly when running NM sessions. This is likely due to both of them using different X servers that did not access the ATI graphics card directly.</p>
<p>We have also not seen this issue when running the DAQ on gluon01, the RHEL7 computer, albeit it has been used very little for the DAQ. It does not have the file /etc/ati/atiapfuser.blb installed though like the RHEL6 machines.</p>
<p>The file /etc/ati/atiapfuser.blb has now been removed from gluon02-gluon05.</p>
</div>
<div class="attachment-box">
</div>
</article> </div>
</div>
</body>
</html>