<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi Wouter, <br>
</p>
<p>Tried that ... no difference. I think this is related to the way
zmq handles (or doesn't) sockets.<br>
</p>
<p>Thanks,</p>
<p>Michael <br>
</p>
<div class="moz-cite-prefix">On 2025-01-21 10:37 a.m., Wouter
Deconinck wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAEPWh+btk5CEzALySJoyp+e_RFLu7hauXeNuq7jKAW26SgpzoQ@mail.gmail.com">
<div dir="auto">
<div>Bandaid: I think you are running into the ulimit on maximum
file descriptors. If you don't care much about that, and you
have root access, you can increase it. A factor 10 doesn't
seem impossible for the system to handle. It's in
/etc/sysctl.conf for permanent change. sysctl command for
temporary change.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Still gotta figure out why the fds aren't
released.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Wouter</div>
<div dir="auto"><br>
<br>
<div class="gmail_quote gmail_quote_container" dir="auto">
<div dir="ltr" class="gmail_attr">On Tue, Jan 21, 2025,
10:25 AM Michael Gericke via Moller_intdet <<a href="mailto:moller_intdet@jlab.org" moz-do-not-send="true" class="moz-txt-link-freetext">moller_intdet@jlab.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<span style="font-family:Calibri"><span style="font-size:12pt">
<table style="background-color:#fbe3b3;border:1px solid black">
<tbody>
<tr>
<td><font style="color:#462c1d"><strong>Caution!</strong>
This message was sent from outside the
University of Manitoba.<span></span></font></td>
</tr>
</tbody>
</table>
</span></span><br>
<div>
<p>Yes, googling was, of course, my first attempt as
well ...</p>
<p><br>
</p>
<p>My understanding is that epoll.cxx is mostly used
for socket management and if I were to create a
bunch of open sockets (without closing them) I would
understand the crash.<br>
But the only place where I open sockets repeatedly
is in the same loop where I close them as well (e.g.
zmq_close(data_socket) : "The
<em>zmq_close()</em> function shall destroy the
socket referenced by the <em>socket</em>
argument.").</p>
<p>Either I am not doing something correctly or there
is a bug in that function with respect to the OS
side.</p>
<p><br>
</p>
<p>Thanks,</p>
<p><br>
</p>
<p>Michael <br>
</p>
<div><br>
</div>
<div>On 2025-01-21 10:08 a.m., Robert Michaels wrote:<br>
</div>
<blockquote type="cite">
<span style="font-family:Calibri"><span style="font-size:12pt">
<table style="background-color:#fbe3b3;border:1px solid black">
<tbody>
<tr>
<td><font style="color:#462c1d"><strong>Caution!</strong>
This message was sent from outside the
University of Manitoba.<span></span></font></td>
</tr>
</tbody>
</table>
</span></span><br>
<div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I haven't encountered this, but I notice that if
you Google "Too many open files
(src/epoll.cpp:38)" there are several
suggestions. My experience with Googling is
that there may be 10 ideas and it ends up that 1
of them works.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
yours</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Bob</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_-381916802176522365Signature">
<div id="m_-381916802176522365divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px;margin-bottom:0px">Dr.
Robert Michaels</p>
<p style="margin-top:0px;margin-bottom:0px">Staff
Scientist, Jefferson Lab</p>
<p style="margin-top:0px;margin-bottom:0px"><a href="http://userweb.jlab.org/~rom" style="margin-top:0px;margin-bottom:0px" target="_blank" rel="noreferrer" moz-do-not-send="true" class="moz-txt-link-freetext">http://userweb.jlab.org/~rom</a><br>
</p>
</div>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-381916802176522365divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b>
Moller_daq
<a href="mailto:moller_daq-bounces@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true"><moller_daq-bounces@jlab.org></a>
on behalf of Michael Gericke
<a href="mailto:Michael.Gericke@umanitoba.ca" target="_blank" rel="noreferrer" moz-do-not-send="true"><Michael.Gericke@umanitoba.ca></a><br>
<b>Sent:</b> Tuesday, January 21, 2025 11:01
AM<br>
<b>To:</b> Zuhal Seyma Demiroglu <a href="mailto:zdemirog@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true">
<zdemirog@jlab.org></a>; Ciprian Gal <a href="mailto:ciprian@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true">
<ciprian@jlab.org></a>; Malte Wilfert
<a href="mailto:mwilfert@uni-mainz.de" target="_blank" rel="noreferrer" moz-do-not-send="true">
<mwilfert@uni-mainz.de></a>; <a href="mailto:moller_daq@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true" class="moz-txt-link-freetext">
moller_daq@jlab.org</a> <a href="mailto:moller_daq@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true">
<moller_daq@jlab.org></a>; <a href="mailto:moller_analysis@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true" class="moz-txt-link-freetext">
moller_analysis@jlab.org</a> <a href="mailto:moller_analysis@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true">
<moller_analysis@jlab.org></a>; <a href="mailto:bryerton@gmail.com" target="_blank" rel="noreferrer" moz-do-not-send="true" class="moz-txt-link-freetext">
bryerton@gmail.com</a> <a href="mailto:bryerton@gmail.com" target="_blank" rel="noreferrer" moz-do-not-send="true">
<bryerton@gmail.com></a>; <a href="mailto:moller_intdet@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true" class="moz-txt-link-freetext">
moller_intdet@jlab.org</a> <a href="mailto:moller_intdet@jlab.org" target="_blank" rel="noreferrer" moz-do-not-send="true">
<moller_intdet@jlab.org></a><br>
<b>Subject:</b> [Moller_daq] [EXTERNAL]
Question: ADC board software - Too many files
open core dump</font>
<div> </div>
</div>
<div>
<p>Hi everyone,</p>
<p>Sorry if you get this multiple times (I am
sending to several email lists with
significant overlap).<br>
</p>
<p>I have an annoying problem. For the PMT
testing, I want to run through a given set of
tests for each PMT in one sitting (DAQ and
analysis program running continuously), which
<br>
means starting the ADC DAQ on the computer end
once and (currently - ideally) letting it
collect upwards of about 1000 5 second long
runs at a time for each PMT. The program<br>
runs fine for about 180 5 second sets and then
core dumps with "Too many open files
(src/epoll.cpp:38)".</p>
<p>If you have encountered this problem and
figured out how to solve it (beyond the band
aid suggestions given in online posts), can
you please let me know.</p>
<p>Thanks,</p>
<p>Michael <br>
</p>
<p><br>
</p>
<p>Some more details ...</p>
<p><br>
</p>
<p>I am writing both root files and binary
files, for the raw data. The code uses a
separate thread to write the root trees, while
it keeps getting data from the ADC board
continuously.
<br>
</p>
<p>The thread that writes the root trees and
stores them to file is started only once and
within it, ROOT files are opened and closed in
the same loop. The main process writes<br>
the raw data file and handles the
communication with the ADC board. The raw data
files are written in a function that both
opens and closes each file that is being
written.<br>
</p>
<p>There are two zmq sockets for communication
with the ADC (control and data) and each is
opened and closed for each 5 second data chunk
that is being received from the ADC<br>
(each single run).</p>
<p>I scoured forums and various information
sources, but most of posts I find suggest that
once needs to increase the nofile parameter in
<code>/proc/sys/fs/file-max </code>
or similar. <br>
I have done that, but it doesn't really change
anything. It just allows me to take more runs,
but doesn't solve the problem.
<br>
</p>
<p>In principle, I should be able to run this
indefinitely (aside from storage space
issues).</p>
<p>Of course I could just stop the process and
restart periodically (collecting fewer runs at
a time), but I want to move through a series
of PMT voltages in one run series, with<br>
as little time as possible between changing
the HV. <br>
</p>
<p>I have encountered this crash every time I
start the program and run for extended periods
of time (1 hour or more) and it is driving me
crazy, because I think the program is written<br>
such that there should not be "many" open
files at a time (I think maybe at most 5 at
any given time).<br>
</p>
<p>The only way around this at the moment is to
use a script that restarts the ADC software
end after some number of runs before this
crash occurs. That's okay, but doesn't resolve
<br>
the basic problem.</p>
<p>Somehow I feel like there is either a bug in
ZMQ or I am doing something wrong (likely the
latter, but some posts I found seem to suggest
the possibility of the former).</p>
<p>I don't want to throw a bunch of code at you.
For now, I am just wondering if any here has
encountered a similar issue and could guess
what the problem is.</p>
<p>If anyone wants to take a look (if you have
time), the code is located here:</p>
<p><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mtgericke_MOLLER-2DIntElec-2DPMTGain-2DMeas&d=DwMDaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=oEOVH2JDjF9X3FqUtWH-nt4bOXao4u2S63djv3IksSA&m=h63pm6kCVRZsJZDv6Qm3IadomS8gZlz-u_jJh_kb06vZdHak21HYvLepNBpUh1Zl&s=LOn7q6CUDdVUcA7WtGX-sjMjpFun4LiiFijDNN2RqTM&e=" target="_blank" rel="noreferrer" moz-do-not-send="true">https://github.com/mtgericke/MOLLER-IntElec-PMTGain-Meas</a></p>
<p>The relevant file is CMData.cxx <br>
</p>
<p>The relevant functions are:</p>
<p>void CMData::StartDataCollection() - file
lines 712-733 , 758-781, </p>
<p>void *CMData::GetServerData(void *vargp) -
file line 826</p>
<p>void* CMData::FillRootTreeThread(void *vargp)
- file lines 1070, 1293</p>
<p>void* CMData::GetSocket(SockType type) -
file line 458</p>
<p><br>
</p>
<br>
<p><img alt="" width="1173" height="733" src="cid:ii_19489b6c088cc1103321" moz-do-not-send="true"></p>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</body>
</html>