<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
<!--
html{color:#555555;}body{line-height:1.5;font-family:'Trebuchet MS','Helvetica Neue',Arial,Helvetica,sans-serif;font-size:87.5%;}h1{font-size:1.6em;}h2.field-label{display:inline-block;font-size:1em;padding-right:5px;min-width:10em;margin:0.3em;}.problem_report{line-height:1.5;max-width:60em;}fieldset.problem_report.resolved
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVQ4jWP8//8/AyWAiZACd3f3/xYWFrht+f//P1a84t3e/0obff4rbfT5D1GGXR0LuoEr3+/7X3W4n2gvwA0gVSOKAcqbfPGGpImJCU45JgYGBoa7fpsZ22wLSbadgYGBgRE9GrF55Vf2BYbHjx8zYjWB0ljAcAGGExkZ/0MtwuoCggmJEBh4AzBS4pMnT/7fuXOH4dKlSwwnT56EiwcGBv43MDBgMDExYdDX12eQkZGBhAlyiC5YsOA/AwMDUXjLli3/iYoFQgAA+pSxZrXofD0AAAAASUVORK5CYII=);background-repeat:no-repeat;padding-left:18px;}fieldset.problem_report.needs_attention
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABmJLR0QA/wD/AP+gvaeTAAAA9ElEQVR42sWTvUoDQRSFv9wMKWxSBVmzdhZJIwTWv9pyLWxTpbE1kBeJPoLxBazzBgGFKNqlHXAhsITUw1y7sMpmjER0YJrDPWcO556BPzhjYPJjlhfT82LUpxcK6Lo5U0JMgcu56tXy7BQajeBDpkAcAEdz1W6uyrLdYieK2DMmKCArcqczpH/ddc0msy+OkyQJC4h3N0ynx+Q5ALtUNs5q5U+8e+R+VPFi4kjk3NZqd++qUK+TZVnYwSfAOyvejeLXt/2qVG/dYoG1dqseBNco27bs/wXKWhIDB8AhcFLAH4Bn4Al4AUqT7RVC++6mv/JVPwDi3VGzomYvyAAAAABJRU5ErkJggg==);background-repeat:no-repeat;padding-left:18px;}.problem_report div.field-items{display:inline-block;}div.date-vitals p{font-size:87.5%;}a{text-decoration:none;}.Readme a:link,.Readme a:visited,.Readme
a:active{color:red;}
-->
</style>
</head>
<body id="mimemail-body" class="elog-logentry-notify">
<div id="center">
<div id="main">
<style>
<!--/*--><![CDATA[/* ><!--*/
div.field-vitals{
margin: 0.5em 0;
}
div.field-vitals .field-type-taxonomy-term-reference {
margin: 0.1em 0;
}
article.comment {
padding-left: 10px;
}
article.comment.odd {
background-color: #EEEEEE;
}
article.comment.even {
background-color: #DDDDDD;
}
div.node-content.logentry table{
width: auto;
border-collapse: collapse;
border-spacing: 0;
border-width: 1px;
}
div.node-content.logentry th{
border: inherit;
}
div.node-content.logentry blockquote{
background-color: #FFFFFF;
}
div.node-content.logentry caption{
font-size: 1em;
font-weight: normal;
}
table.field-vitals{
margin-top: 1em;
margin-bottom: 1em;
font-size: 87.5%;
}
table.field-vitals th{
vertical-align: middle;
text-align: left;
width: 15%;
padding: 0.1em;
}
table.field-vitals td{
vertical-align: middle;
text-align: left;
width: auto;
padding: 0.1em;
}
table.field-vitals td li {
margin-left: 0;
list-style-type: none;
list-style-image: none;
}
table.downtime {
width: 30em;
margin-bottom: 1em;
border: 1px black dotted;
}
table.downtime th {
text-align: center;
}
table.downtime td {
text-align: center;
}
tr.caption th {
border-bottom: none;
}
table.downtime tfoot{
background-color:#EEEEEE;
}
div.field-name-body{
margin: 1em 0;
font-size: 110%;
}
div.date-vitals p{
margin: .1em 0;
}
article div.ctools-collapsible-container{
margin-left: -5px;
clear: both;
}
#comment-form{
margin-left: 5px;
border: graytext outset medium;
-moz-border-radius: 15px;
border-radius: 15px;
padding: 1em;
}
div.comments-form-box {
margin-top: 2em;
margin-bottom: 5em;
}
h3.comment-title {
/* display: none; */
}
p.author-datetime{
font-weight: bold;
}
/*--><!]]>*/
</style><article id="node-361903" class="node node-logentry article ia-n clearfix" role="article"><header class="node-header"><h1 class="node-title" rel="nofollow">
<a href="https://logbooks.jlab.org/entry/3289159" rel="bookmark">Large Scale DAQ testing</a>
</h1>
</header><div class="date-vitals">
<p class="author-datetime">
Lognumber <a href="https://logbooks.jlab.org/entry/3289159" class="lognumber" data-lognumber="3289159">3289159</a>. Submitted by <a href="https://logbooks.jlab.org/user/davidl">davidl</a> on <time datetime="2014-07-03T11:47:34-0400" pubdate="pubdate">Thu, 07/03/2014 - 11:47</time>. </p>
<table class="field-vitals"><tr><th>Logbooks: </th><td><a href="https://logbooks.jlab.org/book/hdlog">HDLOG</a></td></tr><tr><th>Tags: </th><td><a href="https://logbooks.jlab.org/tag/daq">DAQ</a></td></tr><tr><th>Entry Makers: </th><td>abbottd, timmer, gurjyan, furletov, davidl</td></tr></table></div>
<div class="logentry node-content">
<p>Today we continued our regular testing/development of the large scale DAQ system in Hall-D. We again used the 50 crate system and the softROC system where the mcROL is used that reads simulated data from an EVIO file and does not access the VME backplane and does not use the Trigger system.</p>
<p>Today we started with gluon50 and having the IB network interface turned off before starting anything. (Previously, we'd used gluon53). </p>
<p>The configuration runs 4 Data Concentrators, one Secondary Event Builder, and one Event Recorder all on the same host machine. The 50 ROCs are run on their respective hosts.</p>
<p>1st attempt:<br />
- Components couldn’t connect to platform. Not sure why.</p>
<p>2nd attempt:<br />
- Some messages indicating X11 windows couldn’t open, but all windows appear to have opened.<br />
- Message was ”Warning: No xauth data”, “error in locking authority file /home/furletov/.xauthority”<br />
- Components are waiting for connections:<br />
“Socket connect error: Connection timed out”<br />
“client thread 2: error recreating rc client’s TCP send socket”<br />
- Components are slowly connecting one at a time.<br />
- eventually killed all components by hand (actually, via script)</p>
<p>3rd attempt:<br />
- Use configuration with fewer ROCs “bcal_n4_sf” (4 ROCs)<br />
- Everything came up right away.<br />
- Configure failed due to “Failed Communication with the client”</p>
<p>Sergey noted that this configuration was working fine just before 9am.<br />
Vardan said he committed some changes to CODA at 9am. We are going to back out those changes.<br />
(reverted Afecs to revision 12773)</p>
<p>4th attempt:<br />
- Using CODA version prior to Vardan’s commit this morning.<br />
- Configure succeed<br />
- Download succeeded<br />
- Prestart failed: ROCs state “roc-prestart: ET initialization failed”</p>
<p> ET ports:<br />
SEB0 23911<br />
DCFCAL 23913<br />
DCBCAL 23915<br />
DCSTPS 23917<br />
DCFDC 23919</p>
<p>5th attempt:<br />
- Reset followed by configure<br />
- Configure succeeded<br />
- Prestart failed due to ET initialization again (all ROCs)<br />
- Problem was direct connection from ROCs set to use gluon53 in jcedit<br />
- Also issue with file in COOL that stores connection hosts. Need to delete by hand<br />
if hosts changed or restart all processes.</p>
<p>6th attempt:<br />
- ROCBCAL2 failed because ROC died. This appears to have happened during reset.<br />
- All other components seemed OK</p>
<p>7th attempt:<br />
- Restarted ROCBCAL2 manually<br />
- Failed at download because of unresolved libexpat.so link to mcROL.so</p>
<p>8th attempt:<br />
- Recompiled mcROL.so on rocdev1 with fix that uses local file from RAM disk rather than NFS mounted one.<br />
- Restarted all components from scratch<br />
- Configure succeeded<br />
- Download succeeded<br />
- Prestart succeeded<br />
- Go succeeded<br />
- Data flowed until we tried restarting softROCcontroller with 10k events per chunk</p>
<p>9th attempt:<br />
- Reset<br />
- Configure succeeded<br />
- Download succeeded<br />
- softROCcontroller started with 5000 events per chunk<br />
- Prestart succeeded<br />
- Go succeeded<br />
- DCBCAL fails after several thousand events with “etDeadException” </p>
<p>10th attempt:<br />
- Reset and restarted without changing anything<br />
- Ran for ~31k events and event rate fell to zero. All components still reporting active<br />
- Most ROCs had exactly 31k events but all BCAL crates had fewer events<br />
- BCAL ROCs use same DC so there may be some back pressure issue there.</p>
<p>We were unable to thoroughly diagnose the problem leading to the data rate stopping<br />
during the last attempts. (We needed to hand the DAQ systems back to the detector groups.)<br />
It may have been due to the BCAL DC getting into a state where it was unable to receive<br />
events from the ROCs, causing them to go into a holding pattern. It was decided that we should<br />
have another look at the softROC code, but plan on doing testing with more traditional ROLs<br />
on Tuesday to see if the same problem occurs.We will try to arrange use of the TS for<br />
that test period.</p>
</div>
<div class="attachment-box">
</div>
</article> </div>
</div>
</body>
</html>