<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
<!--
html{color:#555555;}body{line-height:1.5;font-family:'Trebuchet MS','Helvetica Neue',Arial,Helvetica,sans-serif;font-size:87.5%;}h1{font-size:1.6em;}h2.field-label{display:inline-block;font-size:1em;padding-right:5px;min-width:10em;margin:0.3em;}.problem_report{line-height:1.5;max-width:60em;}fieldset.problem_report.resolved
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVQ4jWP8//8/AyWAiZACd3f3/xYWFrht+f//P1a84t3e/0obff4rbfT5D1GGXR0LuoEr3+/7X3W4n2gvwA0gVSOKAcqbfPGGpImJCU45JgYGBoa7fpsZ22wLSbadgYGBgRE9GrF55Vf2BYbHjx8zYjWB0ljAcAGGExkZ/0MtwuoCggmJEBh4AzBS4pMnT/7fuXOH4dKlSwwnT56EiwcGBv43MDBgMDExYdDX12eQkZGBhAlyiC5YsOA/AwMDUXjLli3/iYoFQgAA+pSxZrXofD0AAAAASUVORK5CYII=);background-repeat:no-repeat;padding-left:18px;}fieldset.problem_report.needs_attention
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABmJLR0QA/wD/AP+gvaeTAAAA9ElEQVR42sWTvUoDQRSFv9wMKWxSBVmzdhZJIwTWv9pyLWxTpbE1kBeJPoLxBazzBgGFKNqlHXAhsITUw1y7sMpmjER0YJrDPWcO556BPzhjYPJjlhfT82LUpxcK6Lo5U0JMgcu56tXy7BQajeBDpkAcAEdz1W6uyrLdYieK2DMmKCArcqczpH/ddc0msy+OkyQJC4h3N0ynx+Q5ALtUNs5q5U+8e+R+VPFi4kjk3NZqd++qUK+TZVnYwSfAOyvejeLXt/2qVG/dYoG1dqseBNco27bs/wXKWhIDB8AhcFLAH4Bn4Al4AUqT7RVC++6mv/JVPwDi3VGzomYvyAAAAABJRU5ErkJggg==);background-repeat:no-repeat;padding-left:18px;}.problem_report div.field-items{display:inline-block;}div.date-vitals p{font-size:87.5%;}a{text-decoration:none;}.Readme a:link,.Readme a:visited,.Readme
a:active{color:red;}
-->
</style>
</head>
<body id="mimemail-body" class="elog-logentry-notify">
<div id="center">
<div id="main">
<style>
<!--/*--><![CDATA[/* ><!--*/
div.field-vitals{
margin: 0.5em 0;
}
div.field-vitals .field-type-taxonomy-term-reference {
margin: 0.1em 0;
}
article.comment {
padding-left: 10px;
}
article.comment.odd {
background-color: #EEEEEE;
}
article.comment.even {
background-color: #DDDDDD;
}
div.node-content.logentry table{
width: auto;
border-collapse: collapse;
border-spacing: 0;
border-width: 1px;
}
div.node-content.logentry th{
border: inherit;
}
div.node-content.logentry blockquote{
background-color: #FFFFFF;
}
div.node-content.logentry caption{
font-size: 1em;
font-weight: normal;
}
table.field-vitals{
margin-top: 1em;
margin-bottom: 1em;
font-size: 87.5%;
}
table.field-vitals th{
vertical-align: middle;
text-align: left;
width: 15%;
padding: 0.1em;
}
table.field-vitals td{
vertical-align: middle;
text-align: left;
width: auto;
padding: 0.1em;
}
table.field-vitals td li {
margin-left: 0;
list-style-type: none;
list-style-image: none;
}
table.downtime {
width: 30em;
margin-bottom: 1em;
border: 1px black dotted;
}
table.downtime th {
text-align: center;
}
table.downtime td {
text-align: center;
}
tr.caption th {
border-bottom: none;
}
table.downtime tfoot{
background-color:#EEEEEE;
}
div.field-name-body{
margin: 1em 0;
font-size: 110%;
}
div.date-vitals p{
margin: .1em 0;
}
article div.ctools-collapsible-container{
margin-left: -5px;
clear: both;
}
#comment-form{
margin-left: 5px;
border: graytext outset medium;
-moz-border-radius: 15px;
border-radius: 15px;
padding: 1em;
}
div.comments-form-box {
margin-top: 2em;
margin-bottom: 5em;
}
h3.comment-title {
/* display: none; */
}
p.author-datetime{
font-weight: bold;
}
/*--><!]]>*/
</style><article id="node-362351" class="node node-logentry article ia-n clearfix" role="article"><header class="node-header"><h1 class="node-title" rel="nofollow">
<a href="https://logbooks.jlab.org/entry/3289607" rel="bookmark">DAQ testing</a>
</h1>
</header><div class="date-vitals">
<p class="author-datetime">
Lognumber <a href="https://logbooks.jlab.org/entry/3289607" class="lognumber" data-lognumber="3289607">3289607</a>. Submitted by <a href="https://logbooks.jlab.org/user/davidl">davidl</a> on <time datetime="2014-07-15T11:14:52-0400" pubdate="pubdate">Tue, 07/15/2014 - 11:14</time>. </p>
<table class="field-vitals"><tr><th>Logbooks: </th><td><a href="https://logbooks.jlab.org/book/hdlog">HDLOG</a></td></tr><tr><th>Tags: </th><td><a href="https://logbooks.jlab.org/tag/daq">DAQ</a></td></tr><tr><th>Entry Makers: </th><td><a href="mailto:furletov@jlab.org">furletov@jlab.org</a>, timmer, abbottd</td></tr></table></div>
<div class="logentry node-content">
<p>- Initial testing<br />
- Created hd_all_mc3 configuration based on hd_all_mc2 but without FCAL crates<br />
and 4 BCAL crates which appear to be in use. This leaves a configuration with 33 crates<br />
(including roctrig1)<br />
- rcm.sh hung initially while trying to connect to gluon50-daq from gluon03.<br />
Edited ~/.rcm.ini to make it use gluon50 instead of gluon50-daq<br />
- RCM GUI started, but refused to launch hd_all_mc3 configuration since the<br />
/home/hdops/CDAQ/daq_dev_vers/daq/config/hd_all_mc3 directory did not exist.<br />
This message only showed up in the original terminal used to launch rcm.sh<br />
and should probably show up somewhere else in the GUI. It also needed a<br />
file “Default” to be present in that directory. I created and empty one and it<br />
then proceeded with launching the components.<br />
- rcgui died quickly (before window came up). This turns out to be because the<br />
platform would not start locally because one was already running for the FCAL.<br />
Made /home/hdops/CDAQ/daq_dev_vers/daq/config/hd_all_mc3/hosts file<br />
and listed all components to run on gluon51.<br />
- Stopped and then started components again via RCM. Components launched<br />
and rcgui came up. Selected “Connect” and then hit “Configure”.<br />
- Most components configured, but SEB and DCBCAL failed. Then all of the<br />
non-ROC component windows disappeared. (ROCS did not have windows<br />
to start with since they were started with “use xterms” unchecked<br />
- Tried again with no changes.<br />
- Configure succeeded<br />
- Download succeeded<br />
- Prestart failed: The SEB could not connect to the ER’s ET system. The connection<br />
type was set to “Direct” but the IP address corresponded to gluon52 while<br />
the ER was running on gluon51<br />
- Set correct IP address in jcedit and tried again<br />
- Configure failed:<br />
- “CodaRcPrestart service failed”<br />
- “transition failed. SEB0 failed”<br />
- Hit configure again without restarting anything<br />
- Configure succeeded<br />
- Download succeeded<br />
- Prestart failed: same “CodaRcPrestart service failed” message in dialog.<br />
- Restart all components<br />
- Configure succeeded<br />
- Download succeeded<br />
- Prestart failed: DC’s cannot connect to multi-cast port.<br />
- Change all DC links to use “direct” to 127.0.0.1<br />
- Hit Configure without restarting anything.<br />
- Configure failed with “transition failed. DCFDC failed”<br />
- Hit Configure once more without restarting anything.<br />
- Configure succeeded<br />
- Download succeeded<br />
- Prestart failed: “CodaRcPrestart service failed”. Noticed “EtDeadException”s on<br />
all DC windows.<br />
- Restart all components<br />
- Notice that updating on RCM ROCs window does nothing. Tried hitting “stop servers”<br />
followed by “start servers”, but still all ROCs are colored grey.<br />
- Tried starting components anyway, but all ROCs fail to connect. Likely they<br />
are not being started because the shm_serv is down.<br />
- Several minutes later, the ROCs seem to be reporting again. (Perhaps Sergey was<br />
restarting something from his office??) Called Sergey. It turns out that he did restart<br />
the master server from a different account.<br />
- Problems getting all components to configure. Always one ROC failing.<br />
- Stopped all components, waited a couple of minutes and restarted them.<br />
- Try again:<br />
- Configure succeeded<br />
- Download succeeded<br />
- Prestart succeeded<br />
- Go succeeded<br />
- Forgot to start softROCcontroller. Hit end.<br />
- End failed, but that’s expected since ROCS are not aligned.<br />
- Reset and start again.<br />
- Configure succeeded<br />
- Download succeeded<br />
- Prestart succeeded<br />
- start softROCcontroller<br />
- Go succeeded<br />
- Update softROCcontroller to use 2000 events</p>
<p> —— gluon51 crashed ——<br />
- Called Paul Letta. Tried flipping the newly installed NMI switch<br />
- Crash state was locked to the point that NMI could not initiate a reboot + kernel dump</p>
<p> - Switched to gluon49 to try and cause crash there<br />
- Ran for >10minutes without crash.<br />
- Tried reducing pre scale in mcROL from 1000 to 500<br />
- Unable to start because user somov has started using some of the TOF crates<br />
- Remove TOF crates from configuration and start again<br />
- Rate is now doubled to 4kHz<br />
- Ran for 20 minutes on gluon49 and no crash</p>
<p> - Multi-cast test<br />
- Switched back to using multi-cast from SEB-ER. Success<br />
- Switched to using gluon49 and multi-cast worked.<br />
- Switched to using gluon46 and multi-cast worked.<br />
- Switched to using gluon50. Components can't connect since platform is already running.<br />
- Found platform still running on gluon49. Killed it.<br />
- Switched to using gluon51. multi-cast failed.</p>
<p> - RAID disk mounts switched on all gluons to use fast NIC (infiniband)</p>
</div>
<div class="attachment-box">
</div>
</article> </div>
</div>
</body>
</html>