<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
<!--
html{color:#555555;}body{line-height:1.5;font-family:'Trebuchet MS','Helvetica Neue',Arial,Helvetica,sans-serif;font-size:87.5%;}h1{font-size:1.6em;}h2.field-label{display:inline-block;font-size:1em;padding-right:5px;min-width:10em;margin:0.3em;}.problem_report{line-height:1.5;max-width:60em;}fieldset.problem_report.resolved
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVQ4jWP8//8/AyWAiZACd3f3/xYWFrht+f//P1a84t3e/0obff4rbfT5D1GGXR0LuoEr3+/7X3W4n2gvwA0gVSOKAcqbfPGGpImJCU45JgYGBoa7fpsZ22wLSbadgYGBgRE9GrF55Vf2BYbHjx8zYjWB0ljAcAGGExkZ/0MtwuoCggmJEBh4AzBS4pMnT/7fuXOH4dKlSwwnT56EiwcGBv43MDBgMDExYdDX12eQkZGBhAlyiC5YsOA/AwMDUXjLli3/iYoFQgAA+pSxZrXofD0AAAAASUVORK5CYII=);background-repeat:no-repeat;padding-left:18px;}fieldset.problem_report.needs_attention
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABmJLR0QA/wD/AP+gvaeTAAAA9ElEQVR42sWTvUoDQRSFv9wMKWxSBVmzdhZJIwTWv9pyLWxTpbE1kBeJPoLxBazzBgGFKNqlHXAhsITUw1y7sMpmjER0YJrDPWcO556BPzhjYPJjlhfT82LUpxcK6Lo5U0JMgcu56tXy7BQajeBDpkAcAEdz1W6uyrLdYieK2DMmKCArcqczpH/ddc0msy+OkyQJC4h3N0ynx+Q5ALtUNs5q5U+8e+R+VPFi4kjk3NZqd++qUK+TZVnYwSfAOyvejeLXt/2qVG/dYoG1dqseBNco27bs/wXKWhIDB8AhcFLAH4Bn4Al4AUqT7RVC++6mv/JVPwDi3VGzomYvyAAAAABJRU5ErkJggg==);background-repeat:no-repeat;padding-left:18px;}.problem_report div.field-items{display:inline-block;}div.date-vitals p{font-size:87.5%;}a{text-decoration:none;}.Readme a:link,.Readme a:visited,.Readme
a:active{color:red;}
-->
</style>
</head>
<body id="mimemail-body" class="elog-logentry-notify">
<div id="center">
<div id="main">
<style>
<!--/*--><![CDATA[/* ><!--*/
div.field-vitals{
margin: 0.5em 0;
}
div.field-vitals .field-type-taxonomy-term-reference {
margin: 0.1em 0;
}
article.comment {
padding-left: 10px;
}
article.comment.odd {
background-color: #EEEEEE;
}
article.comment.even {
background-color: #DDDDDD;
}
div.node-content.logentry table{
width: auto;
border-collapse: collapse;
border-spacing: 0;
border-width: 1px;
}
div.node-content.logentry th{
border: inherit;
}
div.node-content.logentry blockquote{
background-color: #FFFFFF;
}
div.node-content.logentry caption{
font-size: 1em;
font-weight: normal;
}
table.field-vitals{
margin-top: 1em;
margin-bottom: 1em;
font-size: 87.5%;
}
table.field-vitals th{
vertical-align: middle;
text-align: left;
width: 15%;
padding: 0.1em;
}
table.field-vitals td{
vertical-align: middle;
text-align: left;
width: auto;
padding: 0.1em;
}
table.field-vitals td li {
margin-left: 0;
list-style-type: none;
list-style-image: none;
}
table.downtime {
width: 30em;
margin-bottom: 1em;
border: 1px black dotted;
}
table.downtime th {
text-align: center;
}
table.downtime td {
text-align: center;
}
tr.caption th {
border-bottom: none;
}
table.downtime tfoot{
background-color:#EEEEEE;
}
div.field-name-body{
margin: 1em 0;
font-size: 110%;
}
div.date-vitals p{
margin: .1em 0;
}
article div.ctools-collapsible-container{
margin-left: -5px;
clear: both;
}
#comment-form{
margin-left: 5px;
border: graytext outset medium;
-moz-border-radius: 15px;
border-radius: 15px;
padding: 1em;
}
div.comments-form-box {
margin-top: 2em;
margin-bottom: 5em;
}
h3.comment-title {
/* display: none; */
}
p.author-datetime{
font-weight: bold;
}
/*--><!]]>*/
</style><article id="node-1114437" class="node node-logentry contextual-links-region article ia-n clearfix" role="article"><header class="node-header"><h1 class="node-title">
<a href="https://logbooks.jlab.org/entry/4041402" rel="bookmark">RC Daily Summary - Wed 14th Sept.</a>
</h1>
</header><div class="contextual-links-wrapper"><ul class="contextual-links"><li><a href="https://logbooks.jlab.org/entry/4041402/edit?destination=email/send">Edit</a></li><li><a href="https://logbooks.jlab.org/entry/4041402/delete?destination=email/send">Delete</a></li></ul></div>
<div class="date-vitals">
<p class="author-datetime">
Lognumber <a href="https://logbooks.jlab.org/entry/4041402" class="lognumber" data-lognumber="4041402">4041402</a>. Submitted by <a href="https://logbooks.jlab.org/user/daria">daria</a> on <time datetime="2022-09-14T12:05:06-0400" pubdate="pubdate"><a href="https://logbooks.jlab.org/entries?start_date=1663167906&end_date=1663175106&book=HBLOG">Wed, 09/14/2022 - 12:05</a></time>. </p>
<table class="field-vitals"><tr><th>Logbooks: </th><td><a href="https://logbooks.jlab.org/book/hblog">HBLOG</a></td></tr><tr><th>Entry Makers: </th><td>daria</td></tr></table></div>
<div class="logentry node-content">
<p>Dear all, </p>
<p>Oh boy, where to start... </p>
<p>The ND3 running finished well yesterday, two MVT trips, one DAQ crash and Sergey made a change to the PROD66_PIN configuration. We took two slightly shorter runs at the end so that we could pin the very last one before changing targets (16968). </p>
<p>At 5pm we changed from the ND3 target to an empty cell, with James B., Victoria and Derek. Filled the cell with LHe. Plain sailing, done in 1h30. </p>
<p>Then it all started.</p>
<p>First the hall dropped into Restricted Lockdown (apparently it's a thing) when we crowded into the "escape hatch" between the two exit doors from the hall, returning keys for the exit, and the totally unprotected "emergency escape" button got accidentally pressed. There was no ARM on shift, so they had to call RadCon and do a full sweep. Lost just over an 1h.</p>
<p>Then the RICH threw a major tantrum (loss of communication, several tiles not coming back at all regardless of LV power cycling or recovery procedures), several debugging stages from Nathan, finally a roc_reboot of rich4 fixed it. </p>
<p>Sent beam back on target at 8pm, first with 7 nA to check that all fine -- saw that LHe wasn't filling the cell fully: <a href="https://logbooks.jlab.org/entry/4041178">https://logbooks.jlab.org/entry/4041178</a><br />
We had set the level at 70%, which was "full" for the 15mm cell. This is also what we ran the 20mm ND3 cell with, but for the 20mm cell this is not enough. Increased level to 74%, cell looked full: <a href="https://logbooks.jlab.org/entry/4041180">https://logbooks.jlab.org/entry/4041180</a> <a href="https://logbooks.jlab.org/entry/4041196">https://logbooks.jlab.org/entry/4041196</a></p>
<p>We then increased the beam to 13 nA, checked that DC R1 occupancies were balanced (and around 2-3% at the peak) and decided to go for the max recommended: 20 nA, with beam blocker in. </p>
<p>Five minutes later, at 9pm, the crew chief took away our beam and told us that he has a strict order from up above to not let us run with a beam blocker! It appears there were concerns of activating ground water when a lot of power is deposited in the beam blocker, which went all the way up to DOE, who imposed restrictions (but not a ban) until more checks can be done -- and the powers-that-be over here decided to stop *any* running with the beam blocker until further notice. What followed were frantic phone-calls to MCC, to the crew chief's boss, to Eugene, back to MCC, etc. I was getting a hard and angry "no" so Eugene talked to MCC himself and they reached a compromise: we can take 5h of beam at 20 nA with the beam blocker in. Here's more on the back-story: <a href="https://logbooks.jlab.org/entry/4041183">https://logbooks.jlab.org/entry/4041183</a> <a href="https://logbooks.jlab.org/entry/4041190">https://logbooks.jlab.org/entry/4041190</a></p>
<p>While this chaos was going on, I asked the MCC for 13 nA (no beam blocker) so we could at least get some data. Not for long -- 15min later Valery called to tell us we have lost the ECAL VTP Sector 5 Trigger: <a href="https://logbooks.jlab.org/entry/4041186">https://logbooks.jlab.org/entry/4041186</a><br />
Sergey did the usual magic.</p>
<p>Brief delay while we discuss with the target experts whether we should run the empty cell at the same level of LHe as the ND3 data was taken with, but since LHe is a superfluid, it flows over the ND3 beads filling the entire cell, so even with a 70% LHe level the expectation is that the cell was "full". Hence the decision to fill the empty one to the top, too. </p>
<p>By 9.50pm, RICH is running like a Swiss clock, the DAQ is also pretending to, we're allowed 20 nA, we finally start production. And then rocs start crashing. Sergey tells us three different VTP Triggers crashed and needed resetting. We stop beam to save the precious 5h.</p>
<p>11.20pm, take umpteenth. Start another run. Sergey calls to tell us that L3 helicity delay read-out has crashed, but we can run without it (as it's empty target). Decide to just continue -- it should get reset with the configuration reload for the following run. </p>
<p>Just as the DAQ got fixed, we noticed that the LHe level in the target bath was oscillating like crazy: <a href="https://logbooks.jlab.org/entry/4041211">https://logbooks.jlab.org/entry/4041211</a></p>
<p>In consultation with Victoria and both Jameses over the phone, empirically find the sweet-spot at 73%. We can't go higher as the level is then in the blind-zone of the last sensor, higher still it's on it and very near full capacity of the bath, where it begins to overflow and slosh about sending the run valve that controls the flow into spasms. Not good. At 73% we're literally right at the very edge of the top of the cell. It's a tight fit -- 20 mm cell in a 22mm space. <a href="https://logbooks.jlab.org/entry/4041227">https://logbooks.jlab.org/entry/4041227</a> <a href="https://logbooks.jlab.org/entry/4041223">https://logbooks.jlab.org/entry/4041223</a> <a href="https://logbooks.jlab.org/entry/4041225">https://logbooks.jlab.org/entry/4041225</a></p>
<p>At 2.20am the shift leader changed over to empty target with no LHe by setting the bath level to 35%. It turns out the target is remarkably symmetrical -- the same problem occurs at the bottom as at the top: <a href="https://logbooks.jlab.org/entry/4041331">https://logbooks.jlab.org/entry/4041331</a> <a href="https://logbooks.jlab.org/entry/4041332">https://logbooks.jlab.org/entry/4041332</a><br />
Took a while to find the sweet spot with target expert and eventually also me. We settled on 40%, where the run valve stopped panicking -- appears to be just below the target cell (on RasterMon): <a href="https://logbooks.jlab.org/entry/4041258">https://logbooks.jlab.org/entry/4041258</a></p>
<p>RICH had another tantrum in the night, feeling forgotten -- recovery procedure brought it back. MVT also tripped a few times.</p>
<p>At 4.10am the empty cell - no LHe run was started, we got 45 min at 20 nA, then moved to 14 nA without beam-blocker when our 5h total was up. Beam went off at 6.30am for maintenance day.</p>
<p>By the skin of our teeth we managed to get 4h of decent beam for "empty target, full LHe", run 16975: 42.3 M events. Also a total of 6 M events on "empty target, no LHe", runs 16976, 77, 79.<br /><a href="https://logbooks.jlab.org/entry/4041269">https://logbooks.jlab.org/entry/4041269</a></p>
<p>We have accelerator maintenance from 6.30am today until about 6pm. The hall is in restricted access. Solenoid is at full field, torus is ramped down.<br />
The target group started changing the empty target to NH2 just after 9am.<br />
There's some engineering maintenance being done on the beamline vacuum and on the dome lights.<br />
Dan Carman is also checking the PMTs for He poisoning: <a href="https://logbooks.jlab.org/entry/4041366">https://logbooks.jlab.org/entry/4041366</a> </p>
<p>In total, we took 965 M "good run" trigger on positive polarised ND3 and 2,000 M triggers on negative ND3. Sebastian estimates that we accumulated 8% of FT-out data in 10% of FT-out time :-)</p>
<p>RC meeting at 2.15pm today! And Florian will be your new RC from this afternoon :-)</p>
<p>All the best,</p>
<p>Daria,<br />
Your finally pretty exhausted RC</p>
<p>******** Stats since the start of ND3 with ELMO ********</p>
<p>Target irradiation, anneal.</p>
<p>Half-wave plate OUT, target polarisation POSITIVE, good-run triggers: 467 M (16902 - 16908)</p>
<p>Half-wave plate IN, target polarisation POSITIVE, good-run triggers: 498 M (16910 - 16922)</p>
<p>Half-wave plate IN, target polarisation NEGATIVE, good-run triggers: 638 M (16923 - 16832)</p>
<p>Half-wave plate OUT, target polarisation NEGATIVE, good-run triggers: 335 M (16933 - 16944)</p>
<p>Target anneal.</p>
<p>Half-wave plate OUT, target polarisation NEGATIVE: good-run triggers where NMR pol > 20%: 514 M (16945 - 16959) </p>
<p>--- details in past RC summaries.</p>
<p>---------<br />
Half-wave plate IN, target polarisation NEGATIVE:</p>
<p>16960: 77.2 M (DAQ crash at end)<br />
16961: 5.2 M (pinned. Stopped for RICH prob.)<br />
16963: 100.3 M (pinned)<br />
16964: 100.1 M<br />
16965: 94.7 M<br />
16967: 61.5 M<br />
16968: 73.8 M (pinned)</p>
<p>TOTAL triggers from "good" runs: 513 M</p>
<p>-----<br />
Empty target with LHe</p>
<p>16969 is junk -- adjusting LHe level.<br />
16975: 42.3 M (the LHe level had been taken very briefly a bit below full in the middle of the run, but really very briefly)</p>
<p>-----<br />
Empty target, no LHe</p>
<p>16976: 0.45 M (20 nA)<br />
16978: 0.57 M (20 nA)<br />
16979: 5 M (14 nA)</p>
</div>
<div class="attachment-box">
</div>
</article> </div>
</div>
</body>
</html>