<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div class="moz-cite-prefix">Thanks a lot Fernando,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Cheers,</div>
<div class="moz-cite-prefix"> Alex</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 11/19/19 1:19 PM, Fernando Barbosa wrote:<br>
</div>
<blockquote type="cite" cite="mid:BY5PR09MB38756B3CCDCE93FA7C336248C54C0@BY5PR09MB3875.namprd09.prod.outlook.com">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.xxmsoplaintext, li.xxmsoplaintext, div.xxmsoplaintext
{mso-style-name:x_xmsoplaintext;
mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:541670135;
mso-list-type:hybrid;
mso-list-template-ids:1554133944 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:1118448082;
mso-list-type:hybrid;
mso-list-template-ids:801041788 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello Alex,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I will make sure we also keep track of this other module. Certainly, we will need to have a better assessment of the protocol followed by the Winner crates upon
interlock.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks and best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Fernando<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Alexander Somov
<a class="moz-txt-link-rfc2396E" href="mailto:somov@jlab.org"><somov@jlab.org></a>
<br>
<b>Sent:</b> Tuesday, November 19, 2019 11:17 AM<br>
<b>To:</b> Fernando Barbosa <a class="moz-txt-link-rfc2396E" href="mailto:barbosa@jlab.org">
<barbosa@jlab.org></a>; Hovanes Egiyan <a class="moz-txt-link-rfc2396E" href="mailto:hovanes.egiyan@gmail.com">
<hovanes.egiyan@gmail.com></a>; Timothy Whitlatch <a class="moz-txt-link-rfc2396E" href="mailto:whitey@jlab.org">
<whitey@jlab.org></a>; <a class="moz-txt-link-abbreviated" href="mailto:halld-cal@jlab.org">
halld-cal@jlab.org</a><br>
<b>Subject:</b> Re: [Halld-cal] Minutes from yesterday's meeting<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hello Fernando, <br>
<br>
Thanks for taking care of this. <br>
<br>
There are 5 modules with the DAC problems, not 4, see log 3741916<br>
(one module was released later by RadCon, it's in 118).<br>
<br>
There is another module (from fadc spares) with the suspicious <br>
channel.<br>
<br>
<br>
The voltage spike could be the problem. Though, the spike could <br>
occur during the crate interlock trip (see Hovanes email), caused <br>
by temperature, i.e., this accident can be indirectly related to<br>
heat.<br>
<br>
(potentially it could have happened during the general power <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">outage in the hall, which is out of our control).<br>
<br>
<br>
Modules were replaced with the crate switched off unless calorimeter <br>
guys did something before. Modules were identified dead before <br>
replacing, the sequence is: identify the problem and replace <br>
the module.<br>
<br>
<br>
We should try to prevent it from happening again, replacing <br>
modules is an expensive procedure.<br>
<br>
<br>
Cheers,<br>
Alex<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On 11/19/19 9:27 AM, Fernando Barbosa wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello Hovanes,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">That’s good news as I also had a suspicion that heat was not the issue here.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We had 5 fADC250s that were replaced. The symptoms were:
</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span style="mso-list:Ignore">1)<span style="font:7.0pt
"Times New Roman"">
</span></span><!--[endif]--><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">One had one channel that baseline (DAC setting) could not be adjusted.</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span style="mso-list:Ignore">2)<span style="font:7.0pt
"Times New Roman"">
</span></span><!--[endif]--><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">On the other four, the baseline setting for all 16 channels in a module reported a timeout error.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Bill Gunning just tested all of them and here is what he found:</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo4"><!--[if !supportLists]--><span style="mso-list:Ignore">1)<span style="font:7.0pt
"Times New Roman"">
</span></span><!--[endif]--><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The one with the one “faulty” channel is fully operational. He tested it multiple times and it is still operating correctly. We will re-test again just
to be sure.</span><o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo4"><!--[if !supportLists]--><span style="mso-list:Ignore">2)<span style="font:7.0pt
"Times New Roman"">
</span></span><!--[endif]--><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The other 4 had their fuses blown on the 12 V supply (all supply feeds are fused on each module) and their input filter capacitors shorted, likely caused
by voltage spikes. This supply powers the 16-channel DAC among other sections of the board. This fault is unusual and it could be caused by hotswap, i.e. replacing modules with the crate on. I don’t think this has been the case but I will check the logs and
with Alex further.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The conclusion is that these faults are not heat related but power related (the 4 units). My recommendation is still to keep the fans fixed at 4,000 RPM, which
is more than adequate, and between 3,000 RPM and 5,000 RPM for those crates in the PID loop.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks and best regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Fernando</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Hovanes Egiyan
<a href="mailto:hovanes.egiyan@gmail.com" moz-do-not-send="true"><hovanes.egiyan@gmail.com></a>
<br>
<b>Sent:</b> Monday, November 18, 2019 1:58 PM<br>
<b>To:</b> Fernando Barbosa <a href="mailto:barbosa@jlab.org" moz-do-not-send="true">
<barbosa@jlab.org></a>; Timothy Whitlatch <a href="mailto:whitey@jlab.org" moz-do-not-send="true">
<whitey@jlab.org></a>; Alexander Somov <a href="mailto:somov@jlab.org" moz-do-not-send="true">
<somov@jlab.org></a>; <a href="mailto:halld-cal@jlab.org" moz-do-not-send="true">
halld-cal@jlab.org</a><br>
<b>Subject:</b> Re: [Halld-cal] Minutes from yesterday's meeting</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p>Hi Fernando, Tim. <o:p></o:p></p>
<p> <o:p></o:p></p>
<p>not only the crates are set to turn off at high temperatures in the hall, but <o:p>
</o:p></p>
<p>the crates that were online were actually turned off by EPICS during that <o:p>
</o:p></p>
<p>"open door" incident. See log entry <a href="https://logbooks.jlab.org/entry/3734190" moz-do-not-send="true">
https://logbooks.jlab.org/entry/3734190</a><o:p></o:p></p>
<p>The current high temperature limit in EPICS is 82F <o:p></o:p></p>
<p>as measured by PLC in front and behind the solenoid magnet, not 80F. <o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Note that some crates might shut off power to the bus after (or before) EPICS does using
<o:p></o:p></p>
<p>crate's internal interlock. Also, EPICS interlock only shuts power to the basket, the crates
<o:p></o:p></p>
<p>internal interlock will most likely still get engaged if temperature keeps rising after EPICS
<o:p></o:p></p>
<p>shuts the power off, and will take the crate's fan-tray offline. I do not know what exactly
<o:p></o:p></p>
<p>triggers the crates internal interlock, probably input or output temperature to the fans. Therefore ,<o:p></o:p></p>
<p>I do not know the order these interlock trip, it might be crate-dependent based on the
<o:p></o:p></p>
<p>environment in the hall and around the crate. <o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Hovanes .<o:p></o:p></p>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
<div>
<p class="MsoNormal">On 11/18/19 9:55 AM, Fernando Barbosa wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello Tim,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I am not certain but I will talk to Hovanes.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Fernando</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Timothy Whitlatch
<a href="mailto:whitey@jlab.org" moz-do-not-send="true"><whitey@jlab.org></a> <br>
<b>Sent:</b> Monday, November 18, 2019 9:17 AM<br>
<b>To:</b> Fernando Barbosa <a href="mailto:barbosa@jlab.org" moz-do-not-send="true">
<barbosa@jlab.org></a>; Alexander Somov <a href="mailto:somov@jlab.org" moz-do-not-send="true">
<somov@jlab.org></a>; <a href="mailto:halld-cal@jlab.org" moz-do-not-send="true">
halld-cal@jlab.org</a><br>
<b>Cc:</b> Hovanes Egiyan <a href="mailto:hovanes.egiyan@unh.edu" moz-do-not-send="true">
<hovanes.egiyan@unh.edu></a><br>
<b>Subject:</b> Re: [Halld-cal] Minutes from yesterday's meeting</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Hi Fernando,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">I was under the impression that the crates already are set to turn off above 80F</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black">Cheers,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper">
<p><span style="font-family:"Calibri",sans-serif;color:black">Tim Whitlatch </span><o:p></o:p></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Hall D Engineer</span><o:p></o:p></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Jefferson Lab</span><o:p></o:p></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">757-269-5087</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div class="MsoNormal" style="text-align:center" align="center">
<hr width="98%" size="2" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Fernando Barbosa <<a href="mailto:barbosa@jlab.org" moz-do-not-send="true">barbosa@jlab.org</a>><br>
<b>Sent:</b> Monday, November 18, 2019 8:56 AM<br>
<b>To:</b> Timothy Whitlatch <<a href="mailto:whitey@jlab.org" moz-do-not-send="true">whitey@jlab.org</a>>; Alexander Somov <<a href="mailto:somov@jlab.org" moz-do-not-send="true">somov@jlab.org</a>>;
<a href="mailto:halld-cal@jlab.org" moz-do-not-send="true">halld-cal@jlab.org</a> <<a href="mailto:halld-cal@jlab.org" moz-do-not-send="true">halld-cal@jlab.org</a>><br>
<b>Cc:</b> Fernando Barbosa <<a href="mailto:barbosa@jlab.org" moz-do-not-send="true">barbosa@jlab.org</a>><br>
<b>Subject:</b> RE: [Halld-cal] Minutes from yesterday's meeting</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello Tim,</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks for reviewing the logs. This certainly could explain the increased failure rate of these modules. I will follow up closely on the diagnostics of these
5 fADC250s that failed to see if there is anything that points to an overheating condition.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It would also be very helpful if Alex could check the DAQ logs for the temperature readout of the FPGAs for this same period.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">One area that needs to be addressed in this regard is to implement a protocol that turns crates off when the temperature in the hall reaches some set point:
the crates would stay off until the temperature in the hall would decrease below the set point and someone would then need to re-enable them (an administrative control). I will talk to Hovanes and Nick.</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best regards,</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Fernando</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Timothy Whitlatch <<a href="mailto:whitey@jlab.org" moz-do-not-send="true">whitey@jlab.org</a>>
<br>
<b>Sent:</b> Monday, November 18, 2019 7:54 AM<br>
<b>To:</b> Alexander Somov <<a href="mailto:somov@jlab.org" moz-do-not-send="true">somov@jlab.org</a>>; Fernando Barbosa <<a href="mailto:barbosa@jlab.org" moz-do-not-send="true">barbosa@jlab.org</a>>;
<a href="mailto:halld-cal@jlab.org" moz-do-not-send="true">halld-cal@jlab.org</a><br>
<b>Subject:</b> Re: [Halld-cal] Minutes from yesterday's meeting</span><o:p></o:p></p>
</div>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<p class="xmsonormal"><span style="font-family:"Calibri",sans-serif;color:black">Hi All,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Calibri",sans-serif;color:black">In looking back at the archive data, there is only 1 time the hall reached above 80F this spring through fall. October 12 the temp reached 84F and was above 80F for about 12 hours.
This is when the overhead door was left open after a tour.</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Calibri",sans-serif;color:black">Cheers,</span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div id="x_Signature">
<div id="x_divtagdefaultwrapper">
<p><span style="font-family:"Calibri",sans-serif;color:black">Tim Whitlatch </span><o:p></o:p></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Hall D Engineer</span><o:p></o:p></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">Jefferson Lab</span><o:p></o:p></p>
<p><span style="font-family:"Calibri",sans-serif;color:black">757-269-5087</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="xmsonormal"><span style="font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
</div>
<div class="MsoNormal" style="text-align:center" align="center">
<hr width="98%" size="2" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="xmsonormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> Halld-cal <<a href="mailto:halld-cal-bounces@jlab.org" moz-do-not-send="true">halld-cal-bounces@jlab.org</a>>
on behalf of Alexander Somov <<a href="mailto:somov@jlab.org" moz-do-not-send="true">somov@jlab.org</a>><br>
<b>Sent:</b> Friday, November 15, 2019 6:57 PM<br>
<b>To:</b> Fernando Barbosa <<a href="mailto:barbosa@jlab.org" moz-do-not-send="true">barbosa@jlab.org</a>>;
<a href="mailto:halld-cal@jlab.org" moz-do-not-send="true">halld-cal@jlab.org</a> <<a href="mailto:halld-cal@jlab.org" moz-do-not-send="true">halld-cal@jlab.org</a>><br>
<b>Subject:</b> Re: [Halld-cal] Minutes from yesterday's meeting</span> <o:p></o:p></p>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal" style="margin-bottom:12.0pt">Hi Fernando,<br>
<br>
As we have discussed, the temperature accident happened<br>
not the first time. After the first accident, when we <br>
melted a CTP module by running a fan speed at ~2000 rpms, <br>
I took many trigger-related (and other) crates from the <br>
PID control and ran them at 4500+ rpm.<br>
<br>
After the overheating problems during last year, Hai Dong <br>
suggested to run the fan at about 5000 rpm for the sake of <br>
CTPs, and I agree with him. There should be enough margin <br>
to cool the crate down if the temperature in the hall <br>
increases.<br>
<br>
<br>
The CCAL crate, positioned on the same FCAL platform <br>
runs at about 5000 rpm for more than a year, I didn't <br>
have any problems with it so far.<br>
<br>
<br>
I belive that the PID fan threshold should be set to at <br>
least 4500 rpm for all crates (indeed it's a choice of <br>
sub-detector groups, who should be responsible for <br>
damaged electronics as well).<br>
<br>
<br>
<br>
Cheers,<br>
Sasha<br>
<br>
<br>
P.S. Indeed, there could be other not temperature-related <br>
issues, but this is the most likely one.<br>
<br>
<br>
Rad damage can be an issue as well, but then modules should <br>
fail "continuously" during the run, likely with various symptoms, <br>
which seems to be not the case. We got 5 modules dead after <br>
the run. The rad dose in the tagger hall is also significantly <br>
larger than in Hall D; I haven't seen such problems with fadcs <br>
there.<o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal"> <o:p></o:p></p>
</div>
<div>
<p class="xmsonormal">On 11/15/19 3:10 PM, Fernando Barbosa wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hello Elton, all,</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Some notes regarding the paragraph below:</span><o:p></o:p></p>
<p class="xxmsoplaintext" style="margin-left:.5in;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">a)</span><span style="font-size:7.0pt">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The overheating is not an issue unless the hall temperature increases significantly above the specs. For a full description, refer to GlueX-doc-2752. My recommendation has always been to
operate the fans fixed at 4,000 RPM or in the 3,000 RPM to 5,000 RPM range under PID control.</span><o:p></o:p></p>
<p class="xxmsoplaintext" style="margin-left:.5in;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">b)</span><span style="font-size:7.0pt">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">During the Summer, we experienced some unexplained failures. Nick recently informed me that there were two instances when the temperature in the hall was excessive duw to HVAC failures or
door open for long periods of time. Nick has suggested some additional interlock controls. It would be instructive if someone could check the environmental logs and others since the end of the Spring run.</span><o:p></o:p></p>
<p class="xxmsoplaintext" style="margin-left:.5in;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">c)</span><span style="font-size:7.0pt">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The last sentence is not correct. Spreading the modules apart will not provide better cooling as the thermal radiative losses are insignificant compared to the convective cooling efficiency.</span><o:p></o:p></p>
<p class="xxmsoplaintext" style="margin-left:.5in;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">d)</span><span style="font-size:7.0pt">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">There is also the possibility of radiation damage on the D2 platform due to changing experimental conditions (prototype tests of the COMCAL and others). This was also brought up by CAEN
during maintenance of failed HV controllers. I have Alexandre D. to look into this in detail from logs of radiation detectors he manages.</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Best regards,</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Fernando</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><img style="width:11.9166in;height:1.7083in" id="x_x_Picture_x0020_1" src="cid:part27.373648BC.871E9735@jlab.org" class="" width="1144" height="164" border="0"></span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-----Original Message-----<br>
From: Halld-cal <a href="mailto:halld-cal-bounces@jlab.org" moz-do-not-send="true">
<halld-cal-bounces@jlab.org></a> On Behalf Of Elton Smith<br>
Sent: Friday, November 15, 2019 1:25 PM<br>
To: <a href="mailto:halld-cal@jlab.org" moz-do-not-send="true">halld-cal@jlab.org</a><br>
Subject: [Halld-cal] Minutes from yesterday's meeting</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Dear collaborators,</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I have posted the minutes from yesterday's meeting on the wiki at
<a href="https://halldweb.jlab.org/wiki/index.php/Nov_14,_2019_Calorimeter#Minutes" moz-do-not-send="true">
<span style="color:windowtext;text-decoration:none">https://halldweb.jlab.org/wiki/index.php/Nov_14,_2019_Calorimeter#Minutes</span></a>.</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Cheers, Elton.</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">--</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Elton Smith</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Jefferson Lab MS 12H3</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">12000 Jefferson Ave STE 4</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Newport News, VA 23606</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">(757)269-7625</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">(757)269-6331 fax</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">_______________________________________________</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Halld-cal mailing list</span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><a href="mailto:Halld-cal@jlab.org" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none">Halld-cal@jlab.org</span></a></span><o:p></o:p></p>
<p class="xxmsoplaintext"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><a href="https://mailman.jlab.org/mailman/listinfo/halld-cal" moz-do-not-send="true"><span style="color:windowtext;text-decoration:none">https://mailman.jlab.org/mailman/listinfo/halld-cal</span></a></span><o:p></o:p></p>
</div>
<p class="xmsonormal" style="margin-bottom:12.0pt"> <o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Halld-cal mailing list<o:p></o:p></pre>
<pre><a href="mailto:Halld-cal@jlab.org" moz-do-not-send="true">Halld-cal@jlab.org</a><o:p></o:p></pre>
<pre><a href="https://mailman.jlab.org/mailman/listinfo/halld-cal" moz-do-not-send="true">https://mailman.jlab.org/mailman/listinfo/halld-cal</a><o:p></o:p></pre>
</blockquote>
<p> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Halld-cal mailing list<o:p></o:p></pre>
<pre><a href="mailto:Halld-cal@jlab.org" moz-do-not-send="true">Halld-cal@jlab.org</a><o:p></o:p></pre>
<pre><a href="https://mailman.jlab.org/mailman/listinfo/halld-cal" moz-do-not-send="true">https://mailman.jlab.org/mailman/listinfo/halld-cal</a><o:p></o:p></pre>
</blockquote>
</blockquote>
<p><o:p> </o:p></p>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>