<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
<!--
html{color:#555555;}body{line-height:1.5;font-family:'Trebuchet MS','Helvetica Neue',Arial,Helvetica,sans-serif;font-size:87.5%;}h1{font-size:1.6em;}h2.field-label{display:inline-block;font-size:1em;padding-right:5px;min-width:10em;margin:0.3em;}.problem_report{line-height:1.5;max-width:60em;}fieldset.problem_report.resolved
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVQ4jWP8//8/AyWAiZACd3f3/xYWFrht+f//P1a84t3e/0obff4rbfT5D1GGXR0LuoEr3+/7X3W4n2gvwA0gVSOKAcqbfPGGpImJCU45JgYGBoa7fpsZ22wLSbadgYGBgRE9GrF55Vf2BYbHjx8zYjWB0ljAcAGGExkZ/0MtwuoCggmJEBh4AzBS4pMnT/7fuXOH4dKlSwwnT56EiwcGBv43MDBgMDExYdDX12eQkZGBhAlyiC5YsOA/AwMDUXjLli3/iYoFQgAA+pSxZrXofD0AAAAASUVORK5CYII=);background-repeat:no-repeat;padding-left:18px;}fieldset.problem_report.needs_attention
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABmJLR0QA/wD/AP+gvaeTAAAA9ElEQVR42sWTvUoDQRSFv9wMKWxSBVmzdhZJIwTWv9pyLWxTpbE1kBeJPoLxBazzBgGFKNqlHXAhsITUw1y7sMpmjER0YJrDPWcO556BPzhjYPJjlhfT82LUpxcK6Lo5U0JMgcu56tXy7BQajeBDpkAcAEdz1W6uyrLdYieK2DMmKCArcqczpH/ddc0msy+OkyQJC4h3N0ynx+Q5ALtUNs5q5U+8e+R+VPFi4kjk3NZqd++qUK+TZVnYwSfAOyvejeLXt/2qVG/dYoG1dqseBNco27bs/wXKWhIDB8AhcFLAH4Bn4Al4AUqT7RVC++6mv/JVPwDi3VGzomYvyAAAAABJRU5ErkJggg==);background-repeat:no-repeat;padding-left:18px;}.problem_report div.field-items{display:inline-block;}div.date-vitals p{font-size:87.5%;}a{text-decoration:none;}.Readme a:link,.Readme a:visited,.Readme
a:active{color:red;}
-->
</style>
</head>
<body id="mimemail-body" class="elog-logentry-notify">
<div id="center">
<div id="main">
<style>
<!--/*--><![CDATA[/* ><!--*/
div.field-vitals{
margin: 0.5em 0;
}
div.field-vitals .field-type-taxonomy-term-reference {
margin: 0.1em 0;
}
article.comment {
padding-left: 10px;
}
article.comment.odd {
background-color: #EEEEEE;
}
article.comment.even {
background-color: #DDDDDD;
}
div.node-content.logentry table{
width: auto;
border-collapse: collapse;
border-spacing: 0;
border-width: 1px;
}
div.node-content.logentry th{
border: inherit;
}
div.node-content.logentry blockquote{
background-color: #FFFFFF;
}
div.node-content.logentry caption{
font-size: 1em;
font-weight: normal;
}
table.field-vitals{
margin-top: 1em;
margin-bottom: 1em;
font-size: 87.5%;
}
table.field-vitals th{
vertical-align: middle;
text-align: left;
width: 15%;
padding: 0.1em;
}
table.field-vitals td{
vertical-align: middle;
text-align: left;
width: auto;
padding: 0.1em;
}
table.field-vitals td li {
margin-left: 0;
list-style-type: none;
list-style-image: none;
}
table.downtime {
width: 30em;
margin-bottom: 1em;
border: 1px black dotted;
}
table.downtime th {
text-align: center;
}
table.downtime td {
text-align: center;
}
tr.caption th {
border-bottom: none;
}
table.downtime tfoot{
background-color:#EEEEEE;
}
div.field-name-body{
margin: 1em 0;
font-size: 110%;
}
div.date-vitals p{
margin: .1em 0;
}
article div.ctools-collapsible-container{
margin-left: -5px;
clear: both;
}
#comment-form{
margin-left: 5px;
border: graytext outset medium;
-moz-border-radius: 15px;
border-radius: 15px;
padding: 1em;
}
div.comments-form-box {
margin-top: 2em;
margin-bottom: 5em;
}
h3.comment-title {
/* display: none; */
}
p.author-datetime{
font-weight: bold;
}
/*--><!]]>*/
</style>
<article id="node-1456000" class="node node-logentry article ia-n clearfix" role="article">
<header class="node-header">
<h1 class="node-title">
<a href="https://logbooks.jlab.org/entry/4382940" rel="bookmark">ECal heaters all turned off at 19:50</a>
</h1>
</header>
<div class="date-vitals">
<p class="author-datetime">
Lognumber <a href="https://logbooks.jlab.org/entry/4382940" class="lognumber" data-lognumber="4382940">4382940</a>. Submitted by <a href="https://logbooks.jlab.org/user/jonesdc">jonesdc</a> on <time datetime="2025-05-24T22:31:06-0400" pubdate="pubdate"><a href="https://logbooks.jlab.org/entries?start_date=1748136666&end_date=1748143866&book=HALOG">Sat, 05/24/2025 - 22:31</a></time>. </p>
<table class="field-vitals">
<tr><th>Logbooks: </th><td><a href="https://logbooks.jlab.org/book/halog">HALOG</a></td></tr>
<tr><th>Entry Makers: </th><td>jonesdc, chandan</td></tr>
</table>
</div>
<div class="logentry node-content">
<p>Around 19:50 all ECal heaters lost power simultaneously. About 25 minutes later the low temperature alarm went off and the shift crew called Chandan. Chandan checked for issues and found nothing obvious and called me. I came in and found the computer system all functional, and the process controllers still set properly, but I noticed the watchdog heartbeat on the GUI was frozen and the variable had lost connection. This heartbeat is part of the safety system that checks to see if the cRIO computer is still functioning. Every few seconds the computer on the expansion chassis down in hall sends a pulse to a watchdog process controller to signify it is still active. If those pulses stop, the watchdog process controller opens a relay that stops power to all zones. This is apparently what happened and even though the computer system as a whole seemed to continue working well, the part that sent out the heartbeat failed. The watchdog controller then opened the relay and temperatures started dropping immediately. I called Marc McMullen and consulted with him and he suggested rebooting the cRIO in the hall remotely. I did this logging in to hareboot39 and cycling outlet 8. When the controls came back up, everything was working fine again and temperatures started to recover. Front crystal temperatures are above 200degC once again and will continue to recover several hours before reaching their stable setpoint at 210degC. All is well once more and with the shift crew monitoring this, we were able to catch it before the temperatures dropped very far below 200degC.</p>
</div>
<div class="attachment-box">
<div class="image-wrapper"><img view_mode="elog_email" class="image-style-plentybig" src="https://logbooks.jlab.org/files/styles/plentybig/public/2025/05/4382940/Screenshot from 2025-05-24 22-26-51.png" width="960" height="562" alt="" /></div> </div>
</article>
</div>
</div>
</body>
</html>