<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
<!--
html{color:#555555;}body{line-height:1.5;font-family:'Trebuchet MS','Helvetica Neue',Arial,Helvetica,sans-serif;font-size:87.5%;}h1{font-size:1.6em;}h2.field-label{display:inline-block;font-size:1em;padding-right:5px;min-width:10em;margin:0.3em;}.problem_report{line-height:1.5;max-width:60em;}fieldset.problem_report.resolved
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAy0lEQVQ4jWP8//8/AyWAiZACd3f3/xYWFrht+f//P1a84t3e/0obff4rbfT5D1GGXR0LuoEr3+/7X3W4n2gvwA0gVSOKAcqbfPGGpImJCU45JgYGBoa7fpsZ22wLSbadgYGBgRE9GrF55Vf2BYbHjx8zYjWB0ljAcAGGExkZ/0MtwuoCggmJEBh4AzBS4pMnT/7fuXOH4dKlSwwnT56EiwcGBv43MDBgMDExYdDX12eQkZGBhAlyiC5YsOA/AwMDUXjLli3/iYoFQgAA+pSxZrXofD0AAAAASUVORK5CYII=);background-repeat:no-repeat;padding-left:18px;}fieldset.problem_report.needs_attention
legend{background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABmJLR0QA/wD/AP+gvaeTAAAA9ElEQVR42sWTvUoDQRSFv9wMKWxSBVmzdhZJIwTWv9pyLWxTpbE1kBeJPoLxBazzBgGFKNqlHXAhsITUw1y7sMpmjER0YJrDPWcO556BPzhjYPJjlhfT82LUpxcK6Lo5U0JMgcu56tXy7BQajeBDpkAcAEdz1W6uyrLdYieK2DMmKCArcqczpH/ddc0msy+OkyQJC4h3N0ynx+Q5ALtUNs5q5U+8e+R+VPFi4kjk3NZqd++qUK+TZVnYwSfAOyvejeLXt/2qVG/dYoG1dqseBNco27bs/wXKWhIDB8AhcFLAH4Bn4Al4AUqT7RVC++6mv/JVPwDi3VGzomYvyAAAAABJRU5ErkJggg==);background-repeat:no-repeat;padding-left:18px;}.problem_report div.field-items{display:inline-block;}div.date-vitals p{font-size:87.5%;}a{text-decoration:none;}.Readme a:link,.Readme a:visited,.Readme
a:active{color:red;}
-->
</style>
</head>
<body id="mimemail-body" class="elog-logentry-notify">
<div id="center">
<div id="main">
<style>
<!--/*--><![CDATA[/* ><!--*/
div.field-vitals{
margin: 0.5em 0;
}
div.field-vitals .field-type-taxonomy-term-reference {
margin: 0.1em 0;
}
article.comment {
padding-left: 10px;
}
article.comment.odd {
background-color: #EEEEEE;
}
article.comment.even {
background-color: #DDDDDD;
}
div.node-content.logentry table{
width: auto;
border-collapse: collapse;
border-spacing: 0;
border-width: 1px;
}
div.node-content.logentry th{
border: inherit;
}
div.node-content.logentry blockquote{
background-color: #FFFFFF;
}
div.node-content.logentry caption{
font-size: 1em;
font-weight: normal;
}
table.field-vitals{
margin-top: 1em;
margin-bottom: 1em;
font-size: 87.5%;
}
table.field-vitals th{
vertical-align: middle;
text-align: left;
width: 15%;
padding: 0.1em;
}
table.field-vitals td{
vertical-align: middle;
text-align: left;
width: auto;
padding: 0.1em;
}
table.field-vitals td li {
margin-left: 0;
list-style-type: none;
list-style-image: none;
}
table.downtime {
width: 30em;
margin-bottom: 1em;
border: 1px black dotted;
}
table.downtime th {
text-align: center;
}
table.downtime td {
text-align: center;
}
tr.caption th {
border-bottom: none;
}
table.downtime tfoot{
background-color:#EEEEEE;
}
div.field-name-body{
margin: 1em 0;
font-size: 110%;
}
div.date-vitals p{
margin: .1em 0;
}
article div.ctools-collapsible-container{
margin-left: -5px;
clear: both;
}
#comment-form{
margin-left: 5px;
border: graytext outset medium;
-moz-border-radius: 15px;
border-radius: 15px;
padding: 1em;
}
div.comments-form-box {
margin-top: 2em;
margin-bottom: 5em;
}
h3.comment-title {
/* display: none; */
}
p.author-datetime{
font-weight: bold;
}
/*--><!]]>*/
</style>
<article id="node-1476982" class="node node-logentry article ia-n clearfix" role="article">
<header class="node-header">
<h1 class="node-title">
<a href="https://logbooks.jlab.org/entry/4403918" rel="bookmark">ECal Controls failure mode not caught by alarms</a>
</h1>
</header>
<div class="date-vitals">
<p class="author-datetime">
Lognumber <a href="https://logbooks.jlab.org/entry/4403918" class="lognumber" data-lognumber="4403918">4403918</a>. Submitted by <a href="https://logbooks.jlab.org/user/jonesdc">jonesdc</a> on <time datetime="2025-06-23T09:14:02-0400" pubdate="pubdate"><a href="https://logbooks.jlab.org/entries?start_date=1750680842&end_date=1750688042&book=HALOG">Mon, 06/23/2025 - 09:14</a></time>. </p>
<p class="last-update">
Last updated on <time datetime="2025-06-23T09:29:26-0400" pubdate="pubdate">Mon, 06/23/2025 - 09:29</time> </p>
<table class="field-vitals">
<tr><th>Logbooks: </th><td><a href="https://logbooks.jlab.org/book/halog">HALOG</a></td></tr>
<tr><th>Entry Makers: </th><td>jonesdc</td></tr>
</table>
</div>
<div class="logentry node-content">
<p>Last evening around 5PM (Sunday) the second controls card that reads out 16 channel temperatures for zones 17-32 suffered a radiation failure that is not currently set to alarm on. All temperatures dropped by about 200degC and continued to read (not frozen as we often see). With all 16 zones reading well under temperature, currents in these 16 zones went to maximum output. Temperatures continued to read and fluctuate but 200 degC low. The front plate heater temperatures rose well above their setpoints and the maximum at which the software would open the relay but the system read them around 50degC(see attachment 2). This morning shortly before 7AM Marc McMullen called me and pointed out that we have a problem. I immediately rebooted the system in the Hall remotely, and the temperature readings came back. Many of the front plate heater temperatures were well above our design threshold of 300degC. Note that the system was NOT so warm that it created a danger, but it does mean that things were warmer than we want them to be. Heat at these temperatures could cause premature yellowing of the glue attaching the light guides to crystals, for example. During our first test where the thermocouples were being read out erroneously as type J, we were likely well above the temperatures reached last night. Interestingly, the ambient temperature thermocouples used by the safety system in the hot front region barely noticed the 16 zone temperature increase. The top ambient temperature went up only 5 degC and the middle ambient barely at all, so it would have to be significantly hotter for the independent process controllers to alarm and stop all current. </p>
<p>The crystal temperatures in these zones showed increases in temperature associated with this event even when their partner thermcouples on the heaters did not (attachment 1). It is probably prudent to add further alarm conditions for both high and low temperatures on crystals as well as heaters used in the controls. Since all failures seem to occur by a full readout card of 16 zones at a time (1-16, 17-32 or 33-48), I plan to implement a new EPICS variable that averages the temperatures over these three groups of 16 and places high and low alarms on the averages for heaters, front crystal temperatures and rear crystal temperatures. It seems unlikely that all three readout cards will fail at the same time.</p>
</div>
<div class="attachment-box">
<div class="image-wrapper"><img view_mode="elog_email" class="image-style-plentybig" src="https://logbooks.jlab.org/files/styles/plentybig/public/2025/06/4403918/Screenshot from 2025-06-23 09-13-39.png" width="960" height="474" alt="" /></div> <div class="image-wrapper"><img view_mode="elog_email" class="image-style-plentybig" src="https://logbooks.jlab.org/files/styles/plentybig/public/2025/06/4403918/Screenshot from 2025-06-23 09-13-07.png" width="960" height="439" alt="" /></div> </div>
</article>
</div>
</div>
</body>
</html>