<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<title>Jefferson Lab Lessons Learned : Print Lesson</title>
<base href='https://misportal.jlab.org/ll/'>
<!--[if IE]>
<style type="text/css">
#page_content input[type=checkbox], #page_content input[type=radio] {
background-color: transparent;
border: 0;
}
</style>
<![endif]-->
</head>
<body>
<div id="doc3" class="yui-t7">
<div class='round' id='pageMessages'>
<!-- -->
</div>
<div id="lesson" class="yui-gc">
<style type="text/css">
pre {
white-space: -moz-pre-wrap;
white-space: -pre-wrap;
white-space: -o-pre-wrap;
white-space: pre-wrap;
word-wrap: break-word;
overflow: hidden;
margin-left: 10px;
}
</style>
<div id="topbar">
<div id="lesson_title">
<table width="100%">
<tr>
<td><a href="https://misportal.jlab.org/ll/index.jsf?function=view&lessonId=917">C100 Zones 2L22, 2L23 and 2L24 Power Outage and Machine Recovery - TJNAF</a></td>
</tr>
</table>
</div>
</div>
<table width="100%">
<tr>
<td valign="top" width="100%">
<div id="maintextcol" class="yui-u first">
<h2 style="white-space:nowrap">Statement of Lessons Learned</h2>
<pre>(For a copy of the original Repair Assessment Report, see the attachment)\
OSHA 1910.334(b)(2), Reclosing Circuits After Protective Device Operation:
¿After a circuit is de-energized by the automatic operation of a circuit protective device, the circuit shall not be manually reenergized until it has been determined that the equipment and circuit can be safely energized. The repetitive manual reclosing of circuit breakers or reenergizing circuits through replaced fuses shall be prohibited. When it is determined from the design of the circuit and the overcurrent devices involved that the automatic operation of a device was caused by an overload rather than a fault condition, examination of the circuit or connected equipment shall not be required before the circuit is reenergized.¿</pre>
<h2 style="white-space:nowrap">Discussion of Activities</h2>
<pre>On Sunday, March 8th 2015, at approximately 04:10, a 480 VAC circuit breaker in panel SWBD SL4-C1 tripped. The Personnel Safety System (PSS) status of the accelerator dropped from Beam Permit to Power Permit. Communications were lost with 2L22, 2L23 and 2L24. Also, the Fast Shut Down (FSD) card for the south linac C100 zones and multiple vacuum valves (iocsl23) went unresolved. Resolution of power was fairly straightforward, but machine recovery proved to be problematic due to several issues.
Excerpt from Elog: Downtime Event 1529
¿Lost contact with three zones in the SL (2L22-2L24). Update: Got the zones back, but the control system didn't come back fully. We can control the zones manually, but the "one button" features don't work. This includes resetting cavities from faults. We would have to go to each cavity's XOP page and reset them. A lot of time was spent on 2L22 due to not realizing the heater was off after everything else came back. We would try to bring the zone back up, only for it to drop on us. Once the heater was turned back on, we were golden. EESRF had to bypass 2L22-4. We Re-LEM'd and did TRIM up/down. We are in the process of sending beam around and checking phases.¿
Total Downtime attributed to this event ¿ 8 Hours 10 minutes</pre>
<h2 style="white-space:nowrap">Analysis</h2>
<pre>The root cause for the trip has not been determined, however, it should be noted the breaker has not tripped prior to or after this event occurred. The only clues for the breaker trip were an indication of a ¿Short Circuit¿ and that the breaker provides GFCI protection. The power strips in the low level racks are separate phases; since the FC chassis were on a single strip, possible excessive current load on a single phase might have caused SWBD SL4-C1 to trip. Upon resetting the breaker and after redistribution of load in the low level racks, no fault indications were incurred.
The conclusion is the breaker performed as designed ¿ the cause for the breaker trip is surmised to be the load on the breaker. Besides the possibility of single phase overload, there is so far no determination as to the direct cause for the breaker trip. </pre>
<h2 style="white-space:nowrap">Recommended Actions</h2>
<pre>Immediate Actions:
¿ Investigation/troubleshooting of equipment being down
¿ FC chassis power cables re-distributed (in order to balance out the load on the two power strips)
Additional Actions:
¿ Repair Investigation Team convened
¿ Facility work request to have circuit breaker checked
¿ 480Vac circuit breaker resetting guidance (see attachment)</pre>
<h2 style="white-space:nowrap">JLab Preventive Measures</h2>
<pre>¿ Jefferson Lab electrical worker training
¿ Equipment specific training (qualified worker as determined by group leaders)
¿ Facility maintenance and operations (FMO) regularly scheduled PMs on panelboards</pre>
</div>
<div id="comments_div">
<a name="comments"></a>
<h2>Comments</h2>
<h5 style="white-space:nowrap">11/3/2015 10:49:44 AM by Bailey, Mary Jo</h5>
<pre>Submitted to those with SAF603A: ELECTRICAL SAFETY AWARENESS</pre>
</div>
</td>
<td valign="top">
<div class="yui-u">
<fieldset>
<legend>Summary</legend>
<table>
<tr class="row">
<td class="label" nowrap>Lesson ID:</td>
<td class="dataval">917</td>
</tr>
<tr class="row">
<td class="label" nowrap>Status:</td>
<td class="dataval">OK</td>
</tr>
<tr class="row">
<td class="label" nowrap>Doc ID:</td>
<td class="dataval">2015-JLAB-917</td>
</tr>
<tr class="row">
<td class="label" nowrap>Priority:</td>
<td class="dataval">Info</td>
</tr>
<tr class="row">
<td class="label" nowrap>Safety Related:</td>
<td class="dataval">YES</td>
</tr>
<tr class="row">
<td class="label" nowrap>Originator:</td>
<td class="dataval">Bailey, Mary Jo</td>
</tr>
<tr class="row">
<td class="label" nowrap>Issued:</td>
<td class="dataval">11/3/2015 10:46:44 AM</td>
</tr>
<tr class="row">
<td class="label" nowrap>Approved By:</td>
<td class="dataval">Bailey, Mary Jo</td>
</tr>
<tr class="row">
<td class="label" nowrap>Approved On:</td>
<td class="dataval">11/3/2015 10:49:44 AM</td>
</tr>
<tr class="row">
<td class="label">Source:</td>
<td class="dataval">
ATLis
</td>
</tr>
<tr class="row">
<td class="label" nowrap>Location:</td>
<td class="dataval">TJNAF</td>
</tr>
<tr class="row">
<td class="label" nowrap>Cost Savings:</td>
<td class="dataval"></td>
</tr>
<tr class="row">
<td class="label" nowrap>Contact:</td>
<td class="dataval">Todd Kujawa (269-7006 or kujawa@jlab.org)</td>
</tr>
<tr class="row">
<td class="label" nowrap>Queued Emails:</td>
<td class="dataval">0</td>
</tr>
<tr class="row">
<td class="label" nowrap>Sent Emails:</td>
<td class="dataval">0</td>
</tr>
<tr class="row">
<td class="label" nowrap>Viewings:</td>
<td class="dataval">1 times</td>
</tr>
</table>
</fieldset>
<fieldset>
<legend>Attachments</legend>
<a name="attachments"></a>
<ul class="lesson_attribute_listing">
<li><a target="_blank" href="https://misportal.jlab.org/ll/downloadAttachment?attachmentId=4142">480 Vac_Circuit_breaker_resetting.docx</a></li>
<li><a target="_blank" href="downloadAttachment?attachmentId=4143">C100 Zones Power Event - final_draft.pdf</a></li>
</ul>
</fieldset>
<fieldset>
<legend>Hazard Issues</legend>
<ul class="lesson_attribute_listing">
<li>Electricity</li>
</ul>
</fieldset>
<fieldset>
<legend>Skills</legend>
<ul class="lesson_attribute_listing">
<li>SAF603A: ELECTRICAL SAFETY AWARENESS</li>
</ul>
</fieldset>
<fieldset>
<legend>Distribution/Notification</legend>
<ul class="lesson_attribute_listing">
<li>*Division Safety Officers (DSOs)</li>
<li>*Safety Wardens</li>
<li>*DOE Notification</li>
<li>*ESH&Q Liaisons</li>
</ul>
</fieldset>
<!--
<fieldset>
<legend>Applicable Orgs</legend>
<ul class="lesson_attribute_listing">
</ul>
</fieldset>
-->
</div>
</td>
</tr>
</table>
<script language="javascript">
var commentTextarea = document.getElementById("commentForm:commentText");
if (commentTextarea){
var validationErrorFound = commentTextarea.parentNode.innerHTML.indexOf("Validation Error: Value is required.") > -1;
if (validationErrorFound){
// add hash if necessary
if (window.location.hash == null || window.location.hash == "" || window.location.hash == "#"){
window.location.hash = "#comments";
}
}
else {
//JSF likes to hold on to the values entered into the form...so blank them out
commentTextarea.value = "";
commentTextarea.form.elements["commentForm:commentType"].selectedIndex = 0;
var numStars = commentTextarea.form.elements["commentForm:stars"].length;
for (var ii=0; ii<numStars; ii++){
commentTextarea.form.elements["commentForm:stars"][ii].checked = false;
}
}
}
</script>
</script>
</div>
<br>
<br>
</div>
</body>
</html>
<br>
<br>
Do you have a lesson learned you would like to contribute? Contact the <a href="mailto:mbailey@jlab.org,sjsmith@jlab.org,cjohnson@jlab.org">Lesson Learned Coordinator</a>.