<html><head><style type="text/css">
/*
Copyright (c) 2011, Yahoo! Inc. All rights reserved.
Code licensed under the BSD License:
http://developer.yahoo.com/yui/license.html
version: 2.9.0
*/
html{color:#000;background:#FFF}body,div,dl,dt,dd,ul,ol,li,h1,h2,h3,h4,h5,h6,pre,code,form,fieldset,legend,input,button,textarea,select,p,blockquote,th,td{margin:0;padding:0}table{border-collapse:collapse;border-spacing:0}fieldset,img{border:0}address,button,caption,cite,code,dfn,em,input,optgroup,option,select,strong,textarea,th,var{font:inherit}del,ins{text-decoration:none}li{list-style:none}caption,th{text-align:left}h1,h2,h3,h4,h5,h6{font-size:100%;font-weight:normal}q:before,q:after{content:''}abbr,acronym{border:0;font-variant:normal}sup{vertical-align:baseline}sub{vertical-align:baseline}legend{color:#000}body{font:13px/1.231 arial,helvetica,clean,sans-serif;*font-size:small;*font:x-small}select,input,textarea,button{font:99% arial,helvetica,clean,sans-serif}table{font-size:inherit;font:100%}pre,code,kbd,samp,tt{font-family:monospace;*font-size:108%;line-height:100%}body{text-align:center}#doc,#doc2,#doc3,#doc4,.yui-t1,.yui-t2,.yui-t3,.yui-t4,.yui-t5,.yui-t6,.yui-t7{margin:auto;text-align:left;width:57.69em;*width:56.25em}#doc2{width:73.076em;*width:71.25em}#doc3{margin:auto 10px;width:auto}#doc4{width:74.923em;*width:73.05em}.yui-b{position:relative}.yui-b{_position:static}#yui-main .yui-b{position:static}#yui-main,.yui-g .yui-u .yui-g{width:100%}.yui-t1 #yui-main,.yui-t2 #yui-main,.yui-t3 #yui-main{float:right;margin-left:-25em}.yui-t4 #yui-main,.yui-t5 #yui-main,.yui-t6 #yui-main{float:left;margin-right:-25em}.yui-t1 .yui-b{float:left;width:12.30769em;*width:12.00em}.yui-t1 #yui-main .yui-b{margin-left:13.30769em;*margin-left:13.05em}.yui-t2 .yui-b{float:left;width:13.8461em;*width:13.50em}.yui-t2 #yui-main .yui-b{margin-left:14.8461em;*margin-left:14.55em}.yui-t3 .yui-b{float:left;width:23.0769em;*width:22.50em}.yui-t3 #yui-main .yui-b{margin-left:24.0769em;*margin-left:23.62em}.yui-t4 .yui-b{float:right;width:13.8456em;*width:13.50em}.yui-t4 #yui-main .yui-b{margin-right:14.8456em;*margin-right:14.55em}.yui-t5 .yui-b{float:right;width:18.4615em;*width:18.00em}.yui-t5 #yui-main .yui-b{margin-right:19.4615em;*margin-right:19.125em}.yui-t6 .yui-b{float:right;width:23.0769em;*width:22.50em}.yui-t6 #yui-main .yui-b{margin-right:24.0769em;*margin-right:23.62em}.yui-t7 #yui-main .yui-b{display:block;margin:0 0 1em 0}#yui-main .yui-b{float:none;width:auto}.yui-gb .yui-u,.yui-g .yui-gb .yui-u,.yui-gb .yui-g,.yui-gb .yui-gb,.yui-gb .yui-gc,.yui-gb .yui-gd,.yui-gb .yui-ge,.yui-gb .yui-gf,.yui-gc .yui-u,.yui-gc .yui-g,.yui-gd .yui-u{float:left}.yui-g .yui-u,.yui-g .yui-g,.yui-g .yui-gb,.yui-g .yui-gc,.yui-g .yui-gd,.yui-g .yui-ge,.yui-g .yui-gf,.yui-gc .yui-u,.yui-gd .yui-g,.yui-g .yui-gc .yui-u,.yui-ge .yui-u,.yui-ge .yui-g,.yui-gf .yui-g,.yui-gf .yui-u{float:right}.yui-g div.first,.yui-gb div.first,.yui-gc div.first,.yui-gd div.first,.yui-ge div.first,.yui-gf div.first,.yui-g .yui-gc div.first,.yui-g .yui-ge div.first,.yui-gc div.first div.first{float:left}.yui-g .yui-u,.yui-g .yui-g,.yui-g .yui-gb,.yui-g .yui-gc,.yui-g .yui-gd,.yui-g .yui-ge,.yui-g .yui-gf{width:49.1%}.yui-gb .yui-u,.yui-g .yui-gb .yui-u,.yui-gb .yui-g,.yui-gb .yui-gb,.yui-gb .yui-gc,.yui-gb .yui-gd,.yui-gb .yui-ge,.yui-gb .yui-gf,.yui-gc .yui-u,.yui-gc .yui-g,.yui-gd .yui-u{width:32%;margin-left:1.99%}.yui-gb .yui-u{*margin-left:1.9%;*width:31.9%}.yui-gc div.first,.yui-gd .yui-u{width:66%}.yui-gd div.first{width:32%}.yui-ge div.first,.yui-gf .yui-u{width:74.2%}.yui-ge .yui-u,.yui-gf div.first{width:24%}.yui-g .yui-gb div.first,.yui-gb div.first,.yui-gc div.first,.yui-gd div.first{margin-left:0}.yui-g .yui-g .yui-u,.yui-gb .yui-g .yui-u,.yui-gc .yui-g .yui-u,.yui-gd .yui-g .yui-u,.yui-ge .yui-g .yui-u,.yui-gf .yui-g .yui-u{width:49%;*width:48.1%;*margin-left:0}.yui-g .yui-g .yui-u{width:48.1%}.yui-g .yui-gb div.first,.yui-gb .yui-gb div.first{*margin-right:0;*width:32%;_width:31.7%}.yui-g .yui-gc div.first,.yui-gd .yui-g{width:66%}.yui-gb .yui-g div.first{*margin-right:4%;_margin-right:1.3%}.yui-gb .yui-gc div.first,.yui-gb .yui-gd div.first{*margin-right:0}.yui-gb .yui-gb .yui-u,.yui-gb .yui-gc .yui-u{*margin-left:1.8%;_margin-left:4%}.yui-g .yui-gb .yui-u{_margin-left:1.0%}.yui-gb .yui-gd .yui-u{*width:66%;_width:61.2%}.yui-gb .yui-gd div.first{*width:31%;_width:29.5%}.yui-g .yui-gc .yui-u,.yui-gb .yui-gc .yui-u{width:32%;_float:right;margin-right:0;_margin-left:0}.yui-gb .yui-gc div.first{width:66%;*float:left;*margin-left:0}.yui-gb .yui-ge .yui-u,.yui-gb .yui-gf .yui-u{margin:0}.yui-gb .yui-gb .yui-u{_margin-left:.7%}.yui-gb .yui-g div.first,.yui-gb .yui-gb div.first{*margin-left:0}.yui-gc .yui-g .yui-u,.yui-gd .yui-g .yui-u{*width:48.1%;*margin-left:0}.yui-gb .yui-gd div.first{width:32%}.yui-g .yui-gd div.first{_width:29.9%}.yui-ge .yui-g{width:24%}.yui-gf .yui-g{width:74.2%}.yui-gb .yui-ge div.yui-u,.yui-gb .yui-gf div.yui-u{float:right}.yui-gb .yui-ge div.first,.yui-gb .yui-gf div.first{float:left}.yui-gb .yui-ge .yui-u,.yui-gb .yui-gf div.first{*width:24%;_width:20%}.yui-gb .yui-ge div.first,.yui-gb .yui-gf .yui-u{*width:73.5%;_width:65.5%}.yui-ge div.first .yui-gd .yui-u{width:65%}.yui-ge div.first .yui-gd div.first{width:32%}#hd:after,#bd:after,#ft:after,.yui-g:after,.yui-gb:after,.yui-gc:after,.yui-gd:after,.yui-ge:after,.yui-gf:after{content:"";display:block;clear:both}#hd,#bd,#ft,.yui-g,.yui-gb,.yui-gc,.yui-gd,.yui-ge,.yui-gf{zoom:1}/*
Copyright (c) 2011, Yahoo! Inc. All rights reserved.
Code licensed under the BSD License:
http://developer.yahoo.com/yui/license.html
version: 2.9.0
*/
body{margin:10px}h1{font-size:138.5%}h2{font-size:123.1%}h3{font-size:108%}h1,h2,h3{margin:1em 0}h1,h2,h3,h4,h5,h6,strong,dt{font-weight:bold}optgroup{font-weight:normal}abbr,acronym{border-bottom:1px dotted #000;cursor:help}em{font-style:italic}del{text-decoration:line-through}blockquote,ul,ol,dl{margin:1em}ol,ul,dl{margin-left:2em}ol{list-style:decimal outside}ul{list-style:disc outside}dl dd{margin-left:1em}th,td{border:1px solid #000;padding:.5em}th{font-weight:bold;text-align:center}caption{margin-bottom:.5em;text-align:center}sup{vertical-align:super}sub{vertical-align:sub}p,fieldset,table,pre{margin-bottom:1em}button,input[type="checkbox"],input[type="radio"],input[type="reset"],input[type="submit"]{padding:1px}img{-ms-interpolation-mode:bicubic}/* The item_header styles define the strapline across the top of the
individual entry pages */
.item_header {
border: 1px solid;
margin-bottom: 2px;
margin-top: 2px;
padding-top: 1px;
padding-bottom: 1px;
height: Auto;
}
.item_header .leftmost{
margin-right: 2em;
margin-left: 0.5em;
color: #6889AE;
font-size: 1.4em;
}
.item_header a {
color: #6889AE;
margin-right: 1em;
}
#logbooks_str a {
color: #6889AE;
margin: 0;
font-size: 1.4em;
}
.item_header .elog_id {
margin-left: 0.5em;
display:inline-table;
width: 200px;
color: #6889AE ;
font-weight: bold;
font-size: 1.4em;
text-align: left;
}
.item_header a {
display: inline;
}
.item_header .title {
margin-right: 2em;
margin-left: 0.5em;
color: #6889AE;
font-weight: bold;
font-size: 1.4em;
text-align: left;
}
.item_header .datefield{
margin-left: 0.5em;
display:inline-table;
width: 200px;
color: #6889AE;
font-size: 1.4em;
clear: none;
width: 200px;
}
.item_header table,td{
border: 0;
padding: 2px;
}
div.elog_body {
clear: both;
}
.elog_body pre {
line-height: 18.2px;
font-size:108%;
font-style:normal;
}
.downtime_table {
float:left;
width: auto;
text-align: left;
/*margin:5px auto;*/
margin: 2em 0;
padding:2px;
border: 1px solid;
vertical-align: middle;
}
.downtime_table th{
padding-right: 0.5em;
font-weight: bold;
text-align: center;
}
.downtime_table td.dt1{
padding: 0.5em;
font-weight: normal;
text-align: center;
border: 1px solid;
}
.downtime_table td.dt2{
padding-top: 0.5em;
padding-left:2px;
font-weight: normal;
text-align: left;
border: none;
}
div.opspr_history{
margin-top: 1em;
}
table.opspr_line_item{
margin:0;
background-color: white;
width: 100%;
padding: 0;
}
ul.floatlinks{
float:right;
font-weight:bold;
margin:0.3em;
clear:both;
}
.floatlinks li{
float:right;
font-weight:bold;
margin:0.3em;
clear:both;
list-style-type:none;
}
</style>
</head><body class="yui-skin-sam">
<div id="doc3" class="yui-t7">
<div id="bd">
<div id="yui-main">
<div class="yui-b">
<div id="elog_item">
<div class=one_item><div class=item_header><table><tr><td><span class=leftmost><a href="http://opsweb.acc.jlab.org/CSUEApps/elog/entry/1787277">1787277</a></span></td><td colspan=2><span class=title>IOC failure and recovery summary</span></td></tr><tr><td width=200px;><span class=datefield>25-May-2012 11:30</span></td><td width=350px;><span class=leftmost>T_Mcguckin|P_Kjeldsen|S_Higgins|S_Windham</span></td><td><span id=logbooks_str><a href=http://opsweb.acc.jlab.org/CSUEApps/elog/elog.php?logbook=SLOG>SLOG</a> | <a href=http://opsweb.acc.jlab.org/CSUEApps/elog/elog.php?logbook=PEPPOLOG>PEPPOLOG</a> | <a href=http://opsweb.acc.jlab.org/CSUEApps/elog/elog.php?logbook=ELOG>ELOG</a> | <a href=http://opsweb.acc.jlab.org/CSUEApps/elog/elog.php?logbook=ACELOG>ACELOG</a></span></td></tr></table>
</div>
<ul class=floatlinks>
<li>Elog Backlink: <a href=http://opsweb.acc.jlab.org/CSUEApps/elog/entry/1787278>Numerous iocs in the injector, MCC & for PSS readbacks are unresolved.</a></li></ul><div class=elog_body><pre wrap>Software on-call got called in because injector IOCs were not responding (iocin3, iocpss, etc.). SysAdmin on-call was called in shortly afterwards.
Had a lot of trouble tracking down source of problem due to lack of diagnostic tools (most notably Wireshark, to look at network packet traffic). In process of troubleshooting rebooted opsrv and opsbat3 with no effect. CDEV name server had to be restarted on opsrv.
Finally were able to get Wireshark setup on an OPS machine to look at network traffic at about 1100 and found iocse4 was screaming across network and causing a network storm. Iocse4 had been experiencing errors in its hourly save files since 5/18. Once this IOC was rebooted recovery could begin.
Wireshark has been installed on opsl00 (yum install wireshark-gnome) for future use.
Weiwei restarted CDEV name server on opsrv.
iocse6 and iocse13 are still down, but not needed for PEPPO run. Software on-call and Suhring are aware of issues with these IOCs.
</pre></div></div></div></div></div></div></div></body>