[Ace] IOC Console Lockups [was Re: IOC Channel Access]

[Brian Bevins]

Currently iocin7, iocmodiag, and iocnl3 are all the state where the 
console is non-functional but they are pingable and reachable by channel 
access. Any diagnostics we can try?


On 05/22/17 16:05, Brian Bevins wrote:
> I think it's very likely that this is frequently an ioc issue, and I 
> don't advocate rushing anything out to address it that might weaken 
> security. I had hoped that there was a port-by-port control that would 
> allow resetting the serial bus, in the hope that it would be easy to 
> expose and helpful.
>
> I am fine with monitoring this for now and trying to characterize how 
> often it happens. It's not unlikely that I see the same hung consoles 
> over and over when I'm routinely connecting to them in large numbers.
>
> It might be possible ti implement something in the system database 
> that would allow triggering a serial reset from the ioc end via 
> channel access, if it is often the ioc that is hung. Once the ioc and 
> the console server aren't talking, it's hard to diagnose what's going 
> on without starting to just reboot things.
>
> --Brian
>
> On 05/22/17 13:22, Anthony Cuffe wrote:
>> Brian,
>>
>> I don't think that this is always the console server itself. Haven't 
>> some of the IOCs exhibited behaviour where their console is hung by 
>> some previous process like the iocaudit utility and the only way to 
>> resolve it is to reboot the IOC?
>>
>> We are not against creating a mechanism to do reboot the console 
>> servers.  It is just not a trivial change.  We will have to enable a 
>> login shell for the vxuser account.  I want to look over this 
>> configuration before pushing it out everywhere.
>>
>>
>> +-------------------------+
>> | Anthony Cuffe           |
>> | voice  : 757 269-6213   |
>> | e-mail : cuffe at jlab.org |
>> +-------------------------+
>>
>> On 05/22/2017 12:20 PM, Brian Bevins wrote:
>>> A hung console server is actually a pretty frequent event. Usually 
>>> we work around it, by power cycling iocs vs. soft reboots, using 
>>> telnet, etc.
>>>
>>> When I run tools that go out and touch all the iocs, there are 
>>> always hung consoles.
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Ace mailing list
>>> Ace at jlab.org
>>> https://mailman.jlab.org/mailman/listinfo/ace
>>>
>
>


-- 
Brian S. Bevins, PE
Computer Scientist / Mechanical Engineer
Thomas Jefferson National Accelerator Facility