david.barker@LIGO.ORG - posted 19:00, Thursday 23 January 2020 - last comment - 22:33, Thursday 23 January 2020(54682)
h1pslctrl0 computer not accessible
Ed, Jason, Dave:
It looks like the Beckhoff OPC computer in the diode room is not accessible over the network. This machine is called h1pslctrl0.
At first we wondered if the CER network switch had failed, but the digital cameras are working, only the diode room computer is down.
Jason is investigating remotely and Ed is going into the diode room to check on the computer.
opened FRS14121
Ed found h1pslctrl0 powered up with a system message on the console. It problem started at 18:11 PST (02:11 Friday UTC).
This is the running log I had started:
02:12 Intenion Bit Commissioning - 357 EPICS channels are lost
From Google translate:
"Twin CAT OPC Server has encountered a problem and needs to close
If you haven't saved your work yet, data may be lost.
Please also report this problem to Microsoft
A problem report that you can send us has been created. We will process this report confidentially and anonymously.
To see what data the report contains, click here"
The LASER_PWR node could not connect to a channel that I thought that we had removed back on Dec. 4th (Camilla alog53682), and I confirmed that it is not in the current userapps code. I can't tell at the moment, but perhaps this node just hasn't ever been reloaded since then to take in the new version of the code that no longer uses that channel for this exact reason.
I had Ed check that a reload happened tonight and we will see if this connection error comes back.
To confirm this does not look like a network switch problem. At this time all digital video cameras are working, h1pslctrl0 is responding to pings but its IOC is not responding to channel connection requests. It would appear the problem is with the twincat/IOC software on h1pslctrl0 and is intermittent.
Earlier this evening in my first attempt to get guardian to ignore the LASER_PWR errors, I added this node to the exceptions list in sys/h1/guardian/IFO_NODE_LIST.py and reloaded the IFO node. Now that we think we have correctly deleted the link between guardian and h1pslctrl0, I have undone my change by reverting IFO_NODE_LIST.py and reloading the IFO node.
Tomorrow we will check that the LASER_PWR node had its code changed to remove the AMP_PWR4 channel but this was waiting for a reload.
While we are having connection problems with h1pslctrl0, the EDC on the CDS overview will continue to show 357 disconnected channels. If this number is larger then something else has failed.
R. Savage, J. Oberling, T. Shaffer, D. Barker, E. Merilh
The PSL Beckhoff PC (h1pslctrl0) is running normally and the laser is running normally; the laser control program is operating as usual and I can manipulate it (change menus, turn watchdogs off/on). It looks like a failure of the OPC IOC Shell that sends the PSL Beckhoff channels to Epics. We cleared the error message and shut down the OPC server; at this point it appears the computer became pingable (it wasn't previously) and we can log into it remotely (which we couldn't previously). We need to restart the OPC server, but noticed that the keyboard on the Beckhoff PC is frozen; it's an old PS/2 connected keyboard so we can't simply unplug it and plug it back in (PS/2 connections aren't hot swappable), it would require a restart of the PC. At this point we decided to wait until the next target of opportunity (lockloss, earthquake, etc) to attempt to restart the OPC IOC shell (if the restart causes further problems, we would rather be fresh to deal with it than tired). Due to this, we will have no Epics monitoring or trending of the laser-based PSL channels until this restart takes place; PMC, ISS, and FSS are all run directly through Epics and not the PSL Beckhoff PC, so are available.
The issue stopping the IFO from going into Observe had to do with a failure of the Laser_Pwr guardian node due to the failure of the OPC IOC shell; the node was complaining that it lost connection to H1:PSL-AMP_PWR4, which comes into Epics via the PSL OPC IOC shell (it used to look at this channel to confirm if the laser was running). TJ, live from Pasadena, looked through the Laser_Pwr node and saw no reference to this channel in the code. At this point our theory is that the old version of the Laser_Pwr guardian node, looking at H1:PSL-AMP_PWR4, was running in this node. Back in December Camilla had edited the node to read H1:PSL-PWR_HPL_DC_LP_OUTPUT (which is not read through the PSL Beckhoff PC); this was done to avoid just this situation. To the best we can tell, it appears the guardian node wasn't reloaded at the next available opportunity and was therefore still running the old code. Once it was running the new code, the Laser_Pwr node worked without problem and the IFO had no issues returning to NLN.