Note to Operators:

for this weekend we are running the PSL Diode room OPC Beckhoff computer (h1pslctrl0) in a reduced state; its EPICS IOC is not running. Since you now cannot use MEDM to determine if the computer is operational, I've added a ping check to the CDS Hardware Status EPICS. This will go RED if the computer is not pingable, in which case the Errors string will report 'h1pslctrl0 is down' on the HW STATUS MEDM.

If this error is seen, please contact Jason.

Attached snapshot shows the HW status LED at the bottom of the CDS Overview, and the STATUS MEDM which opened by pressing the 'HW' button.

Notes after locking (this was a mid-shift post I forgot to post.  Ha.):

• Had a large EQ (6.7 Turkey) take down all three detectors; H1 down at 10:21am LOCAL. 
• Target Of Opportunity Window:  10:22 - 11:22 (for OMC dark meas, PSL computer retrieval, Forklifting around LVEA)
• 11:22 Relocking started.
• 12:00 Back to OBSERVING.
• 11:40 Corner station winds start to hover around 20mph.

opened FRS14121

Ed found h1pslctrl0 powered up with a system message on the console. It problem started at 18:11 PST (02:11 Friday UTC).

This is the running log I had started:

02:12 Intenion Bit Commissioning - 357 EPICS channels are lost

• H1 still locked at Nominal Low Noise and   and showing good range
  • Observatory mode switched to
• Dave B. contacted.
  • Suspicions of crashed network switch ruled out because digital cameras still active
  • All channels are PSL related
• Jason O. contacted
  • He is currently looking into the issue remotely
    • Neither of them can log in (see image)

From Google translate:

"Twin CAT OPC Server has encountered a problem and needs to close

If you haven't saved your work yet, data may be lost.

Please also report this problem to Microsoft

A problem report that you can send us has been created. We will process this report confidentially and anonymously.

To see what data the report contains, click here"

The LASER_PWR node could not connect to a channel that I thought that we had removed back on Dec. 4th (Camilla alog53682), and I confirmed that it is not in the current userapps code. I can't tell at the moment, but perhaps this node just hasn't ever been reloaded since then to take in the new version of the code that no longer uses that channel for this exact reason. 

I had Ed check that a reload happened tonight and we will see if this connection error comes back.

To confirm this does not look like a network switch problem. At this time all digital video cameras are working, h1pslctrl0 is responding to pings but its IOC is not responding to channel connection requests. It would appear the problem is with the twincat/IOC software on h1pslctrl0 and is intermittent.

Earlier this evening in my first attempt to get guardian to ignore the LASER_PWR errors, I added this node to the exceptions list in sys/h1/guardian/IFO_NODE_LIST.py and reloaded the IFO node. Now that we think we have correctly deleted the link between guardian and h1pslctrl0, I have undone my change by reverting IFO_NODE_LIST.py and reloading the IFO node.

Tomorrow we will check that the LASER_PWR node had its code changed to remove the AMP_PWR4 channel but this was waiting for a reload.

While we are having connection problems with h1pslctrl0, the EDC on the CDS overview will continue to show 357 disconnected channels. If this number is larger then something else has failed.

R. Savage, J. Oberling, T. Shaffer, D. Barker, E. Merilh

The PSL Beckhoff PC (h1pslctrl0) is running normally and the laser is running normally; the laser control program is operating as usual and I can manipulate it (change menus, turn watchdogs off/on).  It looks like a failure of the OPC IOC Shell that sends the PSL Beckhoff channels to Epics.  We cleared the error message and shut down the OPC server; at this point it appears the computer became pingable (it wasn't previously) and we can log into it remotely (which we couldn't previously).  We need to restart the OPC server, but noticed that the keyboard on the Beckhoff PC is frozen; it's an old PS/2 connected keyboard so we can't simply unplug it and plug it back in (PS/2 connections aren't hot swappable), it would require a restart of the PC.  At this point we decided to wait until the next target of opportunity (lockloss, earthquake, etc) to attempt to restart the OPC IOC shell (if the restart causes further problems, we would rather be fresh to deal with it than tired).  Due to this, we will have no Epics monitoring or trending of the laser-based PSL channels until this restart takes place; PMC, ISS, and FSS are all run directly through Epics and not the PSL Beckhoff PC, so are available.

The issue stopping the IFO from going into Observe had to do with a failure of the Laser_Pwr guardian node due to the failure of the OPC IOC shell; the node was complaining that it lost connection to H1:PSL-AMP_PWR4, which comes into Epics via the PSL OPC IOC shell (it used to look at this channel to confirm if the laser was running).  TJ, live from Pasadena, looked through the Laser_Pwr node and saw no reference to this channel in the code.  At this point our theory is that the old version of the Laser_Pwr guardian node, looking at H1:PSL-AMP_PWR4, was running in this node.  Back in December Camilla had edited the node to read H1:PSL-PWR_HPL_DC_LP_OUTPUT (which is not read through the PSL Beckhoff PC); this was done to avoid just this situation. To the best we can tell, it appears the guardian node wasn't reloaded at the next available opportunity and was therefore still running the old code.  Once it was running the new code, the Laser_Pwr node worked without problem and the IFO had no issues returning to NLN.

Here is a sketch of the wind fence anemometer locations wrt the fence and the EndX building.

Also there are a couple photos of the cable route up the slope through the tumbleweeds (they are over head height) and of the sensors with the fence.