Actually all the front-end models are on a single computer.  Either the power for the I/O chassis has tripped, or the communications was dropped for a few cycles. If you reboot the PSL front-end computer (h1psl0), that would fix the latter.  If that fails, you need to power down the PSL front-end computer, then power-cycle the I/O chassis (resetting DC power supply if needed), then booting the PSL front-end computer.
   If that fails, then we need more investigation.

I have Jim Batch on the phone to troubleshoot

The IOP chassis was not visible from the computer, so I remotely removed the h1psl0 computer from the dolphin network and powered down the computer.  Ed power cycled the I/O chassis and powered the computer back on.  The cards in the I/O chassis are now visible.  Timing on the h1ioppsl0 model has gone "negative", meaning the DAQ data is bad until it works it's way back positive. But the PSL computer show be able to control the PSL OK.  

There is no known cause for this event.

trending h1ioppsl0's state_word shows that at 08:44 PST the FE and ADC bits faulted. It looks like a timing issue which glitched the IOP into a non-recoverable state.

IRIG-B is still in a slow recovery, DAQ data will be invalid until it completes.

Just spoke to Ed and Jonathan.  The flow rate now looks green.  It's possible that this has been triggered by other computers going down.  I can't get remote read-only MEDM access to check the status of the system.  Ed will call me if they can't key the lasers back on at the controller.

Ed and Dave got my remote access turned on so I could log in and take a look.  It appears both TCS lasers are in a good state but are not outputing power.  Power supply voltage is good.  Flow rate is good.  Temperature is good.  Current is around 0.7A indicating that they are getting RF and are ready to laze.  The Beckhoff screen shows red indicators on several read back channels, and it looks like this may be the cause of the lasers not coming on.

I've spoken to Dave and Ed again and they are going to take it from here since it looks like a Beckhoff issue. I'll be available by phone if needed.  The laser chassis will probably need to be re-keyed off and on once this has been fixed.

16:22 Also noticed that tw0 is "white". Not certain at this time how long it has been that way

h1tw0 is currently down while we are rebuilding its raid.

Actually this looks more like a conversion error between signed and unsigned words for the Fahrenheit readbacks. If you subtract 2^16/10 from these numbers, it makes sense! Filed FRS 6963.

Updated plot for the past 2 hours.  Continuing to "trend" downward.  Looks a little suspicious how the trend smoothed out around 37.

This EQ was large enough (10^2 um/s in the EQ band according to the BLRMS) that it tripped every ISI platform and several SUSes.  I switched ISI_CONF to LARGE_EQ_NOBRSXY, but it didn't seem to help the end station ISIs.

This was just over a 9 hour lock.